zoukankan      html  css  js  c++  java
  • flume 容错和负载均衡3

    集群采用3台机器:

    host1 load-balance设置  host2 机器1  host3 机器2


    其中,host1 机器配置:

    #Define a memory channel called c1 on a1  
    a1.channels = c1
    a1.sources = r1
    a1.sinks = k1 k2
    a1.sinkgroups = g1
    a1.sinkgroups.g1.sinks = k1 k2
    a1.sinkgroups.g1.processor.type = load_balance
    a1.sinkgroups.g1.processor.selector = round_robin
    a1.sinkgroups.g1.processor.backoff = true
    a1.channels.c1.type = file
    a1.channels.c1.checkpointDir = /tmp/flume/loadcheckpoint
    a1.channels.c1.dataDirs = /tmp/flume/loaddata
    a1.sources.r1.channels = c1
    a1.sources.r1.type = avro
    a1.sources.r1.bind = 0.0.0.0
    a1.sources.r1.port = 41415

    a1.sinks.k1.channel = c1
    a1.sinks.k1.type = avro
    a1.sinks.k1.hostname = host2
    a1.sinks.k1.port = 41414

    a1.sinks.k2.channel = c1
    a1.sinks.k2.type = AVRO
    a1.sinks.k2.hostname = host3
    a1.sinks.k2.port = 41414


    host2 机器配置:

    a2.channels = c1  
    a2.sources = r1
    a2.sinks = k1
    a2.channels.c1.type = FILE
    a1.channels.c1.checkpointDir = /tmp/flume/checkpoint
    a1.channels.c1.dataDirs = /tmp/flume/data
    a2.sources.r1.channels = c1
    a2.sources.r1.type = AVRO
    a2.sources.r1.bind = 0.0.0.0
    a2.sources.r1.port = 41414
    a2.sinks.k1.channel = c1
    a2.sinks.k1.type = file_roll
    a2.sinks.k1.sink.directory = /tmp/load/
    a2.sinks.k1.sink.rollInterval = 0

    host3 机器配置:

    a2.channels = c1  
    a2.sources = r1
    a2.sinks = k1
    a2.channels.c1.type = FILE
    a1.channels.c1.checkpointDir = /tmp/flume/checkpoint
    a1.channels.c1.dataDirs = /tmp/flume/data
    a2.sources.r1.channels = c1
    a2.sources.r1.type = AVRO
    a2.sources.r1.bind = 0.0.0.0
    a2.sources.r1.port = 41414
    a2.sinks.k1.channel = c1
    a2.sinks.k1.type = file_roll
    a2.sinks.k1.sink.directory = /tmp/load/
    a2.sinks.k1.sink.rollInterval = 0


    客户端机器暂时使用flume-ng的agent发送,配置如下:

    # Define a memory channel called c1 on a1  
    a1.channels.c1.type = file
    a1.channels.c1.checkpointDir = /tmp/flume/checkpoint
    a1.channels.c1.dataDirs = /tmp/flume/data
    a1.sources.r1.channels = c1
    a1.sources.r1.type = exec
    a1.sources.r1.command = cat /tmp/linux.log
    a1.sinks.k1.type = avro
    a1.sinks.k1.channel = c1
    a1.sinks.k1.hostname = host1
    a1.sinks.k1.port = 41415
    a1.channels = c1
    a1.sources = r1
    a1.sinks = k1


    客户端/tmp/linux.log文件3G左右,发送给host1。

    启动:

    host2:

    bin/flume-ng agent -c conf -f conf/load-sink1.conf -n a2

    host3:

    bin/flume-ng agent -c conf -f conf/load-sink2.conf -n a2

    host1:

    bin/flume-ng agent -c conf -f conf/load-balance.conf -n a1

    客户端:

    bin/flume-ng agent -c conf -f conf/client.conf -n a1

    注意:启动方式最好是从下往上启动。即:先启动host2和host3,然后启动host1,最后启动client。

    测试过程中,可以随时将host2或host3停止,过一段时间再启动。

    这样,就测试了flume-ng的load-balance和failover功能。

  • 相关阅读:
    在dll里malloc/new/cvCreate分配内存,在exe里free/Releases释放内存时会出错。
    其原因可能是堆被损坏,这说明 100BloodCellSegTest.exe 中或它所加载的任何 DLL 中有 Bug。
    牛顿方法(Newton's Method)
    广义线性模型(Generalized Linear Models)
    逻辑回归(Logistic Regression)
    局部加权回归、欠拟合、过拟合(Locally Weighted Linear Regression、Underfitting、Overfitting)
    java并发编程实战《三》互斥锁(上)
    关于java链接装载的思考
    java并发编程实战《二》java内存模型
    (转)技术人的王者路,看看你在哪个段位?
  • 原文地址:https://www.cnblogs.com/wjsshide/p/5032781.html
Copyright © 2011-2022 走看看