zoukankan      html  css  js  c++  java
  • flume 容错和负载均衡3

    集群采用3台机器:

    host1 load-balance设置  host2 机器1  host3 机器2


    其中,host1 机器配置:

    #Define a memory channel called c1 on a1  
    a1.channels = c1
    a1.sources = r1
    a1.sinks = k1 k2
    a1.sinkgroups = g1
    a1.sinkgroups.g1.sinks = k1 k2
    a1.sinkgroups.g1.processor.type = load_balance
    a1.sinkgroups.g1.processor.selector = round_robin
    a1.sinkgroups.g1.processor.backoff = true
    a1.channels.c1.type = file
    a1.channels.c1.checkpointDir = /tmp/flume/loadcheckpoint
    a1.channels.c1.dataDirs = /tmp/flume/loaddata
    a1.sources.r1.channels = c1
    a1.sources.r1.type = avro
    a1.sources.r1.bind = 0.0.0.0
    a1.sources.r1.port = 41415

    a1.sinks.k1.channel = c1
    a1.sinks.k1.type = avro
    a1.sinks.k1.hostname = host2
    a1.sinks.k1.port = 41414

    a1.sinks.k2.channel = c1
    a1.sinks.k2.type = AVRO
    a1.sinks.k2.hostname = host3
    a1.sinks.k2.port = 41414


    host2 机器配置:

    a2.channels = c1  
    a2.sources = r1
    a2.sinks = k1
    a2.channels.c1.type = FILE
    a1.channels.c1.checkpointDir = /tmp/flume/checkpoint
    a1.channels.c1.dataDirs = /tmp/flume/data
    a2.sources.r1.channels = c1
    a2.sources.r1.type = AVRO
    a2.sources.r1.bind = 0.0.0.0
    a2.sources.r1.port = 41414
    a2.sinks.k1.channel = c1
    a2.sinks.k1.type = file_roll
    a2.sinks.k1.sink.directory = /tmp/load/
    a2.sinks.k1.sink.rollInterval = 0

    host3 机器配置:

    a2.channels = c1  
    a2.sources = r1
    a2.sinks = k1
    a2.channels.c1.type = FILE
    a1.channels.c1.checkpointDir = /tmp/flume/checkpoint
    a1.channels.c1.dataDirs = /tmp/flume/data
    a2.sources.r1.channels = c1
    a2.sources.r1.type = AVRO
    a2.sources.r1.bind = 0.0.0.0
    a2.sources.r1.port = 41414
    a2.sinks.k1.channel = c1
    a2.sinks.k1.type = file_roll
    a2.sinks.k1.sink.directory = /tmp/load/
    a2.sinks.k1.sink.rollInterval = 0


    客户端机器暂时使用flume-ng的agent发送,配置如下:

    # Define a memory channel called c1 on a1  
    a1.channels.c1.type = file
    a1.channels.c1.checkpointDir = /tmp/flume/checkpoint
    a1.channels.c1.dataDirs = /tmp/flume/data
    a1.sources.r1.channels = c1
    a1.sources.r1.type = exec
    a1.sources.r1.command = cat /tmp/linux.log
    a1.sinks.k1.type = avro
    a1.sinks.k1.channel = c1
    a1.sinks.k1.hostname = host1
    a1.sinks.k1.port = 41415
    a1.channels = c1
    a1.sources = r1
    a1.sinks = k1


    客户端/tmp/linux.log文件3G左右,发送给host1。

    启动:

    host2:

    bin/flume-ng agent -c conf -f conf/load-sink1.conf -n a2

    host3:

    bin/flume-ng agent -c conf -f conf/load-sink2.conf -n a2

    host1:

    bin/flume-ng agent -c conf -f conf/load-balance.conf -n a1

    客户端:

    bin/flume-ng agent -c conf -f conf/client.conf -n a1

    注意:启动方式最好是从下往上启动。即:先启动host2和host3,然后启动host1,最后启动client。

    测试过程中,可以随时将host2或host3停止,过一段时间再启动。

    这样,就测试了flume-ng的load-balance和failover功能。

  • 相关阅读:
    springboot中多端口启动(这里也适用于https既443端口)
    Gson的fromJson()方法(从Json相关对象到Java实体或转换成List集合)
    [C++] STL源码中学到的 Traits 编程技法的应用
    【解决问题】UNIAPP、VUE 中DATA 数组更新后页面不同步动态渲染更新
    selenium操作chrome时的一些配置
    在react项目中使用fetch 和 JWT进行权限验证(转)
    滚动条的设置(样式包括宽度,颜色等)zhuan
    css3 box-shadow阴影(外阴影与外发光)图示讲解(zhuan)
    js 截断字符串 超过几个字加...
    create-react-app 打包部署
  • 原文地址:https://www.cnblogs.com/wjsshide/p/5032781.html
Copyright © 2011-2022 走看看