04_Flume多节点load_balance实践

1、负载均衡场景

 1）初始：上游Agent通过round_robin selector, 将event轮流发送给下游Collecotor1, Collector2

 2）故障： 关闭Collector1进程来模拟故障，Agent1由于配置了backoff, 会将Collecotor1暂时从发送列表中移除，event全部发送给Collector2

3） 恢复： 重启Collector1进程， Collector1在经历最大timeout后，重新进入发送列表；之后的event再次分发给Collector1/2

2、节点配置

2.1 上游Agent的flume配置

# 05-flume-loadbalance-client
# agent name: a1
# source: exec
# channel: memory
# sink: k1 k2, each set to avro type to link to next-level collector


# 01 define source,channel,sink name
a1.sources = r1
a1.channels = c1
a1.sinks = k1 k2

# 02 define source
a1.sources.r1.type = exec
a1.sources.r1.command = tail -f /root/flume_test/server.log


# 03 define sink,each connect to next-level collector via hostname and port
a1.sinks.k1.type = avro
a1.sinks.k1.hostname = slave1   # 上游avro sink绑定到下游主机，RPC
a1.sinks.k1.port = 4444

a1.sinks.k2.type = avro
a1.sinks.k2.hostname = slave2   # 上游avro sink绑定到下游主机, PRC
a1.sinks.k2.port = 4444

# 04 define sinkgroups, sink will be seleced for event distribution based on selecotr 
a1.sinkgroups = g1
a1.sinkgroups.g1.sinks = k1 k2

a1.sinkgroups.g1.processor.type = load_balance
a1.sinkgroups.g1.processor.selector = round_robin

# 节点失效，则将节点从sinkgroup中移除一段时间
a1.sinkgroups.g1.processor.backoff = true
# 将节点从sinkgroups中移除的时间，millisecond
# 节点被暂时移除，selector就不会尝试向节点发送数据，能一定程度提高event分发速度，但event可能会分发的不均衡
a1.sinkgroups.g1.processor.selector.maxTimeOut = 10000


# 05 define channel 
a1.channels.c1.type = memory
# number of events in memory queue 
a1.channels.c1.capacity = 1000  
# number of events for 1 commit(commit events to memory queue)
a1.channels.c1.transactioncapacity = 100

# 06 bind source,sink to channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1
a1.sinks.k2.channel = c1

2.2 下游Collector1的flume配置

# 01 specify agent,source,sink,channel
a1.sources = r1
a1.sinks = k1
a1.channels = c1

# 02 avro source,connect to local port 4444
a1.sources.r1.type = avro         # 下游avro source绑定到本机端口，端口要和上游Agent中的配置值保持一致
a1.sources.r1.bind = slave1
a1.sources.r1.port = 4444

# 03 logger sink
a1.sinks.k1.type = logger 

# 04 channel,memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

# 05 bind source,sink to channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

2.3 下游Collecotor2的flume配置

# 01 specify agent,source,sink,channel
a1.sources = r1
a1.sinks = k1
a1.channels = c1

# 02 avro source,connect to local port 4444 
a1.sources.r1.type = avro         # 下游avro source绑定到本机端口，端口要和上游Agent中的配置值保持一致
a1.sources.r1.bind = slave2
a1.sources.r1.port = 4444

# 03 logger sink
a1.sinks.k1.type = logger 

# 04 channel,memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

# 05 bind source,sink to channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

3、启动各个节点上的flume agent

启动Collector1

# ./bin/flume-ng agent --conf conf --conf-file ./conf/flume-failover-server.properties --name a1 -Dflume.root.logger=INFO,console

启动Collector2

# ./bin/flume-ng agent --conf conf --conf-file ./conf/flume-failover-server.properties --name a1 -Dflume.root.logger=INFO,console

启动上游的Agent

# ./bin/flume-ng agent --conf conf --conf-file ./conf/flume-loadbalance-client.properties --name a1 -Dflume.root.logger=INFO,console

注意：需要先将下游的Collector节点启动，再启动Agent；否则Agent启动，但下游Collector没有启动，Agent会发现没有可用的下游节点，从而产生报错

4、故障模拟

1） 故障前，向Agent所在机器的log文件，通过管道的形式追加数据，看看event是否轮询的发往了Collector1, Collecotor2

Agent上追加如下数据

Collector1接收并打印到Console的Event    （2,4,7）

Collector2接收并打印到Console的Event   （1,4,5,6,8）

总结： Flume的round_robin分发，如果是小测试集，分发结果并不是严格的round_robin. 会出现某些节点被分发的次数多，某些节点被分发的次数少的情况

 2）模拟故障，将Collector1的进程kill

 3）再次在Agent上进行数据追加，查看此时event是否全部分发给Collector2

Collector2此时接收全部event, 并打印到Console

 注意1个细节

 当Collector1故障的时候，Agent发送event时会提示1个Sink不可用，并尝试下一个Sink进行Event发送

4） 恢复Collector1， 查看Event此时的分发结果

Agent上追加数据

Collector1分发得到的数据

Collector2分发得到的数据

5、负载均衡场景下的官方配置参考