zoukankan      html  css  js  c++  java
  • 05_ Flume多级Agent之间串联案例

    多级agent之间串联:

       从tail命令获取数据发送到avro端口,另一个节点可配置一个avro源来获取数据,发送外部存储

    启动两个Agent节点:

      使用scp命令拷贝flume安装包到另一台虚拟机;scp -r /usr/local/src/flume/ shizhan3:$PWD

    1.在shizhan2虚拟上配置采集方案:tail-avro-avro-logger.con 

    ##################
    # Name the components on this agent
    a1.sources = r1
    a1.sinks = k1
    a1.channels = c1
    
    # Describe/configure the source
    a1.sources.r1.type = exec
    a1.sources.r1.command = tail -F /home/hadoop/log/test.log
    a1.sources.r1.channels = c1
    
    # Describe the sink
    #绑定的不是本机, 是另外一台机器的服务地址, sink端的avro是一个发送端, avro的客户端, 往shizhan3这个机器上发
    a1.sinks = k1
    a1.sinks.k1.type = avro
    a1.sinks.k1.channel = c1
    a1.sinks.k1.hostname = shizhan3
    a1.sinks.k1.port = 4141
    a1.sinks.k1.batch-size = 2
    
    # Use a channel which buffers events in memory
    a1.channels.c1.type = memory
    a1.channels.c1.capacity = 1000
    a1.channels.c1.transactionCapacity = 100
    
    # Bind the source and sink to the channel
    a1.sources.r1.channels = c1
    a1.sinks.k1.channel = c1

    2.在shizhan3虚拟上配置采集方案:avro-hdfs.conf(从avro端口接收数据,下沉到logger)

    # Name the components on this agent
    a1.sources = r1
    a1.sinks = k1
    a1.channels = c1
    
    # Describe/configure the source
    #source中的avro组件是接收者服务, 绑定本机
    a1.sources.r1.type = avro
    a1.sources.r1.channels = c1
    a1.sources.r1.bind = 0.0.0.0
    a1.sources.r1.port = 4141
    
    # Describe the sink
    a1.sinks.k1.type = logger
    
    # Use a channel which buffers events in memory
    a1.channels.c1.type = memory
    a1.channels.c1.capacity = 1000
    a1.channels.c1.transactionCapacity = 100
    
    # Bind the source and sink to the channel
    a1.sources.r1.channels = c1
    a1.sinks.k1.channel = c1

    启动shizhan3的Agent采集方案采集数据:将日志信息打印输出到屏幕

      bin/flume-ng agent -c conf -f conf/avro-hdfs.conf -n al -Dflume.root.logger=INFO,console

    往日志文件中不断输入内容,并启动启动shizhan2的Agent采集方案采集数据

      

     

  • 相关阅读:
    使用wps跨ubuntu和windows编辑“.doc”文档。报错“编辑受限”,乱码,软件卡死,强制关闭
    【Python】断言功能Assertion
    【Python】hasattr() getattr() setattr() 使用方法详解
    【python】self & cls
    【Python】2.x与3​​.x版本的选用&版本间的区别
    【python】 del 的用法
    【python】参数中的*args和**kwargs
    解方程AX=b与矩阵分解:奇异值分解 特征值分解 QR分解 三角分解 Cholesky分解
    【python】dist-packages和site-packages的区别
    【python】脚本连续发送QQ邮件
  • 原文地址:https://www.cnblogs.com/yaboya/p/9317187.html
Copyright © 2011-2022 走看看