zoukankan      html  css  js  c++  java
  • flume kafka 配置指南

    1、官方网站也有配置:

     https://flume.apache.org/FlumeUserGuide.html#kafka-source

    2、clodera 官方配置

    https://www.cloudera.com/documentation/kafka/2-0-x/topics/kafka_flume.html

    1

     tier1.sources  = source1
     tier1.channels = channel1
     tier1.sinks = sink1
     
     tier1.sources.source1.type = org.apache.flume.source.kafka.KafkaSource
     tier1.sources.source1.zookeeperConnect = zk01.example.com:2181
     tier1.sources.source1.topic = weblogs
     tier1.sources.source1.groupId = flume
     tier1.sources.source1.channels = channel1
     tier1.sources.source1.interceptors = i1
     tier1.sources.source1.interceptors.i1.type = timestamp
     tier1.sources.source1.kafka.consumer.timeout.ms = 100
     
     tier1.channels.channel1.type = memory
     tier1.channels.channel1.capacity = 10000
     tier1.channels.channel1.transactionCapacity = 1000
     
     tier1.sinks.sink1.type = hdfs
     tier1.sinks.sink1.hdfs.path = /tmp/kafka/%{topic}/%y-%m-%d
     tier1.sinks.sink1.hdfs.rollInterval = 5
     tier1.sinks.sink1.hdfs.rollSize = 0
     tier1.sinks.sink1.hdfs.rollCount = 0
     tier1.sinks.sink1.hdfs.fileType = DataStream
     tier1.sinks.sink1.channel = channel1

    2

    tier1.sources  = source1
     tier1.channels = channel1
     tier1.sinks = sink1
     
     tier1.sources.source1.type = exec
     tier1.sources.source1.command = /usr/bin/vmstat 1
     tier1.sources.source1.channels = channel1
     
     tier1.channels.channel1.type = memory
     tier1.channels.channel1.capacity = 10000
     tier1.channels.channel1.transactionCapacity = 1000
     
     tier1.sinks.sink1.type = org.apache.flume.sink.kafka.KafkaSink
     tier1.sinks.sink1.topic = sink1
     tier1.sinks.sink1.brokerList = kafka01.example.com:9092,kafka02.example.com:9092
     tier1.sinks.sink1.channel = channel1
     tier1.sinks.sink1.batchSize = 20

    3

    tier1.sources = source1
    tier1.channels = channel1
    tier1.sinks = sink1
    
    tier1.sources.source1.type = exec
    tier1.sources.source1.command = /usr/bin/vmstat 1
    tier1.sources.source1.channels = channel1
    
    tier1.channels.channel1.type = org.apache.flume.channel.kafka.KafkaChannel
    tier1.channels.channel1.capacity = 10000
    tier1.channels.channel1.transactionCapacity = 1000
    tier1.channels.channel1.brokerList = kafka02.example.com:9092,kafka03.example.com:9092
    tier1.channels.channel1.topic = channel2
    tier1.channels.channel1.zookeeperConnect = zk01.example.com:2181
    tier1.channels.channel1.parseAsFlumeEvent = true
    
    tier1.sinks.sink1.type = hdfs
    tier1.sinks.sink1.hdfs.path = /tmp/kafka/channel
    tier1.sinks.sink1.hdfs.rollInterval = 5
    tier1.sinks.sink1.hdfs.rollSize = 0
    tier1.sinks.sink1.hdfs.rollCount = 0
    tier1.sinks.sink1.hdfs.fileType = DataStream
    tier1.sinks.sink1.channel = channel1

    总结:

     这样配置以后我们可以通过监控,或者消费kafka console打印效果

    /bin/kafka-console-consumer.sh --bootstrap-server master:9092 --topic topicTest2 --from-beginning

  • 相关阅读:
    CISM国际注册信息安全经理认证与其他认证的差异
    CISD注册信息安全开发人员
    ISACA率先将技能网络安全培训与基于实际操作的考试和认证相结合
    信息安全意识
    2015版CISM国际注册安全经理中文教材
    如何持久建立信息安全意识宣贯/不解释
    北京CISSP免费考前模拟辅导讲座
    信息安全泄露越来越冲“钱”去
    利用碎片时间了解虚拟化安全---第一部分
    如何修改WAMP中mysql默认空密码
  • 原文地址:https://www.cnblogs.com/cbugs/p/9672462.html
Copyright © 2011-2022 走看看