zoukankan      html  css  js  c++  java
  • flume1.5的几种conf配置

    flume版本: 1.5.2

    source:netcat

    查看flume监听端口是否打开: netstat -tunlp | grep 44444

    # Name the components on this agent
    a1.sources = r1
    a1.sinks = k1
    a1.channels = c1
    
    # Describe/configure the source
    a1.sources.r1.type = netcat
    a1.sources.r1.bind = 127.0.0.1
    a1.sources.r1.port = 44444
    
    # Describe the sink
    a1.sinks.k1.type = file_roll
    a1.sinks.k1.channel = c1
    a1.sinks.k1.sink.directory = /var/log/flume
    
    # Use a channel which buffers events in memory
    a1.channels.c1.type = memory
    a1.channels.c1.capacity = 10000
    a1.channels.c1.transactionCapacity = 1000

    sources: exec tail

    flume1.7开始支持TAILDIR, 1.5的flume不支持

    # Name the components on this agent
    a1.sources = r1
    a1.sinks = k1
    a1.channels = c1
    
    # Use a channel which buffers events in memory
    a1.channels.c1.type = memory
    a1.channels.c1.capacity = 10000
    a1.channels.c1.transactionCapacity = 1000
    
    # Describe/configure the source
    a1.sources.r1.type = exec
    a1.sources.r1.command = tail -F /var/log/flume/append.txt
    a1.sources.r1.channels = c1
    
    # Describe the sink
    a1.sinks.k1.type = file_roll
    a1.sinks.k1.sink.directory = /var/log/flume
    a1.sinks.k1.channel = c1

    channel: kafkachannel

    flume1.5与1.7的区别参见:记录kafkachannel遇到的坑

    # Name the components on this agent
    a1.sources = r1
    a1.sinks = k1
    a1.channels = c1
    
    # Use a channel which buffers events in memory
    a1.channels.c1.type = org.apache.flume.channel.kafka.KafkaChannel
    a1.channels.c1.brokerList = hadoop102:9092,hadoop103:9092,hadoop104:9092
    a1.channels.c1.zookeeperConnect=hadoop102:2181,hadoop103:2181,hadoop104:2181
    a1.channels.c1.topic = test
    a1.sources.r1.channels = c1
    
    # Describe/configure the source
    a1.sources.r1.type = exec
    a1.sources.r1.command = tail -F /var/log/flume/append.txt
    a1.sources.r1.channels = c1
    
    # Describe the sink
    a1.sinks.k1.type = file_roll
    a1.sinks.k1.sink.directory = /var/log/flume
    a1.sinks.k1.channel = c1

    hdp中集成的kafka端口不是9092, 而是6667

    参考

    Flume 1.5.0 User Guide — Apache Flume documentation

  • 相关阅读:
    MinGW GCC 7.1.0 2017年6月份出炉啦
    java面试题-框架篇九
    spring-AOP原理
    spring的bean管理(注解)
    23种设计模式(1)-单例模式
    SSH框架面试题集锦
    JQuery基础
    实现用户注册
    spring与hibernate的整合
    spring-IOC理解1
  • 原文地址:https://www.cnblogs.com/bitbitbyte/p/13202798.html
Copyright © 2011-2022 走看看