zoukankan      html  css  js  c++  java
  • flume1.5的几种conf配置

    flume版本: 1.5.2

    source:netcat

    查看flume监听端口是否打开: netstat -tunlp | grep 44444

    # Name the components on this agent
    a1.sources = r1
    a1.sinks = k1
    a1.channels = c1
    
    # Describe/configure the source
    a1.sources.r1.type = netcat
    a1.sources.r1.bind = 127.0.0.1
    a1.sources.r1.port = 44444
    
    # Describe the sink
    a1.sinks.k1.type = file_roll
    a1.sinks.k1.channel = c1
    a1.sinks.k1.sink.directory = /var/log/flume
    
    # Use a channel which buffers events in memory
    a1.channels.c1.type = memory
    a1.channels.c1.capacity = 10000
    a1.channels.c1.transactionCapacity = 1000

    sources: exec tail

    flume1.7开始支持TAILDIR, 1.5的flume不支持

    # Name the components on this agent
    a1.sources = r1
    a1.sinks = k1
    a1.channels = c1
    
    # Use a channel which buffers events in memory
    a1.channels.c1.type = memory
    a1.channels.c1.capacity = 10000
    a1.channels.c1.transactionCapacity = 1000
    
    # Describe/configure the source
    a1.sources.r1.type = exec
    a1.sources.r1.command = tail -F /var/log/flume/append.txt
    a1.sources.r1.channels = c1
    
    # Describe the sink
    a1.sinks.k1.type = file_roll
    a1.sinks.k1.sink.directory = /var/log/flume
    a1.sinks.k1.channel = c1

    channel: kafkachannel

    flume1.5与1.7的区别参见:记录kafkachannel遇到的坑

    # Name the components on this agent
    a1.sources = r1
    a1.sinks = k1
    a1.channels = c1
    
    # Use a channel which buffers events in memory
    a1.channels.c1.type = org.apache.flume.channel.kafka.KafkaChannel
    a1.channels.c1.brokerList = hadoop102:9092,hadoop103:9092,hadoop104:9092
    a1.channels.c1.zookeeperConnect=hadoop102:2181,hadoop103:2181,hadoop104:2181
    a1.channels.c1.topic = test
    a1.sources.r1.channels = c1
    
    # Describe/configure the source
    a1.sources.r1.type = exec
    a1.sources.r1.command = tail -F /var/log/flume/append.txt
    a1.sources.r1.channels = c1
    
    # Describe the sink
    a1.sinks.k1.type = file_roll
    a1.sinks.k1.sink.directory = /var/log/flume
    a1.sinks.k1.channel = c1

    hdp中集成的kafka端口不是9092, 而是6667

    参考

    Flume 1.5.0 User Guide — Apache Flume documentation

  • 相关阅读:
    PostgreSQL pg_hba.conf 文件简析
    Centos 查看端口占用情况
    Nginx 从0开始学
    windows 杀死端口号对应进程
    MyBatis基础-05-缓存
    MyBatis基础-04-动态sql
    MyBatis基础-02
    SpringMVC基础-14-SpringMVC与Spring整合
    SpringMVC基础-13-SpringMVC运行流程
    SpringMVC基础-12-异常处理
  • 原文地址:https://www.cnblogs.com/bitbitbyte/p/13202798.html
Copyright © 2011-2022 走看看