zoukankan      html  css  js  c++  java
  • flume:spooldir采集日志,kafka输出的配置问题

    flume配置:

    #DBFile
    DBFile.sources = sources1  
    DBFile.sinks = sinks1  
    DBFile.channels = channels1  
      
    # DBFile-DB-Source 
    DBFile.sources.sources1.type = spooldir
    DBFile.sources.sources1.spoolDir =/var/log/apache/flumeSpool//db
    DBFile.sources.sources1.inputCharset=utf-8
      
    # DBFile-Sink  
    DBFile.sinks.sinks1.type = org.apache.flume.sink.kafka.KafkaSink  
    DBFile.sinks.sinks1.topic = DBFile
    DBFile.sinks.sinks1.brokerList = hdp01:6667,hdp02:6667,hdp07:6667
    DBFile.sinks.sinks1.requiredAcks = 1  
    DBFile.sinks.sinks1.batchSize = 2000  
      
    # DBFile-Channel
    DBFile.channels.channels1.type = memory
    DBFile.channels.channels1.capacity = 10000
    DBFile.channels.channels1.transactionCapacity = 1000
    
    # DBFile-Source And Sink to the channel
    DBFile.sources.sources1.channels = channels1
    DBFile.sinks.sinks1.channel = channels1

           故障现象:第一次上传文件时,flume能很快处理文件,后面上传还是显示文件未处理。如果重启flume服务,又能立刻处理。

    经测试,问题的原因在这个配置上:DBFile.sinks.sinks1.requiredAcks = -1

          requiredAcks 的官方解释:How many replicas must acknowledge a message before its considered successfully written. Accepted values are 0 (Never wait for acknowledgement),

    1 (wait for leader only), -1 (wait for all replicas) Set this to -1 to avoid data loss in some cases of leader failure.

        把这个值改为1就好了。

  • 相关阅读:
    HttpClient 用法
    HttpURLConnection 用法
    Android 真机连接本地PC服务器
    ArrayList VS Vector
    BufferedReader br=new BufferedReader(new InputStreamReader(System.in));
    Implicit super constructor xx() is undefined for default constructor
    layout_gravity
    gravity
    eclipse 建多层包
    RFC 4585 基于实时传输控制协议(RTCP)反馈的扩展RTP(RTP/AVPF)配置文件
  • 原文地址:https://www.cnblogs.com/huiy/p/6128029.html
Copyright © 2011-2022 走看看