zoukankan      html  css  js  c++  java
  • flume:spooldir采集日志,kafka输出的配置问题

    flume配置:

    #DBFile
    DBFile.sources = sources1  
    DBFile.sinks = sinks1  
    DBFile.channels = channels1  
      
    # DBFile-DB-Source 
    DBFile.sources.sources1.type = spooldir
    DBFile.sources.sources1.spoolDir =/var/log/apache/flumeSpool//db
    DBFile.sources.sources1.inputCharset=utf-8
      
    # DBFile-Sink  
    DBFile.sinks.sinks1.type = org.apache.flume.sink.kafka.KafkaSink  
    DBFile.sinks.sinks1.topic = DBFile
    DBFile.sinks.sinks1.brokerList = hdp01:6667,hdp02:6667,hdp07:6667
    DBFile.sinks.sinks1.requiredAcks = 1  
    DBFile.sinks.sinks1.batchSize = 2000  
      
    # DBFile-Channel
    DBFile.channels.channels1.type = memory
    DBFile.channels.channels1.capacity = 10000
    DBFile.channels.channels1.transactionCapacity = 1000
    
    # DBFile-Source And Sink to the channel
    DBFile.sources.sources1.channels = channels1
    DBFile.sinks.sinks1.channel = channels1

           故障现象:第一次上传文件时,flume能很快处理文件,后面上传还是显示文件未处理。如果重启flume服务,又能立刻处理。

    经测试,问题的原因在这个配置上:DBFile.sinks.sinks1.requiredAcks = -1

          requiredAcks 的官方解释:How many replicas must acknowledge a message before its considered successfully written. Accepted values are 0 (Never wait for acknowledgement),

    1 (wait for leader only), -1 (wait for all replicas) Set this to -1 to avoid data loss in some cases of leader failure.

        把这个值改为1就好了。

  • 相关阅读:
    jQuery daily
    jQuery daily
    spring事务管理
    AspectJ AOP切面编程(XML配置)
    springAOP思想
    spring与web整合(交鸡肋,因为有前台框架封装了servlet)
    spring复杂对象注入四种方式
    spring的Bean注入和P标签使用
    spring Bean的作用域、生命周期和后处理器
    IoC容器实例化Bean三种方式
  • 原文地址:https://www.cnblogs.com/huiy/p/6128029.html
Copyright © 2011-2022 走看看