zoukankan      html  css  js  c++  java
  • org.apache.flume.ChannelException: Take list for MemoryTransaction, capacity 100 full, consider committing more frequently, increasing capacity, or increasing thread count

    flume在抽取MySQL数据到kafka时报错,如下

    [SinkRunner-PollingRunner-DefaultSinkProcessor] ERROR org.apache.flume.sink.kafka.KafkaSink - Failed to publish events
    org.apache.flume.ChannelException: Take list for MemoryTransaction, capacity 100 full, consider committing more frequently, increasing capacity, or increasing thread count
        at org.apache.flume.channel.MemoryChannel$MemoryTransaction.doTake(MemoryChannel.java:96)
        at org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
        at org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
        at org.apache.flume.sink.kafka.KafkaSink.process(KafkaSink.java:161)
        at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:67)
        at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:145)
        at java.lang.Thread.run(Thread.java:745)
    [SinkRunner-PollingRunner-DefaultSinkProcessor] ERROR org.apache.flume.SinkRunner - Unable to deliver event. Exception follows.
    org.apache.flume.EventDeliveryException: Failed to publish events
        at org.apache.flume.sink.kafka.KafkaSink.process(KafkaSink.java:264)
        at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:67)
        at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:145)
        at java.lang.Thread.run(Thread.java:745)
    Caused by: org.apache.flume.ChannelException: Take list for MemoryTransaction, capacity 100 full, consider committing more frequently, increasing capacity, or increasing thread count
        at org.apache.flume.channel.MemoryChannel$MemoryTransaction.doTake(MemoryChannel.java:96)
        at org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
        at org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
        at org.apache.flume.sink.kafka.KafkaSink.process(KafkaSink.java:161)
        ... 3 more
        
    
    MemoryChannel的事务的Put队列满
    解决:将channel的capacity的值调大即可
    a1.channels.ch-1.capacity = 10000
    注意:Flume配置每行后边不能加注释,否则会导致配置不生效
  • 相关阅读:
    sqlite数据库的基本操作
    java-正则表达式判断移动联通电信手机号
    基金新手常识
    Android 心跳包心跳连接 如何实现android和服务器长连接呢?推送消息的原理
    java 设计模式之模板方法
    Android 自定义view实现水波纹效果
    Android中pendingIntent的深入理解
    CentOS 7.X下 -- 配置nginx正向代理支持https
    自动化运维工具saltstack04 -- 之jinja模板
    自动化运维工具saltstack03 -- 之SaltStack的数据系统
  • 原文地址:https://www.cnblogs.com/EnzoDin/p/10156485.html
Copyright © 2011-2022 走看看