zoukankan      html  css  js  c++  java
  • Kafka生产环境中的错误

      最近在处理日志收集任务时,发现前端服务器用flume进行收集,逐步把所有的服务器都增加上。增加的差不多时。Kafka报了如下类似错误:

    709 [main] WARN  kafka.producer.async.DefaultEventHandler  - Produce request with correlation id 214 failed due to [datasift,0]: kafka.common.MessageSizeTooLargeException
    4869 [main] WARN  kafka.producer.async.DefaultEventHandler  - Produce request with    correlation id 217 failed due to [datasift,0]: kafka.common.MessageSizeTooLargeException
    5035 [main] WARN  kafka.producer.async.DefaultEventHandler  - Produce request with   correlation id 220 failed due to [datasift,0]: kafka.common.MessageSizeTooLargeException
    5198 [main] WARN  kafka.producer.async.DefaultEventHandler  - Produce request with correlation id 223 failed due to [datasift,0]: kafka.common.MessageSizeTooLargeException
    5305 [main] ERROR kafka.producer.async.DefaultEventHandler  - Failed to send requests for topics datasift with correlation ids in [213,224]
    
    kafka.common.FailedToSendMessageException: Failed to send messages after 3 tries.
    at kafka.producer.async.DefaultEventHandler.handle(Unknown Source)
    at kafka.producer.Producer.send(Unknown Source)
    at kafka.javaapi.producer.Producer.send(Unknown Source)
    

     查了各种资料,发现是由于Kafka走的默认配置。发送的消息超过了最大字节导致的。修改如下参数即可:

    • replica.fetch.max.bytes - this will allow for the replicas in the brokers to send messages within the cluster and make sure the messages are replicated correctly. If this is too small, then the message will never be replicated, and therefore, the consumer will never see the message because the message will never be committed (fully replicated).
    • message.max.bytes - this is the largest size of the message that can be received by the broker from a producer.
  • 相关阅读:
    前端优化,注意的一些东西
    php遍历memcache的方法
    mysql增量备份
    CI框架安装
    sphinx:received zerosized searchd response
    SEO思考:逆水行舟 不进则退
    利用GNUstep在windows下编写objectc
    Centos5.3下安装memcached
    想卖网站?我来给你指条明路
    时髦的互联网公司都在用什么技术?
  • 原文地址:https://www.cnblogs.com/moonandstar08/p/6642228.html
Copyright © 2011-2022 走看看