zoukankan      html  css  js  c++  java
  • Kafka生产环境中的错误

      最近在处理日志收集任务时,发现前端服务器用flume进行收集,逐步把所有的服务器都增加上。增加的差不多时。Kafka报了如下类似错误:

    709 [main] WARN  kafka.producer.async.DefaultEventHandler  - Produce request with correlation id 214 failed due to [datasift,0]: kafka.common.MessageSizeTooLargeException
    4869 [main] WARN  kafka.producer.async.DefaultEventHandler  - Produce request with    correlation id 217 failed due to [datasift,0]: kafka.common.MessageSizeTooLargeException
    5035 [main] WARN  kafka.producer.async.DefaultEventHandler  - Produce request with   correlation id 220 failed due to [datasift,0]: kafka.common.MessageSizeTooLargeException
    5198 [main] WARN  kafka.producer.async.DefaultEventHandler  - Produce request with correlation id 223 failed due to [datasift,0]: kafka.common.MessageSizeTooLargeException
    5305 [main] ERROR kafka.producer.async.DefaultEventHandler  - Failed to send requests for topics datasift with correlation ids in [213,224]
    
    kafka.common.FailedToSendMessageException: Failed to send messages after 3 tries.
    at kafka.producer.async.DefaultEventHandler.handle(Unknown Source)
    at kafka.producer.Producer.send(Unknown Source)
    at kafka.javaapi.producer.Producer.send(Unknown Source)
    

     查了各种资料,发现是由于Kafka走的默认配置。发送的消息超过了最大字节导致的。修改如下参数即可:

    • replica.fetch.max.bytes - this will allow for the replicas in the brokers to send messages within the cluster and make sure the messages are replicated correctly. If this is too small, then the message will never be replicated, and therefore, the consumer will never see the message because the message will never be committed (fully replicated).
    • message.max.bytes - this is the largest size of the message that can be received by the broker from a producer.
  • 相关阅读:
    2017面试记录
    Bzoj1079:[SCOI2008]着色方案
    Bzoj1046: [HAOI2007]上升序列
    Luogu1121:环状最大两段子段和
    BZOJ 4698: [SDOI2008]Sandy的卡片
    [SPOJ]DISUBSTR:Distinct Substrings&[SPOJ]SUBST1:New Distinct Substrings
    HiHocoder1419 : 后缀数组四·重复旋律4&[SPOJ]REPEATS:Repeats
    Bzoj2946:[POI2000] 最长公共子串
    HiHocoder1415 : 后缀数组三·重复旋律3 & Poj2774:Long Long Message
    POJ3261:Milk Patterns
  • 原文地址:https://www.cnblogs.com/moonandstar08/p/6642228.html
Copyright © 2011-2022 走看看