zoukankan      html  css  js  c++  java
  • kafka-重复消费-1

    错误如下:

    Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member. This means that the time between subsequent calls to poll() was longer than the configured session.timeout.ms, which typically implies that the poll loop is spending too much time message processing. You can address this either by increasing the session timeout or by reducing the maximum size of batches returned in poll() with max.poll.records.

    造成的问题:假如consumer.properties配置中max.poll.records=40  (一次最多拉取40条数据)  session.timeout.ms=30000    (会话时间)

    假设kafka此时一次拉取了40条数据,但在处理第31条的时候抛出了如上的异常,就会导致,本次offset不会提交,完了这40条消息都会在接下来的某刻被再次消费,这其中就包含了其实已经消费了的30条数据

    原因:the poll loop is spending too much time message processing, the time between subsequent calls to poll() was longer than the configured session.timeout.ms,好吧其实是一个意思!

    意思就是说poll下来数据后,处理这些数据的时间比 session.timeout.ms配置的时间要长,从而导致the group has already rebalanced

    解决办法是最后一句话:You can address this either by increasing the session timeout or by reducing the maximum size of batches returned in poll() with max.poll.records.

    即要不增大 session.timeout.ms,要不减小max.poll.records ,至于具体配置为多少,得看你处理一条消息花费多长时间 x,需要满足 x乘以max.poll.records < session.timeout.ms

  • 相关阅读:
    五步搞定Android开发环境部署!
    WebBrowser JS回调delphi的方法 (简单通用)
    Delphi根据方法名调用方法
    Delphi TstringList Stringlist的特殊用法
    delphi 获取webbrowser的cookies给Idhttp用
    HttpHelper类登录淘宝联盟并下载淘宝客订单xls
    我的常用笔记(GetAndroid,ADBDemo,GetSJ,GetTB)
    P1025 数的划分
    P1969 积木大赛
    P2038 无线网络发射器选址
  • 原文地址:https://www.cnblogs.com/chinano1/p/9308000.html
Copyright © 2011-2022 走看看