zoukankan      html  css  js  c++  java
  • kafka.common.ConsumerRebalanceFailedException异常解决

    kafka.common.ConsumerRebalanceFailedException: group_dd-1446432618163-2746a209 can't rebalance after 10 retries
      at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:432)
      at kafka.consumer.ZookeeperConsumerConnector.kafka$consumer$ZookeeperConsumerConnector$$reinitializeConsumer(ZookeeperConsumerConnector.scala:722)
      at kafka.consumer.ZookeeperConsumerConnector.consume(ZookeeperConsumerConnector.scala:212)
      at kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreams(ZookeeperConsumerConnector.scala:80)
      at kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreams(ZookeeperConsumerConnector.scala:92)
      at com.symboltech.mine.ConsumerUtil.start(ConsumerUtil.java:40)
      at com.symboltech.mine.KafkaConsumer.sdf(KafkaConsumer.java:58)



    解决办法:
    1.配置zk问题(kafka的consumer配置)
    zookeeper.session.timeout.ms=5000
    zookeeper.connection.timeout.ms=10000
    rebalance.backoff.ms=2000
    rebalance.max.retries=10



    官方解释:
    consumer rebalancing fails (you will see ConsumerRebalanceFailedException): This is due to conflicts when two consumers are trying to own the same topic partition. The log will show you what caused the conflict (search for "conflict in ").
    If your consumer subscribes to many topics and your ZK server is busy, this could be caused by consumers not having enough time to see a consistent view of all consumers in the same group. If this is the case, try Increasing rebalance.max.retries and rebalance.backoff.ms.
    Another reason could be that one of the consumers is hard killed. Other consumers during rebalancing won't realize that consumer is gone after zookeeper.session.timeout.ms time. In the case, make sure that rebalance.max.retries * rebalance.backoff.ms > zookeeper.session.timeout.ms.

  • 相关阅读:
    创建zabbix监控,添加监控机(server、agent、windows)
    KVM虚拟化平台简介及环境部署
    利用nginx的stream模块实现内网端口的转发代理
    Mysql主主同步+keepalived实现高可用
    基于 MHA 的 MySQL 高可用方案
    mysql基于Amoeba(变形虫)实现读写分离
    mysql主从异步复制
    基于mysqld_multi实现MySQL多实例多进程配置
    基于Haproxy+Keepalived构建高可用负载均衡集群
    yum 安装报错The GPG keys listed for the "CentOS-7
  • 原文地址:https://www.cnblogs.com/dailidong/p/7571179.html
Copyright © 2011-2022 走看看