zoukankan      html  css  js  c++  java

  • 使用 kafkat 在线扩缩容 kafka replicas

    本文档应用环境为 kafka-0.8.2.0, 其余版本请先行测试

    场景

    线上很多 kafka 的 topic 的副本数为1,这样的设置丧失了 kafka 高可用的特性,所以我们需要把 topic 的副本数至少设置为2(当前3台 kafka)

     我们使用 kafkat 这个工具,这个工具是 ruby 写的,所以我们先安装 ruby 环境

    参考 https://ruby-china.org/wiki/rvm-guide

    这时候你应该已经有了 gem 命令

    继续参考https://github.com/airbnb/kafkat 把 kafkat 装上, 并配置好你的 kafka 环境参数到 ~/.kafkatcfg

    实操

    我们修改 topic 的 replicas 为2

    [root@001 kafka]# kafkat partitions IM_APP_BINSPECT_MSG_TOPIC
    Topic       Partition   Leader      Replicas                        ISRs
    IM_APP_BINSPECT_MSG_TOPIC   0       2       [2]                         [2]
    IM_APP_BINSPECT_MSG_TOPIC   1       3       [3]                         [3]
    IM_APP_BINSPECT_MSG_TOPIC   2       1       [1]                         [1]
    IM_APP_BINSPECT_MSG_TOPIC   3       2       [2]                         [2]
    IM_APP_BINSPECT_MSG_TOPIC   4       3       [3]                         [3]
    IM_APP_BINSPECT_MSG_TOPIC   5       1       [1]                         [1]
    IM_APP_BINSPECT_MSG_TOPIC   6       2       [2]                         [2]
    IM_APP_BINSPECT_MSG_TOPIC   7       3       [3]                         [3]
    IM_APP_BINSPECT_MSG_TOPIC   8       1       [1]                         [1]
    IM_APP_BINSPECT_MSG_TOPIC   9       2       [2]                         [2]
    IM_APP_BINSPECT_MSG_TOPIC   10      3       [3]                         [3]
    IM_APP_BINSPECT_MSG_TOPIC   11      1       [1]                         [1]
    IM_APP_BINSPECT_MSG_TOPIC   12      2       [2]                         [2]
    IM_APP_BINSPECT_MSG_TOPIC   13      3       [3]                         [3]
    IM_APP_BINSPECT_MSG_TOPIC   14      1       [1]                         [1]
    IM_APP_BINSPECT_MSG_TOPIC   15      2       [2]                         [2]
    IM_APP_BINSPECT_MSG_TOPIC   16      3       [3]                         [3]
    IM_APP_BINSPECT_MSG_TOPIC   17      1       [1]                         [1]
    IM_APP_BINSPECT_MSG_TOPIC   18      2       [2]                         [2]
    IM_APP_BINSPECT_MSG_TOPIC   19      3       [3]                         [3]
    [root@001 kafka]# kafkat set-replication-factor IM_APP_BINSPECT_MSG_TOPIC --newrf 2 --brokers 1,2,3
    This operation executes the following assignments:
    Topic       Partition   Replicas
    IM_APP_BINSPECT_MSG_TOPIC   0       [2, 1]
    IM_APP_BINSPECT_MSG_TOPIC   1       [3, 2]
    IM_APP_BINSPECT_MSG_TOPIC   2       [1, 3]
    IM_APP_BINSPECT_MSG_TOPIC   3       [2, 1]
    IM_APP_BINSPECT_MSG_TOPIC   4       [3, 2]
    IM_APP_BINSPECT_MSG_TOPIC   5       [1, 3]
    IM_APP_BINSPECT_MSG_TOPIC   6       [2, 1]
    IM_APP_BINSPECT_MSG_TOPIC   7       [3, 2]
    IM_APP_BINSPECT_MSG_TOPIC   8       [1, 3]
    IM_APP_BINSPECT_MSG_TOPIC   9       [2, 1]
    IM_APP_BINSPECT_MSG_TOPIC   10      [3, 2]
    IM_APP_BINSPECT_MSG_TOPIC   11      [1, 3]
    IM_APP_BINSPECT_MSG_TOPIC   12      [2, 1]
    IM_APP_BINSPECT_MSG_TOPIC   13      [3, 2]
    IM_APP_BINSPECT_MSG_TOPIC   14      [1, 3]
    IM_APP_BINSPECT_MSG_TOPIC   15      [2, 1]
    IM_APP_BINSPECT_MSG_TOPIC   16      [3, 2]
    IM_APP_BINSPECT_MSG_TOPIC   17      [1, 3]
    IM_APP_BINSPECT_MSG_TOPIC   18      [2, 1]
    IM_APP_BINSPECT_MSG_TOPIC   19      [3, 2]
    Proceed (y/n)?
    y
    Beginning.
    Started.
    [root@001 kafka]# kafkat partitions IM_APP_BINSPECT_MSG_TOPIC
    Topic       Partition   Leader      Replicas                        ISRs
    IM_APP_BINSPECT_MSG_TOPIC   0       2       [2, 1]                          [2]
    IM_APP_BINSPECT_MSG_TOPIC   1       3       [3, 2]                          [3, 2]
    IM_APP_BINSPECT_MSG_TOPIC   2       1       [1, 3]                          [1]
    IM_APP_BINSPECT_MSG_TOPIC   3       2       [2, 1]                          [2]
    IM_APP_BINSPECT_MSG_TOPIC   4       3       [3, 2]                          [3, 2]
    IM_APP_BINSPECT_MSG_TOPIC   5       1       [1, 3]                          [1]
    IM_APP_BINSPECT_MSG_TOPIC   6       2       [2, 1]                          [2]
    IM_APP_BINSPECT_MSG_TOPIC   7       3       [3, 2]                          [3, 2]
    IM_APP_BINSPECT_MSG_TOPIC   8       1       [1, 3]                          [1]
    IM_APP_BINSPECT_MSG_TOPIC   9       2       [2, 1]                          [2]
    IM_APP_BINSPECT_MSG_TOPIC   10      3       [3, 2]                          [3, 2]
    IM_APP_BINSPECT_MSG_TOPIC   11      1       [1, 3]                          [1]
    IM_APP_BINSPECT_MSG_TOPIC   12      2       [2, 1]                          [2]
    IM_APP_BINSPECT_MSG_TOPIC   13      3       [3, 2]                          [3, 2]
    IM_APP_BINSPECT_MSG_TOPIC   14      1       [1, 3]                          [1]
    IM_APP_BINSPECT_MSG_TOPIC   15      2       [2, 1]                          [2]
    IM_APP_BINSPECT_MSG_TOPIC   16      3       [3, 2]                          [3, 2]
    IM_APP_BINSPECT_MSG_TOPIC   17      1       [1, 3]                          [1]
    IM_APP_BINSPECT_MSG_TOPIC   18      2       [2, 1]                          [2]
    IM_APP_BINSPECT_MSG_TOPIC   19      3       [3, 2]                          [3]

    解释

    上面一共执行了3个命令

    1. kafkat partitions IM_APP_BINSPECT_MSG_TOPIC   查看 topic 的分区情况
    2. kafkat set-replication-factor IM_APP_BINSPECT_MSG_TOPIC --newrf 2 --brokers 1,2,3   重新设置副本数为2  --newrf 2  , 你有几个 broker就在--brokers 后面指定
    3. kafkat partitions IM_APP_BINSPECT_MSG_TOPIC    检查是否生效


    根据 topic 数据量最后完成的时间也不同(大概几分钟)

    这是最后的完成情况

    [root@001 kafka]# kafkat partitions IM_APP_BINSPECT_MSG_TOPIC
    Topic       Partition   Leader      Replicas                        ISRs
    IM_APP_BINSPECT_MSG_TOPIC   0       2       [2, 1]                          [2, 1]
    IM_APP_BINSPECT_MSG_TOPIC   1       3       [3, 2]                          [3, 2]
    IM_APP_BINSPECT_MSG_TOPIC   2       1       [1, 3]                          [1, 3]
    IM_APP_BINSPECT_MSG_TOPIC   3       2       [2, 1]                          [2, 1]
    IM_APP_BINSPECT_MSG_TOPIC   4       3       [3, 2]                          [3, 2]
    IM_APP_BINSPECT_MSG_TOPIC   5       1       [1, 3]                          [1, 3]
    IM_APP_BINSPECT_MSG_TOPIC   6       2       [2, 1]                          [2, 1]
    IM_APP_BINSPECT_MSG_TOPIC   7       3       [3, 2]                          [3, 2]
    IM_APP_BINSPECT_MSG_TOPIC   8       1       [1, 3]                          [1, 3]
    IM_APP_BINSPECT_MSG_TOPIC   9       2       [2, 1]                          [2, 1]
    IM_APP_BINSPECT_MSG_TOPIC   10      3       [3, 2]                          [3, 2]
    IM_APP_BINSPECT_MSG_TOPIC   11      1       [1, 3]                          [1, 3]
    IM_APP_BINSPECT_MSG_TOPIC   12      2       [2, 1]                          [2, 1]
    IM_APP_BINSPECT_MSG_TOPIC   13      3       [3, 2]                          [3, 2]
    IM_APP_BINSPECT_MSG_TOPIC   14      1       [1, 3]                          [1, 3]
    IM_APP_BINSPECT_MSG_TOPIC   15      2       [2, 1]                          [2, 1]
    IM_APP_BINSPECT_MSG_TOPIC   16      3       [3, 2]                          [3, 2]
    IM_APP_BINSPECT_MSG_TOPIC   17      1       [1, 3]                          [1, 3]
    IM_APP_BINSPECT_MSG_TOPIC   18      2       [2, 1]                          [2, 1]
    IM_APP_BINSPECT_MSG_TOPIC   19      3       [3, 2]                          [3, 2]
     
  • 相关阅读:
    字符串替换
    字符串查找
    字符串比较
    字节与字符串相互转换
    1365. How Many Numbers Are Smaller Than the Current Number
    1486. XOR Operation in an Array
    1431. Kids With the Greatest Number of Candies
    1470. Shuffle the Array
    1480. Running Sum of 1d Array
    【STM32H7教程】第56章 STM32H7的DMA2D应用之刷色块，位图和Alpha混合
  • 原文地址：https://www.cnblogs.com/txwsqk/p/9661266.html
Copyright © 2011-2022 走看看