zoukankan      html  css  js  c++  java

  • 使用 kafkat 在线扩缩容 kafka replicas

    本文档应用环境为 kafka-0.8.2.0, 其余版本请先行测试

    场景

    线上很多 kafka 的 topic 的副本数为1,这样的设置丧失了 kafka 高可用的特性,所以我们需要把 topic 的副本数至少设置为2(当前3台 kafka)

     我们使用 kafkat 这个工具,这个工具是 ruby 写的,所以我们先安装 ruby 环境

    参考 https://ruby-china.org/wiki/rvm-guide

    这时候你应该已经有了 gem 命令

    继续参考https://github.com/airbnb/kafkat 把 kafkat 装上, 并配置好你的 kafka 环境参数到 ~/.kafkatcfg

    实操

    我们修改 topic 的 replicas 为2

    [root@001 kafka]# kafkat partitions IM_APP_BINSPECT_MSG_TOPIC
    Topic       Partition   Leader      Replicas                        ISRs
    IM_APP_BINSPECT_MSG_TOPIC   0       2       [2]                         [2]
    IM_APP_BINSPECT_MSG_TOPIC   1       3       [3]                         [3]
    IM_APP_BINSPECT_MSG_TOPIC   2       1       [1]                         [1]
    IM_APP_BINSPECT_MSG_TOPIC   3       2       [2]                         [2]
    IM_APP_BINSPECT_MSG_TOPIC   4       3       [3]                         [3]
    IM_APP_BINSPECT_MSG_TOPIC   5       1       [1]                         [1]
    IM_APP_BINSPECT_MSG_TOPIC   6       2       [2]                         [2]
    IM_APP_BINSPECT_MSG_TOPIC   7       3       [3]                         [3]
    IM_APP_BINSPECT_MSG_TOPIC   8       1       [1]                         [1]
    IM_APP_BINSPECT_MSG_TOPIC   9       2       [2]                         [2]
    IM_APP_BINSPECT_MSG_TOPIC   10      3       [3]                         [3]
    IM_APP_BINSPECT_MSG_TOPIC   11      1       [1]                         [1]
    IM_APP_BINSPECT_MSG_TOPIC   12      2       [2]                         [2]
    IM_APP_BINSPECT_MSG_TOPIC   13      3       [3]                         [3]
    IM_APP_BINSPECT_MSG_TOPIC   14      1       [1]                         [1]
    IM_APP_BINSPECT_MSG_TOPIC   15      2       [2]                         [2]
    IM_APP_BINSPECT_MSG_TOPIC   16      3       [3]                         [3]
    IM_APP_BINSPECT_MSG_TOPIC   17      1       [1]                         [1]
    IM_APP_BINSPECT_MSG_TOPIC   18      2       [2]                         [2]
    IM_APP_BINSPECT_MSG_TOPIC   19      3       [3]                         [3]
    [root@001 kafka]# kafkat set-replication-factor IM_APP_BINSPECT_MSG_TOPIC --newrf 2 --brokers 1,2,3
    This operation executes the following assignments:
    Topic       Partition   Replicas
    IM_APP_BINSPECT_MSG_TOPIC   0       [2, 1]
    IM_APP_BINSPECT_MSG_TOPIC   1       [3, 2]
    IM_APP_BINSPECT_MSG_TOPIC   2       [1, 3]
    IM_APP_BINSPECT_MSG_TOPIC   3       [2, 1]
    IM_APP_BINSPECT_MSG_TOPIC   4       [3, 2]
    IM_APP_BINSPECT_MSG_TOPIC   5       [1, 3]
    IM_APP_BINSPECT_MSG_TOPIC   6       [2, 1]
    IM_APP_BINSPECT_MSG_TOPIC   7       [3, 2]
    IM_APP_BINSPECT_MSG_TOPIC   8       [1, 3]
    IM_APP_BINSPECT_MSG_TOPIC   9       [2, 1]
    IM_APP_BINSPECT_MSG_TOPIC   10      [3, 2]
    IM_APP_BINSPECT_MSG_TOPIC   11      [1, 3]
    IM_APP_BINSPECT_MSG_TOPIC   12      [2, 1]
    IM_APP_BINSPECT_MSG_TOPIC   13      [3, 2]
    IM_APP_BINSPECT_MSG_TOPIC   14      [1, 3]
    IM_APP_BINSPECT_MSG_TOPIC   15      [2, 1]
    IM_APP_BINSPECT_MSG_TOPIC   16      [3, 2]
    IM_APP_BINSPECT_MSG_TOPIC   17      [1, 3]
    IM_APP_BINSPECT_MSG_TOPIC   18      [2, 1]
    IM_APP_BINSPECT_MSG_TOPIC   19      [3, 2]
    Proceed (y/n)?
    y
    Beginning.
    Started.
    [root@001 kafka]# kafkat partitions IM_APP_BINSPECT_MSG_TOPIC
    Topic       Partition   Leader      Replicas                        ISRs
    IM_APP_BINSPECT_MSG_TOPIC   0       2       [2, 1]                          [2]
    IM_APP_BINSPECT_MSG_TOPIC   1       3       [3, 2]                          [3, 2]
    IM_APP_BINSPECT_MSG_TOPIC   2       1       [1, 3]                          [1]
    IM_APP_BINSPECT_MSG_TOPIC   3       2       [2, 1]                          [2]
    IM_APP_BINSPECT_MSG_TOPIC   4       3       [3, 2]                          [3, 2]
    IM_APP_BINSPECT_MSG_TOPIC   5       1       [1, 3]                          [1]
    IM_APP_BINSPECT_MSG_TOPIC   6       2       [2, 1]                          [2]
    IM_APP_BINSPECT_MSG_TOPIC   7       3       [3, 2]                          [3, 2]
    IM_APP_BINSPECT_MSG_TOPIC   8       1       [1, 3]                          [1]
    IM_APP_BINSPECT_MSG_TOPIC   9       2       [2, 1]                          [2]
    IM_APP_BINSPECT_MSG_TOPIC   10      3       [3, 2]                          [3, 2]
    IM_APP_BINSPECT_MSG_TOPIC   11      1       [1, 3]                          [1]
    IM_APP_BINSPECT_MSG_TOPIC   12      2       [2, 1]                          [2]
    IM_APP_BINSPECT_MSG_TOPIC   13      3       [3, 2]                          [3, 2]
    IM_APP_BINSPECT_MSG_TOPIC   14      1       [1, 3]                          [1]
    IM_APP_BINSPECT_MSG_TOPIC   15      2       [2, 1]                          [2]
    IM_APP_BINSPECT_MSG_TOPIC   16      3       [3, 2]                          [3, 2]
    IM_APP_BINSPECT_MSG_TOPIC   17      1       [1, 3]                          [1]
    IM_APP_BINSPECT_MSG_TOPIC   18      2       [2, 1]                          [2]
    IM_APP_BINSPECT_MSG_TOPIC   19      3       [3, 2]                          [3]

    解释

    上面一共执行了3个命令

    1. kafkat partitions IM_APP_BINSPECT_MSG_TOPIC   查看 topic 的分区情况
    2. kafkat set-replication-factor IM_APP_BINSPECT_MSG_TOPIC --newrf 2 --brokers 1,2,3   重新设置副本数为2  --newrf 2  , 你有几个 broker就在--brokers 后面指定
    3. kafkat partitions IM_APP_BINSPECT_MSG_TOPIC    检查是否生效


    根据 topic 数据量最后完成的时间也不同(大概几分钟)

    这是最后的完成情况

    [root@001 kafka]# kafkat partitions IM_APP_BINSPECT_MSG_TOPIC
    Topic       Partition   Leader      Replicas                        ISRs
    IM_APP_BINSPECT_MSG_TOPIC   0       2       [2, 1]                          [2, 1]
    IM_APP_BINSPECT_MSG_TOPIC   1       3       [3, 2]                          [3, 2]
    IM_APP_BINSPECT_MSG_TOPIC   2       1       [1, 3]                          [1, 3]
    IM_APP_BINSPECT_MSG_TOPIC   3       2       [2, 1]                          [2, 1]
    IM_APP_BINSPECT_MSG_TOPIC   4       3       [3, 2]                          [3, 2]
    IM_APP_BINSPECT_MSG_TOPIC   5       1       [1, 3]                          [1, 3]
    IM_APP_BINSPECT_MSG_TOPIC   6       2       [2, 1]                          [2, 1]
    IM_APP_BINSPECT_MSG_TOPIC   7       3       [3, 2]                          [3, 2]
    IM_APP_BINSPECT_MSG_TOPIC   8       1       [1, 3]                          [1, 3]
    IM_APP_BINSPECT_MSG_TOPIC   9       2       [2, 1]                          [2, 1]
    IM_APP_BINSPECT_MSG_TOPIC   10      3       [3, 2]                          [3, 2]
    IM_APP_BINSPECT_MSG_TOPIC   11      1       [1, 3]                          [1, 3]
    IM_APP_BINSPECT_MSG_TOPIC   12      2       [2, 1]                          [2, 1]
    IM_APP_BINSPECT_MSG_TOPIC   13      3       [3, 2]                          [3, 2]
    IM_APP_BINSPECT_MSG_TOPIC   14      1       [1, 3]                          [1, 3]
    IM_APP_BINSPECT_MSG_TOPIC   15      2       [2, 1]                          [2, 1]
    IM_APP_BINSPECT_MSG_TOPIC   16      3       [3, 2]                          [3, 2]
    IM_APP_BINSPECT_MSG_TOPIC   17      1       [1, 3]                          [1, 3]
    IM_APP_BINSPECT_MSG_TOPIC   18      2       [2, 1]                          [2, 1]
    IM_APP_BINSPECT_MSG_TOPIC   19      3       [3, 2]                          [3, 2]
     
  • 相关阅读:
    EF 错误解决
    TortoiseHg 学习笔记 （转）
    Mysql 命令行 使用 （转）
    2017-9-3 时间字符串格式化（转）
    2017-8-25 c# 获取url参数的五种方法（转）
    alert 的使用方法
    表单关键字查询写法
    Mysql和Mysqli的区别
    php MySQL中 增、删、改、查的写法格式
    一维、二维数组 与 常用的返回数组 以及 fetch_all与fetch_row的区别
  • 原文地址：https://www.cnblogs.com/txwsqk/p/9661266.html
Copyright © 2011-2022 走看看