redis集群节点添加与删除

添加和删除节点的流程

1.新节点添加槽位
2.源节点中的数据进行迁移
3.源节点数据迁移完毕
4.迁移下一个槽位的数据，依次循环

1.添加节点

1.准备新机器

[root@db02 ~]# mkdir /service/redis/{6381,6382}
[root@db02 ~]# vim /service/redis/6381/redis.conf
[root@db02 ~]# vim /service/redis/6382/redis.conf

# 启动新节点
[root@db02 ~]# redis-server /service/redis/6381/redis.conf 
[root@db02 ~]# redis-server /service/redis/6382/redis.conf

2.将新节点添加到集群

[root@db01 ~]# redis-trib.rb add-node 172.16.1.52:6381 172.16.1.51:6379
# 或者
[root@db01 ~]# redis-cli -p 6379 -h 172.16.1.51 cluster meet 172.16.1.52:6381

# 查看节点信息
[root@db01 ~]# redis-trib.rb info 172.16.1.51:6379
[root@db01 ~]# redis-cli -p 6379 -h 172.16.1.51 cluster nodes

3.重新分配槽位

[root@db01 ~]# redis-trib.rb reshard 172.16.1.51:6379
# 你想移动多少个槽位到新节点
How many slots do you want to move (from 1 to 16384)? 4096
# 新节点的ID是什么
What is the receiving node ID? a298dbd22c10b8492d9ff4295504c50666f4fb2e
# 输入源节点的ID，如果是所有节点直接使用all
Please enter all the source node IDs.
  Type 'all' to use all the nodes as source nodes for the hash slots.
  Type 'done' once you entered all the source nodes IDs.
source node #1：all
# 你确定要这么分配？
Do you want to proceed with the proposed reshard plan (yes/no)? yes

# 分配完毕，查看分配结果
[root@db01 ~]# redis-cli -p 6379 -h 172.16.1.51 cluster nodes
e4794215d9d3548e9c514c10626ce618be19ebfb 172.16.1.53:6380 slave 5ad7bd957133eac9c3a692b35f8ae72258cf0ece 0 1596769469815 5 connected
d27553035a3e91c78d375208c72b756e9b2523d4 172.16.1.53:6379 master - 0 1596769468805 3 connected 12288-16383
5ad7bd957133eac9c3a692b35f8ae72258cf0ece 172.16.1.51:6379 myself,master - 0 0 1 connected 1365-5460
fee551a90c8646839f66fa0cd1f6e5859e9dd8e0 172.16.1.52:6380 slave d27553035a3e91c78d375208c72b756e9b2523d4 0 1596769467797 4 connected
1d10edbc5ed08f85d2afc21cd338b023b9dd61b4 172.16.1.51:6380 slave 7c79559b280db9d9c182f3a25c718efe9e934fc7 0 1596769469513 6 connected
a298dbd22c10b8492d9ff4295504c50666f4fb2e 172.16.1.52:6381 master - 0 1596769468302 7 connected 0-1364 5461-6826 10923-12287
7c79559b280db9d9c182f3a25c718efe9e934fc7 172.16.1.52:6379 master - 0 1596769468302 2 connected 6827-10922


[root@db01 ~]# redis-trib.rb info 172.16.1.51:6379
172.16.1.51:6379 (5ad7bd95...) -> 499 keys | 4096 slots | 1 slaves.
172.16.1.53:6379 (d2755303...) -> 501 keys | 4096 slots | 1 slaves.
172.16.1.52:6381 (a298dbd2...) -> 502 keys | 4096 slots | 0 slaves.
172.16.1.52:6379 (7c79559b...) -> 498 keys | 4096 slots | 1 slaves.
[OK] 2000 keys in 4 masters.
0.12 keys per slot on average.

4.添加新节点的副本

[root@db02 ~]# redis-cli -h 172.16.1.52 -p 6382 cluster replicate a298dbd22c10b8492d9ff4295504c50666f4fb2e
# 或者
[root@db01 ~]# redis-trib.rb add-node --slave --master-id a298dbd22c10b8492d9ff4295504c50666f4fb2e 172.16.1.52:6382 172.16.1.51:6379
			  ######集群中任意一台机器######
		
[root@db01 ~]# redis-cli -p 6379 -h 172.16.1.51 cluster nodes
[root@db01 ~]# redis-trib.rb info 172.16.1.51:6379
172.16.1.51:6379 (5ad7bd95...) -> 499 keys | 4096 slots | 1 slaves.
172.16.1.53:6379 (d2755303...) -> 501 keys | 4096 slots | 1 slaves.
172.16.1.52:6381 (a298dbd2...) -> 502 keys | 4096 slots | 1 slaves.
172.16.1.52:6379 (7c79559b...) -> 498 keys | 4096 slots | 1 slaves.
[OK] 2000 keys in 4 masters.
0.12 keys per slot on average.

# 调整主从，尽量保证每一台机器的副本都不于主节点在一台机器

5.模拟故障

#分配槽位的过程中
[root@db01 ~]# redis-trib.rb reshard 172.16.1.51:6379

#执行Ctrl+c

#查看集群状态一些命令看不出来有错
[root@db01 ~]# redis-cli -p 6379 -h 172.16.1.51 cluster info
[root@db01 ~]# redis-cli -p 6379 -h 172.16.1.51 cluster nodes
[root@db01 ~]# redis-trib.rb info 172.16.1.51:6379

####必须使用工具检查集群####
[root@db01 ~]# redis-trib.rb check 172.16.1.51:6379

####错误一：172.16.1.52:6379节点正在导出槽位，172.16.1.52:6381节点正在导入槽位####
>>> Check for open slots...
[WARNING] Node 172.16.1.52:6381 has slots in importing state (6885).
[WARNING] Node 172.16.1.52:6379 has slots in migrating state (6885).
[WARNING] The following slots are open: 6885
>>> Check slots coverage...

####错误二：172.16.1.52:6379节点正在导出槽位####
>>> Check for open slots...
[WARNING] Node 172.16.1.52:6379 has slots in migrating state (6975).
[WARNING] The following slots are open: 6975
>>> Check slots coverage...

####错误三：####
>>> Check for open slots...
[WARNING] Node 172.16.1.52:6381 has slots in importing state (7093).
[WARNING] The following slots are open: 7093
>>> Check slots coverage...

6.修复故障

# 使用fix修复集群
[root@db01 ~]# redis-trib.rb fix 172.16.1.52:6379

# 将槽位平均分配
1.平均之前
[root@db01 ~]# redis-trib.rb info 172.16.1.51:6379
172.16.1.51:6379 (5ad7bd95...) -> 499 keys | 4096 slots | 1 slaves.
172.16.1.53:6379 (d2755303...) -> 501 keys | 4096 slots | 1 slaves.
172.16.1.52:6381 (a298dbd2...) -> 648 keys | 5320 slots | 1 slaves.
172.16.1.52:6379 (7c79559b...) -> 352 keys | 2872 slots | 1 slaves.
[OK] 2000 keys in 4 masters.
0.12 keys per slot on average.

2.平均分配（如果节点之间槽位数量相差较小，不会平均分配）
[root@db01 ~]# redis-trib.rb rebalance 172.16.1.51:6379
>>> Performing Cluster Check (using node 172.16.1.51:6379)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
>>> Rebalancing across 4 nodes. Total weight = 4
Moving 1224 slots from 172.16.1.52:6381 to 172.16.1.52:6379

3平均分配之后
[root@db01 ~]# redis-trib.rb info 172.16.1.51:6379
172.16.1.51:6379 (5ad7bd95...) -> 499 keys | 4096 slots | 1 slaves.
172.16.1.53:6379 (d2755303...) -> 501 keys | 4096 slots | 1 slaves.
172.16.1.52:6381 (a298dbd2...) -> 492 keys | 4096 slots | 1 slaves.
172.16.1.52:6379 (7c79559b...) -> 508 keys | 4096 slots | 1 slaves.
[OK] 2000 keys in 4 masters.
0.12 keys per slot on average.

2.删除节点

1.重新分配槽位

# 重新分配
[root@db01 ~]# redis-trib.rb reshard 172.16.1.51:6379
How many slots do you want to move (from 1 to 16384)? 1365   #要分配的槽位数量
What is the receiving node ID? #接收槽位的节点id
Please enter all the source node IDs.
  Type 'all' to use all the nodes as source nodes for the hash slots.
  Type 'done' once you entered all the source nodes IDs.
Source node #1: 要删除的节点id
Source node #2: done
Do you want to proceed with the proposed reshard plan (yes/no)? yes

# 第二次重新分配
[root@db01 ~]# redis-trib.rb reshard 172.16.1.51:6379
How many slots do you want to move (from 1 to 16384)? 1365
What is the receiving node ID? 7c79559b280db9d9c182f3a25c718efe9e934fc7
Please enter all the source node IDs.
  Type 'all' to use all the nodes as source nodes for the hash slots.
  Type 'done' once you entered all the source nodes IDs.
Source node #1:a298dbd22c10b8492d9ff4295504c50666f4fb2e
Source node #2:done
Do you want to proceed with the proposed reshard plan (yes/no)? yes

# 第三次重新分配
[root@db01 ~]# redis-trib.rb reshard 172.16.1.51:6379
How many slots do you want to move (from 1 to 16384)? 1366
What is the receiving node ID? d27553035a3e91c78d375208c72b756e9b2523d4
Please enter all the source node IDs.
  Type 'all' to use all the nodes as source nodes for the hash slots.
  Type 'done' once you entered all the source nodes IDs.
Source node #1:a298dbd22c10b8492d9ff4295504c50666f4fb2e
Source node #2:done
Do you want to proceed with the proposed reshard plan (yes/no)? yes

#####也可以直接全部分配一台机器，然后重新均分槽位#####

2.查看重新分配以后的节点信息

[root@db01 ~]# redis-trib.rb info 172.16.1.51:6379
172.16.1.51:6379 (5ad7bd95...) -> 664 keys | 5461 slots | 1 slaves.
172.16.1.53:6379 (d2755303...) -> 665 keys | 5462 slots | 2 slaves.
172.16.1.52:6381 (a298dbd2...) -> 0 keys | 0 slots | 0 slaves.
172.16.1.52:6379 (7c79559b...) -> 671 keys | 5461 slots | 1 slaves.
[OK] 2000 keys in 4 masters.
0.12 keys per slot on average.

3.删除节点

# 删除没有数据的主节点
[root@db01 ~]# redis-trib.rb del-node 172.16.1.52:6381 a298dbd22c10b8492d9ff4295504c50666f4fb2e
>>> Removing node a298dbd22c10b8492d9ff4295504c50666f4fb2e from cluster 172.16.1.52:6381
>>> Sending CLUSTER FORGET messages to the cluster...
>>> SHUTDOWN the node.

# 删除从节点
[root@db01 ~]# redis-trib.rb del-node 172.16.1.52:6382 47e3638a203488218d8c62deb82e768598977ba4
>>> Removing node 47e3638a203488218d8c62deb82e768598977ba4 from cluster 172.16.1.52:6382
>>> Sending CLUSTER FORGET messages to the cluster...
>>> SHUTDOWN the node.

# 删除时不能删除有数据的主节点，从节点可以随便删除
[root@db01 ~]# redis-trib.rb del-node 172.16.1.53:6379 d27553035a3e91c78d375208c72b756e9b2523d4
>>> Removing node d27553035a3e91c78d375208c72b756e9b2523d4 from cluster 172.16.1.53:6379
[ERR] Node 172.16.1.53:6379 is not empty! Reshard data away and try again.