redis 4.0.9 cluster + failover

os: centos7.4
redis:4.0.9
ruby:2.3.4

ip 规划

redis1    192.168.56.101
redis2    192.168.56.102

用两台虚拟机模拟6个redis节点。
分别为
192.168.56.101 的 9001、9002、9003 端口，
192.168.56.102 的 9001、9002、9003 端口。

连接redis

# redis-cli -h 192.168.56.101 -p 9001 -c
192.168.56.101:9001>
192.168.56.101:9001>
192.168.56.101:9001> cluster info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:6
cluster_my_epoch:1
cluster_stats_messages_ping_sent:23175
cluster_stats_messages_pong_sent:25275
cluster_stats_messages_sent:48450
cluster_stats_messages_ping_received:25270
cluster_stats_messages_pong_received:23175
cluster_stats_messages_meet_received:5
cluster_stats_messages_received:48450

192.168.56.101:9001> cluster nodes
cf22edbc6463b8b31536819297ac50e9e39175ad 192.168.56.102:9003@19003 slave 8240d131c58c8b687c11462e03825bdba00af53b 0 1526649623970 6 connected
8240d131c58c8b687c11462e03825bdba00af53b 192.168.56.101:9001@19001 myself,master - 0 1526649623000 1 connected 0-5460
12e2353c9f0b085d2684da46ba0dc69575d68603 192.168.56.102:9001@19001 slave e38d78fc68b69d4e5f347cf96f31a9cb6a02dccf 0 1526649621000 7 connected
2b48fd08a8ec55eba5bfa0d163d02c8462637302 192.168.56.102:9002@19002 slave 0bdd8387a662984e95f5103522f52e75eceaab69 0 1526649624972 5 connected
0bdd8387a662984e95f5103522f52e75eceaab69 192.168.56.101:9002@19002 master - 0 1526649622000 2 connected 10923-16383
e38d78fc68b69d4e5f347cf96f31a9cb6a02dccf 192.168.56.101:9003@19003 master - 0 1526649622963 7 connected 5461-10922

kill某个redis master 进程

# ps -ef|grep -i redis
root     11301     1  0 21:09 ?        00:00:00 redis-server 192.168.56.101:9001 [cluster]
root     11330     1  0 21:10 ?        00:00:00 redis-server 192.168.56.101:9002 [cluster]
root     11340     1  0 21:10 ?        00:00:00 redis-server 192.168.56.101:9003 [cluster]
root     11358  1851  0 21:11 pts/0    00:00:00 redis-cli -h 192.168.56.101 -p 9001 -c
root     11484  5873  0 21:21 pts/1    00:00:00 grep --color=auto -i redis

# kill 11301

192.168.56.101:9002> cluster nodes
8240d131c58c8b687c11462e03825bdba00af53b 192.168.56.101:9001@19001 master,fail - 1526649753764 1526649753061 1 disconnected
12e2353c9f0b085d2684da46ba0dc69575d68603 192.168.56.102:9001@19001 slave e38d78fc68b69d4e5f347cf96f31a9cb6a02dccf 0 1526649779186 7 connected
0bdd8387a662984e95f5103522f52e75eceaab69 192.168.56.101:9002@19002 myself,master - 0 1526649778000 2 connected 10923-16383
e38d78fc68b69d4e5f347cf96f31a9cb6a02dccf 192.168.56.101:9003@19003 master - 0 1526649776000 7 connected 5461-10922
2b48fd08a8ec55eba5bfa0d163d02c8462637302 192.168.56.102:9002@19002 slave 0bdd8387a662984e95f5103522f52e75eceaab69 0 1526649777173 5 connected
cf22edbc6463b8b31536819297ac50e9e39175ad 192.168.56.102:9003@19003 master - 0 1526649778179 8 connected 0-5460

可以看到 192.168.56.101:9001 现在是 master,fail 状态

kill 之前redis的状态

master 192.168.56.101:9001、192.168.56.101:9002、192.168.56.101:9003
slave  192.168.56.102:9001、192.168.56.102:9002、192.168.56.102:9003

kill 之后redis的状态

master 192.168.56.101:9002、192.168.56.101:9003、192.168.56.102:9003
slave  192.168.56.102:9001、192.168.56.102:9002

master,fail 192.168.56.101:9001

可以看出，192.168.56.102:9003 充当了新的master，也可以推断kill之前，192.168.56.102:9003 是 192.168.56.101:9001 的slave

从输出求证，确实是这样

cf22edbc6463b8b31536819297ac50e9e39175ad 192.168.56.102:9003@19003 slave 8240d131c58c8b687c11462e03825bdba00af53b 0 1526649623970 6 connected
8240d131c58c8b687c11462e03825bdba00af53b 192.168.56.101:9001@19001 myself,master - 0 1526649623000 1 connected 0-5460

注意 8240d131c58c8b687c11462e03825bdba00af53b

redis的slave在master down后，可以自动提升为master。这个很重要。

重新启动被kill的那个redis

# redis-server /usr/redis/redis-cluster-data/9001/redis.conf
[root@pgsql1 9001]# redis-cli -h 192.168.56.101 -p 9001 -c
192.168.56.101:9001>
192.168.56.101:9001> cluster nodes
12e2353c9f0b085d2684da46ba0dc69575d68603 192.168.56.102:9001@19001 slave e38d78fc68b69d4e5f347cf96f31a9cb6a02dccf 0 1526650968543 7 connected
2b48fd08a8ec55eba5bfa0d163d02c8462637302 192.168.56.102:9002@19002 slave 0bdd8387a662984e95f5103522f52e75eceaab69 0 1526650969546 5 connected
cf22edbc6463b8b31536819297ac50e9e39175ad 192.168.56.102:9003@19003 master - 0 1526650967000 8 connected 0-5460
8240d131c58c8b687c11462e03825bdba00af53b 192.168.56.101:9001@19001 myself,slave cf22edbc6463b8b31536819297ac50e9e39175ad 0 1526650967000 1 connected
e38d78fc68b69d4e5f347cf96f31a9cb6a02dccf 192.168.56.101:9003@19003 master - 0 1526650969000 7 connected 5461-10922
0bdd8387a662984e95f5103522f52e75eceaab69 192.168.56.101:9002@19002 master - 0 1526650966534 2 connected 10923-16383
192.168.56.101:9001> set peiyb1 abc
-> Redirected to slot [6161] located at 192.168.56.101:9003
OK
192.168.56.101:9001> 

此时，redis各节点的状态为

master 192.168.56.101:9002、192.168.56.101:9003、192.168.56.102:9003
slave  192.168.56.102:9001、192.168.56.102:9002、192.168.56.101:9001

可以看出，新启动的redis 192.168.56.101:9001@19001 状态为slave，并没有抢夺master的权限。
此外，在slave上操作 set 时，能自动计算 hash slot ，抛给对应的master执行。