redis集群搭建

 

需求

搭建一个redis集群，支持故障恢复,。

Redis内置的一个集群需要至少3个主节点，所以搭建规模为3主3个从的集群用于测试。节点信息列表：

 

role	ip	port	bus port	remark
master1	192.168.201.209	7000	17000	主节点1
master2	192.168.201.209	7001	17001	主节点2
master3	192.168.201.209	7002	17002	主节点3
replica1	192.168.201.209	7003	17003	从节点1
replica1	192.168.201.209	7004	17004	从节点2
replica1	192.168.201.209	7005	17005	从节点3
sentinel1	192.168.201.209	6000	-	哨兵1
sentinel2	192.168.201.209	6001	-	哨兵2
sentinel3	192.168.201.209	6002	-	哨兵3

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

环境：

• 镜像redis:alpine，redis的版本为5.0.3
• macOS 10.13.4
• Docker 18.03.1-ce

 

 

 

docker中用到的配置文件命名

➜  config pwd
/Users/cymin/redis/config
➜  config tree .
.
├── m1.conf
├── m2.conf
├── m3.conf
├── r1.conf
├── r2.conf
├── r3.conf
├── s1.conf
├── s2.conf
└── s3.conf
 
0 directories, 9 files

　　

Note that:

• Masters are called M1, M2, M3, ..., Mn.
• Slaves are called R1, R2, R3, ..., Rn (R stands for replica).
• Sentinels are called S1, S2, S3, ..., Sn.

 

 

3个redis master实例配置

➜  config cat m1.conf
cluster-announce-ip 192.168.201.209
cluster-announce-port 7000
cluster-announce-bus-port 17000
 
cluster-enabled yes
cluster-config-file "/tmp/nodes.conf"
cluster-node-timeout 5000
appendonly yes

　　

剩余的master端口依次+1, cluster-announce-bus-port的配置+10000.

说明：

• cluster-announce-ip 告诉redis实例外部映射地址
• cluster-announce-port 告诉redis实例外部映射端口
• cluster-announce-bus-port 开启集群总线端口,值=cluster-announce-port +10000
• masterauth | requirepass 密码，master和replica需要配置为一样的密码
• 其他都是基本配置,相关文档：https://redis.io/topics/cluster-tutorial

 

 

 

3个redis replica(slave)实例配置

➜  config cat r1.conf
cluster-announce-ip 192.168.201.209
cluster-announce-port 7003
cluster-announce-bus-port 17003
 
cluster-enabled yes
cluster-config-file "/tmp/nodes.conf"
cluster-node-timeout 5000
appendonly yes

　　

剩余的cluster-announce-port端口依次+1，cluster-announce-bus-port的配置+10000。

sentinel实例之间的通讯端口默认是26379.

相关文档：https://redis.io/topics/replication

 

 

 

3个哨兵实例配置

➜  config cat s1.conf
sentinel announce-ip 192.168.201.209
sentinel announce-port 6000
 
sentinel monitor master1 192.168.201.209 7000 2
sentinel down-after-milliseconds master1 60000
sentinel failover-timeout master1 180000
sentinel parallel-syncs master1 1
 
sentinel monitor master2 192.168.201.209 7001 2
sentinel down-after-milliseconds master2 60000
sentinel failover-timeout master2 180000
sentinel parallel-syncs master2 1
 
sentinel monitor master3 192.168.201.209 7002 2
sentinel down-after-milliseconds master3 60000
sentinel failover-timeout master3 180000
sentinel parallel-syncs master3 1　

剩余的announce-port配置依次加1.

说明：每个哨兵只需要监控master节点即可，配置项说明见文档：https://redis.io/topics/sentinel

 

 

 

启动redis实例和哨兵

run.sh

# 3个master
docker run --name master1 -tid --network service -v $HOME/redis/config:/opt/config -p 7000:6379 -p 17000:16379  redis:alpine redis-server /opt/config/m1.conf
docker run --name master2 -tid --network service -v $HOME/redis/config:/opt/config -p 7001:6379 -p 17001:16379  redis:alpine redis-server /opt/config/m2.conf
docker run --name master3 -tid --network service -v $HOME/redis/config:/opt/config -p 7002:6379 -p 17002:16379  redis:alpine redis-server /opt/config/m3.conf
 
# 3个replica
docker run --name replica1 -tid --network service -v $HOME/redis/config:/opt/config -p 7003:6379 -p 17003:16379  redis:alpine redis-server /opt/config/r1.conf
docker run --name replica2 -tid --network service -v $HOME/redis/config:/opt/config -p 7004:6379 -p 17004:16379  redis:alpine redis-server /opt/config/r2.conf
docker run --name replica3 -tid --network service -v $HOME/redis/config:/opt/config -p 7005:6379 -p 17005:16379  redis:alpine redis-server /opt/config/r3.conf
 
# 3个sentinel
docker run --name sentinel1 -tid --network service -v $HOME/redis/config:/opt/config -p 6000:26379 redis:alpine redis-sentinel /opt/config/s1.conf
docker run --name sentinel2 -tid --network service -v $HOME/redis/config:/opt/config -p 6001:26379 redis:alpine redis-sentinel /opt/config/s2.conf
docker run --name sentinel3 -tid --network service -v $HOME/redis/config:/opt/config -p 6002:26379 redis:alpine redis-sentinel /opt/config/s3.conf
 

　　

 

 

创建集群

docker exec -it master1 sh
/data # redis-cli --cluster create 192.168.201.209:7000 192.168.201.209:7001 192.168.201.209:7002 192.168.201.209:7003 192.168.201.209:7004 192.168.201.209:7005 --cluster-replicas 1

最后收到"[OK] All 16384 slots covered.”的日志时，集群搭建成功。

 

 

 

连接集群中某个redis实例

redis-cli -c -h 192.168.201.209 -p 7000 

　　

查看集群状态

redis-cli cluster nodes

 

 

查看某个节点的主从状态

redis-cli -h 192.168.201.209 -p 7000 info Replication

 

 

查看某个哨兵的状态

redis-cli -h 192.168.201.209 -p 6000 info Sentinel

　　

 

故障自动恢复测试

# 集群节点状态
/data # redis-cli cluster nodes
b22123911ffed94e6827a94c55891a4488dbbf5c 192.168.201.209:7003@17003 slave 507f3ec4b24d8d380b4da0c107214dae5b4c08b5 0 1547716358000 4 connected
da63ff7af84ba911d905a8a2e0494520e1eaa6c9 192.168.201.209:7004@17004 slave 4de60b987e84de81a6fce10d68f4b1ba0d2ac28a 0 1547716357525 5 connected
507f3ec4b24d8d380b4da0c107214dae5b4c08b5 192.168.201.209:7001@17001 myself,master - 0 1547716357000 2 connected 5461-10922
e9f1294a466d3c2e4c3b40d59c8237c346c7fed7 192.168.201.209:7005@17005 slave 7d0d00d7710fc416ac2c3f44ea86b306b498038e 0 1547716357014 6 connected
4de60b987e84de81a6fce10d68f4b1ba0d2ac28a 192.168.201.209:7002@17002 master - 0 1547716358535 3 connected 10923-16383
7d0d00d7710fc416ac2c3f44ea86b306b498038e 192.168.201.209:7000@17000 master - 0 1547716356511 1 connected 0-5460
 
# 停止master1
docker stop master1
 
# 集群节点状态
/data # redis-cli cluster nodes
b22123911ffed94e6827a94c55891a4488dbbf5c 192.168.201.209:7003@17003 slave 507f3ec4b24d8d380b4da0c107214dae5b4c08b5 0 1547716754542 4 connected
da63ff7af84ba911d905a8a2e0494520e1eaa6c9 192.168.201.209:7004@17004 slave 4de60b987e84de81a6fce10d68f4b1ba0d2ac28a 0 1547716754542 5 connected
507f3ec4b24d8d380b4da0c107214dae5b4c08b5 192.168.201.209:7001@17001 myself,master - 0 1547716753000 2 connected 5461-10922
e9f1294a466d3c2e4c3b40d59c8237c346c7fed7 192.168.201.209:7005@17005 master - 0 1547716754000 7 connected 0-5460
4de60b987e84de81a6fce10d68f4b1ba0d2ac28a 192.168.201.209:7002@17002 master - 0 1547716754946 3 connected 10923-16383
7d0d00d7710fc416ac2c3f44ea86b306b498038e 192.168.201.209:7000@17000 master,fail - 1547716421213 1547716420405 1 disconnected
 
# 重新启动master1
docker restart master1
 
# 集群节点状态
/data # redis-cli cluster nodes
b22123911ffed94e6827a94c55891a4488dbbf5c 192.168.201.209:7003@17003 slave 507f3ec4b24d8d380b4da0c107214dae5b4c08b5 0 1547716837000 4 connected
da63ff7af84ba911d905a8a2e0494520e1eaa6c9 192.168.201.209:7004@17004 slave 4de60b987e84de81a6fce10d68f4b1ba0d2ac28a 0 1547716837000 5 connected
507f3ec4b24d8d380b4da0c107214dae5b4c08b5 192.168.201.209:7001@17001 myself,master - 0 1547716836000 2 connected 5461-10922
e9f1294a466d3c2e4c3b40d59c8237c346c7fed7 192.168.201.209:7005@17005 master - 0 1547716838053 7 connected 0-5460
4de60b987e84de81a6fce10d68f4b1ba0d2ac28a 192.168.201.209:7002@17002 master - 0 1547716837138 3 connected 10923-16383
7d0d00d7710fc416ac2c3f44ea86b306b498038e 192.168.201.209:7000@17000 slave e9f1294a466d3c2e4c3b40d59c8237c346c7fed7 0 1547716838153 7 connected

观察当id为7d0d00d7710fc416ac2c3f44ea86b306b498038e的master节点挂掉的时候，其下id为e9f1294a466d3c2e4c3b40d59c8237c346c7fed7的slave的节点自动升级为master节点，当之前的master又重新启动的时候自动变成slave节点。

 

 

增加一个节点

# 增加一个redis实例，replica4端口映射到本地为7006
docker run --name replica4 -tid --network service -v $HOME/redis/config:/opt/config -p 7006:6379 -p 17006:16379  redis:alpine redis-server /opt/config/r4.conf
 
# 给id为4de60b987e84de81a6fce10d68f4b1ba0d2ac28a增加一个slave
redis-cli --cluster add-node 192.168.201.209:7006 192.168.201.209:7002 --cluster-slave --cluster-master-id 4de60b987e84de81a6fce10d68f4b1ba0d2ac28a
 
# 查看集群节点状态
/data # redis-cli cluster nodes
b22123911ffed94e6827a94c55891a4488dbbf5c 192.168.201.209:7003@17003 slave 507f3ec4b24d8d380b4da0c107214dae5b4c08b5 0 1547718761000 4 connected
da63ff7af84ba911d905a8a2e0494520e1eaa6c9 192.168.201.209:7004@17004 slave 4de60b987e84de81a6fce10d68f4b1ba0d2ac28a 0 1547718761820 5 connected
507f3ec4b24d8d380b4da0c107214dae5b4c08b5 192.168.201.209:7001@17001 myself,master - 0 1547718761000 2 connected 5461-10922
e9f1294a466d3c2e4c3b40d59c8237c346c7fed7 192.168.201.209:7005@17005 master - 0 1547718761000 7 connected 0-5460
435a5e171bd4ffc02e862d174609fa0d07045925 192.168.201.209:7006@17006 slave 4de60b987e84de81a6fce10d68f4b1ba0d2ac28a 0 1547718761312 3 connected
4de60b987e84de81a6fce10d68f4b1ba0d2ac28a 192.168.201.209:7002@17002 master - 0 1547718760000 3 connected 10923-16383
7d0d00d7710fc416ac2c3f44ea86b306b498038e 192.168.201.209:7000@17000 slave e9f1294a466d3c2e4c3b40d59c8237c346c7fed7 0 1547718761517 7 connected
 

　　

 

 

 

删除一个节点

/data # redis-cli --cluster del-node 192.168.201.209:7000 435a5e171bd4ffc02e862d174609fa0d07045925
>>> Removing node 435a5e171bd4ffc02e862d174609fa0d07045925 from cluster 192.168.201.209:7000
>>> Sending CLUSTER FORGET messages to the cluster...
>>> SHUTDOWN the node.

# 查看集群节点状态
/data # redis-cli cluster nodes
b22123911ffed94e6827a94c55891a4488dbbf5c 192.168.201.209:7003@17003 slave 507f3ec4b24d8d380b4da0c107214dae5b4c08b5 0 1547719609601 4 connected
da63ff7af84ba911d905a8a2e0494520e1eaa6c9 192.168.201.209:7004@17004 slave 4de60b987e84de81a6fce10d68f4b1ba0d2ac28a 0 1547719610512 5 connected
507f3ec4b24d8d380b4da0c107214dae5b4c08b5 192.168.201.209:7001@17001 myself,master - 0 1547719611000 2 connected 5461-10922
e9f1294a466d3c2e4c3b40d59c8237c346c7fed7 192.168.201.209:7005@17005 master - 0 1547719611626 7 connected 0-5460
4de60b987e84de81a6fce10d68f4b1ba0d2ac28a 192.168.201.209:7002@17002 master - 0 1547719610512 3 connected 10923-16383
7d0d00d7710fc416ac2c3f44ea86b306b498038e 192.168.201.209:7000@17000 slave e9f1294a466d3c2e4c3b40d59c8237c346c7fed7 0 1547719610613 7 connected
 

　　

 

 

数据同步测试

/data # redis-cli -c -h 192.168.201.209 -p 7000
192.168.201.209:7000> keys *
(empty list or set)
192.168.201.209:7000> set name cymin
-> Redirected to slot [5798] located at 192.168.201.209:7001
OK
192.168.201.209:7001>
/data # redis-cli -c -h 192.168.201.209 -p 7001
192.168.201.209:7001> keys *
1) "name"
192.168.201.209:7001>
/data # redis-cli -c -h 192.168.201.209 -p 7002
192.168.201.209:7002> keys *
(empty list or set)
192.168.201.209:7002> get name
-> Redirected to slot [5798] located at 192.168.201.209:7001
“cymin”
192.168.201.209:7001>
192.168.201.209:7001> get afawfgawfgw
-> Redirected to slot [15081] located at 192.168.201.209:7002
(nil)
192.168.201.209:7002> get feawgwegewgew
-> Redirected to slot [7926] located at 192.168.201.209:7001
(nil)
192.168.201.209:7001>

观察数据读写均在master节点，如果在replica节点中读写key，则会自动重定向到对应的master节点中。

 

 

集群密码设置

进入各个(6个)实例进行依次设置:

redis-cli -c -h 192.168.201.209 -p 7000
127.0.0.1:7000> config set masterauth passwd123
127.0.0.1:7000> config set requirepass passwd123

 

# 重新进入实例，写入配置

redis-cli -a abc123 -c -h 192.168.201.209 -p 7000
127.0.0.1:7000> config rewrite

 

 

 

java连接集群事例

Jar:

compile group: 'redis.clients', name: 'jedis', version: '3.0.1'

　　 

使用jedis在代码中连接redis集群，因为是集群，可以自动发现，也同样只要一个地址就行了

import org.apache.commons.pool2.impl.GenericObjectPoolConfig;
import redis.clients.jedis.HostAndPort;
import redis.clients.jedis.JedisCluster;
 
import java.util.HashSet;
import java.util.Set;
 
public class RedisTest {
    public static void main(String[] args) {
        testWithPassword();
    }
 
    public static void test() {
        Set<HostAndPort> jedisClusterNodes = new HashSet<>();
        jedisClusterNodes.add(new HostAndPort("192.168.201.208", 7000));
        JedisCluster jc = new JedisCluster(jedisClusterNodes);
        String value = jc.get("name");
        System.out.println(value);
    }
 
 
    public static void testWithPassword() {
        Set<HostAndPort> jedisClusterNodes = new HashSet<>();
        jedisClusterNodes.add(new HostAndPort("192.168.201.208", 7000));
        GenericObjectPoolConfig gopc = new GenericObjectPoolConfig();
        gopc.setMaxTotal(32);
        gopc.setMaxIdle(4);
        JedisCluster jc = new JedisCluster(jedisClusterNodes, 10000, 10000, 3,"abc123", gopc);
        String value = jc.get("name");
        System.out.println(value);
    }
}

　　

 

 

 

 

The end.