Redis Cluster部署手册

Redis Cluster：Redis Cluster提供了一种真正的实时的高可用解决方案，在Redis Cluster集群中，任意一个节点的宕机均不会对服务产生影响，slave角色的节点可以无缝接替宕机的master节点对外继续提供服务。

例如，在一个三主三从的Redis Cluster中，Redis Cluster供定义了16384(0~16383)个slots（槽），并将这些slots均匀分布Redis Cluster的三个主节点中，如0~5000个slots属于主节点A，5001~10000个slots属于主节点B，10001~16383个slot属于主节点C，Redis中任意一个key经过CRC16校验后的校验值始终在0~16383范围中，Redis Cluster根据该key的校验值将该Key存入和该slot值一致的节点上，A1、B1、C1分别为A、B、C的从节点，自动从对应的主节点同步slots信息和slots中的key数据，当主节点宕机后，从节点立刻执行故障转义变成主节点继续提供服务。

一个三主三从的Redis Cluster如下所示

我们以三主三从的架构进行Redis Cluster环境部署

主机名	IP地址	角色	安装目录	版本
redis-master-01	172.16.101.54	Master	/usr/local/redis	Version 3.12
redis-master-02	172.16.101.55	Master
redis-master-03	172.16.101.56	Master
redis-slave-01	172.16.101.58	Slave
redis-slave-02	172.16.101.59	Slave
redis-slave-03	172.16.101.60	Slave

一.各节点系统环境配置

1.新建运行redis的用户和组

# groupadd -r dba
# useradd -r -g dba -G root redis
# cat /home/redis/.bash_profile
................
PATH=/usr/local/redis/src:$PATH:$HOME/.local/bin:$HOME/bin
................

2.系统内核参数优化

# cat /etc/sysctl.conf
.......................
vm.overcommit_memory=1
vm.swappiness=1
fs.aio-max-nr=1048576
fs.file-max= 7672460
net.ipv4.ip_local_port_range=9000 65500
net.core.rmem_default=262144
net.core.rmem_max=4194304
net.core.wmem_default=262144
net.core.wmem_max=1048586
net.core.somaxconn = 2048
kernel.sem= 50100 64128000 50100 1280
kernel.shmall=5242880
kernel.shmmax=12884901888
.......................

3. redis运行资源优化

# cat /etc/security/limits.conf
..................................
redis               soft    nproc           65536
redis               hard    nproc           65536
redis               soft    nofile          65536
redis               hard    nofile          65536
..................................

4. 大页面文件优化

# echo "echo never > /sys/kernel/mm/transparent_hugepage/enabled" >> /etc/rc.local

二.各节点redis安装

# mkdir /usr/local/redis
# chown -R redis.dba /usr/local/redis
$ cd /usr/local/redis
$ tar -zxf redis-3.2.12.tar.gz 
$ mv redis-3.2.12/* .
$ rm -rf redis-3.2.12*
$ make
$ mkdir {data,log,conf}
$ mv redis.conf conf

三. 配置各节点redis配置文件，如下所示，这里仅列出主要参数

bind 0.0.0.0
protected-mode no
port 6379
tcp-backlog 1024
timeout 0
tcp-keepalive 0
daemonize yes
supervised systemd
pidfile /var/run/redis_6379.pid
loglevel notice
logfile "/usr/local/redis/log/redis.log"
dir "/usr/local/redis/data"
slave-read-only yes
repl-ping-slave-period 3
repl-timeout 15
slave-priority 100
min-slaves-to-write 1
min-slaves-max-lag 10
rename-command flushall ""
rename-command flushdb ""
rename-command save ""
rename-command keys ""
maxclients 10000
maxmemory 4gb
maxmemory-policy allkeys-lru
appendonly yes
appendfilename "appendonly.aof"
appendfsync everysec
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 5000
cluster-slave-validity-factor 10
cluster-migration-barrier 1
cluster-require-full-coverage yes

四.将redis配置为系统服务

# cat /usr/lib/systemd/system/redis.service 
[Unit]
Description=Redis In-Memory Data Store
Documentation=https://redis.io/documentation
After=syslog.target
After=network.target

[Service]
Type=notify
User=redis
Group=dba
Restart=always

# Disable OOM kill on the Redis
OOMScoreAdjust=-1000
Environment=PG_OOM_ADJUST_FILE=/proc/self/oom_score_adj
Environment=PG_OOM_ADJUST_VALUE=0

ExecStart=/usr/local/redis/src/redis-server /usr/local/redis/conf/redis.conf
ExecStop=/bin/kill -15 $MAINPID
LimitNOFILE=65536
 

[Install]
WantedBy=multi-user.target

# systemctl daemon-reload
# systemctl enable redis
# systemctl start redis
# systemctl status redis

 各节点启动成功后，会自动生成一个长度为40字节的唯一的node-id,且在默认开启集群功能后，各节点均认为自己是主节点，从log中我们可以看出

172.16.101.54

25630:M 12 Apr 00:59:59.477 * No cluster configuration found, I'm 89a83af689b194b18c7f5c0dae105c329a6a831f

172.16.101.55

6232:M 12 Apr 00:59:59.450 * No cluster configuration found, I'm 6c9558d05f3d5a1b2dac4ebe31d581b1581f6112

172.16.101.56

28099:M 12 Apr 00:59:59.478 * No cluster configuration found, I'm 684908fde2ccc85fa2caf2fbbbfaa0e9f470cf04

172.16.101.58

25490:M 12 Apr 01:01:07.832 * No cluster configuration found, I'm 310edd63d50d3d1412e6413b9c4d2ff31e2fae5f

172.16.101.59

27836:M 12 Apr 01:01:07.835 * No cluster configuration found, I'm 63eef92da475574c62485543ac57898d13618520

172.16.101.60

15314:M 12 Apr 01:01:07.839 * No cluster configuration found, I'm c7095b2f9adf4adada4fb0afd953ab7439ccc058

 查看当前集群节点信息

redis-master-01:6379> info cluster
# Cluster
cluster_enabled:1
redis-master-01:6379> cluster info
cluster_state:fail
cluster_slots_assigned:0
cluster_slots_ok:0
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:1
cluster_size:0
cluster_current_epoch:0
cluster_my_epoch:0
cluster_stats_messages_sent:0
cluster_stats_messages_received:0
redis-master-01:6379> cluster nodes
89a83af689b194b18c7f5c0dae105c329a6a831f :6379 myself,master - 0 0 0 connected

通过以上可以看出

• 当前集群功能已经开启，cluster_enabled:1
• 集群功能fail状态，cluster_state:fail
• 集群未分配任何slots，cluster_slots_assigned:0
• 集群中只能查看自身节点的简单信息，cluster_known_nodes:1、 89a83af689b194b18c7f5c0dae105c329a6a831f :6379 myself,master - 0 0 0 connected

五.集群部署

1. 将各个节点加入集群

可在任意一个主节点操作，如在redis-master-01节点上，将其他所有节点添加到集群中

redis-master-01:6379> cluster meet 172.16.101.55 6379
redis-master-01:6379> cluster meet 172.16.101.56 6379
redis-master-01:6379> cluster meet 172.16.101.58 6379
redis-master-01:6379> cluster meet 172.16.101.59 6379
redis-master-01:6379> cluster meet 172.16.101.60 6379

添加完之后再次查看集群节点信息

redis-master-01:6379> cluster info
.............................
cluster_known_nodes:6
.............................

redis-master-01:6379> cluster nodes
6c9558d05f3d5a1b2dac4ebe31d581b1581f6112 172.16.101.55:6379 master - 0 1586625723577 1 connected
c7095b2f9adf4adada4fb0afd953ab7439ccc058 172.16.101.60:6379 master - 0 1586625721574 0 connected
684908fde2ccc85fa2caf2fbbbfaa0e9f470cf04 172.16.101.56:6379 master - 0 1586625723076 2 connected
63eef92da475574c62485543ac57898d13618520 172.16.101.59:6379 master - 0 1586625722075 4 connected
89a83af689b194b18c7f5c0dae105c329a6a831f 172.16.101.54:6379 myself,master - 0 0 5 connected
310edd63d50d3d1412e6413b9c4d2ff31e2fae5f 172.16.101.58:6379 master - 0 1586625723577 3 connected

2. 为各个主节点分配slots

[redis@redis-master-01 ~]$ redis-cli -h redis-master-01 cluster addslots {0..5000}
[redis@redis-master-01 ~]$ redis-cli -h redis-master-02 cluster addslots {5001..10000}
[redis@redis-master-01 ~]$ redis-cli -h redis-master-03 cluster addslots {10001..16383}

这里需要注意的点：

• 如果一个slot已经分配给一个主节点后，无法再分配给另外一个节点的，否则会报错“(error) ERR Slot * is already busy”
• slot的最大数量为16384（0~16383），如果分配的数量大于16384，会报错“(error) ERR Invalid or out of range slot”

再次查看当前集群信息

redis-master-01:6379> cluster info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:5
cluster_my_epoch:5
cluster_stats_messages_sent:2573
cluster_stats_messages_received:2573

redis-master-01:6379> cluster nodes
6c9558d05f3d5a1b2dac4ebe31d581b1581f6112 172.16.101.55:6379 master - 0 1586626222109 1 connected 5001-10000
c7095b2f9adf4adada4fb0afd953ab7439ccc058 172.16.101.60:6379 master - 0 1586626221105 0 connected
684908fde2ccc85fa2caf2fbbbfaa0e9f470cf04 172.16.101.56:6379 master - 0 1586626222612 2 connected 10001-16383
63eef92da475574c62485543ac57898d13618520 172.16.101.59:6379 master - 0 1586626220600 4 connected
89a83af689b194b18c7f5c0dae105c329a6a831f 172.16.101.54:6379 myself,master - 0 0 5 connected 0-5000
310edd63d50d3d1412e6413b9c4d2ff31e2fae5f 172.16.101.58:6379 master - 0 1586626221607 3 connected

通过以上可以看出，当前集群共分配了16384个slots，且16484个slots均在线，每个主节点分配的slots范围如“cluster nodes”返回值中所示。我们并未给172.16.101.58/59/60节点分配slots，所以上图我们看到该三个节点依然没有任何slots产生。

此时，一个三主节点的集群已经可以正常提供服务，但是任何一个主节点宕机之后都会导致集群失效。

3. 为每个主节点添加对应的从节点

添加从节点需要在每个从节点上操作

将redis-slave-01添加为主节点redis-master-01的从节点，在redis-slave-01上操作

redis-slave-01:6379> cluster replicate 89a83af689b194b18c7f5c0dae105c329a6a831f

其中89a83af689b194b18c7f5c0dae105c329a6a831f为主节点redis-master-01的node-id

同理将redis-slave-02/03添加为redis-master-02/03的从节点

redis-slave-02:6379> cluster replicate 6c9558d05f3d5a1b2dac4ebe31d581b1581f6112

redis-slave-03:6379> cluster replicate 684908fde2ccc85fa2caf2fbbbfaa0e9f470cf04

 添加完成之后，我们再次查看集群信息

[redis@redis-master-01 ~]$ redis-cli -h redis-master-01 cluster nodes | sort -k 2
89a83af689b194b18c7f5c0dae105c329a6a831f 172.16.101.54:6379 myself,master - 0 0 5 connected 0-5000
6c9558d05f3d5a1b2dac4ebe31d581b1581f6112 172.16.101.55:6379 master - 0 1586627102928 1 connected 5001-10000
684908fde2ccc85fa2caf2fbbbfaa0e9f470cf04 172.16.101.56:6379 master - 0 1586627103932 2 connected 10001-16383
310edd63d50d3d1412e6413b9c4d2ff31e2fae5f 172.16.101.58:6379 slave 89a83af689b194b18c7f5c0dae105c329a6a831f 0 1586627103932 5 connected
63eef92da475574c62485543ac57898d13618520 172.16.101.59:6379 slave 6c9558d05f3d5a1b2dac4ebe31d581b1581f6112 0 1586627102426 4 connected
c7095b2f9adf4adada4fb0afd953ab7439ccc058 172.16.101.60:6379 slave 684908fde2ccc85fa2caf2fbbbfaa0e9f470cf04 0 1586627102928 2 connected

此时，一个三主三从的Redis Cluster已经部署完成。