Redis Cluster集群

一、环境说明

服务器IP地址	服务器版本	Redis版本	Redis-master	Redis-slave
10.10.18.10	centos7.5	5.0	6379	6380
10.10.18.11	centos7.5	5.0	6379	6380
10.10.18.12	centos7.5	5.0	6379	6380

搭建redis集群，建议至少准备3台服务器，共搭建6个节点，3个master,3个slave，并且要求3个master节点不能全部跑到同一台服务器上，保证节点安全，3台服务器的配置相同， 每个节点运行两个端口。第一列做主库，第二列做备库

每个Redis群集节点都需要打开两个TCP连接。用于为客户端提供服务的普通Redis TCP端口，例如6379，加上通过向数据端口添加10000获得的端口，因此示例中为16379。

默认数据存放结构图：

这个结构会随着机器的故障自动调整master的角色，有可能原来是slave的角色，会变成master。

三个slave可以都宕机，数据没有问题

Redis Cluster提供了一种运行Redis安装的方法，其中数据 在多个Redis节点之间自动分片。

Redis Cluster还在分区期间提供一定程度的可用性，实际上是在某些节点发生故障或无法通信时继续运行的能力。但是，如果发生较大的故障（例如，当大多数主设备不可用时），群集将停止运行。

二、系统环境配置

系统参数配置

修改最大可打开文件数

修改文件/etc/security/limits.conf

* soft nofile 102400
* hard nofile 102400

TCP监听队列大小

文件/etc/sysctl.conf中增加一行

net.core.somaxconn = 32767
vm.overcommit_memory=1

然后执行命令“sysctl -p”以生效。

三、安装部署

下载redis5.0以上的版本

http://download.redis.io/releases/redis-5.0.3.tar.gz

tar -xzvf redis-5.0.3.tar.gz –C /usr/local
cd /usr/local/redis-5.0.3
yum -y install gcc
make MALLOC=libc
mkdir –p /data/redis_data/
make install PREFIX=/data/redis_data/
mkdir –p /data/redis_data/6379/

cp /usr/local/redis-5.0.3/redis.conf /data/redis_data/6379/

修改：/data/redis_data/6379/redis.conf

bind 10.10.18.10
protected-mode no
port 6379
daemonize yes
cluster-enabled yes
cluster-config-file  /data/redis_data/6379/node.conf     #这个文件是自动生成的文件
cluster-node-timeout 5000
pidfile /data/redis_data/6379/redis.pid
logfile  "/data/redis_data/6379/redis.log"
dir /data/redis_data/6379

tcp-backlog 511
timeout 0
tcp-keepalive 300
supervised no
loglevel notice
databases 16
always-show-logo yes
save 900 1
save 300 10
save 60 10000
stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes
dbfilename dump.rdb
replica-serve-stale-data yes
replica-read-only yes
repl-diskless-sync no
repl-diskless-sync-delay 5
repl-disable-tcp-nodelay no
replica-priority 100
lazyfree-lazy-eviction no
lazyfree-lazy-expire no
lazyfree-lazy-server-del no
replica-lazy-flush no
appendonly yes
appendfilename "appendonly.aof"
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
aof-load-truncated yes
aof-use-rdb-preamble yes
lua-time-limit 5000
slowlog-log-slower-than 10000
slowlog-max-len 128
latency-monitor-threshold 0
notify-keyspace-events ""
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
list-max-ziplist-size -2
list-compress-depth 0
set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
hll-sparse-max-bytes 3000
stream-node-max-bytes 4096
stream-node-max-entries 100
activerehashing yes
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit replica 256mb 64mb 60
client-output-buffer-limit pubsub 32mb 8mb 60
hz 10
dynamic-hz yes
aof-rewrite-incremental-fsync yes
rdb-save-incremental-fsync yes

修改：/data/redis_data/6379/redis.conf

mkdir –p /data/redis_data/6380/
cp /data/redis_data/6379/redis.conf  /data/redis_data/6380/
sed -i 's/6379/6380/g' /data/redis_data/6380/redis.conf

编写启动脚本 /data/redis_data/start-redis.sh

#!/bin/sh
REDIS_HOME=/data/redis_data
$REDIS_HOME/bin/redis-server $REDIS_HOME/6379/redis.conf
$REDIS_HOME/bin/redis-server $REDIS_HOME/6380/redis.conf

chmod +x /data/redis_data/start-redis.sh

开启服务

查看各服务器的进程情况

[root@redis01]# netstat -anpt |grep redis
tcp        0      0 10.10.18.12:16379       0.0.0.0:*               LISTEN      4328/redis-server 1 
tcp        0      0 10.10.18.12:16380       0.0.0.0:*               LISTEN      4330/redis-server 1 
tcp        0      0 10.10.18.12:6379        0.0.0.0:*               LISTEN      4328/redis-server 1 
tcp        0      0 10.10.18.12:6380        0.0.0.0:*               LISTEN      4330/redis-server 1

创建启动集群脚本  /data/redis_data/redis-cluster.sh

#!/bin/bash
/data/redis_data/bin/redis-cli --cluster create 10.10.18.10:6379 10.10.18.11:6379 10.10.18.12:6379 10.10.18.10:6380 10.10.18.11:6380 10.10.18.12:6380 --cluster-replicas 1
参数说明
--cluster create：表示创建redis集群
--cluster-replicas 1：表示为集群中的每一个主节点指定一个从节点，即一比一的复制。

三台服务器上都使用以上的安装步骤。

在任意一台服务器上执行，开启集群服务：

[root@redis01]# sh redis-cluster.sh

结果

>>> Performing hash slots allocation on 6 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica 10.10.18.11:6380 to 10.10.18.10:6379
Adding replica 10.10.18.10:6380 to 10.10.18.11:6379
Adding replica 10.10.18.12:6380 to 10.10.18.12:6379
>>> Trying to optimize slaves allocation for anti-affinity
[OK] Perfect anti-affinity obtained!
M: c81b9adc68d5373237e9eaf3e5809bef42c5471a 10.10.18.10:6379
   slots:[0-5460] (5461 slots) master
M: c4a495f621576910d934fa04da0896b39c0adf7a 10.10.18.11:6379
   slots:[5461-10922] (5462 slots) master
M: baa5b1e2a769da1e37e3a4179ca362c9a26698c8 10.10.18.12:6379
   slots:[10923-16383] (5461 slots) master
S: 8d315d8cc7f42afc5ac641003714f5e285822bff 10.10.18.10:6380
   replicates c4a495f621576910d934fa04da0896b39c0adf7a
S: a7096a413407e77431c0d852c5106860d6d3184a 10.10.18.11:6380
   replicates baa5b1e2a769da1e37e3a4179ca362c9a26698c8
S: 7f0c0858330bf107276d27c10f3e9c600aefa8ad 10.10.18.12:6380
   replicates c81b9adc68d5373237e9eaf3e5809bef42c5471a
Can I set the above configuration? (type 'yes' to accept): yes         这里输入yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
....
>>> Performing Cluster Check (using node 10.10.18.10:6379)
M: c81b9adc68d5373237e9eaf3e5809bef42c5471a 10.10.18.10:6379
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
M: c4a495f621576910d934fa04da0896b39c0adf7a 10.10.18.11:6379
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
S: 8d315d8cc7f42afc5ac641003714f5e285822bff 10.10.18.10:6380
   slots: (0 slots) slave
   replicates c4a495f621576910d934fa04da0896b39c0adf7a
S: 7f0c0858330bf107276d27c10f3e9c600aefa8ad 10.10.18.12:6380
   slots: (0 slots) slave
   replicates c81b9adc68d5373237e9eaf3e5809bef42c5471a
S: a7096a413407e77431c0d852c5106860d6d3184a 10.10.18.11:6380
   slots: (0 slots) slave
   replicates baa5b1e2a769da1e37e3a4179ca362c9a26698c8
M: baa5b1e2a769da1e37e3a4179ca362c9a26698c8 10.10.18.12:6379
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

查看各服务器redis的状态

[root@redis01]# netstat -anpt | grep redis

结果：

tcp        0      0 10.10.18.12:16379       0.0.0.0:*               LISTEN      4328/redis-server 1 
tcp        0      0 10.10.18.12:16380       0.0.0.0:*               LISTEN      4330/redis-server 1 
tcp        0      0 10.10.18.12:6379        0.0.0.0:*               LISTEN      4328/redis-server 1 
tcp        0      0 10.10.18.12:6380        0.0.0.0:*               LISTEN      4330/redis-server 1 
tcp        0      0 10.10.18.12:46543       10.10.18.11:16380       ESTABLISHED 4330/redis-server 1 
tcp        0      0 10.10.18.12:16379       10.10.18.10:40956       ESTABLISHED 4328/redis-server 1 
tcp        0      0 10.10.18.12:16380       10.10.18.11:40559       ESTABLISHED 4330/redis-server 1 
tcp        0      0 10.10.18.12:45872       10.10.18.10:16379       ESTABLISHED 4330/redis-server 1 
tcp        0      0 10.10.18.12:45755       10.10.18.10:6379        ESTABLISHED 4330/redis-server 1 
tcp        0      0 10.10.18.12:16379       10.10.18.12:36457       ESTABLISHED 4328/redis-server 1 
tcp        0      0 10.10.18.12:39163       10.10.18.10:16379       ESTABLISHED 4328/redis-server 1 
tcp        0      0 10.10.18.12:43185       10.10.18.10:16380       ESTABLISHED 4328/redis-server 1 
tcp        0      0 10.10.18.12:42254       10.10.18.10:16380       ESTABLISHED 4330/redis-server 1 
tcp        0      0 10.10.18.12:35352       10.10.18.11:16379       ESTABLISHED 4330/redis-server 1 
tcp        0      0 10.10.18.12:41761       10.10.18.11:16379       ESTABLISHED 4328/redis-server 1 
tcp        0      0 10.10.18.12:36457       10.10.18.12:16379       ESTABLISHED 4330/redis-server 1 
tcp        0      0 10.10.18.12:16380       10.10.18.11:37557       ESTABLISHED 4330/redis-server 1 
tcp        0      0 10.10.18.12:6379        10.10.18.11:46722       ESTABLISHED 4328/redis-server 1 
tcp        0      0 10.10.18.12:16380       10.10.18.10:39318       ESTABLISHED 4330/redis-server 1 
tcp        0      0 10.10.18.12:16379       10.10.18.10:35113       ESTABLISHED 4328/redis-server 1 
tcp        0      0 10.10.18.12:16380       10.10.18.10:40872       ESTABLISHED 4330/redis-server 1 
tcp        0      0 10.10.18.12:16380       10.10.18.12:34377       ESTABLISHED 4330/redis-server 1 
tcp        0      0 10.10.18.12:16379       10.10.18.11:37717       ESTABLISHED 4328/redis-server 1 
tcp        0      0 10.10.18.12:43984       10.10.18.11:16380       ESTABLISHED 4328/redis-server 1 
tcp        0      0 10.10.18.12:16379       10.10.18.11:45994       ESTABLISHED 4328/redis-server 1 
tcp        0      0 10.10.18.12:34377       10.10.18.12:16380       ESTABLISHED 4328/redis-server 1

四、测试

设置redis密码

[root@redis01 bin]# /data/redis_data/bin/redis-cli -h 10.10.18.10 -p 6379 -c 

结果：

10.10.18.10:6379> keys *
(empty list or set)
10.10.18.10:6379> config set masterauth redis20190628
OK
10.10.18.10:6379> config set requirepass redis20190628
OK
10.10.18.10:6379> exit

查看所有的节点信息：

[root@redis01]# /data/redis_data/bin/redis-cli -h 10.10.18.10 -p 6380 -c -a 'redis20190628' cluster nodes

结果：

Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
c4a495f621576910d934fa04da0896b39c0adf7a 10.10.18.11:6379@16379 master - 0 1551081545000 2 connected 5461-10922
c81b9adc68d5373237e9eaf3e5809bef42c5471a 10.10.18.10:6379@16379 master - 0 1551081544292 1 connected 0-5460
8d315d8cc7f42afc5ac641003714f5e285822bff 10.10.18.10:6380@16380 myself,slave c4a495f621576910d934fa04da0896b39c0adf7a 0 1551081543000 4 connected
baa5b1e2a769da1e37e3a4179ca362c9a26698c8 10.10.18.12:6379@16379 master - 0 1551081543000 3 connected 10923-16383
7f0c0858330bf107276d27c10f3e9c600aefa8ad 10.10.18.12:6380@16380 slave c81b9adc68d5373237e9eaf3e5809bef42c5471a 0 1551081544292 6 connected
a7096a413407e77431c0d852c5106860d6d3184a 10.10.18.11:6380@16380 slave baa5b1e2a769da1e37e3a4179ca362c9a26698c8 0 1551081543292 5 connected

添加测试数据

在任意一台机器上执行插入数据，然后在其它任意节点上都能获取到对应的值

[root@redis01]# /data/redis_data/bin/redis-cli -h 10.10.18.10 -p 6380 -c -a 'redis20190628'   
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
10.10.18.10:6380> set test10_6380 test
[root@redis01]# /data/redis_data/bin/redis-cli -h 10.10.18.10 -p 6379 -c -a 'redis20190628'  
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
10.10.18.10:6379> get test10_6380
"test"

[root@redis02]# /data/redis_data/bin/redis-cli -h 10.10.18.11 -p 6380 -c -a 'redis20190628' 
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
10.10.18.11:6380> get test10_6380
-> Redirected to slot [2317] located at 10.10.18.10:6379
"test"