redis4.0集群部署

环境
OS:Centos 7
db:4.0.14
主 从
192.168.1.118:7001 192.168.1.85:8001
192.168.1.118:7002 192.168.1.85:8001
192.168.1.118:7003 192.168.1.85:8001

1.下载安装介质
我这里下载的是redis-4.0.14.tar.gz
下载地址:
http://download.redis.io/releases/

-------------------------192.168.1.118上操作------------------------
2.解压介质并安装
[root@localhost soft]# tar -xvf redis-4.0.14.tar.gz
[root@localhost soft]# cd redis-4.0.14
[root@localhost redis-4.0.14]# make
[root@localhost redis-4.0.14]# make test
没有问题的话，会输入如下信息
o/ All tests passed without errors!

Cleanup: may take some time... OK
make[1]: Leaving directory `/soft/redis-4.0.14/src'
You have mail in /var/spool/mail/root

安装到指定的目录,默认是安装在/usr/bin目录,我们这里统一安装到/home/hxl/redis目录,该目录不用提前创建
[root@localhost redis-4.0.14]# cd src
[root@localhost src]# make PREFIX=/home/hxl/redis install
Hint: It's a good idea to run 'make test' ;)
INSTALL install
INSTALL install
INSTALL install
INSTALL install
INSTALL install

拷贝 ./scr/redis-trib.rb到指定的目录,该脚本是用来创建集群用的
[root@localhost src]# cp redis-trib.rb /home/hxl/redis/

3.创建目录并修改相应权限
[root@localhost src]# mkdir -p /home/hxl/redis/conf
[root@localhost src]# mkdir -p /home/hxl/redis/data
[root@localhost src]# mkdir -p /home/hxl/redis/logs
[root@localhost src]# mkdir -p /home/hxl/redis/run
[root@localhost src]# mkdir -p /home/hxl/redis/data/redis_7001
[root@localhost src]# mkdir -p /home/hxl/redis/data/redis_7002
[root@localhost src]# mkdir -p /home/hxl/redis/data/redis_7003
修改权限
[root@localhost hxl]# cd /home/hxl
[root@localhost hxl]# chown -R hxl.hxl ./redis

4.创建配置文件
su - hxl
vi /home/hxl/redis/conf/7001.conf
内如如下:
daemonize yes
bind 192.168.1.118
protected-mode yes
port 7001
tcp-backlog 511
timeout 300
tcp-keepalive 300
supervised no
pidfile "/home/hxl/redis/run/redis_7001.pid"
loglevel notice
logfile "/home/hxl/redis/logs/redis_7001.log"
databases 16
save 900 1
save 300 10
save 60 10000
stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes
dbfilename "dump.rdb"
dir "/home/hxl/redis/data/redis_7001"
slave-serve-stale-data yes
slave-read-only yes
repl-diskless-sync no
repl-diskless-sync-delay 5
repl-disable-tcp-nodelay no
slave-priority 100
appendonly no
appendfilename "appendonly.aof"
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
aof-load-truncated yes
lua-time-limit 5000
cluster-enabled yes
cluster-config-file "7001.conf"
cluster-node-timeout 5000
cluster-slave-validity-factor 10
cluster-migration-barrier 1
cluster-require-full-coverage yes
slowlog-log-slower-than 10000
slowlog-max-len 128
latency-monitor-threshold 0
notify-keyspace-events ""
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
list-max-ziplist-size -2
list-compress-depth 0
set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
hll-sparse-max-bytes 3000
activerehashing yes
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit slave 256mb 64mb 60
client-output-buffer-limit pubsub 32mb 8mb 60
hz 10
aof-rewrite-incremental-fsync yes
masterauth "oracleabc2018"
requirepass "oracleabc2018"

拷贝7001.conf生成7002.conf,7003.conf配置文件
-bash-4.2$ cp 7001.conf 7002.conf
-bash-4.2$ cp 7001.conf 7003.conf

替换修改7002.conf和7003.conf文件内容
sed -i 's/7001/7002/g' /home/hxl/redis/conf/7002.conf
sed -i 's/7001/7003/g' /home/hxl/redis/conf/7003.conf

-------------------------192.168.1.85上操作----------------------
1.解压介质并安装
[root@localhost soft]# tar -xvf redis-4.0.14.tar.gz
[root@localhost soft]# cd redis-4.0.14
[root@localhost redis-4.0.14]# make
[root@localhost redis-4.0.14]# make test

跟上面的节点一样安装到相同的目录
[root@localhost redis-4.0.14]# cd src
[root@localhost src]# make PREFIX=/home/hxl/redis install
[root@localhost src]# cp redis-trib.rb /home/hxl/redis/

2.创建目录并修改相应权限
[root@localhost src]# mkdir -p /home/hxl/redis/conf
[root@localhost src]# mkdir -p /home/hxl/redis/data
[root@localhost src]# mkdir -p /home/hxl/redis/logs
[root@localhost src]# mkdir -p /home/hxl/redis/run
[root@localhost src]# mkdir -p /home/hxl/redis/data/redis_8001
[root@localhost src]# mkdir -p /home/hxl/redis/data/redis_8002
[root@localhost src]# mkdir -p /home/hxl/redis/data/redis_8003
修改权限
[root@localhost hxl]# cd /home/hxl
[root@localhost hxl]# chown -R hxl.hxl ./redis

3.创建配置文件
从192.168.1.118拷贝一份过来进行修改
192.168.1.118上操作
-bash-4.2$ scp 7001.conf hxl@192.168.1.85:/home/hxl/redis/conf/

192.168.1.85上操作
-bash-4.2$ cd /home/hxl/redis/conf
-bash-4.2$ mv 7001.conf 8001.conf
-bash-4.2$ cp 8001.conf 8002.conf
-bash-4.2$ cp 8001.conf 8003.conf

替换内容
sed -i 's/7001/8001/g' /home/hxl/redis/conf/8001.conf
sed -i 's/7001/8002/g' /home/hxl/redis/conf/8002.conf
sed -i 's/7001/8003/g' /home/hxl/redis/conf/8003.conf

修改每个参数文件的bind 192.168.1.118项,让其指向192.1681.85
bind 192.168.1.85

-----------------启动---------------------------------
192.168.1.118
su - hxl
/home/hxl/redis/bin/redis-server /home/hxl/redis/conf/7001.conf
/home/hxl/redis/bin/redis-server /home/hxl/redis/conf/7002.conf
/home/hxl/redis/bin/redis-server /home/hxl/redis/conf/7003.conf

192.168.1.85
su - hxl
/home/hxl/redis/bin/redis-server /home/hxl/redis/conf/8001.conf
/home/hxl/redis/bin/redis-server /home/hxl/redis/conf/8002.conf
/home/hxl/redis/bin/redis-server /home/hxl/redis/conf/8003.conf

查看进程情况
-bash-4.2$ ps -ef|grep redis
hxl 18012 1 0 11:01 ? 00:00:00 /home/hxl/redis/bin/redis-server 192.168.1.118:7001 [cluster]
hxl 18092 1 0 11:02 ? 00:00:00 /home/hxl/redis/bin/redis-server 192.168.1.118:7002 [cluster]
hxl 18104 1 0 11:02 ? 00:00:00 /home/hxl/redis/bin/redis-server 192.168.1.118:7003 [cluster]

-bash-4.2$ ps -ef|grep redis
hxl 28570 1 0 11:03 ? 00:00:00 /home/hxl/redis/bin/redis-server 192.168.1.85:8001 [cluster]
hxl 28614 1 0 11:03 ? 00:00:00 /home/hxl/redis/bin/redis-server 192.168.1.85:8002 [cluster]
hxl 28628 1 0 11:03 ? 00:00:00 /home/hxl/redis/bin/redis-server 192.168.1.85:8003 [cluster]
hxl 28980 26742 0 11:06 pts/2 00:00:00 grep --color=auto redis

----------------------------配置集群-------------------------------------
在其中一台机器操作,我这里是在192.168.1.118上操作
/home/hxl/redis/redis-trib.rb create --replicas 1 192.168.1.118:7001 192.168.1.118:7002 192.168.1.118:7003 192.168.1.85:8001 192.168.1.85:8002 192.168.1.85:8003
>>> Creating cluster
[ERR] Sorry, can't connect to node 192.168.1.118:7001
这里报错，因为我们的每个redis都配置了密码，尝试注释掉密码后启动，等创建完集群后再启用密码

停掉每个节点
192.168.1.118
./redis-cli -h 192.168.1.118 -a oracleabc2018 -p 7001 shutdown
./redis-cli -h 192.168.1.118 -a oracleabc2018 -p 7002 shutdown
./redis-cli -h 192.168.1.118 -a oracleabc2018 -p 7003 shutdown

192.168.1.85
./redis-cli -h 192.168.1.85 -a oracleabc2018 -p 8001 shutdown
./redis-cli -h 192.168.1.85 -a oracleabc2018 -p 8002 shutdown
./redis-cli -h 192.168.1.85 -a oracleabc2018 -p 8003 shutdown

把每个节点的配置文件如下两项先注释掉
##masterauth "oracleabc2018"
##requirepass "oracleabc2018"

然后再次启动.

重新创建集群
-bash-4.2$ /home/hxl/redis/redis-trib.rb create --replicas 1 192.168.1.118:7001 192.168.1.118:7002 192.168.1.118:7003 192.168.1.85:8001 192.168.1.85:8002 192.168.1.85:8003
>>> Creating cluster
>>> Performing hash slots allocation on 6 nodes...
Using 3 masters:
192.168.1.118:7001
192.168.1.85:8001
192.168.1.118:7002
Adding replica 192.168.1.85:8003 to 192.168.1.118:7001
Adding replica 192.168.1.118:7003 to 192.168.1.85:8001
Adding replica 192.168.1.85:8002 to 192.168.1.118:7002
M: 09389886a0105f0e9900f176a78e24fde83c4218 192.168.1.118:7001
slots:0-5460 (5461 slots) master
M: 60e8f33b3b2aad3ab1d5738970722cfbacaf56c9 192.168.1.118:7002
slots:10923-16383 (5461 slots) master
S: 24669175d5904e82e6cde65969da8096eaa797a3 192.168.1.118:7003
replicates d023b8befee4fef15953b4240d56d9e274fea2f5
M: d023b8befee4fef15953b4240d56d9e274fea2f5 192.168.1.85:8001
slots:5461-10922 (5462 slots) master
S: 0c0328bbc2a3be779fce1c2b6ff0c96edf0bebf6 192.168.1.85:8002
replicates 60e8f33b3b2aad3ab1d5738970722cfbacaf56c9
S: c6993c7b31ccc93cbc9bd86e34428a01363c9de6 192.168.1.85:8003
replicates 09389886a0105f0e9900f176a78e24fde83c4218
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join....
>>> Performing Cluster Check (using node 192.168.1.118:7001)
M: 09389886a0105f0e9900f176a78e24fde83c4218 192.168.1.118:7001
slots:0-5460 (5461 slots) master
1 additional replica(s)
S: 0c0328bbc2a3be779fce1c2b6ff0c96edf0bebf6 192.168.1.85:8002
slots: (0 slots) slave
replicates 60e8f33b3b2aad3ab1d5738970722cfbacaf56c9
M: d023b8befee4fef15953b4240d56d9e274fea2f5 192.168.1.85:8001
slots:5461-10922 (5462 slots) master
1 additional replica(s)
S: 24669175d5904e82e6cde65969da8096eaa797a3 192.168.1.118:7003
slots: (0 slots) slave
replicates d023b8befee4fef15953b4240d56d9e274fea2f5
M: 60e8f33b3b2aad3ab1d5738970722cfbacaf56c9 192.168.1.118:7002
slots:10923-16383 (5461 slots) master
1 additional replica(s)
S: c6993c7b31ccc93cbc9bd86e34428a01363c9de6 192.168.1.85:8003
slots: (0 slots) slave
replicates 09389886a0105f0e9900f176a78e24fde83c4218
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

查看当前集群
-bash-4.2$ ./redis-trib.rb check 192.168.1.118:7001
>>> Performing Cluster Check (using node 192.168.1.118:7001)
M: 09389886a0105f0e9900f176a78e24fde83c4218 192.168.1.118:7001
slots:0-5460 (5461 slots) master
1 additional replica(s)
S: 0c0328bbc2a3be779fce1c2b6ff0c96edf0bebf6 192.168.1.85:8002
slots: (0 slots) slave
replicates 60e8f33b3b2aad3ab1d5738970722cfbacaf56c9
M: d023b8befee4fef15953b4240d56d9e274fea2f5 192.168.1.85:8001
slots:5461-10922 (5462 slots) master
1 additional replica(s)
S: 24669175d5904e82e6cde65969da8096eaa797a3 192.168.1.118:7003
slots: (0 slots) slave
replicates d023b8befee4fef15953b4240d56d9e274fea2f5
M: 60e8f33b3b2aad3ab1d5738970722cfbacaf56c9 192.168.1.118:7002
slots:10923-16383 (5461 slots) master
1 additional replica(s)
S: c6993c7b31ccc93cbc9bd86e34428a01363c9de6 192.168.1.85:8003
slots: (0 slots) slave
replicates 09389886a0105f0e9900f176a78e24fde83c4218
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

停掉集群,然后启用密码登陆
停掉的时候一般先停掉从节点,再停住节点

192.168.1.118
./redis-cli -h 192.168.1.118 -p 7001 shutdown
./redis-cli -h 192.168.1.118 -p 7002 shutdown
./redis-cli -h 192.168.1.118 -p 7003 shutdown

192.168.1.85
./redis-cli -h 192.168.1.85 -p 8001 shutdown
./redis-cli -h 192.168.1.85 -p 8002 shutdown
./redis-cli -h 192.168.1.85 -p 8003 shutdown

重新启动后,查看集群
[hxl@xdm-redis1 bin]$ ./redis-cli -c -h 192.168.1.118 -p 7001 -a oracleabc2018
192.168.1.118:7001> cluster nodes
60e8f33b3b2aad3ab1d5738970722cfbacaf56c9 192.168.1.118:7002@17002 master - 0 1566877167000 2 connected 10923-16383
d023b8befee4fef15953b4240d56d9e274fea2f5 192.168.1.85:8001@18001 slave 24669175d5904e82e6cde65969da8096eaa797a3 0 1566877167000 7 connected
0c0328bbc2a3be779fce1c2b6ff0c96edf0bebf6 192.168.1.85:8002@18002 slave 60e8f33b3b2aad3ab1d5738970722cfbacaf56c9 0 1566877167447 5 connected
24669175d5904e82e6cde65969da8096eaa797a3 192.168.1.118:7003@17003 master - 0 1566877166443 7 connected 5461-10922
09389886a0105f0e9900f176a78e24fde83c4218 192.168.1.118:7001@17001 myself,master - 0 1566877164000 1 connected 0-5460
c6993c7b31ccc93cbc9bd86e34428a01363c9de6 192.168.1.85:8003@18003 slave 09389886a0105f0e9900f176a78e24fde83c4218 0 1566877167547 6 connected

-----------------------------验证---------------------
登录其中一个主节点
[hxl@xdm-redis1 bin]$ ./redis-cli -c -h 192.168.1.118 -p 7001 -a oracleabc2018
设置key值
192.168.1.118:7001> set name 'huangxueliang'
-> Redirected to slot [5798] located at 192.168.1.118:7003
OK

登录一个从节点get该key值,看集群是否同步，从下面的输出可以看出，集群已经同步.
[root@izwz9gxsbwybwg9n5xdi47z src]# ./redis-cli -c -h 127.0.0.1 -p 7003
127.0.0.1:7003> get name
-> Redirected to slot [5798] located at 127.0.0.1:7001
"huangxueliang"

-bash-4.2$ ./redis-cli -c -h 192.168.1.85 -p 8002 -a oracleabc2018
Warning: Using a password with '-a' option on the command line interface may not be safe.
192.168.1.85:8002> get name
-> Redirected to slot [5798] located at 192.168.1.118:7003
"huangxueliang"

 

----------------遇到的问题---------------------
1.redis-trib.rb脚本在有密码验证的情况下无法使用
如下:
/home/hxl/redis/redis-trib.rb create --replicas 1 192.168.1.118:7001 192.168.1.118:7002 192.168.1.118:7003 192.168.1.85:8001 192.168.1.85:8002 192.168.1.85:8003
>>> Creating cluster
[ERR] Sorry, can't connect to node 192.168.1.118:7001

解决办法1:去掉密码认证后再创建集群
解决方法2:修改该脚本client.rb
[root@localhost /]# find / -name client.rb
/usr/share/ruby/xmlrpc/client.rb
/usr/local/rvm/src/ruby-2.4.5/gems/xmlrpc-0.2.1/lib/xmlrpc/client.rb
/usr/local/rvm/gems/ruby-2.4.5/gems/redis-4.1.0/lib/redis/client.rb
/usr/local/rvm/rubies/ruby-2.4.5/lib/ruby/gems/2.4.0/gems/xmlrpc-0.2.1/lib/xmlrpc/client.rb

修改
/usr/local/rvm/gems/ruby-2.4.5/gems/redis-4.1.0/lib/redis/client.rb
password后面的值需要带上双引号

class Redis
class Client

DEFAULTS = {
:url => lambda { ENV["REDIS_URL"] },
:scheme => "redis",
:host => "127.0.0.1",
:port => 6379,
:path => nil,
:timeout => 5.0,
:password => "oracleabc2018",
:db => 0,
:driver => nil,
:id => nil,
:tcp_keepalive => 0,
:reconnect_attempts => 1,
:reconnect_delay => 0,
:reconnect_delay_max => 0.5,
:inherit_socket => false
}

./redis-trib.rb check 192.168.1.118:7001

方法3:修改/home/hxl/redis/redis-trib.rb脚本,找到如下行
@r = Redis.new(:host => @info[:host], :port => @info[:port], :timeout => 60)
修改为
@r = Redis.new(:host => @info[:host], :port => @info[:port], :timeout => 60,:password => "oracleabc2018")