前面使用了哨兵集群搭建单个分片高可用的结构，实现了主从复制和故障转移，但是没有实现分布式。接下来使用Redis集群搭建，实现哨兵集群没有实现的分布式高可用。
redis-cluster搭建需求
下面准备6个redis节点，搭建如图所示的三主三从分布式redis集群，实现两两互联，以master8001为例，它可以连接另外两个主节点，以及三个从节点。搭建过程可以使用原生redis cluster命令（使用方便），也可以使用ruby自带的脚本（需要先安装ruby，还有安装redis和ruby的接口，比较麻烦）。
ruby脚本辅助搭建
使用ruby脚本辅助安装，即使用redis根目录/src/redis-trib.rb脚本来完成，需要先安装ruby。
-rwxrwxr-x. 1 root root 60852 Sep 21 2017 redis-trib.rb
（1）安装ruby，这里解压了压缩包，进行编译和安装。但是安装完只是最基本的一步，后面还有坑，比较麻烦。
# 解压 [root@node01 /home/software]# tar -zxvf ruby-2.3.1.tar.gz # 检查 [root@node01 /home/software/ruby-2.3.1]# . /configure # 编译，安装 [root@node01 /home/software/ruby-2.3.1]# make && make install # 省略具体安装日志，安装完成后使用ruby -v如果能查看到版本号，就安装ok [root@node01 /home/software/ruby-2.3.1]# ruby -v ruby 2.3.1p112 (2016-04-26 revision 54768) [x86_4-linux]
（2）先准备一个redis-cluster.conf启动加载文件，基本配置参考前面博文https://www.cnblogs.com/youngchaolin/p/11983705.html#_label0 ，主要是bind设置不绑定ip、protected mode设置为no、daemonize设置为yes、其他还有pid、logfile、rdb等，保持原设置不变。
（3）配置appendonly为yes，启动aof持久化方式，这是除了rdb外redis提供的第二种持久化方式。
589 # AOF and RDB persistence can be enabled at the same time without problems. 590 # If the AOF is enabled on startup Redis will load the AOF, that is the file 591 # with the better durability guarantees. 592 # 593 # Please check http://redis.io/topics/persistence for more information. 594 # 配置第二种持久化方式aof 595 appendonly yes 596 597 # The name of the append only file (default: "appendonly.aof") 598 # 标识aof持久化文件名，以端口号区分 599 appendfilename "appendonly6379.aof"
其中rdb和aof两种持久化的主要区别为：
a.rdb保存的是具体的key-value数据，如name-messi，aof保存的是操作记录，如set name messi。
b.因为aof保存的数据更全，redis启动默认加载的是aof，rdb启动可以单独开启。
c.rdb持久化频率是按照持久化策略来的，容易造成数据丢失，而aof是每秒保存一次数据，数据不容易丢失。
d.如果对数据的可靠性要求高，使用aof持久化，如果需要一定的数据恢复能力，但是又不需要很高的可靠性，就选择rdb。
（4）开启集群模式，如果不开启无法计算槽道号。
719 # Normal Redis instances can't be part of a Redis Cluster; only nodes that are 720 # started as cluster nodes can. In order to start a Redis instance as a 721 # cluster node enable the cluster support uncommenting the following: 722 # 需设置yes，否则无法计算槽道号，无法创建集群 723 cluster-enabled yes
（5）需要开启集群节点状态记录文件，这个文件会自动更新，每个redis节点都需要一个这样的文件。
725 # Every cluster node has a cluster configuration file. This file is not 726 # intended to be edited by hand. It is created and updated by Redis nodes. 727 # Every Redis Cluster node requires a different cluster configuration file. 728 # Make sure that instances running in the same system do not have 729 # overlapping cluster configuration file names. 730 # 开启集群节点状态记录文件 731 cluster-config-file nodes-6379.conf
（6）将上面的配置文件保存，接下来复制到6个不同的文件夹下，其中文件夹的名字就是redis节点的端口号，根据上图准备8000、8001、8002、8003、8004和8005六个文件夹。复制完成后，修改对应端口号，如8000文件夹的配置文件，将里面全是6379的部分替换为8000，其他类似修改即可。
# 创建目录 [root@node01 /home/software/redis-3.2.11]# mkdir 8000 8001 8002 8003 8004 8005 # 复制文件 [root@node01 /home/software/redis-3.2.11]# cp redis.conf 8000/ [root@node01 /home/software/redis-3.2.11]# cp redis.conf 8001/ [root@node01 /home/software/redis-3.2.11]# cp redis.conf 8002/ [root@node01 /home/software/redis-3.2.11]# cp redis.conf 8003/ [root@node01 /home/software/redis-3.2.11]# cp redis.conf 8004/ [root@node01 /home/software/redis-3.2.11]# cp redis.conf 8005/ # 递归查看是否copy成功 [root@node01 /home/software/redis-3.2.11]# ls -R 800* 8000: redis.conf 8001: redis.conf 8002: redis.conf 8003: redis.conf 8004: redis.conf 8005: redis.conf
# 修改端口略
（7）启动各个节点，发现均是cluster的方式启动，这是创建集群的基础，另外查看集群状态为fail，是因为槽道还没分配的原因，且每个节点cluster nodes查看集群节点信息都只能找到自己一个节点，目前两两互联还没有建立。
# 加载修改后的配置文件启动各个节点 [root@node01 /home/software/redis-3.2.11]# redis-server 8000/redis.conf [root@node01 /home/software/redis-3.2.11]# redis-server 8001/redis.conf [root@node01 /home/software/redis-3.2.11]# redis-server 8002/redis.conf [root@node01 /home/software/redis-3.2.11]# redis-server 8003/redis.conf [root@node01 /home/software/redis-3.2.11]# redis-server 8004/redis.conf [root@node01 /home/software/redis-3.2.11]# redis-server 8005/redis.conf # 查看是否以cluster方式启动，ok [root@node01 /home/software/redis-3.2.11]# ps -ef|grep redis root 16028 1 0 15:43 ? 00:00:00 redis-server *:8000 [cluster] root 16033 1 0 15:43 ? 00:00:00 redis-server *:8001 [cluster] root 16037 1 0 15:43 ? 00:00:00 redis-server *:8002 [cluster] root 16041 1 0 15:43 ? 00:00:00 redis-server *:8003 [cluster] root 16045 1 0 15:43 ? 00:00:00 redis-server *:8004 [cluster] root 16049 1 0 15:43 ? 00:00:00 redis-server *:8005 [cluster] root 16053 1409 0 15:43 pts/0 00:00:00 grep redis # 登录一个节点，查看集群信息 [root@node01 /home/software/redis-3.2.11]# redis-cli -c -p 8000 127.0.0.1:8000> cluster info cluster_state:fail # 只要有一个槽道没分配，就是fail cluster_slots_assigned:0 cluster_slots_ok:0 cluster_slots_pfail:0 cluster_slots_fail:0 cluster_known_nodes:1 # 只能发现自己一个节点，两两互联暂未建立 cluster_size:0 cluster_current_epoch:0 # 集群的纪元 cluster_my_epoch:0 cluster_stats_messages_sent:0 # 两两互联发送数据量 cluster_stats_messages_received:0 # 两两互联接受数据量 # 查看集群节点，发现只有自己一个 127.0.0.1:8000> cluster nodes dc0e2e16888426089de31466ff398679eec81b86 :8000 myself,master - 0 0 0 connected
（8）启动src/redis-trib.rb命令后想查看帮助信息发现报错。这是正常的，这意味着踏入了一个大坑，接下来的需要填坑了，参考文末博文，完成了报错解决。
# 报错 /usr/local/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require': cannot load such file -- redis (LoadError) from /usr/local/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require' from src/redis-trib.rb:25:in `<main>'
（9）上面报错是需要redis的gem包来安装ruby和redis的接口，先安装gem包，网址https://rubygems.org/gems/redis/，选择对应的版本下载，我这里是选择3.2.1版本，下载好后上传到了redis根目录/src目录下了，具体位置放哪不清楚，放src下是可以验证通过的，另外查看博文放根目录也是可以的。
-rw-r--r--. 1 root root 73728 Dec 12 16:51 redis-3.2.1.gem
（10）准备使用gem install redis命令安装接口，发现报错，提示需要安装zlib，网址http://www.zlib.net，本次选择了1.2.11版本。
ERROR: Loading command: install (LoadError) cannot load such file -- zlib ERROR: While executing gem ... (NoMethodError) undefined method `invoke_with_build_args' for nil:NilClass
（11）tar -zxvf解压zlib压缩包后，进入解压目录，先指定安装目录，后编译安装zlib。
# --prefix指定安装目录/usr/local/zlib [root@node01 /home/software/zlib-1.2.11]# ./configure --prefix=/usr/local/zlib Checking for gcc... Checking for shared library support... Building shared library libz.so.1.2.11 with gcc. Checking for size_t... Yes. Checking for off64_t... Yes. Checking for fseeko... Yes. Checking for strerror... Yes. Checking for unistd.h... Yes. Checking for stdarg.h... Yes. Checking whether to use vs[n]printf() or s[n]printf()... using vs[n]printf(). Checking for vsnprintf() in stdio.h... Yes. Checking for return value of vsnprintf()... Yes. Checking for attribute(visibility) support... Yes. You have new mail in /var/spool/mail/root # 编译后安装 [root@node01 /home/software/zlib-1.2.11]# make && make install ...省略 安装成功 chmod 644 /usr/local/zlib/include/zlib.h /usr/local/zlib/include/zconf.h
（12）还需要进入ruby根目录/ext/zlib，先使用ruby extconf.rb命令，执行完后会在当前目录下生成Makefile文件。
[root@node01 /home/software/ruby-2.3.1/ext/zlib]# ll total 892 -rw-r--r--. 1 root root 146 Apr 16 2013 depend -rw-r--r--. 1 root root 1447 Dec 16 2015 extconf.rb
# 使用ruby extconf.rb命令后会生成一个Makefile文件 -rw-r--r--. 1 root root 7656 Dec 12 17:14 Makefile -rw-r--r--. 1 root root 7090 Dec 12 17:14 mkmf.log -rw-r--r--. 1 root root 124168 Dec 19 2015 zlib.c -rw-r--r--. 1 root root 418576 Dec 12 17:18 zlib.o -rwxr-xr-x. 1 root root 336032 Dec 12 17:18 zlib.so
# 安装时需添加配置信息，指定zlib目录下的include和lib，zlib目录就是前面编译指定安装的目录 [root@node01 /home/software/ruby-2.3.1/ext/zlib]# ruby extconf.rb --with-zlib-include=/usr/local/zlib/include/ --with-zlib-lib=/usr/local/zlib/lib
# 省略，编译安装
[root@node01 /home/software/ruby-2.3.1/ext/zlib]# make && make install
接下来编译，本次编译顺利通过了，如果没有通过可能会出现如下报错，还需要修改Makefile内容。
make: *** No rule to make target `/include/ruby.h', needed by `zlib.o'. Stop
修改Makefile内容，将$(top_srcdir)整个换成绝对路径，我这里没有报错，因此没有修改。
zlib.o: $(top_srcdir)/include/ruby.h
修改成：zlib.o: ../../include/ruby.h
最后编译通过会提示如下内容。
[root@node01 /home/software/ruby-2.3.1/ext/zlib]# make && make install compiling zlib.c linking shared-object zlib.so /usr/bin/install -c -m 0755 zlib.so /usr/local/lib/ruby/site_ruby/2.4.0/x86_64-linux
（13）继续安装openssl，如果不安装切换到redis目录使用gem install redis会报''unable to require openssl...''的报错，因此需要继续像安装zlib一样安装openssl。
网址https://www.openssl.org/source/，解压后进入安装目录，执行以下命令，编译安装执行时间比较长。
[root@node01 /home/software/openssl-1.0.2t]# ./config -fPIC --prefix=/usr/local/openssl enable-shared [root@node01 /home/software/openssl-1.0.2t]# ./config -t # 这里会编译很久 [root@node01 /home/software/openssl-1.0.2t]# make && make install
（14）还需进入ruby根目录/ext/openssl下，执行ruby extconf.rb命令，也会在当前目录下生成Makefile文件，执行后光荣的报错了。
[root@node01 /home/software/ruby-2.3.1/ext/openssl]# ruby extconf.rb --with-openssl-include=/usr/local/openssl/include/ --with-openssl-lib=/usr/local/openssl/lib [root@node01 /home/software/ruby-2.3.1/ext/openssl]# make && make install
（15）类似上面安装zlib，也将Makefile文件的$(top_srcdir)全部改成"../.."，这里在使用了":%s/${top_srcdir}/../../g"进行了全局替换。再次编译安装就通过了。
（16）到这里，就差不多可以了，切换目录到redis根目录，然后执行gem install redis，再次报错。没辙了，网上找了一种土办法，先让安装成功。
[root@node01 /home/software/redis-3.2.11]# gem install redis ERROR: Could not find a valid gem 'redis' (>= 0), here is why: Unable to download data from https://rubygems.org/ - SSL_connect returned=1 errno=0 state=error: certificate verify failed (https://api.rubygems.org/specs.4.8.gz)
临时办法，大概意思让https连接改成http连接。
[root@node01 /home/software/redis-3.2.11]# gem sources -r https://rubygems.org source https://rubygems.org not present in cache [root@node01 /home/software/redis-3.2.11]# gem sources -a http://rubygems.org https://rubygems.org is recommended for security over http://rubygems.org Do you want to add this insecure source? [yn] y http://rubygems.org added to sources You have new mail in /var/spool/mail/root
继续执行命令，终于ok了，可谓一波三折，这个方法只能作为参考，换一个redis或ruby又不知道会发生什么报错。
# 继续 [root@node01 /home/software/redis-3.2.11/src]# gem install redis Successfully installed redis-3.2.1 Parsing documentation for redis-3.2.1 Installing ri documentation for redis-3.2.1 Done installing documentation for redis after 0 seconds #成功了 WARNING: Unable to pull data from 'https://rubygems.org/': SSL_connect returned=1 errno=0 state=error: certificate verify failed (https://api.rubygems.org/specs.4.8.gz) 1 gem installed You have new mail in /var/spool/mail/root # 试运行redis-trib.rb脚本，终于ok了 [root@node01 /home/software/redis-3.2.11]# src/redis-trib.rb Usage: redis-trib <command> <options> <arguments ...> create host1:port1 ... hostN:portN --replicas <arg> check host:port info host:port fix host:port --timeout <arg> reshard host:port --from <arg> --to <arg> --slots <arg> --yes --timeout <arg> --pipeline <arg> rebalance host:port --weight <arg> --auto-weights --use-empty-masters --timeout <arg> ...省略
（17）前面准备好了后，就可以使用ruby脚本来开始构建集群了。上面启动后的redis节点都是各自为政互不联系，接下来先将8000、8001、8002作为主节点连起来，并分配槽道号，本次使用默认分配。
# 使用create [root@node01 /home/software/redis-3.2.11]# src/redis-trib.rb create 192.168.200.140:8000 192.168.200.140:8001 192.168.200.140:8002 >>> Creating cluster >>> Performing hash slots allocation on 3 nodes... Using 3 masters: 192.168.200.140:8000 192.168.200.140:8001 192.168.200.140:8002 M: dc0e2e16888426089de31466ff398679eec81b86 192.168.200.140:8000 slots:0-5460 (5461 slots) master M: aa79ace502e5369236b62ed61c0eb43733ddcbde 192.168.200.140:8001 slots:5461-10922 (5462 slots) master M: 719d1dd412faf6b4e1eb348c65c329a262e393e1 192.168.200.140:8002 slots:10923-16383 (5461 slots) master Can I set the above configuration? (type 'yes' to accept): yes >>> Nodes configuration updated >>> Assign a different config epoch to each node >>> Sending CLUSTER MEET messages to join the cluster Waiting for the cluster to join >>> Performing Cluster Check (using node 192.168.200.140:8000) M: dc0e2e16888426089de31466ff398679eec81b86 192.168.200.140:8000 slots:0-5460 (5461 slots) master 0 additional replica(s) M: aa79ace502e5369236b62ed61c0eb43733ddcbde 192.168.200.140:8001 slots:5461-10922 (5462 slots) master 0 additional replica(s) M: 719d1dd412faf6b4e1eb348c65c329a262e393e1 192.168.200.140:8002 slots:10923-16383 (5461 slots) master 0 additional replica(s) [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered. You have new mail in /var/spool/mail/root [root@node01 /home/software/redis-3.2.11]# redis-cli -c -p 8000 # 查看发现集群状态已变成ok，说明槽道没有一个没分配 127.0.0.1:8000> cluster into (error) ERR Wrong CLUSTER subcommand or number of arguments 127.0.0.1:8000> cluster info cluster_state:ok cluster_slots_assigned:16384 cluster_slots_ok:16384 cluster_slots_pfail:0 cluster_slots_fail:0 cluster_known_nodes:3 cluster_size:3 cluster_current_epoch:3 cluster_my_epoch:1 cluster_stats_messages_sent:68 cluster_stats_messages_received:68 # 三个主节点都分配了槽道 127.0.0.1:8000> cluster nodes
# 节点信息
# 第一个为节点id，用40位的十六进制表示
# 第二个为节点ip+端口
# 第三个为角色
# 第四个为主节点id，没有主节点就用'-'表示
# 第五个是与操作相关的时间戳
# 第六个代表序号
# 第七个代表连接状态
# 第八个代表槽道号区间 aa79ace502e5369236b62ed61c0eb43733ddcbde 192.168.200.140:8001 master - 0 1576145625541 2 connected 5461-10922 719d1dd412faf6b4e1eb348c65c329a262e393e1 192.168.200.140:8002 master - 0 1576145623522 3 connected 10923-16383 dc0e2e16888426089de31466ff398679eec81b86 192.168.200.140:8000 myself,master - 0 0 1 connected 0-5460
（18）将8003挂接到8000，8004挂接到8001，8005挂接到8002成为从节点，使用add node命令，这里需要用到两个选项，一个是--salve，代表新加的节点是从，一个是--master-id，指定需要添加的主是谁，最后依次添加slaveHost:slavePort existHost:existPort，后面的节点可以是集群中任意一个集群的已知节点。
# 以添加8003为例 ，注意主节点可以是任意一个主节点，不一定是当前从的主节点
[root@node01 /home/software/redis-3.2.11]# src/redis-trib.rb add-node --slave --master-id dc0e2e16888426089de31466ff398679eec81b86 192.168.200.140:8003 192.168.200.140:8000 >>> Adding node 192.168.200.140:8003 to cluster 192.168.200.140:8000 >>> Performing Cluster Check (using node 192.168.200.140:8000) M: dc0e2e16888426089de31466ff398679eec81b86 192.168.200.140:8000 slots:0-5460 (5461 slots) master 0 additional replica(s) M: aa79ace502e5369236b62ed61c0eb43733ddcbde 192.168.200.140:8001 slots:5461-10922 (5462 slots) master 0 additional replica(s) M: 719d1dd412faf6b4e1eb348c65c329a262e393e1 192.168.200.140:8002 slots:10923-16383 (5461 slots) master 0 additional replica(s) [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered.
# 底层还是调用redis原生命令cluster meet >>> Send CLUSTER MEET to node 192.168.200.140:8003 to make it join the cluster. Waiting for the cluster to join. >>> Configure node as replica of 192.168.200.140:8000. [OK] New node added correctly. You have new mail in /var/spool/mail/root
登录集群客户端查看集群信息，发现搭建成功，实现了三主三从redis集群的搭建。
127.0.0.1:8000> cluster nodes aa79ace502e5369236b62ed61c0eb43733ddcbde 192.168.200.140:8001 master - 0 1576152242486 2 connected 5461-10922 dc0e2e16888426089de31466ff398679eec81b86 192.168.200.140:8000 myself,master - 0 0 1 connected 0-5460 824a999176c12eb6ebd944b75944fd97c65b579d 192.168.200.140:8005 slave 719d1dd412faf6b4e1eb348c65c329a262e393e1 0 1576152244002 3 connected 312accb9be27c48c107a19ab501a75a8c5d321c6 192.168.200.140:8004 slave aa79ace502e5369236b62ed61c0eb43733ddcbde 0 1576152239461 2 connected ada1903bcb5744900916151dd385936ceb6ce049 192.168.200.140:8003 slave dc0e2e16888426089de31466ff398679eec81b86 0 1576152243496 1 connected 719d1dd412faf6b4e1eb348c65c329a262e393e1 192.168.200.140:8002 master - 0 1576152244507 3 connected 10923-16383
ruby脚本简化搭建
可以看出，这个搭建比较繁琐， 如果解决了ruby的报错问题，只需要一个命令就可以完成三主三从redis集群的搭建，下面记录一下。
（1）停止redis所有进程。
[root@node01 /home/software/redis-3.2.11]# redis-cli -c -p 8000 shutdown [root@node01 /home/software/redis-3.2.11]# redis-cli -c -p 8001 shutdown [root@node01 /home/software/redis-3.2.11]# redis-cli -c -p 8002 shutdown [root@node01 /home/software/redis-3.2.11]# redis-cli -c -p 8003 shutdown [root@node01 /home/software/redis-3.2.11]# redis-cli -c -p 8004 shutdown [root@node01 /home/software/redis-3.2.11]# redis-cli -c -p 8005 shutdown [root@node01 /home/software/redis-3.2.11]# ps -ef|grep redis root 3657 18054 0 20:13 pts/2 00:00:00 grep redis
（2）删除rdb和aof持久化文件，删除集群节点状态记录文件。
[root@node01 /home/software/redis-3.2.11]# rm -rf appendonly800* [root@node01 /home/software/redis-3.2.11]# rm -rf dump800* [root@node01 /home/software/redis-3.2.11]# rm -rf nodes-800*
（3）再次启动6个redis节点。
# 启动 [root@node01 /home/software/redis-3.2.11]# redis-server 8000/redis.conf [root@node01 /home/software/redis-3.2.11]# redis-server 8001/redis.conf [root@node01 /home/software/redis-3.2.11]# redis-server 8002/redis.conf [root@node01 /home/software/redis-3.2.11]# redis-server 8003/redis.conf [root@node01 /home/software/redis-3.2.11]# redis-server 8004/redis.conf [root@node01 /home/software/redis-3.2.11]# redis-server 8005/redis.conf [root@node01 /home/software/redis-3.2.11]# redis-cli -c -p 8000 # 还没分配槽道，redis集群状态fail 127.0.0.1:8000> cluster info cluster_state:fail cluster_slots_assigned:0 cluster_slots_ok:0 cluster_slots_pfail:0 cluster_slots_fail:0 cluster_known_nodes:1 cluster_size:0 cluster_current_epoch:0 cluster_my_epoch:0 cluster_stats_messages_sent:0 cluster_stats_messages_received:0
（4）使用ruby脚本src/redis-trib.rb create --replicas 1 六个的节点ip:端口，这一条命令就可以自动搭建了三主三从。
# 一个命令 [root@node01 /home/software/redis-3.2.11]# src/redis-trib.rb create --replicas 1 192.168.200.140:8000 192.168.200.140:8001 192.168.200.140:8002 192.168.200.140:8003 192.168.200.140:8004 192.168.200.140:8005 >>> Creating cluster >>> Performing hash slots allocation on 6 nodes... Using 3 masters: 192.168.200.140:8000 192.168.200.140:8001 192.168.200.140:8002 Adding replica 192.168.200.140:8003 to 192.168.200.140:8000 Adding replica 192.168.200.140:8004 to 192.168.200.140:8001 Adding replica 192.168.200.140:8005 to 192.168.200.140:8002 M: 6537ac1cb42f209600ea20b1d6310e7b789257d4 192.168.200.140:8000 slots:0-5460 (5461 slots) master M: 5da79737cf493085724a689bdb5bd878c9121c07 192.168.200.140:8001 slots:5461-10922 (5462 slots) master M: 49f40898d2d18cc162b00e5f69593f0b91af5766 192.168.200.140:8002 slots:10923-16383 (5461 slots) master S: a92a3eb2cfff488e0af6ca07d9b612f2f3595c0a 192.168.200.140:8003 replicates 6537ac1cb42f209600ea20b1d6310e7b789257d4 S: ea41d1e7a41e5c5b19baa8e6cbac15b7d4d2bdb8 192.168.200.140:8004 replicates 5da79737cf493085724a689bdb5bd878c9121c07 S: bede8e5c11fe947cbaf0e435d822057a7b0fe414 192.168.200.140:8005 replicates 49f40898d2d18cc162b00e5f69593f0b91af5766 Can I set the above configuration? (type 'yes' to accept): yes >>> Nodes configuration updated >>> Assign a different config epoch to each node >>> Sending CLUSTER MEET messages to join the cluster Waiting for the cluster to join... >>> Performing Cluster Check (using node 192.168.200.140:8000) M: 6537ac1cb42f209600ea20b1d6310e7b789257d4 192.168.200.140:8000 slots:0-5460 (5461 slots) master 1 additional replica(s) M: 49f40898d2d18cc162b00e5f69593f0b91af5766 192.168.200.140:8002 slots:10923-16383 (5461 slots) master 1 additional replica(s) S: bede8e5c11fe947cbaf0e435d822057a7b0fe414 192.168.200.140:8005 slots: (0 slots) slave replicates 49f40898d2d18cc162b00e5f69593f0b91af5766 M: 5da79737cf493085724a689bdb5bd878c9121c07 192.168.200.140:8001 slots:5461-10922 (5462 slots) master 1 additional replica(s) S: a92a3eb2cfff488e0af6ca07d9b612f2f3595c0a 192.168.200.140:8003 slots: (0 slots) slave replicates 6537ac1cb42f209600ea20b1d6310e7b789257d4 S: ea41d1e7a41e5c5b19baa8e6cbac15b7d4d2bdb8 192.168.200.140:8004 slots: (0 slots) slave replicates 5da79737cf493085724a689bdb5bd878c9121c07 [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered. You have new mail in /var/spool/mail/root [root@node01 /home/software/redis-3.2.11]# redis-cli -c -p 8000 # 结果一样 127.0.0.1:8000> cluster nodes 49f40898d2d18cc162b00e5f69593f0b91af5766 192.168.200.140:8002 master - 0 1576153283083 3 connected 10923-16383 bede8e5c11fe947cbaf0e435d822057a7b0fe414 192.168.200.140:8005 slave 49f40898d2d18cc162b00e5f69593f0b91af5766 0 1576153285101 6 connected 5da79737cf493085724a689bdb5bd878c9121c07 192.168.200.140:8001 master - 0 1576153286109 2 connected 5461-10922 6537ac1cb42f209600ea20b1d6310e7b789257d4 192.168.200.140:8000 myself,master - 0 0 1 connected 0-5460 a92a3eb2cfff488e0af6ca07d9b612f2f3595c0a 192.168.200.140:8003 slave 6537ac1cb42f209600ea20b1d6310e7b789257d4 0 1576153283585 4 connected ea41d1e7a41e5c5b19baa8e6cbac15b7d4d2bdb8 192.168.200.140:8004 slave 5da79737cf493085724a689bdb5bd878c9121c07 0 1576153284091 5 connected
redis cluster原生命令搭建
以上两种方法，都是依赖ruby脚本，其实它还是基于redis cluster的命令来搭建完成，使用这种方式搭建能更好的理解搭建中经历了什么。
（1）提前准备6个启动的节点，前提也需删除持久化文件，以及节点状态记录文件，跟上面一样。
（2）登录任何一个节点，和其他5个进行meet，这里使用8000的客户端来完成，meet完后任意一个节点都认为自己是主节点。
# 使用cluster meet命令
192.168.200.140:8000> CLUSTER MEET 192.168.200.140 8001 OK 192.168.200.140:8000> CLUSTER MEET 192.168.200.140 8002 OK 192.168.200.140:8000> CLUSTER MEET 192.168.200.140 8003 OK 192.168.200.140:8000> CLUSTER MEET 192.168.200.140 8004 OK 192.168.200.140:8000> CLUSTER MEET 192.168.200.140 8005 OK
# 登录任何一个节点查看，都能得到如下信息 192.168.200.140:8000> cluster nodes fd4c160edc74536d79ea29d239dca43275ec6b5a 192.168.200.140:8004 master - 0 1576155518818 0 connected ced952bc6a47c38756bbc48d85a5f601db620b79 192.168.200.140:8000 myself,master - 0 0 4 connected 2c52d95c3d6d4c396469a81edfc1493d984e0f2d 192.168.200.140:8005 master - 0 1576155517810 5 connected 231fe9df31dc1ccf7cca5ae2fb2313979cd6fa83 192.168.200.140:8001 master - 0 1576155514284 1 connected 7ce388bde879f686fc3c8491175397ca20405565 192.168.200.140:8002 master - 0 1576155515791 2 connected 2e0f23d703874db80373f28b1be8c13f9de4fe6b 192.168.200.140:8003 master - 0 1576155516800 3 connected
（3）给8000，8001和8002分配槽道号，让其作为主节点。分配槽道号的命令为cluster addslots 槽道号...。分配完成后不管登录哪个节点都能看到槽道号信息，
# 槽道号之间有多个需要用空格隔开 [root@node01 /home/software/redis-3.2.11]# redis-cli -c -p 8000 -h 192.168.200.140 192.168.200.140:8000> CLUSTER ADDSLOTS 0 1 2 3 4 5 OK
# 可以看到8000的节点分配了0-5这6个槽道号 192.168.200.140:8000> cluster nodes fd4c160edc74536d79ea29d239dca43275ec6b5a 192.168.200.140:8004 master - 0 1576155898158 0 connected ced952bc6a47c38756bbc48d85a5f601db620b79 192.168.200.140:8000 myself,master - 0 0 4 connected 0-5 2c52d95c3d6d4c396469a81edfc1493d984e0f2d 192.168.200.140:8005 master - 0 1576155896137 5 connected 231fe9df31dc1ccf7cca5ae2fb2313979cd6fa83 192.168.200.140:8001 master - 0 1576155894119 1 connected 7ce388bde879f686fc3c8491175397ca20405565 192.168.200.140:8002 master - 0 1576155897147 2 connected 2e0f23d703874db80373f28b1be8c13f9de4fe6b 192.168.200.140:8003 master - 0 1576155895128 3 connected
一个个写来分配会比较费力气，可以写一个shell脚本来批量分配槽道号，使用for循环来实现。
shell脚本
#分配槽道号的脚本 #!/bin/bash # 8000节点 for slot in {6..5460} do redis-cli -c -p 8000 -h 192.168.200.140 cluster addslots $slot done # 8001节点 for slot in {5461..10922} do redis-cli -c -p 8001 -h 192.168.200.140 cluster addslots $slot done # 8002节点 for slot in {10923..16383} do redis-cli -c -p 8002 -h 192.168.200.140 cluster addslots $slot done # 执行完打印出来结果 echo "分配槽道号结束"
运行脚本执行完结果，ok。
登录一个节点查看集群槽道信息，发现跟分配的预想一样。
[root@node01 /home/software/redis-3.2.11]# redis-cli -c -p 8001 -h 192.168.200.140 192.168.200.140:8001> cluster nodes
# 8001结果，与上面脚本一样结果 231fe9df31dc1ccf7cca5ae2fb2313979cd6fa83 192.168.200.140:8001 myself,master - 0 0 1 connected 5461-10922 fd4c160edc74536d79ea29d239dca43275ec6b5a 192.168.200.140:8004 master - 0 1576157229386 0 connected ced952bc6a47c38756bbc48d85a5f601db620b79 192.168.200.140:8000 master - 0 1576157228376 4 connected 0-5460 7ce388bde879f686fc3c8491175397ca20405565 192.168.200.140:8002 master - 0 1576157229386 2 connected 10923-16383 2e0f23d703874db80373f28b1be8c13f9de4fe6b 192.168.200.140:8003 master - 0 1576157227368 3 connected 2c52d95c3d6d4c396469a81edfc1493d984e0f2d 192.168.200.140:8005 master - 0 1576157230393 5 connected
（4）将8003、8004和8005角色转换为从节点，使用cluster replicate 主节点id 命令。最后查看集群信息，发现成功的分配了槽道号。
[root@node01 /home/software/redis-3.2.11]# redis-cli -c -p 8003 -h 192.168.200.140 # 成为8000的从节点 192.168.200.140:8003> cluster replicate ced952bc6a47c38756bbc48d85a5f601db620b79 OK 192.168.200.140:8003> quit [root@node01 /home/software/redis-3.2.11]# redis-cli -c -p 8004 -h 192.168.200.140 # 成为8001的从节点 192.168.200.140:8004> cluster replicate 231fe9df31dc1ccf7cca5ae2fb2313979cd6fa83 OK 192.168.200.140:8004> quit You have new mail in /var/spool/mail/root [root@node01 /home/software/redis-3.2.11]# redis-cli -c -p 8005 -h 192.168.200.140 # 成为8002的从节点 192.168.200.140:8005> cluster replicate 7ce388bde879f686fc3c8491175397ca20405565 OK # 结果ok 192.168.200.140:8005> cluster nodes 2c52d95c3d6d4c396469a81edfc1493d984e0f2d 192.168.200.140:8005 myself,slave 7ce388bde879f686fc3c8491175397ca20405565 0 0 5 connected 7ce388bde879f686fc3c8491175397ca20405565 192.168.200.140:8002 master - 0 1576157733223 2 connected 10923-16383 2e0f23d703874db80373f28b1be8c13f9de4fe6b 192.168.200.140:8003 slave ced952bc6a47c38756bbc48d85a5f601db620b79 0 1576157732216 4 connected ced952bc6a47c38756bbc48d85a5f601db620b79 192.168.200.140:8000 master - 0 1576157735238 4 connected 0-5460 231fe9df31dc1ccf7cca5ae2fb2313979cd6fa83 192.168.200.140:8001 master - 0 1576157736242 1 connected 5461-10922 fd4c160edc74536d79ea29d239dca43275ec6b5a 192.168.200.140:8004 slave 231fe9df31dc1ccf7cca5ae2fb2313979cd6fa83 0 1576157734230 1 connected
最后集群状态为ok，槽道全部分配完成，形成了三主三从的结构。
192.168.200.140:8005> cluster info cluster_state:ok cluster_slots_assigned:16384 cluster_slots_ok:16384 cluster_slots_pfail:0 cluster_slots_fail:0
# 6个节点 cluster_known_nodes:6 # 3个主
cluster_size:3 cluster_current_epoch:5 cluster_my_epoch:2 cluster_stats_messages_sent:5093 cluster_stats_messages_received:5093
redis cluster高可用
现在在8000节点上保存数据，然后将8000的主节点宕机，8003从节点会顶替上来成为主节点，并获取到8000的槽道信息。
[root@node01 /home/software/redis-3.2.11]# redis-cli -c -p 8000 -h 192.168.200.140 192.168.200.140:8000> set age 28 OK 192.168.200.140:8000> get age "28" 192.168.200.140:8000> quit [root@node01 /home/software/redis-3.2.11]# redis-cli -c -p 8003 -h 192.168.200.140 192.168.200.140:8003> get age -> Redirected to slot [741] located at 192.168.200.140:8000 "28" 192.168.200.140:8000> shutdown not connected> quit You have new mail in /var/spool/mail/root [root@node01 /home/software/redis-3.2.11]# redis-cli -c -p 8003 -h 192.168.200.140 # 8000宕机后，8003暂时还是从 192.168.200.140:8003> cluster nodes fd4c160edc74536d79ea29d239dca43275ec6b5a 192.168.200.140:8004 slave 231fe9df31dc1ccf7cca5ae2fb2313979cd6fa83 0 1576158134892 1 connected 2e0f23d703874db80373f28b1be8c13f9de4fe6b 192.168.200.140:8003 myself,slave ced952bc6a47c38756bbc48d85a5f601db620b79 0 0 3 connected ced952bc6a47c38756bbc48d85a5f601db620b79 192.168.200.140:8000 master - 1576158120865 1576158119755 4 disconnected 0-5460 2c52d95c3d6d4c396469a81edfc1493d984e0f2d 192.168.200.140:8005 slave 7ce388bde879f686fc3c8491175397ca20405565 0 1576158132873 5 connected 231fe9df31dc1ccf7cca5ae2fb2313979cd6fa83 192.168.200.140:8001 master - 0 1576158133883 1 connected 5461-10922 7ce388bde879f686fc3c8491175397ca20405565 192.168.200.140:8002 master - 0 1576158131864 2 connected 10923-16383 # 过一会，8003变为主，并且获得槽道信息 192.168.200.140:8003> cluster nodes fd4c160edc74536d79ea29d239dca43275ec6b5a 192.168.200.140:8004 slave 231fe9df31dc1ccf7cca5ae2fb2313979cd6fa83 0 1576158149019 1 connected 2e0f23d703874db80373f28b1be8c13f9de4fe6b 192.168.200.140:8003 myself,master - 0 0 6 connected 0-5460 ced952bc6a47c38756bbc48d85a5f601db620b79 192.168.200.140:8000 master,fail - 1576158120865 1576158119755 4 disconnected 2c52d95c3d6d4c396469a81edfc1493d984e0f2d 192.168.200.140:8005 slave 7ce388bde879f686fc3c8491175397ca20405565 0 1576158148008 5 connected 231fe9df31dc1ccf7cca5ae2fb2313979cd6fa83 192.168.200.140:8001 master - 0 1576158146999 1 connected 5461-10922 7ce388bde879f686fc3c8491175397ca20405565 192.168.200.140:8002 master - 0 1576158150028 2 connected 10923-16383 # 数据也备份了 192.168.200.140:8003> get age "28"
redis cluster添加和删除节点
再补充添加和删除节点的操作，关于删除节点，它只能删除从节点或没有槽道管理权的节点，由于8000已经没有了槽道管理权，将8000重启后，下面使用ruby脚本删除，命令为del-node ip:port 节点id。
# 确认8000开启
[root@node01 /home/software/redis-3.2.11]# ps -ef|grep redis root 4170 1 0 20:51 ? 00:00:13 redis-server *:8001 [cluster] root 4174 1 0 20:51 ? 00:00:11 redis-server *:8002 [cluster] root 4178 1 0 20:51 ? 00:00:08 redis-server *:8003 [cluster] root 4182 1 0 20:51 ? 00:00:08 redis-server *:8004 [cluster] root 4186 1 0 20:51 ? 00:00:08 redis-server *:8005 [cluster] root 7231 1 0 22:09 ? 00:00:00 redis-server *:8000 [cluster] root 7298 18054 0 22:13 pts/2 00:00:00 grep redis
# 删除8000 [root@node01 /home/software/redis-3.2.11]# src/redis-trib.rb del-node 192.168.200.140:8000 94f634b2d364d560d3f8fb6e88fa874cf2b493f6 >>> Removing node 94f634b2d364d560d3f8fb6e88fa874cf2b493f6 from cluster 192.168.200.140:8000 >>> Sending CLUSTER FORGET messages to the cluster...
# 关闭节点 >>> SHUTDOWN the node. You have new mail in /var/spool/mail/root
发现删除节点后，还自动将节点关闭了，因为8000节点的状态记录文件nodes-8000.conf还记录着8000是连接状态，但是其他节点的这个文件已经没了8000的信息，为了集群数据的不混乱，需要将8000立即关闭。
既然删除了，尝试启动8000后添加到集群，需要使用add-node newHost:newPort existHost:existPort命令，类似添加从节点命令只是没有选项。发现启动8000后再添加进集群会报错。
# 确认开启8000 [root@node01 /home/software/redis-3.2.11]# ps -ef|grep redis root 4170 1 0 20:51 ? 00:00:14 redis-server *:8001 [cluster] root 4174 1 0 20:51 ? 00:00:11 redis-server *:8002 [cluster] root 4178 1 0 20:51 ? 00:00:08 redis-server *:8003 [cluster] root 4182 1 0 20:51 ? 00:00:08 redis-server *:8004 [cluster] root 4186 1 0 20:51 ? 00:00:08 redis-server *:8005 [cluster] root 7400 1 0 22:21 ? 00:00:00 redis-server *:8000 [cluster] root 7422 18054 0 22:22 pts/2 00:00:00 grep redis You have new mail in /var/spool/mail/root # 添加8000到集群 [root@node01 /home/software/redis-3.2.11]# src/redis-trib.rb add-node 192.168.200.140:8000 192.168.200.140:8001 >>> Adding node 192.168.200.140:8000 to cluster 192.168.200.140:8001 >>> Performing Cluster Check (using node 192.168.200.140:8001) M: 231fe9df31dc1ccf7cca5ae2fb2313979cd6fa83 192.168.200.140:8001 slots:5461-10922 (5462 slots) master 1 additional replica(s) S: fd4c160edc74536d79ea29d239dca43275ec6b5a 192.168.200.140:8004 slots: (0 slots) slave replicates 231fe9df31dc1ccf7cca5ae2fb2313979cd6fa83 M: 7ce388bde879f686fc3c8491175397ca20405565 192.168.200.140:8002 slots:10923-16383 (5461 slots) master 1 additional replica(s) M: 2e0f23d703874db80373f28b1be8c13f9de4fe6b 192.168.200.140:8003 slots:0-5460 (5461 slots) master 0 additional replica(s) S: 2c52d95c3d6d4c396469a81edfc1493d984e0f2d 192.168.200.140:8005 slots: (0 slots) slave replicates 7ce388bde879f686fc3c8491175397ca20405565 [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered. # 提示报错，提示8000可能有数据，或者8000里有其他集群的信息 [ERR] Node 192.168.200.140:8000 is not empty. Either the node already knows other nodes (check with CLUSTER NODES) or contains some key in database 0.
这里删除了8000的rdb和aof持久化文件，以及状态文件后，重启8000节点再次尝试添加进去集群发现成功，但是添加进去的节点为没有槽道管理权的主节点。
[root@node01 /home/software/redis-3.2.11]# src/redis-trib.rb add-node 192.168.200.140:8000 192.168.200.140:8001 >>> Adding node 192.168.200.140:8000 to cluster 192.168.200.140:8001 >>> Performing Cluster Check (using node 192.168.200.140:8001) M: 231fe9df31dc1ccf7cca5ae2fb2313979cd6fa83 192.168.200.140:8001 slots:5461-10922 (5462 slots) master 1 additional replica(s) S: fd4c160edc74536d79ea29d239dca43275ec6b5a 192.168.200.140:8004 slots: (0 slots) slave replicates 231fe9df31dc1ccf7cca5ae2fb2313979cd6fa83 M: 7ce388bde879f686fc3c8491175397ca20405565 192.168.200.140:8002 slots:10923-16383 (5461 slots) master 1 additional replica(s) M: 2e0f23d703874db80373f28b1be8c13f9de4fe6b 192.168.200.140:8003 slots:0-5460 (5461 slots) master 0 additional replica(s) S: 2c52d95c3d6d4c396469a81edfc1493d984e0f2d 192.168.200.140:8005 slots: (0 slots) slave replicates 7ce388bde879f686fc3c8491175397ca20405565 [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered. >>> Send CLUSTER MEET to node 192.168.200.140:8000 to make it join the cluster. [OK] New node added correctly.
8000的信息
fd4c160edc74536d79ea29d239dca43275ec6b5a 192.168.200.140:8004 slave 231fe9df31dc1ccf7cca5ae2fb2313979cd6fa83 0 1576161337438 1 connected # 没有槽道管理权的主节点 c41dbe9595ae83725d1322b032736fd198b26c49 192.168.200.140:8000 myself,master - 0 0 0 connected 2c52d95c3d6d4c396469a81edfc1493d984e0f2d 192.168.200.140:8005 slave 7ce388bde879f686fc3c8491175397ca20405565 0 1576161336432 2 connected 231fe9df31dc1ccf7cca5ae2fb2313979cd6fa83 192.168.200.140:8001 master - 0 1576161336432 1 connected 5461-10922 2e0f23d703874db80373f28b1be8c13f9de4fe6b 192.168.200.140:8003 master - 0 1576161339456 6 connected 0-5460 7ce388bde879f686fc3c8491175397ca20405565 192.168.200.140:8002 master - 0 1576161338448 2 connected 10923-16383
以上是redis-cluster集群搭建的基本知识，后面继续补充槽道知识。
参考博文：
（1）《Redis设计与实践》
（2）https://blog.csdn.net/qq_26710443/article/details/82724268 ruby安装
（3）https://www.jianshu.com/p/c38369097448
（4）https://www.cnblogs.com/xuliangxing/p/7146868.html ruby脚本不能执行解决，报错主要参考文
（5）https://blog.csdn.net/ck3207/article/details/90404952 openssl无法编译