Redis+Keepalived高可用环境部署记录

Keepalived 实现VRRP（虚拟路由冗余）协议，从路由级别实现VIP切换，可以完全避免类似heartbeat脑裂问题，可以很好的实现主从、主备、互备方案，尤其是无状态业务，有状态业务就需要额外花些功夫了。既然Mysql可以使用Keepalived很好的做到主从切换，那么Redis自然可以使用这种方式实现高可用。

Redis主从实现完全没有Mysql成熟，仅仅是可用而已，经过测试主从也不是那么完全不靠谱，主要问题在于同步连接断开之后需要重新全量同步，如果频繁进行会对主服务带来很大性能影响。 但现实中主从机器往往要求放在一个机柜同一台交换设备下，网络闪断情况极低；再者主从同步在同步数量量大情况下，需要将缓存区调得足够大，不然也容易造成连接断开。
实现切换逻辑如下：A和B两台机器
1）A 、B机器依次启动，A机作为主、B机为从。
2）主A机挂掉，B机接管业务并作为主。
3）A机起来，作为从SLAVEOF B。
4）B机挂掉，A机再切回主。

在Keepalived 有两个角色：Master(一个)、Backup（多个），如果设置一个为Master，但Master挂了后再起来，必然再次业务又一次切换，这对于有状态服务是不可接受的。解决方案就是两台机器都设置为Backup，而且优先级高的Backup设置为nopreemt 不抢占。

部署记录：

0）服务器信息

192.168.10.205    redis-master     需要安装redis(3.2.0版本)、keepalived(1.3.2版本)
192.168.10.206    redis-slave      需要安装redis(3.2.0版本)、keepalived(1.3.2版本)
192.168.10.230    VIP

关闭两个节点机器的iptables和selinux（两个节点上都要操作）
[root@redis-master ~]# /etc/init.d/iptables stop
[root@redis-master ~]# vim /etc/sysconfig/selinux
......
SELINUX=disabled
[root@redis-master ~]# setenforce 0
[root@redis-master ~]# getenforce 
Permissive

1）安装redis服务及主从配置（两个节点机都要操作）

[root@redis-master ~]# cd /usr/local/src/
[root@redis-master src]# wget http://download.redis.io/releases/redis-3.2.0.tar.gz
[root@redis-master src]# tar -zvxf redis-3.2.0.tar.gz
[root@redis-master src]# cd redis-3.2.0
[root@redis-master redis-3.2.0]# make

添加相关文件及命令
[root@redis-master redis-3.2.0]# mkdir -p /usr/local/redis/bin/ 
[root@redis-master redis-3.2.0]# cd src
[root@redis-master src]# cp redis-benchmark redis-check-aof redis-check-rdb redis-cli redis-server redis-sentinel /usr/local/redis/bin/
[root@redis-master src]# cd ../
[root@redis-master redis-3.2.0]# cp redis.conf /etc/

添加redis启动脚本
[root@redis-master redis-3.2.0]# vim /etc/init.d/redis 
#!/bin/bash
#chkconfig: 2345 10 90
#description: Start and Stop redis

REDISPORT=6379

EXEC=/usr/local/redis/bin/redis-server

REDIS_CLI=/usr/local/redis/bin/redis-cli 

PIDFILE=/var/run/redis.pid

CONF="/etc/redis.conf"

case "$1" in

start)

if [ -f $PIDFILE ]

then

echo "$PIDFILE exists, process is already running or crashed"

else

echo "Starting Redis server..."

$EXEC $CONF

fi

if [ "$?"="0" ]

then

echo "Redis is running..."

fi

;;

stop)

if [ ! -f $PIDFILE ]

then

echo "$PIDFILE does not exist, process is not running"

else

PID=$(cat $PIDFILE)

echo "Stopping ..."

$REDIS_CLI -p $REDISPORT SHUTDOWN

while [ -x ${PIDFILE} ]

do

echo "Waiting for Redis to shutdown ..."

sleep 1

done

echo "Redis stopped"

fi

;;

restart|force-reload)

${0} stop

${0} start

;;

*)

echo "Usage: /etc/init.d/redis {start|stop|restart|force-reload}" >&2

exit 1

esac

添加执行权限
[root@redis-master redis-3.2.0]# chmod 755 /etc/init.d/redis

设置开机自启动
[root@redis-master redis-3.2.0]# chkconfig --add redis
[root@redis-master redis-3.2.0]# chkconfig redis on

创建redis状态日志
[root@redis-master redis-3.2.0]# mkdir /var/log/redis/
[root@redis-master redis-3.2.0]# touch /var/log/redis/redis.log

redis主从配置（先看下redis-master主节点的配置）
[root@redis-master redis-3.2.0]# vim /etc/redis.conf
.......
port 6379
.......
daemonize yes              #这个修改为yes
.......
bind 0.0.0.0               #绑定的主机地址。说明只能通过这个ip地址连接本机的redis。最好绑定0.0.0.0；注意这个不能配置成127.0.0.1，否则复制会失败！用0.0.0.0或者本机ip地址都可以
.......
pidfile /var/run/redis.pid
.......
logfile /var/log/redis/redis.log
.......
dir  /var/redis/redis      #redis数据目录
....... 
appendonly yes              #启用AOF持久化方式
appendfilename "appendonly.aof"       #AOF文件的名称，默认为appendonly.aof
appendfsync everysec        #每秒钟强制写入磁盘一次，在性能和持久化方面做了很好的折中，是受推荐的方式。
.....
save 900 1                 #启用RDB快照功能，默认就是启用的
save 300 10
save 60 10000              ＃即在多少秒的时间内，有多少key被改变的数据添加到.rdb文件里
.......
slave-serve-stale-data yes    #默认就会开启
slave-read-only yes
......
dbfilename dump.rdb       ＃快照文件名称
......

另一个从节点redis-slave的redis.conf配置和上面基本差不多，只是多了下面一行配置：
slaveof 192.168.10.205 6379

接着创建redis的数据目录
[root@redis-master redis-3.2.0]# mkdir -p /var/redis/redis

然后启动两个节点的redis服务
[root@redis-master redis-3.2.0]# /etc/init.d/redis start
Starting Redis server...
Redis is running...
[root@redis-master redis-3.2.0]# lsof -i:6379
COMMAND     PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
redis-ser 17265 root    4u  IPv4  59068      0t0  TCP *:6379 (LISTEN)

2）Keepalived安装（两个节点机都要操作）

[root@redis-master ~]# yum -y install openssl openssl-devel
[root@redis-master ~]# cd /usr/local/src/
[root@redis-master src]# wget http://www.keepalived.org/software/keepalived-1.4.0.tar.gz
[root@redis-master src]# tar -zvxf keepalived-1.4.0.tar.gz
[root@redis-master src]# cd keepalived-1.4.0
[root@redis-master keepalived-1.4.0]# ./configure && make && make install

文件配置
[root@redis-master keepalived-1.4.0]# mkdir /etc/keepalived
[root@redis-master keepalived-1.4.0]# mkdir /usr/local/keepalived/scripts/ -p
[root@redis-master keepalived-1.4.0]# cp /usr/local/src/keepalived-1.4.0/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/
[root@redis-master keepalived-1.4.0]# cp /usr/local/src/keepalived-1.4.0/keepalived/etc/init.d/keepalived /etc/init.d/
[root@redis-master keepalived-1.4.0]# cp /usr/local/sbin/keepalived /usr/sbin
[root@redis-master keepalived-1.4.0]# cp /usr/local/src/keepalived-1.4.0/keepalived/etc/sysconfig/keepalived /etc/sysconfig/

设置开机启动
[root@redis-master keepalived-1.4.0]# chmod +x /etc/init.d/keepalived
[root@redis-master keepalived-1.4.0]# chkconfig --add keepalived
[root@redis-master keepalived-1.4.0]# chkconfig keepalived on

redis主从配置简单说明

redis的主从复制实现简单却功能强大，其具有以下特点：
1）一个master支持多个slave连接，slave可以接受其他slave的连接
2）主从同步时，master和slave都是非阻塞的

redis主从复制可以用来：
1）data redundancy（数据冗余）
2）slave作为master的扩展，提供一些read-only的服务
3）可以将数据持久化放在slave做，从而提升master性能

通过简单的配置slave（master端无需配置），用户就能使用redis的主从复制，即只需在slave端的redis.conf文件中配置下面一行：
slaveof <masterip> <masterport>
表示该redis服务作为slave，masterip和masterport分别为master 的ip和port

其他配置：
masterauth <master-password>
如果master设置了安全密码，则此处设置为相应的密码

slave-serve-stale-data yes  当slave丢失master或者同步正在进行时，如果发生对slave的服务请求：
slave-serve-stale-data设置为yes则slave依然正常提供服务
slave-serve-stale-data设置为no则slave返回client错误："SYNC with master in progress"

repl-ping-slave-period 10
slave发送PINGS到master的时间间隔

repl-timeout 60
IO超时时间

3）redis+keepalived配置

a）先进行redis-master主节点的高可用配置
[root@redis-master ~]# cp /etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf.bak
[root@redis-master ~]# vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived

global_defs {
  router_id redis-master
}

vrrp_script chk_redis {
  script "/usr/local/keepalived/scripts/redis_check.sh 127.0.0.1 6379"        #监控脚本  
  interval 2                                    #监控时间
  timeout 2                                     #超时时间
  fall 3
}

vrrp_instance redis {
  state BACKUP                              
  interface eth0
  lvs_sync_daemon_interface eth0
  virtual_router_id 202
  priority 150                              #权重值                  
  nopreempt                                 #nopreempt：设置不抢占，这里只能设置在state为backup的节点上，而且这个节点的优先级必须比另外节点的高
  advert_int 1

  authentication {                          #all node must same              
    auth_type PASS                            #加密    
    auth_pass 1111                            #密码 
  }

  virtual_ipaddress {
    192.168.10.230                            #VIP地址                  
  }

  track_script {
    chk_redis
  }

notify_master "/usr/local/keepalived/scripts/redis_master.sh 127.0.0.1 192.168.10.206 6379"
notify_backup "/usr/local/keepalived/scripts/redis_backup.sh 127.0.0.1 192.168.10.206 6379"
notify_fault /usr/local/keepalived/scripts/redis_fault.sh
notify_stop /usr/local/keepalived/scripts/redis_stop.sh
}

b）接着进行redis-slave从节点的高可用配置
[root@redis-slave ~]# cp /etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf.bak
[root@redis-slave ~]# vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived

global_defs {
  router_id redis-slave
}

vrrp_script chk_redis{
  script "/usr/local/keepalived/scripts/redis_check.sh 127.0.0.1 6379"                            
  interval 2                                     
  timeout 2                                     
  fall 3                                    
}

vrrp_instance redis {
  state BACKUP                                     
  interface eth0                                
  lvs_sync_daemon_interface eth0                      
  virtual_router_id 202
  priority  100                               
  nopreempt 
  advert_int 1

  authentication {   
    auth_type PASS                               
    auth_pass 1111                         
  }

  virtual_ipaddress {
    192.168.10.230                                     
  }

  track_script {
  chk_redis
  }

notify_master "/usr/local/keepalived/scripts/redis_master.sh 127.0.0.1 192.168.10.205 6379"
notify_backup "/usr/local/keepalived/scripts/redis_backup.sh 127.0.0.1 192.168.10.205 6379"
notify_fault /usr/local/keepalived/scripts/redis_fault.sh
notify_stop /usr/local/keepalived/scripts/redis_stop.sh
}

c）在redis-master和redis-slave两个节点机器上都要创建监控脚本（下面几个脚本，在两个节点上都要同样配置）
首先配置监控脚本
[root@redis-master ~]# vim /usr/local/keepalived/scripts/redis_check.sh 
#!/bin/bash
ALIVE=`/usr/local/redis/bin/redis-cli -h $1 -p $2 PING`
LOGFILE="/var/log/keepalived-redis-check.log"
echo "[CHECK]" >> $LOGFILE
date >> $LOGFILE
if [ $ALIVE == "PONG" ]; then :
   echo "Success: redis-cli -h $1 -p $2 PING $ALIVE" >> $LOGFILE 2>&1
    exit 0
else
    echo "Failed:redis-cli -h $1 -p $2 PING $ALIVE " >> $LOGFILE 2>&1
    exit 1
fi

需要注意的是：
以下负责运作的关键脚本：
notify_master /usr/local/keepalived/scripts/redis_master.sh
notify_backup /usr/local/keepalived/scripts/redis_backup.sh
notify_fault /usr/local/keepalived/scripts/redis_fault.sh
notify_stop /usr/local/keepalived/scripts/redis_stop.sh

因为Keepalived在转换状态时会依照状态来呼叫：
当进入Master状态时会呼叫notify_master
当进入Backup状态时会呼叫notify_backup
当发现异常情况时进入Fault状态呼叫notify_fault
当Keepalived程序终止时则呼叫notify_stop

   温馨提示：
   以上的keepalived.conf文件中的切换模式设置为nopreempt，意思是：
   不抢占VIP资源，此种模式要是所有的节点都必须设置为state BACKUP模式！
   需要注意无论主备服务器都需要设置为BACKUP，与以往KeepAlived的配置不同，其目的就是防止主服务器恢复后重新抢回VIP，导致Redis切换从而影响稳定。

接着在redis-master主节点上创建notity_master与notify_backup脚本：
[root@redis-master ~]# vim /usr/local/keepalived/scripts/redis_master.sh
#!/bin/bash
REDISCLI="/usr/local/redis/bin/redis-cli -h $1 -p $3"
LOGFILE="/var/log/keepalived-redis-state.log"
echo "[master]" >> $LOGFILE
date >> $LOGFILE
echo "Being master...." >> $LOGFILE 2>&1
echo "Run SLAVEOF cmd ... " >> $LOGFILE
$REDISCLI SLAVEOF $2 $3 >> $LOGFILE  2>&1

#echo "SLAVEOF $2 cmd can't excute ... " >> $LOGFILE
sleep 10                                               #延迟10秒以后待数据同步完成后再取消同步状态
echo "Run SLAVEOF NO ONE cmd ..." >> $LOGFILE
$REDISCLI SLAVEOF NO ONE >> $LOGFILE 2>&1

[root@redis-master ~]# vim /usr/local/keepalived/scripts/redis_backup.sh
#!/bin/bash
REDISCLI="/usr/local/redis/bin/redis-cli"
LOGFILE="/var/log/keepalived-redis-state.log"
echo "[BACKUP]" >> $LOGFILE
date >> $LOGFILE
echo "Being slave...." >> $LOGFILE 2>&1
echo "Run SLAVEOF cmd ..." >> $LOGFILE 2>&1
$REDISCLI SLAVEOF $2 $3 >> $LOGFILE
sleep 100                                             #延迟100秒以后待数据同步完成后再取消同步状态
exit(0)

[root@redis-master ~]# vim /usr/local/keepalived/scripts/redis_fault.sh
#!/bin/bash
LOGFILE="/var/log/keepalived-redis-state.log"
echo "[fault]" >> $LOGFILE
date >> $LOGFILE

[root@redis-master ~]# vim /usr/local/keepalived/scripts/redis_stop.sh
#!/bin/bash
LOGFILE="/var/log/keepalived-redis-state.log"
echo "[stop]" >> $LOGFILE
date >> $LOGFILE

[root@redis-master ~]# chmod 755 /usr/local/keepalived/scripts/*.sh
[root@redis-master ~]# ll /usr/local/keepalived/scripts/
total 20
-rwxr-xr-x. 1 root root 283 May  7 07:20 redis_backup.sh
-rwxr-xr-x. 1 root root 360 May  7 07:12 redis_check.sh
-rwxr-xr-x. 1 root root 102 May  7 07:22 redis_fault.sh
-rwxr-xr-x. 1 root root 445 May  7 07:16 redis_master.sh
-rwxr-xr-x. 1 root root 101 May  7 07:23 redis_stop.sh

将redis-master主节点上的上面5个脚本直接复制到redis-slave节点上即可。
[root@redis-master ~]# rsync -e "ssh -p22" -avpgolr /usr/local/keepalived/scripts/*.sh root@192.168.10.206:/usr/local/keepalived/scripts/

到redis-slave从节点上查看脚本：
[root@redis-slave ~]# ll /usr/local/keepalived/scripts/
total 20
-rwxr-xr-x. 1 root root 283 May  7 07:20 redis_backup.sh
-rwxr-xr-x. 1 root root 360 May  7 07:12 redis_check.sh
-rwxr-xr-x. 1 root root 102 May  7 07:22 redis_fault.sh
-rwxr-xr-x. 1 root root 445 May  7 07:16 redis_master.sh
-rwxr-xr-x. 1 root root 101 May  7 07:23 redis_stop.sh

d）设置环境变量（两个节点上都要设置）
[root@redis-master ~]# vim /etc/profile
......
export PATH=$PATH:/usr/local/redis/bin
[root@redis-master ~]# source /etc/profile

e）启动两个节点上的keepalived服务
[root@redis-master ~]# /etc/init.d/keepalived start
Starting keepalived:                                       [  OK  ]
[root@redis-master ~]# ps -ef|grep keepalived
root     32509     1  0 07:29 ?        00:00:00 keepalived -D
root     32510 32509  0 07:29 ?        00:00:00 keepalived -D
root     32512 32509  0 07:29 ?        00:00:00 keepalived -D
root     32515 32512  0 07:29 ?        00:00:00 keepalived -D
root     32517 32515  0 07:29 ?        00:00:00 /bin/bash /usr/local/keepalived/scripts/redis_backup.sh 127.0.0.1 192.168.10.206 6379
root     32529 14122  0 07:29 pts/1    00:00:00 grep keepalived

[root@redis-slave ~]# /etc/init.d/keepalived start
Starting keepalived:                                       [  OK  ]
[root@redis-slave ~]# ps -ef|grep keepalived
root     22277     1  0 07:29 ?        00:00:00 keepalived -D
root     22278 22277  0 07:29 ?        00:00:00 keepalived -D
root     22279 22277  0 07:29 ?        00:00:00 keepalived -D
root     22283 22279  0 07:29 ?        00:00:00 keepalived -D
root     22284 22283  0 07:29 ?        00:00:00 /bin/bash /usr/local/keepalived/scripts/redis_backup.sh 127.0.0.1 192.168.10.205 6379
root     22289 10868  0 07:29 pts/1    00:00:00 grep keepalived

查看下redis-master主节点，发现vip资源已经有了
[root@redis-master ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:b1:9c:93 brd ff:ff:ff:ff:ff:ff
    inet 192.168.10.205/24 brd 192.168.10.255 scope global eth0
    inet 192.168.10.230/32 scope global eth0
    inet6 fe80::5054:ff:feb1:9c93/64 scope link 
       valid_lft forever preferred_lft forever

4）redis+keepalived主从高可用故障切换测试

a）分别启动redis-master和redis-slave两个节点的redis和keepalived服务（如上已启动）
  
b）尝试通过VIP连接Redis:
[root@redis-master ~]# redis-cli -h 192.168.10.230 INFO|grep role
role:master
[root@redis-master ~]# redis-cli -h 192.168.10.205 INFO|grep role
role:master
[root@redis-master ~]# redis-cli -h 192.168.10.206 INFO|grep role
role:slave
连接成功，Slave也连接上来了。
  
c）尝试插入一些数据：
[root@redis-master ~]# redis-cli -h 192.168.10.230 SET Hello Redis
OK
  
从VIP读取数据
[root@redis-master ~]# redis-cli -h 192.168.10.230 GET Hello
"Redis"
  
从redis-master主节点读取数据
[root@redis-master ~]# redis-cli -h 192.168.10.205 GET Hello
"Redis"
  
从redis-slave从节点读取数据
[root@redis-master ~]# redis-cli -h 192.168.10.206 GET Hello
"Redis"
  
e）然后开始模拟故障产生：
将redis-master主节点上的redis进程杀死：
[root@redis-master ~]# ps -ef|grep redis
root      4500 14122  0 08:04 pts/1    00:00:00 grep redis
root     17265     1  0 04:00 ?        00:00:07 /usr/local/redis/bin/redis-server 0.0.0.0:6379 
[root@redis-master ~]# kill -9 17265
[root@redis-master ~]# ps -ef|grep redis
root      4514 14122  0 08:04 pts/1    00:00:00 grep redis
  
查看redis-master主节点上的Keepalived日志
[root@redis-master ~]# tail -f /var/log/keepalived-redis-state.log
OK
[master]
Mon May  7 07:29:17 CST 2018
Being master....
Run SLAVEOF cmd ...
OK Already connected to specified master
Run SLAVEOF NO ONE cmd ...
OK
[fault]
Mon May  7 08:05:00 CST 2018
  
同时redis-slave从节点上的日志显示：
[root@redis-slave ~]# tail -f /var/log/keepalived-redis-state.log
Being slave....
Run SLAVEOF cmd ...
OK
[master]
Mon May  7 08:05:02 CST 2018
Being master....
Run SLAVEOF cmd ...
OK Already connected to specified master
Run SLAVEOF NO ONE cmd ...
OK
  
然后我们可以发现，redis-slave从节点已经接管服务，并且担任Master的角色了。
redis-slave从节点上已经接管过来VIP资源了（大概需要等待2秒左右的时间，vip资源就切过来了）
[root@redis-slave ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:dd:84:6b brd ff:ff:ff:ff:ff:ff
    inet 192.168.10.206/24 brd 192.168.10.255 scope global eth0
    inet 192.168.10.230/32 scope global eth0
    inet6 fe80::5054:ff:fedd:846b/64 scope link
       valid_lft forever preferred_lft forever
  
[root@redis-slave ~]# redis-cli -h 192.168.10.230 INFO|grep role
role:master
[root@redis-slave ~]# redis-cli -h 192.168.10.205 INFO|grep role
Could not connect to Redis at 192.168.10.205:6379: Connection refused
[root@redis-slave ~]# redis-cli -h 192.168.10.206 INFO|grep role
role:master
  
=======================================================================
然后再恢复redis-master主节点的redis进程
[root@redis-master ~]# /etc/init.d/redis start
/var/run/redis.pid exists, process is already running or crashed
Redis is running...
[root@redis-master ~]# rm -f /var/run/redis.pid
[root@redis-master ~]# /etc/init.d/redis start
Starting Redis server...
Redis is running...
[root@redis-master ~]# ps -ef|grep redis
root      4969     1  0 08:08 ?        00:00:00 /usr/local/redis/bin/redis-server 0.0.0.0:6379 
root      4977  4976  0 08:08 ?        00:00:00 /bin/bash /usr/local/keepalived/scripts/redis_backup.sh 127.0.0.1 192.168.10.206 6379
root      4987 14122  0 08:08 pts/1    00:00:00 grep redis
[root@redis-master ~]# lsof -i:6379
COMMAND    PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
redis-ser 4969 root    4u  IPv4  93698      0t0  TCP *:6379 (LISTEN)
redis-ser 4969 root    6u  IPv4  93709      0t0  TCP 192.168.10.205:43299->192.168.10.206:6379 (ESTABLISHED)
  
查看redis-master上的Keepalived日志
[root@redis-master ~]# tail -f /var/log/keepalived-redis-state.log
OK Already connected to specified master
Run SLAVEOF NO ONE cmd ...
OK
[fault]
Mon May  7 08:05:00 CST 2018
[BACKUP]
Mon May  7 08:08:34 CST 2018
Being slave....
Run SLAVEOF cmd ...
OK
  
查看redis-slave上的Keepalived日志
[root@redis-slave ~]# tail -f /var/log/keepalived-redis-state.log
Being slave....
Run SLAVEOF cmd ...
OK
[master]
Mon May  7 08:05:02 CST 2018
Being master....
Run SLAVEOF cmd ...
OK Already connected to specified master
Run SLAVEOF NO ONE cmd ...
OK
  
[root@redis-master ~]# redis-cli -h 192.168.10.230 INFO|grep role
role:master
[root@redis-master ~]# redis-cli -h 192.168.10.205 INFO|grep role
role:slave
[root@redis-master ~]# redis-cli -h 192.168.10.206 INFO|grep role
role:master
  
发现redis-master的redis服务再次启动后，redis-master主节点成为salve角色了，redis-slave从节点还是master角色。
当redis-slave节点宕机或redis服务关闭后，redis-master节点再次接管服务成为master角色，如此循环~~
  
关闭redis-slave从节点的reids服务
[root@redis-slave ~]# ps -ef|grep redis
root     15407     1  0 04:00 ?        00:00:10 /usr/local/redis/bin/redis-server 0.0.0.0:6379 
root     22900 10868  0 08:11 pts/1    00:00:00 grep redis
[root@redis-slave ~]# kill -9 15407
[root@redis-slave ~]# ps -ef|grep redis
root     22902 10868  0 08:11 pts/1    00:00:00 grep redis
  
查看redis-slave上的Keepalived日志
[root@redis-slave ~]# tail -f /var/log/keepalived-redis-state.log
.......
[stop]          //测试时发现，当redis-slave的redis服务关闭后，还需要重启或关闭keepalived，才能将vip资源漂到redis-master节点上，所以日志里也就会出现这个stop信息
Mon May  7 09:25:03 CST 2018
[BACKUP]
Mon May  7 09:25:04 CST 2018
Being slave....
Run SLAVEOF cmd ...
OK
 
查看redis-master上的Keepalived日志
[root@redis-master ~]# tail -f /var/log/keepalived-redis-state.log
.......
[master]
Mon May  7 09:25:03 CST 2018
Being master....
Run SLAVEOF cmd ...
OK Already connected to specified master
Run SLAVEOF NO ONE cmd ...
OK
 
查看redis-master，发现VIP资源已经接管过来了（如果没有按时切过来的话，只需重启或关闭redis-slave节点那边的keepalived服务即可）
[root@redis-master ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:b1:9c:93 brd ff:ff:ff:ff:ff:ff
    inet 192.168.10.205/24 brd 192.168.10.255 scope global eth0
    inet 192.168.10.230/32 scope global eth0
    inet6 fe80::5054:ff:feb1:9c93/64 scope link
       valid_lft forever preferred_lft forever
  
[root@redis-master ~]# redis-cli -h 192.168.10.230 INFO|grep role
role:master
[root@redis-master ~]# redis-cli -h 192.168.10.205 INFO|grep role
role:master
[root@redis-master ~]# redis-cli -h 192.168.10.206 INFO|grep role
Could not connect to Redis at 192.168.10.206:6379: Connection refused
  
发现redis-maste节点已经转变为master角色了。
  
同样，当reids-slave节点的redis服务重新启动后，它将成为slave角色。
[root@redis-slave ~]# /etc/init.d/redis start
/var/run/redis.pid exists, process is already running or crashed
Redis is running...
[root@redis-slave ~]# rm -f /var/run/redis.pid
[root@redis-slave ~]# /etc/init.d/redis start
Starting Redis server...
Redis is running...
[root@redis-slave ~]# lsof -i:6379
COMMAND     PID USER   FD   TYPE  DEVICE SIZE/OFF NODE NAME
redis-ser 23244 root    4u  IPv4 3049509      0t0  TCP *:6379 (LISTEN)
redis-ser 23244 root    6u  IPv4 3049513      0t0  TCP dns.kevin.cn:44931->192.168.10.205:6379 (ESTABLISHED)
  
[root@redis-master ~]# redis-cli -h 192.168.10.230 INFO|grep role
role:master
[root@redis-master ~]# redis-cli -h 192.168.10.205 INFO|grep role
role:master
[root@redis-master ~]# redis-cli -h 192.168.10.206 INFO|grep role
role:slave