keepalived+lvs 部署

环境配置：

node	service	VIP
node2010(192.168.20.10)	keepalived(master)+lvs	192.168.20.22
node2011(192.168.20.11)	keepalived(backup)+lvs
node2012(192.168.20.12)	httpd(RS)

lvs相关内容请翻看另外内容

安装

$ yum install -y curl gcc openssl-devel libnl3-devel net-snmp-devel
$ tar xf keepalived-2.0.19.tar.gz
$ cd keepalived-2.0.19
$ ./configure --prefix=/usr/local/keepalived
$ make -j 4 && make install
$ tree -L 3 /usr/local/keepalived
/usr/local/keepalived
├── bin
│   └── genhash
├── etc
│   ├── keepalived
│   │   ├── keepalived.conf  #主配置文件
│   │   └── samples
│   └── sysconfig
│       └── keepalived
├── sbin
│   └── keepalived
└── share
    ├── doc
    │   └── keepalived
    ├── man
    │   ├── man1
    │   ├── man5
    │   └── man8
    └── snmp
        └── mibs

#添加systemd
$ vim /usr/lib/systemd/system/keepalived.service
[Unit]
Description=LVS and VRRP High Availability Monitor
After=network-online.target syslog.target
Wants=network-online.target

[Service]
Type=forking
PIDFile=/run/keepalived.pid
KillMode=process
EnvironmentFile=/usr/local/keepalived/etc/sysconfig/keepalived
ExecStart=/usr/local/keepalived/sbin/keepalived  -f /usr/local/keepalived/etc/keepalived/keepalived.conf
ExecReload=/bin/kill -HUP $MAINPID

[Install]
WantedBy=multi-user.target

$ systemctl daemon-reload && systemctl enable keepalived.service 

配置keepalived.conf

$ cat /usr/local/keepalived/etc/keepalived/keepalived.conf
!Configuration File for keepalived

global_defs {     //全局定义，邮件配置。
#        notification_email {
#        localhost@root.com
#}
#        notification_email_from root
#        smtp_server 127.0.0.1   
#        smtp_connect_timeout 30
        router_id LVS_DEVEL   //标识虚拟路由ID,主从得不同
        vrrp_mcast_group4 224.0.10.10  //组播地址，用来发送VRRP报文
#        enable_script_security
#        script_user root    //脚本执行者，建议使用非root来执行
}
#vrrp_script chk_schedown {  //自检可以通过第三方监控程序zabbix,prometheus
#        script "[ -e /etc/keepalived/down ] && exit 1 || exit 0"
#        interval 1
#        weight -11
#        fall 2
#        rise 1
#}
#vrrp_script chk_nginx {     //此处检测nginx只需在主上有即可，如果从上也有话，就会导致主上的nginx，down的一瞬间，又起来了，优先级混乱
#        script "`killall -0 nginx` && exit 0 || exit 1" 
#        interval 1
#        weight -11
#        fall 2
#        rise 1
#}

vrrp_instance VI_1 {               
        state MASTER         //主，BACKUP为备
        interface ens192     //虚拟IP绑定的网卡
        virtual_router_id 10 //虚拟路由ID组，这个主从得相同
        priority 100         //优先级，高的为MASTER，BACKUP需比此小
        advert_int 1         //一秒一次VRRP报文
        authentication {     //VRRP报文简单加密
                auth_type PASS    
                auth_pass 12345678
        }
        virtual_ipaddress { //VIP地址，也可写多个
                192.168.20.22 
        }

#        track_script {          //调用脚本，做健康状态检测
#                chk_schedown
#                #chk_nginx
#        }
#notify_master "/etc/keepalived/notify.sh master"
#notify_backup "/etc/keepalived/notify.sh backup"
#notify_fault "/etc/keepalived/notify.sh fault"
notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master" #状态发生变化时执行的脚本
}

#keepalived与lvs结合时的配置
virtual_server fwmark 2 { #IP port(VIP)、fwmark int(防火墙标记)、group string(虚拟服务器组)
        delay_loop 6      //延迟定时器用于服务轮询 
        lb_algo wrr       //调度算法   
        lb_kind DR        //lvs模式
#        nat_mask 255.255.255.0   //新版本keepavlied不支持以下两种配置
#        persistence_timeout 0    
        protocol TCP                       
        real_server 192.168.20.3 {    //后端RS
                weight 1              //权重
                TCP_CHECK {           //RS健康检测，检测方式有：HTTP_GET|SSL_GET|TCP_CHECK|SMTP_CHECK|MISC_CHECK          
                        connect_timeout 10   
                        #nb_get_retry 3
                        delay_before_retry 3
                        connect_port 80
                }
        }
        real_server 192.168.20.12 {
                weight 3
                TCP_CHECK {
                        connect_timeout 10
                        #nb_get_retry 3
                        delay_before_retry 3
                        connect_port 80
                }
        }
}

脚本(主要作用是添加iptables mark 标记)

mark标记是为了使用lvs能够映射多个端口至RS

$ cat /usr/local/keepalived/etc/keepalived/notify.sh
#!/bin/bash

VIP=192.168.20.22

case $1 in 
master)
  iptables -t mangle -L > /tmp/iptables
  mangle_rule=`grep "multiport dports http,https MARK set 0x2" /tmp/iptables | wc -l`
  if [ $mangle_rule -ne 1 ];then
    iptables -t mangle -F
    iptables -t mangle -A PREROUTING -d $VIP -p tcp -m multiport --dports 80,443 -j MARK --set-mark 2
    [ $? -ne 0 ] && echo "iptables write false!!!";systemctl stop keepalived.service  
  fi
    ;;

*)
  echo "Please write iptables to add mangle rules. "
esac

启动服务

$ systemctl start keepalived.service //MASTER 机器查看
$ ip a
...
2: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:0c:29:7c:ab:08 brd ff:ff:ff:ff:ff:ff
    inet 192.168.20.10/24 brd 192.168.20.255 scope global ens192
       valid_lft forever preferred_lft forever
    inet 192.168.20.22/32 scope global ens192    #VIP已经加入
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fe7c:ab08/64 scope link 
       valid_lft forever preferred_lft forever

$ ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
FWM  2 wrr
  -> 192.168.20.12:0              Route   3      0          0    #ipvs 规则已经加入，这里192.168.20.3未安装httpd,所以TCP检测失败次数超过设置后将规则移除

$ iptables -t mangle -L   #防火墙标记添加，这样http,https端口都可以转发至RS上
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         
MARK       tcp  --  anywhere             node2010             multiport dports http,https MARK set 0x2
...


#从内网其它机器访问VIP
$~]# curl http://192.168.20.22
12

抓包查看VRRP通信过程

$ tcpdump  -i ens192 host 224.0.10.10
11:56:25.574398 IP node2010 > 224.0.10.10: VRRPv2, Advertisement, vrid 10, prio 100, authtype simple, intvl 1s, length 20
11:56:26.574467 IP node2010 > 224.0.10.10: VRRPv2, Advertisement, vrid 10, prio 100, authtype simple, intvl 1s, length 20
...

关闭MASTER，并查看VIP是否漂移至BACKUP

node2010$ systemctl stop keepalived   

node2011$  less /var/log/message
Jan 16 17:40:43 node2011 Keepalived_vrrp[25872]: (VI_1) Backup received priority 0 advertisement   #Backup主机接收到优先级为0的VRRP
Jan 16 17:40:44 node2011 Keepalived_vrrp[25872]: (VI_1) Entering MASTER STATE                      #0 < 90 ,于是BACKUP转换成MASTER

node2011$ ip a
2: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:0c:29:f3:c4:42 brd ff:ff:ff:ff:ff:ff
    inet 192.168.20.11/24 brd 192.168.20.255 scope global ens192
       valid_lft forever preferred_lft forever
    inet 192.168.20.22/32 scope global ens192
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fef3:c442/64 scope link 
       valid_lft forever preferred_lft forever

node2011$ ipvsadm -ln      #切换成功，VIP已经漂移
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
FWM  2 wrr
  -> 192.168.20.12:0              Route   3      0          0   

监控

最有效的监控方式一、使用自带的邮件发送功能，二、监控日志，状态转换会写日志也会触发脚本。脚本和第三方监控程序还是会有时间间隔。
建议使用服务自带的邮件功能即可，实时性最好。同时也可以用prometheus对进程状态进行监控。双管其下。

结语

在操作过程中，可以通过arp -a来查看VIP与对应MAC地址是否相匹配。一但出现不匹配那就是你配置问题了。

参考资料：

• https://weizhimiao.github.io/2017/02/11/keepalived%E9%85%8D%E7%BD%AE%E6%95%B4%E7%90%86/