LVS粗讲

该文章不对 LVS 和 keepalived 的理论展开介绍，本次做的是拓扑中红色虚线中的部分

主机配置

lvs/172.16.186.111/1G/NAT/1U/80G/VIP:172.16.186.109/CentOS7.9.2009
web1/172.16.186.112/1G/NAT/1U/80G/VIP:172.16.186.109/CentOS7.9.2009
web2/172.16.186.113/1G/NAT/1U/80G/VIP:172.16.186.109/CentOS7.9.2009
keepalived/172.16.186.114/NAT/1G/1U/80G/VIP:172.16.186.109/CentOS7.9.2009

lvs端设置

lvs端网络设置
[root@lvs ~]# systemctl stop firewalld && systemctl disable firewalld
[root@lvs ~]# setenforce 0
[root@lvs ~]# sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config
[root@lvs ~]# systemctl stop NetworkManager && systemctl disable NetworkManager
[root@lvs ~]# cd /etc/sysconfig/network-scripts/
[root@lvs network-scripts]# cp ifcfg-ens33{,:0}       ===> 在物理网卡上开一个子接口，0可自定义
[root@lvs network-scripts]# vim ifcfg-ens33:0         ===> 配置子接口
TYPE="Ethernet"
BOOTPROTO="static"
NAME="ens33:0"
DEVICE="ens33:0"                                      ===> 这里和子接口文件名匹配
ONBOOT="yes"
IPADDR="172.16.186.109"                               ===> 这里写vip
PREFIX="24"                                           ===> 掩码必须写


[root@lvs network-scripts]# cd
[root@lvs ~]# systemctl restart network
[root@lvs ~]# ifconfig                                ===> 使用ip a查看网卡时186.109的地址会在ens33网卡配置中


安装lvs集群管理工具
[root@lvs ~]# mount /dev/sr0 /mnt/usb1                ===>挂载光盘
[root@lvs ~]# yum -y install /mnt/usb1/Packages/ipvsadm-1.27-8.el7.x86_64.rpm    ===> 未有互联网时使用该方式，有互联网可直接yum安装

配置web端

所有web端网络配置相同

web1端网络设置
[root@web* ~]# systemctl stop firewalld && systemctl disable firewalld
[root@web* ~]# setenforce 0
[root@web* ~]# sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config
[root@web* ~]# systemctl stop NetworkManager && systemctl disable NetworkManager
[root@web* ~]# cd /etc/sysconfig/network-scripts/
[root@web* network-scripts]# cp ifcfg-lo{,:0}          ===> 在回环口上开一个子接口
[root@web* network-scripts]# vim ifcfg-lo:0            ===> 配置回环子接口
DEVICE=lo:0                      ===> 名字要改
IPADDR=172.16.186.109            ===> vip地址
NETMASK=255.255.255.255          ===> 全部是255,表示自己是一个网段
NETWORK=127.0.0.0
BROADCAST=127.255.255.255
ONBOOT=yes
NAME=loopback

启动lo:0网卡
[root@web* network-scripts]# ifup lo:0
[root@web* network-scripts]# ifconfig         ===> 使用ip a查看时，子接口地址在lo网卡中

所有web端设置arp相应级别
arp-ignore：arp响应级别(是否接受外来的请求)
0：只要本机配置了ip就能响应请求
1：请求的目标地址到达对应的网络接口才会响应请求

arp-announce：arp通过行为(是否主动向外宣告自己)
0：本机上任何网络接口都向外通告，所有的网卡都能接受到通告
1：尽可能避免网卡与不匹配的目标进行通过
2：只在本网卡通告

开始设置(所有web端设置相同)
[root@web* ~]# vim /etc/sysctl.conf
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.default.arp_ignore = 1
net.ipv4.conf.default.arp_announce = 2
net.ipv4.conf.lo.arp_ignore = 1
net.ipv4.conf.lo.arp_announce = 2

[root@web* ~]# sysctl -p

添加临时路由
作用：会接收到lvs集群IP(186.109)的数据报文，接收到后会交给lo:0网卡处理
[root@web* ~]# route add -host 172.16.186.109 dev lo:0
注：如没有route命令可用yum安装net-tools工具

[root@web* ~]# route -n | grep "186.109"
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         172.16.186.2    0.0.0.0         UG    100    0        0 ens33
172.16.186.0    0.0.0.0         255.255.255.0   U     100    0        0 ens33
172.16.186.109  0.0.0.0         255.255.255.255 UH    0      0        0 lo        ===> 所有来自186.109的请求都送到lo:0网卡处理


永久添加路由
[root@web* ~]# echo "route add -host 172.16.186.109 dev lo:0">>/etc/rc.local 


安装httpd
[root@web* ~]# yum -y install httpd
[root@web1 ~]# echo "<h1>This is web1</h1>" >> /var/www/html/index.html
[root@web2 ~]# echo "<h1>This is web2</h1>" >> /var/www/html/index.html
[root@web* ~]# systemctl start httpd && systemctl enable httpd

配置IPVS集群

创建ipvs集群
[root@lvs ~]# ipvsadm -A -t 172.16.186.109:80 -s rr                    ===> rr 为轮询模式

往集群中添加节点
[root@lvs ~]# ipvsadm -a -t 172.16.186.109:80 -r 172.16.186.112:80 -g     ===> -g是dr模式，-m是NAT模式，-i是tun模式
[root@lvs ~]# ipvsadm -a -t 172.16.186.109:80 -r 172.16.186.113:80 -g

查看集群情况
[root@lvs ~]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  172.16.186.109:80 rr
  -> 172.16.186.112:80            Route   1      0          0         
  -> 172.16.186.113:80            Route   1      0          0 


查看集群状态
[root@lvs ~]# ipvsadm -ln --stats
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port               Conns   InPkts  OutPkts  InBytes OutBytes
  -> RemoteAddress:Port
TCP  172.16.186.109:80                   4       33        0     6617        0
  -> 172.16.186.112:80                   2       17        0     3188        0
  -> 172.16.186.113:80                   2       16        0     3429        0

注：OutPkts、OutBytes这两列为数据包出去时的记录，这里看到出去时没有经过lvs，lvs集群模式为dr


非集群内的且和集群中同一个网段的IP使访问186.109进行测试
http://172.16.186.109
注：多次访问集群地址，但一直是访问的是后台的一台机器，并没有对后端的2台机器进行轮询，这是因为lvs的持久化机制，默认持久化时间是300秒，可使用下面命令进行查询详情
[root@lvs ~]# man ipvsadm | grep -w "--persistent"

可修改持久的时间(该操作仅限于看效果，生产中不用设置)
[root@lvs ~]# ipvsadm -E -t 172.16.186.109:80 -s rr -p 3      ===>设置持久时间为3秒

再次查看集群情况
[root@lvs ~]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  172.16.186.109:80 rr persistent 3                           ====>这里已经将持久时间进行了改变
  -> 172.16.186.112:80            Route   1      0          1         
  -> 172.16.186.113:80            Route   1      0          0  


再次访问集群地址发现还是不行，这是因为还有一个tcp/udp的超时时间未设置
[root@lvs ~]# man ipvsadm
--set tcp tcpfin udp
     Change the timeout values used for IPVS connections. This command always takes 3 parameters,  representing  the  timeout   val‐
     ues  (in  seconds)  for  TCP  sessions, TCP sessions after receiving a  FIN packet, and  UDP  packets, respectively.  A timeout
     value 0 means that the current timeout value of the  corresponding  entry  is preserved.


设置tcp/udp的超时时间
[root@lvs ~]# ipvsadm --set 1 1 1        ===> 分别表示TCP会话时间、接收FIN数据包后的TCP会话时间、UDP数据包的ues时间

查看超时时间
[root@lvs ~]# ipvsadm -lnc
IPVS connection entries
pro expire state       source             virtual            destination

注：这里还没有请求，这时刷新一次web页面(是刷新一次)，再查看超时的情况会得到如下回显

[root@lvs ~]# ipvsadm -lnc
IPVS connection entries
pro expire state       source             virtual            destination
TCP 00:00  NONE        172.16.186.1:0     172.16.186.109:80  172.16.186.112:80
注：expire列是超时时间，因为设置的TCP会话、接收FIN数据包后的TCP会话、UDP数据包的ues都是1秒所以这里会很快超时，进而expire显示的为0，再次ipvsadm -lnc查看即显示为空


如在web页面上刷新了n次则expire显示的时间依然是1秒后失效，如执行下面命令时且不断刷新页面会得到类似下面的回显
[root@lvs ~]# for (( i = 0; i < 20; i=(i+1) ));do ipvsadm -lnc && sleep 1;done
IPVS connection entries
pro expire state       source             virtual            destination
TCP 00:59  NONE        172.16.186.1:0     172.16.186.109:80  172.16.186.112:80
IPVS connection entries                                                            ===> 第一个周期
pro expire state       source             virtual            destination
TCP 00:58  NONE        172.16.186.1:0     172.16.186.109:80  172.16.186.112:80
IPVS connection entries                                                            ===> 第二个周期
pro expire state       source             virtual            destination
TCP 00:59  NONE        172.16.186.1:0     172.16.186.109:80  172.16.186.112:80
TCP 00:00  ESTABLISHED 172.16.186.1:44234 172.16.186.109:80  172.16.186.112:80     ===> 1秒内新建了2个连接
TCP 00:00  ESTABLISHED 172.16.186.1:44236 172.16.186.109:80  172.16.186.112:80
IPVS connection entries                                                            ===> 第三个周期
pro expire state       source             virtual            destination
TCP 00:58  NONE        172.16.186.1:0     172.16.186.109:80  172.16.186.112:80
TCP 00:00  ESTABLISHED 172.16.186.1:44234 172.16.186.109:80  172.16.186.112:80
IPVS connection entries                                                            ===> 第四个周期...
pro expire state       source             virtual            destination
TCP 00:57  NONE        172.16.186.1:0     172.16.186.109:80  172.16.186.112:80
TCP 00:00  ESTABLISHED 172.16.186.1:44234 172.16.186.109:80  172.16.186.112:80
IPVS connection entries
pro expire state       source             virtual            destination
TCP 00:56  NONE        172.16.186.1:0     172.16.186.109:80  172.16.186.112:80
TCP 00:00  ESTABLISHED 172.16.186.1:44234 172.16.186.109:80  172.16.186.112:80
IPVS connection entries
pro expire state       source             virtual            destination
TCP 00:55  NONE        172.16.186.1:0     172.16.186.109:80  172.16.186.112:80
TCP 00:00  ESTABLISHED 172.16.186.1:44234 172.16.186.109:80  172.16.186.112:80
IPVS connection entries
pro expire state       source             virtual            destination
TCP 00:54  NONE        172.16.186.1:0     172.16.186.109:80  172.16.186.112:80
TCP 00:00  ESTABLISHED 172.16.186.1:44234 172.16.186.109:80  172.16.186.112:80
IPVS connection entries
pro expire state       source             virtual            destination
TCP 00:53  NONE        172.16.186.1:0     172.16.186.109:80  172.16.186.112:80
TCP 00:00  ESTABLISHED 172.16.186.1:44234 172.16.186.109:80  172.16.186.112:80
IPVS connection entries
pro expire state       source             virtual            destination
TCP 00:52  NONE        172.16.186.1:0     172.16.186.109:80  172.16.186.112:80

注：因为设置了TCP会话、接收FIN数据包后的TCP会话、UDP数据包的ues都为1秒,也就是说一个连接中TCP会话、接收FIN数据包后的TCP会话、UDP数据包的ues都处理完毕后才会进行重新调度，即约为4秒一轮的重新调度

keepalived安装

lvs端(keepalived主)配置keepalived
[root@lvs ~]# mkdir /usr/local/keepalived
[root@lvs ~]# yum -y install gcc gcc-c++ openssl-devel libnl libnl-devel
[root@lvs ~]# wget https://www.keepalived.org/software/keepalived-2.2.2.tar.gz
[root@lvs ~]# tar -zxvf keepalived-2.2.2.tar.gz
[root@lvs ~]# cd keepalived-2.2.2
[root@keepalived keepalived-2.2.2]# ./configure --prefix=/usr/local/keepalived --with-init=SYSV
[root@keepalived keepalived-2.2.2]# make && make install

# 以下都使用的绝对路径就不带主机名了
cp /usr/local/keepalived/etc/rc.d/init.d/keepalived  /etc/init.d/
cp /usr/local/keepalived/etc/sysconfig/keepalived  /etc/sysconfig
cp /usr/local/keepalived/sbin/keepalived  /usr/sbin
cp /usr/local/keepalived/etc/keepalived/keepalived.conf  /etc/keepalived.conf
chmod +x /etc/init.d/keepalived 
chkconfig --level 2345 keepalived on

vim /etc/sysconfig/keepalived
KEEPALIVED_OPTIONS="-f /etc/keepalived.conf -D -S 0"

配置keepalived
[root@keepalived keepalived-2.2.2]# cp /etc/keepalived.conf{,.bak}
[root@keepalived keepalived-2.2.2]# vim /etc/keepalived.conf
! Configuration File for keepalived

global_defs {
   router_id LVS_186.110
}

# 一个vrrp_instance就是定义一个虚拟路由器，实例名称
vrrp_instance VI_1 {
    state MASTER                 ===> 是主还是备
    interface ens33              ===> 监听的哪个网卡，通告选举使用哪个接口进行
    virtual_router_id 41         ===> 路由标识，一般不改，也可写成当前主机的主机名或IP
    priority 100                 ===> ID还是虚拟MAC最后一段地址的信息，取值范围0-255
    advert_int 1                 ===> 通告频率(秒)
    authentication {             ===> 通信认证机制，这里可选择是明文认证还有一种是加密认证
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {          ===> 设置虚拟VIP地址
        172.16.186.109
    }
}

#========================= 此处可配置检测脚本，该文档未配置 ==================================
vrrp_script check_nginx {
    script "/etc/keepalived/check_nginx.sh"
}

在 /etc/keepalived/目录中编写好check_nginx.sh脚本后需要给该脚本添加 x 权限
如需使用邮件则global_defs段中的一切不要删除并且完整配置，该文档未配置且未使用邮件功能

脚本示例
cat > /etc/keepalived/check_nginx.sh  << EOF
#!/bin/bash
count=$(ps -ef | grep nginx | egrep -cv "grep | $$")
if [ "$count" -eq 0 ];then
    xxxxx
else
    xxxxx
fi
EOF
#========================================================================================

# 配置集群地址访问的IP+port，端口和nginx保持一致
virtual_server 172.16.186.109 80 {
    delay_loop 6                ===> 健康检查的时间,秒
    lb_algo rr                  ===> 负载均衡的算法,默认是轮询(rr|wrr|lc|wlc|lblc|sh|dh)
    lb_kind DR                  ===> LVS的模式，NAT|TUN|DR
    persistence_timeout 5       ===> 持久连接超时时间,默认是50
    protocol TCP                ===> 使用的协议 -t

    # 负载均衡的后端真实服务器,也就是nginx节点的具体IP地址
    real_server 172.16.186.111 80 {    ===> web1端的
        weight 1                       ==> 轮询的默认权重
        TCP_CHECK {                    ===> 如果是应用服务器不是WEB服务器，就用TCP_CHECK检查MSIC_CHECK|SMTP_CHEKC|TCP_CHECK|SSL_GET|HTTP_GET
          connect_port 80              ===> 检测的80端口
          connect_timeout 2            ===> 超时时间2秒
          nb_get_retry 2               ===> 重试次数2次
          delay_before_retry 3         ===> 间隔时间3秒
        }
    }
    real_server 172.16.186.112 80 {      ===> web2 端的
        weight 1
        TCP_CHECK {
          connect_port 80
          connect_timeout 2
          nb_get_retry 2
          delay_before_retry 3
        }
    }
}    
    

# 清空当前ipvs的规则    
[root@lvs keepalived-2.2.2]# ipvsadm -C
[root@lvs keepalived-2.2.2]# ipvsadm -ln
注：清空后将无规则


# 启动keepalived
[root@lvs keepalived-2.2.2]# systemctl restart keepalived
[root@lvs keepalived-2.2.2]# systemctl status keepalived
● keepalived.service - SYSV: Start and stop Keepalived
   Loaded: loaded (/etc/rc.d/init.d/keepalived; bad; vendor preset: disabled)
   Active: active (running) since 四 2021-07-22 03:51:17 EDT; 6s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 11560 ExecStart=/etc/rc.d/init.d/keepalived start (code=exited, status=0/SUCCESS)
 Main PID: 11567 (keepalived)
   CGroup: /system.slice/keepalived.service
           ├─11567 keepalived -f /etc/keepalived.conf -D -S 0
           ├─11569 keepalived -f /etc/keepalived.conf -D -S 0
           └─11570 keepalived -f /etc/keepalived.conf -D -S 0

7月 22 03:51:20 lvs Keepalived_vrrp[11570]: (VI_1) Receive advertisement timeout
7月 22 03:51:20 lvs Keepalived_vrrp[11570]: (VI_1) Entering MASTER STATE
7月 22 03:51:20 lvs Keepalived_vrrp[11570]: (VI_1) setting VIPs.
7月 22 03:51:20 lvs Keepalived_vrrp[11570]: (VI_1) Sending/queueing gratuitous ARPs on ens33 for 172.16.186.109
7月 22 03:51:20 lvs Keepalived_vrrp[11570]: Sending gratuitous ARP on ens33 for 172.16.186.109
7月 22 03:51:20 lvs Keepalived_vrrp[11570]: Sending gratuitous ARP on ens33 for 172.16.186.109
7月 22 03:51:20 lvs Keepalived_vrrp[11570]: Sending gratuitous ARP on ens33 for 172.16.186.109
7月 22 03:51:20 lvs Keepalived_vrrp[11570]: Sending gratuitous ARP on ens33 for 172.16.186.109
7月 22 03:51:20 lvs Keepalived_vrrp[11570]: Sending gratuitous ARP on ens33 for 172.16.186.109
7月 22 03:51:20 lvs Keepalived_healthcheckers[11569]: TCP connection to [172.16.186.112]:tcp:80 success.


[root@lvs keepalived-2.2.2]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  172.16.186.109:80 rr persistent 5
  -> 172.16.186.111:80            Route   1      0          0         
  -> 172.16.186.112:80            Route   1      0          0 

以上是主keepalived配置完毕,接下来配置从的keepalived

从keepalived配置

从主keepalived上发送包和配置文件到从的上面
[root@lvs ~]# scp -r keepalived-2.2.2 root@172.16.186.114:~
[root@lvs ~]# scp /etc/keepalived.conf root@172.16.186.114:/etc


回到从的keepalived节点上开始配置LVS 和 keepalived
安装lvs (只需安装不用配置规则)
[root@keepalived ~]# mount /dev/sr0 /mnt/usb1        ===> 挂载光盘
[root@keepalived ~]# yum -y install /mnt/usb1/Packages/ipvsadm-1.27-8.el7.x86_64.rpm       ===> 如有互联网可不挂载直接yum安装ipvsadm

配置keepalived
[root@keepalived ~]# mkdir /usr/local/keepalived
[root@keepalived ~]# yum -y install gcc gcc-c++ openssl-devel libnl libnl-devel
[root@keepalived ~]# cd keepalived-2.2.2
[root@keepalived keepalived-2.2.2]# ./configure --prefix=/usr/local/keepalived --with-init=SYSV
[root@keepalived keepalived-2.2.2]# make && make install

以下都使用的绝对路径，主机名和当前路径加到一块太长，这里就不带了
cp /usr/local/keepalived/etc/rc.d/init.d/keepalived  /etc/init.d/
cp /usr/local/keepalived/etc/sysconfig/keepalived  /etc/sysconfig
cp /usr/local/keepalived/sbin/keepalived  /usr/sbin
cp /usr/local/keepalived/etc/keepalived/keepalived.conf  /etc/keepalived.conf

chmod +x /etc/init.d/keepalived 
chkconfig --level 2345 keepalived on


修改keepalived启动选项
vim /etc/sysconfig/keepalived
KEEPALIVED_OPTIONS="-f /etc/keepalived.conf -D -S 0"


keepalived配置文件主的和从的不是一模一样，所以这里要改
vim /etc/keepalived.conf
! Configuration File for keepalived

global_defs {
   router_id LVS_186.114      ===> 需要修改，可自定义
}

vrrp_instance VI_1 {
    state BACKUP              ===> 改成备机
    interface ens33
    virtual_router_id 41
    priority 90               ===> 权重值,应比主的权重值小
....
   ....
其他所有都不用改也不用动


# 清空当前ipvs的规则    
[root@lvs keepalived-2.2.2]# ipvsadm -C
[root@lvs keepalived-2.2.2]# ipvsadm -ln
注：清空后将无规则


# 启动keepalived
[root@lvs keepalived-2.2.2]# systemctl restart keepalived
[root@lvs keepalived-2.2.2]# systemctl status keepalived
● keepalived.service - SYSV: Start and stop Keepalived
   Loaded: loaded (/etc/rc.d/init.d/keepalived; bad; vendor preset: disabled)
   Active: active (running) since 四 2021-07-22 16:36:45 CST; 4s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 16580 ExecStart=/etc/rc.d/init.d/keepalived start (code=exited, status=0/SUCCESS)
 Main PID: 16587 (keepalived)
   CGroup: /system.slice/keepalived.service
           ├─16587 keepalived -f /etc/keepalived.conf -D -S 0
           ├─16589 keepalived -f /etc/keepalived.conf -D -S 0
           └─16590 keepalived -f /etc/keepalived.conf -D -S 0

7月 22 16:36:45 keepalived Keepalived_vrrp[16590]: Registering Kernel netlink reflector
7月 22 16:36:45 keepalived Keepalived_vrrp[16590]: Registering Kernel netlink command channel
7月 22 16:36:45 keepalived Keepalived_vrrp[16590]: Assigned address 172.16.186.114 for interface ens33
7月 22 16:36:45 keepalived Keepalived_vrrp[16590]: Assigned address fe80::b77e:df59:787b:674b for interface ens33
7月 22 16:36:45 keepalived Keepalived_vrrp[16590]: Registering gratuitous ARP shared channel
7月 22 16:36:45 keepalived Keepalived_vrrp[16590]: (VI_1) removing VIPs.
7月 22 16:36:45 keepalived Keepalived_vrrp[16590]: (VI_1) Entering BACKUP STATE (init)
7月 22 16:36:45 keepalived Keepalived_vrrp[16590]: VRRP sockpool: [ifindex(  2), family(IPv4), proto(112), fd(13,14)]
7月 22 16:36:45 keepalived Keepalived[16587]: Startup complete
7月 22 16:36:48 keepalived Keepalived_healthcheckers[16589]: TCP connection to [172.16.186.111]:tcp:80 success.


查看ipvs
[root@keepalived ~]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  172.16.186.109:80 rr persistent 5
  -> 172.16.186.111:80            Route   1      0          0         
  -> 172.16.186.112:80            Route   1      0          0 

测试keepalived主备

在未关闭主keepalive端的时先看下状态
[root@lvs ~]# systemctl status keepalived
● keepalived.service - SYSV: Start and stop Keepalived
   Loaded: loaded (/etc/rc.d/init.d/keepalived; bad; vendor preset: disabled)
   Active: active (running) since 四 2021-07-22 04:55:11 EDT; 1h 1min ago
     Docs: man:systemd-sysv-generator(8)
  Process: 11617 ExecStop=/etc/rc.d/init.d/keepalived stop (code=exited, status=0/SUCCESS)
  Process: 11676 ExecStart=/etc/rc.d/init.d/keepalived start (code=exited, status=0/SUCCESS)
 Main PID: 11683 (keepalived)
   CGroup: /system.slice/keepalived.service
           ├─11683 keepalived -f /etc/keepalived.conf -D -S 0
           ├─11685 keepalived -f /etc/keepalived.conf -D -S 0
           └─11686 keepalived -f /etc/keepalived.conf -D -S 0

7月 22 05:32:17 lvs Keepalived_vrrp[11686]: Sending gratuitous ARP on ens33 for 172.16.186.109
7月 22 05:32:17 lvs Keepalived_vrrp[11686]: Sending gratuitous ARP on ens33 for 172.16.186.109
7月 22 05:32:17 lvs Keepalived_vrrp[11686]: Sending gratuitous ARP on ens33 for 172.16.186.109
7月 22 05:32:17 lvs Keepalived_vrrp[11686]: Sending gratuitous ARP on ens33 for 172.16.186.109
7月 22 05:32:22 lvs Keepalived_vrrp[11686]: (VI_1) Sending/queueing gratuitous ARPs on ens33 for 172.16.186.109
7月 22 05:32:22 lvs Keepalived_vrrp[11686]: Sending gratuitous ARP on ens33 for 172.16.186.109
7月 22 05:32:22 lvs Keepalived_vrrp[11686]: Sending gratuitous ARP on ens33 for 172.16.186.109
7月 22 05:32:22 lvs Keepalived_vrrp[11686]: Sending gratuitous ARP on ens33 for 172.16.186.109
7月 22 05:32:22 lvs Keepalived_vrrp[11686]: Sending gratuitous ARP on ens33 for 172.16.186.109
7月 22 05:32:22 lvs Keepalived_vrrp[11686]: Sending gratuitous ARP on ens33 for 172.16.186.109


将lvs端的keepalived服务关闭
[root@lvs ~]# systemctl stop keepalived

再来看Keepalived的状态，确定该服务已停止
[root@lvs ~]# systemctl status keepalived
● keepalived.service - SYSV: Start and stop Keepalived
   Loaded: loaded (/etc/rc.d/init.d/keepalived; bad; vendor preset: disabled)
   Active: inactive (dead) since 四 2021-07-22 04:41:10 EDT; 2min 47s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 11617 ExecStop=/etc/rc.d/init.d/keepalived stop (code=exited, status=0/SUCCESS)
  Process: 11560 ExecStart=/etc/rc.d/init.d/keepalived start (code=exited, status=0/SUCCESS)
 Main PID: 11567 (code=exited, status=0/SUCCESS)

7月 22 04:07:42 lvs Keepalived_vrrp[11570]: Sending gratuitous ARP on ens33 for 172.16.186.109
7月 22 04:07:42 lvs Keepalived_vrrp[11570]: Sending gratuitous ARP on ens33 for 172.16.186.109
7月 22 04:07:42 lvs Keepalived_vrrp[11570]: Sending gratuitous ARP on ens33 for 172.16.186.109
7月 22 04:07:42 lvs Keepalived_vrrp[11570]: Sending gratuitous ARP on ens33 for 172.16.186.109
7月 22 04:41:08 lvs systemd[1]: Stopping SYSV: Start and stop Keepalived...
7月 22 04:41:08 lvs Keepalived[11567]: Stopping
7月 22 04:41:08 lvs Keepalived_vrrp[11570]: (VI_1) sent 0 priority
7月 22 04:41:08 lvs Keepalived_vrrp[11570]: (VI_1) removing VIPs.
7月 22 04:41:10 lvs keepalived[11617]: Stopping keepalived: [  OK  ]
7月 22 04:41:10 lvs systemd[1]: Stopped SYSV: Start and stop Keepalived.            ===> 已停止


# 查看ipvs规则也为空
[root@lvs ~]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn


从keepalived节点上查看
[root@keepalived ~]# systemctl status keepalived
● keepalived.service - SYSV: Start and stop Keepalived
   Loaded: loaded (/etc/rc.d/init.d/keepalived; bad; vendor preset: disabled)
   Active: active (running) since 四 2021-07-22 16:36:45 CST; 11min ago
     Docs: man:systemd-sysv-generator(8)
  Process: 16580 ExecStart=/etc/rc.d/init.d/keepalived start (code=exited, status=0/SUCCESS)
 Main PID: 16587 (keepalived)
   CGroup: /system.slice/keepalived.service
           ├─16587 keepalived -f /etc/keepalived.conf -D -S 0
           ├─16589 keepalived -f /etc/keepalived.conf -D -S 0
           └─16590 keepalived -f /etc/keepalived.conf -D -S 0

7月 22 16:41:09 keepalived Keepalived_vrrp[16590]: Sending gratuitous ARP on ens33 for 172.16.186.109
7月 22 16:41:09 keepalived Keepalived_vrrp[16590]: Sending gratuitous ARP on ens33 for 172.16.186.109
7月 22 16:41:09 keepalived Keepalived_vrrp[16590]: Sending gratuitous ARP on ens33 for 172.16.186.109
7月 22 16:41:09 keepalived Keepalived_vrrp[16590]: Sending gratuitous ARP on ens33 for 172.16.186.109
7月 22 16:41:14 keepalived Keepalived_vrrp[16590]: (VI_1) Sending/queueing gratuitous ARPs on ens33 for 172.16.186.109
7月 22 16:41:14 keepalived Keepalived_vrrp[16590]: Sending gratuitous ARP on ens33 for 172.16.186.109
7月 22 16:41:14 keepalived Keepalived_vrrp[16590]: Sending gratuitous ARP on ens33 for 172.16.186.109
7月 22 16:41:14 keepalived Keepalived_vrrp[16590]: Sending gratuitous ARP on ens33 for 172.16.186.109
7月 22 16:41:14 keepalived Keepalived_vrrp[16590]: Sending gratuitous ARP on ens33 for 172.16.186.109
7月 22 16:41:14 keepalived Keepalived_vrrp[16590]: Sending gratuitous ARP on ens33 for 172.16.186.109        ===> 已正常接管所有请求


修改持久的时间(该操作仅限于看效果，生产中不用设置)
[root@lvs ~]# ipvsadm -E -t 172.16.186.109:80 -s rr -p 3      ===>设置持久时间为3秒


设置tcp/udp的超时时间
[root@lvs ~]# ipvsadm --set 1 1 1 

浏览器访问：http://172.16.186.109    #每隔4秒刷新一下就能看到效果


再将lvs节点上的主keepalived恢复
[root@lvs ~]# systemctl start keepalived 
[root@lvs ~]# systemctl status keepalived                        ====> 会重新接管所有请求
● keepalived.service - SYSV: Start and stop Keepalived
   Loaded: loaded (/etc/rc.d/init.d/keepalived; bad; vendor preset: disabled)
   Active: active (running) since 四 2021-07-22 06:02:05 EDT; 2s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 11744 ExecStop=/etc/rc.d/init.d/keepalived stop (code=exited, status=0/SUCCESS)
  Process: 11764 ExecStart=/etc/rc.d/init.d/keepalived start (code=exited, status=0/SUCCESS)
 Main PID: 11771 (keepalived)
   CGroup: /system.slice/keepalived.service
           ├─11771 keepalived -f /etc/keepalived.conf -D -S 0
           ├─11773 keepalived -f /etc/keepalived.conf -D -S 0
           └─11774 keepalived -f /etc/keepalived.conf -D -S 0

7月 22 06:02:05 lvs Keepalived_vrrp[11774]: Assigned address 172.16.186.110 for interface ens33
7月 22 06:02:05 lvs Keepalived_vrrp[11774]: Assigned address fe80::20c:29ff:fef1:e2f9 for interface ens33
7月 22 06:02:05 lvs Keepalived_vrrp[11774]: Registering gratuitous ARP shared channel
7月 22 06:02:05 lvs Keepalived_vrrp[11774]: (VI_1) removing VIPs.
7月 22 06:02:05 lvs Keepalived_vrrp[11774]: (VI_1) Entering BACKUP STATE (init)
7月 22 06:02:05 lvs Keepalived_vrrp[11774]: VRRP sockpool: [ifindex(  2), family(IPv4), proto(112), fd(13,14)]
7月 22 06:02:05 lvs Keepalived[11771]: Startup complete
7月 22 06:02:06 lvs Keepalived_vrrp[11774]: (VI_1) received lower priority (90) advert from 172.16.186.114 - discarding
7月 22 06:02:07 lvs Keepalived_vrrp[11774]: (VI_1) received lower priority (90) advert from 172.16.186.114 - discarding
7月 22 06:02:08 lvs Keepalived_healthcheckers[11773]: TCP connection to [172.16.186.111]:tcp:80 success.


查看从keepalived状态
[root@keepalived ~]# systemctl status keepalived            ====> 从keepalived会将ipvs移除
● keepalived.service - SYSV: Start and stop Keepalived
   Loaded: loaded (/etc/rc.d/init.d/keepalived; bad; vendor preset: disabled)
   Active: active (running) since 四 2021-07-22 16:36:45 CST; 18min ago
     Docs: man:systemd-sysv-generator(8)
  Process: 16580 ExecStart=/etc/rc.d/init.d/keepalived start (code=exited, status=0/SUCCESS)
 Main PID: 16587 (keepalived)
   CGroup: /system.slice/keepalived.service
           ├─16587 keepalived -f /etc/keepalived.conf -D -S 0
           ├─16589 keepalived -f /etc/keepalived.conf -D -S 0
           └─16590 keepalived -f /etc/keepalived.conf -D -S 0

7月 22 16:41:09 keepalived Keepalived_vrrp[16590]: Sending gratuitous ARP on ens33 for 172.16.186.109
7月 22 16:41:14 keepalived Keepalived_vrrp[16590]: (VI_1) Sending/queueing gratuitous ARPs on ens33 for 172.16.186.109
7月 22 16:41:14 keepalived Keepalived_vrrp[16590]: Sending gratuitous ARP on ens33 for 172.16.186.109
7月 22 16:41:14 keepalived Keepalived_vrrp[16590]: Sending gratuitous ARP on ens33 for 172.16.186.109
7月 22 16:41:14 keepalived Keepalived_vrrp[16590]: Sending gratuitous ARP on ens33 for 172.16.186.109
7月 22 16:41:14 keepalived Keepalived_vrrp[16590]: Sending gratuitous ARP on ens33 for 172.16.186.109
7月 22 16:41:14 keepalived Keepalived_vrrp[16590]: Sending gratuitous ARP on ens33 for 172.16.186.109
7月 22 16:55:14 keepalived Keepalived_vrrp[16590]: (VI_1) Master received advert from 172.16.186.110 with higher priority 100, ours 90
7月 22 16:55:14 keepalived Keepalived_vrrp[16590]: (VI_1) Entering BACKUP STATE
7月 22 16:55:14 keepalived Keepalived_vrrp[16590]: (VI_1) removing VIPs.


当后端的httpd/nginx异常后ipvsadm会自动将该节点移除，当恢复好后ipvsadm会自动再将该节点添加到ipvs集群中

Linux学习QQ群