负载均衡LVS(DR模式)安装实战

1.编译安装ipvsadm

首先从LVS官网下载tarball，解压后make && make install即可。

要注意的是LVS的依赖有：popt-static、libnl、kernel。如果yum源里有就直接安装，如果没有就自行下载，但一定要对应上已有软件如popt或Linux内核的版本。

$ tar -xzvf ipvsadm-1.26.tar.gz
$ cd ipvsadm-1.26
$ make
make -C libipvs
make[1]: Entering directory `/root/lvs/ipvsadm-1.26/libipvs'
gcc -Wall -Wunused -Wstrict-prototypes -g -fPIC -DLIBIPVS_USE_NL  -DHAVE_NET_IP_VS_H -c -o libipvs.o libipvs.c
gcc -Wall -Wunused -Wstrict-prototypes -g -fPIC -DLIBIPVS_USE_NL  -DHAVE_NET_IP_VS_H -c -o ip_vs_nl_policy.o ip_vs_nl_policy.c
ar rv libipvs.a libipvs.o ip_vs_nl_policy.o
ar: creating libipvs.a
a - libipvs.o
a - ip_vs_nl_policy.o
gcc -shared -Wl,-soname,libipvs.so -o libipvs.so libipvs.o ip_vs_nl_policy.o
make[1]: Leaving directory `/root/lvs/ipvsadm-1.26/libipvs'
gcc -Wall -Wunused -Wstrict-prototypes -g  -DVERSION="1.26" -DSCHEDULERS=""rr|wrr|lc|wlc|lblc|lblcr|dh|sh|sed|nq"" -DPE_LIST=""sip"" -DHAVE_POPT -DHAVE_NET_IP_VS_H -c -o ipvsadm.o ipvsadm.c
ipvsadm.c: In function âprint_largenumâ:
ipvsadm.c:1383: warning: field width should have type âintâ, but argument 2 has type âsize_tâ
gcc -Wall -Wunused -Wstrict-prototypes -g  -DVERSION="1.26" -DSCHEDULERS=""rr|wrr|lc|wlc|lblc|lblcr|dh|sh|sed|nq"" -DPE_LIST=""sip"" -DHAVE_POPT -DHAVE_NET_IP_VS_H -c -o config_stream.o config_stream.c
gcc -Wall -Wunused -Wstrict-prototypes -g  -DVERSION="1.26" -DSCHEDULERS=""rr|wrr|lc|wlc|lblc|lblcr|dh|sh|sed|nq"" -DPE_LIST=""sip"" -DHAVE_POPT -DHAVE_NET_IP_VS_H -c -o dynamic_array.o dynamic_array.c
gcc -Wall -Wunused -Wstrict-prototypes -g -o ipvsadm ipvsadm.o config_stream.o dynamic_array.o libipvs/libipvs.a -lpopt -lnl

最后make install，安装ipvsadm、ipvsadm-save等命令，并添加ipvsadm到系统服务，方便了后续使用。

$ make install
make -C libipvs
make[1]: Entering directory `/root/lvs/ipvsadm-1.26/libipvs'
make[1]: Nothing to be done for `all'.
make[1]: Leaving directory `/root/lvs/ipvsadm-1.26/libipvs'
if [ ! -d /sbin ]; then mkdir -p /sbin; fi
install -m 0755 ipvsadm /sbin
install -m 0755 ipvsadm-save /sbin
install -m 0755 ipvsadm-restore /sbin
[ -d /usr/man/man8 ] || mkdir -p /usr/man/man8
install -m 0644 ipvsadm.8 /usr/man/man8
install -m 0644 ipvsadm-save.8 /usr/man/man8
install -m 0644 ipvsadm-restore.8 /usr/man/man8
[ -d /etc/rc.d/init.d ] || mkdir -p /etc/rc.d/init.d
install -m 0755 ipvsadm.sh /etc/rc.d/init.d/ipvsadm

最后，关闭防火墙和selinux：service iptables stop && setenforce 0

2.DR模式配置

2.1 Ipvsadm参数解释

$ ipvsadm -C
$ ipvsadm -A -t 192.168.100.20:80 -s rr 
$ ipvsadm -a -t 192.168.100.20:80 -r 192.168.100.116:80 -g -p60 
$ ipvsadm -a -t 192.168.100.20:80 -r 192.168.100.117:80 -g -p60

ipvsadm参数含义：

• -C：清除已有规则。
• -A：添加VIP服务，后跟服务的访问地址。
• -t：TCP协议，还是UDP协议(-u)。
• -s：负载均衡算法，rr表示RoundRobin。
• -a：添加RealServer到VIP，后跟虚地址。
• -r：添加RealServer到VIP，后跟实地址。
• -g：透传模式(-g表示Direct Routing即DR模式，-i表示ipip封装即Tunneling模式，-m表示Network Access Translation即NAT模式)
• -p：Session粘连，同一客户端的请求在一段时间内都负载到同一RealServer。

2.2 Director配置

使用DR模式必须在主机上开启IP转发。

$ vi /etc/sysctl.conf
    ...
# Controls IP packet forwarding
net.ipv4.ip_forward = 1
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0
net.ipv4.conf.em1.send_redirects = 0
    ...
$ sysctl -p
net.ipv4.ip_forward = 1
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0
net.ipv4.conf.em1.send_redirects = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
kernel.sysrq = 0
kernel.core_uses_pid = 1
net.ipv4.tcp_syncookies = 1
error: "net.bridge.bridge-nf-call-ip6tables" is an unknown key
error: "net.bridge.bridge-nf-call-iptables" is an unknown key
error: "net.bridge.bridge-nf-call-arptables" is an unknown key
kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.shmmax = 68719476736
kernel.shmall = 4294967296
vm.swappiness = 0

网络配置：

$ ifconfig em1:0 192.168.242.101 broadcast 192.168.242.255 netmask 255.255.255.0 up
[root@node12 lvs]# ifconfig
em1       Link encap:Ethernet  HWaddr B0:83:FE:DB:48:DF  
          inet addr:192.168.242.102  Bcast:192.168.242.255  Mask:255.255.255.0
          inet6 addr: fe80::b283:feff:fedb:48df/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          ...

em1:0     Link encap:Ethernet  HWaddr B0:83:FE:DB:48:DF  
          inet addr:192.168.242.101  Bcast:192.168.242.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          Interrupt:40 Memory:95000000-957fffff

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          ...

$ route add -host 192.168.242.101 dev em1:0
$ route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
192.168.242.101 *               255.255.255.255 UH    0      0        0 em1
192.168.242.0   *               255.255.255.0   U     0      0        0 em1
192.168.180.0   *               255.255.255.0   U     0      0        0 em3
link-local      *               255.255.0.0     U     1002   0        0 em1
link-local      *               255.255.0.0     U     1004   0        0 em3
default         192.168.180.253 0.0.0.0         UG    0      0        0 em3

接下来用ipvsadm配置LVS：

$ ipvsadm -A -t 192.168.242.101:8088 -s rr
$ ipvsadm -a -t 192.168.242.101:8088 -r 192.168.242.105:8088 -g
$ ipvsadm -a -t 192.168.242.101:8088 -r 192.168.242.106:8088 -g
$ ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.242.101:8088 rr
  -> 192.168.242.105:8088         Route   1      0          0         
  -> 192.168.242.106:8088         Route   1      0          0         

$ ipvsadm-save > /etc/sysconfig/ipvsadm
$ cat /etc/sysconfig/ipvsadm
-A -t 192.168.242.101:radan-http -s rr
-a -t 192.168.242.101:radan-http -r 192.168.242.105:radan-http -g -w 1
-a -t 192.168.242.101:radan-http -r 192.168.242.106:radan-http -g -w 1

$ service ipvsadm start
Clearing the current IPVS table:                           [  OK  ]
Applying IPVS configuration:                               [  OK  ]

2.4 RealServer配置

首先配置内核屏蔽掉Director发来的ARP消息：

$ vi /etc/sysctl.conf 
    ...
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.em1.arp_ignore = 1
net.ipv4.conf.em1.arp_announce = 2
    ...
$ sysctl -p
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.em1.arp_ignore = 1
net.ipv4.conf.em1.arp_announce = 2
net.ipv4.ip_forward = 1
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
kernel.sysrq = 0
kernel.core_uses_pid = 1
net.ipv4.tcp_syncookies = 1
error: "net.bridge.bridge-nf-call-ip6tables" is an unknown key
error: "net.bridge.bridge-nf-call-iptables" is an unknown key
error: "net.bridge.bridge-nf-call-arptables" is an unknown key
kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.shmmax = 68719476736
kernel.shmall = 4294967296
vm.swappiness = 0

将VIP添加到loopback接口，并且netmask必须是255.255.255.255。

$ ifconfig lo:0 192.168.242.101 broadcast 192.168.242.255 netmask 255.255.255.255 up
$ ifconfig
em1       Link encap:Ethernet  HWaddr XX:XX:XX:XX:XX:XX  
          inet addr:192.168.242.105  Bcast:192.168.242.255  Mask:255.255.255.0
          inet6 addr: xxxx::xxxx:xxxx:xxxx:xxxx/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          ...

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          ...

lo:0      Link encap:Local Loopback  
          inet addr:192.168.242.102  Mask:255.255.255.255
          UP LOOPBACK RUNNING  MTU:16436  Metric:1

$ route add -host 192.168.242.101 dev lo:0
$ route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
192.168.242.101 *               255.255.255.255 UH    0      0        0 lo
192.168.242.0   *               255.255.255.0   U     0      0        0 em1
192.168.180.0   *               255.255.255.0   U     0      0        0 em3
link-local      *               255.255.0.0     U     1002   0        0 em1
link-local      *               255.255.0.0     U     1004   0        0 em3
default         192.168.180.253 0.0.0.0         UG    0      0        0 em3

因为RealServer上不需要安装启动ipvsadm服务，所以我们要手动加载ip_vs到内核(Director上是ipvsadm自动帮我们加载的)：

$ modprobe ip_vs
$ lsmod | grep ip
ip_vs                 125220  0 
ip6t_REJECT             4628  2 
nf_conntrack_ipv6       8748  2 
nf_defrag_ipv6         11182  1 nf_conntrack_ipv6
nf_conntrack           79758  2 nf_conntrack_ipv6,xt_state
ip6table_filter         2889  1 
ip6_tables             18732  1 ip6table_filter
ipv6                  317340  399 ip_vs,ip6t_REJECT,nf_conntrack_ipv6,nf_defrag_ipv6
libcrc32c               1246  2 ip_vs,bnx2x

3.负载均衡测试

在两台RealServer上通过tcpdump tcp -A -ni em1 port 8088监听，不断访问VIP就会发现请求轮询转发到两台RealServer上，实现了负载均衡。注意监听的是em1而不是lo:1。

测试时犯了一个错误，我一直在VIP本机上测试，发现怎么都不通，以为LVS没配好于是就一直配啊配，结果从其他机器访问了一下VIP发现其实早就配好了…

经过这次配置学习，以及上面监听em1而非lo:1和VIP本机访问的问题，都证明了我对LVS和网络基础的不够熟悉，还要强化学习一下！