一、服务器信息
lvs服务器：10.30.20.90
nginx服务器：10.30.8.64,10.30.12.71
vip：10.30.20.253
二、部署
1、nginx服务器部署
# systemctl start docker
# docker pull nginx
# mkdir /data/html
# IP=$(ifconfig eth0|awk -F'[ ]+' '/inet/ {print $3}')
# echo "${IP}" >/data/html/index.html
# docker run -d --network=host -v /data/html:/usr/share/nginx/html nginx
# curl "http://127.0.0.1"
10.30.8.64
2、部署lvs nat模式
# yum install -y ipvsadm ipset conntrack # ip address add 10.30.20.253/32 dev eth0 # ipvsadm -A -t 10.30.20.253:8181 -s rr # ipvsadm -a -t 10.30.20.253:8181 -r 10.30.8.64:80 -m # ipvsadm -a -t 10.30.20.253:8181 -r 10.30.12.71:80 -m # ipvsadm -L -n IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 10.30.20.253:8181 rr -> 10.30.8.64:80 Masq 1 0 0 -> 10.30.12.71:80 Masq 1 0 0 # sysctl net.ipv4.ip_forward=1
3、iptables配置
# iptables -t nat -N KUBE-SERVICES # iptables -t nat -N KUBE-MARK-MASQ # iptables -t nat -N KUBE-POSTROUTING # iptables -t nat -N KUBE-NODE-PORT # ipset create KUBE-NODE-PORT-TCP bitmap:port range 0-65535 # ipset add KUBE-NODE-PORT-TCP 8181 # iptables -t nat -A PREROUTING -j KUBE-SERVICES # iptables -t nat -A KUBE-SERVICES -m addrtype --dst-type LOCAL -j KUBE-NODE-PORT # iptables -t nat -A KUBE-NODE-PORT -p tcp -m set --match-set KUBE-NODE-PORT-TCP dst -j KUBE-MARK-MASQ # iptables -t nat -A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000 # iptables -t nat -A POSTROUTING -j KUBE-POSTROUTING # iptables -t nat -A KUBE-POSTROUTING -m mark --mark 0x4000/0x4000 -j MASQUERADE # sysctl net.ipv4.vs.conntrack=1
4、测试
# curl "http://10.30.20.253:8181" 10.30.12.71 # curl "http://10.30.20.253:8181" 10.30.8.64
三、抓包
1、在nginx服务器
15:07:10.282569 IP 10.30.20.90.55888 > 10.30.8.64.80: Flags [S], seq 3207943442, win 65535, options [mss 1388,nop,wscale 5,nop,nop,TS val 226579042 ecr 0,sackOK,eol], length 0 15:07:10.282650 IP 10.30.8.64.80 > 10.30.20.90.55888: Flags [S.], seq 1357773976, ack 3207943443, win 28960, options [mss 1460,sackOK,TS val 2630379 ecr 226579042,nop,wscale 7], length 0 15:07:10.289414 IP 10.30.20.90.55888 > 10.30.8.64.80: Flags [.], ack 1, win 4128, options [nop,nop,TS val 226579051 ecr 2630379], length 0 15:07:10.290442 IP 10.30.20.90.55888 > 10.30.8.64.80: Flags [P.], seq 1:82, ack 1, win 4128, options [nop,nop,TS val 226579051 ecr 2630379], length 81: HTTP: GET / HTTP/1.1 15:07:10.290470 IP 10.30.8.64.80 > 10.30.20.90.55888: Flags [.], ack 82, win 227, options [nop,nop,TS val 2630386 ecr 226579051], length 0 15:07:10.290849 IP 10.30.8.64.80 > 10.30.20.90.55888: Flags [P.], seq 1:236, ack 82, win 227, options [nop,nop,TS val 2630387 ecr 226579051], length 235: HTTP: HTTP/1.1 200 OK 15:07:10.290974 IP 10.30.8.64.80 > 10.30.20.90.55888: Flags [P.], seq 236:247, ack 82, win 227, options [nop,nop,TS val 2630387 ecr 226579051], length 11: HTTP 15:07:10.298179 IP 10.30.20.90.55888 > 10.30.8.64.80: Flags [.], ack 236, win 4120, options [nop,nop,TS val 226579058 ecr 2630387], length 0 15:07:10.298297 IP 10.30.20.90.55888 > 10.30.8.64.80: Flags [.], ack 247, win 4120, options [nop,nop,TS val 226579058 ecr 2630387], length 0 15:07:10.298306 IP 10.30.20.90.55888 > 10.30.8.64.80: Flags [F.], seq 82, ack 247, win 4120, options [nop,nop,TS val 226579059 ecr 2630387], length 0 15:07:10.298443 IP 10.30.8.64.80 > 10.30.20.90.55888: Flags [F.], seq 247, ack 83, win 227, options [nop,nop,TS val 2630394 ecr 226579059], length 0 15:07:10.305622 IP 10.30.20.90.55888 > 10.30.8.64.80: Flags [.], ack 248, win 4120, options [nop,nop,TS val 226579065 ecr 2630394], length 0
2、在lvs服务器
# conntrack -L|grep 55888 conntrack v1.4.4 (conntrack-tools): 12 flow entries have been shown. tcp 6 111 TIME_WAIT src=10.127.12.138 dst=10.30.20.253 sport=55888 dport=8181 src=10.30.8.64 dst=10.30.20.90 sport=80 dport=55888 [ASSURED] mark=0 use=1
SNAT：10.127.12.138:55888 --> 10.30.20.90.55888
DNAT：10.30.20.253:8181 --> 10.30.8.64.80
参考文档：