LVS-dr

一、LVS-dr的基本原理

   direct routing，它通过修改请求报文的目标MAC地址进行转发。

   请求报文经由director发送至RS，那么就不能让除Director外的RS响应，因此，有三种解决方案：1.在上游路由器进行IP和MAC的绑定，2.在RS上做arptables，3.修改RS主机内核参数，也就是说vip别名在环回口，修改的内核参数能使vip对发来的请求不做处理。

   响应报文不经由director，而是有vip直接响应cip，其中vip通过rip的MAC和gateway直接响应cip。

   dr的基本规则：     

（1）保证前端路由器将目标IP为VIP的请求报文发送给director

　　解决方案：

 　　 静态绑定

  　　arptables

 　 　修改RS主机内核的参数

（2）RS的RIP可以使用私有地址；但也可以使用公网地址

（3）RS跟Director必须在同一物理网络中

（4）请求报文经由Director调度，但响应报文一定不能经由Director；

（5）不支持端口映射

（6）RS可以大多数OS

（7）RS的网关不能指向DIP

二、实验环境，局域网环境

Client：本机windows7
Director：CentOS 7.1
RealServer：node1，node2均为CentOS6.7
Director与RealServer的所有IP均为同一网段

拓扑如下：

三、配置

director：
[root@localhost ~]# ifconfig ens33:0 192.168.1.15/32 broadcast 192.168.1.15 up
[root@localhost ~]# route add -host 192.168.1.15 dev ens33:0
 
RS：
 node1：
    [root@jymlinux ~]# echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore 
    [root@jymlinux ~]# echo 1 > /proc/sys/net/ipv4/conf/eth2/arp_ignore 
    [root@jymlinux ~]# echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce 
    [root@jymlinux ~]# echo 2 > /proc/sys/net/ipv4/conf/eth2/arp_announce 
    [root@jymlinux ~]# ifconfig lo:0 192.168.1.15/32 broadcast 192.168.1.15 up
    [root@jymlinux ~]# route add -host 192.168.1.15 dev lo:0
 node2：
    [root@jymlinux ~]# echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
    [root@jymlinux ~]# echo 1 > /proc/sys/net/ipv4/conf/eth0/arp_ignore
    [root@jymlinux ~]# echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce 
    [root@jymlinux ~]# echo 2 > /proc/sys/net/ipv4/conf/eth0/arp_announce 
    [root@jymlinux ~]# ifconfig lo:0 192.168.1.15/32 broadcast 192.168.1.15 up
    [root@jymlinux ~]# route add -host 192.168.1.15 dev lo:0
     
或写脚本并给予执行权限设置内核参数
[root@jymlinux ~]# vim lvsdrka.sh
#!/bin/bash
#
case $1 in
start)
        echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
        echo 1 > /proc/sys/net/ipv4/conf/eth2/arp_ignore
        echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
        echo 2 > /proc/sys/net/ipv4/conf/eth2/arp_announce
        ;;
stop)
        echo 0 > /proc/sys/net/ipv4/conf/all/arp_ignore
        echo 0 > /proc/sys/net/ipv4/conf/eth2/arp_ignore
        echo 0 > /proc/sys/net/ipv4/conf/all/arp_announce
        echo 0 > /proc/sys/net/ipv4/conf/eth2/arp_announce
        ;;
esac
 
[root@jymlinux ~]# chmod +x lvsdrka.sh 
[root@jymlinux ~]# ./lvsdrka.sh start
[root@jymlinux ~]# cat /proc/sys/net/ipv4/conf/all/arp_ignore  
1
[root@jymlinux ~]# cat /proc/sys/net/ipv4/conf/all/arp_announce 
2

两个内核参数：

    arp_announce：arp通告

　　0：通告全部IP      默认

　　1：尽量避免将非本网络的地址通告给网络中的其他地址

　　2：总是用最佳本地地址通告网络

　arp_ignore：arp响应

　　0：无论从哪个接口请求的，只要主机有这个地址，就会响应    默认

　　1：从哪个接口请求的地址，就从哪个接口的地址响应

　　2,3,4,,5,6,7,8,有9个级别，其余不常用

在lvs-dr中，我们一般使用arp_ignore=1 arp_announce=2

测试web服务

[root@localhost ~]# curl http://192.168.1.20
<h1>this is node1 <h1>
[root@localhost ~]# curl http://192.168.1.21
<h1>this is node2 <h1>

配置集群规则

[root@localhost ~]# ipvsadm -A -t 192.168.1.15:80 -s rr
[root@localhost ~]# ipvsadm -a -t 192.168.1.15:80 -r 192.168.1.20 -g
[root@localhost ~]# ipvsadm -a -t 192.168.1.15:80 -r 192.168.1.21 -g

四、配置http与https双集群服务

此处网段改为192.168.3.0网段。

1、创建私有CA（以Director主机为例）
[root@localhost ~]# cd /etc/pki/CA
[root@localhost CA]# (umask 077; openssl genrsa -out private/cakey.pem 2048)
Generating RSA private key, 2048 bit long modulus
..................+++
.......+++
e is 65537 (0x10001)
[root@localhost CA]# touch index.txt
[root@localhost CA]# echo 01 > serial
[root@localhost CA]# openssl req -new -x509 -key private/cakey.pem -out cacert.pem -days 365
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [XX]:CN
State or Province Name (full name) []:XJ
Locality Name (eg, city) [Default City]:XJ
Organization Name (eg, company) [Default Company Ltd]:JJ
Organizational Unit Name (eg, section) []:Ops
Common Name (eg, your name or your server's hostname) []:CA
Email Address []:ca.admin.com
 
2、RS主机申请证书
[root@jymlinux ~]# cd /etc/httpd/
[root@jymlinux httpd]# mkdir ssl
[root@jymlinux httpd]# cd ssl
[root@jymlinux ssl]# (umask 077; openssl genrsa -out httpd.key 2048)
Generating RSA private key, 2048 bit long modulus
..............+++
..................................+++
e is 65537 (0x10001)
[root@jymlinux ssl]# openssl req -new -key httpd.key -out httpd.csr
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [XX]:CN
State or Province Name (full name) []:XJ
Locality Name (eg, city) [Default City]:XJ
Organization Name (eg, company) [Default Company Ltd]:JJ
Organizational Unit Name (eg, section) []:Ops
Common Name (eg, your name or your server's hostname) []:CA
Email Address []:rs1.admin.com
 
Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:
 
3、将申请的证书发往CA
[root@jymlinux ssl]# scp httpd.csr root@192.168.3.10:/root
The authenticity of host '192.168.3.10 (192.168.3.10)' can't be established.
RSA key fingerprint is ef:85:f8:aa:1c:de:41:5a:fd:93:8d:9f:83:f7:a2:ff.
Are you sure you want to continue connecting (yes/no)? y
Please type 'yes' or 'no': yes
Warning: Permanently added '192.168.3.10' (RSA) to the list of known hosts.
root@192.168.3.10's password: 
httpd.csr                                           100% 1013     1.0KB/s   00:00 
 
4、CA签署并发证
[root@localhost CA]# openssl ca -in /root/httpd.csr -out /root/httpd.crt
Using configuration from /etc/pki/tls/openssl.cnf
Check that the request matches the signature
Signature ok
Certificate Details:
        Serial Number: 1 (0x1)
        Validity
            Not Before: Nov  9 13:48:21 2016 GMT
            Not After : Nov  9 13:48:21 2017 GMT
        Subject:
            countryName               = CN
            stateOrProvinceName       = XJ
            organizationName          = JJ
            organizationalUnitName    = Ops
            commonName                = CA
            emailAddress              = rs1.admin.com
        X509v3 extensions:
            X509v3 Basic Constraints: 
                CA:FALSE
            Netscape Comment: 
                OpenSSL Generated Certificate
            X509v3 Subject Key Identifier: 
                7E:FA:3A:6F:89:28:EF:D1:CF:5C:42:75:50:7B:C6:99:1D:98:91:B6
            X509v3 Authority Key Identifier: 
                keyid:91:9D:0E:8E:86:45:09:DE:C3:3F:63:61:C2:3D:CB:E1:E3:1C:F1:B6
 
Certificate is to be certified until Nov  9 13:48:21 2017 GMT (365 days)
Sign the certificate? [y/n]:y
 
 
1 out of 1 certificate requests certified, commit? [y/n]y
Write out database with 1 new entries
Data Base Updated
[root@localhost CA]# scp /root/httpd.crt root@192.168.3.20:/etc/httpd/ssl/
The authenticity of host '192.168.3.20 (192.168.3.20)' can't be established.
RSA key fingerprint is e5:84:6c:f7:c0:60:3d:0b:39:b6:1e:12:0d:48:8b:07.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.3.20' (RSA) to the list of known hosts.
root@192.168.3.20's password: 
httpd.crt                                           100% 4482     4.4KB/s   00:00  
 
4、安装mod-ssl
[root@jymlinux ~]# yum install mod_ssl
 
5、修改ssl的配置文件
[root@jymlinux ~]# cd /etc/httpd/conf.d/
[root@jymlinux conf.d]# vim ssl.conf 
DocumentRoot "/var/www/html"                       #启用
SSLCertificateFile /etc/httpd/ssl/httpd.crt        #修改证书以及密钥的所在路径
SSLCertificateKeyFile /etc/httpd/ssl/httpd.key
 
6、重启httpd服务
[root@jymlinux conf.d]# service httpd restart
 
7、Director配置ipvsadm规则
#使用iptables在PREROUTING链上的MARK标记将http与https标记为一组
[root@localhost ~]# iptables -t mangle -A PREROUTING -d 192.168.3.15 -p tcp --dport 80 -j MARK --set-mark 10
[root@localhost ~]# iptables -t mangle -A PREROUTING -d 192.168.3.15 -p tcp --dport 443 -j MARK --set-mark 10
 
[root@localhost ~]# ipvsadm -A -f 10 -s rr
[root@localhost ~]# ipvsadm -a -f 10 -r 192.168.3.20 -g
[root@localhost ~]# ipvsadm -a -f 10 -r 192.168.3.21 -g