Haproxy+Keepalived+Nginx

Haproxy+Keepalived+Nginx

高可用负载均衡

官方网站:
http://haproxy.com/
http://www.keepalived.org/
http://nginx.org/

实验环境
OS：CentOS6.5 x64
软件版本：
haproxy-1.5.3
keepalived-1.2.13
nginx-1.6.2
主机：
haproxy VIP(keepalived): 192.168.8.100
haproxy1+keepalive-master: 192.168.8.80,192.168.7.70
haproxy2+keepalive-backup: 192.168.8.81,192.168.7.71
nginx1: 192.168.7.100
nginx2: 192.168.7.200




Nginx
请参看nginx-1.6.2+php-5.5.19+mariadb-10.0.14

HAProxy
一.安装编译依赖库
[root@haproxy1 ~]# yum -y install pcre-devel openssl-devel zlib-devel

二.编译安装
[root@haproxy1 ~]# tar -xvf haproxy-1.5.3.tar.gz -C /usr/local/src/
[root@haproxy1 ~]# cd /usr/local/src/haproxy-1.5.3/
[root@haproxy1 haproxy-1.5.3]# make TARGET=linux2628 USE_PCRE=1 USE_OPENSSL=1 USE_ZLIB=1
[root@haproxy1 haproxy-1.5.3]# make install

三.配置
1.init控制脚本
[root@haproxy1 haproxy-1.5.3]# ln -s /usr/local/sbin/haproxy* /usr/sbin/
[root@haproxy1 haproxy-1.5.3]# cp examples/haproxy.init /etc/init.d/haproxy
[root@haproxy1 haproxy-1.5.3]# chmod 755 /etc/init.d/haproxy
[root@haproxy1 ~]# chkconfig --add haproxy
[root@haproxy1 ~]# chkconfig haproxy on
[root@haproxy1 ~]# chkconfig --list haproxy
haproxy            0:off    1:off    2:on    3:on    4:on    5:on    6:off

2.配置文件/etc/haproxy/haproxy.cfg

http://www.haproxy.org/download/1.7/doc/configuration.txt

[root@haproxy1 haproxy-1.5.3]# mkdir /etc/haproxy
[root@haproxy1 haproxy-1.5.3]# mkdir /usr/share/haproxy
[root@haproxy1 haproxy-1.5.3]# cp examples/haproxy.cfg /etc/haproxy
[root@haproxy1 haproxy-1.5.3]# vim /etc/haproxy/haproxy.cfg
# this config needs haproxy-1.1.28 or haproxy-1.2.1

global
        log 127.0.0.1   local0
        log 127.0.0.1   local1 notice
        #log loghost    local0 info
        maxconn 4096
        chroot /usr/share/haproxy
        uid 99
        gid 99
        daemon
        #debug
        #quiet

defaults
        log     global
        mode    http
        option  httplog
        option  dontlognull
        retries 3

       stats enable    

       stats uri /haproxy-stats

       stats refresh 10s

       stats realm Haproxy statistic

       stats auth hadmin:foo.123

#redispatch
        maxconn 2000
        timeout connect 5000
        timeout client  50000
        timeout server  50000

listen  nginx-http 0.0.0.0:80
        cookie  SERVERID rewrite
        balance roundrobin
        server  nginx1 192.168.7.100:80 cookie nginx1 check inter 2000 rise 2 fall 5
        server  nginx2 192.168.7.200:80 cookie nginx2 check inter 2000 rise 2 fall 5
listen  nginx-https 0.0.0.0:443
        cookie  SERVERID rewrite
        balance roundrobin
        server  nginx1 192.168.7.100:443 cookie nginx1 check inter 2000 rise 2 fall 5
        server  nginx2 192.168.7.200:443 cookie nginx2 check inter 2000 rise 2 fall 5
补充：如果是调度数据库，如Redis, MariaDB, MongoDB等非web应用时，则非要将模式改为mode tcp，同时cookie也会失效，重启服务后会提示，所以也要删除对应的cookie选项。
[root@haproxy1 haproxy]# /etc/init.d/haproxy restart
Shutting down haproxy:                                     [  OK  ]
Starting haproxy:                                          [  OK  ]
[root@haproxy1 haproxy]# netstat -tunlp|grep haproxy
tcp        0      0 0.0.0.0:80                  0.0.0.0:*                   LISTEN      3768/haproxy       
tcp        0      0 0.0.0.0:443                 0.0.0.0:*                   LISTEN      3768/haproxy       
udp        0      0 0.0.0.0:55509               0.0.0.0:*                               3768/haproxy  
到这里已经可以通过浏览器访问192.168.8.80，haproxy默认会轮巡地将请求调度给后端的nginx1和nginx2
在haproxy2上作同样的操作，配置文件也可以拷贝到haproxy2直接使用，但访问地址就变为192.168.8.81。
这里只是做最简单的配置，更多的高度算法及配置参数请参看官方文档即大神们的blog。

如果调度器haproxy工作停止，则整个服务全部停线，所以又引入高可用方案，即haproxy1和haproxy2负责调度以负载均衡，这两台调度器又作HA通过VIP对外提供服务，以保证服务的高可用。这里采用主流的keepalived

在配置文件defaults章节加入stats相关配置即可启用haproxy-status的dashboard,支持auth-basic认证,可以看到负载状态http://192.168.8.100:5000/haproxy-stats




Keepalived
请参看LVS+Keepalived DR模式（附脚本）
一. 安装编译依赖库
[root@haproxy1 ~]#yum –y install kernel-devel gcc openssl-devel popt-devel libnl-devel net-snmp-devel

二.编译安装
[root@haproxy1 ~]#tar -xvf keepalived-1.2.13.tar.gz -C /usr/local/src
[root@haproxy1 ~]#cd /usr/local/src/keepalived-1.2.13
[root@haproxy1 ~]#./configure
--prefix=/usr
--sysconf=/etc
--with-kernel-dir=/usr/src/kernels/$(uname -r)
--enable-snmp
--enable-sha1
标红的部分一定要有，否则不能编译成功
[root@haproxy1 ~]#make && make install

 

三. init脚本
[root@haproxy1 ~]#cp /usr/local/src/keepalived-1.2.13/keepalived/etc/init.d/keepalived.init /etc/rc.d/rc3.d/S99keepalived
[root@haproxy1 ~]# cp /usr/local/src/keepalived-1.2.13/keepalived/etc/init.d/keepalived.init /etc/rc.d/rc5.d/S99keepalived
[root@haproxy1 ~]#chkconfig keepalived on
[root@haproxy1 ~]# chkconfig --list keepalived
keepalived 0:off 1:off 2:on 3:on 4:on 5:on 6:off

四.配置keepalived
1.haproxy健康检查脚本haproxy_check
[root@haproxy1 keepalived]# pwd
/etc/keepalived
[root@haproxy1 keepalived]# mkdir scripts
[root@haproxy1 keepalived]# vim scripts/haproxy_check.sh
[root@haproxy1 keepalived]# chmod +x scripts/haproxy_check.sh 
############################################
#!/bin/bash
#To install check haproxy status automatically
#Made by liujun,2014/12/20
############################################
#Define system environment PATH
export PATH=$PATH

flag=$(pidof haproxy)

if [ "$flag" == "" ];then
        /etc/init.d/haproxy start
        sleep 3
        flag_wait=$(pidof haproxy)
        if [ "$flag_wait" == "" ];then
        /etc/init.d/keepalived stop
        fi
fi
2.主配置文件/etc/keepalived/keepalived.conf
[root@haproxy1 keepalived]# vim keepalived.conf
! Configuration File for keepalived
vrrp_script haproxy_check {
    script "/etc/keepalived/scripts/haproxy_check.sh"
    interval 2
    weight 2
}
vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 51
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.8.100
    }
    track_script {
        haproxy_check    }
}
[root@haproxy1 keepalived]# /etc/init.d/keepalived restart
Stopping keepalived:                                       [  OK  ]
Starting keepalived:                                       [  OK  ]
搞定，己经可以通过VIP192.168.8.100来访问后端的nginx了，现在的VIP在haproxy1上面
[root@haproxy1 keepalived]# ip addr list
1: lo: mtu 16436 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
2: eth0: mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:01:00:01 brd ff:ff:ff:ff:ff:ff
    inet 192.168.8.80/24 brd 192.168.8.255 scope global eth0
    inet 192.168.8.100/32 scope global eth0
3: eth1: mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:02:00:01 brd ff:ff:ff:ff:ff:ff
    inet 192.168.7.70/24 brd 192.168.7.255 scope global eth1
再配置haproxy2，直接将keepalived.conf和haproxy_check.sh脚本copy到haproxy2对应的目录，修改keepalived.conf中的以下两行，使haproxy2作为haproxy1的BACKUP
state BACKUP
priority 50


测试
一.在haproxy1上手动停掉haproxy
[root@haproxy1 ~]# /etc/init.d/haproxy stop
Shutting down haproxy:                                     [  OK  ]
[root@haproxy1 ~]# /etc/init.d/haproxy status
haproxy (pid  31824) is running...
发现，健康检查脚本会每隔2秒钟检查一次，如果haproxy没有进程号则通过管控脚本启动，再等3秒钟，如果还是没有启动，则停止keepalived让BACKUP来接管。说明keepalive在持续监控haproxy的健康状态。

二.直接停止keepalived
[root@haproxy1 ~]# /etc/init.d/keepalived stop
Stopping keepalived:                                       [  OK  ]
[root@haproxy1 ~]# tail -f /var/log/messages
Dec 20 17:07:19 sentinel Keepalived_vrrp[31768]: VRRP_Script(haproxy_check) timed out
Dec 20 17:07:19 sentinel Keepalived_vrrp[31768]: Process [31814] didn't respond to SIGTERM
Dec 20 17:07:19 sentinel Keepalived_vrrp[31768]: VRRP_Script(haproxy_check) succeeded
Dec 20 17:12:02 sentinel Keepalived[31765]: Stopping Keepalived v1.2.13 (12/20,2014)
Dec 20 17:12:02 sentinel Keepalived_vrrp[31768]: VRRP_Instance(VI_1) sending 0 priority
Dec 20 17:12:02 sentinel Keepalived_vrrp[31768]: VRRP_Instance(VI_1) removing protocol VIPs.
Dec 20 17:12:02 sentinel Keepalived_healthcheckers[31767]: Netlink reflector reports IP 192.168.8.100 removed
可以看到，VIP从MASTER节点上被移除，而飘到了BACKUP节点
[root@haproxy2 ~]# tail -f /var/log/messages
Dec 20 17:12:03 master Keepalived_vrrp[7103]: VRRP_Instance(VI_1) Transition to MASTER STATE
Dec 20 17:12:04 master Keepalived_vrrp[7103]: VRRP_Instance(VI_1) Entering MASTER STATE
Dec 20 17:12:04 master Keepalived_vrrp[7103]: VRRP_Instance(VI_1) setting protocol VIPs.
Dec 20 17:12:04 master Keepalived_healthcheckers[7102]: Netlink reflector reports IP 192.168.8.100 added
Dec 20 17:12:04 master Keepalived_vrrp[7103]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.8.100
Dec 20 17:12:09 master Keepalived_vrrp[7103]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.8.100
[root@haproxy2 ~]# ip add list
1: lo: mtu 16436 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
2: eth1: mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:01:00:02 brd ff:ff:ff:ff:ff:ff
    inet 192.168.7.71/24 brd 192.168.7.255 scope global eth1
3: eth0: mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:02:00:02 brd ff:ff:ff:ff:ff:ff
    inet 192.168.8.81/24 brd 192.168.8.255 scope global eth0
    inet 192.168.8.100/32 scope global eth0


三.keepalived MASTER恢复启动
[root@haproxy1 ~]# /etc/init.d/keepalived start
Starting keepalived:                                       [  OK  ]
[root@haproxy1 ~]# tail -f /var/log/messages
Dec 20 17:14:27 sentinel Keepalived[32292]: Starting Keepalived v1.2.13 (12/20,2014)
Dec 20 17:14:27 sentinel Keepalived[32293]: Starting Healthcheck child process, pid=32294
Dec 20 17:14:27 sentinel Keepalived[32293]: Starting VRRP child process, pid=32296
Dec 20 17:14:27 sentinel Keepalived_vrrp[32296]: Netlink reflector reports IP 192.168.8.80 added
Dec 20 17:14:27 sentinel Keepalived_vrrp[32296]: Netlink reflector reports IP 192.168.7.70 added
Dec 20 17:14:27 sentinel Keepalived_vrrp[32296]: Netlink reflector reports IP 192.168.8.80 added
Dec 20 17:14:27 sentinel Keepalived_vrrp[32296]: Netlink reflector reports IP 192.168.7.70 added
Dec 20 17:14:27 sentinel Keepalived_vrrp[32296]: Registering Kernel netlink reflector
Dec 20 17:14:27 sentinel Keepalived_vrrp[32296]: Registering Kernel netlink command channel
Dec 20 17:14:27 sentinel Keepalived_vrrp[32296]: Registering gratuitous ARP shared channel
Dec 20 17:14:27 sentinel Keepalived_healthcheckers[32294]: Netlink reflector reports IP 192.168.8.80 added
Dec 20 17:14:27 sentinel Keepalived_healthcheckers[32294]: Netlink reflector reports IP 192.168.7.70 added
Dec 20 17:14:27 sentinel Keepalived_healthcheckers[32294]: Netlink reflector reports IP 192.168.8.80 added
Dec 20 17:14:27 sentinel Keepalived_healthcheckers[32294]: Netlink reflector reports IP 192.168.7.70 added
Dec 20 17:14:27 sentinel Keepalived_healthcheckers[32294]: Registering Kernel netlink reflector
Dec 20 17:14:27 sentinel Keepalived_healthcheckers[32294]: Registering Kernel netlink command channel
Dec 20 17:14:27 sentinel Keepalived_vrrp[32296]: Opening file '/etc/keepalived/keepalived.conf'.
Dec 20 17:14:27 sentinel Keepalived_vrrp[32296]: Configuration is using : 63268 Bytes
Dec 20 17:14:27 sentinel Keepalived_vrrp[32296]: Using LinkWatch kernel netlink reflector...
Dec 20 17:14:27 sentinel Keepalived_vrrp[32296]: VRRP sockpool: [ifindex(2), proto(112), unicast(0), fd(10,11)]
Dec 20 17:14:27 sentinel Keepalived_vrrp[32296]: VRRP_Instance(VI_1) Transition to MASTER STATE
Dec 20 17:14:27 sentinel Keepalived_vrrp[32296]: VRRP_Instance(VI_1) Received lower prio advert, forcing new election
Dec 20 17:14:27 sentinel Keepalived_vrrp[32296]: VRRP_Script(haproxy_check) succeeded
Dec 20 17:14:28 sentinel Keepalived_vrrp[32296]: VRRP_Instance(VI_1) Entering MASTER STATE
Dec 20 17:14:28 sentinel Keepalived_vrrp[32296]: VRRP_Instance(VI_1) setting protocol VIPs.
Dec 20 17:14:28 sentinel Keepalived_vrrp[32296]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.8.100
Dec 20 17:14:32 sentinel Keepalived_healthcheckers[32294]: Opening file '/etc/keepalived/keepalived.conf'.
Dec 20 17:14:32 sentinel Keepalived_healthcheckers[32294]: Configuration is using : 5556 Bytes
Dec 20 17:14:32 sentinel Keepalived_healthcheckers[32294]: Using LinkWatch kernel netlink reflector...
Dec 20 17:14:32 sentinel Keepalived_healthcheckers[32294]: Netlink reflector reports IP 192.168.8.100 added
Dec 20 17:14:33 sentinel Keepalived_vrrp[32296]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.8.100
因为优先级比BACKUP高，所以VIP又飘了回来。

ok，实验结束。