【原创】学习日记5：nginx负载均衡（三）2012.01.10 nginx+keepalievd，实现负载均衡和故障点切换。keepalived双机热备。

接上文  学习日记4：nginx负载均衡（二）2012.01.08

......

到此，整体工作还差一个备用nginx proxy，其中涉及到keepalived，做心跳。

　　　

黄线以上做完了。

本次主要做的是1加一台从nginx_proxy ，暂且命令为front2（为从）。之前的nginx_proxy命令为front（为主）

1 在主nginx上配置keepalived。

cd /usr/local/src/
wget http://keepalived.org/software/keepalived-1.2.7.tar.gz

tar -xzvf keepalived-1.2.7.tar.gz
cd keepalived-1.2.7/
./configure --prefix=/usr/local/keepalived

这时候会报错的！

configure: error: 
  !!! OpenSSL is not properly installed on your system. !!!
  !!! Can not include OpenSSL headers files.            !!!

缺少gcc，安装下面扩展

yum install gcc gcc-c++ autoconf automake

继续编译，下面还会报错，类似这样的错误。

make: *** No targets specified and no makefile found.  Stop.

configure: error: ! OpenSSL is not properly installed on your system. configure: error: !

 对此，我们安装以下扩展即可解决。

yum -y install openssl-devel

yum -y install popt-devel

在编译

./configure --prefix=/usr/local/keepalived

这时候，会出现下面的结果，这个结果就表明是ok的。到此你的keepalived安装是成功的！

Keepalived configuration
------------------------
Keepalived version       : 1.2.7
Compiler                 : gcc
Compiler flags           : -g -O2
Extra Lib                : -lpopt -lssl -lcrypto 
Use IPVS Framework       : Yes
IPVS sync daemon support : Yes
IPVS use libnl           : No
Use VRRP Framework       : Yes
Use VRRP VMAC            : Yes
SNMP support             : No
Use Debug flags          : No

继续下面的操作

make
make install
cp /usr/local/keepalived/sbin/keepalived /usr/sbin/
cp /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/
cp /usr/local/keepalived/etc/rc.d/init.d/keepalived /etc/init.d/
mkdir /etc/keepalived
cd /etc/keepalived
vim keepalived.conf

　

keepalived.conf配置文件比较重要

front1 ：主nginx的配置文件如下：

! Configuration File for keepalived
global_defs {
   router_id LVS_01
}
vrrp_instance VI_1 {
    state MASTER  #master 主nginx
    interface eth1
    virtual_router_id 51
    mcast_src_ip 192.168.65.133 #本机ip地址也即主nginx地址
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.65.128 #vip 虚拟出来的ip地址 要求在内网中未被使用
    }
}

到此，主nginx配置完成。同样的操作，在从nginx上也配置一遍。

不过从nginx的配置文件如下：

[root@localhost nginx]# vim /etc/keepalived/keepalived.conf 

! Configuration File for keepalived
global_defs {
   router_id LVS_01
}
vrrp_instance VI_1 {
    state BACKUP #slave 
    interface eth2
    virtual_router_id 51
    mcast_src_ip 192.168.65.137 #slave ip
    priority 80
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.65.128 #vip
    }
}

重启两边的keepalived服务：

[root@localhost conf.d]# service keepalived restart
Stopping keepalived:                                       [FAILED]
Starting keepalived:                                       [  OK  ]

主nginx上，用

[root@localhost conf.d]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:20:e9:20 brd ff:ff:ff:ff:ff:ff
    inet 192.168.65.133/24 brd 192.168.65.255 scope global eth1
    inet 192.168.65.128/32 scope global eth1
    inet6 fe80::20c:29ff:fe20:e920/64 scope link 
       valid_lft forever preferred_lft forever

 到此，我们的vip算是真正的绑定上了。

查看keepalived的日志：

root@localhost conf.d]# tail -f /var/log/messages 
Jan  9 18:53:06 localhost Keepalived_healthcheckers[3355]: Using LinkWatch kernel netlink reflector...
Jan  9 18:53:06 localhost Keepalived_vrrp[3356]: Configuration is using : 34824 Bytes
Jan  9 18:53:06 localhost Keepalived_vrrp[3356]: Using LinkWatch kernel netlink reflector...
Jan  9 18:53:06 localhost Keepalived_vrrp[3356]: VRRP sockpool: [ifindex(2), proto(112), fd(11,12)]
Jan  9 18:53:06 localhost Keepalived_vrrp[3356]: VRRP_Instance(VI_1) Transition to MASTER STATE
Jan  9 18:53:07 localhost Keepalived_vrrp[3356]: VRRP_Instance(VI_1) Entering MASTER STATE
Jan  9 18:53:07 localhost Keepalived_vrrp[3356]: VRRP_Instance(VI_1) setting protocol VIPs.
Jan  9 18:53:07 localhost avahi-daemon[1765]: Registering new address record for 192.168.65.128 on eth1.IPv4.
Jan  9 18:53:07 localhost Keepalived_healthcheckers[3355]: Netlink reflector reports IP 192.168.65.128 added
Jan  9 18:53:07 localhost Keepalived_vrrp[3356]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth1 for 192.168.65.128
Jan  9 18:53:12 localhost Keepalived_vrrp[3356]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth1 for 192.168.65.128

从nginx上，同样

[root@localhost nginx]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:a7:81:c7 brd ff:ff:ff:ff:ff:ff
    inet 192.168.65.137/24 brd 192.168.65.255 scope global eth2
    inet6 fe80::20c:29ff:fea7:81c7/64 scope link 
       valid_lft forever preferred_lft forever

我们发现并没有绑定vip 128

同时；

[root@localhost nginx]# tail -f /var/log/messages 
Jan  9 19:16:25 localhost Keepalived_healthcheckers[22420]: Registering Kernel netlink reflector
Jan  9 19:16:25 localhost Keepalived_healthcheckers[22420]: Registering Kernel netlink command channel
Jan  9 19:16:25 localhost Keepalived_vrrp[22421]: Opening file '/etc/keepalived/keepalived.conf'.
Jan  9 19:16:25 localhost Keepalived_healthcheckers[22420]: Opening file '/etc/keepalived/keepalived.conf'.
Jan  9 19:16:25 localhost Keepalived_vrrp[22421]: Configuration is using : 34822 Bytes
Jan  9 19:16:25 localhost Keepalived_healthcheckers[22420]: Configuration is using : 4821 Bytes
Jan  9 19:16:25 localhost Keepalived_vrrp[22421]: Using LinkWatch kernel netlink reflector...
Jan  9 19:16:25 localhost Keepalived_vrrp[22421]: VRRP_Instance(VI_1) Entering BACKUP STATE
Jan  9 19:16:25 localhost Keepalived_vrrp[22421]: VRRP sockpool: [ifindex(2), proto(112), fd(11,12)]
Jan  9 19:16:25 localhost Keepalived_healthcheckers[22420]: Using LinkWatch kernel netlink reflector...

貌似从nginx似乎出了什么问题，先不着急。

这时候我们停止掉主nginx的服务，再看看效果怎么样！

killall nginx

再观察从nginx的状态变化

[root@localhost nginx]# tail -f /var/log/messages 
Jan  9 19:16:25 localhost Keepalived_healthcheckers[22420]: Registering Kernel netlink reflector
Jan  9 19:16:25 localhost Keepalived_healthcheckers[22420]: Registering Kernel netlink command channel
Jan  9 19:16:25 localhost Keepalived_vrrp[22421]: Opening file '/etc/keepalived/keepalived.conf'.
Jan  9 19:16:25 localhost Keepalived_healthcheckers[22420]: Opening file '/etc/keepalived/keepalived.conf'.
Jan  9 19:16:25 localhost Keepalived_vrrp[22421]: Configuration is using : 34822 Bytes
Jan  9 19:16:25 localhost Keepalived_healthcheckers[22420]: Configuration is using : 4821 Bytes
Jan  9 19:16:25 localhost Keepalived_vrrp[22421]: Using LinkWatch kernel netlink reflector...
Jan  9 19:16:25 localhost Keepalived_vrrp[22421]: VRRP_Instance(VI_1) Entering BACKUP STATE
Jan  9 19:16:25 localhost Keepalived_vrrp[22421]: VRRP sockpool: [ifindex(2), proto(112), fd(11,12)]
Jan  9 19:16:25 localhost Keepalived_healthcheckers[22420]: Using LinkWatch kernel netlink reflector...
Jan  9 19:17:33 localhost Keepalived_vrrp[22421]: VRRP_Instance(VI_1) Transition to MASTER STATE
Jan  9 19:17:34 localhost Keepalived_vrrp[22421]: VRRP_Instance(VI_1) Entering MASTER STATE
Jan  9 19:17:34 localhost Keepalived_vrrp[22421]: VRRP_Instance(VI_1) setting protocol VIPs.
Jan  9 19:17:34 localhost Keepalived_vrrp[22421]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth2 for 192.168.65.128
Jan  9 19:17:34 localhost avahi-daemon[1810]: Registering new address record for 192.168.65.128 on eth2.IPv4.
Jan  9 19:17:34 localhost Keepalived_healthcheckers[22420]: Netlink reflector reports IP 192.168.65.128 added
Jan  9 19:17:39 localhost Keepalived_vrrp[22421]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth2 for 192.168.65.128

从第33秒后发现，备用nginx已经转为master状态。

再看

[root@localhost nginx]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:a7:81:c7 brd ff:ff:ff:ff:ff:ff
    inet 192.168.65.137/24 brd 192.168.65.255 scope global eth2
    inet 192.168.65.128/32 scope global eth2
    inet6 fe80::20c:29ff:fea7:81c7/64 scope link 
       valid_lft forever preferred_lft forever

发现没有？vip飘过来了！

针对Keepalived的不足,用check_nginx.sh来监控nginx进程，实现真正意义上的负载均衡高可用.
此脚本思路其实也很简单，即放置在后台一直监控nginx进程；如进程消失，尝试重启nginx，
如是失败则立即停掉本机的keepalived服务，让另一台负载均衡器接手.

vi /shell/nginx_pid.sh
###################################################
#!/bin/bash
while :
    do
    nginxpid=`ps -C nginx --no-header | wc -l`

 if [ $nginxpid -eq 0 ]
   then
   /usr/local/nginx-0.8.53/sbin/nginx
   sleep 5
 if [ $nginxpid -eq 0 ]
    then
    /etc/init.d/keepalived stop
    fi
    fi
    sleep 5
done
###################################################

我们仍然可以继续访问我们的站点。至此。故障点切换宣告结束。