Linux集群

集群介绍

根据功能划分为两大类：高可用和负载均衡

(1)高可用集群通常为两台服务器，台工作，另外一台作为准备，当提供服务的机器宕机，另外一台将接替继续提供服务。

实现高可用的开源软件有：heartbeat,keepalived

(2)负载均衡集群：需要有一台服务器作为分发器，它负责吧用户的请求分发给后端的服务器处理，在这个集群里，除了分发器外，就是给用户提供服务的服务器了，这些服务器数量最少为2

实现负载均衡的开源软件有LVS，keepalived，haproxy，nginx，商业的有F5，Netscaler

keepalived介绍

heartbeat在centos6上有一些bug，影响实验效果。所以我们不介意使用heartbeat。

keepalived通过VRRP（虚拟路由冗余协议）来实现高可用。

在这个协议里会将多台功能相同的路由器组成一个小组，这个小组里会有一个master角色和N（N>=1）个backup角色。

工作原理：mster会通过组播的形式向各个backup发送VRRP协议的数据包，当backup收不到master发来的VRRP数据包时，就会认为master宕机了，此时就需要根据哥哥backup的优先级来决定谁成为新的mater。

keepalived要有三个模块，分别是core，check，vrrp。其中core模块为keepalived的核心，负责主进程的启动，维护以及全局配置文件的加载和解析，check模块负责健康检查，vrrp模块是用来实现VRRP协议的。

用keepalived配置高可用集群

1.准备两台机器

master:192.168.133.88

backup:192.168.133.66

2.两台机.器都安装keepalived

[root@sunyujun01 ~]# yum install -y keepalived

3.两台机器都安装Nginx，如果yum install -y nginx 提示没有nginx包，需要安装epel后再安装nginx包

[root@sunyujun01 ~]# yum install -y epel-release.noarch

[root@sunyujun01 ~]# yum install -y nginx

4.启动nginx

[root@sunyujun01 conf.d]# systemctl start nginx
[root@sunyujun01 conf.d]# ps aux|grep nginx
root       5273  0.0  0.2 122892  2100 ?        Ss   21:44   0:00 nginx: master process /usr/sbin/nginx
nginx      5274  0.3  0.3 123356  3128 ?        S    21:44   0:00 nginx: worker process
root       5276  0.0  0.0 112664   932 pts/4    S+   21:44   0:00 grep --color=auto nginx
[root@sunyujun01 conf.d]# 

5.keepalived+nginx实现web高可用

Keepalived的作用是检测服务器的状态，如果有一台web服务器宕机，或工作出现故障，Keepalived将检测到，并将有故障的服务器从系统中剔除，同时使用其他服务器代替该服务器的工作，当服务器工作正常后Keepalived自动将服务器加入到服务器群中，这些工作全部自动完成，不需要人工干涉，需要人工做的只是修复故障的服务器。

6.配置master上的keepalived配置文件

[root@sunyujun01 keepalived]# > keepalived.conf
[root@sunyujun01 keepalived]# vim keepalived.conf

　　

global_defs {
   notification_email {            （出现问题发邮件的邮箱）
    278108678@qq.com
   } 
   notification_email_from 278108678@qq.com         （由哪一个邮箱地址发出，可以引用第三方）   
   smtp_server 127.0.0.1                         
   smtp_connect_timeout 30 
   router_id LVS_DEVEL
}  
vrrp_script chk_nginx {                     （检测服务是否正常，还记得之前说过的keepalived的check模块吗？）
    script "/usr/local/sbin/check_ng.sh"         （需要写一个检测脚本，可以自定义）
    interval 3                           （检测间断3秒）
}   
vrrp_instance VI_1 {
    state MASTER                         （定义角色，是master还是backup）
    interface ens33                       （通过哪一个网卡想backup发送VRRP协议数据包）
    virtual_router_id 51                    （定义路由器的id，master要和backup保持一致）
    priority 100                          （权重，master和backup的权重是不一样的）
    advert_int 1
    authentication {                        （认证相关的配置）
        auth_type PASS                       （认证的类型是PASS密码）
        auth_pass sunyujun                 （定义密码是多少）
    }   
    virtual_ipaddress {  （定义VIP，也就是虚拟IP，这个IP是由keepalived给服务器配置的，服务器靠这个VIP对外提供服务）
        192.168.133.200
    }   
    track_script {                         （加载检查脚本）
        chk_nginx                  
    }   
}   

7.配置keepalived-nginx检查脚本，路径就是在上面配置文件里写的

[root@sunyujun01 keepalived]# vi /usr/local/sbin/check_ng.sh
[root@sunyujun01 keepalived]# 
#!/bin/bash
#时间变量，用于记录日志
d=`date --date today +%Y%m%d_%H:%M:%S`
#计算nginx进程数量
n=`ps -C nginx --no-heading|wc -l`
#如果进程为0，则启动nginx，并且再次检测nginx进程数量，
#如果还为0，说明nginx无法启动，此时需要关闭keepalived
if [ $n -eq "0" ]; then
        /etc/init.d/nginx start (如果用yum安装的nginx需要修改为 systemctl start nginx)
        n2=`ps -C nginx --no-heading|wc -l`
        if [ $n2 -eq "0"  ]; then
                echo "$d nginx down,keepalived will stop" >> /var/log/check_ng.log
                systemctl stop keepalived
        fi
fi

8.权限变更，不改权限文件就无法自动加载这个脚本，从而导致不能启动keepalive服务　　

[root@sunyujun01 ~]# chmod 755 /usr/local/sbin/check_ng.sh
[root@sunyujun01 ~]# 

9.测试keepalived和nginx服务停止掉自动启动，结果是可以，在这里我们要注意，如果有问题。他的错误日志在/var/log/messages里。要关闭防火墙和selinux，或者更改规则。

[root@sunyujun01 ~]# setenforce 0
[root@sunyujun01 ~]# iptables -F
[root@sunyujun01 ~]# 

　　

[root@sunyujun01 ~]# systemctl stop firewalld
[root@sunyujun01 ~]# 

　　

[root@sunyujun01 ~]# systemctl start keepalived
[root@sunyujun01 ~]# ps aux|grep keepalived
root      46589  0.8  0.1 120704  1476 ?        Ss   22:54   0:10 /usr/sbin/keepalived -D
root      46590  0.0  0.3 122776  3044 ?        S    22:54   0:00 /usr/sbin/keepalived -D
root      49699  0.3  0.2 131640  2956 ?        S    22:59   0:03 /usr/sbin/keepalived -D
root      51578  0.0  0.0 112664   936 pts/1    S+   23:14   0:00 grep --color=auto keepalived
[root@sunyujun01 ~]# ps aux|grep nginx
root      50794  0.0  0.2 122892  2104 ?        Ss   23:07   0:00 nginx: master process /usr/sbin/nginx
nginx     50795  0.0  0.3 123356  3132 ?        S    23:07   0:00 nginx: worker process
root      51640  0.0  0.0 112664   932 pts/1    R+   23:14   0:00 grep --color=auto nginx
[root@sunyujun01 ~]# systemctl stop nginx
[root@sunyujun01 ~]# ps aux|grep nginx
root      51748  0.0  0.2 122892  2104 ?        Ss   23:15   0:00 nginx: master process /usr/sbin/nginx
nginx     51749  0.0  0.3 123356  3132 ?        S    23:15   0:00 nginx: worker process
root      51754  1.0  0.0 112664   928 pts/1    R+   23:15   0:00 grep --color=auto nginx
[root@sunyujun01 ~]# 

10.查看ip 192.168.133.200/32

[root@sunyujun01 ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:33:1b:3e brd ff:ff:ff:ff:ff:ff
    inet 192.168.133.88/24 brd 192.168.133.255 scope global ens33
       valid_lft forever preferred_lft forever
    inet 192.168.133.200/32 scope global ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::b646:159d:d0ac:4cbe/64 scope link 
       valid_lft forever preferred_lft forever
[root@sunyujun01 ~]# 　　

11.配置从机backup：（检查主从机子的防火墙和selinux的规则。虚拟环境下我们就关闭它）

12.关闭防火墙

[root@sunyujun02 ~]# iptables -nvL
[root@sunyujun02 ~]# iptables -F
[root@sunyujun02 ~]# systemctl stop firewalld
[root@sunyujun02 ~]# getenforce 
Enforcing
[root@sunyujun02 ~]# setenforce 0
[root@sunyujun02 ~]# getenforce 
Permissive
[root@sunyujun02 ~]# 

13.编辑从配置文件

[root@sunyujun02 ~]# > /etc/keepalived/keepalived.conf 
[root@sunyujun02 ~]# vim /etc/keepalived/keepalived.conf 
[root@sunyujun02 ~]# 

global_defs {
   notification_email {
     aming@aminglinux.com
   }
   notification_email_from root@aminglinux.com
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id LVS_DEVEL
}
vrrp_script chk_nginx {
    script "/usr/local/sbin/check_ng.sh"
    interval 3
}
vrrp_instance VI_1 {
    state BACKUP                           （这里和主不一样，角色为backup）
    interface ens33
    virtual_router_id 51                      （这里要和主保持一样）
    priority 90                            （权重要比主低）
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass litongyao123                    （密码要和主一样）
    }
    virtual_ipaddress {
        192.168.52.99                          （虚拟vip要和主一样）
    }
    track_script {
        chk_nginx
    }   
}  

14.编辑监控nginx服务的脚本　　

[root@sunyujun02 ~]# vim /usr/local/sbin/check_ng.sh
[root@sunyujun02 ~]# 
#时间变量，用于记录日志
d=`date --date today +%Y%m%d_%H:%M:%S`
#计算nginx进程数量
n=`ps -C nginx --no-heading|wc -l`
#如果进程为0，则启动nginx，并且再次检测nginx进程数量，
#如果还为0，说明nginx无法启动，此时需要关闭keepalived
if [ $n -eq "0" ]; then
        systemctl start nginx                （这里和主的区别，因为我们从机是Yum安装的）
        n2=`ps -C nginx --no-heading|wc -l`
        if [ $n2 -eq "0"  ]; then
                echo "$d nginx down,keepalived will stop" >> /var/log/check_ng.log
                systemctl stop keepalived
        fi
fi

15.增加权限，启动keepalived

[root@sunyujun02 ~]# chmod 755 !$
chmod 755 /usr/local/sbin/check_ng.sh
[root@sunyujun02 ~]# systemctl start keepalived
[root@sunyujun02 ~]# ps aux|grep keep
root       2610  0.0  0.1 120704  1400 ?        Ss   23:56   0:00 /usr/sbin/keepalived -D
root       2611  0.3  0.3 122776  3056 ?        S    23:56   0:00 /usr/sbin/keepalived -D
root       2612  0.7  0.2 127100  2784 ?        S    23:56   0:00 /usr/sbin/keepalived -D
root       2683  0.0  0.0 112664   932 pts/1    S+   23:56   0:00 grep --color=auto keep
[root@sunyujun02 ~]# 

16.测试主服务器nginx是否通

　

17.修改从默认页文件title

[root@sunyujun02 ~]# cat /usr/share/nginx/html/index.html 

18.从服务器

19.用200服务器访问，发现访问的是主服务器（vip在主上）

20.我们在从上停掉nginx服务试试，发现nginx服务又起来了

[root@sunyujun02 ~]# ps aux|grep nginx
root       2637  0.0  0.2 122892  2104 ?        Ss   3月05   0:00 nginx: master process /usr/sbin/nginx
nginx      2638  0.0  0.3 123356  3572 ?        S    3月05   0:00 nginx: worker process
root       5534  0.0  0.0 112664   932 pts/1    S+   00:13   0:00 grep --color=auto nginx
[root@sunyujun02 ~]# systemctl stop nginx
[root@sunyujun02 ~]# ps aux|grep nginx
root       5785  0.0  0.2 122892  2104 ?        Ss   00:15   0:00 nginx: master process /usr/sbin/nginx
nginx      5786  0.0  0.3 123356  3132 ?        S    00:15   0:00 nginx: worker process
root       5791  0.0  0.0 112664   932 pts/1    S+   00:15   0:00 grep --color=auto nginx
[root@sunyujun02 ~]# 

测试

1.目前vip在master上

[root@sunyujun01 ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:33:1b:3e brd ff:ff:ff:ff:ff:ff
    inet 192.168.133.88/24 brd 192.168.133.255 scope global ens33
       valid_lft forever preferred_lft forever
    inet 192.168.133.200/32 scope global ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::b646:159d:d0ac:4cbe/64 scope link 
       valid_lft forever preferred_lft forever
[root@sunyujun01 ~]# 

[root@sunyujun02 ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:b2:c6:90 brd ff:ff:ff:ff:ff:ff
    inet 192.168.133.66/24 brd 192.168.133.255 scope global ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::d721:b5f6:8bf1:60d/64 scope link 
       valid_lft forever preferred_lft forever
[root@sunyujun02 ~]# 

2.将主上vrrp出去的包用防火墙封掉（这招测试不好使 哈哈）

[root@sunyujun02 ~]# iptables -I OUTPUT -p vrrp -j DROP
[root@sunyujun02 ~]# iptables -nvL
Chain INPUT (policy ACCEPT 58 packets, 3328 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 21 packets, 1980 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DROP       112  --  *      *       0.0.0.0/0            0.0.0.0/0           
[root@sunyujun02 ~]# 

3.测试主机宕机场景，直接停掉主机的keepalived服务，发现主200的ip释放了，从上有了200ip

[root@sunyujun01 ~]# systemctl stop keepalived
[root@sunyujun01 ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:33:1b:3e brd ff:ff:ff:ff:ff:ff
    inet 192.168.133.88/24 brd 192.168.133.255 scope global ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::b646:159d:d0ac:4cbe/64 scope link 
       valid_lft forever preferred_lft forever
[root@sunyujun01 ~]# 

[root@sunyujun02 ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:b2:c6:90 brd ff:ff:ff:ff:ff:ff
    inet 192.168.133.66/24 brd 192.168.133.255 scope global ens33
       valid_lft forever preferred_lft forever
    inet 192.168.133.200/32 scope global ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::d721:b5f6:8bf1:60d/64 scope link 
       valid_lft forever preferred_lft forever
[root@sunyujun02 ~]# 

4.查看从上的日志

[root@sunyujun02 ~]# tail -n 20 /var/log/messages
Mar  6 00:15:24 sunyujun02 systemd: Starting The nginx HTTP and reverse proxy server...
Mar  6 00:15:24 sunyujun02 nginx: nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
Mar  6 00:15:24 sunyujun02 nginx: nginx: configuration file /etc/nginx/nginx.conf test is successful
Mar  6 00:15:24 sunyujun02 systemd: Started The nginx HTTP and reverse proxy server.
Mar  6 00:21:48 sunyujun02 kernel: perf: interrupt took too long (83390 > 79618), lowering kernel.perf_event_max_sample_rate to 2000
Mar  6 00:33:30 sunyujun02 Keepalived_vrrp[2612]: VRRP_Instance(VI_1) Transition to MASTER STATE
Mar  6 00:33:31 sunyujun02 Keepalived_vrrp[2612]: VRRP_Instance(VI_1) Entering MASTER STATE
Mar  6 00:33:31 sunyujun02 Keepalived_vrrp[2612]: VRRP_Instance(VI_1) setting protocol VIPs.
Mar  6 00:33:31 sunyujun02 Keepalived_vrrp[2612]: Sending gratuitous ARP on ens33 for 192.168.133.200
Mar  6 00:33:31 sunyujun02 Keepalived_vrrp[2612]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on ens33 for 192.168.133.200
Mar  6 00:33:31 sunyujun02 Keepalived_vrrp[2612]: Sending gratuitous ARP on ens33 for 192.168.133.200
Mar  6 00:33:31 sunyujun02 Keepalived_vrrp[2612]: Sending gratuitous ARP on ens33 for 192.168.133.200
Mar  6 00:33:31 sunyujun02 Keepalived_vrrp[2612]: Sending gratuitous ARP on ens33 for 192.168.133.200
Mar  6 00:33:31 sunyujun02 Keepalived_vrrp[2612]: Sending gratuitous ARP on ens33 for 192.168.133.200
Mar  6 00:33:36 sunyujun02 Keepalived_vrrp[2612]: Sending gratuitous ARP on ens33 for 192.168.133.200
Mar  6 00:33:36 sunyujun02 Keepalived_vrrp[2612]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on ens33 for 192.168.133.200
Mar  6 00:33:36 sunyujun02 Keepalived_vrrp[2612]: Sending gratuitous ARP on ens33 for 192.168.133.200
Mar  6 00:33:36 sunyujun02 Keepalived_vrrp[2612]: Sending gratuitous ARP on ens33 for 192.168.133.200
Mar  6 00:33:36 sunyujun02 Keepalived_vrrp[2612]: Sending gratuitous ARP on ens33 for 192.168.133.200
Mar  6 00:33:36 sunyujun02 Keepalived_vrrp[2612]: Sending gratuitous ARP on ens33 for 192.168.133.200
[root@sunyujun02 ~]# 

5.访问200ip，发现切换到从上了

6.再次启动从机，发现200 vip又回来了

[root@sunyujun01 ~]# systemctl start keepalived
[root@sunyujun01 ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:33:1b:3e brd ff:ff:ff:ff:ff:ff
    inet 192.168.133.88/24 brd 192.168.133.255 scope global ens33
       valid_lft forever preferred_lft forever
    inet 192.168.133.200/32 scope global ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::b646:159d:d0ac:4cbe/64 scope link 
       valid_lft forever preferred_lft forever
[root@sunyujun01 ~]# 

　

7.这就是高可用集群，配置集群时，priority值越大，优先级越高　　

四、keeplived+MySQL

http://blog.51cto.com/lizhenliang/1362313