DRBD+Heartbeat+mysql 高可用性负载均衡

一、安装配置DRBD：

DRBD1的IP地址为：192.168.2.203

DRBD2的IP地址为：192.168.2.204

安装内核包：

uname –a ##查看本系统的内核版本是多少

注意：安装的内核包要和所显示的系统内核版本一直，所以最好是安装系统光盘自带的内核包

mkdir /mnt/cdrom

mount /dev/cdrom /mnt/cdrom  #挂载系统光盘到/mnt/cdrom下

cd /mnt/cdrom/Packages

rpm –ivh kernel-devel-2.6.32-504.el6.x86_64.rpm

rpm –ivh kernel-headers-2.6.32-504.el6.x86_64.rpm

yum install gcc gcc-c++ make glibc flex perl -y

两台服务器上都安装DRBD：

wget http://oss.linbit.com/drbd/8.4/drbd-8.4.4.tar.gz

tar -xzvf drbd-8.4.4.tar.gz

cd drbd-8.4.4

./configure --prefix=/usr/local/drbd --with-km

make KDIR=/usr/src/kernels/2.6.32-504.el6.x86_64/

make install

mkdir -p /usr/local/drbd/var/run

cp /usr/local/drbd/etc/rc.d/init.d/drbd  /etc/rc.d/init.d/

chkconfig –add drbd

chkconfig drbd on

加载drbd模块

modprobe drbd

检查是否加载了drbd模块

lsmod|grep drbd

备注：遇到此种错误，直接yum install perl –y

参数配置：(drbd1,drbd2)

vi /usr/local/drbd/etc/drbd.conf

include "drbd.d/global_common.conf";

include "drbd.d/*.res";

resource r0{

    on drbd1 {

     device /dev/drbd0;

     disk  /dev/sdb1;

     address 192.168.2.203:7788;

     meta-disk internal;

}

    on drbd2 {

     device /dev/drbd0;

     disk /dev/sdb1;

     address 192.168.2.204:7788;

     meta-disk internal;

}

}

vi /usr/local/drbd/etc/drbd.d/global_common.conf

global {

        usage-count yes;

       }

common {

       

        startup {

                wfc-timeout 0; degr-wfc-timeout 120;

        }       

        disk {

              

                 on-io-error detach;

         }

              syncer {

             rate 600K;

       }

}

mknod /dev/drbd0 b 147 0

drbdadm create-md r0

两服务器同时启动drbd服务：

service drbd start

此时查看两台服务器的drbd状态均是这样：

drbd driver loaded OK; device status:

version: 8.4.4 (api:1/proto:86-101)

GIT-hash: 74402fecf24da8e5438171ee8c19e28627e1c98a build by root@drbd1, 2015-12-29 16:21:36

m:res  cs         ro                  ds                     p  mounted  fstype

0:r0   Connected  Secondary /Secondary  Inconsistent/ Inconsistent  C

备注：这里ro:Secondary/Secondary表示两台主机的状态都是备机状态，ds是磁盘状态，显示的状态内容为“Inconsistent不一致”，这是因为DRBD无法判断哪一方为主机，应以哪一方的磁盘数据作为标准。

在drbd1上配置为主节点：

drbdsetup /dev/drbd0 primary –force

查看两台服务器的drbd状态则为这样：

Drbd1：

drbd driver loaded OK; device status:

version: 8.4.4 (api:1/proto:86-101)

GIT-hash: 74402fecf24da8e5438171ee8c19e28627e1c98a build by root@drbd1, 2015-12-29 16:21:36

m:res  cs         ro                 ds                 p  mounted  fstype

0:r0   Connected  Primary/Secondary  UpToDate/UpToDate  C

Drbd2:

drbd driver loaded OK; device status:

version: 8.4.4 (api:1/proto:86-101)

GIT-hash: 74402fecf24da8e5438171ee8c19e28627e1c98a build by root@dbrd2, 2015-12-29 16:14:14

m:res  cs         ro                 ds                 p  mounted  fstype

0:r0   Connected  Secondary/Primary  UpToDate/UpToDate  C

ro在主从服务器上分别显示 Primary/Secondary和Secondary/Primary
ds显示UpToDate/UpToDate
表示主从配置成功。

挂载DRBD：(drbd1)
从刚才的状态上看到mounted和fstype参数为空，所以我们这步开始挂载DRBD到系统目录/drbd

mkfs.ext4 /dev/drbd0

mke2fs 1.41.12 (17-May-2010)

Filesystem label=

OS type: Linux

Block size=4096 (log=2)

Fragment size=4096 (log=2)

Stride=0 blocks, Stripe width=0 blocks

1310720 inodes, 5241029 blocks

262051 blocks (5.00%) reserved for the super user

First data block=0

Maximum filesystem blocks=4294967296

160 block groups

32768 blocks per group, 32768 fragments per group

8192 inodes per group

Superblock backups stored on blocks:

        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,

        4096000

Writing inode tables: done                           

Creating journal (32768 blocks): done

Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 36 mounts or

180 days, whichever comes first.  Use tune2fs -c or -i to override.

mkdir /drbd

mount /dev/drbd0 /drbd

注：Secondary节点上不允许对DRBD设备进行任何操作，包括挂载；所有的读写操作只能在Primary节点上进行，只有当Primary节点挂掉时，Secondary节点才能提升为Primary节点，并自动挂载DRBD继续工作。

成功挂载后的DRBD状态：(drbd1)

drbd driver loaded OK; device status:

version: 8.4.4 (api:1/proto:86-101)

GIT-hash: 74402fecf24da8e5438171ee8c19e28627e1c98a build by root@drbd1, 2015-12-29 16:21:36

m:res  cs         ro                 ds                 p  mounted  fstype

0:r0   Connected  Primary/Secondary  UpToDate/UpToDate  C  /drbd    ext4

二、安装mysql：

yum install libaio cmake ncurses-devel -y

tar -xzvf mysql-5.6.13.tar.gz

cd mysql-5.6.13

/usr/sbin/groupadd mysql

/usr/sbin/useradd -g mysql mysql

ln -s /usr/local/mysql/lib/libmysqlclient.so.18.1.0 /usr/lib64/libmysqlclient.so.18

ln -s /usr/local/mysql/lib/libmysqlclient.so.18.1.0 /usr/lib/libmysqlclient.so.18

ln -s /usr/local/mysql/lib/libmysqlclient.so.18.1.0 /usr/local/lib64/libmysqlclient.so.18

ln -s /usr/local/mysql/lib/libmysqlclient.so.18.1.0 /usr/local/lib/libmysqlclient.so.18

/usr/bin/cmake .-DCMAKE_INSTALL_PREFIX=/usr/local/mysql -DWITH_INNOBASE_STORAGE_ENGINE=1 -DWITH_ARCHIVE_STORAGE_ENGINE=1 -DWITH_BLACKHOLE_STORAGE_ENGINE=1 -DWITH_FEDERATED_STORAGE_ENGINE=1 -DWITH_PARTITION_STORAGE_ENGINE=1 -DENABLED_LOCAL_INFILE=0 -DEXTRA_CHARSETS=all -DDEFAULT_CHARSET=utf8 -DDEFAULT_COLLATION=utf8_general_ci -DMYSQL_USER=mysql -DINSTALL_LAYOUT=STANDALONE -DENABLED_PROFILING=ON -DMYSQL_MAINTAINER_MODE=OFF -DWITH_DEBUG=OFF

make

make install

chown -R mysql:mysql /usr/local/mysql

chown -R mysql:mysql /drbd/data/mysql

cp support-files/my-default.cnf /etc/my.cnf

cp support-files/mysql.server /etc/init.d/mysqld

chmod 755 /etc/init.d/mysqld

vi /etc/my.cnf （两台服务器均增加以下两行）

datadir=/drbd/data/mysql

basedir=/usr/local/mysql

/usr/local/mysql/scripts/mysql_install_db --user=mysql --datadir=/drbd/data/mysql/ --basedir=/usr/local/mysql

ls /drbd/data/mysql/（从服务器不需要初始化数据库）

三、手动切换drbd的主从

卸载主服务器上的磁盘

umount /drbd

service drbd status（主服务器）

drbd driver loaded OK; device status:

version: 8.4.4 (api:1/proto:86-101)

GIT-hash: 74402fecf24da8e5438171ee8c19e28627e1c98a build by root@drbd1, 2015-12-29 16:21:36

m:res  cs         ro                 ds                 p  mounted  fstype

0:r0   Connected  Primary/Secondary  UpToDate/UpToDate  C

drbdadm secondary all   #将主手动改为从

service drbd status（主服务器）

drbd driver loaded OK; device status:

version: 8.4.4 (api:1/proto:86-101)

GIT-hash: 74402fecf24da8e5438171ee8c19e28627e1c98a build by root@drbd1, 2015-12-29 16:21:36

m:res  cs         ro                   ds                 p  mounted  fstype

0:r0   Connected  Secondary/Secondary  UpToDate/UpToDate  C

drbdadm  primary all    #将从手动改为主

service drbd status（从服务器）

drbd driver loaded OK; device status:

version: 8.4.4 (api:1/proto:86-101)

GIT-hash: 74402fecf24da8e5438171ee8c19e28627e1c98a build by root@dbrd2, 2015-12-29 16:14:14

m:res  cs         ro                 ds                 p  mounted  fstype

0:r0   Connected  Primary/Secondary  UpToDate/UpToDate  C

mount /dev/drbd0 /drbd   #重新挂载磁盘

ls /drbd/    #mysql初始化数据已经存在了

注意：DRBD脑裂后的处理

当DRBD出现脑裂后，会导致drbd两边的磁盘数据不一致，在确定要作为从的节点上切换成secondary，并放弃该资源的数据：

drbdadm secondary r0

drbdadm -- --discard-my-data connect r0

在要作为primary的节点重新连接secondary（如果这个节点当前的连接状态为WFConnection的话，可以省略），使用如下命令连接：

drbdadm connect r0

三、配置Heartbeat：

yum install heartbeat –y

cp /usr/share/doc/heartbeat-3.0.4/ha.cf /etc/ha.d/ha.cf

cp /usr/share/doc/heartbeat-3.0.4/haresources /etc/ha.d/haresources

cp /usr/share/doc/heartbeat-3.0.4/authkeys /etc/ha.d/authkeys

（DRBD1）

配置ha.cf文件，添加以下配置：

echo ''>/etc/ha.d/ha.cf

vi /etc/ha.d/ha.cf

logfile /var/log/ha-log     #指定heartbeat日志文件的位置

logfacility     local0    

keepalive 2              #心跳发送时间间隔

deadtime 15             #备用节点15s内没有检测到主机心跳，确认对方故障

warntime 5              #警告5次

initdead 30              #守护进程启动30s后，启动服务资源

ucast eth1 192.168.2.204   #指定对方网卡及IP地址

auto_failback off         #当primary节点切换到sencondary节点之后，primary节点恢复正常，不进行切回操作，因为切换一次的成本很高

node   drbd1          #定义两个节点的主机名，一行写一个

node   drbd2

（DRBD2）

配置ha.cf文件，添加以下配置：

echo ''>/etc/ha.d/ha.cf

vi /etc/ha.d/ha.cf

logfile /var/log/ha-log     #指定heartbeat日志文件的位置

logfacility     local0    

keepalive 2              #心跳发送时间间隔

deadtime 15             #备用节点15s内没有检测到主机心跳，确认对方故障

warntime 5              #警告5次

initdead 30              #守护进程启动30s后，启动服务资源

ucast eth1 192.168.2.203   #指定对方网卡及IP地址

auto_failback off         #当primary节点切换到sencondary节点之后，primary节点恢复正常，不进行切回操作，因为切换一次的成本很高

node   drbd1          #定义两个节点的主机名，一行写一个

node   drbd2

编辑双机互联验证文件authkeys，添加以下内容：(drbd1,drbd2)

vi /etc/ha.d/authkeys

auth 1

1 crc

给验证文件600权限:

chmod 600 /etc/ha.d/authkeys

编辑集群资源文件：(drbd1,drbd2)

vi  /etc/ha.d/haresources

drbd1 192.168.2.200/24/eth1 drbddisk::r0 Filesystem::/dev/drbd0::/drbd::ext4 mysqld

创建DRBD脚本文件drbddisk:(两台服务器)

注意：

此处是一个大坑，因为默认yum安装Heartbeat，不会在/etc/ha.d/resource.d/创建drbddisk脚本，估计是版本太新了吧。记得前两年都不会这样的。囧。而且也无法在安装后从本地其他路径找到该文件。此处也是因为启动Heartbeat后无法PING通虚IP，最后通过查看/var/log/ha-log日志，找到一行ERROR: Cannot locate resource script drbddisk，然后进而到/etc/ha.d/resource.d/路径下发现竟然没有drbddisk脚本，最后在google上找到该代码，创建该脚本，终于测试通过：

vi /etc/ha.d/resource.d/drbddisk

#!/bin/bash

#

# This script is inteded to be used as resource script by heartbeat

#

# Copright 2003-2008 LINBIT Information Technologies

# Philipp Reisner, Lars Ellenberg

#

###

DEFAULTFILE="/etc/default/drbd"

DRBDADM="/sbin/drbdadm"

if [ -f $DEFAULTFILE ]; then

 . $DEFAULTFILE

fi

if [ "$#" -eq 2 ]; then

 RES="$1"

 CMD="$2"

else

 RES="all"

 CMD="$1"

fi

## EXIT CODES

# since this is a "legacy heartbeat R1 resource agent" script,

# exit codes actually do not matter that much as long as we conform to

#  http://wiki.linux-ha.org/HeartbeatResourceAgent

# but it does not hurt to conform to lsb init-script exit codes,

# where we can.

#  http://refspecs.linux-foundation.org/LSB_3.1.0/

#LSB-Core-generic/LSB-Core-generic/iniscrptact.html

####

drbd_set_role_from_proc_drbd()

{

local out

if ! test -e /proc/drbd; then

ROLE="Unconfigured"

return

fi

dev=$( $DRBDADM sh-dev $RES )

minor=${dev#/dev/drbd}

if [[ $minor = *[!0-9]* ]] ; then

# sh-minor is only supported since drbd 8.3.1

minor=$( $DRBDADM sh-minor $RES )

fi

if [[ -z $minor ]] || [[ $minor = *[!0-9]* ]] ; then

ROLE=Unknown

return

fi

if out=$(sed -ne "/^ *$minor: cs:/ { s/:/ /g; p; q; }" /proc/drbd); then

set -- $out

ROLE=${5%/**}

: ${ROLE:=Unconfigured} # if it does not show up

else

ROLE=Unknown

fi

}

case "$CMD" in

   start)

# try several times, in case heartbeat deadtime

# was smaller than drbd ping time

try=6

while true; do

$DRBDADM primary $RES && break

let "--try" || exit 1 # LSB generic error

sleep 1

done

;;

   stop)

# heartbeat (haresources mode) will retry failed stop

# for a number of times in addition to this internal retry.

try=3

while true; do

$DRBDADM secondary $RES && break

# We used to lie here, and pretend success for anything != 11,

# to avoid the reboot on failed stop recovery for "simple

# config errors" and such. But that is incorrect.

# Don't lie to your cluster manager.

# And don't do config errors...

let --try || exit 1 # LSB generic error

sleep 1

done

;;

   status)

if [ "$RES" = "all" ]; then

   echo "A resource name is required for status inquiries."

   exit 10

fi

ST=$( $DRBDADM role $RES )

ROLE=${ST%/**}

case $ROLE in

Primary|Secondary|Unconfigured)

# expected

;;

*)

# unexpected. whatever...

# If we are unsure about the state of a resource, we need to

# report it as possibly running, so heartbeat can, after failed

# stop, do a recovery by reboot.

# drbdsetup may fail for obscure reasons, e.g. if /var/lock/ is

# suddenly readonly.  So we retry by parsing /proc/drbd.

drbd_set_role_from_proc_drbd

esac

case $ROLE in

Primary)

echo "running (Primary)"

exit 0 # LSB status "service is OK"

;;

Secondary|Unconfigured)

echo "stopped ($ROLE)"

exit 3 # LSB status "service is not running"

;;

*)

# NOTE the "running" in below message.

# this is a "heartbeat" resource script,

# the exit code is _ignored_.

echo "cannot determine status, may be running ($ROLE)"

exit 4 #  LSB status "service status is unknown"

;;

esac

;;

   *)

echo "Usage: drbddisk [resource] {start|stop|status}"

exit 1

;;

esac

exit 0

给文件drbddisk权限：

chmod 755 /etc/ha.d/resource.d/drbddisk

配置好heartbeat之后，需要将mysql从自启动服务器中去掉，因为主heartbeat启动的时候会挂载drdb文件系统以及启动mysql，切换的时候会将主上的mysql停止并卸载文件系统，从上会挂载文件系统，并启动mysql。因此需要做如下操作（两台服务器）：

chkconfig mysqld off

chkconfig heartbeat off

chkconfig drbd off     

vi /etc/rc.local 

#!/bin/sh

#

# This script will be executed *after* all the other init scripts.

# You can put your own initialization stuff in here if you don't

# want to do the full Sys V style init stuff.

touch /var/lock/subsys/local

modprobe drbd              #必须先加载模块，这也是因为将启动命令放在这里的原因

/etc/init.d/drbd start

/etc/init.d/heartbeat start

到这里heartbeat+drbd+mysql高可用环境就搭建结束了。接下来进行测试。

高可用性测试：

在第一台服务器上面启动mysql服务（drbd1）

/etc/init.d/mysqld start

Starting MySQL.... SUCCESS!

/etc/init.d/drbd status

drbd driver loaded OK; device status:

version: 8.4.4 (api:1/proto:86-101)

GIT-hash: 74402fecf24da8e5438171ee8c19e28627e1c98a build by root@drbd1, 2015-12-29 16:21:36

m:res  cs         ro                 ds                      p  mounted  fstype

0:r0   Connected  Primary/Secondary  UpToDate/UpToDate  C   /drbd      ext4

在两台服务器上面启动heartbeat

[root@drbd1 ~]#/etc/init.d/heartbeat start

Starting High-Availability services: INFO:  Resource is stopped

Done.

[root@drbd2 ~]#/etc/init.d/heartbeat start

[root@drbd1 ~]#ip addr | grep eth1

2: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000

    inet 192.168.2.203/24 brd 192.168.2.255 scope global eth1

    inet 192.168.2.200/24 brd 192.168.2.255 scope global secondary eth1

测试方法：

1.停掉master上的mysqld，看看是否切换(因为heartheat不检查服务的可用性，因此需要通过而外的脚本来实现)。

2.停掉master的heartheat看看是否能正常切换。 

3.停掉master的网络或者直接将master系统shutdown，看看能否正常切换。 

4.启动master的heartbeat看看是否能正常切换回来。 

5.重新启动master看看能否切换过程是否OK。 

注意：这里说的切换是不是已经将mysql停掉、是否卸载了文件系统等等。

停止master（drbd1）上的heartbeat来测试是否会自动切换，这里除了第一条无法实现，其他的都可以切换：

[root@drbd1 ~]# /etc/init.d/heartbeat stop

Stopping High-Availability services: Done.

[root@drbd1 ~]# df –HT

Filesystem           Type   Size   Used   Avail    Use%    Mounted on

/dev/mapper/VolGroup-lv_root

                     ext4   7.0G   4.4G    2.3G     66%        /

tmpfs                tmpfs  249M     0  249M     0%         /dev/shm

/dev/sda1            ext4   500M   29M  445M    6%         /boot

[root@drbd1 ~]# service drbd status

drbd driver loaded OK; device status:

version: 8.4.4 (api:1/proto:86-101)

GIT-hash: 74402fecf24da8e5438171ee8c19e28627e1c98a build by root@drbd1, 2015-12-29 16:21:36

m:res  cs              ro                 ds                 p  mounted  fstype

0:r0   Connected  Secondary/Primary  UpToDate/UpToDate  C

查看drbd2从服务器上是否切换过来了：

[root@drbd2 ~]# service drbd status

drbd driver loaded OK; device status:

version: 8.4.4 (api:1/proto:86-101)

GIT-hash: 74402fecf24da8e5438171ee8c19e28627e1c98a build by root@dbrd2, 2015-12-29 16:14:14

m:res  cs              ro                 ds                 p  mounted  fstype

0:r0   Connected  Primary/Secondary  UpToDate/UpToDate  C  /drbd    ext4

[root@drbd2 ~]# df -HT

Filesystem           Type   Size  Used   Avail  Use%  Mounted on

/dev/mapper/VolGroup-lv_root

                     ext4   7.0G  4.4G   2.3G   66%      /

tmpfs                tmpfs  249M     0  249M   0%  /dev/shm

/dev/sda1            ext4   500M   29M  445M   6% /boot

/dev/drbd0           ext4    21G  162M   20G   1% /drbd

[root@drbd2 ~]# ip addr

eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000

    link/ether 00:0c:29:71:c8:7e brd ff:ff:ff:ff:ff:ff

    inet 192.168.2.204/24 brd 192.168.2.255 scope global eth1

    inet 192.168.2.200/24 brd 192.168.2.255 scope global secondary eth1

    inet6 fe80::20c:29ff:fe71:c87e/64 scope link 

       valid_lft forever preferred_lft forever