Twemproxy+Keepalived+Redis

Twemproxy+Keepalived+Redis

方网站:
http://www.redis.io/
http://www.redis.cn
https://github.com/twitter/twemproxy
http://www.keepalived.org/
https://github.com/LittlePeng/redis-monitor
https://github.com/kontera-technologies/nutcracker-web
参考文档：
https://github.com/twitter/twemproxy/blob/master/notes/recommendation.md#liveness
http://www.cnblogs.com/lulu/archive/2013/06/10/3130878.html
http://os.51cto.com/art/201408/447247_all.htm
http://www.oschina.net/translate/twemproxy-a-twitter-redis-proxy
http://www.cnblogs.com/haoxinyue/p/redis.html
http://www.oschina.net/question/861681_135017?sort=time&p=1
http://wenku.baidu.com/link?url=xZ8niKIvxQY1QUPFMWh4Jry58MLuzyXeMwVfBoT8OcvMNuWxNlAh_ajf5e5V14rLq5ma33cBONOfML4I8mYFQcG5Otry9GlJ2p_Zxmf_-Fa

实验环境

OS:CentOS 6.5 x64

软件版本：

nutcracker-0.4.0

keepalived-1.2.13

redis-2.8.18

主机：

twemproxy VIP(keepalived): 192.168.8.100

twemproxy_01+keepalive-master: 192.168.8.80,192.168.7.70

twemproxy_02+keepalive-backup: 192.168.8.81,192.168.7.71

redis_01: 192.168.7.100

redis_02: 192.168.7.200

目录

Redis
一.安装编译依赖库
二.编译安装
三.配置redis sysV管控脚本

Twemproxy
一.安装编译依赖库
二.升级autoconf
三.编译安装
四.nutcracker sysV init脚本
五.配置文件
六.启服务

Keepalived
一.安装编译依赖库
二.编译安装
三. 配置keepalived sysV init脚本
四.配置keepalived
1.twemproxy健康检查脚本twemproxy_check
2.主配置文件/etc/keepalived/keepalived.conf

测试
一.测试nutcracker是否生效
二.在twemproxy_01上停掉nutcracker
三.直接停止keepalived Master
四.keepalived MASTER恢复启动

Redis

一.安装编译依赖库

[root@redis_01 ~]# yum -y install gcc gcc-c++ make tcl-devel

 

二.编译安装

[root@redis_01 ~]# tar -xvf redis-2.8.18.tar.gz -C /usr/local/src/

[root@redis_01 ~]# cd /usr/local/src/redis-2.8.18/

[root@redis_01 redis-2.8.18]# make && make install

 

三.配置redis sysV管控脚本

[root@redis_01 redis-2.8.18]# cd utils/

[root@redis_01 utils]# ls

build-static-symbols.tcl  mkrelease.sh           redis-sha1.rb

generate-command-help.rb  redis-copy.rb          speed-regression.tcl

hyperloglog               redis_init_script      whatisdoing.sh

install_server.sh         redis_init_script.tpl

[root@redis_01 utils]# ./install_server.sh

Welcome to the redis service installer

This script will help you easily set up a running redis server

 

Please select the redis port for this instance: [6379]

Selecting default: 6379

Please select the redis config file name [/etc/redis/6379.conf]

Selected default - /etc/redis/6379.conf

Please select the redis log file name [/var/log/redis_6379.log]

Selected default - /var/log/redis_6379.log

Please select the data directory for this instance [/var/lib/redis/6379]

Selected default - /var/lib/redis/6379

Please select the redis executable path [/usr/local/bin/redis-server]

Selected config:

Port           :6379

Config file    : /etc/redis/6379.conf

Log file       :/var/log/redis_6379.log

Data dir       : /var/lib/redis/6379

Executable     : /usr/local/bin/redis-server

Cli Executable : /usr/local/bin/redis-cli

Is this ok? Then press ENTER to go on or Ctrl-C to abort.

Copied /tmp/6379.conf => /etc/init.d/redis_6379

Installing service...

Successfully added to chkconfig!

Successfully added to runlevels 345!

Starting Redis server...

Installation successful!

默认的话，一路回车即可。init脚本会自动生成

[root@redis_01 utils]# mv /etc/init.d/redis_6379 /etc/init.d/redis

[root@redis_01 utils]# /etc/init.d/redis status

Redis is running (4796)

[root@redis_01 utils]# /etc/init.d/redis restart

Stopping ...

Redis stopped

Starting Redis server...

[root@redis_01 utils]# chkconfig --add redis

[root@redis_01 utils]# chkconfig redis on

[root@redis_01 utils]# chkconfig --list redis

redis_6379         0:off    1:off    2:on    3:on    4:on    5:on    6:off

[root@redis_01 utils]# netstat -tunlp|grep redis

tcp        0      0 0.0.0.0:6379                0.0.0.0:*                   LISTEN      4812/redis-server *

可以看到，redis己成功运行并监听在6379端口上。

 

对于redis_02，作为redis_01的Slave需要在redis配置文件中指定Master主机

slaveof 192.168.8.81 6379

 

补充：

可以置客户端访问密码
Master配置文件中加入
requirepass test123

Slave配置文件中加入
requirepass test123
masterauth test123

 

Twemproxy

Twitter开源的c版本proxy，用来代理后端的redis和memcached

一.安装编译依赖库

[root@twemproxy_01 ~]#wget https://github.com/twitter/twemproxy/archive/master.zip -O twemproxy-master.zip

[root@twemproxy_01 ~]#yum -y install gcc gcc-c++ autoconf automake libtool

 

二.升级autoconf

[root@twemproxy_01 ~]# rpm -e autoconf --nodeps

twemproxy要求autoconf>=2.64，而CentOS6.5 x64默认自带的为2.63，所以必需更新。

[root@twemproxy_01 ~]# wget http://ftp.gnu.org/gnu/autoconf/autoconf-2.69.tar.xz

[root@twemproxy_01 ~]# tar -xvf autoconf-2.69.tar.xz -C /usr/local/src/

[root@twemproxy_01 ~]# cd /usr/local/src/autoconf-2.69/

root@jun-live:autoconf-2.69#./configure && make && make install

[root@twemproxy_01 ~]# ln -s /usr/local/bin/autoconf /usr/bin/

[root@twemproxy_01 ~]# ln -s /usr/local/bin/autoreconf /usr/bin/

[root@twemproxy_01 ~]# autoconf --version

autoconf (GNU Autoconf) 2.69

Copyright (C) 2012 Free Software Foundation, Inc.

License GPLv3+/Autoconf: GNU GPL version 3 or later

,

This is free software: you are free to change and redistribute it.

There is NO WARRANTY, to the extent permitted by law.

 

Written by David J. MacKenzie and Akim Demaille.

 

三.编译安装

[root@twemproxy_01 ~]# cd /usr/local/src/

[root@twemproxy_01 src]# unzip /root/twemproxy-master.zip

[root@twemproxy_01 ~]# cd /usr/local/src/twemproxy-master

[root@twemproxy_01 twemproxy-master]# autoreconf -fvi

[root@twemproxy_01 twemproxy-master]# ./configure --enable-debug=log && make && make install

 

四.nutcracker sysV init脚本

[root@twemproxy_01 twemproxy-master]# cd /usr/local/src/twemproxy-master/scripts

[root@twemproxy_01 scripts]# cp nutcracker.init /etc/init.d/nutcracker

[root@twemproxy_01 scripts]# chmod 755 /etc/init.d/nutcracker

[root@twemproxy_01 ~]# chkconfig --add nutcracker

[root@twemproxy_01 ~]# chkconfig nutcracker on

[root@twemproxy_01 ~]# chkconfig --list nutcracker

nutcracker     0:off      1:off      2:on       3:on       4:on       5:on       6:off

 

五.配置文件

[root@twemproxy_01 twemproxy-master]# cp conf/nutcracker.yml /etc/nutcracker

[root@twemproxy_01 ~]# cd /etc/nutcracker/

[root@twemproxy_01 nutcracker]# vim nutcracker.yml

alpha:

  listen: 0.0.0.0:22121

  hash: fnv1a_64

  hash_tag: "{}"

  distribution: ketama

  auto_eject_hosts: true

  timeout: 400

  redis: true

  server_retry_timeout: 300000

  server_failure_limit: 3

  servers:

   - 192.168.7.100:6379:1 server1

   - 192.168.7.200:6379:1 server2

可以将配置文件copy到twemproxy_02上即可

说明：因为client最终通过VIP来访问，所以这里nutcracker监听所有接口

 

六.启服务

指定日志文件位置，方便排错

[root@twemproxy_01 scripts]# vim /etc/init.d/nutcracker
OPTIONS="-d -c /etc/nutcracker/nutcracker.yml -o /var/log/twemproxy/nutcracker.log"

[root@twemproxy_01 scripts]# mkdir /var/log/twemproxy
[root@twemproxy_01 scripts]# touch /var/log/twemproxy/nutcracker.log
[root@twemproxy_01 scripts]# chmod 777 /var/log/twemproxy/nutcracker.log
[root@twemproxy_01 scripts]# /etc/init.d/nutcracker start
Starting nutcracker:                                       [  OK  ]
[root@twemproxy_01 nutcracker]# tail -f /var/log/twemproxy/nutcracker.log
[2015-01-04 13:56:50.838] nc.c:187 nutcracker-0.4.0 built for Linux 2.6.32-431.el6.x86_64 x86_64 started on pid 28015
[2015-01-04 13:56:50.842] nc.c:192 run, rabbit run / dig that hole, forget the sun / and when at last the work is done / don't sit down / it's time to dig another one
^C

[root@twemproxy_01 ~]# /etc/init.d/nutcracker restart

Stopping nutcracker: [  OK  ]

Starting nutcracker: [  OK  ]

[root@twemproxy_01 ~]# netstat -tunlp|grep nutcracker

tcp        0      0 0.0.0.0:22222               0.0.0.0:*                   LISTEN      2505/nutcracker    

tcp        0      0 192.168.8.80:22121          0.0.0.0:*                   LISTEN      2505/nutcracker   

如果大批量部署，可以编译rpm包

[root@twemproxy_01 ~]# yum -y install rpm-build

[root@twemproxy_01 ~]# unzip twemproxy-master.zip

[root@twemproxy_01 ~]# cp twemproxy-master/scripts/nutcracker.spec .

[root@twemproxy_01 ~]# mv twemproxy-master nutcracker-0.4.0

[root@twemproxy_01 SOURCES]# tar -zcvf nutcracker-0.4.0.tar.gz nutcracker-0.4.0/

[root@twemproxy_01 ~]# ls

nutcracker-0.4.0    nutcracker.spec          twemproxy-master.zip

必要时可以修改nutcracker.spec以解决依赖关系等

Version: 0.4.0

BuildRequires: autoconf

[root@twemproxy_01 ~]# rpmbuild -ba nutcracker.spec

error: File /root/rpmbuild/SOURCES/nutcracker-0.4.0.tar.gz: No such file or directory

[root@twemproxy_01 ~]# cp nutcracker-0.4.0.tar.gz rpmbuild/SOURCES/

[root@twemproxy_01 ~]# rpmbuild -ba nutcracker.spec

[root@twemproxy_01 ~]# rpm -ivh rpmbuild/RPMS/x86_64/nutcracker-0.4.0-1.x86_64.rpm

Preparing...                ########################################### [100%]

   1:nutcracker             ############################serial8250: too much work for irq4

########################################### [100%]

 

Keepalived

一.安装编译依赖库

[root@twemproxy_01 ~]#yum –y install kernel-devel gcc openssl-devel popt-devel libnl-devel net-snmp-devel

 

二.编译安装

[root@twemproxy_01 ~]#tar -xvf keepalived-1.2.13.tar.gz -C /usr/local/src

[root@twemproxy_01 ~]#cd /usr/local/src/keepalived-1.2.13

[root@twemproxy_01 ~]#./configure

--prefix=/usr

--sysconf=/etc

--with-kernel-dir=/usr/src/kernels/$(uname -r)

--enable-snmp

--enable-sha1

标红的部分一定要有，否则不能编译成功

[root@twemproxy_01 ~]#make && make install

 

三. 配置keepalived sysV init脚本

[root@twemproxy_01 ~]#cp /usr/local/src/keepalived-1.2.13/keepalived/etc/init.d/keepalived.init /etc/rc.d/rc3.d/S99keepalived

[root@twemproxy_01 ~]# cp /usr/local/src/keepalived-1.2.13/keepalived/etc/init.d/keepalived.init /etc/rc.d/rc5.d/S99keepalived

[root@twemproxy_01 ~]#chkconfig keepalived on

[root@twemproxy_01 ~]# chkconfig --list keepalived

keepalived 0:off 1:off 2:on 3:on 4:on 5:on 6:off

 

四.配置keepalived

1.twemproxy健康检查脚本twemproxy_check

[root@twemproxy_01 keepalived]# pwd

/etc/keepalived

[root@twemproxy_01 keepalived]# mkdir scripts

[root@twemproxy_01 keepalived]# vim scripts/twemproxy_check.sh

[root@twemproxy_01 keepalived]# chmod +x scripts/twemproxy_check.sh

############################################

#!/bin/bash

#To install check twemproxy status automatically

#Made by liujun,2014/12/31

############################################

#Define system environment PATH

export PATH=$PATH

flag=$(pidof nutcracker)

if [ "$flag" == "" ];then

        /etc/init.d/nutcracker start

        sleep 3

        flag_wait=$(pidof nutcracker)

        if [ "$flag_wait" == "" ];then

        /etc/init.d/keepalived stop

        fi

fi

2.主配置文件/etc/keepalived/keepalived.conf

[root@twemproxy_01 keepalived]# vim keepalived.conf

! Configuration File for keepalived

vrrp_script twemproxy_check {

    script "/etc/keepalived/scripts/twemproxy_check.sh"

    interval 2

    weight 2

}

vrrp_instance VI_1 {

    state MASTER

    interface eth0

    virtual_router_id 51

    priority 100

    advert_int 1

    authentication {

        auth_type PASS

        auth_pass 1111

    }

    virtual_ipaddress {

        192.168.8.100

    }

    track_script {

        twemproxy_check    }

}

[root@twemproxy_01 keepalived]# /etc/init.d/keepalived restart

Stopping keepalived:                                       [  OK  ]

Starting keepalived:                                       [  OK  ]

搞定，己经可以通过VIP192.168.8.100来访问后端的redis了，现在的VIP在twemproxy_01上面

[root@twemproxy_01 keepalived]# ip addr

1: lo: mtu 16436 qdisc noqueue state UNKNOWN

    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

    inet 127.0.0.1/8 scope host lo

2: eth0: mtu 1500 qdisc pfifo_fast state UP qlen 1000

    link/ether 52:54:00:01:00:01 brd ff:ff:ff:ff:ff:ff

    inet 192.168.8.80/24 brd 192.168.8.255 scope global eth0

    inet 192.168.8.100/32 scope global eth0

3: eth1: mtu 1500 qdisc pfifo_fast state UP qlen 1000

    link/ether 52:54:00:02:00:01 brd ff:ff:ff:ff:ff:ff

    inet 192.168.7.70/24 brd 192.168.7.255 scope global eth1

再配置twemproxy_02，直接将keepalived.conf和twemproxy_check.sh脚本copy到twemproxy_02对应的目录，修改keepalived.conf中的以下两行，使twemproxy_02作为twemproxy_01的BACKUP

state BACKUP

priority 50

 

 

测试

一.测试nutcracker是否生效

twemproxy_01

root@jun-live:~#redis-cli  -h 192.168.8.80 -p 22121
192.168.8.80:22121> help
redis-cli 2.8.18
Type: "help @" to get a list of commands in
      "help " for help on
      "help " to get a list of possible help topics
      "quit" to exit
192.168.8.80:22121> quit

twemproxy_02

root@jun-live:~#redis-cli  -h 192.168.8.81 -p 22121
192.168.8.81:22121> help
redis-cli 2.8.18
Type: "help @" to get a list of commands in
      "help " for help on
      "help " to get a list of possible help topics
      "quit" to exit
192.168.8.81:22121> quit

VIP
root@jun-live:~#redis-cli  -h 192.168.8.100 -p 22121
192.168.8.100:22121> help
redis-cli 2.8.18
Type: "help @" to get a list of commands in
      "help " for help on
      "help " to get a list of possible help topics
      "quit" to exit
192.168.8.100:22121> quit

二.在twemproxy_01上停掉nutcracker

[root@twemproxy_01 ~]# /etc/init.d/nutcracker stop

Stopping nutcracker: [  OK  ]

[root@twemproxy_01 ~]# /etc/init.d/nutcracker status

nutcracker (pid 5448) is running...

发现，健康检查脚本会每隔2秒钟检查一次，如果nutcracker没有进程号则通过管控脚本启动，再等3秒钟，如果还是没有启动，则停止keepalived让BACKUP来接管。说明keepalive在持续监控nutcracker的健康状态

 

三.直接停止keepalived Master

[root@twemproxy_01 ~]# /etc/init.d/keepalived stop

Stopping keepalived: [  OK  ]

[root@twemproxy_01 ~]# tail -f /var/log/messages

Dec 31 09:25:59 twemproxy_01 Keepalived[1144]: Stopping Keepalived v1.2.13 (12/20,2014)

Dec 31 09:25:59 twemproxy_01 Keepalived_vrrp[1146]: VRRP_Instance(VI_1) sending 0 priority

Dec 31 09:25:59 twemproxy_01 Keepalived_vrrp[1146]: VRRP_Instance(VI_1) removing protocol VIPs.

可以看到，VIP从MASTER节点上被移除，而飘到了BACKUP节点

[root@twemproxy_02 ~]# tail -f /var/log/messages

Dec 31 09:25:59 twemproxy_02 Keepalived_vrrp[1153]: VRRP_Instance(VI_1) Transition to MASTER STATE

Dec 31 09:26:00 twemproxy_02 Keepalived_vrrp[1153]: VRRP_Instance(VI_1) Entering MASTER STATE

Dec 31 09:26:00 twemproxy_02 Keepalived_vrrp[1153]: VRRP_Instance(VI_1) setting protocol VIPs.

Dec 31 09:26:00 twemproxy_02 Keepalived_healthcheckers[1152]: Netlink reflector reports IP 192.168.8.100 added

Dec 31 09:26:00 twemproxy_02 Keepalived_vrrp[1153]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.8.100

Dec 31 09:26:05 twemproxy_02 Keepalived_vrrp[1153]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.8.100

[root@twemproxy_02 ~]# ip addr

1: lo: mtu 16436 qdisc noqueue state UNKNOWN

    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

    inet 127.0.0.1/8 scope host lo

2: eth1: mtu 1500 qdisc pfifo_fast state UP qlen 1000

    link/ether 52:54:00:01:00:02 brd ff:ff:ff:ff:ff:ff

    inet 192.168.7.71/24 brd 192.168.7.255 scope global eth1

3: eth0: mtu 1500 qdisc pfifo_fast state UP qlen 1000

    link/ether 52:54:00:02:00:02 brd ff:ff:ff:ff:ff:ff

    inet 192.168.8.81/24 brd 192.168.8.255 scope global eth0

    inet 192.168.8.100/32 scope global eth0

 

四.keepalived MASTER恢复启动

[root@twemproxy_01 ~]# /etc/init.d/keepalived start

Starting keepalived: [  OK  ]

Dec 31 09:39:15 twemproxy_01 Keepalived[6048]: Starting Keepalived v1.2.13 (12/20,2014)

Dec 31 09:39:15 twemproxy_01 Keepalived[6049]: Starting Healthcheck child process, pid=6050

Dec 31 09:39:15 twemproxy_01 Keepalived[6049]: Starting VRRP child process, pid=6052

Dec 31 09:39:15 twemproxy_01 Keepalived_vrrp[6052]: Netlink reflector reports IP 192.168.8.80 added

Dec 31 09:39:15 twemproxy_01 Keepalived_vrrp[6052]: Netlink reflector reports IP 192.168.7.70 added

Dec 31 09:39:15 twemproxy_01 Keepalived_vrrp[6052]: Netlink reflector reports IP 192.168.8.80 added

Dec 31 09:39:15 twemproxy_01 Keepalived_vrrp[6052]: Netlink reflector reports IP 192.168.7.70 added

Dec 31 09:39:15 twemproxy_01 Keepalived_vrrp[6052]: Registering Kernel netlink reflector

Dec 31 09:39:15 twemproxy_01 Keepalived_vrrp[6052]: Registering Kernel netlink command channel

Dec 31 09:39:15 twemproxy_01 Keepalived_vrrp[6052]: Registering gratuitous ARP shared channel

Dec 31 09:39:15 twemproxy_01 Keepalived_healthcheckers[6050]: Netlink reflector reports IP 192.168.8.80 added

Dec 31 09:39:15 twemproxy_01 Keepalived_healthcheckers[6050]: Netlink reflector reports IP 192.168.7.70 added

Dec 31 09:39:15 twemproxy_01 Keepalived_healthcheckers[6050]: Netlink reflector reports IP 192.168.8.80 added

Dec 31 09:39:15 twemproxy_01 Keepalived_healthcheckers[6050]: Netlink reflector reports IP 192.168.7.70 added

Dec 31 09:39:15 twemproxy_01 Keepalived_healthcheckers[6050]: Registering Kernel netlink reflector

Dec 31 09:39:15 twemproxy_01 Keepalived_healthcheckers[6050]: Registering Kernel netlink command channel

Dec 31 09:39:15 twemproxy_01 Keepalived_vrrp[6052]: Opening file '/etc/keepalived/keepalived.conf'.

Dec 31 09:39:15 twemproxy_01 Keepalived_vrrp[6052]: Configuration is using : 63284 Bytes

Dec 31 09:39:15 twemproxy_01 Keepalived_vrrp[6052]: Using LinkWatch kernel netlink reflector...

Dec 31 09:39:15 twemproxy_01 Keepalived_vrrp[6052]: VRRP sockpool: [ifindex(2), proto(112), unicast(0), fd(10,11)]

Dec 31 09:39:15 twemproxy_01 Keepalived_vrrp[6052]: VRRP_Script(twemproxy_check) succeeded

Dec 31 09:39:15 twemproxy_01 Keepalived_vrrp[6052]: VRRP_Instance(VI_1) Transition to MASTER STATE

Dec 31 09:39:15 twemproxy_01 Keepalived_vrrp[6052]: VRRP_Instance(VI_1) Received lower prio advert, forcing new election

Dec 31 09:39:16 twemproxy_01 Keepalived_vrrp[6052]: VRRP_Instance(VI_1) Entering MASTER STATE

Dec 31 09:39:16 twemproxy_01 Keepalived_vrrp[6052]: VRRP_Instance(VI_1) setting protocol VIPs.

Dec 31 09:39:16 twemproxy_01 Keepalived_vrrp[6052]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.8.100

Dec 31 09:39:20 twemproxy_01 Keepalived_healthcheckers[6050]: Opening file '/etc/keepalived/keepalived.conf'.

Dec 31 09:39:20 twemproxy_01 Keepalived_healthcheckers[6050]: Configuration is using : 5568 Bytes

Dec 31 09:39:20 twemproxy_01 Keepalived_healthcheckers[6050]: Using LinkWatch kernel netlink reflector...

Dec 31 09:39:20 twemproxy_01 Keepalived_healthcheckers[6050]: Netlink reflector reports IP 192.168.8.100 added

Dec 31 09:39:21 twemproxy_01 Keepalived_vrrp[6052]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.8.100

因为优先级比BACKUP高，所以VIP又飘了回来。

 

ok，实验结束。