利用gearman同步mysql数据到redis

一.Gearman

1.Gearman是一个分发任务的程序框架.

2.体系:
a.client:发送一个job
b.server:找到合适的worker,把job交给该worker
c.worker:处理job

二.安装Gearmand服务

1.Gearman

a.官网
http://gearman.org/

b.官网下载:
https://launchpad.net/gearmand

2.安装依赖库

# yum install -y boost boost-devel gperf libevent-devel libuuid-devel

　　

ps:要求boost版本至少大于1.39,系统之前安装的是1.3;所以需要升级

3.升级boost到1.55.0

a.下载boost_1_55_0.tar.bz2

http://downloads.sourceforge.net/project/boost/boost/1.55.0/boost_1_55_0.tar.bz2?r=http%3A%2F%2Fsourceforge.net%2Fprojects%2Fboost%2Ffiles%2Fboost%2F1.55.0%2F&ts=1385953406&use_mirror=softlayer-ams

b.解压

c. 安装

./bootstrap.sh
./b2 
./b2 install

　　

4.安装Gearman

a.解压
b.编译

./configure --prefix=/usr/local/gearmand --with-boost=/usr/local/boost/ --with-boost-libdir=/usr/local/boost/lib

　　

c.make

出现错误:error: cinttypes: No such file or directory

安装:gcc44等

yum install gcc44 gcc44-c++ libstdc++44-devel -y

　　

在/etc/profile中加入
export CC=/usr/bin/gcc44 or export CC=/usr/bin/gcc
export CXX=/usr/bin/g++44

#source /etc/profile

　　

d.再次make
报错:

error: 'SQLITE_OPEN_READWRITE' was not declared in this scope

　　

e.安装新的sqlite

#wget http://www.sqlite.org/2014/sqlite-autoconf-3080704.tar.gz
#tar zxvf sqlite-autoconf-3080704.tar.gz
#cd sqlite-autoconf-3080704
#./configure --prefix=/usr/local/sqlite
#make && make install

　　

f.再次配置编译安装

#./configure --prefix=/usr/local/gearmand --with-boost=/usr/local/boost/   --with-boost-libdir=/usr/local/boost/lib   --with-sqlite3=/usr/local/sqlite
#make

　　

出现错误undefined reference to `sqlite3_prepare_v2'

#ln -s /usr/local/sqlite/lib/libsqlite3.so.0.8.6 /usr/lib64

*如果/usr/lib64下存在libsqlite3.so.0.8.6,将其重命名

三.启动Gearman

1.

/usr/local/gearmand/sbin/gearmand -d -u root -L 127.0.0.1 --log-file=/var/log/gearmand.log

　　

报错:
error while loading shared libraries: libboost_program_options.so.1.55.0: cannot open shared object file: No such file or directory

#vi /etc/ld.so.conf.d/gearman.conf

　　

插入以下内容
/usr/local/boost/lib/
/usr/local/gearman/lib/

执行

#ldconfig

　　

2.参数详解
-b，--backlog= 储备的监听连接数量
-d, --daemon 后台运行
-f, --file-descriptors= 文件描述符的数量
-h, --help 帮助
-j, --job-retries= 在ob server移除不可用job之前运行的次数，防止不断运行导致其他可用worker崩溃。默认没有限制
-l, -log-file= 日志文件存放位置(默认记录最简单日志)
-L, --listen= 监听的IP，默认全部接受
-p, --port= 指定监听端口
-P, --pid-file= 指定进程ID写入位置
-r, --protocol= 加载协议模块
-q, --queue-type= 指定持久化队列
-t, --threads= 使用的I/9线程数量。默认为0
-u, --user= 启动后，切换到指定用户
-v, --verbose 增加一级详细程度
-V, --version 显示版本信息

3.监控
#watch -n 1 "(echo status; sleep 0.1) | nc 127.0.0.1 4730"

四.安装php-gearman

1.wget http://pecl.php.NET/get/gearman-1.1.1.tgz
2.解压
3.#/usr/local/php/bin/phpize
4.#./configure --with-gearman=/usr/local/gearmand/ --with-php-config=/usr/local/php/bin/php-config
5.make && make install
6.在php.ini中加入
extension=gearman.so
7.用#php -i | grep gearman或者php -m | grep gearman来验证

五.安装lib_mysqludf_json 和 gearman-mysql-udf

1.安装lib_mysqludf_json

wget  https://github.com/mysqludf/lib_mysqludf_json/archive/master.zip
mv master master.zip
unzip master.zip
cd lib_mysqludf_json-master
rm -rf lib_mysqludf_json.so 
gcc $(/usr/local/mysql56-server/bin/mysql_config --cflags) -shared -fPIC -o lib_mysqludf_json.so lib_mysqludf_json.c

　　

2.拷贝lib_mysqludf_json.so到mysql的plugin目录下

3.安装gearman-mysql-udf

wget https://launchpad.net/gearman-mysql-udf/trunk/0.6/+download/gearman-mysql-udf-0.6.tar.gz   
tar xf gearman-mysql-udf-0.6.tar.gz -C ./
cd gearman-mysql-udf-0.6
./configure --with-mysql=/usr/local/mysql56-server/bin/mysql_config --libdir=/usr/local/mysql56-server/lib/plugin/ 
make && make install

　　

报错:
configure: error: At least version 0.33 of libgearman is required for gearman-mysql-udf

找不到libgearman,指定路径,加入--with-libgearman-prefix

./configure --with-mysql=/usr/local/mysql56-server/bin/mysql_config --libdir=/usr/local/mysql56-server/lib/plugin/ --with-libgearman-prefix=/usr/local/gearmand

　　

六.创建function

mysql> CREATE FUNCTION json_object RETURNS STRING SONAME 'lib_mysqludf_json.so';
mysql> CREATE FUNCTION gman_do_background RETURNS STRING SONAME 'libgearman_mysql_udf.so';
mysql> CREATE FUNCTION gman_servers_set RETURNS STRING SONAME 'libgearman_mysql_udf.so';

　　

七.在mysql中设置Gearman Server

1.

SELECT gman_servers_set('127.0.0.1:4730');

　　

2.建立一个触发器(更新记录时将数据更新到redis)

CREATE TRIGGER `test_mysql_redis` AFTER UPDATE ON `test` FOR EACH ROW SET @ret=gman_do_background('syncToRedis', json_object(NEW.id as `id`, NEW.name as `name`));

　　

八.查看job及更新数据到redis

1.

watch -n 1 "(echo status; sleep 0.1) | nc 127.0.0.1 4730"

　　

syncToRedis 1 0 0

syncToRedis后面三列的意思是:
1个正在运行的job, 队列为空, 0个可用的woker

2.编写worker, gearman_workder.php

<?php
    $worker = new GearmanWorker();
    $worker->addServer();
    $worker->addFunction('syncToRedis', 'syncToRedis');

    $redis = new Redis();
    $redis->connect('127.0.0.1', '6379');
    $redis->select(9);

    while($worker->work());

    function syncToRedis($job)
    {
        global $redis;
        $workString = $job->workload();
        $work = json_decode($workString);
        if(!isset($work->id)){
           return false;
        }
        $redis->set($work->id, $workString);
    }

　　

然后运行gearman_workder.php,去redis中验证,看数据是否有同步.