zoukankan      html  css  js  c++  java
  • CentOS的Gearman安装

    背景:用PHP做一些简单的上传是没有任何的问题,但是要做断点上传好像也是没有大问题,但要是并发的切片加断点上传可能就会有问题了哟。
    第一个问题是合并问题:如果一上传就合并,PHP老半天不返回是一个方面(是PHP超时还是Nginx超时?),这样势必会造成客户端因没返回,没法再启动新的上传进程,如果用php的追加写文件内存太大还会退出的可能,用上gearmand后出现异步同时写,可能加快了传输效率的同时IO可能会比较密集(假如同一时间并发上传的人多的情况。)。
    第二个问题是先后问题:如果并发时出现了传一半时最后一片先上传上来了怎么办?如果直接追加进文件是会不按顺序出错的,如果放在那儿不管,那啥时候管,这也是一个问题。
    第三个问题是锁的问题:并发时如果同时两片到了,像PHP这种如何去结合锁的功能呢?据我所知有memcache的add函数是原子操作,可以利用,那就设计到锁多久,是不是又回到第一个问题,返回时间得浪费多长时间呢。
    基于上面三点,看来PHP做这活不是太靠谱,于是引出了gearman这个模型,相当于异步处理,再集合PHP的memcache锁(memcache放内存,可能得用ttserver才行,协议一样的。),PHP上传逻辑处理,Nginx上传插件,才有可能做一个较为靠谱且稳定的分片,断点加多进程(多线程)上传服务搭建。

    一、从源码安装gearmand遇到的各种外部代码版本及yum版本太低的导致各种编译不过的问题,历程相当的麻烦,特别是boost和这个libevent默认的yum install非常低,都给编译了,把rpm所强制删除了,最后才成功,再就是g++编译版本,得export后才能编译,否则一堆问题,都在下面有描写。
    出现第一个问题及处理办法:
    libgearman-server/plugins/queue/sqlite/instance.cc: In member function 'bool gearmand::queue::Instance::_sqlite_prepare(const std::string&, sqlite3_stmt**)':
    libgearman-server/plugins/queue/sqlite/instance.cc:125: error: 'sqlite3_prepare_v2' was not declared in this scope
    libgearman-server/plugins/queue/sqlite/instance.cc: In member function 'gearmand_error_t gearmand::queue::Instance::init()':
    libgearman-server/plugins/queue/sqlite/instance.cc:224: error: 'SQLITE_OPEN_READWRITE' was not declared in this scope
    libgearman-server/plugins/queue/sqlite/instance.cc:224: error: 'SQLITE_OPEN_CREATE' was not declared in this scope
    libgearman-server/plugins/queue/sqlite/instance.cc:224: error: 'sqlite3_open_v2' was not declared in this scope
    make[1]: *** [libgearman-server/plugins/queue/sqlite/libgearman_server_libgearman_server_la-instance.lo] Error 1
    将队列存放在sqlite3或postgresql。这货是用的sqlite存队列啊:
    yum search sqlite
    yum install sqlite.x86_64 sqlite-devel.x86_64
    还是不行,参考:
    安装:https://justwinit.cn/post/7709/ 后,
    最后在编译Gearman时带上 --with-sqlite3=/usr/local/sqlite3,告诉编译器应该使用这个新的sqlite即可。


    出现第二个问题及处理办法:
    ./libgearman-1.0/gearman.h:53:23: error: cinttypes: No such file or directory
    命令:
    yum install gcc44 gcc44-c++ libstdc++44-devel -y
    然后在环境变量里加入:
    export CC=/usr/bin/gcc44 or export CC=/usr/bin/gcc
    export CXX=/usr/bin/g++44
    保存退出后执行:
    source /etc/profile
    删除gearmand-0.34文件夹重新进行编译.
    重新进行编译后执行make这步......
    在后面有详细的说明,可以不source直接设置环境变量,因为编译后也不一定要这个版本的gcc的。



    出现第三个问题的处理办法:
    ibgearman-server/plugins/queue/mysql/queue.cc:430: error: 'mysql_error' was not declared in this scope
    libgearman-server/plugins/queue/mysql/queue.cc: In function 'gearmand_error_t _mysql_queue_replay(gearman_server_st*, void*, gearmand_error_t (*)(gearman_server_st*, void*, const char*, size_t, const char*, size_t, const void*, size_t, gearman_job_priority_t, int64_t), void*)':
    libgearman-server/plugins/queue/mysql/queue.cc:446: error: 'MYSQL_RES' was not declared in this scope
    libgearman-server/plugins/queue/mysql/queue.cc:446: error: 'result' was not declared in this scope
    libgearman-server/plugins/queue/mysql/queue.cc:447: error: 'MYSQL_ROW' was not declared in this scope

    [root@test gearmand-1.1.12]# ./configure --help|grep mysql
      --with-mysql=[ARG]      use MySQL client library [default=yes], optionally
                              specify path to mysql_config
                  Full path to mysql_config program

    看样子是想用mysql做队列queue的:


    加上还是报错,去了得了,不用mysql做队列,有sqlite足够了,
    --without-mysql就好了,来自:
    https://bugs.launchpad.net/gearmand/+bug/1327038 说的:


    出现第四个问题的解决办法:
    /home/xiangdong/software/gearmand-1.1.12/bin/gearadmin.cc:129: undefined reference to `boost::program_options::options_description::m_default_line_length'
    /home/xiangdong/software/gearmand-1.1.12/bin/gearadmin.cc:129: undefined reference to `boost::program_options::options_description::options_description(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int, unsigned int)'
    bin/gearadmin.o: In function `boost::program_options::basic_command_line_parser<char>::run()':
    /usr/local/include/boost/program_options/detail/parsers.hpp:107: undefined reference to `boost::program_options::detail::cmdline::get_canonical_option_prefix()'
    bin/gearadmin.o: In function `boost::program_options::basic_command_line_parser<char>::extra_parser(boost::function1<std::pair<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>)':
    /usr/local/include/boost/program_options/detail/parsers.hpp:77: undefined reference to `boost::program_options::detail::cmdline::set_additional_parser(boost::function1<std::pair<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>)'
    collect2: ld returned 1 exit status
    make[1]: *** [bin/gearadmin] Error 1
    make[1]: Leaving directory `/home/xiangdong/software/gearmand-1.1.12'


    [root@test gearmand-1.1.12]# ./configure --help|grep boost
      --with-boost[=ARG]      use Boost library from a standard location
      --with-boost-libdir=LIB_DIR
                              Force given directory for boost libraries. Note that
                              fails and you know exactly where your boost
      --with-boost-program-options[=special-lib]
                              use the program options library from boost - it is
                              --with-boost-program-options=boost_program_options-gcc-mt-1_33_1

    指定boost的库:
    从这儿得知在这儿呢:  http://justwinit.cn/post/7706/
    系统默认会将include拷贝到/usr/local/include/boost/中
    将lib拷贝到/usr/local/lib下
    yum install boost-devel  [实践失败]
    回头了解下boost的结构,rpm包是啥样的,如下所示,版本号1.33,有可能是因为lddconfig里默认配置指向这个旧版本了,如下:
    vi /etc/ld.so.conf


    [root@test ~]# rpm -ql boost-1.33.1-16.el5_9
    /usr/lib64/*.so.*  //各种so文件
    /usr/lib/*.so.*     //各种so文件
    rpm -ql boost-devel-1.33.1-16.el5_9
    /usr/include/boost  里面是各种分类目录,里面是hpp头文件
    algorithm/
    archive/
    assign/
    bind/
    compatibility
    config/
    devel包里还有静态a文件及动态so文件:
    /usr/lib/libboost_test_exec_monitor.a
    /usr/lib/libboost_test_exec_monitor.so

    如下实践试试:
    cd /home/xiangdong/software/boost_1_57_0
    ./bootstrap.sh --prefix=/usr/local/boost-1.57
    vi tools/build/v2/user-config.jam
    文件尾部增加(没找到,直接./b2 ./b2 install 即可!):
    using mpi  
    ./b2
    ./b2 install


    Boost headers version >= 1.39.0
    [root@localdomain gearmand-1.1.12]# yum search boost
    [root@localdomain gearmand-1.1.12]# yum install boost.x86_64
    [root@localdomain gearmand-1.1.12]# yum install yum install boost-devel.x86_64
    旧的yum源,直接干死,用源码装个试试:



    出现第五个问题的解决办法:
    /home/xiangdong/software/gearmand-1.1.12/libgearman-server/gearmand_con.cc:677: undefined reference to `event_initialized'
    collect2: ld returned 1 exit status
    libevent版本太老了,更新新的版本:
    安装最新版本的libevent,并执行以下操作:(备份老版本的/usr/lib64/libevent.so)
    ln -s /usr/lib64/libevent-2.0.so.5 /usr/lib64/libevent.so
    删除老旧的libevent的rpm包后,安装最新的方法实践如下:
    [root@test software]# cp  /usr/lib64/libevent.so /usr/lib64/libevent.so.bak.old
    [root@test libevent-2.0.21-stable]# tar -zxvf libevent-2.0.21-stable.tar.gz
    [root@test libevent-2.0.21-stable]# cd libevent-2.0.21-stable
    [root@test libevent-2.0.21-stable]# ./configure --prefix=/usr/local/libevent-2.0.21-stable


    ls -al /usr/lib | grep libevent
    rpm -qa|grep libevent
    libevent-1.4.13-1
    libevent-devel-1.4.13-1
    libevent-devel-1.4.13-1
    libevent-1.4.13-1

    rpm -e --allmatches --nodeps libevent-1.4.13-1 libevent-devel-1.4.13-1
    [root@test libevent-2.0.21-stable]# rpm -e --allmatches --nodeps libevent-1.4.13-1 libevent-devel-1.4.13-1
    [root@test libevent-2.0.21-stable]#

    卸载后看还有没:
    [root@test libevent-2.0.21-stable]# ls -al /usr/lib | grep libevent  
    lrwxrwxrwx   1 root root       31 Dec 30 17:17 libevent-1.1a.so.1 -> /usr/lib/libevent-1.1a.so.1.0.2
    -rw-r--r--   1 root root    31596 Jan  7  2007 libevent-1.1a.so.1.0.2

    rm -Rf  /usr/lib/libevent-1.1a.so.1.0.2 /usr/lib/libevent-1.1a.so.1 //删除干净
    [root@test libevent-2.0.21-stable]# rm -Rf  /usr/lib/libevent-1.1a.so.1.0.2 /usr/lib/libevent-1.1a.so.1
    [root@test libevent-2.0.21-stable]#

    cd /home/xiangdong/software/libevent-2.0.21-stable  //最新安装包
    ./configure --prefix=/usr   //libevent会安装到 /usr/lib 或 /usr/local/lib 下
    #./configure --prefix=/usr/local/libevent-2.0.21-stable
    [root@test libevent-2.0.21-stable]# ./configure --prefix=/usr && make && make install
    测试libevent是否安装成功:ls -al /usr/lib | grep libevent(或 ls -al /usr/local/lib | grep libevent)
    ls -al /usr/lib | grep libevent
    lrwxrwxrwx   1 root root       21 Dec 31 09:44 libevent-2.0.so.5 -> libevent-2.0.so.5.1.9
    -rwxr-xr-x   1 root root  1065819 Dec 31 09:44 libevent-2.0.so.5.1.9
    -rw-r--r--   1 root root  1678970 Dec 31 09:44 libevent.a
    lrwxrwxrwx   1 root root       26 Dec 31 09:44 libevent_core-2.0.so.5 -> libevent_core-2.0.so.5.1.9

    如果libevent的安装目录为/usr/local/lib下,则还需要建立 libevent-1.4.so.2 到 /usr/lib 的软连接,这样其他程序运行时才能找到libevent库:ln -s /usr/local/lib/libevent-1.4.so.2  /usr/lib/libevent-1.4.so.2

    ls /usr/lib/libevent-2.0.so.5
    /usr/lib/libevent-2.0.so.5

    干掉旧的libevent的残留的动态so文件:
    ls -al /usr/lib64 | grep libevent
    lrwxrwxrwx   1 root root       19 Dec 31 09:40 libevent-1.4.so.2 -> libevent.so.bak.old
    -rwxr-xr-x   1 root root   104296 Dec 31 09:22 libevent.so.bak.old
    [root@test libevent-2.0.21-stable]# rm -Rf /usr/lib64/libevent-1.4.so.2
    参考来自:http://blog.sina.com.cn/s/blog_4b93170a0100mbm9.html

    编译成功结果如下:


    启动gearmand的方法:
    [root@test gearmand-1.1.12]# whereis gearmand
    gearmand: /usr/local/sbin/gearmand
    [root@test ~]# mkdir -p /usr/local/gearman/log

    ls /usr/local/sqlite3/bin/
    sqlite3

    *特别提醒,(mysql性能上可能在做队列上并不比sqlite强到哪儿去)
    使用sqlite更加简单方便 :
    /usr/local/sbin/gearmand -l /usr/local/gearman/log/trace2.log --verbose INFO -p 4830 -q libsqlite3 --libsqlite3-db /usr/local/sqlite/bin/gearman --libsqlite3-table gearman_queue -d
    很遗憾,我到现在也没能编译出对MySQL的持久化

    [root@test ~]# /usr/local/sbin/gearmand -l /usr/local/gearman/log/trace2.log --verbose INFO -p 4830 -q libsqlite3 --libsqlite3-db /usr/local/sqlite/bin/gearman --libsqlite3-table gearman_queue -d
    /usr/local/sbin/gearmand: Error while initializing the queue : libsqlite3

    原来是sqlite的位置不对,修改路径后就OK了:

    启动成功:
    [root@test sqlite3]# /usr/local/sbin/gearmand -l /usr/local/gearman/log/trace2.log --verbose INFO -p 4830 -q libsqlite3 --libsqlite3-db /usr/local/sqlite3/bin/gearman --libsqlite3-table gearman_queue -d
    [root@test sqlite3]#
    端口存在了,说明是真的启动成功了:
    [root@test sqlite3]# netstat -atlunp|grep 4830
    tcp        0      0 0.0.0.0:4830                0.0.0.0:*                   LISTEN      17820/gearmand      
    tcp        0      0 :::4830                     :::*                        LISTEN      17820/gearmand  

    也就是说gearmand启动时候自己建立一个sqlite的表,如下:
    [root@test sqlite3]# cat /usr/local/sqlite3/bin/gearman
    SQLite format 3@  ''blegearman_queuegearman_queueCREATE TABLE gearman_queue ( unique_key TEXT, function_name TEXT, priority INTEGER, data BLOB, when_to_run INTEGER, PRIMARY KEY (unique_key, function_name))9M'indexsqlite_autoindex_gearman_queue_1gearman_queue

    经过一陈折腾,说明这个gearmand对boost库的要求高,对libevent的版本也要求高,不是能经过yum install能解决得了的,得自己安装后指定,再就是对编译器的版本也是有要求的,也就是说相对来说其安装比较苛刻一些,如Lamp架构相比较的话,难度要高一些。
    ——————————————————————————————————————————————————————————————
    总之,一大堆问题,这货居然还这么流行听说Gearman最初用于LiveJournal的图片resize功能,由于图片resize需要消耗大量计算资 源,因此需要调度到后端多台服务器执行,完成任务之后返回前端再呈现到界面。,看来广大分发系统还真得靠它啊,其它各种小问题不断,参考:
    http://www.111cn.net/sys/CentOS/65334.htm

    二、PHP的gearmand扩展编译及遇到问题解决实践如下:
    PHP的gearmand折腾扩展,这块试了好几个版本的扩展要么configure不过去,要么是lib里却啥,但最后还是以当前最新的版本gearman-1.1.2.tgz给解决了:

    ./configure --with-php-config=/usr/local/php/bin/php-config
    出现如下所示的配置错误:
    configure: WARNING: You will need re2c 0.13.4 or later if you want to regenerate PHP parsers.
    checking for gawk... gawk
    checking whether to enable gearman support... yes, shared
    found in /usr/local
    checking for gearman_create in -lgearman... no
    configure: error: wrong libgearman version or lib not found

    问题一:
    gearman的PHP扩展时出现configure: WARNING: You will need re2c 0.13.4 or later if you want to regenerate PHP parsers.警告信息。
    警告信息说明需要安装 re2c
    执行下面命令安装:
    wget http://sourceforge.net/projects/re2c/files/re2c/0.13.5/re2c-0.13.5.tar.gz/download
    2014-12-31 11:01:59 (999 KB/s) - `re2c-0.13.5.tar.gz' saved [782725/782725]
    tar -zxvf re2c-0.13.5.tar.gz
    cd re2c-0.13.5
    ./configure && make && make install //安装成功,警告得到了解决,第二个依旧存在。
    它的意思是好像没有找到lib,不对是找到了没有找到里面的某个叫gearman_create的函数,这个就奇怪了,找下:
    [root@test /]# find . -name "*gearman*.so*"
    ./usr/local/lib/libgearman.so.8.0.0
    ./usr/local/lib/libgearman.so
    ./usr/local/lib/libgearman.so.8

    [root@test /]# vi /etc/ld.so.conf
    include ld.so.conf.d/*.conf
    /lib
    /lib64
    /usr/lib                                                                                                                                                                                            
    /usr/lib64
    /usr/local/lib
    /usr/local/lib
    有这个位置的啊。
    [root@test /]# ldconfig  //重新lib的位置载入。

    [root@test gearman-0.4.0]# phpize
    Configuring for:
    PHP Api Version:         20090626
    Zend Module Api No:      20090626
    Zend Extension Api No:   220090626
    [root@test gearman-0.4.0]# ./configure --with-php-config=/usr/local/php/bin/php-config

    问题依旧,去官网看一下呗,http://gearman.org/download/:
    文字摘录:PHP
    There are two PHP client/worker libraries, one which is a pure PHP extension and one which wraps the libgearman C library.
    Gearman Extension
    A PHP extension that wraps the libgearman C library.
    Gearman PHP Extension (1.0.2) (Source)
    官方的PHP扩展高V1.0.2,我这个是gearman-0.4.0太旧了,好像还有更高的gearman-1.1.2.tgz,http://pecl.php.net/package/gearman,放上官网这个上去重新试下了:
    tar -zxvf gearman-1.0.2.tgz
    cd gearman-1.0.2
    [root@test gearman-1.0.2]# phpize
    ./configure --with-php-config=/usr/local/php/bin/php-config   //这个没有问题
    [root@test gearman-1.0.2]# make && make install
    /home/xiangdong/software/gearman-1.0.2/php_gearman.c: In function ‘zm_startup_gearman’:
    /home/xiangdong/software/gearman-1.0.2/php_gearman.c:4638: error: ‘GEARMAN_MAGIC_TEXT’ undeclared (first use in this function)
    /home/xiangdong/software/gearman-1.0.2/php_gearman.c:4638: error: (Each undeclared identifier is reported only once
    还是差啥么子玩意,换个更新的看下,gearman-1.1.2.tgz 实践OK了,如下:
    Libraries have been installed in:
       /home/xiangdong/software/gearman-1.1.2/modules

    If you ever happen to want to link against installed libraries
    in a given directory, LIBDIR, you must either use libtool, and
    specify the full pathname of the library, or use the `-LLIBDIR'
    flag during linking and do at least one of the following:
       - add LIBDIR to the `LD_LIBRARY_PATH' environment variable
         during execution
       - add LIBDIR to the `LD_RUN_PATH' environment variable
         during linking
       - use the `-Wl,--rpath -Wl,LIBDIR' linker flag
       - have your system administrator add LIBDIR to `/etc/ld.so.conf'

    See any operating system documentation about shared libraries for
    more information, such as the ld(1) and ld.so(8) manual pages.
    ----------------------------------------------------------------------

    Build complete.
    Don't forget to run 'make test'.

    [root@test gearman-1.1.2]# make install
    Installing shared extensions:     /usr/local/php/lib/php/extensions/no-debug-non-zts-20090626/
    真在的:
    ls  /usr/local/php/lib/php/extensions/no-debug-non-zts-20090626/gearman.so
    /usr/local/php/lib/php/extensions/no-debug-non-zts-20090626/gearman.so
    在php.ini里加上这个扩展就成了。Successful....EOF


    [root@test gearman-1.1.2]# php -v
    PHP 5.3.10 (cli) (built: Feb  3 2012 14:04:56)
    Copyright (c) 1997-2012 The PHP Group
    Zend Engine v2.3.0, Copyright (c) 1998-2012 Zend Technologies
        with eAccelerator v0.9.6.1, Copyright (c) 2004-2010 eAccelerator, by eAccelerator

    [root@test gearman-1.1.2]# vi /usr/local/php/etc/php.ini
    extension = "gearman.so"  

    [root@test gearman-1.1.2]# php -m|grep gearman
    gearman



    三、代码编写task的分发及client的及简单代码实践OK及其情况实践Ok:
    cd /usr/local/gearman/
    [root@test gearman]# mkdir codesTest
    [root@test gearman]# cd codesTest/
    [root@test codesTest]# vi myworker.php


    以守护进程方式启动worker:
    # nohup php myworker.php >/dev/null 2>&1 &
    [root@test codesTest]# nohup php myworker.php >/dev/null 2>&1 &
    [1] 31238

    一个发送任务处理请求的client:
    vi  myclient.php
    [root@test codesTest]# vi  myclient.php


    运行myclient.php:
    [root@test codesTest]# php myclient.php "Jose"
    hello, Jose
    H:test.local:2

    一个发出并行处理任务请求的例子tasksclient:
    [root@test codesTest]# vi  tasksclient.php


    运行tasksclient:
    [root@test codesTest]# php tasksclient.php
    Completed task:: id :2 , handled result:hello, John
    Completed task:: id :1 , handled result:hello, Jose

    查看worker及队列情况:
    (echo "workers"; sleep 0.1) | nc 192.168.109.8 4830
    [root@test codesTest]#  (echo "workers"; sleep 0.1) | nc 192.168.109.8 4830
    34 192.168.109.8 - : sayhello logMsg
    35 192.168.109.8 - :
    .
    查看当前队列情况(显示格式FUNCTION TOTAL RUNNING AVAILABLE_WORKERS):
    (echo "status"; sleep 0.1) | nc 192.168.109.8 4830
    [root@test codesTest]# (echo "status"; sleep 0.1) | nc 192.168.109.8 4830
    sayhello        0       0       1
    logMsg  0       0       1
    .

    监控php的守护进程(没实践), 可以开多个,并同时监控:
    关于Worker守护进程的启动和监控
    上面例子中直接启动worker脚本作为守护进程,无法监控到worker进程是否存活.
    使用Unix进程监控supervisord则可轻松解决这个问题.
    将如下配置添加到supervisord的配置中,即可实现由supervisord来启动和监控myworker.
    编辑配置文件 vi /etc/supervisord.conf
    [program:myworker]
    command=/usr/local/php5415/bin/php myworker.php
    process_name=%(program_name)s_%(process_num)02d
    ;修改numprocs即可同时开启多个worker进程
    numprocs=1
    directory=/usr/local/gearman/codesTest
    autostart=true
    autorestart=true
    user=gearmand
    stopsignal=KILL
    Posted in Daemon / Worker, Gearman.
    Tagged gearman.

    监控这块看:
    http://jingyan.baidu.com/article/375c8e198d1b1425f2a2290c.html

    上面这块的PHP扩展代码及实践来自:
    http://www.51itstudy.com/30114.html
    http://codego.net/384693/

    _________下面是摘录别的文章不一定自己配置时可完全过得过,得按自己环境作配置安装,只是用作参考!_______
    Gearman的目的在于用PHP扩展分发,于是PHP扩展如何安装:
    tar zxvf gearman-1.1.1.tgz
    cd gearman-1.1.1
    /opt/local/php/bin/phpize
    ./configure --with-php-config=/opt/local/php/bin/php-config --with-gearman
    make
    make install
    编辑 php.ini
    vi php.ini
    增加
    extension = "gearman.so"
    重启php

    启动gearmand 服务
    gearmand -L 10.6.0.6 -p 4730  -u root -l /var/log/gearmand.log -d
    其他参数请  gearmand --help

    摘自:http://jicki.blog.51cto.com/1323993/1177487

    通常,多语言多系统之间的集成是个大问题,一般来说,人们多半会采用WebService的方式来处理此类集成问题,但不管采用何种风格的WebService,如RPC风格,或者REST风格,其本身都有一定的复杂性。相比之下,Gearman也能实现类似的作用,而且更简单易用。

    一个Gearman请求的处理过程涉及三个角色:Client -> Job -> Worker。

    Client:请求的发起者,可以是C,PHP,Perl,MySQL UDF等等。
    Job:请求的调度者,用来负责协调把Client发出的请求转发给合适的Work。
    Worker:请求的处理者,可以是C,PHP,Perl等等。

    因为Client,Worker并不限制用一样的语言,所以有利于多语言多系统之间的集成。

    甚至我们通过增加更多的Worker,可以很方便的实现应用程序的分布式负载均衡架构。

    下面看看如何安装运行一个例子,条件所限,我们把Client,Job,Worker三个角色运行在一台服务器上:

    安装libevent:
    wget http://www.monkey.org/~provos/libevent-1.4.12-stable.tar.gz
    tar zxvf libevent-1.4.12-stable.tar.gz
    cd libevent-1.4.12-stable/
    ./configure --prefix=/usr
    make && make install
    /sbin/ldconfig
    cd ../



    安装Gearman server and library:

    wget http://launchpad.net/gearmand/trunk/0.9/+download/gearmand-0.9.tar.gz
    tar zxvf gearmand-0.9.tar.gz
    cd gearmand-0.9
    ./configure
    make
    make install
    /sbin/ldconfig
    cd ../


    安装Gearman PHP extension:

    wget http://pecl.php.net/get/gearman-0.5.0.tgz
    tar zxvf gearman-0.5.0.tgz
    cd gearman-0.5.0
    /usr/local/webserver/php/bin/phpize
    ./configure --with-php-config=/usr/local/webserver/php/bin/php-config --with-gearman
    make
    make install
    cd ../

    编辑php.ini配置文件加载相应模块并使之生效:

    extension = "gearman.so"

    启动Job:

    gearmand -d

    如果当前用户是root的话,则需要这样操作:

    gearmand -d -u root

    缺省会使用4730端口,下面会用到。

    注意:如果找不到gearmand命令的路径,别忘了用whereis gearmand确认。



    我们可以用 ps 指令來查看启动是否成功:

    ps aux |grep gearman
    编写Worker:

    worker.php文件内容如下:

    <?php
    $worker= new GearmanWorker();
    $worker->addServer('127.0.0.1', 4730);
    $worker->addFunction('reverse', 'my_reverse_function');

    while ($worker->work());

    function my_reverse_function($job)
    {
        return strrev($job->workload());
    }
    ?>

    设置后台运行work:

    #php /var/www/html/worker.php &

    编写Client:

    client.php文件内容如下:

    <?php
    $client= new GearmanClient();
    $client->addServer('127.0.0.1', 4730);
    echo $client->do('reverse', 'Hello World!'), "n";
    ?>

    运行client:

    #php /var/www/html/client.php

    输出:!dlroW olleH



    首先, PHP Gearman Extension 提供了一个名为GearmanClient 的类别,它可以让程序安排工作给 Job Server 。

    而 addServer 方法表示要通知的是哪些 Job Server ,也就是说如果有多台 Job Server 的话,就可以透过 addServer 新增。

    然后我们将要呼叫哪个 Worker 以及该 Worker 所需要的数据,利用 GearmanClient 的doBackground 方法传送过去。 doBackground 方法顾名思义就是在背景执行, Client 在丢出需求后就可以继续处理其他的程序。

    =============

    $client->doBackground('sendEmail', serialize($emailData));
    echo "Email sending is done.n";

    ============

    doBackground 方法的第一个参数是告诉 Job Server 要执行哪个功能,而这个功能则是由 Worker 提供的;要注意是,这个参数只是识别用的,并不是真正的函式名称。而第二个参数是要传给 Worker 的数据,它必须是个字符串;因此如果要传送的是数组的话,我们就要用 PHP 的 serialize 函式来对这些资料做串行化。

    出于方便的考虑,Worker,Client使用的都是PHP,但这并不影响演示,实际应用中,你完全可以通过Gearman集成不同语言实现的Worker,Client。或许此时你还想了解前面提到的负载均衡功能:很简单,只要增加多个Worker即可,你可以按照worker.php的样子多写几个类似的文件,并设置不同的返回值用以识别演示效果。然后依次启动这几个Worker文件,并多次使用client.php去请求,你就会发现Job会把Client请求转发给不同的Worker。

    以守护进程启动
    gearmand -L 192.168.0.1 -p 4730 -u root -d

    命令行工具

    如果你觉得安装PHP之类的东西太麻烦的话,你也可以仅仅通过命令行工具来体验Gearman的功能:

    启动Worker:gearman -w -f wc -- wc -l &
    运行Client:gearman -f wc < /etc/passwd

    具体可以参考官方文档。
    摘自:http://www.codesky.net/article/201107/174139.html
    安装时会出现一个小问题:
    configure出现以下错误 checking for gperf... no configure: error: could not find gperf. 解决办法: #yum search gperf #yum install gperf.x86_64.
    gperf.x86_64 : A perfect hash function generator.

    Gearman 安装使用 以及 问题处理  
    http://deapge.blog.163.com/blog/static/1113114792013929380253


    gearman.h:53:23: error: cinttypes: No such file or directory
    tr1/cinttypes: No such file or directory
    错误代码:
    make[1]: Entering directory `/usr/local/src/gearmand-0.38'
      CXX    libgearman/libgearman_libgearman_la-actions.lo
    In file included from ./libgearman/common.h:44,
                     from libgearman/actions.cc:39:
    ./libgearman-1.0/gearman.h:53:27: error: tr1/cinttypes: No such file or directory
    make[1]: *** [libgearman/libgearman_libgearman_la-actions.lo] Error 1
    make[1]: Leaving directory `/usr/local/src/gearmand-0.38'
    解决:[root@localdomain gearmand-0.38]# yum install gcc44 gcc44-c++
    来自google快照:https://webcache.googleusercontent.com/search?q=cache:GhqVv51gppAJ:www.livingelsewhere.net/2012/08/22/gearman-instal/+&cd=1&hl=zh-CN&ct=clnk&gl=cn
    google来自:http://www.livingelsewhere.net/2012/08/22/gearman-instal/

    解决./libgearman-1.0/gearman.h:53:23: error: cinttypes: No such file or directory 实践OK如下:
    后补来自:http://blog.slogra.com/post-234.html
    如果出现configure: error: Unable to find libevent,则输入命令:
    yum -y install libevent libevent-devel
    然后重新configure.
    没有问题后再执行
    make
    这步出现./libgearman-1.0/gearman.h:53:27: error: tr1/cinttypes: No such file or directory错误,可以推断是gcc编译器的问题,执行命令:
    yum install gcc44 gcc44-c++ libstdc++44-devel -y
    然后在环境变量里加入:
    export CC=/usr/bin/gcc44 or export CC=/usr/bin/gcc
    export CXX=/usr/bin/g++44
    保存退出后执行:
    source /etc/profile
    删除gearmand-0.34文件夹重新进行编译.
    重新进行编译后执行make这步,如图
    点击查看原图
    没有报错的话才能继续执行
    make install
    如果没有报错的话,那么恭喜你可以继续安装gearman了.

    看下yum后新的gcc在不在,在的:
    yum install gcc44 gcc44-c++ libstdc++44-devel -y

    Total download size: 4.1 M
    Downloading Packages:
    libstdc++44-devel-4.4.7-1.el5.i386.rpm      | 4.1 MB     00:00    
    Transaction Test Succeeded
    Running Transaction
      Installing     : libstdc++44-devel        1/1
    Installed:
      libstdc++44-devel.i386 0:4.4.7-1.el5        
    Complete!

    方法二:
    http://www.cnblogs.com/cenalulu/archive/2013/04/03/2998140.html
    关键信息:error: tr1/cinttypes: No such file or directory

    报错原因:libmemcached需要 gcc 4.2 以上版本才可编译,而centos 5.4 的gcc版本只有4.1 ,详见:https://bugs.launchpad.net/libmemcached/+bug/1076181

    解决方法:安装gcc44的扩展包,详见:http://gearman.info/build/centos5-8.html

    解决步骤引用如下:

    wget http://mirrors.ustc.edu.cn/fedora/epel/5/i386/epel-release-5-4.noarch.rpm
    rpm -ivh epel-release-5-4.noarch.rpm
    yum -y -q install boost141-devel
    ln -s /usr/include/boost141/boost/ /usr/include/boost
    export LDFLAGS="-L/usr/lib64/boost141"
    export LD_LIBRARY_PATH=/usr/lib64/boost141:$LD_LIBRARY_PATH
    yum -y -q install e2fsprogs-devel e2fsprogs
    yum -y -q install gcc44 gcc44-c++
    export CC="gcc44"
    export CXX="g++44"

    所有操作完成后,重新执行:

    ./configure --enable-memaslap && make && make install

    安装完成!!

    实践如下问题:
    [root@test software]# rpm -ivh epel-release-5-4.noarch.rpm
    rpm: error while loading shared libraries: libbeecrypt.so.6: cannot open shared object file: No such file or directory


    http://pecl.php.net/package/gearman
    http://sourceforge.net/projects/boost/?source=pdlp
    提示没有安装boost libraries,那么安装之。
    下载地址:http://sourceforge.net/projects/boost/?source=pdlp
    关于boost的了解,可以到官网:
    http://www.boost.org/doc/libs/1_55_0/more/getting_started/index.html
    http://www.boost.org/doc/libs/1_55_0/more/getting_started/windows.html

    Boost库的安装步骤参照:(漫长的编译过程……)
    Linux下安装gearman的方法:
    https://justwinit.cn/post/7706/
    ————————————————————————————————————————
    http://www.cnblogs.com/longcpp/archive/2012/06/06/2538251.html
    还有http://m.blog.csdn.net/blog/yan420523/9096871(……晕晕呼呼)
    http://www.cnblogs.com/cocowool/archive/2011/08/18/2144142.html
    接着重新执行上面的./configure
    又出现错误了:

    http://www.cnblogs.com/youlechang123/archive/2013/10/22/3382425.html
    用apt-get install gperf安装了gperf,重新执行./configure,执行一段时间,又出错了:
    =====================================================================
    重点参考:http://blog.chinaunix.net/uid-27177626-id-4325909.html

    参照http://blog.slogra.com/post-234.html解决
    用apt-get install libevent libevent-devel安装缺少的依赖,但是又出错了:ubuntu E: Unable to locate package libevent
    更新源:apt-get apdate  apt-get upgrade
    还是无法安装libevent,只有去官网下载,用源码安装方式安装:http://libevent.org/
    下载之后,./configure make make install
    重新执行./configure,执行一段时间,又出错:

    用apt-get install libuuid无法安装,因此,只有去官网下载:
    http://sourceforge.net/projects/libuuid/
    用源码安装方式安装。
    然后,再次重新执行gearman目录下的./configure。呼呼~~~~~,终于不再出错了,这次顺利执行完毕。

    Boost DownLoad:
    http://jaist.dl.sourceforge.net/project/boost/boost/1.57.0/boost_1_57_0.zip

  • 相关阅读:
    第九篇 正则表达式
    第八篇、递归、装饰器
    第四篇、函数和递归
    第二篇、HTML
    nginx rewrite标签配置以及用户认证配置
    nginx location
    nginx日志配置,以及日志轮询
    nginx别名配置,状态配置,include优化
    第一篇 先用socket模拟web服务器
    第二十八篇、自定义线程池
  • 原文地址:https://www.cnblogs.com/chunguang/p/5891887.html
Copyright © 2011-2022 走看看