zoukankan      html  css  js  c++  java
  • 分布式存储之MogileFS基于Nginx实现负载均衡(Nginx+MogileFS)

    MogileFS分布式文件系统特点:

      1.具有raid的性能

      2.不存在单点故障

      3.简单的命名空间: 每个文件对应一个key:用于domain定义名称空间

          4.不共享任何数据

          5.传输中立,无特殊协议:可以通过NFS或HTTP进行通信

          6.自动文件复制:复制的最小单位不是文件,而class

      7.应用层: 用户空间文件系统,无须特殊的核心组件

    Nginx+MogileFS的好处:

      1、将请求代理至后端MogileFS服务器集群中,能实现负载均衡的效果。

       2、能对后端的tracker节点进行健康检测。

       3、将第三方模块“nginx_mogilefs_module”编译进Nginx中,能实现直接使用key访问对应的文件,如下:

            使用nginx做代理之前:http://192.168.80.137:7500/dev2/0/000/000/0000000007.fid

            使用nginx做代理之后:http://192.168.80.132/image/1.jpg

    MogileFS是由三个组件组成的:

      1、tracker:MogileFS的核心,是一个调度器,服务进程为mogilefsd,职责:删除数据、复制数据、监控、查询等。

      2、database:为tracker存储元素据

      3、数据存储的位置,通常是一个HTTP(webDAV)服务器,用来做数据的创建(put)、删除(delete)、获取(get),监听端口7500, storage节点使用http进行数据传输, 依赖于perbal, 进程为mogstored。

    理想中模型:

    实验架构:

    实验中用到的rpm包:http://pan.baidu.com/s/1c1S0dJM 

    MariaDB节点配置:

    对各mogilefs节点的主机名进行解析:

    [root@www ~]# vim /etc/hosts
         192.168.80.136 mog1.daixiang.com
         192.168.80.137 mog2.daixiang.com
         192.168.80.138 mog3.daixiang.com
                               
    View Code

    二进制包安装MariaDB:

    [root@mariadb mysql]# useradd -r -s /sbin/nologin mysql
    [root@mariadb ~]# tar xf mariadb-10.1.14-linux-x86_64.tar.gz -C /usr/local/
    [root@mariadb ~]# ln -sv /usr/local/mariadb-10.1.14-linux-x86_64 /usr/local/mysql
    [root@mariadb mysql]# chown -R mysql.mysql /usr/local/mysql/
    [root@mariadb mysql]# ./scripts/mysql_install_db --user=mysql --datadir=/data
    [root@mariadb mysql]# cp support-files/mysql.server /etc/rc.d/init.d/mysqld
    [root@mariadb mysql]# chmod +x /etc/rc.d/init.d/mysqld
    [root@mariadb mysql]# cp support-files/my-large.cnf /etc/my.cnf 
    [root@mariadb mysql]# vim /etc/my.cnf
             datadir = /data
    [root@mariadb mysql]# ln -sv /usr/local/mysql/include/ /usr/include/mysql/
    [root@mariadb mysql]# vim /etc/ld.so.conf.d/mysql.conf
             /usr/local/mysql/lib
    [root@mariadb mysql]# vim /etc/profile.d/mysql.sh 
             export PATH=/usr/local/mysql/bin:$PATH
    [root@mariadb mysql]# ldconfig 
    [root@mariadb mysql]# ldconfig -p | grep mysql
    	libmysqld.so.18 (libc6,x86-64) => /usr/local/mysql/lib/libmysqld.so.18
    	libmysqld.so (libc6,x86-64) => /usr/local/mysql/lib/libmysqld.so
    	libmysqlclient_r.so.16 (libc6,x86-64) => /usr/lib64/mysql/libmysqlclient_r.so.16
    	libmysqlclient.so.18 (libc6,x86-64) => /usr/local/mysql/lib/libmysqlclient.so.18
    	libmysqlclient.so.16 (libc6,x86-64) => /usr/lib64/mysql/libmysqlclient.so.16
    	libmysqlclient.so (libc6,x86-64) => /usr/local/mysql/lib/libmysqlclient.so
    	libgalera_smm.so (libc6,x86-64) => /usr/local/mysql/lib/libgalera_smm.so
    

       

    对用户进行授权:

    MariaDB [(none)]> grant all on *.* to 'root'@'192.168.80.%' identified by 'rootpass';
    Query OK, 0 rows affected (0.06 sec)
    
    MariaDB [(none)]> grant all on mogilefs.* to 'moguser'@'192.168.80.%' identified by 'mogpass';
    Query OK, 0 rows affected (0.00 sec)
    
    MariaDB [(none)]> flush privileges;
    Query OK, 0 rows affected (0.00 sec)
    

      

    MogileFS配置:

    [root@mog1 ~]# yum install *.rpm -y
    [root@mog1 ~]# yum install perl-IO-AIO -y
    
    [root@mog2 ~]# yum install *.rpm -y
    [root@mog2 ~]# yum install perl-IO-AIO -y
    
    [root@mog3 ~]# yum install *.rpm -y
    [root@mog3 ~]# yum install perl-IO-AIO -y 

    修改mogilefsd进程的配置文件:

     1 [root@mog1 ~]# vim /etc/mogilefs/mogilefsd.conf
     2 
     3 # Enable daemon mode to work in background and use syslog
     4 daemonize = 1                                           #以守护进程的形式运行
     5 # Where to store the pid of the daemon (must be the same in the init script)
     6 pidfile = /var/run/mogilefsd/mogilefsd.pid
     7 # Database connection information
     8 db_dsn = DBI:mysql:mogilefs:host=192.168.80.135      #定义数据库名为mogilefs和数据库服务器地址
     9 db_user = moguser                                     #定义管理此数据库的用户名
    10 db_pass = mogpass                                     #定义密码
    11 # IP:PORT to listen on for mogilefs client requests
    12 listen = 0.0.0.0:7001                                  #定义监听的地址和端口
    13 # Optional, if you don't define the port above.
    14 conf_port = 7001                                                            
    15 # Number of query workers to start by default.
    16 query_jobs = 10                                         #定义启动查询线程个数
    17 # Number of delete workers to start by default.
    18 delete_jobs = 1                                          #定义启动删除线程个数
    19 # Number of replicate workers to start by default.
    20 replicate_jobs = 5                                       #定义启动复制线程个数
    21 # Number of reaper workers to start by default.
    22 # (you don't usually need to increase this)
    23 reaper_jobs = 1                            #响应客户端请求,在磁盘失败后将请求重新放到队列中
    24 # Number of fsck workers to start by default.
    25 # (these can cause a lot of load when fsck'ing)
    26 #fsck_jobs = 1                              #对磁盘进行检测,默认没有启动
    27 # Minimum amount of space to reserve in megabytes
    28 # default: 100                               #默认保留100M空间
    29 # Consider setting this to be larger than the largest file you
    30 # would normally be uploading.
    31 #min_free_space = 200                #最小空闲空间为200M
    32 # Number of seconds to wait for a storage node to respond.
    33 # default: 2                                   #等待存储节点相应的时间,默认为2s
    34 # Keep this low, so busy storage nodes are quickly ignored.
    35 #node_timeout = 2                        #等待节点的超时时长
    36 # Number of seconds to wait to connect to a storage node.
    37 # default: 2                                     #连接存储节点的超时时长
    38 # Keep this low so overloaded nodes get skipped.
    39 #conn_timeout = 2                         
    40 # Allow replication to use the secondary node get port,
    41 # if you have apache or similar configured for GET's
    42 #repl_use_get_port = 1
    View Code

    设置tracker对应的数据库,生成mogilefs库: 

    [root@mog1 ~]# mogdbsetup --dbhost=192.168.80.135 --dbport=3306 --dbname=mogilefs --dbrootuser=root --dbrootpass=rootpass --dbuser=moguser --dbpass=mogpass --yes
    
    #多个tracker共享一个库,所以其他tracker节点上就不需要在做此操作,直接修改配置文件/etc/mogilefs/mogilefs.conf就可以

        注意:此处有一bug,本人也不知道是什么原因造成的:在生成mogilefs库的时候会出现错误,说root用户没对mogilefs库的访问权限(“Failed to grant privileges: Access denied for user 'root'@'192.168.80.%' to database 'mogilefs'”),但是再一次执行上面的设置数据库的命令时就会成功。前提是确保mariadb对用户授权是正确的。如果有哪位大神知道是什么原因造成的,求指点。。  

     到MariaDB节点上查看mogilefs库是否成功生成:

    [root@Mariadb ~]# mysql  
     1 MariaDB [(none)]> show databases;
     2 +--------------------+
     3 | Database           |
     4 +--------------------+
     5 | information_schema |
     6 | mogilefs           |
     7 | mysql              |
     8 | performance_schema |
     9 | test               |
    10 +--------------------+
    11 5 rows in set (0.00 sec)
    12 
    13 MariaDB [(none)]> use mogilefs;
    14 Database changed
    15 MariaDB [mogilefs]> show tables;
    16 +----------------------+
    17 | Tables_in_mogilefs   |
    18 +----------------------+
    19 | checksum             |
    20 | class                |
    21 | device               |
    22 | domain               |
    23 | file                 |
    24 | file_on              |
    25 | file_on_corrupt      |
    26 | file_to_delete       |
    27 | file_to_delete2      |
    28 | file_to_delete_later |
    29 | file_to_queue        |
    30 | file_to_replicate    |
    31 | fsck_log             |
    32 | host                 |
    33 | server_settings      |
    34 | tempfile             |
    35 | unreachable_fids     |
    36 +----------------------+
    37 17 rows in set (0.00 sec)
    View Code

    启动mogilefsd进程:

    [root@mog1 ~]# service mogilefsd start
    1 [root@mog1 ~]# ss -tnl | grep '7001'
    2 LISTEN     0      128                       *:7001                     *:*  
    View Code

    对mogstored进程进行配置:

    [root@mog1 ~]# vim /etc/mogilefs/mogstored.conf
    
    maxconns = 10000
    httplisten = 0.0.0.0:7500
    mgmtlisten = 0.0.0.0:7501
    docroot = /dfs/mogdata
    View Code

    将mogilefsd进程和mogstored进程的配置文件都同步到其他mog节点上去:

    [root@mog1 ~]# scp /etc/mogilefs/mogilefsd.conf 192.168.80.137:/etc/mogilefs/
    [root@mog1 ~]# scp /etc/mogilefs/mogilefsd.conf 192.168.80.138:/etc/mogilefs/
    
    [root@mog1 ~]# scp /etc/mogilefs/mogstored.conf 192.168.80.137:/etc/mogilefs/
    [root@mog1 ~]# scp /etc/mogilefs/mogstored.conf 192.168.80.138:/etc/mogilefs/
    

    在存储节点上创建挂载点

    [root@mog1 ~]# mkdir /dfs/mogdata/dev1
    [root@mog1 ~]# chown -R mogilefs.mogilefs /dfs/mogdata/
    
    [root@mog2 ~]# mkdir /dfs/mogdata/dev2 -pv
    [root@mog2 ~]# chown -R mogilefs.mogilefs /dfs/mogdata/
    
    [root@mog3 ~]# mkdir /dfs/mogdata/dev3 -pv
    [root@mog3 ~]# chown -R mogilefs.mogilefs /dfs/mogdata/
    
    
        注意:在生产环境中,需要将存储磁盘挂载在/dfs/mogdata目录下,当块磁盘挂载成功之后,需要在磁盘上创建共享块设备,即这里博主就不挂载磁盘了,直接创建共享块设备。 
                挂载磁盘的配置方法:
                              [root@mog1 ~]# mkdir /dfs/mogdata/
                              [root@mog1 ~]# mount -t ext4 /dev/sdb1 /dfs/mogdata/
                              [root@mog1 ~]# mkdir /dfs/mogdata/dev1
                              [root@mog1 ~]# chown -R mogilefs,mogilefs /dfs/mogdata/
                              
    

    启动mogstored进程:

    [root@mog1 ~]# service mogstored start
    [root@mog2 ~]# service mogstored start
    [root@mog3 ~]# service mogstored start
    
    [root@mog1 ~]# ss -tnlp | grep 'mogstored'
    LISTEN     0      128                       *:7500                     *:*      users:(("mogstored",2288,4))
    LISTEN     0      128                       *:7501                     *:*      users:(("mogstored",2288,9))
    
    [root@mog2 ~]# ss -tnlp | grep 'mogstored'
    LISTEN     0      128                       *:7500                     *:*      users:(("mogstored",2288,4))
    LISTEN     0      128                       *:7501                     *:*      users:(("mogstored",2288,9))
    
    [root@mog3 ~]# ss -tnlp | grep 'mogstored'
    LISTEN     0      128                       *:7500                     *:*      users:(("mogstored",2288,4))
    LISTEN     0      128                       *:7501                     *:*      users:(("mogstored",2288,9))
    View Code

      

    对各节点进行管理:

    [root@mog1 ~]# mogadm --trackers=192.168.80.136:7001 host add mog1 --ip=192.168.80.136 --status=alive
    [root@mog1 ~]# mogadm --trackers=192.168.80.136:7001 device add mog1 001 --status=alive
    
    [root@mog1 ~]# mogadm --trackers=192.168.80.136:7001 host add mog2 --ip=192.168.80.137 --status=alive
    [root@mog1 ~]# mogadm --trackers=192.168.80.136:7001 device add mog2 002 --status=alive
    
    [root@mog1 ~]# mogadm --trackers=192.168.80.136:7001 host add mog3 --ip=192.168.80.138 --status=alive
    [root@mog1 ~]# mogadm --trackers=192.168.80.136:7001 device add mog3 003 --status=alive
    
    [root@mog1 ~]# mogadm domain add linux1 
    [root@mog1 ~]# mogadm domain add python1
    
    [root@mog1 ~]# mogadm class add linux1 class1 --mindevcount=3
    [root@mog1 ~]# mogadm class add linux1 class2 --mindevcount=2
    [root@mog1 ~]# mogadm class add python1 dx1 --mindevcount=2
    
    [root@mog1 ~]# mogadm check
    
    Checking trackers...
      127.0.0.1:7001 ... OK
    
    Checking hosts...
      [ 1] mog1 ... OK
      [ 2] mog2 ... OK
      [ 3] mog3 ... OK
    
    Checking devices...
      host device         size(G)    used(G)    free(G)   use%   ob state   I/O%
      ---- ------------ ---------- ---------- ---------- ------ ---------- -----
      [ 1] dev1            16.509      3.901     12.608  23.63%  writeable   4.4
      [ 2] dev2            16.509      3.901     12.608  23.63%  writeable   0.3
      [ 3] dev3            16.509      3.897     12.612  23.60%  writeable   0.0
      ---- ------------ ---------- ---------- ---------- ------
                 total:    49.527     11.699     37.828  23.62%
    
    
    
    [root@mog1 ~]# mogadm domain list 
     domain               class                mindevcount   replpolicy   hashtype
    -------------------- -------------------- ------------- ------------ -------
     linux1               class1                    3        MultipleHosts() NONE   
     linux1               class2                    2        MultipleHosts() NONE   
     linux1               default                   2        MultipleHosts() NONE   
    
     python1              default                   2        MultipleHosts() NONE   
     python1              dx1                       2        MultipleHosts() NONE   
    View Code

      详细介绍其查看上一篇博客:"分布式存储之MogileFS分布式文件系统简单应用"

    上传文件进行测试:

    [root@mog1 ~]# mogupload --trackers=192.168.80.136:7001 --domain=linux1 --key='/1.jpg' --file='/root/centos.jpg'
    [root@mog1 ~]# mogfileinfo --tracker=192.168.80.137:7001 --domain=linux1 --key='/1.jpg'
    
    [root@mog1 ~]# mogupload --trackers=192.168.80.136:7001 --domain=python1 --key='/fstab.html' --file='/etc/fstab'
    [root@mog1 ~]# mogfileinfo --tracker=192.168.80.137:7001 --domain=python1 --key='/fstab.html'  
    [root@mog1 ~]# mogfileinfo --tracker=192.168.80.137:7001 --domain=linux1 --key='1.jpg'
    - file: 1.jpg
         class:              default
      devcount:                    2
        domain:               linux1
           fid:                    5
           key:                1.jpg
        length:               134783
     - http://192.168.80.137:7500/dev2/0/000/000/0000000005.fid
     - http://192.168.80.136:7500/dev1/0/000/000/0000000005.fid
    View Code

    配置Nginx:

    [root@nginx ~]# # yum groupinstall "Development Tools" "Server Platform Deveopment" -y
    [root@nginx ~]# # yum install openssl-devel pcre-devel -y
    [root@nginx ~]# useradd -r nginx
    [root@nginx ~]# tar xf nginx_mogilefs_module-1.0.4.tar.gz 
    [root@nginx ~]# tar xf nginx-1.10.0.tar.gz 
    [root@nginx ~]# cd nginx-1.10.0
    [root@nginx nginx-1.10.0]#  ./configure 
      --prefix=/usr/local/nginx 
      --sbin-path=/usr/sbin/nginx 
      --conf-path=/etc/nginx/nginx.conf 
      --error-log-path=/var/log/nginx/error.log 
      --http-log-path=/var/log/nginx/access.log 
      --pid-path=/var/run/nginx/nginx.pid  
      --lock-path=/var/lock/nginx.lock 
      --user=nginx 
      --group=nginx 
      --with-http_ssl_module 
      --with-http_flv_module 
      --with-http_stub_status_module 
      --with-http_gzip_static_module 
      --http-client-body-temp-path=/var/tmp/nginx/client/ 
      --http-proxy-temp-path=/var/tmp/nginx/proxy/ 
      --http-fastcgi-temp-path=/var/tmp/nginx/fcgi/ 
      --http-uwsgi-temp-path=/var/tmp/nginx/uwsgi 
      --http-scgi-temp-path=/var/tmp/nginx/scgi 
      --with-pcre 
      --with-debug 
      --add-module=/root/nginx_mogilefs_module-1.0.4    #添加的第三方模块,实现直接使用key作为uri进行访问
    
    [root@nginx nginx-1.10.0]#  make && make install
    

      

    提供nginx启动脚本:

    [root@nginx ~]# vim /etc/rc.d/init.d/nginx
    
    #!/bin/sh
    #
    # nginx - this script starts and stops the nginx daemon
    #
    # chkconfig:   - 85 15 
    # description:  Nginx is an HTTP(S) server, HTTP(S) reverse 
    #               proxy and IMAP/POP3 proxy server
    # processname: nginx
    # config:      /etc/nginx/nginx.conf
    # config:      /etc/sysconfig/nginx
    # pidfile:     /var/run/nginx.pid
     
    # Source function library.
    . /etc/rc.d/init.d/functions
     
    # Source networking configuration.
    . /etc/sysconfig/network
     
    # Check that networking is up.
    [ "$NETWORKING" = "no" ] && exit 0
     
    nginx="/usr/sbin/nginx"
    prog=$(basename $nginx)
     
    NGINX_CONF_FILE="/etc/nginx/nginx.conf"
     
    [ -f /etc/sysconfig/nginx ] && . /etc/sysconfig/nginx
     
    lockfile=/var/lock/subsys/nginx
     
    make_dirs() {
       # make required directories
       user=`nginx -V 2>&1 | grep "configure arguments:" | sed 's/[^*]*--user=([^ ]*).*/1/g' -`
       options=`$nginx -V 2>&1 | grep 'configure arguments:'`
       for opt in $options; do
           if [ `echo $opt | grep '.*-temp-path'` ]; then
               value=`echo $opt | cut -d "=" -f 2`
               if [ ! -d "$value" ]; then
                   # echo "creating" $value
                   mkdir -p $value && chown -R $user $value
               fi
           fi
       done
    }
     
    start() {
        [ -x $nginx ] || exit 5
        [ -f $NGINX_CONF_FILE ] || exit 6
        make_dirs
        echo -n $"Starting $prog: "
        daemon $nginx -c $NGINX_CONF_FILE
        retval=$?
        echo
        [ $retval -eq 0 ] && touch $lockfile
        return $retval
    }
     
    stop() {
        echo -n $"Stopping $prog: "
        killproc $prog -QUIT
        retval=$?
        echo
        [ $retval -eq 0 ] && rm -f $lockfile
        return $retval
    }
     
    restart() {
        configtest || return $?
        stop
        sleep 1
        start
    }
     
    reload() {
        configtest || return $?
        echo -n $"Reloading $prog: "
        killproc $nginx -HUP
        RETVAL=$?
        echo
    }
     
    force_reload() {
        restart
    }
     
    configtest() {
      $nginx -t -c $NGINX_CONF_FILE
    }
     
    rh_status() {
        status $prog
    }
     
    rh_status_q() {
        rh_status >/dev/null 2>&1
    }
     
    case "$1" in
        start)
            rh_status_q && exit 0
            $1
            ;;
        stop)
            rh_status_q || exit 0
            $1
            ;;
        restart|configtest)
            $1
            ;;
        reload)
            rh_status_q || exit 7
            $1
            ;;
        force-reload)
            force_reload
            ;;
        status)
            rh_status
            ;;
        condrestart|try-restart)
            rh_status_q || exit 0
                ;;
        *)
            echo $"Usage: $0 {start|stop|status|restart|condrestart|try-restart|reload|force-reload|configtest}"
            exit 2
    esac
    View Code
    [root@nginx ~]# chmod +x /etc/rc.d/init.d/nginx  

    修改nginx配置文件如下:

    http {
    
        upstream images {
            server 192.168.80.136:7001;
            server 192.168.80.137:7001;
            server 192.168.80.138:7001;
        }
    
        server {
            listen       80;
            server_name  localhost;
            location /image {
                mogilefs_tracker images;
                mogilefs_domain linux1;
                mogilefs_methods GET;
                mogilefs_noverify on;
                mogilefs_pass {
                    proxy_pass $mogilefs_path;
                    proxy_hide_header Content-Type;
                    proxy_buffering off;
                }
            }
    
            location /files {
                mogilefs_tracker images;
                mogilefs_domain python1;
                mogilefs_methods GET;
                mogilefs_noverify on;
                mogilefs_pass {
                    proxy_pass $mogilefs_path;
                    proxy_hide_header Content-Type;
                    proxy_buffering off;
                }
            }
        }
    }   
    View Code

    启动nginx:

    [root@nginx ~]# service nginx start
    

    访问测试:

     

     

      

      

     

  • 相关阅读:
    记一道乘法&加法线段树(模版题)
    2021CCPC网络赛(重赛)题解
    Codeforces Round #747 (Div. 2)题解
    F. Mattress Run 题解
    Codeforces Round #744 (Div. 3) G题题解
    AtCoder Beginner Contest 220部分题(G,H)题解
    Educational Codeforces Round 114 (Rated for Div. 2)题解
    Codeforces Global Round 16题解
    Educational Codeforces Round 113 (Rated for Div. 2)题解
    AtCoder Beginner Contest 182 F
  • 原文地址:https://www.cnblogs.com/daixiang/p/5610012.html
Copyright © 2011-2022 走看看