MariaDB Galera Cluster环境搭建及高可用测试

一、服务器概况
Galera Cluster需要至少三个节点，在此次实验过程中，三个节点IP地址：
192.168.56.101
192.168.56.102
192.168.56.103
OS为centos 7.2
服务器配置：
4G 内存，2核CPU，20G数据磁盘空间。

关闭防火墙:

service iptables stop
systemctl stop firewalld
chkconfig iptables off

关闭SELinux:

/etc/selinux/config中的SELINUX=disabled

修改主机名

vi /etc/hostname MariaDB01 MariaDB02 MariaDB03
systemctl restart systemd-hostnamed

三台主机上加入/etc/hosts:

192.168.56.101 MariaDB01
192.168.56.102 MariaDB02
192.168.56.103 MariaDB03

二、MariaDB Galera Cluster软件二进制安装

  官网下载：https://downloads.mariadb.org/   10.0.29

  初始化节点1

su - mysql
cd /app/mysql/
vi my.cnf

--my.cnf
[client]
port = 3306
socket = /tmp/mysql.sock
[mysqld]
port = 3306
socket = /tmp/mysql.sock
character-set-server=utf8
skip-name-resolve
basedir=/app/mysql
datadir=/data/mysqldata
back_log = 50
binlog_cache_size = 1M
max_heap_table_size = 64M
read_buffer_size = 2M
read_rnd_buffer_size = 16M
sort_buffer_size = 8M
join_buffer_size = 8M
thread_cache_size = 8
thread_concurrency = 8
ft_min_word_len = 4
binlog_format=ROW
default-storage-engine=innodb
innodb_autoinc_lock_mode=2
thread_stack = 240K
transaction_isolation = REPEATABLE-READ
tmp_table_size = 64M
log-bin=/data/binlog/mysql-bin
slow_query_log
long_query_time = 2
log_error=/data/logs/error.log
slow_query_log_file=/data/logs/slow.log
server-id = 2
key_buffer_size = 32M
bulk_insert_buffer_size = 64M
myisam_sort_buffer_size = 128M
myisam_max_sort_file_size = 10G
myisam_repair_threads = 1
myisam_recover
innodb_additional_mem_pool_size = 16M
innodb_buffer_pool_size = 1G
innodb_data_file_path = ibdata1:500M;ibdata2:16M:autoextend
innodb_file_per_table = 1
innodb_support_xa = on
sync_binlog = 0
table_definition_cache=65535
table_open_cache=4000
max_connections=3100
max_user_connections=3000
max_connect_errors=65536
max_allowed_packet=16M
innodb_strict_mode = 1
innodb_open_files = 4000
innodb_file_format = Barracuda
innodb_file_format_max = Barracuda
innodb_flush_method = O_DIRECT
expire_logs_days=3
innodb_data_home_dir = /data/mysqldata
innodb_write_io_threads = 8
innodb_read_io_threads = 8
innodb_thread_concurrency = 16
innodb_flush_log_at_trx_commit = 1
innodb_log_buffer_size = 128M
innodb_log_file_size = 256M
innodb_log_files_in_group = 3
innodb_log_group_home_dir = /data/redolog
innodb_max_dirty_pages_pct = 90
innodb_flush_method=O_DIRECT
innodb_lock_wait_timeout = 120

[galera]
bind-address=0.0.0.0
wsrep_provider=none
wsrep_node_address='192.168.56.101'
wsrep_provider=/app/mariadb-galera-10.0.29-linux/lib/galera/libgalera_smm.so
wsrep_provider_options='gcache.size=1999M'
wsrep_cluster_name='galera_cluster'
wsrep_cluster_address ='gcomm://192.168.56.101,192.168.56.102,192.168.56.103'
wsrep_node_name='MariaDB01'
wsrep_slave_threads=4
wsrep_certify_nonPK=1
wsrep_max_ws_rows=131072
wsrep_max_ws_size=1073741824
wsrep_debug=0
wsrep_convert_LOCK_to_trx=0
wsrep_retry_autocommit=1
wsrep_auto_increment_control=1
wsrep_replicate_myisam=1
wsrep_drupal_282555_workaround=0
wsrep_causal_reads=0
wsrep_sst_method=rsync
wsrep_sst_auth= sst_user:dbpass

[mysqldump]
quick
max_allowed_packet = 16M
[mysql]
no-auto-rehash
[myisamchk]
key_buffer_size = 512M
sort_buffer_size = 512M
read_buffer = 8M
write_buffer = 8M
[mysqlhotcopy]
interactive-timeout
[mysqld_safe]
open-files-limit = 8192

　　

./scripts/mysql_install_db --defaults-file=my.cnf 
./bin/mysqld_safe --defaults-file=my.cnf --wsrep-new-cluster &

      初始化节点2、3

./scripts/mysql_install_db --defaults-file=my.cnf 
./bin/mysqld_safe --defaults-file=my.cnf &

     为集群创建帐号

mysql> DELETE FROM mysql.user WHERE user='';
mysql> GRANT ALL ON *.* TO 'root'@'%' IDENTIFIED BY 'dbpass';
mysql> GRANT USAGE ON *.* to sst_user@'%' IDENTIFIED BY 'dbpass';
mysql> GRANT ALL PRIVILEGES on *.* to sst_user@'%';
mysql> FLUSH PRIVILEGES;

 三、日常维护

       1. 集群启动
         启动第一个节点：service mysql start --wsrep-new-cluster
         启动其它节点：server mysql start
         关闭节点：server mysql stop
       2. 通过查看4567端口确认集群是否启动
         netstat -plantu | grep mysqld
       3. 查询 galera插件是否已启用

SHOW status LIKE 'wsrep_ready';
SHOW VARIABLES LIKE 'wsrep_cluster_address';

       4. 检查集群是否正常

          查看Galera集群状态: mysql> show status like 'wsrep%';

(1)SHOW GLOBAL STATUS LIKE 'wsrep_cluster_state_uuid'; 集群中每个节点的值必须一致。
(2)SHOW GLOBAL STATUS LIKE 'wsrep_cluster_size'; 显示集群的规模，此例应该显示”3”。
(3)SHOW GLOBAL STATUS LIKE 'wsrep_cluster_status'; 显示”Primary”为正常。
(4)SHOW GLOBAL STATUS LIKE 'wsrep_ready'; 显示”ON”为正常。
(5) SHOW GLOBAL STATUS LIKE 'wsrep_connected'; 显示”ON”为正常。
(6) SHOW GLOBAL STATUS LIKE 'wsrep_local_state_comment'; 显示” Joined”为正常。

        cluster完整性检测

        status 变量:
        wsrep_cluster_state_uuid: cluster中的所有node的该变量的值必须一样, 不一样表示node没有连接到cluster。
        wsrep_cluster_conf_id: 次变量用来表示node是否在它相应的cluster中。cluster中所有node的该变量值应该一样， 不一样则表示nodes被分隔开了, node恢复的时候该变量也会恢复。
        wsrep_cluster_size: 表示cluster中有多少node节点， 等于预期的数量则表示所有node连接到了cluster。
        wsrep_cluster_status: 正常情况下值为Primary, 如果不为Primay,则该node当前不能操作(归咎于多成员关系的改变和quorum的缺失), 同时也可能满足split-brain的条件。
        如果cluster中没有node 连接上(connected) PC(就是所有node属于同一部分,但是node都是non-primary状态),

        wsrep_sst_method的参数说明：

Method      Speed    Blocks Donor?    Available on Live Node    Type       Requires root access to database server?
mysqldump   slow     yes              yes                       logical    donor and joiner
rsync       fastest  yes              no                        physical   none
xtrabackup  fast     briefly          no                        physical   donor only

      rsync在数据同步（SST和IST)的时候，速度最快，但是会锁住提供数据的节点，xtrabackup只会短暂的锁住节点。

 四、高可用测试

     1、节点数据同步写入

     2、故障节点修复后数据自行修复

附录

MariaDB Galera Cluste安装报错处理

170525 23:19:48 [ERROR] WSREP: wsrep_load(): dlopen(): libssl.so.6: cannot open shared object file: No such file or directory
170525 23:19:48 [ERROR] WSREP: wsrep_load(/app/mariadb-galera-10.0.29-linux/lib/galera/libgalera_smm.so) failed: Invalid argument (22). Reverting to no provider.
cd /app/mysql/lib/galera/
ldd ./libgalera_smm.so
cd /usr/lib64
ln -s libcrypto.so.1.0.1e libcrypto.so.6
ln -s libssl.so.1.0.1e libssl.so.6

170526 0:04:07 [Warning] WSREP: Gap in state sequence. Need state transfer.
170526 0:04:07 [Note] WSREP: Running: 'wsrep_sst_rsync --role 'joiner' --address '192.168.56.103' --datadir '/data/mysqldata/' --defaults-file '/app/mariadb-galera-10.0.29-linux/my.cnf' --defaults-group-suffix '' --parent '2832' --binlog '/data/binlog/mysql-bin' '
which: no rsync in (/usr/sbin:/sbin:/app/mariadb-galera-10.0.29-linux//bin:/app/mysql/bin:/bin:/usr/bin:/usr/local/bin:/usr/sbin)
170526 0:04:07 [ERROR] WSREP: Failed to read 'ready <addr>' from: wsrep_sst_rsync --role 'joiner' --address '192.168.56.103' --datadir '/data/mysqldata/' --defaults-file '/app/mariadb-galera-10.0.29-linux/my.cnf' --defaults-group-suffix '' --parent '2832' --binlog '/data/binlog/mysql-bin'
Read: ''rsync' not found in PATH'
170526 0:04:07 [ERROR] WSREP: Process completed with error: wsrep_sst_rsync --role 'joiner' --address '192.168.56.103' --datadir '/data/mysqldata/' --defaults-file '/app/mariadb-galera-10.0.29-linux/my.cnf' --defaults-group-suffix '' --parent '2832' --binlog '/data/binlog/mysql-bin' : 2 (No such file or directory)
170526 0:04:07 [ERROR] WSREP: Failed to prepare for 'rsync' SST. Unrecoverable.
170526 0:04:07 [ERROR] Aborting
yum install rsync.x86_64

which: no lsof in (/usr/sbin:/sbin:/app/mariadb-galera-10.0.29-linux//bin:/app/mysql/bin:/bin:/usr/bin:/usr/local/bin:/usr/sbin:/app/xtrabackup/percona-xtrabackup/bin)
170526 10:13:28 [ERROR] WSREP: Failed to read 'ready <addr>' from: wsrep_sst_rsync --role 'joiner' --address '192.168.56.103' --datadir '/data/mysqldata/' --defaults-file '/app/mariadb-galera-10.0.29-linux/my.cnf' --defaults-group-suffix '' --parent '2821' --binlog '/data/binlog/mysql-bin'
Read: ''lsof' not found in PATH'
170526 10:13:28 [ERROR] WSREP: Process completed with error: wsrep_sst_rsync --role 'joiner' --address '192.168.56.103' --datadir '/data/mysqldata/' --defaults-file '/app/mariadb-galera-10.0.29-linux/my.cnf' --defaults-group-suffix '' --parent '2821' --binlog '/data/binlog/mysql-bin' : 2 (No such file or directory)
170526 10:13:28 [ERROR] WSREP: Failed to prepare for 'rsync' SST. Unrecoverable.
170526 10:13:28 [ERROR] Aborting
yum install lsof