开篇说明
Galera Cluster 其实可以在每个结点进行读写操作，没有什么读写分离的概念。本文重点是讲 MariaDB 如何组建 Galera Cluster 环境，以及如何在MyCat中进行MariaDB集群多机联动的配置。
当然不用MyCAT作为数据库集群的前置也OK，还可用HA LVS等解决方案。
环境说明
mac os 是宿主机，使用 vmare fusion 开三台 centos 7
mariadb 10.1.10
下载最新版本的 mariadb 去 http://mariadb.org/ 下载sourcecode包
0 其他准备
由于centos7 开始已经使用 firewall 来替换 iptables ，可以用以下命令进行相关操作
# systemctl start firewalld # 启动,
# systemctl enable firewalld # 开机启动
# systemctl stop firewalld # 关闭
# systemctl disable firewalld # 取消开机启动]
firewall-cmd --zone=public --add-port=8080/tcp
更多firewall操作参考 http://www.lmyw.net.cn/?p=596
设定selinux等级，注意如果搞挂了linux启动不了
vi /etc/selinux/config
SELINUX=disabled
1 配置Linux hosts文件
vi /etc/hosts
在文件中加入：
192.168.31.187 db01 db01.kaye0110.com
192.168.31.188 db02 db02.kaye0110.com
192.168.31.189 db03 db03.kaye0110.com
2 检查是否已安装了mariadb 并进行删除
rpm -qa|grep mariadb
rpm -qa|grep mysql
yum -y remove xxxxx
3 开始安装mariadb mysql
3.0 安装相关需要用到插件
yum -y install cmake gcc gcc-c++ autoconf automake zlib* libxml* ncurses ncurses-devel libtool libtool-ltdl-devel* make bison bison-devel openssl-devel libevent-devel libaio libaio-devel pam-devel boost boost-devel valgrind-devel libnl-devel popt-devel popt-static bzr
yum -y install perl perl-devel libaio libaio-devel perl-Time-HiRes perl-DBD-MySQL
yum -y install libev
yum install perl-Digest-MD5
3.1 习惯把应用程度装到/usr/local/mariadb目录下
tar xvf mariadb-10.1.10.tar.gz
cd mariadb-10.1.10
3.2 编译安装
/*
cmake . -LH –查看cmake支持的mysql相关参数
shell> cmake . -L # overview
shell> cmake . -LH # overview with help text
shell> cmake . -LAH # all params with help text
shell> ccmake . # interactive display
重新编译时，需要清除旧的对象文件和缓存信息
# make clean
# rm -f CMakeCache.txt
**/
由于当前linux环境下已安装了一个普通版本的mysql 占用3307 端口
本次安装时将目录定为 /usr/local/mariadb/mariadb-10.1.10-13306
cmake -DCMAKE_INSTALL_PREFIX=/usr/local/mariadb/mariadb-10.1.10-13306 -DMYSQL_UNIX_ADDR=/usr/local/mariadb/mariadb-10.1.10-13306/socket/mysql.sock -DWITH_INNOBASE_STORAGE_ENGINE=1 -DWITH_MYISAM_STORAGE_ENGINE=1 -DWITH_SPHINX_STORAGE_ENGINE=1 -DWITH_READLINE=1 -DMYSQL_DATADIR=/usr/local/mariadb/mariadb-10.1.10-13306/data -DMYSQL_TCP_PORT=13306 -DENABLED_LOCAL_INFILE=1 -DWITH_EXTRA_CHARSETS=all -DDEFAULT_CHARSET=utf8 -DDEFAULT_COLLATION=utf8_general_ci -DEXTRA_CHARSETS=all -DWITH_WSREP=1 -DWITH_INNODB_DISALLOW_WRITES=1
慢慢等一会，时间够去看一集美剧了。
make -j4
make install
3.3 将目录权限分配给mysql用户
chown -R mysql:mysql mariadb-10.1.10-13306
3.4 初始化mariadb mysql
进入mariadb 安装目录 ，将通用配置好的mysql config文件复制出来，参数根据不同的环境进行调整 。
/usr/local/mariadb/mariadb-10.1.10-13306> cp ./support-files/my-small.cnf ./my.cnf
执行 scripts/mysql_install_db 脚本初始化。
./mysql_install_db --defaults-file=/usr/local/mariadb/mariadb-10.1.10-13306/my.cnf --basedir=/usr/local/mariadb/mariadb-10.1.10-13306 --datadir=/usr/local/mariadb/mariadb-10.1.10-13306/data --user=mysql
WARNING: The host 'miwifi-r1cm-srv' could not be looked up with resolveip.
This probably means that your libc libraries are not 100 % compatible
with this binary MariaDB version. The MariaDB daemon, mysqld, should work
normally with the exception that host name resolving will not work.
This means that you should use IP addresses instead of hostnames
when specifying MariaDB privileges !
Installing MariaDB/MySQL system tables in '/usr/local/mariadb/mariadb-10.1.10-13306/data' ...
2016-01-24 11:30:31 140448078133312 [Note] /usr/local/mariadb/mariadb-10.1.10-13306/bin/mysqld (mysqld 10.1.10-MariaDB) starting as process 29848 ...
2016-01-24 11:30:32 140448078133312 [Note] InnoDB: Using mutexes to ref count buffer pool pages
2016-01-24 11:30:32 140448078133312 [Note] InnoDB: The InnoDB memory heap is disabled
2016-01-24 11:30:32 140448078133312 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2016-01-24 11:30:32 140448078133312 [Note] InnoDB: Memory barrier is not used
2016-01-24 11:30:32 140448078133312 [Note] InnoDB: Compressed tables use zlib 1.2.7
2016-01-24 11:30:32 140448078133312 [Note] InnoDB: Using Linux native AIO
2016-01-24 11:30:32 140448078133312 [Note] InnoDB: Using SSE crc32 instructions
2016-01-24 11:30:32 140448078133312 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2016-01-24 11:30:32 140448078133312 [Note] InnoDB: Completed initialization of buffer pool
2016-01-24 11:30:32 140448078133312 [Note] InnoDB: The first specified data file ./ibdata1 did not exist: a new database to be created!
2016-01-24 11:30:32 140448078133312 [Note] InnoDB: Setting file ./ibdata1 size to 12 MB
2016-01-24 11:30:32 140448078133312 [Note] InnoDB: Database physically writes the file full: wait...
2016-01-24 11:30:32 140448078133312 [Note] InnoDB: Setting log file ./ib_logfile101 size to 48 MB
2016-01-24 11:30:32 140448078133312 [Note] InnoDB: Setting log file ./ib_logfile1 size to 48 MB
2016-01-24 11:30:32 140448078133312 [Note] InnoDB: Renaming log file ./ib_logfile101 to ./ib_logfile0
2016-01-24 11:30:32 140448078133312 [Warning] InnoDB: New log files created, LSN=45883
2016-01-24 11:30:32 140448078133312 [Note] InnoDB: Doublewrite buffer not found: creating new
2016-01-24 11:30:32 140448078133312 [Note] InnoDB: Doublewrite buffer created
2016-01-24 11:30:32 140448078133312 [Note] InnoDB: 128 rollback segment(s) are active.
2016-01-24 11:30:32 140448078133312 [Warning] InnoDB: Creating foreign key constraint system tables.
2016-01-24 11:30:32 140448078133312 [Note] InnoDB: Foreign key constraint system tables created
2016-01-24 11:30:32 140448078133312 [Note] InnoDB: Creating tablespace and datafile system tables.
2016-01-24 11:30:32 140448078133312 [Note] InnoDB: Tablespace and datafile system tables created.
2016-01-24 11:30:32 140448078133312 [Note] InnoDB: Waiting for purge to start
2016-01-24 11:30:32 140448078133312 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.26-76.0 started; log sequence number 0
2016-01-24 11:30:33 140447628183296 [Note] InnoDB: Dumping buffer pool(s) not yet started
OK
Filling help tables...
2016-01-24 11:30:35 139723365349440 [Note] /usr/local/mariadb/mariadb-10.1.10-13306/bin/mysqld (mysqld 10.1.10-MariaDB) starting as process 29888 ...
2016-01-24 11:30:35 139723365349440 [Note] InnoDB: Using mutexes to ref count buffer pool pages
2016-01-24 11:30:35 139723365349440 [Note] InnoDB: The InnoDB memory heap is disabled
2016-01-24 11:30:35 139723365349440 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2016-01-24 11:30:35 139723365349440 [Note] InnoDB: Memory barrier is not used
2016-01-24 11:30:35 139723365349440 [Note] InnoDB: Compressed tables use zlib 1.2.7
2016-01-24 11:30:35 139723365349440 [Note] InnoDB: Using Linux native AIO
2016-01-24 11:30:35 139723365349440 [Note] InnoDB: Using SSE crc32 instructions
2016-01-24 11:30:35 139723365349440 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2016-01-24 11:30:35 139723365349440 [Note] InnoDB: Completed initialization of buffer pool
2016-01-24 11:30:35 139723365349440 [Note] InnoDB: Highest supported file format is Barracuda.
2016-01-24 11:30:35 139723365349440 [Note] InnoDB: 128 rollback segment(s) are active.
2016-01-24 11:30:35 139723365349440 [Note] InnoDB: Waiting for purge to start
2016-01-24 11:30:35 139723365349440 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.26-76.0 started; log sequence number 1616799
2016-01-24 11:30:35 139722915084032 [Note] InnoDB: Dumping buffer pool(s) not yet started
OK
Creating OpenGIS required SP-s...
2016-01-24 11:30:37 140529693435968 [Note] /usr/local/mariadb/mariadb-10.1.10-13306/bin/mysqld (mysqld 10.1.10-MariaDB) starting as process 29918 ...
2016-01-24 11:30:38 140529693435968 [Note] InnoDB: Using mutexes to ref count buffer pool pages
2016-01-24 11:30:38 140529693435968 [Note] InnoDB: The InnoDB memory heap is disabled
2016-01-24 11:30:38 140529693435968 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2016-01-24 11:30:38 140529693435968 [Note] InnoDB: Memory barrier is not used
2016-01-24 11:30:38 140529693435968 [Note] InnoDB: Compressed tables use zlib 1.2.7
2016-01-24 11:30:38 140529693435968 [Note] InnoDB: Using Linux native AIO
2016-01-24 11:30:38 140529693435968 [Note] InnoDB: Using SSE crc32 instructions
2016-01-24 11:30:38 140529693435968 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2016-01-24 11:30:38 140529693435968 [Note] InnoDB: Completed initialization of buffer pool
2016-01-24 11:30:38 140529693435968 [Note] InnoDB: Highest supported file format is Barracuda.
2016-01-24 11:30:38 140529693435968 [Note] InnoDB: 128 rollback segment(s) are active.
2016-01-24 11:30:38 140529693435968 [Note] InnoDB: Waiting for purge to start
2016-01-24 11:30:38 140529693435968 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.26-76.0 started; log sequence number 1616809
2016-01-24 11:30:38 140529174046464 [Note] InnoDB: Dumping buffer pool(s) not yet started
OK
To start mysqld at boot time you have to copy
support-files/mysql.server to the right place for your system
PLEASE REMEMBER TO SET A PASSWORD FOR THE MariaDB root USER !
To do so, start the server, then issue the following commands:
'/usr/local/mariadb/mariadb-10.1.10-13306/bin/mysqladmin' -u root password 'new-password'
'/usr/local/mariadb/mariadb-10.1.10-13306/bin/mysqladmin' -u root -h miwifi-r1cm-srv password 'new-password'
Alternatively you can run:
'/usr/local/mariadb/mariadb-10.1.10-13306/bin/mysql_secure_installation'
which will also give you the option of removing the test
databases and anonymous user created by default. This is
strongly recommended for production servers.
See the MariaDB Knowledgebase at http://mariadb.com/kb or the
MySQL manual for more instructions.
You can start the MariaDB daemon with:
cd '/usr/local/mariadb/mariadb-10.1.10-13306' ; /usr/local/mariadb/mariadb-10.1.10-13306/bin/mysqld_safe --datadir='/usr/local/mariadb/mariadb-10.1.10-13306/data'
You can test the MariaDB daemon with mysql-test-run.pl
cd '/usr/local/mariadb/mariadb-10.1.10-13306/mysql-test' ; perl mysql-test-run.pl
Please report any problems at http://mariadb.org/jira
The latest information about MariaDB is available at http://mariadb.org/.
You can find additional information about the MySQL part at:
http://dev.mysql.com
Support MariaDB development by buying support/new features from MariaDB
Corporation Ab. You can contact us about this at sales@mariadb.com.
Alternatively consider joining our community based development effort:
http://mariadb.com/kb/en/contributing-to-the-mariadb-project/
3.5 启动mariadb
cd /usr/local/mariadb/mariadb-10.1.10-13306
/usr/local/mariadb/mariadb-10.1.10-13306/bin/mysqld_safe --datadir='/usr/local/mariadb/mariadb-10.1.10-13306/data' &
3.6 登陆mysql client 修改root用户密码 , 建立远程登陆用户
./mysql -u root -p -P 13306
MariaDB [(none)]> SET PASSWORD FOR root@localhost = PASSWORD('root123');
Query OK, 0 rows affected (0.00 sec)
MariaDB [test]> CREATE USER 'root'@'%' IDENTIFIED BY 'root123';
Query OK, 0 rows affected (0.00 sec)
MariaDB [test]> select host,user,password from mysql.user;
+-----------------+------+-------------------------------------------+
| host | user | password |
+-----------------+------+-------------------------------------------+
| localhost | root | *FAAFFE644E901CFAFAEC7562415E5FAEC243B8B2 |
| miwifi-r1cm-srv | root | |
| 127.0.0.1 | root | |
| ::1 | root | |
| localhost | | |
| miwifi-r1cm-srv | | |
| % | root | *FAAFFE644E901CFAFAEC7562415E5FAEC243B8B2 |
+-----------------+------+-------------------------------------------+
7 rows in set (0.00 sec)
4 安装galera cluster
wget http://cznic.dl.sourceforge.net/project/scons/scons/2.3.5/scons-2.3.5.tar.gz
tar zxvf scons-2.3.5.tar.gz
cd scons-2.3.5
python2.6 setup.py install
下载rpm 安装包，centos 7 64x 下载地址：http://releases.galeracluster.com/centos/7/x86_64/galera-3-25.3.14-2.el7.x86_64.rpm
rpm -ivh galera-3-25.3.14-2.el7.x86_64.rpm
cp /usr/lib64/galera-3/libgalera_smm.so /usr/local/mariadb/mariadb-10.1.10-13306/lib/plugin/
4.1 galera cluster 同步方法主要有以下几种
rsync、xtrabackup、mydupm ，这里就讲下网上主要推荐的xtrabackup安装测略
rpm -ivh percona-xtrabackup-2.3.3-1.el7.x86_64.rpm
警告：percona-xtrabackup-2.3.3-1.el7.x86_64.rpm: 头V4 DSA/SHA1 Signature, 密钥 ID cd2efd2a: NOKEY
准备中... ################################# [100%]
正在升级/安装...
1:percona-xtrabackup-2.3.3-1.el7 ################################# [100%]
5 配置mariadb galera cluster 环境
5.1 新增数据同步专用用户
mysql> GRANT USAGE ON *.* to dbsync@'%' IDENTIFIED BY 'dbsync123';
mysql> GRANT ALL PRIVILEGES on *.* to dbsync@'%';
mysql> FLUSH PRIVILEGES;
mysql> quit
5.2 调整mariadb my.cnf
进入mariadb目录 ，建立 my.cnf.d 目录， 并复制 wsrep.cnf 进入目录
在my.cnf中加入
!includedir /usr/local/mariadb/mariadb-10.1.10-13306/my.cnf.d
5.3 配置wsrep.cnf
node 1
[mysqld]
wsrep_on=ON
binlog_format=ROW
default-storage-engine=innodb
innodb_autoinc_lock_mode=2
innodb_locks_unsafe_for_binlog=1
innodb_flush_log_at_trx_commit = 2 #可以提高性能，galera保证不丢数据
query_cache_size=0
query_cache_type=0
wsrep_provider=/usr/local/mariadb/mariadb-10.1.10-13306/lib/plugin/libgalera_smm.so #修改
wsrep_cluster_name="galera_cluster" #修改
wsrep_cluster_address="gcomm://192.168.31.187,192.168.31.188,192.168.31.189,192.168.31.190" #修改
wsrep_node_address=192.168.31.187 #修改
wsrep_node_name=db01 #修改
wsrep_slave_threads=4
wsrep_certify_nonPK=1
wsrep_max_ws_rows=131072
wsrep_max_ws_size=1073741824
wsrep_debug=0
wsrep_convert_LOCK_to_trx=0
wsrep_retry_autocommit=1
wsrep_auto_increment_control=1
wsrep_drupal_282555_workaround=0
wsrep_causal_reads=0
wsrep_sst_method=xtrabackup #可以修改#rsync,xtrabackup,mysqldump xtrabackup-v2
wsrep_sst_auth= dbsync: dbsync123 #修改
node 2
[mysqld]
wsrep_on=ON
binlog_format=ROW
default-storage-engine=innodb
innodb_autoinc_lock_mode=2
innodb_locks_unsafe_for_binlog=1
innodb_flush_log_at_trx_commit = 2 #可以提高性能，galera保证不丢数据
query_cache_size=0
query_cache_type=0
wsrep_provider=/usr/local/mariadb/mariadb-10.1.10-13306/lib/plugin/libgalera_smm.so #修改
wsrep_cluster_name="galera_cluster" #修改
wsrep_cluster_address="gcomm://192.168.31.187,192.168.31.188,192.168.31.189,192.168.31.190" #修改
wsrep_node_address=192.168.31.187 #修改
wsrep_node_name=db01 #修改
wsrep_slave_threads=4
wsrep_certify_nonPK=1
wsrep_max_ws_rows=131072
wsrep_max_ws_size=1073741824
wsrep_debug=0
wsrep_convert_LOCK_to_trx=0
wsrep_retry_autocommit=1
wsrep_auto_increment_control=1
wsrep_drupal_282555_workaround=0
wsrep_causal_reads=0
wsrep_sst_method=xtrabackup #可以修改#rsync,xtrabackup,mysqldump xtrabackup-v2
wsrep_sst_auth= dbsync: dbsync123 #修改
node 3
[mysqld]
wsrep_on=ON
binlog_format=ROW
default-storage-engine=innodb
innodb_autoinc_lock_mode=2
innodb_locks_unsafe_for_binlog=1
innodb_flush_log_at_trx_commit = 2 #可以提高性能，galera保证不丢数据
query_cache_size=0
query_cache_type=0
wsrep_provider=/usr/local/mariadb/mariadb-10.1.10-13306/lib/plugin/libgalera_smm.so #修改
wsrep_cluster_name="galera_cluster" #修改
wsrep_cluster_address="gcomm://192.168.31.187,192.168.31.188,192.168.31.189,192.168.31.190" #修改
wsrep_node_address=192.168.31.187 #修改
wsrep_node_name=db01 #修改
wsrep_slave_threads=4
wsrep_certify_nonPK=1
wsrep_max_ws_rows=131072
wsrep_max_ws_size=1073741824
wsrep_debug=0
wsrep_convert_LOCK_to_trx=0
wsrep_retry_autocommit=1
wsrep_auto_increment_control=1
wsrep_drupal_282555_workaround=0
wsrep_causal_reads=0
wsrep_sst_method=xtrabackup #可以修改#rsync,xtrabackup,mysqldump xtrabackup-v2
wsrep_sst_auth= dbsync: dbsync123 #修改
6 启动并验证
6.1 启动首台服务
/usr/local/mariadb/mariadb-10.1.10-13306/bin/mysqld_safe --defaults-file=/usr/local/mariadb/mariadb-10.1.10-13306/my.cnf --datadir=/usr/local/mariadb/mariadb-10.1.10-13306/data --log-error=/usr/local/mariadb/mariadb-10.1.10-13306/data/mariadb-error.log --wsrep-cluster-address="gcomm://" &
6.2 启动其他服务
/usr/local/mariadb/mariadb-10.1.10-13306/bin/mysqld_safe --defaults-file=/usr/local/mariadb/mariadb-10.1.10-13306/my.cnf --datadir=/usr/local/mariadb/mariadb-10.1.10-13306/data --log-error=/usr/local/mariadb/mariadb-10.1.10-13306/data/mariadb-error.log &
6.3 数据库验证
6.3.1 参数验证
主机查询联动参数
mysql>SHOW VARIABLES LIKE 'wsrep_cluster_address';
+-----------------------+----------+
| Variable_name | Value |
+-----------------------+----------+
| wsrep_cluster_address | gcomm:// |
+-----------------------+----------+
1 row in set (0.00 sec)
从机查询联动参数
MariaDB [(none)]> SHOW VARIABLES LIKE 'wsrep_cluster_address';
+-----------------------+------------------------------------------------------+
| Variable_name | Value |
+-----------------------+------------------------------------------------------+
| wsrep_cluster_address | gcomm://192.168.31.187,192.168.31.212,192.168.31.150 |
+-----------------------+------------------------------------------------------+
1 row in set (0.00 sec)
可以用下面的命令查询基他配置
MariaDB [(none)]>SHOW STATUS LIKE 'wsrep%';
6.3.2 数据库操作验证
主机新建schema
MariaDB [(none)]> create database sync_test;
Query OK, 1 row affected (0.01 sec)
从机create table，并新增记录
MariaDB [(none)]> use sync_test
Database changed
MariaDB [sync_test]> CREATE TABLE `t_new_table` ( `id` INT NOT NULL, `context` VARCHAR(45) NULL, PRIMARY KEY (`id`));
Query OK, 0 rows affected (0.03 sec)
MariaDB [sync_test]> insert into t_new_table (id,context) values ('1','hello world');
Query OK, 1 row affected (0.01 sec)
MariaDB [sync_test]> select * from t_new_table;
+----+-------------+
| id | context |
+----+-------------+
| 1 | hello world |
+----+-------------+
1 row in set (0.00 sec)
在主机 和 3号机上验证数据
MariaDB [sync_test]> select * from t_new_table;
+----+-------------+
| id | context |
+----+-------------+
| 1 | hello world |
+----+-------------+
1 row in set (0.00 sec)
OK，到这里为止 galera cluster 总算是搭初步建完成， 后面再补 mycat 集成 、HA 集成方案吧。
其他相关文档可供参考 ：
《Galera 10.0.20 on CentOS 6.6》http://blog.csdn.net/yangzhawen/article/details/46788927
《centos6.6下编译安装mariadb-10.0.20》 http://blog.csdn.net/ligaofeng/article/details/47173557
《xtrabackup 安装、备份、还原及错误处理 教程》http://blog.163.com/ji_1006/blog/static/10612341201382355716623/
《MySQL的Galera Cluster配置说明》 http://www.360doc.com/content/13/0817/15/834950_307820923.shtml
【2016年1月31日 补充】
今天验证了数据库单结点脱离集群环境再加入后数据里补全的验证，有几个坑在这里面补充下：
1）wsrep_sst_method=xtrabackup 建议改为 xtrabackup-v2 或直接改为 rsync 模式。
2）如果发现有问题的话，可以看下 innodb backup log 文件（具体文件在datadir目录下），里面会有具体的数据备份时发现的问题。
3）注意防火墙 5678 端口、4444端口，最简单的做法是直接把firewall给关掉~~当然生产环境这样做不是很好。