Preface
I've installed MasterHA yesterday,Now let's test the master-slave switch and failover feature.
Framework
| Hostname | IP | Port | Identity | OS Version | MySQL Version |
| zlm2 | 192.168.1.101 | 3306 | master | CentOS 7.0 | 5.7.21 |
| zlm3 | 192.168.1.102 | 3306 | slave/mha-manager | CentOS 7.0 | 5.7.21 |
| null | 192.168.1.200 | null | vip | null | null |
Procedure
Test 1：Manual master switchover
Check state of MHA-manager on zlm3.
1 [root@zlm3 07:35:00 ~] 2 #masterha_check_ssh --conf=/etc/masterha/app1.conf 3 Fri Aug 3 07:37:13 2018 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping. 4 Fri Aug 3 07:37:13 2018 - [info] Reading application default configuration from /etc/masterha/app1.conf.. 5 Fri Aug 3 07:37:13 2018 - [info] Reading server configuration from /etc/masterha/app1.conf.. 6 Fri Aug 3 07:37:13 2018 - [info] Starting SSH connection tests.. 7 Fri Aug 3 07:37:13 2018 - [debug] 8 Fri Aug 3 07:37:13 2018 - [debug] Connecting via SSH from root@192.168.1.101(192.168.1.101:22) to root@192.168.1.102(192.168.1.102:22).. 9 Fri Aug 3 07:37:13 2018 - [debug] ok. 10 Fri Aug 3 07:37:14 2018 - [debug] 11 Fri Aug 3 07:37:13 2018 - [debug] Connecting via SSH from root@192.168.1.102(192.168.1.102:22) to root@192.168.1.101(192.168.1.101:22).. 12 Fri Aug 3 07:37:13 2018 - [debug] ok. 13 Fri Aug 3 07:37:14 2018 - [info] All SSH connection tests passed successfully. 14 15 [root@zlm3 07:37:14 ~] 16 #masterha_check_repl --conf=/etc/masterha/app1.conf --global_conf=/etc/masterha/masterha_default.conf 17 Fri Aug 3 07:37:37 2018 - [info] Reading default configuration from /etc/masterha/masterha_default.conf.. 18 Fri Aug 3 07:37:37 2018 - [info] Reading application default configuration from /etc/masterha/app1.conf.. 19 Fri Aug 3 07:37:37 2018 - [info] Reading server configuration from /etc/masterha/app1.conf.. 20 Fri Aug 3 07:37:37 2018 - [info] MHA::MasterMonitor version 0.56. 21 Fri Aug 3 07:37:38 2018 - [info] GTID failover mode = 1 22 Fri Aug 3 07:37:38 2018 - [info] Dead Servers: 23 Fri Aug 3 07:37:38 2018 - [info] Alive Servers: 24 Fri Aug 3 07:37:38 2018 - [info] 192.168.1.101(192.168.1.101:3306) 25 Fri Aug 3 07:37:38 2018 - [info] 192.168.1.102(192.168.1.102:3306) 26 Fri Aug 3 07:37:38 2018 - [info] Alive Slaves: 27 Fri Aug 3 07:37:38 2018 - [info] 192.168.1.102(192.168.1.102:3306) Version=5.7.21-log (oldest major version between slaves) log-bin:enabled 28 Fri Aug 3 07:37:38 2018 - [info] GTID ON 29 Fri Aug 3 07:37:38 2018 - [info] Replicating from 192.168.1.101(192.168.1.101:3306) 30 Fri Aug 3 07:37:38 2018 - [info] Primary candidate for the new Master (candidate_master is set) 31 Fri Aug 3 07:37:38 2018 - [info] Current Alive Master: 192.168.1.101(192.168.1.101:3306) 32 Fri Aug 3 07:37:38 2018 - [info] Checking slave configurations.. 33 Fri Aug 3 07:37:38 2018 - [info] read_only=1 is not set on slave 192.168.1.102(192.168.1.102:3306). 34 Fri Aug 3 07:37:38 2018 - [info] Checking replication filtering settings.. 35 Fri Aug 3 07:37:38 2018 - [info] binlog_do_db= , binlog_ignore_db= 36 Fri Aug 3 07:37:38 2018 - [info] Replication filtering check ok. 37 Fri Aug 3 07:37:38 2018 - [info] GTID (with auto-pos) is supported. Skipping all SSH and Node package checking. 38 Fri Aug 3 07:37:38 2018 - [info] Checking SSH publickey authentication settings on the current master.. 39 ssh_exchange_identification: Connection closed by remote host 40 Fri Aug 3 07:37:38 2018 - [warning] HealthCheck: SSH to 192.168.1.101 is NOT reachable. 41 Fri Aug 3 07:37:38 2018 - [info] 42 192.168.1.101(192.168.1.101:3306) (current master) 43 +--192.168.1.102(192.168.1.102:3306) 44 45 Fri Aug 3 07:37:38 2018 - [info] Checking replication health on 192.168.1.102.. 46 Fri Aug 3 07:37:38 2018 - [info] ok. 47 Fri Aug 3 07:37:38 2018 - [info] Checking master_ip_failover_script status: 48 Fri Aug 3 07:37:38 2018 - [info] /etc/masterha/master_ip_failover --command=status --ssh_user=root --orig_master_host=192.168.1.101 --orig_master_ip=192.168.1.101 --orig_master_port=3306 --orig_master_ssh_port=3306 49 Fri Aug 3 07:37:38 2018 - [info] OK. 50 Fri Aug 3 07:37:38 2018 - [warning] shutdown_script is not defined. 51 Fri Aug 3 07:37:38 2018 - [info] Got exit code 0 (Not master dead). 52 53 MySQL Replication Health is OK. 54 55 [root@zlm3 07:40:03 ~] 56 #Fri Aug 3 07:40:03 2018 - [info] Reading default configuration from /etc/masterha/masterha_default.conf.. 57 Fri Aug 3 07:40:03 2018 - [info] Reading application default configuration from /etc/masterha/app1.conf.. 58 Fri Aug 3 07:40:03 2018 - [info] Reading server configuration from /etc/masterha/app1.conf.. 59 ssh_exchange_identification: Connection closed by remote host 60 ^C 61 62 [root@zlm3 07:40:11 ~] 63 #masterha_check_status --conf=/etc/masterha/app1.conf 64 app1 (pid:5628) is running(0:PING_OK), master:192.168.1.101
Switch master to slave and make it become a new slave of new master.
1 [root@zlm3 08:21:27 ~] 2 #masterha_master_switch --conf=/etc/masterha/app1.conf --global_conf=/etc/masterha/masterha_default.conf --master_state=alive --new_master_host=192.168.1.102 --orig_master_is_new_slave --running_updates_limit=60 3 Fri Aug 3 08:21:29 2018 - [info] MHA::MasterRotate version 0.56. 4 Fri Aug 3 08:21:29 2018 - [info] Starting online master switch.. 5 Fri Aug 3 08:21:29 2018 - [info] 6 Fri Aug 3 08:21:29 2018 - [info] * Phase 1: Configuration Check Phase.. 7 Fri Aug 3 08:21:29 2018 - [info] 8 Fri Aug 3 08:21:29 2018 - [info] Reading default configuration from /etc/masterha/masterha_default.conf.. 9 Fri Aug 3 08:21:29 2018 - [info] Reading application default configuration from /etc/masterha/app1.conf.. 10 Fri Aug 3 08:21:29 2018 - [info] Reading server configuration from /etc/masterha/app1.conf.. 11 Fri Aug 3 08:21:30 2018 - [info] GTID failover mode = 1 12 Fri Aug 3 08:21:30 2018 - [info] Current Alive Master: 192.168.1.101(192.168.1.101:3306) 13 Fri Aug 3 08:21:30 2018 - [info] Alive Slaves: 14 Fri Aug 3 08:21:30 2018 - [info] 192.168.1.102(192.168.1.102:3306) Version=5.7.21-log (oldest major version between slaves) log-bin:enabled 15 Fri Aug 3 08:21:30 2018 - [info] GTID ON 16 Fri Aug 3 08:21:30 2018 - [info] Replicating from 192.168.1.101(192.168.1.101:3306) 17 Fri Aug 3 08:21:30 2018 - [info] Primary candidate for the new Master (candidate_master is set) 18 19 It is better to execute FLUSH NO_WRITE_TO_BINLOG TABLES on the master before switching. Is it ok to execute on 192.168.1.101(192.168.1.101:3306)? (YES/no): yes 20 Fri Aug 3 08:21:33 2018 - [info] Executing FLUSH NO_WRITE_TO_BINLOG TABLES. This may take long time.. 21 Fri Aug 3 08:21:33 2018 - [info] ok. 22 Fri Aug 3 08:21:33 2018 - [info] Checking MHA is not monitoring or doing failover.. 23 Fri Aug 3 08:21:33 2018 - [error][/usr/share/perl5/vendor_perl/MHA/MasterRotate.pm, ln142] Getting advisory lock failed on the current master. MHA Monitor runs on the current master. Stop MHA Manager/Monitor and try again. 24 Fri Aug 3 08:21:33 2018 - [error][/usr/share/perl5/vendor_perl/MHA/ManagerUtil.pm, ln177] Got ERROR: at /usr/bin/masterha_master_switch line 53. 25 26 //It means that we should stop MHA-manager when donging switchover master. 27 28 [root@zlm3 08:21:33 ~] 29 #masterha_stop --conf=/etc/masterha/app1.conf --global_conf=/etc/masterha/masterha_default.conf 30 Stopped app1 successfully. 31 [1]+ Exit 1 masterha_manager --conf=/etc/masterha/app1.conf --global_conf=/etc/masterha/masterha_default.conf 32 33 [root@zlm3 08:28:07 ~] 34 #masterha_master_switch --conf=/etc/masterha/app1.conf --global_conf=/etc/masterha/masterha_default.conf --master_state=alive --new_master_host=192.168.1.102 --orig_master_is_new_slave --running_updates_limit=60 35 Fri Aug 3 08:28:21 2018 - [info] MHA::MasterRotate version 0.56. 36 Fri Aug 3 08:28:21 2018 - [info] Starting online master switch.. 37 Fri Aug 3 08:28:21 2018 - [info] 38 Fri Aug 3 08:28:21 2018 - [info] * Phase 1: Configuration Check Phase.. 39 Fri Aug 3 08:28:21 2018 - [info] 40 Fri Aug 3 08:28:21 2018 - [info] Reading default configuration from /etc/masterha/masterha_default.conf.. 41 Fri Aug 3 08:28:21 2018 - [info] Reading application default configuration from /etc/masterha/app1.conf.. 42 Fri Aug 3 08:28:21 2018 - [info] Reading server configuration from /etc/masterha/app1.conf.. 43 Fri Aug 3 08:28:22 2018 - [info] GTID failover mode = 1 44 Fri Aug 3 08:28:22 2018 - [info] Current Alive Master: 192.168.1.101(192.168.1.101:3306) 45 Fri Aug 3 08:28:22 2018 - [info] Alive Slaves: 46 Fri Aug 3 08:28:22 2018 - [info] 192.168.1.102(192.168.1.102:3306) Version=5.7.21-log (oldest major version between slaves) log-bin:enabled 47 Fri Aug 3 08:28:22 2018 - [info] GTID ON 48 Fri Aug 3 08:28:22 2018 - [info] Replicating from 192.168.1.101(192.168.1.101:3306) 49 Fri Aug 3 08:28:22 2018 - [info] Primary candidate for the new Master (candidate_master is set) 50 51 It is better to execute FLUSH NO_WRITE_TO_BINLOG TABLES on the master before switching. Is it ok to execute on 192.168.1.101(192.168.1.101:3306)? (YES/no): yes 52 Fri Aug 3 08:28:25 2018 - [info] Executing FLUSH NO_WRITE_TO_BINLOG TABLES. This may take long time.. 53 Fri Aug 3 08:28:25 2018 - [info] ok. 54 Fri Aug 3 08:28:25 2018 - [info] Checking MHA is not monitoring or doing failover.. 55 Fri Aug 3 08:28:25 2018 - [info] Checking replication health on 192.168.1.102.. 56 Fri Aug 3 08:28:25 2018 - [info] ok. 57 Fri Aug 3 08:28:25 2018 - [info] 192.168.1.102 can be new master. 58 Fri Aug 3 08:28:25 2018 - [info] 59 From: 60 192.168.1.101(192.168.1.101:3306) (current master) 61 +--192.168.1.102(192.168.1.102:3306) 62 63 To: 64 192.168.1.102(192.168.1.102:3306) (new master) 65 +--192.168.1.101(192.168.1.101:3306) 66 67 Starting master switch from 192.168.1.101(192.168.1.101:3306) to 192.168.1.102(192.168.1.102:3306)? (yes/NO): yes 68 Fri Aug 3 08:28:31 2018 - [info] Checking whether 192.168.1.102(192.168.1.102:3306) is ok for the new master.. 69 Fri Aug 3 08:28:31 2018 - [info] ok. 70 Fri Aug 3 08:28:31 2018 - [info] 192.168.1.101(192.168.1.101:3306): SHOW SLAVE STATUS returned empty result. To check replication filtering rules, temporarily executing CHANGE MASTER to a dummy host. 71 Fri Aug 3 08:28:31 2018 - [info] 192.168.1.101(192.168.1.101:3306): Resetting slave pointing to the dummy host. 72 Fri Aug 3 08:28:31 2018 - [info] ** Phase 1: Configuration Check Phase completed. 73 Fri Aug 3 08:28:31 2018 - [info] 74 Fri Aug 3 08:28:31 2018 - [info] * Phase 2: Rejecting updates Phase.. 75 Fri Aug 3 08:28:31 2018 - [info] 76 Fri Aug 3 08:28:31 2018 - [info] Executing master ip online change script to disable write on the current master: 77 Fri Aug 3 08:28:31 2018 - [info] /etc/masterha/master_ip_online_change --command=stop --orig_master_host=192.168.1.101 --orig_master_ip=192.168.1.101 --orig_master_port=3306 --orig_master_user='zlm' --orig_master_password='zlmzlm' --new_master_host=192.168.1.102 --new_master_ip=192.168.1.102 --new_master_port=3306 --new_master_user='zlm' --new_master_password='zlmzlm' --orig_master_ssh_user=root --new_master_ssh_user=root --orig_master_ssh_port=3306 --new_master_ssh_port=3306 --orig_master_is_new_slave 78 Unknown option: new_master_ssh_port 79 Fri Aug 3 08:28:32 2018 116409 Set read_only on the new master.. ok. 80 Fri Aug 3 08:28:32 2018 125643 drop vip 10.33.101.239.. 81 ssh_exchange_identification: Connection closed by remote host 82 Fri Aug 3 08:28:32 2018 142948 Waiting all running 1 threads are disconnected.. (max 1500 milliseconds) 83 {'Time' => '13435','db' => undef,'Id' => '21','User' => 'repl','State' => 'Master has sent all binlog to slave; waiting for more updates','Command' => 'Binlog Dump GTID','Info' => undef,'Host' => 'zlm3:40535'} 84 Fri Aug 3 08:28:32 2018 646769 Waiting all running 1 threads are disconnected.. (max 1000 milliseconds) 85 {'Time' => '13435','db' => undef,'Id' => '21','User' => 'repl','State' => 'Master has sent all binlog to slave; waiting for more updates','Command' => 'Binlog Dump GTID','Info' => undef,'Host' => 'zlm3:40535'} 86 Fri Aug 3 08:28:33 2018 149221 Waiting all running 1 threads are disconnected.. (max 500 milliseconds) 87 {'Time' => '13436','db' => undef,'Id' => '21','User' => 'repl','State' => 'Master has sent all binlog to slave; waiting for more updates','Command' => 'Binlog Dump GTID','Info' => undef,'Host' => 'zlm3:40535'} 88 Fri Aug 3 08:28:33 2018 650816 Set read_only=1 on the orig master.. ok. 89 Fri Aug 3 08:28:33 2018 653323 Waiting all running 1 queries are disconnected.. (max 500 milliseconds) 90 {'Time' => '13436','db' => undef,'Id' => '21','User' => 'repl','State' => 'Master has sent all binlog to slave; waiting for more updates','Command' => 'Binlog Dump GTID','Info' => undef,'Host' => 'zlm3:40535'} 91 Fri Aug 3 08:28:34 2018 154965 Killing all application threads.. 92 Fri Aug 3 08:28:34 2018 167919 done. 93 Fri Aug 3 08:28:34 2018 - [info] ok. 94 Fri Aug 3 08:28:34 2018 - [info] Locking all tables on the orig master to reject updates from everybody (including root): 95 Fri Aug 3 08:28:34 2018 - [info] Executing FLUSH TABLES WITH READ LOCK.. 96 Fri Aug 3 08:28:34 2018 - [info] ok. 97 Fri Aug 3 08:28:34 2018 - [info] Orig master binlog:pos is mysql-bin.000050:2361. 98 Fri Aug 3 08:28:34 2018 - [info] Waiting to execute all relay logs on 192.168.1.102(192.168.1.102:3306).. 99 Fri Aug 3 08:28:34 2018 - [info] master_pos_wait(mysql-bin.000050:2361) completed on 192.168.1.102(192.168.1.102:3306). Executed 0 events. 100 Fri Aug 3 08:28:34 2018 - [info] done. 101 Fri Aug 3 08:28:34 2018 - [info] Getting new master's binlog name and position.. 102 Fri Aug 3 08:28:34 2018 - [info] mysql-bin.000003:2321 103 Fri Aug 3 08:28:34 2018 - [info] All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='192.168.1.102', MASTER_PORT=3306, MASTER_AUTO_POSITION=1, MASTER_USER='repl', MASTER_PASSWORD='xxx'; 104 Fri Aug 3 08:28:34 2018 - [info] Executing master ip online change script to allow write on the new master: 105 Fri Aug 3 08:28:34 2018 - [info] /etc/masterha/master_ip_online_change --command=start --orig_master_host=192.168.1.101 --orig_master_ip=192.168.1.101 --orig_master_port=3306 --orig_master_user='zlm' --orig_master_password='zlmzlm' --new_master_host=192.168.1.102 --new_master_ip=192.168.1.102 --new_master_port=3306 --new_master_user='zlm' --new_master_password='zlmzlm' --orig_master_ssh_user=root --new_master_ssh_user=root --orig_master_ssh_port=3306 --new_master_ssh_port=3306 --orig_master_is_new_slave 106 Unknown option: new_master_ssh_port 107 Fri Aug 3 08:28:34 2018 327146 Set read_only=0 on the new master. 108 Fri Aug 3 08:28:34 2018 328259Add vip 10.33.101.239 on p3p1.. 109 ssh_exchange_identification: Connection closed by remote host 110 Fri Aug 3 08:28:34 2018 - [info] ok. 111 Fri Aug 3 08:28:34 2018 - [info] 112 Fri Aug 3 08:28:34 2018 - [info] * Switching slaves in parallel.. 113 Fri Aug 3 08:28:34 2018 - [info] 114 Fri Aug 3 08:28:34 2018 - [info] Unlocking all tables on the orig master: 115 Fri Aug 3 08:28:34 2018 - [info] Executing UNLOCK TABLES.. 116 Fri Aug 3 08:28:34 2018 - [info] ok. 117 Fri Aug 3 08:28:34 2018 - [info] Starting orig master as a new slave.. 118 Fri Aug 3 08:28:34 2018 - [info] Resetting slave 192.168.1.101(192.168.1.101:3306) and starting replication from the new master 192.168.1.102(192.168.1.102:3306).. 119 Fri Aug 3 08:28:34 2018 - [info] Executed CHANGE MASTER. 120 Fri Aug 3 08:28:35 2018 - [info] Slave started. 121 Fri Aug 3 08:28:35 2018 - [info] All new slave servers switched successfully. 122 Fri Aug 3 08:28:35 2018 - [info] 123 Fri Aug 3 08:28:35 2018 - [info] * Phase 5: New master cleanup phase.. 124 Fri Aug 3 08:28:35 2018 - [info] 125 Fri Aug 3 08:28:35 2018 - [info] 192.168.1.102: Resetting slave info succeeded. 126 Fri Aug 3 08:28:35 2018 - [info] Switching master to 192.168.1.102(192.168.1.102:3306) completed successfully. 127 128 [root@zlm3 08:28:35 ~] 129 #
Check the master-slave replication status.
1 //New master(original slave) 2 (zlm@192.168.1.102 3306)[(none)]>show master status; 3 +------------------+----------+--------------+------------------+------------------------------------------------+ 4 | File | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set | 5 +------------------+----------+--------------+------------------+------------------------------------------------+ 6 | mysql-bin.000003 | 2321 | | | 1b7181ee-6eaf-11e8-998e-080027de0e0e:1-3730259 | 7 +------------------+----------+--------------+------------------+------------------------------------------------+ 8 1 row in set (0.00 sec) 9 10 (zlm@192.168.1.102 3306)[(none)]>show slave statusG 11 Empty set (0.00 sec) 12 13 //New slave(original master) 14 (zlm@192.168.1.101 3306)[(none)]>show master status; 15 +------------------+----------+--------------+------------------+------------------------------------------------+ 16 | File | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set | 17 +------------------+----------+--------------+------------------+------------------------------------------------+ 18 | mysql-bin.000050 | 2361 | | | 1b7181ee-6eaf-11e8-998e-080027de0e0e:1-3730259 | 19 +------------------+----------+--------------+------------------+------------------------------------------------+ 20 1 row in set (0.01 sec) 21 22 (zlm@192.168.1.101 3306)[(none)]>show slave statusG 23 *************************** 1. row *************************** 24 Slave_IO_State: Waiting for master to send event 25 Master_Host: 192.168.1.102 26 Master_User: repl 27 Master_Port: 3306 28 Connect_Retry: 60 29 Master_Log_File: mysql-bin.000003 30 Read_Master_Log_Pos: 2321 31 Relay_Log_File: relay-bin.000002 32 Relay_Log_Pos: 398 33 Relay_Master_Log_File: mysql-bin.000003 34 Slave_IO_Running: Yes 35 Slave_SQL_Running: Yes 36 Replicate_Do_DB: 37 Replicate_Ignore_DB: 38 Replicate_Do_Table: 39 Replicate_Ignore_Table: 40 Replicate_Wild_Do_Table: 41 Replicate_Wild_Ignore_Table: 42 Last_Errno: 0 43 Last_Error: 44 Skip_Counter: 0 45 Exec_Master_Log_Pos: 2321 46 Relay_Log_Space: 591 47 Until_Condition: None 48 Until_Log_File: 49 Until_Log_Pos: 0 50 Master_SSL_Allowed: No 51 Master_SSL_CA_File: 52 Master_SSL_CA_Path: 53 Master_SSL_Cert: 54 Master_SSL_Cipher: 55 Master_SSL_Key: 56 Seconds_Behind_Master: 0 57 Master_SSL_Verify_Server_Cert: No 58 Last_IO_Errno: 0 59 Last_IO_Error: 60 Last_SQL_Errno: 0 61 Last_SQL_Error: 62 Replicate_Ignore_Server_Ids: 63 Master_Server_Id: 1023306 64 Master_UUID: 842ea497-9551-11e8-83ca-080027de0e0e 65 Master_Info_File: mysql.slave_master_info 66 SQL_Delay: 0 67 SQL_Remaining_Delay: NULL 68 Slave_SQL_Running_State: Slave has read all relay log; waiting for more updates 69 Master_Retry_Count: 86400 70 Master_Bind: 71 Last_IO_Error_Timestamp: 72 Last_SQL_Error_Timestamp: 73 Master_SSL_Crl: 74 Master_SSL_Crlpath: 75 Retrieved_Gtid_Set: 76 Executed_Gtid_Set: 1b7181ee-6eaf-11e8-998e-080027de0e0e:1-3730259 77 Auto_Position: 1 78 Replicate_Rewrite_DB: 79 Channel_Name: 80 Master_TLS_Version: 81 1 row in set (0.00 sec)
Check the log of MasterHA on zlm3.
1 [root@zlm3 08:28:35 ~] 2 #cd /var/log/masterha/app1 3 4 [root@zlm3 08:29:12 /var/log/masterha/app1] 5 #cat app1.log 6 Fri Aug 3 07:39:13 2018 - [info] MHA::MasterMonitor version 0.56. 7 Fri Aug 3 07:39:14 2018 - [info] GTID failover mode = 1 8 Fri Aug 3 07:39:14 2018 - [info] Dead Servers: 9 Fri Aug 3 07:39:14 2018 - [info] Alive Servers: 10 Fri Aug 3 07:39:14 2018 - [info] 192.168.1.101(192.168.1.101:3306) 11 Fri Aug 3 07:39:14 2018 - [info] 192.168.1.102(192.168.1.102:3306) 12 Fri Aug 3 07:39:14 2018 - [info] Alive Slaves: 13 Fri Aug 3 07:39:14 2018 - [info] 192.168.1.102(192.168.1.102:3306) Version=5.7.21-log (oldest major version between slaves) log-bin:enabled 14 Fri Aug 3 07:39:14 2018 - [info] GTID ON 15 Fri Aug 3 07:39:14 2018 - [info] Replicating from 192.168.1.101(192.168.1.101:3306) 16 Fri Aug 3 07:39:14 2018 - [info] Primary candidate for the new Master (candidate_master is set) 17 Fri Aug 3 07:39:14 2018 - [info] Current Alive Master: 192.168.1.101(192.168.1.101:3306) 18 Fri Aug 3 07:39:14 2018 - [info] Checking slave configurations.. 19 Fri Aug 3 07:39:14 2018 - [info] read_only=1 is not set on slave 192.168.1.102(192.168.1.102:3306). 20 Fri Aug 3 07:39:14 2018 - [info] Checking replication filtering settings.. 21 Fri Aug 3 07:39:14 2018 - [info] binlog_do_db= , binlog_ignore_db= 22 Fri Aug 3 07:39:14 2018 - [info] Replication filtering check ok. 23 Fri Aug 3 07:39:14 2018 - [info] GTID (with auto-pos) is supported. Skipping all SSH and Node package checking. 24 Fri Aug 3 07:39:14 2018 - [info] Checking SSH publickey authentication settings on the current master.. 25 Fri Aug 3 07:39:14 2018 - [warning] HealthCheck: SSH to 192.168.1.101 is NOT reachable. 26 Fri Aug 3 07:39:14 2018 - [info] 27 192.168.1.101(192.168.1.101:3306) (current master) 28 +--192.168.1.102(192.168.1.102:3306) 29 30 Fri Aug 3 07:39:14 2018 - [info] Checking master_ip_failover_script status: 31 Fri Aug 3 07:39:14 2018 - [info] /etc/masterha/master_ip_failover --command=status --ssh_user=root --orig_master_host=192.168.1.101 --orig_master_ip=192.168.1.101 --orig_master_port=3306 --orig_master_ssh_port=3306 32 Fri Aug 3 07:39:14 2018 - [info] OK. 33 Fri Aug 3 07:39:14 2018 - [warning] shutdown_script is not defined. 34 Fri Aug 3 07:39:14 2018 - [info] Set master ping interval 1 seconds. 35 Fri Aug 3 07:39:14 2018 - [warning] secondary_check_script is not defined. It is highly recommended setting it to check master reachability from two or more routes. 36 Fri Aug 3 07:39:14 2018 - [info] Starting ping health check on 192.168.1.101(192.168.1.101:3306).. 37 Fri Aug 3 07:39:14 2018 - [info] Ping(SELECT) succeeded, waiting until MySQL doesn't respond.. 38 Fri Aug 3 07:39:27 2018 - [info] Got terminate signal. Exit. 39 Fri Aug 3 07:40:03 2018 - [info] MHA::MasterMonitor version 0.56. 40 Fri Aug 3 07:40:04 2018 - [info] GTID failover mode = 1 41 Fri Aug 3 07:40:04 2018 - [info] Dead Servers: 42 Fri Aug 3 07:40:04 2018 - [info] Alive Servers: 43 Fri Aug 3 07:40:04 2018 - [info] 192.168.1.101(192.168.1.101:3306) 44 Fri Aug 3 07:40:04 2018 - [info] 192.168.1.102(192.168.1.102:3306) 45 Fri Aug 3 07:40:04 2018 - [info] Alive Slaves: 46 Fri Aug 3 07:40:04 2018 - [info] 192.168.1.102(192.168.1.102:3306) Version=5.7.21-log (oldest major version between slaves) log-bin:enabled 47 Fri Aug 3 07:40:04 2018 - [info] GTID ON 48 Fri Aug 3 07:40:04 2018 - [info] Replicating from 192.168.1.101(192.168.1.101:3306) 49 Fri Aug 3 07:40:04 2018 - [info] Primary candidate for the new Master (candidate_master is set) 50 Fri Aug 3 07:40:04 2018 - [info] Current Alive Master: 192.168.1.101(192.168.1.101:3306) 51 Fri Aug 3 07:40:04 2018 - [info] Checking slave configurations.. 52 Fri Aug 3 07:40:04 2018 - [info] read_only=1 is not set on slave 192.168.1.102(192.168.1.102:3306). 53 Fri Aug 3 07:40:04 2018 - [info] Checking replication filtering settings.. 54 Fri Aug 3 07:40:04 2018 - [info] binlog_do_db= , binlog_ignore_db= 55 Fri Aug 3 07:40:04 2018 - [info] Replication filtering check ok. 56 Fri Aug 3 07:40:04 2018 - [info] GTID (with auto-pos) is supported. Skipping all SSH and Node package checking. 57 Fri Aug 3 07:40:04 2018 - [info] Checking SSH publickey authentication settings on the current master.. 58 Fri Aug 3 07:40:04 2018 - [warning] HealthCheck: SSH to 192.168.1.101 is NOT reachable. 59 Fri Aug 3 07:40:04 2018 - [info] 60 192.168.1.101(192.168.1.101:3306) (current master) 61 +--192.168.1.102(192.168.1.102:3306) 62 63 Fri Aug 3 07:40:04 2018 - [info] Checking master_ip_failover_script status: 64 Fri Aug 3 07:40:04 2018 - [info] /etc/masterha/master_ip_failover --command=status --ssh_user=root --orig_master_host=192.168.1.101 --orig_master_ip=192.168.1.101 --orig_master_port=3306 --orig_master_ssh_port=3306 65 Fri Aug 3 07:40:04 2018 - [info] OK. 66 Fri Aug 3 07:40:04 2018 - [warning] shutdown_script is not defined. 67 Fri Aug 3 07:40:04 2018 - [info] Set master ping interval 1 seconds. 68 Fri Aug 3 07:40:04 2018 - [warning] secondary_check_script is not defined. It is highly recommended setting it to check master reachability from two or more routes. 69 Fri Aug 3 07:40:04 2018 - [info] Starting ping health check on 192.168.1.101(192.168.1.101:3306).. 70 Fri Aug 3 07:40:04 2018 - [info] Ping(SELECT) succeeded, waiting until MySQL doesn't respond.. 71 Fri Aug 3 08:28:07 2018 - [info] Got terminate signal. Exit.
Test 2：Manual master failover
Execute "masterha_master_switch" script again to generate a failover on zlm2.
1 [root@zlm2 10:11:10 ~] 2 #masterha_master_switch --conf=/etc/masterha/app1.conf --global_conf=/etc/masterha/masterha_default.conf --dead_master_host=192.168.1.102 --master_state=dead --new_master_host=192.168.1.101 --ignore_last_failover 3 --dead_master_ip=<dead_master_ip> is not set. Using 192.168.1.102. 4 --dead_master_port=<dead_master_port> is not set. Using 3306. 5 Fri Aug 3 10:11:50 2018 - [info] Reading default configuration from /etc/masterha/masterha_default.conf.. 6 Fri Aug 3 10:11:50 2018 - [info] Reading application default configuration from /etc/masterha/app1.conf.. 7 Fri Aug 3 10:11:50 2018 - [info] Reading server configuration from /etc/masterha/app1.conf.. 8 Fri Aug 3 10:11:50 2018 - [info] MHA::MasterFailover version 0.56. 9 Fri Aug 3 10:11:50 2018 - [info] Starting master failover. 10 Fri Aug 3 10:11:50 2018 - [info] 11 Fri Aug 3 10:11:50 2018 - [info] * Phase 1: Configuration Check Phase.. 12 Fri Aug 3 10:11:50 2018 - [info] 13 Fri Aug 3 10:11:51 2018 - [info] GTID failover mode = 1 14 Fri Aug 3 10:11:51 2018 - [info] Dead Servers: 15 Fri Aug 3 10:11:51 2018 - [error][/usr/share/perl5/vendor_perl/MHA/MasterFailover.pm, ln187] None of server is dead. Stop failover. 16 Fri Aug 3 10:11:51 2018 - [error][/usr/share/perl5/vendor_perl/MHA/ManagerUtil.pm, ln177] Got ERROR: at /usr/bin/masterha_master_switch line 53. 17 18 //Stop mysqld of master on zlm3. 19 [root@zlm3 10:13:56 ~] 20 #mysqladmin shutdown 21 22 [root@zlm3 10:14:04 ~] 23 #ps aux|grep mysqld 24 mysql 5368 0.0 19.6 1110812 200292 pts/0 Sl 04:44 0:09 mysqld --defaults-file=/data/mysql/mysql3306/my.cnf 25 root 8827 0.0 0.0 112640 960 pts/0 R+ 10:14 0:00 grep --color=auto mysqld 26 27 [root@zlm3 10:14:08 ~] 28 #ps aux|grep mysqld 29 mysql 5368 0.0 19.6 1110812 200292 pts/0 Sl 04:44 0:09 mysqld --defaults-file=/data/mysql/mysql3306/my.cnf 30 root 8833 0.0 0.0 112640 960 pts/0 R+ 10:14 0:00 grep --color=auto mysqld 31 32 [root@zlm3 10:14:12 ~] 33 #ps aux|grep mysqld 34 mysql 5368 0.0 19.1 995088 194692 pts/0 Sl 04:44 0:09 mysqld --defaults-file=/data/mysql/mysql3306/my.cnf 35 root 8839 0.0 0.0 112640 960 pts/0 R+ 10:14 0:00 grep --color=auto mysqld 36 37 [root@zlm3 10:14:23 ~] 38 #ps aux|grep mysqld 39 root 8854 0.0 0.0 112640 960 pts/0 R+ 10:14 0:00 grep --color=auto mysqld 40 41 //Execute the above command again on zlm2. 42 [root@zlm2 10:15:43 ~] 43 #masterha_master_switch --conf=/etc/masterha/app1.conf --global_conf=/etc/masterha/masterha_default.conf --dead_master_host=192.168.1.102 --master_state=dead --new_master_host=192.168.1.101 --ignore_last_failover 44 --dead_master_ip=<dead_master_ip> is not set. Using 192.168.1.102. 45 --dead_master_port=<dead_master_port> is not set. Using 3306. 46 Fri Aug 3 10:15:43 2018 - [info] Reading default configuration from /etc/masterha/masterha_default.conf.. 47 Fri Aug 3 10:15:43 2018 - [info] Reading application default configuration from /etc/masterha/app1.conf.. 48 Fri Aug 3 10:15:43 2018 - [info] Reading server configuration from /etc/masterha/app1.conf.. 49 Fri Aug 3 10:15:43 2018 - [info] MHA::MasterFailover version 0.56. 50 Fri Aug 3 10:15:43 2018 - [info] Starting master failover. 51 Fri Aug 3 10:15:43 2018 - [info] 52 Fri Aug 3 10:15:43 2018 - [info] * Phase 1: Configuration Check Phase.. 53 Fri Aug 3 10:15:43 2018 - [info] 54 Fri Aug 3 10:15:44 2018 - [info] GTID failover mode = 1 55 Fri Aug 3 10:15:44 2018 - [info] Dead Servers: 56 Fri Aug 3 10:15:44 2018 - [info] 192.168.1.102(192.168.1.102:3306) 57 Fri Aug 3 10:15:44 2018 - [info] Checking master reachability via MySQL(double check)... 58 Fri Aug 3 10:15:44 2018 - [info] ok. 59 Fri Aug 3 10:15:44 2018 - [info] Alive Servers: 60 Fri Aug 3 10:15:44 2018 - [info] 192.168.1.101(192.168.1.101:3306) 61 Fri Aug 3 10:15:44 2018 - [info] Alive Slaves: 62 Fri Aug 3 10:15:44 2018 - [info] 192.168.1.101(192.168.1.101:3306) Version=5.7.21-log (oldest major version between slaves) log-bin:enabled 63 Fri Aug 3 10:15:44 2018 - [info] GTID ON 64 Fri Aug 3 10:15:44 2018 - [info] Replicating from 192.168.1.102(192.168.1.102:3306) 65 Fri Aug 3 10:15:44 2018 - [info] Primary candidate for the new Master (candidate_master is set) 66 Master 192.168.1.102(192.168.1.102:3306) is dead. Proceed? (yes/NO): yes 67 Fri Aug 3 10:15:46 2018 - [info] Starting GTID based failover. 68 Fri Aug 3 10:15:46 2018 - [info] 69 Fri Aug 3 10:15:46 2018 - [info] ** Phase 1: Configuration Check Phase completed. 70 Fri Aug 3 10:15:46 2018 - [info] 71 Fri Aug 3 10:15:46 2018 - [info] * Phase 2: Dead Master Shutdown Phase.. 72 Fri Aug 3 10:15:46 2018 - [info] 73 ssh: connect to host 192.168.1.102 port 3306: Connection refused 74 Fri Aug 3 10:15:46 2018 - [warning] HealthCheck: SSH to 192.168.1.102 is NOT reachable. 75 Fri Aug 3 10:15:46 2018 - [info] Forcing shutdown so that applications never connect to the current master.. 76 Fri Aug 3 10:15:46 2018 - [info] Executing master IP deactivation script: 77 Fri Aug 3 10:15:46 2018 - [info] /etc/masterha/master_ip_failover --orig_master_host=192.168.1.102 --orig_master_ip=192.168.1.102 --orig_master_port=3306 --command=stop --orig_master_ssh_port=3306 78 ssh: connect to host 192.168.1.102 port 3306: Connection refused 79 Fri Aug 3 10:15:48 2018 - [info] done. 80 Fri Aug 3 10:15:48 2018 - [warning] shutdown_script is not set. Skipping explicit shutting down of the dead master. 81 Fri Aug 3 10:15:48 2018 - [info] * Phase 2: Dead Master Shutdown Phase completed. 82 Fri Aug 3 10:15:48 2018 - [info] 83 Fri Aug 3 10:15:48 2018 - [info] * Phase 3: Master Recovery Phase.. 84 Fri Aug 3 10:15:48 2018 - [info] 85 Fri Aug 3 10:15:48 2018 - [info] * Phase 3.1: Getting Latest Slaves Phase.. 86 Fri Aug 3 10:15:48 2018 - [info] 87 Fri Aug 3 10:15:48 2018 - [info] The latest binary log file/position on all slaves is mysql-bin.000003:2321 88 Fri Aug 3 10:15:48 2018 - [info] Latest slaves (Slaves that received relay log files to the latest): 89 Fri Aug 3 10:15:48 2018 - [info] 192.168.1.101(192.168.1.101:3306) Version=5.7.21-log (oldest major version between slaves) log-bin:enabled 90 Fri Aug 3 10:15:48 2018 - [info] GTID ON 91 Fri Aug 3 10:15:48 2018 - [info] Replicating from 192.168.1.102(192.168.1.102:3306) 92 Fri Aug 3 10:15:48 2018 - [info] Primary candidate for the new Master (candidate_master is set) 93 Fri Aug 3 10:15:48 2018 - [info] The oldest binary log file/position on all slaves is mysql-bin.000003:2321 94 Fri Aug 3 10:15:48 2018 - [info] Oldest slaves: 95 Fri Aug 3 10:15:48 2018 - [info] 192.168.1.101(192.168.1.101:3306) Version=5.7.21-log (oldest major version between slaves) log-bin:enabled 96 Fri Aug 3 10:15:48 2018 - [info] GTID ON 97 Fri Aug 3 10:15:48 2018 - [info] Replicating from 192.168.1.102(192.168.1.102:3306) 98 Fri Aug 3 10:15:48 2018 - [info] Primary candidate for the new Master (candidate_master is set) 99 Fri Aug 3 10:15:48 2018 - [info] 100 Fri Aug 3 10:15:48 2018 - [info] * Phase 3.3: Determining New Master Phase.. 101 Fri Aug 3 10:15:48 2018 - [info] 102 Fri Aug 3 10:15:48 2018 - [info] 192.168.1.101 can be new master. 103 Fri Aug 3 10:15:48 2018 - [info] New master is 192.168.1.101(192.168.1.101:3306) 104 Fri Aug 3 10:15:48 2018 - [info] Starting master failover.. 105 Fri Aug 3 10:15:48 2018 - [info] 106 From: 107 192.168.1.102(192.168.1.102:3306) (current master) 108 +--192.168.1.101(192.168.1.101:3306) 109 110 To: 111 192.168.1.101(192.168.1.101:3306) (new master) 112 113 Starting master switch from 192.168.1.102(192.168.1.102:3306) to 192.168.1.101(192.168.1.101:3306)? (yes/NO): yes 114 Fri Aug 3 10:15:56 2018 - [info] New master decided manually is 192.168.1.101(192.168.1.101:3306) 115 Fri Aug 3 10:15:56 2018 - [info] 116 Fri Aug 3 10:15:56 2018 - [info] * Phase 3.3: New Master Recovery Phase.. 117 Fri Aug 3 10:15:56 2018 - [info] 118 Fri Aug 3 10:15:56 2018 - [info] Waiting all logs to be applied.. 119 Fri Aug 3 10:15:56 2018 - [info] done. 120 Fri Aug 3 10:15:56 2018 - [info] Getting new master's binlog name and position.. 121 Fri Aug 3 10:15:56 2018 - [info] mysql-bin.000051:190 122 Fri Aug 3 10:15:56 2018 - [info] All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='192.168.1.101', MASTER_PORT=3306, MASTER_AUTO_POSITION=1, MASTER_USER='repl', MASTER_PASSWORD='xxx'; 123 Fri Aug 3 10:15:56 2018 - [info] Master Recovery succeeded. File:Pos:Exec_Gtid_Set: mysql-bin.000051, 190, 1b7181ee-6eaf-11e8-998e-080027de0e0e:1-3730259 124 Fri Aug 3 10:15:56 2018 - [info] Executing master IP activate script: 125 Fri Aug 3 10:15:56 2018 - [info] /etc/masterha/master_ip_failover --command=start --ssh_user=root --orig_master_host=192.168.1.102 --orig_master_ip=192.168.1.102 --orig_master_port=3306 --new_master_host=192.168.1.101 --new_master_ip=192.168.1.101 --new_master_port=3306 --new_master_user='zlm' --new_master_password='zlmzlm' --orig_master_ssh_port=3306 --new_master_ssh_port=3306 126 Unknown option: new_master_ssh_port 127 Set read_only=0 on the new master. 128 ssh_exchange_identification: Connection closed by remote host 129 Fri Aug 3 10:15:56 2018 - [info] OK. 130 Fri Aug 3 10:15:56 2018 - [info] ** Finished master recovery successfully. 131 Fri Aug 3 10:15:56 2018 - [info] * Phase 3: Master Recovery Phase completed. 132 Fri Aug 3 10:15:56 2018 - [info] 133 Fri Aug 3 10:15:56 2018 - [info] * Phase 4: Slaves Recovery Phase.. 134 Fri Aug 3 10:15:56 2018 - [info] 135 Fri Aug 3 10:15:56 2018 - [info] 136 Fri Aug 3 10:15:56 2018 - [info] * Phase 4.1: Starting Slaves in parallel.. 137 Fri Aug 3 10:15:56 2018 - [info] 138 Fri Aug 3 10:15:56 2018 - [info] All new slave servers recovered successfully. 139 Fri Aug 3 10:15:56 2018 - [info] 140 Fri Aug 3 10:15:56 2018 - [info] * Phase 5: New master cleanup phase.. 141 Fri Aug 3 10:15:56 2018 - [info] 142 Fri Aug 3 10:15:56 2018 - [info] Resetting slave info on the new master.. 143 Fri Aug 3 10:15:56 2018 - [info] 192.168.1.101: Resetting slave info succeeded. 144 Fri Aug 3 10:15:56 2018 - [info] Master failover to 192.168.1.101(192.168.1.101:3306) completed successfully. 145 Fri Aug 3 10:15:56 2018 - [info] 146 147 ----- Failover Report ----- 148 149 app1: MySQL Master failover 192.168.1.102(192.168.1.102:3306) to 192.168.1.101(192.168.1.101:3306) succeeded 150 151 Master 192.168.1.102(192.168.1.102:3306) is down! 152 153 Check MHA Manager logs at zlm2 for details. 154 155 Started manual(interactive) failover. 156 Invalidated master IP address on 192.168.1.102(192.168.1.102:3306) 157 Selected 192.168.1.101(192.168.1.101:3306) as a new master. 158 192.168.1.101(192.168.1.101:3306): OK: Applying all logs succeeded. 159 192.168.1.101(192.168.1.101:3306): OK: Activated master IP address. 160 192.168.1.101(192.168.1.101:3306): Resetting slave info succeeded. 161 Master failover to 192.168.1.101(192.168.1.101:3306) completed successfully.
Check the status of new master on zlm2.
1 (zlm@192.168.1.101 3306)[(none)]>show master status; 2 +------------------+----------+--------------+------------------+------------------------------------------------+ 3 | File | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set | 4 +------------------+----------+--------------+------------------+------------------------------------------------+ 5 | mysql-bin.000051 | 190 | | | 1b7181ee-6eaf-11e8-998e-080027de0e0e:1-3730259 | 6 +------------------+----------+--------------+------------------+------------------------------------------------+ 7 1 row in set (0.00 sec) 8 9 (zlm@192.168.1.101 3306)[(none)]>show slave statusG 10 Empty set (0.00 sec)
Check the file and log on MasterHA.
1 [root@zlm2 10:15:56 ~] 2 #cd /var/log/masterha/app1 3 4 [root@zlm2 10:20:04 /var/log/masterha/app1] 5 #ls -l 6 total 4 7 -rw-r--r-- 1 root root 0 Aug 3 10:15 app1.failover.complete 8 -rw-r--r-- 1 root root 3883 Aug 2 11:29 app1.log 9 10 //The option of "--ignore_last_failover" can neglect the influence of existence of "app1.failover.complete".Otherwise,the failover operation will be terminated by error. 11 //This file will be created after a failover operation and it will be created only on the original slave who wants to become a new master. 12 13 [root@zlm2 10:20:05 /var/log/masterha/app1] 14 #cat app1.log 15 Thu Aug 2 11:12:03 2018 - [info] MHA::MasterMonitor version 0.56. 16 Thu Aug 2 11:12:04 2018 - [error][/usr/share/perl5/vendor_perl/MHA/ServerManager.pm, ln301] Got MySQL error when connecting 192.168.1.101(192.168.1.101:3306) :1045:Access denied for user 'root'@'zlm2' (using password: NO), but this is not a MySQL crash. Check MySQL server settings. 17 at /usr/share/perl5/vendor_perl/MHA/ServerManager.pm line 297. 18 Thu Aug 2 11:12:04 2018 - [error][/usr/share/perl5/vendor_perl/MHA/ServerManager.pm, ln301] Got MySQL error when connecting 192.168.1.102(192.168.1.102:3306) :1045:Access denied for user 'root'@'zlm2' (using password: NO), but this is not a MySQL crash. Check MySQL server settings. 19 at /usr/share/perl5/vendor_perl/MHA/ServerManager.pm line 297. 20 Thu Aug 2 11:12:05 2018 - [error][/usr/share/perl5/vendor_perl/MHA/ServerManager.pm, ln309] Got fatal error, stopping operations 21 Thu Aug 2 11:12:05 2018 - [error][/usr/share/perl5/vendor_perl/MHA/MasterMonitor.pm, ln424] Error happened on checking configurations. at /usr/share/perl5/vendor_perl/MHA/MasterMonitor.pm line 326. 22 Thu Aug 2 11:12:05 2018 - [error][/usr/share/perl5/vendor_perl/MHA/MasterMonitor.pm, ln523] Error happened on monitoring servers. 23 Thu Aug 2 11:12:05 2018 - [info] Got exit code 1 (Not master dead). 24 Thu Aug 2 11:13:56 2018 - [info] MHA::MasterMonitor version 0.56. 25 Thu Aug 2 11:13:57 2018 - [info] GTID failover mode = 1 26 Thu Aug 2 11:13:57 2018 - [info] Dead Servers: 27 Thu Aug 2 11:13:57 2018 - [info] Alive Servers: 28 Thu Aug 2 11:13:57 2018 - [info] 192.168.1.101(192.168.1.101:3306) 29 Thu Aug 2 11:13:57 2018 - [info] 192.168.1.102(192.168.1.102:3306) 30 Thu Aug 2 11:13:57 2018 - [info] Alive Slaves: 31 Thu Aug 2 11:13:57 2018 - [info] 192.168.1.102(192.168.1.102:3306) Version=5.7.21-log (oldest major version between slaves) log-bin:enabled 32 Thu Aug 2 11:13:57 2018 - [info] GTID ON 33 Thu Aug 2 11:13:57 2018 - [info] Replicating from 192.168.1.101(192.168.1.101:3306) 34 Thu Aug 2 11:13:57 2018 - [info] Primary candidate for the new Master (candidate_master is set) 35 Thu Aug 2 11:13:57 2018 - [info] Current Alive Master: 192.168.1.101(192.168.1.101:3306) 36 Thu Aug 2 11:13:57 2018 - [info] Checking slave configurations.. 37 Thu Aug 2 11:13:57 2018 - [info] read_only=1 is not set on slave 192.168.1.102(192.168.1.102:3306). 38 Thu Aug 2 11:13:57 2018 - [info] Checking replication filtering settings.. 39 Thu Aug 2 11:13:57 2018 - [info] binlog_do_db= , binlog_ignore_db= 40 Thu Aug 2 11:13:57 2018 - [info] Replication filtering check ok. 41 Thu Aug 2 11:13:57 2018 - [info] GTID (with auto-pos) is supported. Skipping all SSH and Node package checking. 42 Thu Aug 2 11:13:57 2018 - [info] Checking SSH publickey authentication settings on the current master.. 43 Thu Aug 2 11:13:57 2018 - [warning] HealthCheck: SSH to 192.168.1.101 is NOT reachable. 44 Thu Aug 2 11:13:57 2018 - [info] 45 192.168.1.101(192.168.1.101:3306) (current master) 46 +--192.168.1.102(192.168.1.102:3306) 47 48 Thu Aug 2 11:13:57 2018 - [info] Checking master_ip_failover_script status: 49 Thu Aug 2 11:13:57 2018 - [info] /etc/masterha/master_ip_failover --command=status --ssh_user=root --orig_master_host=192.168.1.101 --orig_master_ip=192.168.1.101 --orig_master_port=3306 --orig_master_ssh_port=3306 50 Thu Aug 2 11:13:57 2018 - [info] OK. 51 Thu Aug 2 11:13:57 2018 - [warning] shutdown_script is not defined. 52 Thu Aug 2 11:13:57 2018 - [info] Set master ping interval 1 seconds. 53 Thu Aug 2 11:13:57 2018 - [warning] secondary_check_script is not defined. It is highly recommended setting it to check master reachability from two or more routes. 54 Thu Aug 2 11:13:57 2018 - [info] Starting ping health check on 192.168.1.101(192.168.1.101:3306).. 55 Thu Aug 2 11:13:57 2018 - [info] Ping(SELECT) succeeded, waiting until MySQL doesn't respond.. 56 Thu Aug 2 11:29:51 2018 - [info] Got terminate signal. Exit.
Test3: Automation of master switchover.
Repair the salve replication on zlm3.
1 [root@zlm3 10:14:24 ~] 2 #mysql 3 Welcome to the MySQL monitor. Commands end with ; or g. 4 Your MySQL connection id is 3 5 Server version: 5.7.21-log MySQL Community Server (GPL) 6 7 Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved. 8 9 Oracle is a registered trademark of Oracle Corporation and/or its 10 affiliates. Other names may be trademarks of their respective 11 owners. 12 13 Type 'help;' or 'h' for help. Type 'c' to clear the current input statement. 14 15 (zlm@192.168.1.102 3306)[(none)]>change master to 16 -> master_host='192.168.1.101', 17 -> master_port=3306, 18 -> master_user='repl', 19 -> master_password='repl4slave', 20 -> master_auto_position=1; 21 Query OK, 0 rows affected, 2 warnings (0.02 sec) 22 23 (zlm@192.168.1.102 3306)[(none)]>show slave statusG 24 *************************** 1. row *************************** 25 Slave_IO_State: Waiting for master to send event 26 Master_Host: 192.168.1.101 27 Master_User: repl 28 Master_Port: 3306 29 Connect_Retry: 60 30 Master_Log_File: mysql-bin.000051 31 Read_Master_Log_Pos: 190 32 Relay_Log_File: relay-bin.000002 33 Relay_Log_Pos: 355 34 Relay_Master_Log_File: mysql-bin.000051 35 Slave_IO_Running: Yes 36 Slave_SQL_Running: Yes 37 Replicate_Do_DB: 38 Replicate_Ignore_DB: 39 Replicate_Do_Table: 40 Replicate_Ignore_Table: 41 Replicate_Wild_Do_Table: 42 Replicate_Wild_Ignore_Table: 43 Last_Errno: 0 44 Last_Error: 45 Skip_Counter: 0 46 Exec_Master_Log_Pos: 190 47 Relay_Log_Space: 548 48 Until_Condition: None 49 Until_Log_File: 50 Until_Log_Pos: 0 51 Master_SSL_Allowed: No 52 Master_SSL_CA_File: 53 Master_SSL_CA_Path: 54 Master_SSL_Cert: 55 Master_SSL_Cipher: 56 Master_SSL_Key: 57 Seconds_Behind_Master: 0 58 Master_SSL_Verify_Server_Cert: No 59 Last_IO_Errno: 0 60 Last_IO_Error: 61 Last_SQL_Errno: 0 62 Last_SQL_Error: 63 Replicate_Ignore_Server_Ids: 64 Master_Server_Id: 1013306 65 Master_UUID: 1b7181ee-6eaf-11e8-998e-080027de0e0e 66 Master_Info_File: mysql.slave_master_info 67 SQL_Delay: 0 68 SQL_Remaining_Delay: NULL 69 Slave_SQL_Running_State: Slave has read all relay log; waiting for more updates 70 Master_Retry_Count: 86400 71 Master_Bind: 72 Last_IO_Error_Timestamp: 73 Last_SQL_Error_Timestamp: 74 Master_SSL_Crl: 75 Master_SSL_Crlpath: 76 Retrieved_Gtid_Set: 77 Executed_Gtid_Set: 1b7181ee-6eaf-11e8-998e-080027de0e0e:1-3730259 78 Auto_Position: 1 79 Replicate_Rewrite_DB: 80 Channel_Name: 81 Master_TLS_Version: 82 1 row in set (0.00 sec)
Start MasterHA-manager.
1 [root@zlm3 10:48:00 /var/log/masterha/app1] 2 #nohup masterha_manager --conf=/etc/masterha/app1.conf --global_conf=/etc/masterha/masterha_default.conf & 3 [1] 9265 4 nohup: ignoring input and appending output to ‘nohup.out’ 5 6 [root@zlm3 10:48:12 /var/log/masterha/app1] 7 #ls -l 8 total 24 9 -rw-r--r-- 1 root root 16370 Aug 3 10:48 app1.log 10 -rw-r--r-- 1 root root 35 Aug 3 10:48 app1.master_status.health //This file is created only when MasterHA-manager is running.It will continuously record the health status between slave and master. 11 -rw------- 1 root root 371 Aug 3 10:48 nohup.out 12 13 [root@zlm3 10:48:14 /var/log/masterha/app1] 14 #cat nohup.out 15 Fri Aug 3 10:48:12 2018 - [info] Reading default configuration from /etc/masterha/masterha_default.conf.. 16 Fri Aug 3 10:48:12 2018 - [info] Reading application default configuration from /etc/masterha/app1.conf.. 17 Fri Aug 3 10:48:12 2018 - [info] Reading server configuration from /etc/masterha/app1.conf.. 18 ssh_exchange_identification: Connection closed by remote host 19 20 [root@zlm3 10:48:26 /var/log/masterha/app1] 21 #cat app1.master_status.health 22 9265 0:PING_OK master:192.168.1.101 23 [root@zlm3 10:48:31 /var/log/masterha/app1] 24 #ps aux|grep manager 25 root 9265 0.6 2.1 299172 21516 pts/1 S 10:48 0:00 perl /usr/bin/masterha_manager --conf=/etc/masterha/app1.conf --global_conf=/etc/masterha/masterha_default.conf 26 root 9332 0.0 0.0 112640 960 pts/1 R+ 10:48 0:00 grep --color=auto manager
Kill mysqld on zlm2 to pretend the master is dead.
1 [root@zlm2 10:54:31 /var/log/masterha/app1] 2 #pkill mysqld 3 4 [root@zlm2 10:55:28 /var/log/masterha/app1] 5 #ps aux|grep mysqld 6 root 6067 0.0 0.0 112640 960 pts/1 R+ 10:55 0:00 grep --color=auto mysqld
Observe the app1.log on zlm3.
1 [root@zlm3 10:54:59 /var/log/masterha/app1] 2 #echo ''> app1.log 3 4 [root@zlm3 10:55:17 /var/log/masterha/app1] 5 #tail -f app1.log 6 7 Fri Aug 3 10:55:29 2018 - [warning] Got error on MySQL select ping: 2006 (MySQL server has gone away) 8 Fri Aug 3 10:55:29 2018 - [info] Executing SSH check script: exit 0 9 Fri Aug 3 10:55:29 2018 - [warning] HealthCheck: SSH to 192.168.1.101 is NOT reachable. 10 Fri Aug 3 10:55:30 2018 - [warning] Got error on MySQL connect: 2003 (Can't connect to MySQL server on '192.168.1.101' (111)) 11 Fri Aug 3 10:55:30 2018 - [warning] Connection failed 2 time(s).. 12 Fri Aug 3 10:55:31 2018 - [warning] Got error on MySQL connect: 2003 (Can't connect to MySQL server on '192.168.1.101' (111)) 13 Fri Aug 3 10:55:31 2018 - [warning] Connection failed 3 time(s).. 14 Fri Aug 3 10:55:32 2018 - [warning] Got error on MySQL connect: 2003 (Can't connect to MySQL server on '192.168.1.101' (111)) 15 Fri Aug 3 10:55:32 2018 - [warning] Connection failed 4 time(s).. 16 Fri Aug 3 10:55:32 2018 - [warning] Master is not reachable from health checker! 17 Fri Aug 3 10:55:32 2018 - [warning] Master 192.168.1.101(192.168.1.101:3306) is not reachable! 18 Fri Aug 3 10:55:32 2018 - [warning] SSH is NOT reachable. 19 Fri Aug 3 10:55:32 2018 - [info] Connecting to a master server failed. Reading configuration file /etc/masterha/masterha_default.conf and /etc/masterha/app1.conf again, and trying to connect to all servers to check server status.. 20 Fri Aug 3 10:55:32 2018 - [info] Reading default configuration from /etc/masterha/masterha_default.conf.. 21 Fri Aug 3 10:55:32 2018 - [info] Reading application default configuration from /etc/masterha/app1.conf.. 22 Fri Aug 3 10:55:32 2018 - [info] Reading server configuration from /etc/masterha/app1.conf.. 23 Fri Aug 3 10:55:33 2018 - [info] GTID failover mode = 1 24 Fri Aug 3 10:55:33 2018 - [info] Dead Servers: 25 Fri Aug 3 10:55:33 2018 - [info] 192.168.1.101(192.168.1.101:3306) 26 Fri Aug 3 10:55:33 2018 - [info] Alive Servers: 27 Fri Aug 3 10:55:33 2018 - [info] 192.168.1.102(192.168.1.102:3306) 28 Fri Aug 3 10:55:33 2018 - [info] Alive Slaves: 29 Fri Aug 3 10:55:33 2018 - [info] 192.168.1.102(192.168.1.102:3306) Version=5.7.21-log (oldest major version between slaves) log-bin:enabled 30 Fri Aug 3 10:55:33 2018 - [info] GTID ON 31 Fri Aug 3 10:55:33 2018 - [info] Replicating from 192.168.1.101(192.168.1.101:3306) 32 Fri Aug 3 10:55:33 2018 - [info] Primary candidate for the new Master (candidate_master is set) 33 Fri Aug 3 10:55:33 2018 - [info] Checking slave configurations.. 34 Fri Aug 3 10:55:33 2018 - [info] read_only=1 is not set on slave 192.168.1.102(192.168.1.102:3306). 35 Fri Aug 3 10:55:33 2018 - [info] Checking replication filtering settings.. 36 Fri Aug 3 10:55:33 2018 - [info] Replication filtering check ok. 37 Fri Aug 3 10:55:33 2018 - [info] Master is down! 38 Fri Aug 3 10:55:33 2018 - [info] Terminating monitoring script. 39 Fri Aug 3 10:55:33 2018 - [info] Got exit code 20 (Master dead). 40 Fri Aug 3 10:55:33 2018 - [info] MHA::MasterFailover version 0.56. 41 Fri Aug 3 10:55:33 2018 - [info] Starting master failover. 42 Fri Aug 3 10:55:33 2018 - [info] 43 Fri Aug 3 10:55:33 2018 - [info] * Phase 1: Configuration Check Phase.. 44 Fri Aug 3 10:55:33 2018 - [info] 45 Fri Aug 3 10:55:34 2018 - [info] GTID failover mode = 1 46 Fri Aug 3 10:55:34 2018 - [info] Dead Servers: 47 Fri Aug 3 10:55:34 2018 - [info] 192.168.1.101(192.168.1.101:3306) 48 Fri Aug 3 10:55:34 2018 - [info] Checking master reachability via MySQL(double check)... 49 Fri Aug 3 10:55:34 2018 - [info] ok. 50 Fri Aug 3 10:55:34 2018 - [info] Alive Servers: 51 Fri Aug 3 10:55:34 2018 - [info] 192.168.1.102(192.168.1.102:3306) 52 Fri Aug 3 10:55:34 2018 - [info] Alive Slaves: 53 Fri Aug 3 10:55:34 2018 - [info] 192.168.1.102(192.168.1.102:3306) Version=5.7.21-log (oldest major version between slaves) log-bin:enabled 54 Fri Aug 3 10:55:34 2018 - [info] GTID ON 55 Fri Aug 3 10:55:34 2018 - [info] Replicating from 192.168.1.101(192.168.1.101:3306) 56 Fri Aug 3 10:55:34 2018 - [info] Primary candidate for the new Master (candidate_master is set) 57 Fri Aug 3 10:55:34 2018 - [info] Starting GTID based failover. 58 Fri Aug 3 10:55:34 2018 - [info] 59 Fri Aug 3 10:55:34 2018 - [info] ** Phase 1: Configuration Check Phase completed. 60 Fri Aug 3 10:55:34 2018 - [info] 61 Fri Aug 3 10:55:34 2018 - [info] * Phase 2: Dead Master Shutdown Phase.. 62 Fri Aug 3 10:55:34 2018 - [info] 63 Fri Aug 3 10:55:34 2018 - [info] Forcing shutdown so that applications never connect to the current master.. 64 Fri Aug 3 10:55:34 2018 - [info] Executing master IP deactivation script: 65 Fri Aug 3 10:55:34 2018 - [info] /etc/masterha/master_ip_failover --orig_master_host=192.168.1.101 --orig_master_ip=192.168.1.101 --orig_master_port=3306 --command=stop --orig_master_ssh_port=3306 66 ssh: connect to host 192.168.1.101 port 3306: Connection refused 67 Fri Aug 3 10:55:37 2018 - [info] done. 68 Fri Aug 3 10:55:37 2018 - [warning] shutdown_script is not set. Skipping explicit shutting down of the dead master. 69 Fri Aug 3 10:55:37 2018 - [info] * Phase 2: Dead Master Shutdown Phase completed. 70 Fri Aug 3 10:55:37 2018 - [info] 71 Fri Aug 3 10:55:37 2018 - [info] * Phase 3: Master Recovery Phase.. 72 Fri Aug 3 10:55:37 2018 - [info] 73 Fri Aug 3 10:55:37 2018 - [info] * Phase 3.1: Getting Latest Slaves Phase.. 74 Fri Aug 3 10:55:37 2018 - [info] 75 Fri Aug 3 10:55:37 2018 - [info] The latest binary log file/position on all slaves is mysql-bin.000051:190 76 Fri Aug 3 10:55:37 2018 - [info] Latest slaves (Slaves that received relay log files to the latest): 77 Fri Aug 3 10:55:37 2018 - [info] 192.168.1.102(192.168.1.102:3306) Version=5.7.21-log (oldest major version between slaves) log-bin:enabled 78 Fri Aug 3 10:55:37 2018 - [info] GTID ON 79 Fri Aug 3 10:55:37 2018 - [info] Replicating from 192.168.1.101(192.168.1.101:3306) 80 Fri Aug 3 10:55:37 2018 - [info] Primary candidate for the new Master (candidate_master is set) 81 Fri Aug 3 10:55:37 2018 - [info] The oldest binary log file/position on all slaves is mysql-bin.000051:190 82 Fri Aug 3 10:55:37 2018 - [info] Oldest slaves: 83 Fri Aug 3 10:55:37 2018 - [info] 192.168.1.102(192.168.1.102:3306) Version=5.7.21-log (oldest major version between slaves) log-bin:enabled 84 Fri Aug 3 10:55:37 2018 - [info] GTID ON 85 Fri Aug 3 10:55:37 2018 - [info] Replicating from 192.168.1.101(192.168.1.101:3306) 86 Fri Aug 3 10:55:37 2018 - [info] Primary candidate for the new Master (candidate_master is set) 87 Fri Aug 3 10:55:37 2018 - [info] 88 Fri Aug 3 10:55:37 2018 - [info] * Phase 3.3: Determining New Master Phase.. 89 Fri Aug 3 10:55:37 2018 - [info] 90 Fri Aug 3 10:55:37 2018 - [info] Searching new master from slaves.. 91 Fri Aug 3 10:55:37 2018 - [info] Candidate masters from the configuration file: 92 Fri Aug 3 10:55:37 2018 - [info] 192.168.1.102(192.168.1.102:3306) Version=5.7.21-log (oldest major version between slaves) log-bin:enabled 93 Fri Aug 3 10:55:37 2018 - [info] GTID ON 94 Fri Aug 3 10:55:37 2018 - [info] Replicating from 192.168.1.101(192.168.1.101:3306) 95 Fri Aug 3 10:55:37 2018 - [info] Primary candidate for the new Master (candidate_master is set) 96 Fri Aug 3 10:55:37 2018 - [info] Non-candidate masters: 97 Fri Aug 3 10:55:37 2018 - [info] Searching from candidate_master slaves which have received the latest relay log events.. 98 Fri Aug 3 10:55:37 2018 - [info] New master is 192.168.1.102(192.168.1.102:3306) 99 Fri Aug 3 10:55:37 2018 - [info] Starting master failover.. 100 Fri Aug 3 10:55:37 2018 - [info] 101 From: 102 192.168.1.101(192.168.1.101:3306) (current master) 103 +--192.168.1.102(192.168.1.102:3306) 104 105 To: 106 192.168.1.102(192.168.1.102:3306) (new master) 107 Fri Aug 3 10:55:37 2018 - [info] 108 Fri Aug 3 10:55:37 2018 - [info] * Phase 3.3: New Master Recovery Phase.. 109 Fri Aug 3 10:55:37 2018 - [info] 110 Fri Aug 3 10:55:37 2018 - [info] Waiting all logs to be applied.. 111 Fri Aug 3 10:55:37 2018 - [info] done. 112 Fri Aug 3 10:55:37 2018 - [info] Getting new master's binlog name and position.. 113 Fri Aug 3 10:55:37 2018 - [info] mysql-bin.000004:190 114 Fri Aug 3 10:55:37 2018 - [info] All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='192.168.1.102', MASTER_PORT=3306, MASTER_AUTO_POSITION=1, MASTER_USER='repl', MASTER_PASSWORD='xxx'; 115 Fri Aug 3 10:55:37 2018 - [info] Master Recovery succeeded. File:Pos:Exec_Gtid_Set: mysql-bin.000004, 190, 1b7181ee-6eaf-11e8-998e-080027de0e0e:1-3730259 116 Fri Aug 3 10:55:37 2018 - [info] Executing master IP activate script: 117 Fri Aug 3 10:55:37 2018 - [info] /etc/masterha/master_ip_failover --command=start --ssh_user=root --orig_master_host=192.168.1.101 --orig_master_ip=192.168.1.101 --orig_master_port=3306 --new_master_host=192.168.1.102 --new_master_ip=192.168.1.102 --new_master_port=3306 --new_master_user='zlm' --new_master_password='zlmzlm' --orig_master_ssh_port=3306 --new_master_ssh_port=3306 118 Unknown option: new_master_ssh_port 119 Set read_only=0 on the new master. 120 ssh_exchange_identification: Connection closed by remote host 121 Fri Aug 3 10:55:37 2018 - [info] OK. 122 Fri Aug 3 10:55:37 2018 - [info] ** Finished master recovery successfully. 123 Fri Aug 3 10:55:37 2018 - [info] * Phase 3: Master Recovery Phase completed. 124 Fri Aug 3 10:55:37 2018 - [info] 125 Fri Aug 3 10:55:37 2018 - [info] * Phase 4: Slaves Recovery Phase.. 126 Fri Aug 3 10:55:37 2018 - [info] 127 Fri Aug 3 10:55:37 2018 - [info] 128 Fri Aug 3 10:55:37 2018 - [info] * Phase 4.1: Starting Slaves in parallel.. 129 Fri Aug 3 10:55:37 2018 - [info] 130 Fri Aug 3 10:55:37 2018 - [info] All new slave servers recovered successfully. 131 Fri Aug 3 10:55:37 2018 - [info] 132 Fri Aug 3 10:55:37 2018 - [info] * Phase 5: New master cleanup phase.. 133 Fri Aug 3 10:55:37 2018 - [info] 134 Fri Aug 3 10:55:37 2018 - [info] Resetting slave info on the new master.. 135 Fri Aug 3 10:55:37 2018 - [info] 192.168.1.102: Resetting slave info succeeded. 136 Fri Aug 3 10:55:37 2018 - [info] Master failover to 192.168.1.102(192.168.1.102:3306) completed successfully. 137 Fri Aug 3 10:55:37 2018 - [info] 138 139 ----- Failover Report ----- 140 141 app1: MySQL Master failover 192.168.1.101(192.168.1.101:3306) to 192.168.1.102(192.168.1.102:3306) succeeded 142 143 Master 192.168.1.101(192.168.1.101:3306) is down! 144 145 Check MHA Manager logs at zlm3:/var/log/masterha/app1/app1.log for details. 146 147 Started automated(non-interactive) failover. 148 Invalidated master IP address on 192.168.1.101(192.168.1.101:3306) 149 Selected 192.168.1.102(192.168.1.102:3306) as a new master. 150 192.168.1.102(192.168.1.102:3306): OK: Applying all logs succeeded. 151 192.168.1.102(192.168.1.102:3306): OK: Activated master IP address. 152 192.168.1.102(192.168.1.102:3306): Resetting slave info succeeded. 153 Master failover to 192.168.1.102(192.168.1.102:3306) completed successfully. 154 155 //Above failover report shows all the evidence and results of automation master switchover.All of the steps are executed successfully.