MHA切换检测日志分析
GTID模式
[root@db03 ~]# tail -f /etc/mha/manager.log
#在MySQL select ping:2006上出错（MySQL服务器已消失）
Tue Nov 19 17:52:00 2019 - [warning] Got error on MySQL select ping: 2006 (MySQL server has gone away)
#获得SSH检查脚本：退出0
Tue Nov 19 17:52:00 2019 - [info] Executing SSH check script: exit 0
# 健康检查：可以访问SSH到10.0.0.51
Tue Nov 19 17:52:00 2019 - [info] HealthCheck: SSH to 10.0.0.51 is reachable.
#在MySQL connect:2003上出现错误（无法连接到“10.0.0.51”（111）上的MySQL服务器）
Tue Nov 19 17:52:02 2019 - [warning] Got error on MySQL connect: 2003 (Can't connect to MySQL server on '10.0.0.51' (111))
#连接失败2次。。
Tue Nov 19 17:52:02 2019 - [warning] Connection failed 2 time(s)..
#在MySQL connect:2003上出现错误（无法连接到“10.0.0.51”（111）上的MySQL服务器）
Tue Nov 19 17:52:04 2019 - [warning] Got error on MySQL connect: 2003 (Can't connect to MySQL server on '10.0.0.51' (111))
#连接失败3次。。
Tue Nov 19 17:52:04 2019 - [warning] Connection failed 3 time(s)..
#在MySQL connect:2003上出现错误（无法连接到“10.0.0.51”（111）上的MySQL服务器）
Tue Nov 19 17:52:06 2019 - [warning] Got error on MySQL connect: 2003 (Can't connect to MySQL server on '10.0.0.51' (111))
#连接失败4次。。
Tue Nov 19 17:52:06 2019 - [warning] Connection failed 4 time(s)..
#无法从健康检查程序访问主服务器！
Tue Nov 19 17:52:06 2019 - [warning] Master is not reachable from health checker!
#无法访问Master 10.0.0.51（10.0.0.51:3306）！
Tue Nov 19 17:52:06 2019 - [warning] Master 10.0.0.51(10.0.0.51:3306) is not reachable!
#SSH是可访问的。
Tue Nov 19 17:52:06 2019 - [warning] SSH is reachable.
#连接到主服务器失败。再次读取配置文件/etc/masterha_default.cnf和/etc/mha/app1.cnf，并尝试连接到所有服务器以检查服务器状态。
Tue Nov 19 17:52:06 2019 - [info] Connecting to a master server failed. Reading configuration file /etc/masterha_default.cnf and /etc/mha/app1.cnf again, and trying to connect to all servers to check server status..
#未找到全局配置文件/etc/masterha/u default.cnf。跳过！
Tue Nov 19 17:52:06 2019 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
#正在从/etc/mha/app1.cnf.读取应用程序默认配置。。
Tue Nov 19 17:52:06 2019 - [info] Reading application default configuration from /etc/mha/app1.cnf..
# 正在从/etc/mha/app1.cnf.读取服务器配置。。
Tue Nov 19 17:52:06 2019 - [info] Reading server configuration from /etc/mha/app1.cnf..
#GTID故障转移模式=1
Tue Nov 19 17:52:07 2019 - [info] GTID failover mode = 1
#死机服务器：
Tue Nov 19 17:52:07 2019 - [info] Dead Servers:
#10.0.0.51（10.0.0.51:3306）
Tue Nov 19 17:52:07 2019 - [info] 10.0.0.51(10.0.0.51:3306)
#活动服务器：
Tue Nov 19 17:52:07 2019 - [info] Alive Servers:
#10.0.0.52（10.0.0.52:3306）
Tue Nov 19 17:52:07 2019 - [info] 10.0.0.52(10.0.0.52:3306)
#10.0.0.54（10.0.0.54:3306）
Tue Nov 19 17:52:07 2019 - [info] 10.0.0.54(10.0.0.54:3306)
#10.0.0.53（10.0.0.53:3306）
Tue Nov 19 17:52:07 2019 - [info] 10.0.0.53(10.0.0.53:3306)
#活着的奴隶：
Tue Nov 19 17:52:07 2019 - [info] Alive Slaves:
#10.0.0.52（10.0.0.52:3306）版本=5.6.44-log（从机之间的主要旧版本）日志箱：
Tue Nov 19 17:52:07 2019 - [info] 10.0.0.52(10.0.0.52:3306) Version=5.6.44-log (oldest major version between slaves) log-bin:enabled
#GTID开启
Tue Nov 19 17:52:07 2019 - [info] GTID ON
#从10.0.0.51复制（10.0.0.51:3306）
Tue Nov 19 17:52:07 2019 - [info] Replicating from 10.0.0.51(10.0.0.51:3306)
#10.0.0.54（10.0.0.54:3306）版本=5.6.44-log（从机之间最旧的主版本）日志箱：启用
Tue Nov 19 17:52:07 2019 - [info] 10.0.0.54(10.0.0.54:3306) Version=5.6.44-log (oldest major version between slaves) log-bin:enabled
#GTID开启
Tue Nov 19 17:52:07 2019 - [info] GTID ON
#从10.0.0.51复制（10.0.0.51:3306）
Tue Nov 19 17:52:07 2019 - [info] Replicating from 10.0.0.51(10.0.0.51:3306)
#10.0.0.53（10.0.0.53:3306）版本=5.6.44-log（从机之间最旧的主版本）日志箱：启用
Tue Nov 19 17:52:07 2019 - [info] 10.0.0.53(10.0.0.53:3306) Version=5.6.44-log (oldest major version between slaves) log-bin:enabled
#GTID开启
Tue Nov 19 17:52:07 2019 - [info] GTID ON
#从10.0.0.51复制（10.0.0.51:3306）
Tue Nov 19 17:52:07 2019 - [info] Replicating from 10.0.0.51(10.0.0.51:3306)
#检查从机配置。。
Tue Nov 19 17:52:07 2019 - [info] Checking slave configurations..
#只读=1未设置在从机10.0.0.52（10.0.0.52:3306）上。
Tue Nov 19 17:52:07 2019 - [info] read_only=1 is not set on slave 10.0.0.52(10.0.0.52:3306).
#只读=1未设置在从机10.0.0.54（10.0.0.54:3306）上。
Tue Nov 19 17:52:07 2019 - [info] read_only=1 is not set on slave 10.0.0.54(10.0.0.54:3306).
#只读=1未设置在从机10.0.0.53（10.0.0.53:3306）上。
Tue Nov 19 17:52:07 2019 - [info] read_only=1 is not set on slave 10.0.0.53(10.0.0.53:3306).
#正在检查复制筛选设置。。
Tue Nov 19 17:52:07 2019 - [info] Checking replication filtering settings..
#复制筛选检查正常。
Tue Nov 19 17:52:07 2019 - [info] Replication filtering check ok.
#主人倒下了！
Tue Nov 19 17:52:07 2019 - [info] Master is down!
#终止监视脚本。
Tue Nov 19 17:52:07 2019 - [info] Terminating monitoring script.
#获得退出代码20（主死亡）。
Tue Nov 19 17:52:07 2019 - [info] Got exit code 20 (Master dead).
#MHA：：主故障转移版本0.56。
Tue Nov 19 17:52:07 2019 - [info] MHA::MasterFailover version 0.56.
#启动主故障转移。
Tue Nov 19 17:52:07 2019 - [info] Starting master failover.
Tue Nov 19 17:52:07 2019 - [info]
#*第1阶段：配置检查阶段。。
Tue Nov 19 17:52:07 2019 - [info] * Phase 1: Configuration Check Phase..
Tue Nov 19 17:52:07 2019 - [info]
#GTID故障转移模式=1
Tue Nov 19 17:52:08 2019 - [info] GTID failover mode = 1
#死机服务器：
Tue Nov 19 17:52:08 2019 - [info] Dead Servers:
#10.0.0.51（10.0.0.51:3306）
Tue Nov 19 17:52:08 2019 - [info] 10.0.0.51(10.0.0.51:3306)
#通过MySQL检查主机可访问性（双重检查…）。。。
Tue Nov 19 17:52:08 2019 - [info] Checking master reachability via MySQL(double check)...
#好。
Tue Nov 19 17:52:08 2019 - [info] ok.
#活动服务器：
Tue Nov 19 17:52:08 2019 - [info] Alive Servers:
#10.0.0.52（10.0.0.52:3306）
Tue Nov 19 17:52:08 2019 - [info] 10.0.0.52(10.0.0.52:3306)
#10.0.0.54（10.0.0.54:3306）
Tue Nov 19 17:52:08 2019 - [info] 10.0.0.54(10.0.0.54:3306)
#10.0.0.53（10.0.0.53:3306）
Tue Nov 19 17:52:08 2019 - [info] 10.0.0.53(10.0.0.53:3306)
#活奴隶：
Tue Nov 19 17:52:08 2019 - [info] Alive Slaves:
#10.0.0.52（10.0.0.52:3306）版本=5.6.44-log（从机之间最旧的主版本）日志箱：启用
Tue Nov 19 17:52:08 2019 - [info] 10.0.0.52(10.0.0.52:3306) Version=5.6.44-log (oldest major version between slaves) log-bin:enabled
#GTID开启
Tue Nov 19 17:52:08 2019 - [info] GTID ON
#从10.0.0.51复制（10.0.0.51:3306）
Tue Nov 19 17:52:08 2019 - [info] Replicating from 10.0.0.51(10.0.0.51:3306)
#10.0.0.54（10.0.0.54:3306）版本=5.6.44-log（从机之间最旧的主版本）日志箱：启用
Tue Nov 19 17:52:08 2019 - [info] 10.0.0.54(10.0.0.54:3306) Version=5.6.44-log (oldest major version between slaves) log-bin:enabled
#GTID开启
Tue Nov 19 17:52:08 2019 - [info] GTID ON
#从10.0.0.51复制（10.0.0.51:3306）
Tue Nov 19 17:52:08 2019 - [info] Replicating from 10.0.0.51(10.0.0.51:3306)
#10.0.0.53（10.0.0.53:3306）版本=5.6.44-log（从机之间最旧的主版本）日志箱：启用
Tue Nov 19 17:52:08 2019 - [info] 10.0.0.53(10.0.0.53:3306) Version=5.6.44-log (oldest major version between slaves) log-bin:enabled
#GTID开启
Tue Nov 19 17:52:08 2019 - [info] GTID ON
#从10.0.0.51复制（10.0.0.51:3306）
Tue Nov 19 17:52:08 2019 - [info] Replicating from 10.0.0.51(10.0.0.51:3306)
#启动基于GTID的故障转移。
Tue Nov 19 17:52:08 2019 - [info] Starting GTID based failover.
Tue Nov 19 17:52:08 2019 - [info]
#**第1阶段：配置检查阶段完成。
Tue Nov 19 17:52:08 2019 - [info] ** Phase 1: Configuration Check Phase completed.
Tue Nov 19 17:52:08 2019 - [info]
#*第2阶段：主停堆阶段。。
Tue Nov 19 17:52:08 2019 - [info] * Phase 2: Dead Master Shutdown Phase..
Tue Nov 19 17:52:08 2019 - [info]
#强制关闭，使应用程序永远无法连接到当前主服务器。。
Tue Nov 19 17:52:08 2019 - [info] Forcing shutdown so that applications never connect to the current master..
#[警告]未设置master_ip_failover_脚本。正在跳过失效的主IP地址
Tue Nov 19 17:52:08 2019 - [warning] master_ip_failover_script is not set. Skipping invalidating dead master IP address.
#[警告]关闭脚本未设置。跳过显式关闭已死亡的主控形状。
Tue Nov 19 17:52:08 2019 - [warning] shutdown_script is not set. Skipping explicit shutting down of the dead master.
#*第2阶段：主停堆阶段完成。
Tue Nov 19 17:52:09 2019 - [info] * Phase 2: Dead Master Shutdown Phase completed.
Tue Nov 19 17:52:09 2019 - [info]
#*第3阶段：主恢复阶段。。
Tue Nov 19 17:52:09 2019 - [info] * Phase 3: Master Recovery Phase..
Tue Nov 19 17:52:09 2019 - [info]
#*第3.1阶段：获取最新的从属阶段。。
Tue Nov 19 17:52:09 2019 - [info] * Phase 3.1: Getting Latest Slaves Phase..
Tue Nov 19 17:52:09 2019 - [info]
#所有从属服务器上的最新二进制日志文件/位置是mysql bin.000005:191
Tue Nov 19 17:52:09 2019 - [info] The latest binary log file/position on all slaves is mysql-bin.000005:191
#检索到Gtid集：6e607afa-0a67-11ea-ac08-000c29f3321a:1-2
Tue Nov 19 17:52:09 2019 - [info] Retrieved Gtid Set: 6e607afa-0a67-11ea-ac08-000c29f3321a:1-2
#最新从属服务器（接收到最新中继日志文件的从属服务器）：
Tue Nov 19 17:52:09 2019 - [info] Latest slaves (Slaves that received relay log files to the latest):
#10.0.0.52（10.0.0.52:3306）版本=5.6.44-log（从机之间最旧的主版本）日志箱：启用
Tue Nov 19 17:52:09 2019 - [info] 10.0.0.52(10.0.0.52:3306) Version=5.6.44-log (oldest major version between slaves) log-bin:enabled
#GTID开启
Tue Nov 19 17:52:09 2019 - [info] GTID ON
#从10.0.0.51复制（10.0.0.51:3306）
Tue Nov 19 17:52:09 2019 - [info] Replicating from 10.0.0.51(10.0.0.51:3306)
#10.0.0.54（10.0.0.54:3306）版本=5.6.44-log（从机之间最旧的主版本）日志箱：启用
Tue Nov 19 17:52:09 2019 - [info] 10.0.0.54(10.0.0.54:3306) Version=5.6.44-log (oldest major version between slaves) log-bin:enabled
#GTID开启
Tue Nov 19 17:52:09 2019 - [info] GTID ON
#从10.0.0.51复制（10.0.0.51:3306）
Tue Nov 19 17:52:09 2019 - [info] Replicating from 10.0.0.51(10.0.0.51:3306)
#10.0.0.53（10.0.0.53:3306）版本=5.6.44-log（从机之间最旧的主版本）日志箱：启用
Tue Nov 19 17:52:09 2019 - [info] 10.0.0.53(10.0.0.53:3306) Version=5.6.44-log (oldest major version between slaves) log-bin:enabled
#GTID开启
Tue Nov 19 17:52:09 2019 - [info] GTID ON
#从10.0.0.51复制（10.0.0.51:3306）
Tue Nov 19 17:52:09 2019 - [info] Replicating from 10.0.0.51(10.0.0.51:3306)
#所有从机上最早的二进制日志文件/位置是mysql bin.000005:191
Tue Nov 19 17:52:09 2019 - [info] The oldest binary log file/position on all slaves is mysql-bin.000005:191
#检索到Gtid集：6e607afa-0a67-11ea-ac08-000c29f3321a:1-2
Tue Nov 19 17:52:09 2019 - [info] Retrieved Gtid Set: 6e607afa-0a67-11ea-ac08-000c29f3321a:1-2
#最老的奴隶：
Tue Nov 19 17:52:09 2019 - [info] Oldest slaves:
#10.0.0.52（10.0.0.52:3306）版本=5.6.44-log（从机之间最旧的主版本）日志箱：启用
Tue Nov 19 17:52:09 2019 - [info] 10.0.0.52(10.0.0.52:3306) Version=5.6.44-log (oldest major version between slaves) log-bin:enabled
#GTID开启
Tue Nov 19 17:52:09 2019 - [info] GTID ON
#从10.0.0.51复制（10.0.0.51:3306）
Tue Nov 19 17:52:09 2019 - [info] Replicating from 10.0.0.51(10.0.0.51:3306)
#10.0.0.54（10.0.0.54:3306）版本=5.6.44-log（从机之间最旧的主版本）日志箱：启用
Tue Nov 19 17:52:09 2019 - [info] 10.0.0.54(10.0.0.54:3306) Version=5.6.44-log (oldest major version between slaves) log-bin:enabled
#GTID开启
Tue Nov 19 17:52:09 2019 - [info] GTID ON
#从10.0.0.51复制（10.0.0.51:3306）
Tue Nov 19 17:52:09 2019 - [info] Replicating from 10.0.0.51(10.0.0.51:3306)
#10.0.0.53（10.0.0.53:3306）版本=5.6.44-log（从机之间最旧的主版本）日志箱：启用
Tue Nov 19 17:52:09 2019 - [info] 10.0.0.53(10.0.0.53:3306) Version=5.6.44-log (oldest major version between slaves) log-bin:enabled
#GTID开启
Tue Nov 19 17:52:09 2019 - [info] GTID ON
#从10.0.0.51复制（10.0.0.51:3306）
Tue Nov 19 17:52:09 2019 - [info] Replicating from 10.0.0.51(10.0.0.51:3306)
Tue Nov 19 17:52:09 2019 - [info]
#*第3.3阶段：确定新的主阶段。。
Tue Nov 19 17:52:09 2019 - [info] * Phase 3.3: Determining New Master Phase..
Tue Nov 19 17:52:09 2019 - [info]
#从奴隶中搜索新的主。。
Tue Nov 19 17:52:09 2019 - [info] Searching new master from slaves..
#配置文件中的候选主控形状：
Tue Nov 19 17:52:09 2019 - [info] Candidate masters from the configuration file:
#非候选硕士：
Tue Nov 19 17:52:09 2019 - [info] Non-candidate masters:
#新主机为10.0.0.52（10.0.0.52:3306）
Tue Nov 19 17:52:09 2019 - [info] New master is 10.0.0.52(10.0.0.52:3306)
#启动主故障转移。。
Tue Nov 19 17:52:09 2019 - [info] Starting master failover..
Tue Nov 19 17:52:09 2019 - [info]
#发件人：
From:
#10.0.0.51（10.0.0.51:3306）（当前主控）
10.0.0.51(10.0.0.51:3306) (current master)
+--10.0.0.52(10.0.0.52:3306)
+--10.0.0.54(10.0.0.54:3306)
+--10.0.0.53(10.0.0.53:3306)
#致：
To:
#10.0.0.52（10.0.0.52:3306）（新母版）
10.0.0.52(10.0.0.52:3306) (new master)
+--10.0.0.54(10.0.0.54:3306)
+--10.0.0.53(10.0.0.53:3306)
Tue Nov 19 17:52:09 2019 - [info]
#*第3.3阶段：新主恢复阶段。。
Tue Nov 19 17:52:09 2019 - [info] * Phase 3.3: New Master Recovery Phase..
Tue Nov 19 17:52:09 2019 - [info]
#等待应用所有日志。。
Tue Nov 19 17:52:09 2019 - [info] Waiting all logs to be applied..
#完成。
Tue Nov 19 17:52:09 2019 - [info] done.
#获取新主人的binlog名称和位置。。
Tue Nov 19 17:52:09 2019 - [info] Getting new master's binlog name and position..
#mysql bin.000002:438
Tue Nov 19 17:52:09 2019 - [info] mysql-bin.000002:438
#所有其他从属服务器应从此处开始复制。语句应该是：CHANGE MASTER
Tue Nov 19 17:52:09 2019 - [info] All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='10.0.0.52', MASTER_PORT=3306, MASTER_AUTO_POSITION=1, MASTER_USER='slave', MASTER_PASSWORD='xxx';
#主恢复成功。文件：Pos:Exec_Gtid_Set:mysql-bin.000002、438、6e607afa-0a67-11ea-ac08-000c29f3321a:1-2
Tue Nov 19 17:52:09 2019 - [info] Master Recovery succeeded. File:Pos:Exec_Gtid_Set: mysql-bin.000002, 438, 6e607afa-0a67-11ea-ac08-000c29f3321a:1-2
#[警告]未设置master_ip_failover_脚本。正在跳过接管新的主IP地址。
Tue Nov 19 17:52:09 2019 - [warning] master_ip_failover_script is not set. Skipping taking over new master IP address.
#**已成功完成主恢复。
Tue Nov 19 17:52:09 2019 - [info] ** Finished master recovery successfully.
#*第3阶段：主恢复阶段完成。
Tue Nov 19 17:52:09 2019 - [info] * Phase 3: Master Recovery Phase completed.
Tue Nov 19 17:52:09 2019 - [info]
#*第4阶段：从机恢复阶段。。
Tue Nov 19 17:52:09 2019 - [info] * Phase 4: Slaves Recovery Phase..
Tue Nov 19 17:52:09 2019 - [info]
Tue Nov 19 17:52:09 2019 - [info]
#*第4.1阶段：并行启动从机。。
Tue Nov 19 17:52:09 2019 - [info] * Phase 4.1: Starting Slaves in parallel..
Tue Nov 19 17:52:09 2019 - [info]
#——主机10.0.0.54（10.0.0.54:3306）上的从机恢复已启动，pid:22464。如果需要时间，请检查tmp log/etc/mha/app1/10.0.0.54_3306_20191119175207.log。。
Tue Nov 19 17:52:09 2019 - [info] -- Slave recovery on host 10.0.0.54(10.0.0.54:3306) started, pid: 22464. Check tmp log /etc/mha/app1/10.0.0.54_3306_20191119175207.log if it takes time..
#]——主机10.0.0.53（10.0.0.53:3306）上的从机恢复已启动，pid:22465。如果需要时间，请检查tmp log/etc/mha/app1/10.0.0.53_3306_20191119175207.log。。
Tue Nov 19 17:52:09 2019 - [info] -- Slave recovery on host 10.0.0.53(10.0.0.53:3306) started, pid: 22465. Check tmp log /etc/mha/app1/10.0.0.53_3306_20191119175207.log if it takes time..
Tue Nov 19 17:52:10 2019 - [info]
#记录来自10.0.0.54的消息。。。
Tue Nov 19 17:52:10 2019 - [info] Log messages from 10.0.0.54 ...
Tue Nov 19 17:52:10 2019 - [info]
#重置从机10.0.0.54（10.0.0.54:3306）并从新主机10.0.0.52（10.0.0.52:3306）开始复制。。
Tue Nov 19 17:52:09 2019 - [info] Resetting slave
10.0.0.54(10.0.0.54:3306) and starting replication from the new master 10.0.0.52(10.0.0.52:3306)..
#执行变更主控。
Tue Nov 19 17:52:09 2019 - [info] Executed CHANGE MASTER.
#从系统启动。
Tue Nov 19 17:52:09 2019 - [info] Slave started.
#gtid_wait（6e607afa-0a67-11ea-ac08-000c29f3321a:1-2）10.0.0.54（10.0.0.54:3306）完成。执行了0个事件。
Tue Nov 19 17:52:09 2019 - [info] gtid_wait(6e607afa-0a67-11ea-ac08-000c29f3321a:1-2) completed on 10.0.0.54(10.0.0.54:3306). Executed 0 events.
#10.0.0.54的日志消息结束。
Tue Nov 19 17:52:10 2019 - [info] End of log messages from 10.0.0.54.
#——主机10.0.0.54（10.0.0.54:3306）上的从机已启动。
Tue Nov 19 17:52:10 2019 - [info] -- Slave on host 10.0.0.54(10.0.0.54:3306) started.
Tue Nov 19 17:52:11 2019 - [info]
#10.0.0.53中的日志消息。。。
Tue Nov 19 17:52:11 2019 - [info] Log messages from 10.0.0.53 ...
Tue Nov 19 17:52:11 2019 - [info]
#重置从机10.0.0.53（10.0.0.53:3306）并从新主机10.0.0.52（10.0.0.52:3306）开始复制。。
Tue Nov 19 17:52:09 2019 - [info] Resetting slave
10.0.0.53(10.0.0.53:3306) and starting replication from the new master 10.0.0.52(10.0.0.52:3306)..
#执行变更主控。
Tue Nov 19 17:52:09 2019 - [info] Executed CHANGE MASTER.
#奴隶启动。
Tue Nov 19 17:52:10 2019 - [info] Slave started.
#gtid_wait（6e607afa-0a67-11ea-ac08-000c29f3321a:1-2）10.0.0.53（10.0.0.53:3306）完成。执行了0个事件。
Tue Nov 19 17:52:10 2019 - [info] gtid_wait(6e607afa-0a67-11ea-ac08-000c29f3321a:1-2) completed on 10.0.0.53(10.0.0.53:3306). Executed 0 events.
#10.0.0.53的日志消息结束。
Tue Nov 19 17:52:11 2019 - [info] End of log messages from 10.0.0.53.
#——主机10.0.0.53（10.0.0.53:3306）上的从机已启动。
Tue Nov 19 17:52:11 2019 - [info] -- Slave on host 10.0.0.53(10.0.0.53:3306) started.
#所有新的从属服务器已成功恢复。
Tue Nov 19 17:52:11 2019 - [info] All new slave servers recovered successfully.
Tue Nov 19 17:52:11 2019 - [info]
#*第5阶段：新的主清理阶段。。
Tue Nov 19 17:52:11 2019 - [info] * Phase 5: New master cleanup phase..
Tue Nov 19 17:52:11 2019 - [info]
#在新主机上重置从机信息。。
Tue Nov 19 17:52:11 2019 - [info] Resetting slave info on the new master..
#10.0.0.52:重置从机信息成功。
Tue Nov 19 17:52:11 2019 - [info] 10.0.0.52: Resetting slave info succeeded.
#主故障转移到10.0.0.52（10.0.0.52:3306）成功完成。
Tue Nov 19 17:52:11 2019 - [info] Master failover to 10.0.0.52(10.0.0.52:3306) completed successfully.
#从/etc/mha/app1.cnf中删除了server1条目。
Tue Nov 19 17:52:11 2019 - [info] Deleted server1 entry from /etc/mha/app1.cnf .
Tue Nov 19 17:52:11 2019 - [info]
#-----故障转移报告-----
----- Failover Report -----
#app1:MySQL主故障转移10.0.0.51（10.0.0.51:3306）到10.0.0.52（10.0.0.52:3306）成功
app1: MySQL Master failover 10.0.0.51(10.0.0.51:3306) to 10.0.0.52(10.0.0.52:3306) succeeded
#大师10.0.0.51（10.0.0.51:3306）倒下了！
Master 10.0.0.51(10.0.0.51:3306) is down!
#有关详细信息，请查看位于db03:/etc/MHA/Manager.log的MHA Manager日志。
Check MHA Manager logs at db03:/etc/mha/manager.log for details.
#已启动自动（非交互式）故障转移。
Started automated(non-interactive) failover.
#选择10.0.0.52（10.0.0.52:3306）作为新的主控形状。
Selected 10.0.0.52(10.0.0.52:3306) as a new master.
#10.0.0.52（10.0.0.52:3306）：确定：应用所有日志成功。
10.0.0.52(10.0.0.52:3306): OK: Applying all logs succeeded.
#10.0.0.54（10.0.0.54:3306）：确定：从机启动，从10.0.0.52（10.0.0.52:3306）复制
10.0.0.54(10.0.0.54:3306): OK: Slave started, replicating from 10.0.0.52(10.0.0.52:3306)
#10.0.0.53（10.0.0.53:3306）：确定：从机启动，从10.0.0.52（10.0.0.52:3306）复制
10.0.0.53(10.0.0.53:3306): OK: Slave started, replicating from 10.0.0.52(10.0.0.52:3306)
#10.0.0.52（10.0.0.52:3306）：重置从机信息成功。
10.0.0.52(10.0.0.52:3306): Resetting slave info succeeded.
#已成功完成到10.0.0.52（10.0.0.52:3306）的主故障转移。
Master failover to 10.0.0.52(10.0.0.52:3306) completed successfully.
普通模式
#发件人：
From:
#10.0.0.51（10.0.0.51:3306）（当前主控）
10.0.0.51(10.0.0.51:3306) (current master)
+--10.0.0.52(10.0.0.52:3306)
+--10.0.0.54(10.0.0.54:3306)
+--10.0.0.53(10.0.0.53:3306)
#致：
To:
#10.0.0.52（10.0.0.52:3306）（新母版）
10.0.0.52(10.0.0.52:3306) (new master)
+--10.0.0.54(10.0.0.54:3306)
+--10.0.0.53(10.0.0.53:3306)
Tue Nov 19 19:22:21 2019 - [info]
#*第3.3阶段：新主差异日志生成阶段。。
Tue Nov 19 19:22:21 2019 - [info] * Phase 3.3: New Master Diff Log Generation Phase..
Tue Nov 19 19:22:21 2019 - [info]
#此服务器具有所有中继日志。不需要从最新的从机生成diff文件。
Tue Nov 19 19:22:21 2019 - [info] This server has all relay logs. No need to generate diff files from the latest slave.
#发送binlog。。
Tue Nov 19 19:22:21 2019 - [info] Sending binlog..
Tue Nov 19 19:22:22 2019 - [info] scp from local:/etc/mha/app1/saved_master_binlog_from_10.0.0.51_3306_20191119192216.binlog to root@10.0.0.52:/var/tmp/saved_master_binlog_from_10.0.0.51_3306_20191119192216.binlog succeeded.
Tue Nov 19 19:22:22 2019 - [info]
#*第3.4阶段：主日志应用阶段。。
Tue Nov 19 19:22:22 2019 - [info] * Phase 3.4: Master Log Apply Phase..
Tue Nov 19 19:22:22 2019 - [info]
#*注意：如果在此阶段发生任何错误，则需要手动恢复。
Tue Nov 19 19:22:22 2019 - [info] *NOTICE: If any error happens from this phase, manual recovery is needed.
#10.0.0.52（10.0.0.52:3306）开始恢复。。
Tue Nov 19 19:22:22 2019 - [info] Starting recovery on 10.0.0.52(10.0.0.52:3306)..
#生成差异成功。
Tue Nov 19 19:22:22 2019 - [info] Generating diffs succeeded.
#等待所有中继日志应用。
Tue Nov 19 19:22:22 2019 - [info] Waiting until all relay logs are applied.
#完成。
Tue Nov 19 19:22:22 2019 - [info] done.
#获取奴隶状态。。
Tue Nov 19 19:22:22 2019 - [info] Getting slave status..
#此从机（10.0.0.52）的Exec_Master_Log_Pos等于Read_Master_Log_Pos（mysql bin.000006:120）。无需从Exec_Master_Log_Pos恢复。
Tue Nov 19 19:22:22 2019 - [info] This slave(10.0.0.52)'s Exec_Master_Log_Pos equals to Read_Master_Log_Pos(mysql-bin.000006:120). No need to recover from Exec_Master_Log_Pos.
#连接到目标从属主机10.0.0.52，运行恢复脚本。。
Tue Nov 19 19:22:22 2019 - [info] Connecting to the target slave host 10.0.0.52, running recover script..
#执行命令：
Tue Nov 19 19:22:22 2019 - [info] Executing command: apply_diff_relay_logs --command=apply --slave_user='mha' --slave_host=10.0.0.52 --slave_ip=10.0.0.52 --slave_port=3306 --apply_files=/var/tmp/saved_master_binlog_from_10.0.0.51_3306_20191119192216.binlog --workdir=/var/tmp --target_version=5.6.44-log --timestamp=20191119192216 --handle_raw_binlog=1 --disable_log_bin=0 --manager_version=0.56 --slave_pass=xxx
Tue Nov 19 19:22:22 2019 - [info]
#MySQL客户端版本是5.6.44。使用--二进制模式。应用差分二进制/中继日志文件/var/tmp/saved_master_binlog_from_10.0.0.51_3306_20191119192216.binlog on 10.0.0.52:3306。这可能需要很长时间。。。
MySQL client version is 5.6.44. Using --binary-mode.
Applying differential binary/relay log files /var/tmp/saved_master_binlog_from_10.0.0.51_3306_20191119192216.binlog on 10.0.0.52:3306. This may take long time...
#应用日志文件成功。
Applying log files succeeded.
#所有中继日志均已成功应用。
Tue Nov 19 19:22:22 2019 - [info] All relay logs were successfully applied.
#获取新主人的binlog名称和位置。。
Tue Nov 19 19:22:22 2019 - [info] Getting new master's binlog name and position..
Tue Nov 19 19:22:22 2019 - [info] mysql-bin.000003:120
#所有其他从机应从此处开始复制。语句应该是：
Tue Nov 19 19:22:22 2019 - [info] All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='10.0.0.52', MASTER_PORT=3306, MASTER_LOG_FILE='mysql-bin.000003', MASTER_LOG_POS=120, MASTER_USER='slave', MASTER_PASSWORD='xxx';
#-[警告]未设置master_ip_failover_脚本。正在跳过接管新的主IP地址。
Tue Nov 19 19:22:22 2019 - [warning] master_ip_failover_script is not set. Skipping taking over new master IP address.
#**已成功完成主恢复。
Tue Nov 19 19:22:22 2019 - [info] ** Finished master recovery successfully.
#*第3阶段：主恢复阶段完成。
Tue Nov 19 19:22:22 2019 - [info] * Phase 3: Master Recovery Phase completed.
Tue Nov 19 19:22:22 2019 - [info]
#*第4阶段：奴隶恢复阶段。。
Tue Nov 19 19:22:22 2019 - [info] * Phase 4: Slaves Recovery Phase..
Tue Nov 19 19:22:22 2019 - [info]
#*第4.1阶段：启动并行从Diff日志生成阶段。。
Tue Nov 19 19:22:22 2019 - [info] * Phase 4.1: Starting Parallel Slave Diff Log Generation Phase..
Tue Nov 19 19:22:22 2019 - [info]
#——主机10.0.0.54（10.0.0.54:3306）上的从属差异文件生成已启动，pid:23478。如果需要时间，请检查tmp log/etc/mha/app1/10.0.0.54_3306_20191119192216.log。。
Tue Nov 19 19:22:22 2019 - [info] -- Slave diff file generation on host 10.0.0.54(10.0.0.54:3306) started, pid: 23478. Check tmp log /etc/mha/app1/10.0.0.54_3306_20191119192216.log if it takes time..
#——主机10.0.0.53（10.0.0.53:3306）上的从属差异文件生成已启动，pid:23479。如果需要时间，请检查tmp log/etc/mha/app1/10.0.0.53_3306_20191119192216.log。。
Tue Nov 19 19:22:22 2019 - [info] -- Slave diff file generation on host 10.0.0.53(10.0.0.53:3306) started, pid: 23479. Check tmp log
/etc/mha/app1/10.0.0.53_3306_20191119192216.log if it takes time..
Tue Nov 19 19:22:23 2019 - [info]
#10.0.0.54中的日志消息。。。
Tue Nov 19 19:22:23 2019 - [info] Log messages from 10.0.0.54 ...
Tue Nov 19 19:22:23 2019 - [info]
#此服务器具有所有中继日志。不需要从最新的从机生成diff文件。
Tue Nov 19 19:22:22 2019 - [info] This server has all relay logs. No need to generate diff files from the latest slave.
#10.0.0.54的日志消息结束。
Tue Nov 19 19:22:23 2019 - [info] End of log messages from 10.0.0.54.
#-10.0.0.54（10.0.0.54:3306）有最新的中继日志事件。
Tue Nov 19 19:22:23 2019 - [info] -- 10.0.0.54(10.0.0.54:3306) has the latest relay log events.
Tue Nov 19 19:22:23 2019 - [info]
#10.0.0.53中的日志消息。。。
Tue Nov 19 19:22:23 2019 - [info] Log messages from 10.0.0.53 ...
Tue Nov 19 19:22:23 2019 - [info]
#此服务器具有所有中继日志。不需要从最新的从机生成diff文件。
Tue Nov 19 19:22:22 2019 - [info] This server has all relay logs. No need to generate diff files from the latest slave.
#10.0.0.53的日志消息结束。
Tue Nov 19 19:22:23 2019 - [info] End of log messages from 10.0.0.53.
#-10.0.0.53（10.0.0.53:3306）有最新的中继日志事件。
Tue Nov 19 19:22:23 2019 - [info] -- 10.0.0.53(10.0.0.53:3306) has the latest relay log events.
#从最新的从机生成中继diff文件成功。
Tue Nov 19 19:22:23 2019 - [info] Generating relay diff files from the latest slave succeeded.
Tue Nov 19 19:22:23 2019 - [info]
#*第4.2阶段：启动并行从日志应用阶段。。
Tue Nov 19 19:22:23 2019 - [info] * Phase 4.2: Starting Parallel Slave Log Apply Phase..
Tue Nov 19 19:22:23 2019 - [info]
#——主机10.0.0.54（10.0.0.54:3306）上的从机恢复已启动，pid:23482。检查tmp日志
Tue Nov 19 19:22:23 2019 - [info] -- Slave recovery on host 10.0.0.54(10.0.0.54:3306) started, pid: 23482. Check tmp log /etc/mha/app1/10.0.0.54_3306_20191119192216.log if it takes time..
Tue Nov 19 19:22:23 2019 - [info] -- Slave recovery on host 10.0.0.53(10.0.0.53:3306) started, pid: 23483. Check tmp log /etc/mha/app1/10.0.0.53_3306_20191119192216.log if it takes time..
Tue Nov 19 19:22:24 2019 - [info]
#记录来自10.0.0.54的消息。。。
Tue Nov 19 19:22:24 2019 - [info] Log messages from 10.0.0.54 ...
Tue Nov 19 19:22:24 2019 - [info]
#发送binlog。。
Tue Nov 19 19:22:23 2019 - [info] Sending binlog..
Tue Nov 19 19:22:23 2019 - [info] scp from local:/etc/mha/app1/saved_master_binlog_from_10.0.0.51_3306_20191119192216.binlog to root@10.0.0.54:/var/tmp/saved_master_binlog_from_10.0.0.51_3306_20191119192216.binlog succeeded.
#10.0.0.54（10.0.0.54:3306）开始恢复。。
Tue Nov 19 19:22:23 2019 - [info] Starting recovery on 10.0.0.54(10.0.0.54:3306)..
#生成差异成功。
Tue Nov 19 19:22:23 2019 - [info] Generating diffs succeeded.
#等待所有中继日志应用。
Tue Nov 19 19:22:23 2019 - [info] Waiting until all relay logs are applied.
#完成。
Tue Nov 19 19:22:23 2019 - [info] done.
#获取奴隶状态。。
Tue Nov 19 19:22:23 2019 - [info] Getting slave status..
#此从机（10.0.0.54）的Exec_Master_Log_Pos等于Read_Master_Log_Pos（mysql bin.000006:120）。无需从Exec_Master_Log_Pos恢复。
Tue Nov 19 19:22:23 2019 - [info] This slave(10.0.0.54)'s Exec_Master_Log_Pos equals to Read_Master_Log_Pos(mysql-bin.000006:120). No need to recover from Exec_Master_Log_Pos.
#连接到目标从属主机10.0.0.54，运行恢复脚本。。
Tue Nov 19 19:22:23 2019 - [info] Connecting to the target slave host 10.0.0.54, running recover script..
#执行命令：
Tue Nov 19 19:22:23 2019 - [info] Executing command: apply_diff_relay_logs --command=apply --slave_user='mha' --slave_host=10.0.0.54 --slave_ip=10.0.0.54 --slave_port=3306 --apply_files=/var/tmp/saved_master_binlog_from_10.0.0.51_3306_20191119192216.binlog --workdir=/var/tmp --target_version=5.6.44-log --timestamp=20191119192216 --handle_raw_binlog=1 --disable_log_bin=0 --manager_version=0.56 --slave_pass=xxx
Tue Nov 19 19:22:24 2019 - [info]
#MySQL客户端版本是5.6.44。使用--二进制模式。应用差分二进制/中继日志文件/var/tmp/saved_master_binlog_from_10.0.0.51_3306_20191119192216.binlog on 10.0.0.54:3306。这可能需要很长时间。。。 应用日志文件成功。
MySQL client version is 5.6.44. Using --binary-mode.
Applying differential binary/relay log files /var/tmp/saved_master_binlog_from_10.0.0.51_3306_20191119192216.binlog on 10.0.0.54:3306. This may take long time...
Applying log files succeeded.
#所有中继日志均已成功应用。
Tue Nov 19 19:22:24 2019 - [info] All relay logs were successfully applied.
#重置从机10.0.0.54（10.0.0.54:3306）并从新主机10.0.0.52（10.0.0.52:3306）开始复制。。
Tue Nov 19 19:22:24 2019 - [info] Resetting slave 10.0.0.54(10.0.0.54:3306) and starting replication from the new master 10.0.0.52(10.0.0.52:3306)..
#执行变更主控。
Tue Nov 19 19:22:24 2019 - [info] Executed CHANGE MASTER.
#奴隶启动。
Tue Nov 19 19:22:24 2019 - [info] Slave started.
#10.0.0.54的日志消息结束。
Tue Nov 19 19:22:24 2019 - [info] End of log messages from 10.0.0.54.
#——主机10.0.0.54（10.0.0.54:3306）上的从机恢复成功。
Tue Nov 19 19:22:24 2019 - [info] -- Slave recovery on host 10.0.0.54(10.0.0.54:3306) succeeded.
Tue Nov 19 19:22:25 2019 - [info]
#记录来自10.0.0.53的消息。。。
Tue Nov 19 19:22:25 2019 - [info] Log messages from 10.0.0.53 ...
Tue Nov 19 19:22:25 2019 - [info]
#发送binlog。。
Tue Nov 19 19:22:23 2019 - [info] Sending binlog..
Tue Nov 19 19:22:24 2019 - [info] scp from local:/etc/mha/app1/saved_master_binlog_from_10.0.0.51_3306_20191119192216.binlog to root@10.0.0.53:/var/tmp/saved_master_binlog_from_10.0.0.51_3306_20191119192216.binlog succeeded.
#10.0.0.53（10.0.0.53:3306）开始恢复。。
Tue Nov 19 19:22:24 2019 - [info] Starting recovery on 10.0.0.53(10.0.0.53:3306)..
#生成差异成功。
Tue Nov 19 19:22:24 2019 - [info] Generating diffs succeeded.
#等待所有中继日志应用。
Tue Nov 19 19:22:24 2019 - [info] Waiting until all relay logs are applied.
#完成。
Tue Nov 19 19:22:24 2019 - [info] done.
#获取奴隶状态。。
Tue Nov 19 19:22:24 2019 - [info] Getting slave status..
#此从机（10.0.0.53）的Exec_Master_Log_Pos等于Read_Master_Log_Pos（mysql bin.000006:120）。无需从Exec_Master_Log_Pos恢复。
Tue Nov 19 19:22:24 2019 - [info] This slave(10.0.0.53)'s Exec_Master_Log_Pos equals to Read_Master_Log_Pos(mysql-bin.000006:120). No need to recover from Exec_Master_Log_Pos.
#连接到目标从属主机10.0.0.53，运行恢复脚本。。
Tue Nov 19 19:22:24 2019 - [info] Connecting to the target slave host 10.0.0.53, running recover script..
#执行命令：
Tue Nov 19 19:22:24 2019 - [info] Executing command: apply_diff_relay_logs --command=apply --slave_user='mha' --slave_host=10.0.0.53 --slave_ip=10.0.0.53 --slave_port=3306 --apply_files=/var/tmp/saved_master_binlog_from_10.0.0.51_3306_20191119192216.binlog --workdir=/var/tmp --target_version=5.6.44-log --timestamp=20191119192216 --handle_raw_binlog=1 --disable_log_bin=0 --manager_version=0.56 --slave_pass=xxx
Tue Nov 19 19:22:24 2019 - [info]
#MySQL客户端版本是5.6.44。使用--二进制模式。应用差分二进制/中继日志文件/var/tmp/saved_master_binlog_from_10.0.0.51_3306_20191119192216.binlog on 10.0.0.53:3306。这可能需要很长时间。。。应用日志文件成功。
MySQL client version is 5.6.44. Using --binary-mode.
Applying differential binary/relay log files /var/tmp/saved_master_binlog_from_10.0.0.51_3306_20191119192216.binlog on 10.0.0.53:3306. This may take long time...
Applying log files succeeded.
#所有中继日志均已成功应用。
Tue Nov 19 19:22:24 2019 - [info] All relay logs were successfully applied.
#重置从机10.0.0.53（10.0.0.53:3306）并从新主机10.0.0.52（10.0.0.52:3306）开始复制。。
Tue Nov 19 19:22:24 2019 - [info] Resetting slave 10.0.0.53(10.0.0.53:3306) and starting replication from the new master 10.0.0.52(10.0.0.52:3306)..
#执行变更主控。
Tue Nov 19 19:22:24 2019 - [info] Executed CHANGE MASTER.
#奴隶启动。
Tue Nov 19 19:22:24 2019 - [info] Slave started.
#10.0.0.53的日志消息结束。
Tue Nov 19 19:22:25 2019 - [info] End of log messages from 10.0.0.53.
#——主机10.0.0.53（10.0.0.53:3306）上的从机恢复成功。
Tue Nov 19 19:22:25 2019 - [info] -- Slave recovery on host 10.0.0.53(10.0.0.53:3306) succeeded.
#所有新的从属服务器已成功恢复。
Tue Nov 19 19:22:25 2019 - [info] All new slave servers recovered successfully.
Tue Nov 19 19:22:25 2019 - [info]
#*第5阶段：新的主清理阶段。。
Tue Nov 19 19:22:25 2019 - [info] * Phase 5: New master cleanup phase..
Tue Nov 19 19:22:25 2019 - [info]
#在新主机上重置从机信息。。
Tue Nov 19 19:22:25 2019 - [info] Resetting slave info on the new master..
#10.0.0.52:重置从机信息成功。
Tue Nov 19 19:22:25 2019 - [info] 10.0.0.52: Resetting slave info succeeded.
#主故障转移到10.0.0.52（10.0.0.52:3306）成功完成。
Tue Nov 19 19:22:25 2019 - [info] Master failover to 10.0.0.52(10.0.0.52:3306) completed successfully.
#从/etc/mha/app1.cnf中删除了server1条目。
Tue Nov 19 19:22:25 2019 - [info] Deleted server1 entry from /etc/mha/app1.cnf .
Tue Nov 19 19:22:25 2019 - [info]
#-----故障转移报告-----
----- Failover Report -----
#app1:MySQL主故障转移10.0.0.51（10.0.0.51:3306）到10.0.0.52（10.0.0.52:3306）成功
app1: MySQL Master failover 10.0.0.51(10.0.0.51:3306) to 10.0.0.52(10.0.0.52:3306) succeeded
#大师10.0.0.51（10.0.0.51:3306）倒下了！
Master 10.0.0.51(10.0.0.51:3306) is down!
#有关详细信息，请查看位于db03:/etc/MHA/Manager.log的MHA Manager日志。
Check MHA Manager logs at db03:/etc/mha/manager.log for details.
#已启动自动（非交互式）故障转移。
Started automated(non-interactive) failover.
#最新的从机10.0.0.52（10.0.0.52:3306）具有用于恢复的所有中继日志。
The latest slave 10.0.0.52(10.0.0.52:3306) has all relay logs for recovery.
#选择10.0.0.52（10.0.0.52:3306）作为新的主控形状。
Selected 10.0.0.52(10.0.0.52:3306) as a new master.
#10.0.0.52（10.0.0.52:3306）：确定：应用所有日志成功。
10.0.0.52(10.0.0.52:3306): OK: Applying all logs succeeded.
#10.0.0.54（10.0.0.54:3306）：此主机具有最新的中继日志事件。
10.0.0.54(10.0.0.54:3306): This host has the latest relay log events.
#10.0.0.53（10.0.0.53:3306）：此主机具有最新的中继日志事件。
10.0.0.53(10.0.0.53:3306): This host has the latest relay log events.
#从最新的从机生成中继diff文件成功。
Generating relay diff files from the latest slave succeeded.
#10.0.0.54（10.0.0.54:3306）：确定：应用所有日志成功。从机启动，从10.0.0.52复制（10.0.0.52:3306）
10.0.0.54(10.0.0.54:3306): OK: Applying all logs succeeded. Slave started, replicating from 10.0.0.52(10.0.0.52:3306)
#10.0.0.53（10.0.0.53:3306）：确定：应用所有日志成功。从机启动，从10.0.0.52复制（10.0.0.52:3306）
10.0.0.53(10.0.0.53:3306): OK: Applying all logs succeeded. Slave started, replicating from 10.0.0.52(10.0.0.52:3306)
#10.0.0.52（10.0.0.52:3306）：重置从机信息成功。已成功完成到10.0.0.52（10.0.0.52:3306）的主故障转移。
10.0.0.52(10.0.0.52:3306): Resetting slave info succeeded.
Master failover to 10.0.0.52(10.0.0.52:3306) completed successfully.