mysql5.7同步复制报错故障处理
# 报错 1060，具体如下
Last_Errno: 1060
Last_Error: Coordinator stopped because there were error(s) in the worker(s). The most recent failure being: Worker 0 failed executing transaction 'ANONYMOUS' at master log mysql-bin.000311, end_log_pos 352951786. See error log and/or performance_schema.replication_applier_status_by_worker table for more details about this failure or others, if any.
# 可以使用命令查看具体的执行sql语句
# 查找 positions 的位置向后10行
/usr/local/percona-server-5.7.14/bin/mysqlbinlog --no-defaults -v -v --base64-output=DECODE-ROWS mysql-bin.000311 | grep -A 10 352951786
# 用这条语句定位不到具体的执行sql
[root@newcms:/data/mysql_data]# /usr/local/percona-server-5.7.14/bin/mysqlbinlog --base64-output=DECODE-ROWS -v --start-position=352951786 --stop-position=352951786 mysql-bin.000311 | more
/*!50530 SET @@SESSION.PSEUDO_SLAVE_MODE=1*/;
/*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/;
DELIMITER /*!*/;
SET @@SESSION.GTID_NEXT= 'AUTOMATIC' /* added by mysqlbinlog */ /*!*/;
DELIMITER ;
# End of log file
/*!50003 SET COMPLETION_TYPE=@OLD_COMPLETION_TYPE*/;
/*!50530 SET @@SESSION.PSEUDO_SLAVE_MODE=0*/;
# 推荐使用如下定位命令：
[root@newcms:/data/mysql_data]# /usr/local/percona-server-5.7.14/bin/mysqlbinlog --no-defaults -v -v --base64-output=DECODE-ROWS mysql-bin.000311 | grep -A 10 352951786
#191202 9:13:24 server id 1189 end_log_pos 352951786 CRC32 0xb6b23b6f Query thread_id=4299701 exec_time=0 error_code=0
SET TIMESTAMP=1575249204/*!*/;
/*!C utf8 *//*!*/;
SET @@session.character_set_client=33,@@session.collation_connection=33,@@session.collation_server=45/*!*/;
# 可以看到关键的执行sql语句，添加了字段，并且新增内容
ALTER TABLE `template_page_9342` ADD COLUMN `content` longtext NOT NULL
/*!*/;
# at 352951786
#191202 9:13:24 server id 1189 end_log_pos 352951851 CRC32 0x5636b59f Anonymous_GTID last_committed=3226 sequence_number=3227
SET @@SESSION.GTID_NEXT= 'ANONYMOUS'/*!*/;
# at 352951851
#191202 9:13:24 server id 1189 end_log_pos 352951929 CRC32 0x8f64d217 Query thread_id=4299701 exec_time=0 error_code=0
SET TIMESTAMP=1575249204/*!*/;
BEGIN
/*!*/;
# at 352951929
#191202 9:13:24 server id 1189 end_log_pos 352952048 CRC32 0x96da7aa4 Table_map: `cms_global`.`template_fields` mapped to number 139
# at 352952048
# 对比主库和从库的表结构，可以看到是一致的，直接跳过即可
STOP SLAVE; SET GLOBAL SQL_SLAVE_SKIP_COUNTER = 1; START SLAVE;
# 主库
mysql> desc template_page_9342;
+-------------+----------------------------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-------------+----------------------------------+------+-----+---------+----------------+
| page_id | int(10) unsigned | NO | PRI | NULL | auto_increment |
| title | char(100) | NO | | NULL | |
| keyword | char(250) | NO | | NULL | |
| description | char(200) | NO | | NULL | |
| url | varchar(255) | NO | | NULL | |
| entity_id | int(10) unsigned | NO | | NULL | |
| entity_name | char(100) | NO | | NULL | |
| is_default | enum('Y','N') | NO | | Y | |
| html_time | datetime | NO | | NULL | |
| action_name | char(20) | NO | | NULL | |
| user_id_a | int(11) unsigned | NO | | NULL | |
| user_id_e | int(11) unsigned | NO | | NULL | |
| add_time | datetime | NO | | NULL | |
| edit_time | datetime | NO | | NULL | |
| user_name | char(20) | NO | | NULL | |
| state | enum('able','disable','deleted') | NO | | able | |
| content | longtext | NO | | NULL | |
| style | longtext | NO | | NULL | |
| script | longtext | NO | | NULL | |
+-------------+----------------------------------+------+-----+---------+----------------+
# 从库
MySQL [cms_global]> desc template_page_9342;
+-------------+----------------------------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-------------+----------------------------------+------+-----+---------+----------------+
| page_id | int(10) unsigned | NO | PRI | NULL | auto_increment |
| title | char(100) | NO | | NULL | |
| keyword | char(250) | NO | | NULL | |
| description | char(200) | NO | | NULL | |
| url | varchar(255) | NO | | NULL | |
| entity_id | int(10) unsigned | NO | | NULL | |
| entity_name | char(100) | NO | | NULL | |
| is_default | enum('Y','N') | NO | | Y | |
| html_time | datetime | NO | | NULL | |
| action_name | char(20) | NO | | NULL | |
| user_id_a | int(11) unsigned | NO | | NULL | |
| user_id_e | int(11) unsigned | NO | | NULL | |
| add_time | datetime | NO | | NULL | |
| edit_time | datetime | NO | | NULL | |
| user_name | char(20) | NO | | NULL | |
| state | enum('able','disable','deleted') | NO | | able | |
| content | longtext | NO | | NULL | |
| style | longtext | NO | | NULL | |
| script | longtext | NO | | NULL | |
+-------------+----------------------------------+------+-----+---------+----------------+
# 在主库中查找binlog报错，ERROR: Error in Log_event::read_log_event(): 'Sanity check failed' 是因为默认的mysqlbinlog命令是yum安装的低版本mysqlbinlog，需要指定我们运行的mysql高版本的命令
[root@newcms:/data/mysql_data]# mysqlbinlog --no-defaults -v -v --base64-output=DECODE-ROWS mysql-bin.000311 | grep -A 10 352951786
ERROR: Error in Log_event::read_log_event(): 'Sanity check failed', data_len: 31, event_type: 35
ERROR: Could not read entry at offset 123: Error in log format or read error.
[root@newcms:/data/mysql_data]# mysqlbinlog --base64-output=DECODE-ROWS -v --start-position=352951786 --stop-position=352951786 mysql-bin.000311 | more
ERROR: Error in Log_event::read_log_event(): 'Sanity check failed', data_len: 65, event_type: 34
ERROR: Could not read entry at offset 352951786: Error in log format or read error.
/*!40019 SET @@session.max_insert_delayed_threads=0*/;
/*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/;
DELIMITER /*!*/;
DELIMITER ;
# End of log file
ROLLBACK /* added by mysqlbinlog */;
/*!50003 SET COMPLETION_TYPE=@OLD_COMPLETION_TYPE*/;
总结：
在MySQL主从同步集群部署中，经常会遇到主从不能同步的问题，以下对常见问题及解决办法进行了归纳列举
数据不一致：包括删除失败、主键重复、更新丢失
例如：
#更新丢失
Last_SQL_Errno: 1032
Last_SQL_Error: Could not execute Update_rows event on table social.test; Can't find record in 'jason', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND;the event's master log mysql-bin.000019, end_log_pos 4563
#主键重复
Last_SQL_Errno: 1062
Last_SQL_Error: Error 'Duplicate entry '1' for key 'PRIMARY'' on query. Default database: 'social'. Query: 'INSERT INTO `social`.`tool`(`createdat`, `updatedat`) VALUES ('2019-06-05 09:11:46', '2019-06-05 09:11:51')'
#删除失败
Last_SQL_Errno: 1032
Last_SQL_Error: Could not execute Delete_rows_v1 event on table social.player_role; Can't find record in 'player_role', Error_code: 1032; handler error HA_ERR_END_OF_FILE; the event's master log mysql-bin.000013, end_log_pos 20064466
在master上，用mysqlbinlog 分析下出错的binlog日志在干什么：
[root@localhost ~]# mysqlbinlog --no-defaults -v -v --base64-output=DECODE-ROWS mysql-bin.000013 | grep -A 10 20064466
#190605 9:12:28 server id 1 end_log_pos 20064466 Delete_rows: table id 442 flags: STMT_END_F
### DELETE FROM `social`.`player_role`
### WHERE
### @1=1 /* INT meta=0 nullable=0 is_null=0 */
### @2=1 /* INT meta=0 nullable=0 is_null=0 */
### @3='2019-06-04 20:10:48' /* DATETIME(0) meta=0 nullable=1 is_null=0 */
### @4='2019-06-04 20:10:48' /* DATETIME(0) meta=0 nullable=1 is_null=0 */
# at 20064466
#190605 9:12:28 server id 1 end_log_pos 20064493 Xid = 414815
COMMIT/*!*/;
# at 20064493
#190605 9:13:00 server id 1 end_log_pos 20064531 GTID 0-1-9152684 trans
/*!100001 SET @@session.gtid_seq_no=9152684*//*!*/;
BEGIN
/*!*/;
# at 20064531
#190605 9:13:00 server id 1 end_log_pos 20064795 Query thread_id=1733 exec_time=0 error_code=0
SET TIMESTAMP=1559697180.6860/*!*/;
对于主键重复／删除失败／更新丢失，更新丢失主要是缺少更新的记录，进行补充下即可，主键重复／删除失败此时可以选择忽略错误，方法如下：
mysql> stop slave;
mysql> set global sql_slave_skip_counter=1;
mysql> start slave;
mysql> show slave statusG
字段不一致：包括字段重复、丢失、不够长等
#字段重复
Last_SQL_Errno: 1060
Last_SQL_Error: Error 'Duplicate column name 'del_flag'' on query. Default database: 'social'. Query: 'ALTER TABLE `social`.`player` ADD COLUMN `del_flag` int(11) NULL AFTER `detail`'
对于数据库字段的同步失败，只需要将主从数据库的字段名称、类型等调整一致即可。之后重启slave，查看主从同步是否恢复
主从表不一致
Last_SQL_Errno: 1146
Last_SQL_Error: Error executing row event: 'Table 'social.player_role' doesn't exist'
主从提示缺少表可以进行添加，多余表可以删除，保证主从表的一致即可。之后重启slave，查看主从同步是否恢复