binlog2sql

binlog2sql使用总结：
binlog2sql是大众点评开源的一款用于解析binlog的工具

 

功能：
1.提取SQL
2.生成回滚SQL

 

关于该工具的使用方法可参考github操作文档：https://github.com/danfengcao/binlog2sql

 

使用该工具的前提：
1.binlog_format为ROW，且binlog_row_image为full或noblog，默认为full
2.必须开启MySQL Server，理由有如下两点：
1)、它是基于binlog_dump协议来获取binlog内容
2)、需要读取server端information_schema.COLUMNS表，获取表结构的元信息，拼接成可视化的sql语句
该工具所需权限如下：
grant select,replication slave,replication client on *.* to
因为是伪装成slave来获取主的二进制事件，故无需对binlog有可读权限

安装binlog2sql前先安装git和pip
安装python3
# tar zxvf Python-3.5.1.tgz
# cd Python-3.5.1
# ./configure
# make && make install
测试
python3 --version
安装epel-release-6-8.noarch.rpm
# yum -y install epel-release-6-8.noarch.rpm
# yum -y install git python-pip
# git clone https://github.com/danfengcao/binlog2sql.git
# cd binlog2sql
# pip3 install -r requirements.txt

 

如果上不了网，需要下载
git clone https://github.com/PyMySQL/PyMySQL
git clone https://github.com/noplay/python-mysql-replication

 

MySQL的配置要开启以下选项：
[mysqld]
server_id = 1
log_bin = /var/log/mysql/mysql-bin.log
max_binlog_size = 1G
binlog_format = row
binlog_row_image = ful

 

提取SQL示例：
# python3 binlog2sql.py -h192.168.1.121 -P3306 -uroot -pmsds007 -dtest --start-file='mysql-bin.000007'
# python binlog2sql.py -h 10.215.48.133 -urecover  -precover123 -P3306 -d bukexuetang -t order_consignee --start-file='mysql-bin.000012' --start-position=484053839 --stop-position=484831465  > recover_20190622.sql
INSERT INTO `test`.`gtidtest`(`id`, `name`) VALUES (1, 'allen'); #start 296 end 468 time 2017-01-31 21:50:02
INSERT INTO `test`.`gtidtest`(`id`, `name`) VALUES (2, 'bob'); #start 499 end 669 time 2017-01-31 21:50:02
INSERT INTO `test`.`gtidtest`(`id`, `name`) VALUES (3, 'click'); #start 700 end 872 time 2017-01-31 21:50:02
INSERT INTO `test`.`gtidtest`(`id`, `name`) VALUES (4, 'david'); #start 903 end 1075 time 2017-01-31 21:50:03
DELETE FROM `test`.`gtidtest` WHERE `id`=3 AND `name`='click' LIMIT 1; #start 1106 end 1278 time 2017-01-31 21:50:11
DELETE FROM `test`.`gtidtest` WHERE `id`=4 AND `name`='david' LIMIT 1; #start 1309 end 1481 time 2017-01-31 21:50:11
INSERT INTO `test`.`gtidtest`(`id`, `name`) VALUES (5, 'eric'); #start 1512 end 1683 time 2017-01-31 21:50:11
INSERT INTO `test`.`gtidtest`(`id`, `name`) VALUES (6, 'fred'); #start 1714 end 1885 time 2017-01-31 21:50:12

 

生成回滚SQL示例：
# python3 binlog2sql.py --flashback -h192.168.1.121 -P3306 -uroot -pmsds007 -dtest --start-file='mysql-bin.000007'
# python binlog2sql.py --flashback -h 10.215.48.133 -urecover  -precover123 -P3306 -d bukexuetang -t order_consignee --start-file='mysql-bin.000012' --start-position=484053839 --stop-position=484831465  > recover_20190622.sql
DELETE FROM `test`.`gtidtest` WHERE `name`='fred' AND `id`=6 LIMIT 1; #start 1714 end 1885 time 2017-01-31 21:50:12
DELETE FROM `test`.`gtidtest` WHERE `name`='eric' AND `id`=5 LIMIT 1; #start 1512 end 1683 time 2017-01-31 21:50:11
INSERT INTO `test`.`gtidtest`(`name`, `id`) VALUES ('david', 4); #start 1309 end 1481 time 2017-01-31 21:50:11
INSERT INTO `test`.`gtidtest`(`name`, `id`) VALUES ('click', 3); #start 1106 end 1278 time 2017-01-31 21:50:11
DELETE FROM `test`.`gtidtest` WHERE `name`='david' AND `id`=4 LIMIT 1; #start 903 end 1075 time 2017-01-31 21:50:03
DELETE FROM `test`.`gtidtest` WHERE `name`='click' AND `id`=3 LIMIT 1; #start 700 end 872 time 2017-01-31 21:50:02
DELETE FROM `test`.`gtidtest` WHERE `name`='bob' AND `id`=2 LIMIT 1; #start 499 end 669 time 2017-01-31 21:50:02
DELETE FROM `test`.`gtidtest` WHERE `name`='allen' AND `id`=1 LIMIT 1; #start 296 end 468 time 2017-01-31 21:50:02

 

总结：
1. 看了下源代码，它本身的核心代码比较少，主要是在pymysqlreplication的基础上进行了二次开发。
   pymysqlreplication实现了MySQL复制协议，可捕捉不同类型的EVENT事件。
   具体可参考：https://github.com/noplay/python-mysql-replication
2. 个人感觉，直接解析文本格式的binlog，也未尝不是一个好办法。
   理由如下：
   1> binlog2sql强烈依赖于MySQL复制协议，如果复制协议发生改变，则该工具将不可用。
       虽然，复制协议发生改变的可能性很小（一般都会保持向前兼容），但相对而言，自带的mysqlbinlog肯定更懂binlog，基于mysqlbinlog解析后的结果进行处理，
       可完全屏蔽复制协议等底层细节。
   2> 用python来解析文本格式的binlog，本身也不是件难事。
       譬如，update语句在binlog中的对应的文本

 

 

内网环境使用该工具：
该工具的使用依赖以下三个包：
PyMySQL==0.7.8
wheel==0.24.0
mysql-replication==0.9
其中，每个包又会依赖其它包，所以安装这些包是一个比较麻烦的事情。
在内网环境下，可手动安装这些包，目前，这些包已下载打包，并上传到百度云盘中，大家可自行下载。
http://pan.baidu.com/s/1qYQ2PPy

 

安装教程：
# tar xvf binlog2sql.tar.gz
# cd binlog2sql/binlog2sql_dependencies/
# tar xvf setuptools-0.6c11.tar.gz
# cd setuptools-0.6c11
# python setup.py install
# cd ..
# tar xvf pip-9.0.1.tar.gz
# cd pip-9.0.1
# python setup.py install
# cd ..
# pip install *.whl mysql-replication-0.9.tar.gz
DEPRECATION: Python 2.6 is no longer supported by the Python core team, please upgrade your Python. A future version of pip will drop
 support for Python 2.6Processing ./mysql-replication-0.9.tar.gz
Processing ./argparse-1.4.0-py2.py3-none-any.whl
Processing ./linecache2-1.0.0-py2.py3-none-any.whl
Requirement already satisfied: PyMySQL==0.7.8 from file:///root/binlog2sql/binlog2sql_dependencies/PyMySQL-0.7.8-py2-none-any.whl in
/usr/lib/python2.6/site-packagesProcessing ./six-1.10.0-py2.py3-none-any.whl
Processing ./traceback2-1.4.0-py2.py3-none-any.whl
Processing ./unittest2-1.1.0-py2.py3-none-any.whl
Processing ./wheel-0.24.0-py2.py3-none-any.whl
Installing collected packages: argparse, linecache2, six, traceback2, unittest2, wheel, mysql-replication
  Running setup.py install for mysql-replication ... done
Successfully installed argparse-1.4.0 linecache2-1.0.0 mysql-replication-0.9 six-1.10.0 traceback2-1.4.0 unittest2-1.1.0 wheel-0.24.0
至此，所有依赖包安装完毕。

 

测试一下
# cd ..
# cd binlog2sql
# python binlog2sql.py -h 192.168.244.20 -u root -p 123456  -d test -t test --start-file mysql-bin.000022
INSERT INTO `test`.`test`(`id`, `name`) VALUES (4, 'd'); #start 4 end 284 time 2017-02-21 10:44:02
INSERT INTO `test`.`test`(`id`, `name`) VALUES (5, 'f'); #start 315 end 479 time 2017-02-21 10:44:05
DELETE FROM `test`.`test` WHERE `id`=4 AND `name`='d' LIMIT 1; #start 510 end 674 time 2017-02-21 10:44:09
UPDATE `test`.`test` SET `id`=5, `name`='t' WHERE `id`=5 AND `name`='f' LIMIT 1; #start 705 end 877 time 2017-02-21 10:44:22

 

# python binlog2sql.py -h 10.10.97.253 -u root -p GOmKCNwIP0HH  -d grampus_db -t gms_company --start-file mysql-bin.000024 --stop-file mysql-bin.000025

# python binlog2sql.py -h 10.10.97.253 -u root -p GOmKCNwIP0HH  -d grampus_db -t gms_replenishment_blacklist --start-file mysql-bin.000033 --stop-file mysql-bin.000036 --start-datetime="2017-03-12 10:00:00" --stop-datetime="2017-03-13 12:00:00"> gms_replenishment_blacklist.sql

# python binlog2sql.py -h10.10.97.253 -P3306 -uroot -p'GOmKCNwIP0HH' -dgrampus_db -tgms_company --start-file='mysql-bin.000023' --start-datetime="2017-03-06 11:00:00" --stop-datetime="2017-03-06 17:30:00" > /data/gms_company_1.sql

# python binlog2sql.py -h10.215.50.101 -P3306 -ubluewhale -p'bluewhale001' -dsusuan --start-file='mysql-bin.003967' --start-datetime="2018-05-09 07:25:00" --stop-datetime="2018-05-09 07:30:00" > /data/gms_company_1.sql

python binlog2sql.py --flashback -h10.215.50.114 -P3315 -ubluewhale -p'bluewhale001' -docr --start-file='mysql-bin.000100'   
--stop-file='mysql-bin.000100' --start-datetime="2018-08-13 20:56:00" --stop-datetime="2018-08-13 21:00:00" > recover.sql    &