zoukankan      html  css  js  c++  java
  • pt-table-checksum检测不出主从差异处理

    几个月前写过pt-table-checksum 3.0.4检测不出主从差异数据,当时的解决方案是使用旧版本,另一个挫方法是自行设置binlog_format='STATEMENT'。现在已经发布到3.0.9版本,结果又遇到相同的坑~
    最近几版pt-table-checksum(已核实3.0.4和3.0.9有问题)在binlog_format='row',且主从存在差异数据时,却检测不出主从差异。原因就是处理过程中主上没有SET @@binlog_format := 'STATEMENT',导致下面两个核心语句不是以statement格式记录,从库不会进行CRC32相关运算,主从永远一致~

    # pt-table-checksum 3.0.4检测不出差异数据
    [root@ZST2 ~]# /usr/local/bin/pt-table-checksum --nocheck-binlog-format --nocheck-replication-filters --recursion-method=hosts --replicate=replcrash.checksums --databases=replcrash --tables=py_user,py_user_innodb --host=192.168.85.132 --port=3306 --user=mydba --password=mysql5721
                TS ERRORS  DIFFS     ROWS  CHUNKS SKIPPED    TIME TABLE
    05-08T09:41:15      0      0        8       1       0   0.257 replcrash.py_user
    05-08T09:41:16      0      0    67740       5       0   1.056 replcrash.py_user_innodb
    
    # 两个核心语句
    REPLACE INTO `replcrash`.`checksums` (db, tbl, chunk, chunk_index, lower_boundary, upper_boundary, this_cnt, this_crc) SELECT 'replcrash', 'py_user', '1', NULL, NULL, NULL, COUNT(*) AS cnt, COALESCE(LOWER(CONV(BIT_XOR(CAST(CRC32(CONCAT_WS('#', `uid`, convert(`name` using utf8mb4), `add_time`, convert(`server_id` using utf8mb4), CONCAT(ISNULL(`name`), ISNULL(`add_time`), ISNULL(`server_id`)))) AS UNSIGNED)), 10, 16)), 0) AS crc FROM `replcrash`.`py_user` /*checksum table*/
    UPDATE `replcrash`.`checksums` SET chunk_time = '0.004092', master_crc = '5abbd632', master_cnt = '8' WHERE db = 'replcrash' AND tbl = 'py_user' AND chunk = '1'
    View Code

    通过对比旧版本(2.2.3)的/usr/local/bin/pt-table-checksum代码,发现只需稍微调整代码即可正常使用

    # pt-table-checksum 3.0.4 只需注释掉第9335行和9364行,这层逻辑应该应用于任何情况下,因此不需要使用if判断
    [root@ZST2 ~]# vim /usr/local/bin/pt-table-checksum
    ...
    9335       #if ( $o->get('check-binlog-format') ) {
    9336         # https://bugs.launchpad.net/percona-toolkit/+bug/919352
    9337         # The tool shouldn't blindly attempt to change binlog_format;
    9338         # instead, it should check if it's already set to STATEMENT.
    9339         # This is becase starting with MySQL 5.1.29, changing the format
    9340         # requires a SUPER user.
    9341         if ( VersionParser->new($dbh) >= '5.1.5' ) {
    9342            $sql = 'SELECT @@binlog_format';
    9343            PTDEBUG && _d($dbh, $sql);
    9344            my ($original_binlog_format) = $dbh->selectrow_array($sql);
    9345            PTDEBUG && _d('Original binlog_format:', $original_binlog_format);
    9346            if ( $original_binlog_format !~ /STATEMENT/i ) {
    9347               $sql = q{/*!50108 SET @@binlog_format := 'STATEMENT'*/};
    9348               eval {
    9349                  PTDEBUG && _d($dbh, $sql);
    9350                  $dbh->do($sql);
    9351               };
    9352               if ( $EVAL_ERROR ) {
    9353                  die "Failed to $sql: $EVAL_ERROR
    "
    9354                     . "This tool requires binlog_format=STATEMENT, "
    9355                     . "but the current binlog_format is set to "
    9356                     ."$original_binlog_format and an error occurred while "
    9357                     . "attempting to change it.  If running MySQL 5.1.29 or newer, "
    9358                     . "setting binlog_format requires the SUPER privilege.  "
    9359                     . "You will need to manually set binlog_format to 'STATEMENT' "
    9360                     . "before running this tool.
    ";
    9361               }
    9362            }
    9363         }
    9364       #}
    ...
    
    # 查看pt版本
    [root@ZST2 ~]# /usr/local/bin/pt-table-checksum --version
    pt-table-checksum 3.0.4
    [root@ZST2 ~]# 
    # 修改后检测出主从不一致
    [root@ZST2 ~]# /usr/local/bin/pt-table-checksum --nocheck-binlog-format --nocheck-replication-filters --recursion-method=hosts --replicate=replcrash.checksums --databases=replcrash --tables=py_user,py_user_innodb --host=192.168.85.132 --port=3306 --user=mydba --password=mysql5721
                TS ERRORS  DIFFS     ROWS  CHUNKS SKIPPED    TIME TABLE
    05-08T09:51:32      0      1        8       1       0   0.030 replcrash.py_user
    05-08T09:51:33      0      0    67740       5       0   0.561 replcrash.py_user_innodb
    [root@ZST2 ~]# 
    
    # general-log信息
    [root@ZST1 ~]# cat /data/mysql/mysql3306/data/ZST1.log |more
    ...
    2018-05-08T01:51:32.315064Z        30 Query     SHOW VARIABLES LIKE 'version%'
    2018-05-08T01:51:32.318337Z        30 Query     SHOW ENGINES
    2018-05-08T01:51:32.319229Z        30 Query     SHOW VARIABLES LIKE 'innodb_version'
    2018-05-08T01:51:32.322518Z        30 Query     SELECT @@binlog_format
    2018-05-08T01:51:32.323019Z        30 Query     /*!50108 SET @@binlog_format := 'STATEMENT'*/
    2018-05-08T01:51:32.323453Z        30 Query     SET SESSION TRANSACTION ISOLATION LEVEL REPEATABLE READ
    2018-05-08T01:51:32.324012Z        30 Query     SHOW VARIABLES LIKE 'wsrep_on'
    2018-05-08T01:51:32.327830Z        30 Query     SELECT @@SERVER_ID
    2018-05-08T01:51:32.328416Z        30 Query     SHOW SLAVE HOSTS
    2018-05-08T01:51:32.344178Z        30 Query     SHOW VARIABLES LIKE 'wsrep_on'
    2018-05-08T01:51:32.347571Z        30 Query     SELECT @@SERVER_ID
    2018-05-08T01:51:32.351692Z        30 Query     SHOW VARIABLES LIKE 'wsrep_on'
    2018-05-08T01:51:32.355034Z        30 Query     SELECT @@SERVER_ID
    2018-05-08T01:51:32.359543Z        30 Query     SHOW DATABASES LIKE 'replcrash'
    2018-05-08T01:51:32.360406Z        30 Query     CREATE DATABASE IF NOT EXISTS `replcrash` /* pt-table-checksum */
    2018-05-08T01:51:32.361538Z        30 Query     USE `replcrash`
    2018-05-08T01:51:32.362069Z        30 Query     SHOW TABLES FROM `replcrash` LIKE 'checksums'
    2018-05-08T01:51:32.364738Z        30 Query     CREATE TABLE IF NOT EXISTS `replcrash`.`checksums` (
         db             CHAR(64)     NOT NULL,
         tbl            CHAR(64)     NOT NULL,
         chunk          INT          NOT NULL,
         chunk_time     FLOAT            NULL,
         chunk_index    VARCHAR(200)     NULL,
         lower_boundary TEXT             NULL,
         upper_boundary TEXT             NULL,
         this_crc       CHAR(40)     NOT NULL,
         this_cnt       INT          NOT NULL,
         master_crc     CHAR(40)         NULL,
         master_cnt     INT              NULL,
         ts             TIMESTAMP    NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
         PRIMARY KEY (db, tbl, chunk),
         INDEX ts_db_tbl (ts, db, tbl)
      ) ENGINE=InnoDB DEFAULT CHARSET=utf8
    2018-05-08T01:51:32.920625Z        30 Query     SHOW GLOBAL STATUS LIKE 'Threads_running'
    2018-05-08T01:51:32.924636Z        30 Query     SELECT CONCAT(@@hostname, @@port)
    ...
    View Code



    如果遇到类似问题,建议开启general_log,查看处理过程,再核验到底什么原因导致检测不出主从差异数据(・ω・)


    09:21 2018/5/10 补充,其实可以在命令行带上--set-vars binlog_format='statement'
    详细说明参考:pt-table-checksum not detecting diffs

    [root@ZST2 ~]# /usr/local/bin/pt-table-checksum --nocheck-binlog-format --nocheck-replication-filters --recursion-method=hosts --replicate=replcrash.checksums --databases=replcrash --tables=py_user,py_user_innodb --host=192.168.85.132 --port=3306 --user=mydba --password=mysql5721 --set-vars binlog_format='statement'
                TS ERRORS  DIFFS     ROWS  CHUNKS SKIPPED    TIME TABLE
    05-10T09:25:01      0      1       11       1       0   0.024 replcrash.py_user
    05-10T09:25:01      0      0    67740       5       0   0.695 replcrash.py_user_innodb
    [root@ZST2 ~]# 
    View Code

    善于思考,善于搜索,不要期盼别人把结果原封不动地送到嘴里~.~

  • 相关阅读:
    [改善Java代码]养成良好习惯,显式声明UID
    [改善Java代码]警惕自增的陷阱
    [改善Java代码]覆写变长方法也循规蹈矩
    [改善Java代码]别让null值和空值威胁到变长方法
    [改善Java代码]避免带有变长参数的方法重载
    [改善Java代码]三元操作符的类型务必一致
    关于Windows下mysql忘记root密码的解决方法
    关于同步VSS服务器上的代码发生Eclipse里面的项目全部不见了
    关于关闭Eclipse的控制台自动跳出
    关于Windows下如何查看端口占用和杀掉进程
  • 原文地址:https://www.cnblogs.com/Uest/p/9007278.html
Copyright © 2011-2022 走看看