zoukankan      html  css  js  c++  java
  • 使用GoldenGate完成MySQL到MySQL的同步

    (一)基础环境配置

      源库 目标库
    操作系统版本 CentOS Linux release 7.4 CentOS Linux release 7.4
    IP地址 192.168.10.11 192.168.10.12
    数据库版本 社区版 5.7.24 社区版 5.7.24
    需要同步的数据库 testdb(所有表) testdb_repl
    GoldenGate版本 OGG-12.2.0.2 OGG-12.2.0.2

    说明:

    1.oracle GoldenGate下载地址:https://edelivery.oracle.com/ 免费注册一个账号即可登入,直接搜索“goldengate for non oracle database”,找到想要的版本下载即可。

    image_thumb1

    2.经过测试,使用OGG-11.2.0.1同步数据库会报错:

    ERROR   OGG-00146  VAM function VAMRead returned unexpected result: error 600 - VAM Client Report <CAUSE OF FAILURE : Sanity Check Failed for events
    WHEN FAILED : While reading log event from binary log
    WHERE FAILED : MySQLBinLog Reader Module
    CONTEXT OF FAILURE : No Information Available!>.

    更换为OGG-12.2.0.2后解决,故本文采用的Ogg版本是12.2.0.2。

    (二)MySQL数据库初始信息

      源库 目标库
    MySQL安装位置(basedir) /usr/local/mysql /usr/local/mysql
    数据存放位置    (datadir) /mysql/data /mysql/data
    二进制日志位置 (log_bin) /mysql/binlog /mysql/binlog

    (三)MySQL数据库开启二进制日志

    要使用ogg抽取数据,必须开启二进制日志。在mysql的配置文件/etc/my.cnf中配置二进制日志,重启数据库生效。

    [mysqld]
    
    # binlog配置
    server_id = 1
    binlog_format=ROW
    log_bin=/mysql/binlog/master-bin
    sync_binlog=1 
    expire_logs_days=1

    (四)源端和目标端安装配置GoldenGate(源端和目标端都要执行

    (4.1)源端和目标端创建安装ogg,实际上ogg并不需要安装,解压就可以使用

    创建ogg安装路径

    [root@masterdb ~]# mkdir /ogg

    将安装包拷贝到该路径下解压:

    [root@masterdb ogg]# ls
    ggs_Linux_x64_MySQL_64bit.tar
    [root@masterdb ogg]# tar -xvf ggs_Linux_x64_MySQL_64bit.tar 
    ./
    ./mysql_checklist.sql
    ...略
    ./libicuuc.so.56.1
    [root@masterdb ogg]# ls
    bcpfmt.tpl              emsclnt                        libggnnzitp.so        mysql.txt
    bcrypt.txt              extract                        libggparam.so         notices.txt
    cachefiledump           freeBSD.txt                    libggperf.so          oggerr
    checkprm                ggcmd                          libggrepo.so          prvtclkm.plb
    chkpt_mysql_create.sql  ggMessage.dat                  libicudata.so.56      replicat
    convchk                 ggparam.dat                    libicudata.so.56.1    retrace
    convprm                 ggsci                          libicui18n.so.56      reverse
    db2cntl.tpl             ggs_Linux_x64_MySQL_64bit.tar  libicui18n.so.56.1    server
    defgen                  help.txt                       libicuuc.so.56        sqlldr.tpl
    demo_mysql_create.sql   keygen                         libicuuc.so.56.1      tcperrs
    demo_mysql_insert.sql   lib                            libmysqlclient.so.18  ucharset.h
    demo_mysql_load.sql     lib12                          libxerces-c-3.1.so    UserExitExamples
    demo_mysql_misc.sql     libantlr3c.so                  logdump               usrdecs.h
    dirout                  libdb-6.1.so                   mgr                   zlib.txt
    dirwww                  libgglog.so                    mysql_checklist.sql

    (4.2)使用ggsci创建ogg相关路径

    [root@masterdb ogg]# pwd
    /ogg
    [root@masterdb ogg]# ./ggsci
    
    Oracle GoldenGate Command Interpreter for MySQL
    Version 12.2.0.2.2 OGGCORE_12.2.0.2.0_PLATFORMS_170630.0419
    Linux, x64, 64bit (optimized), MySQL Enterprise on Jun 30 2017 06:32:17
    Operating system character set identified as UTF-8.
    
    Copyright (C) 1995, 2017, Oracle and/or its affiliates. All rights reserved.
    
    
    
    GGSCI (masterdb) 1> create subdirs
    
    Creating subdirectories under current directory /ogg
    
    Parameter files                /ogg/dirprm: created
    Report files                   /ogg/dirrpt: created
    Checkpoint files               /ogg/dirchk: created
    Process status files           /ogg/dirpcs: created
    SQL script files               /ogg/dirsql: created
    Database definitions files     /ogg/dirdef: created
    Extract data files             /ogg/dirdat: created
    Temporary files                /ogg/dirtmp: created
    Credential store files         /ogg/dircrd: created
    Masterkey wallet files         /ogg/dirwlt: created
    Dump files                     /ogg/dirdmp: created

    (五)源端配置GoldenGate(源端执行

    (5.1)创建数据库用户ogg,该用户用来同步数据

    mysql> GRANT ALL PRIVILEGES ON `testdb`.* TO 'ogg'@'%'IDENTIFIED BY 'ogg';

    (5.2)配置管理进程mgr

    GGSCI (masterdb) 1> edit param mgr
    port 7809  
    dynamicportlist 7840-7939  
    purgeoldextracts /ogg/dirdat/*,usecheckpoints, minkeepdays 1 
    
    
    GGSCI (masterdb) 2> start mgr
    Manager started.
    
    
    GGSCI (masterdb) 3> info all
    
    Program     Status      Group       Lag at Chkpt  Time Since Chkpt
    
    MANAGER     RUNNING

    (5.3)配置捕获进程exta

    GGSCI (masterdb) 5> edit param exta
    extract exta  
    sourcedb testdb@192.168.10.11:3306 userid ogg password ogg  
    exttrail /ogg/dirdat/ea
    discardfile  /ogg/dirrpt/exta.dsc,append  
    TranLogOptions AltLogDest /mysql/binlog/master-bin.index 
    table testdb.*; 
    
    GGSCI (masterdb) 10> add extract exta,tranlog,begin now  
    EXTRACT added.
    
    GGSCI (masterdb) 11> add exttrail /ogg/dirdat/ea,extract exta  
    EXTTRAIL added.

    (5.4)配置投递进程dmpa

    GGSCI (masterdb) 15> edit param dmpa 
    extract dmpa 
    passthru  
    sourcedb testdb@192.168.10.11:3306 userid ogg password ogg  
    rmthost  192.168.10.12,mgrport 7809,compress  
    rmttrail  /ogg/dirdat/da  
    dynamicresolution  
    numfiles 3000  
    table testdb.*; 
    
    GGSCI (masterdb) 16> add extract dmpa ,exttrailsource /ogg/dirdat/ea  
    EXTRACT added.
    
    GGSCI (masterdb) 17> add rmttrail /ogg/dirdat/da,extract dmpa 
    RMTTRAIL added.

    (5.5)创建表定义文件,传送到目标端

    (5.5.1)创建表定义文件

    GGSCI (masterdb) 23> edit param defgen  
    defsfile ./dirdef/testdb.def  
    sourcedb testdb@192.168.10.11:3306 userid ogg, password ogg  
    table testdb.*;  
    
    GGSCI (masterdb) 26> exit

    (5.5.2)生成表定义

    [root@masterdb ogg]# ./defgen paramfile ./dirprm/defgen.prm  
    
    ***********************************************************************
            Oracle GoldenGate Table Definition Generator for MySQL
          Version 12.2.0.2.2 OGGCORE_12.2.0.2.0_PLATFORMS_170630.0419
     Linux, x64, 64bit (optimized), MySQL Enterprise on Jun 30 2017 07:27:58
     
    Copyright (C) 1995, 2017, Oracle and/or its affiliates. All rights reserved.
    
    
                        Starting at 2020-02-08 18:15:40
    ***********************************************************************
    
    Operating System Version:
    Linux
    Version #1 SMP Tue Aug 22 21:09:27 UTC 2017, Release 3.10.0-693.el7.x86_64
    Node: masterdb
    Machine: x86_64
                             soft limit   hard limit
    Address Space Size   :    unlimited    unlimited
    Heap Size            :    unlimited    unlimited
    File Size            :    unlimited    unlimited
    CPU Time             :    unlimited    unlimited
    
    Process id: 2510
    
    ***********************************************************************
    **            Running with the following parameters                  **
    ***********************************************************************
    defsfile ./dirdef/testdb.def
    sourcedb testdb@192.168.10.11:3306 userid ogg, password ***
    table testdb.*;
    Expanding wildcard table specification testdb.*:
    
    Retrieving definition for testdb.test01.
    Retrieving definition for testdb.test02.
    Retrieving definition for testdb.test03.
    
    
    Definitions generated for 3 tables in ./dirdef/testdb.def.

    (5.5.3)传送到目标端

    [root@masterdb ogg]# scp /ogg/dirdef/testdb.def 192.168.10.12:/ogg/dirdef/
    root@192.168.10.12's password: 
    testdb.def                            100% 1785     1.9MB/s   00:00

    (六)目标端配置GoldenGate(目标端执行

    (6.1)创建数据库用户ogg,该用户用来同步数据

    mysql> GRANT ALL PRIVILEGES ON `testdb_repl`.* TO 'ogg'@'%'IDENTIFIED BY 'ogg';

    (6.2)配置管理进程mgr

    GGSCI (slavedb) 3> edit param mgr
    port 7809  
    dynamicportlist 7840-7939  
    purgeoldextracts /ogg/dirdat/*,usecheckpoints, minkeepdays 1 
    
    GGSCI (slavedb) 5> start mgr
    Manager started.
    
    GGSCI (slavedb) 6> info all
    
    Program     Status      Group       Lag at Chkpt  Time Since Chkpt
    
    MANAGER     RUNNING

    (6.3)配置检查表

    GGSCI (slavedb) 9> dblogin sourcedb testdb_repl@192.168.10.12:3306 userid ogg password ogg 
    Successfully logged into database.
    
    GGSCI (slavedb DBLOGIN as ogg) 10> add checkpointtable testdb_repl.checkpoint  
    
    Successfully created checkpoint table testdb_repl.checkpoint.
    
    GGSCI (slavedb DBLOGIN as ogg) 11> edit params ./GLOBALS 
    checkpointtable testdb_repl.checkpoint

    (6.4)配置应用进程repa

    GGSCI (slavedb DBLOGIN as ogg) 13> edit params repa
    
    replicat repa 
    targetdb testdb_repl@192.168.10.12:3306 userid ogg password ogg  
    handlecollisions  
    sourcedefs /ogg/dirdef/testdb.def  
    discardfile /ogg/dirrpt/repa.dsc,purge  
    map testdb.*, target testdb_repl.*;
     

    GGSCI (slavedb DBLOGIN as ogg) 16> add replicat repa,exttrail /ogg/dirdat/da,checkpointtable testdb_repl.checkpoint
    REPLICAT added.

    (七)导入初始化数据

    将源库需要同步的表导出,然后导入到目标数据库

    # 源库导出,并传到目标数据库服务器上
    [root@masterdb ~]# mysqldump -uroot -p123456 -h192.168.10.11  --single-transaction --flush-logs --master-data=2 testdb  > testdb.sql
    mysqldump: [Warning] Using a password on the command line interface can be insecure.
    
    [root@masterdb ~]# ls
    anaconda-ks.cfg  testdb.sql  wgetlog-02-05-20-22:27.log  wgetlog-02-05-20-22:28.log  wget.sh
    
    [root@masterdb ~]# scp testdb.sql root@192.168.10.12:/root
    root@192.168.10.12's password: 
    testdb.sql                                                                                100%   15MB  68.9MB/s   00:00
    # 目标数据库执行导入
    [root@slavedb ~]# mysql -uroot -p123456 testdb_repl < testdb.sql 
    mysql: [Warning] Using a password on the command line interface can be insecure.

    (八)开启ogg同步进程

    (8.1)源端开启捕获进程(源端执行

    由于数据库数据一直在变化,所以对于导出的testdb.sql文件,我们需要记录二进制日志文件的log_file和log_pos

    [root@slavedb ~]# cat testdb.sql |grep "CHANGE MASTER TO MASTER_LOG_FILE"
    -- CHANGE MASTER TO MASTER_LOG_FILE='master-bin.000005', MASTER_LOG_POS=154;

    然后从导出时的binlog开始抽取

    GGSCI (masterdb) 5> alter extract exta,vam,lognum 5,logpos 154
    EXTRACT altered.
    
    
    GGSCI (masterdb) 6> start exta
    
    Sending START request to MANAGER ...
    EXTRACT EXTA starting

    (8.2)开启投递进程(源端执行

    GGSCI (masterdb) 19> start dmpa
    
    Sending START request to MANAGER ...
    EXTRACT DMPA starting

    (8.3)开启应用进程(目标端执行

    GGSCI (slavedb DBLOGIN as ogg) 18> start repa
    
    Sending START request to MANAGER ...
    REPLICAT REPA starting

    (九)确认同步状态

    (9.1)使用info all查看进程是否运行正常

    所有进程状态为“running”代表正常

    源端:

    GGSCI (masterdb) 8> info all
    
    Program     Status      Group       Lag at Chkpt  Time Since Chkpt
    
    MANAGER     RUNNING                                           
    EXTRACT     RUNNING     DMPA        00:00:00      00:00:08    
    EXTRACT     RUNNING     EXTA        00:00:00      00:00:04

    目标端:

    GGSCI (slavedb DBLOGIN as ogg) 23> info all
    
    Program     Status      Group       Lag at Chkpt  Time Since Chkpt
    
    MANAGER     RUNNING                                           
    REPLICAT    RUNNING     REPA        01:14:40      00:00:00

    (9.2)源端手动插入数据,查看目标端是否同步

    源端执行:

    insert into test03 values(1,'a');

    目标端查看数据是否同步过来:

    mysql> use testdb_repl ;
    Database changed
    
    mysql> select * from test03;
    +-----+------+
    | id3 | name |
    +-----+------+
    |   1 | a    |
    +-----+------+
    1 row in set (0.04 sec)

    【完】

  • 相关阅读:
    求正方形的面积和周长
    面向对象练习——校园管理系统
    python--面向对象(最全讲解)
    git删除远程仓库的文件或目录
    git设置忽略文件和目录
    计算器
    织梦标签整理
    比较恶心的360老年版浏览器 文件导出下载
    织梦文章列表样式多元化实现
    php函数整理
  • 原文地址:https://www.cnblogs.com/lijiaman/p/12284544.html
Copyright © 2011-2022 走看看