zoukankan      html  css  js  c++  java
  • 解决kettle在两个mysql之间迁移数据时乱码的问题 和 相关报错 及参数调整, 速度优化

    1. 乱码问题

    编辑目标数据库的链接:

    配置编码参数即可。

    2. 报错 No operations allowed after statement closed. 

    需要调整wait_timeout: set global wait_timeout=1000000;

    3. net_write_timeout 参数也需要调整:set global net_write_timeout='60000'

    kettle在迁移数据时,运行速度很慢,如果数量很大时,需要调整相关参数,不然运行到一半就报错。

    迁移完成之后,可以恢复相关参数。

    4. kettle 加速

     原理是把 单条的insert转换为 批量 batch insert

     To remedy this, in PDI I create a separate, specialized Database Connection I use for batch inserts. Set these two MySQL-specific options on your Database Connection:

    useServerPrepStmts false
    rewriteBatchedStatements true

    Used together, these "fake" batch inserts on the client. Specificially, the insert statements:

    INSERT INTO t (c1,c2) VALUES ('One',1);
    INSERT INTO t (c1,c2) VALUES ('Two',2);
    INSERT INTO t (c1,c2) VALUES ('Three',3);

    will be rewritten into:

    INSERT INTO t (c1,c2) VALUES ('One',1),('Two',2),('Three',3);

    So that the batched rows will be inserted with one statement (and one network round-trip). With this simple change, Table Output is very fast and close to performance of the bulk loader steps.

    置后写入速度有明显提升

    另外 目标 端的mysql可以调整一下参数:

    innodb_flush_log_at_trx_commit = 0

    sync_binlog = 0     

  • 相关阅读:
    洛谷P1613 跑路
    洛谷P2149 Elaxia的路线
    洛谷P3119 草鉴定
    洛谷P1972 HH的项链
    洛谷P2458 保安站岗
    uva10061
    uva579
    uva 127 "Accordian" Patience
    uva10177 (2/3/4)-D Sqr/Rects/Cubes/Boxes?
    uva156
  • 原文地址:https://www.cnblogs.com/digdeep/p/10967887.html
Copyright © 2011-2022 走看看