zoukankan      html  css  js  c++  java
  • Import SQL into MySQL with a progress meter



    There is nice tool called pv

    # On Ubuntu/Debian system
    $ sudo apt-get install pv
    
    # On Redhat/CentOS
    $ sudo yum install pv

    then e.g. you can use it like this

      pv sqlfile.sql | mysql -u xxx -p xxxx dbname

    $ zcat dbpackfile.sql.gz | pv -cN zcat | mysql -uuser -ppass dbname

    Please check UPDATE 2 for my latest version

    ps: check this blog http://blog.larsstrand.org/2011/12/tip-pipe-viewer.html

    UPDATE: seems like above link is broken but I found same article here http://blog.larsstrand.no/2011/12/tip-pipe-viewer.html

    UPDATE 2: Even better solution with FULL progress bar. To do it you need to use 2 build in pv options. One is --progress to indicate progress bar and second is --size to tell pv how large the overall file is.

    pv --progress --size UNPACKED-FILE-SIZE-IN-BYTES

    ..the problem is with .gz original file size. You need somehow get unpacked original file size information without unpacking it self, otherwise you will lost our precious time to unpack this file twice (first time pv and second time zcat). But fortunately you have gzip -l option that contain uncompressed information about our gziped file. Unfortunattly you have it in table format so you need to extract before it can be use it. All together can be seen below:

    gzip -l /path/to/our/database.sql.gz | sed -n 2p | awk '{print $2}'

    Uff.. so the last thing you need to do is just combine all together.

    zcat /path/to/our/database.sql.gz | pv --progress --size `gzip -l %s | sed -n 2p | awk '{print $2}'` | mysql -uuser -ppass dbname

    To make it even nicer you can add progres NAME like this

    zcat /path/to/our/database.sql.gz | pv --progress --size `gzip -l %s | sed -n 2p | awk '{print $2}'` --name '  Importing.. ' | mysql -uuser -ppass dbname

    Final result:

    Importing.. : [===========================================>] 100%


    =================================================================
    #sudo yum install pv
    Loaded plugins: fastestmirror, refresh-packagekit, security
    Loading mirror speeds from cached hostfile
     * base: mirror.neu.edu.cn
     * extras: mirror.neu.edu.cn
     * updates: mirror.neu.edu.cn
    Setting up Install Process
    No package pv available.
    Error: Nothing to do
    ------------------------------------------------------------------
    http://wiki.centos.org/AdditionalResources/Repositories/RPMForge#head-f0c3ecee3dbb407e4eed79a56ec0ae92d1398e01
    wget http://pkgs.repoforge.org/rpmforge-release/rpmforge-release-0.5.3-1.el6.rf.x86_64.rpm
    rpm --import http://apt.sw.be/RPM-GPG-KEY.dag.txt
    rpm -K rpmforge-release-0.5.3-1.el6.rf.x86_64.rpm
    pm -i rpmforge-release-0.5.3-1.el6.rf.x86_64.rpm
    yum install pv
    =================================================================


    ref:
    http://www.ivarch.com/programs/pv.shtml

    http://unix.stackexchange.com/questions/36769/get-a-progress-indicator-when-importing-mysql-databases
    https://major.io/2010/11/24/monitor-mysql-restore-progress-with-pv/
    http://dba.stackexchange.com/questions/17367/how-can-i-monitor-the-progress-of-an-import-of-a-large-sql-file
    http://www.commandlinefu.com/commands/view/5884/import-sql-into-mysql-with-a-progress-meter

  • 相关阅读:
    数据仓库 数据可视化 Hive导出到MySql
    数据仓库 品牌复购率
    数据仓库 转化率及漏斗分析
    数据仓库 GMV成交总额
    数据仓库 DWS层之用户行为宽表
    数据仓库 业务数仓 DWD层
    数据仓库 业务数仓 ODS层
    数据仓库 表的分类与同步策略
    数据仓库 最近七天内连续三天活跃用户数
    CF505E Mr. Kitayuta vs. Bamboos 二分+贪心
  • 原文地址:https://www.cnblogs.com/emanlee/p/4592956.html
Copyright © 2011-2022 走看看