数据库备份及还原

数据库关系到业务的核心，备份数据库进行容灾是必不可少的操作

mysql

使用mysqldump命令备份

mysqldump基本语法：

mysqldump -u username -p dbname table1 table2 ...-> BackupName.sql

其中：

dbname参数表示数据库的名称;
table1和table2参数表示需要备份的表的名称，为空则整个数据库备份;
BackupName.sql参数表设计备份文件的名称,文件名前面可以加上一个绝对路径. 通常将数据库被分成一个后缀名为sql的文件;

直接复制整个数据库目录

简单粗暴, 操作时最好保持数据库服务是停止的，以免影响数据，导致最终无法还原

另外这种方法不适用于InnoDB存储引擎的表，而对于MyISAM存储引擎的表很方便。同时，还原时MySQL的版本最好相同。

和这种方法类似，利用LVM或者btrfs这些文件系统层面的快照可以将硬盘当前状态保存，速度更快，但是需要文件系统支持

mysql的备份脚本

#!/bin/bash

. /etc/profile
. ~/.bashrc

. common.sh

# general variable define
bk_date=`date +'%Y%m%d'`
bk_dir=/data/backup/mysql/
bk_path=/data/backup/mysql/$bk_date
bk_log=/data/backup/log/$bk_date.log
de_date=`date -d"-7 day" +"%Y%m%d"`
dbs=(agreement cqa_test healthy hive hive_dfs hive_gas hive_kcloud hive_new hive_rec kcloud_chat kcloud_data kcloud_operation kcloud_operation_test kcloud_system kcloud_tool lingxi_yunwei mysql oppo public test xiaomi yunwei)

# alias define
shopt -s expand_aliases
alias now="date +'%Y-%m-%d %H:%M:%S'"
alias nowmin="date +'%H%M'"

# logging level
l1=INFO
l2=WARN
l3=CRIT
l4=UNKNOWN

logging(){
        echo "[`now`] $1 $2" >> $bk_log
}

check_bkpath(){
        if [ ! -d $bk_path ]
        then
                logging "$l1" "Backup path $bk_path not exist, make it"
                mkdir -p $bk_path
                [ $? -eq 0 ] && logging "$l1" "Backup path $bk_path make ok" || logging "l3" "Backup path $bk_path make exception"
        fi
}

backup_dbs(){
        cd $bk_path
        for d in ${dbs[@]}
        do
                db=$d.`nowmin`
                logging "$l1" "DB $db backup starting"
                res=`mysqldump -uroot -piflytek! $d > $db.sql 2>&1`
                if [ $? -eq 0 ]; then logging "$l1" "DB $db backup finish"; else logging "$l3" "DB $db backup exception: $res"; return 1; fi
                logging "$l1" "DB $db compress starting"
                res=`tar czf $db.sql.tgz $db.sql 2>&1`
                if [ $? -eq 0 ]; then logging "$l1" "DB $db compress finish"; else logging "$l2" "DB $db compress exception: $res"; return 1; fi
                res=`rm $db.sql 2>&1`
                if [ $? -eq 0 ]; then logging "$l1" "DB $db rm finish"; else logging "$l2" "DB $db rm exception: $res"; return 1; fi
        done
}

rm_history(){
        cd $bk_dir
        ls -l .|awk '{print $NF}'|sort -n|uniq|while read line
        do
                if [[ ${#line} -eq 8 ]] && [[ $line =~ ^[0-9]*.?[0-9]*$ ]]
                then
                        r=`echo $line|awk '{if($1<="'"$de_date"'")print}'`
                        if [ ! -z $r ]
                        then
                                logging "$l1" "$line is expire, will be removed"
                                res=`rm -rf $line 2>&1`
                                [ $? -eq 0 ] && logging "$l1" "$line removed finish" || logging "$l2" "$line removed exception: $res"
                        fi
                fi
        done
}

check_bkpath
mdb "BACKING"
[ $? -eq 0 ] && logging "$l1" "Write BACKING to db" || logging "$l2" "Write BACKING to db Failed"
backup_dbs
[ $? -ne 0 ] && mdb "BACKERR" || mdb "BACKOK"
[ $? -eq 0 ] && logging "$l1" "Write BACKOK/BACKERR to db" || logging "$l2" "Write BACKOK/BACKERR to db Failed"
rm_history

mongodb

备份前的检查

> show dbs
MyDB 0.0625GB
admin (empty)
bruce 0.0625GB
local (empty)
test 0.0625GB
> use MyDB
switched to db MyDB
> db.users.find()
{ "_id" : ObjectId("4e290aa39a1945747b28f1ee"), "a" : 1, "b" : 1 }
{ "_id" : ObjectId("4e2cd2182a65c81f21566318"), "a" : 3, "b" : 5 }
>

整库备份

mongodump -h dbhost -d dbname -o dbdirectory

-h：MongDB所在服务器地址，例如：127.0.0.1，当然也可以指定端口号：127.0.0.1:27017
-d：需要备份的数据库实例，例如：test
-o：备份的数据存放位置，例如：c:datadump，当然该目录需要提前建立，在备份完成后，系统自动在dump目录下建立一个test目录，这个目录里面存放该数据库实例的备份数据。

mongodump的官方说明(可通过mongodump --help查看)：
options:
 --help          produce help message
 -v [ --verbose ]     be more verbose (include multiple times for more
              verbosity e.g. -vvvvv)
 --version        print the program's version and exit
 -h [ --host ] arg    mongo host to connect to ( /s1,s2 for
              sets)
 --port arg        server port. Can also use --host hostname:port
 --ipv6          enable IPv6 support (disabled by default)
 -u [ --username ] arg  username
 -p [ --password ] arg  password
 --dbpath arg       directly access mongod database files in the given
              path, instead of connecting to a mongod server -
              needs to lock the data directory, so cannot be used
              if a mongod is currently accessing the same path
 --directoryperdb     if dbpath specified, each db is in a separate
              directory
 --journal        enable journaling
 -d [ --db ] arg     database to use
 -c [ --collection ] arg collection to use (some commands)
 -o [ --out ] arg (=dump) output directory or "-" for stdout
 -q [ --query ] arg    json query
 --oplog         Use oplog for point-in-time snapshotting
 --repair         try to recover a crashed database
 --forceTableScan     force a table scan (do not use $snapshot)

备注:
--forceTableScan 在内存比较空闲时可以显著加快对冷数据的备份, 备份速度提升接近100倍

整库恢复

mongorestore -h dbhost -d dbname –directoryperdb dbdirectory

-h：MongoDB所在服务器地址
-d：需要恢复的数据库实例，例如：test，当然这个名称也可以和备份时候的不一样，比如test2
--directoryperdb：备份数据所在位置
–drop：恢复的时候，先删除当前数据，然后恢复备份的数据。就是说，恢复后，备份后添加修改的数据都会被删除

mongorestore的官方说明(可通过mongorestore --help查看)：
options:
 --help         produce help message
 -v [ --verbose ]    be more verbose (include multiple times for more
             verbosity e.g. -vvvvv)
 --version        print the program's version and exit
 -h [ --host ] arg    mongo host to connect to ( /s1,s2 for sets)
 --port arg       server port. Can also use --host hostname:port
 --ipv6         enable IPv6 support (disabled by default)
 -u [ --username ] arg  username
 -p [ --password ] arg  password
 --dbpath arg      directly access mongod database files in the given
             path, instead of connecting to a mongod server -
             needs to lock the data directory, so cannot be used
             if a mongod is currently accessing the same path
 --directoryperdb    if dbpath specified, each db is in a separate
             directory
 --journal        enable journaling
 -d [ --db ] arg     database to use
 -c [ --collection ] arg collection to use (some commands)
 --objcheck       validate object before inserting
 --filter arg      filter to apply before inserting
 --drop         drop each collection before import
 --oplogReplay      replay oplog for point-in-time restore
 --oplogLimit arg    exclude oplog entries newer than provided timestamp
             (epoch[:ordinal])
 --keepIndexVersion   don't upgrade indexes to newest version
 --noOptionsRestore   don't restore collection options
 --noIndexRestore    don't restore indexes
 --w arg (=1)      minimum number of replicas per write

单个collection备份

mongoexport -h dbhost -d dbname -c collectionname -f collectionKey -o dbdirectory

-h: MongoDB所在服务器地址
-d: 需要恢复的数据库实例
-c: 需要恢复的集合
-f: 需要导出的字段(省略为所有字段)
-o: 表示导出的文件名

mongoexport的官方说明(可通过mongoexport --help查看)：
 --help          produce help message
 -v [ --verbose ]     be more verbose (include multiple times for more
              verbosity e.g. -vvvvv)
 --version         print the program's version and exit
 -h [ --host ] arg     mongo host to connect to ( /s1,s2 for
              sets)
 --port arg        server port. Can also use --host hostname:port
 --ipv6          enable IPv6 support (disabled by default)
 -u [ --username ] arg   username
 -p [ --password ] arg   password
 --dbpath arg       directly access mongod database files in the given
              path, instead of connecting to a mongod server -
              needs to lock the data directory, so cannot be used
              if a mongod is currently accessing the same path
 --directoryperdb     if dbpath specified, each db is in a separate
              directory
 --journal         enable journaling
 -d [ --db ] arg      database to use
 -c [ --collection ] arg  collection to use (some commands)
 -f [ --fields ] arg    comma separated list of field names e.g. -f
              name,age
 --fieldFile arg      file with fields names - 1 per line
 -q [ --query ] arg    query filter, as a JSON string
 --csv           export to csv instead of json
 -o [ --out ] arg     output file; if not specified, stdout is used
 --jsonArray        output to a json array rather than one object per
              line
 -k [ --slaveOk ] arg (=1) use secondaries for export if available, default
              true
 --forceTableScan     force a table scan (do not use $snapshot)

单个collection恢复

mongoimport -d dbhost -c collectionname –type csv –headerline –file

-type: 指明要导入的文件格式
-headerline: 批明不导入第一行，因为第一行是列名
-file: 指明要导入的文件路径

mongoimport的官方说明(可通过mongoimport --help查看)：
 --help         produce help message
 -v [ --verbose ]    be more verbose (include multiple times for more
             verbosity e.g. -vvvvv)
 --version        print the program's version and exit
 -h [ --host ] arg    mongo host to connect to ( /s1,s2 for sets)
 --port arg       server port. Can also use --host hostname:port
 --ipv6         enable IPv6 support (disabled by default)
 -u [ --username ] arg  username
 -p [ --password ] arg  password
 --dbpath arg      directly access mongod database files in the given
             path, instead of connecting to a mongod server -
             needs to lock the data directory, so cannot be used
             if a mongod is currently accessing the same path
 --directoryperdb    if dbpath specified, each db is in a separate
             directory
 --journal        enable journaling
 -d [ --db ] arg     database to use
 -c [ --collection ] arg collection to use (some commands)
 -f [ --fields ] arg   comma separated list of field names e.g. -f name,age
 --fieldFile arg     file with fields names - 1 per line
 --ignoreBlanks     if given, empty fields in csv and tsv will be ignored
 --type arg       type of file to import. default: json (json,csv,tsv)
 --file arg       file to import from; if not specified stdin is used
 --drop         drop collection first
 --headerline      CSV,TSV only - use first line as headers
 --upsert        insert or update objects that already exist
 --upsertFields arg   comma-separated fields for the query part of the
             upsert. You should make sure this is indexed
 --stopOnError      stop importing at first error rather than continuing
 --jsonArray       load a json array, not one item per line. Currently
             limited to 16MB.

其他导入与导出操作

1. mongoimport -d my_mongodb -c user user.dat

参数说明:

-d 指明使用的库, 本例中为” my_mongodb”

-c 指明要导出的表, 本例中为”user”

可以看到导入数据的时候会隐式创建表结构
2. mongoexport -d my_mongodb -c user -o user.dat

参数说明:

-d 指明使用的库, 本例中为” my_mongodb”

-c 指明要导出的表, 本例中为”user”

-o 指明要导出的文件名, 本例中为”user.dat”

从上面可以看到导出的方式使用的是JSON 的样式.

redis

持久化设置

save 900 1    # 900秒内有至少1个键被更改则进行快照
save 300 10   # 300秒内有至少10个键被更改则进行快照
save 60 10000 # 60秒内有至少10000个键被更改则进行快照

redis 可以直接复制持久化的文件进行备份

#! /bin/bash

PATH=/usr/local/bin:$PATH
redis-cli SAVE

date=$(date +"%Y%m%d")
cp /var/lib/redis/6379/dump.rdb /data01/cache_backup/$date.rdb

echo "done!"

若没有进行持久化可以通过SAVE命令进行持久化

redis 127.0.0.1:6379> SAVE 

另外可以后台进行操作

127.0.0.1:6379> BGSAVE

Background saving started