Rsync
1. rsync intruction
1.1 DESCRIPTION
rsync is a fast, versatile, remote (and local) file-copy tools.
Rsync is a fast and extraordinarily versatile file copying tool. It can copy locally, to/from another host over any remote shell, or to/from a remote rsync daemon. It offers a large number of options that control every aspect of its behavior and permit very flexible specification of the set of files to be copied. It is famous for its delta-transfer algorithm, which reduces the amount of data sent over the network by sending only the differences between the source files and the existing files in the destination. Rsync is widely used for back-ups and mirroring and as an improved copy command for everyday use.
Rsync finds files that need to be transferred using a quick check algorithm (by default) that looks for files that have changed in size or in last-modified time. Any changes in the other preserved attributes (as requested by options) are made on the destination file directly when the quick check indicates that the files data does
not need to be updated.
Some of the additional features of rsync are:
o support for copying links, devices, owners, groups, and permissions
o exclude and exclude-from options similar to GNU tar
o a CVS exclude mode for ignoring the same files that CVS would ignore
o can use any transparent remote shell, including ssh or rsh
o does not require super-user privileges
o pipelining of file transfers to minimize latency costs
o support for anonymous or authenticated rsync daemons (ideal for mirroring)
1.2 SYNOPSIS
Local: rsync [OPTION...] SRC... [DEST]
Access via remote shell:
Pull: rsync [OPTION...] [USER@]HOST:SRC... [DEST]
Push: rsync [OPTION...] SRC... [USER@]HOST:DEST
Access via rsync daemon:
Pull: rsync [OPTION...] [USER@]HOST::SRC... [DEST]
rsync [OPTION...] rsync://[USER@]HOST[:PORT]/SRC... [DEST]
Push: rsync [OPTION...] SRC... [USER@]HOST::DEST
rsync [OPTION...] SRC... rsync://[USER@]HOST[:PORT]/DEST
Usages with just one SRC arg and no DEST arg will list the source files instead of copying.
In a word, rsync is tools that can move data from one driver to another driver
2 setting and using
2.1 check and use it in local
use this command “rpm -qa rsync” then you will see
rsync-3.0.6-9.el6_4.1.x86_64
rsync is default install when preparing Linux system.
so we use it to copy file in local
The common usage is:
rsync [OPTION...] SRC... [DEST]
examples:
rsync /etc/hosts /tmp/
if you want copy directory, you can use “-r” options
rsync -r /tmp /root/
this usage is same as the “copy”
2.2 using based on remote shell
We also can use it by SSH pipelining, so first you should set SSH pipelining
The common usage is:
Pull: rsync [OPTION...] [USER@]HOST:SRC... [DEST]
examples:
rsync -r -e 'ssh -p 52113' ygh_ssh@192.168.98.132:~ /backup/
-e --rsh=COMMAND specify the remote shell to use
Push: rsync [OPTION...] SRC... [USER@]HOST:DEST
examples:
rsync -r -e 'ssh -p 52113' /backup/demo.txt ygh_ssh@192.168.98.132:~
2.3 using based on a rsync daemon(this is a very important way)
2.3.1 preparation works
Before use this usage, we need know some important options
-v --verbose increase verbosity
-z --compress compress file date during the transfer
-a --archive archive mode;equals -rlptgoD (no -H,-A,-X)
-r --recursive recurse into directories
-t --times preserve modification times
-o --owner preserve owner(super user only)
-p --perms preserve permissions
-P --progress show progress during transfer
-D --devices special files
-l --links copy symlinks as symlinks
-a is contain above options
-e --rsh=COMMAND specify the remote shell to use
--exclude=PATTERN exclude files matching PATTERN
--exclude-from=FILE read exclude patterns from FILE
--bwlimit=KBPS limit I/O bandwidth; KBytes per second
--delete delete extraneous files from dest dirs
the options “avz” is usually used
If you want more, you can reference this command “man rsync”
Now new need two machines, one as the client, another is as service to run rsync
daemon
The details information is following:
|
Hostname |
Network card |
Gateway |
Functions |
|
NFS_SERVICE |
192.168.98.132 |
255.255.255.0 |
Run rsync daemon |
|
NFS_CLIENT_1 |
192.168.98.135 |
255.255.255.0 |
Pull or push data |
|
NFS_CLIENT_2 |
192.168.98.134 |
255.255.255.0 |
Pull or push data |
2.3.2 Setting daemon in service
Firstly, create “/etc/rsyncd.conf” file, the file is not existing default, you need to create it..
use this command “vim /etc/rsyncd.conf”,then writing following content
#rsync configuration
#user and group
uid = rsync
gid = rsync
use chroot = no
max connections = 200
timeout = 300
#the file put pid of rsync
pid file = /var/run/rsyncd.pid
#the file put lock information of rsync
lock file = /var/run/rsync.lock
#the logs file
log file = /var/log/rsyncd.log
ignore errors
read only = false
list = false
hosts allow = 255.255.255.0/24
hosts deny = 0.0.0.0/32
#the vitual user
auth users = rsync_backup
##the password is put in this
secrets file = /etc/rsync.password
#exclude file
#exclude= a b
#this is mode
[oldboy]
path = /oldboy/
you can get more information by “man rsyncd.conf”
Secondly, create password file “/etc/rsync.password” then write following content,
then set it’s mod is 600,othwise rsync will give exception.
#username:password
rsync_backup:oldboy
Thirdly, run rsync daemon
use this command “[root@nfs_service ~]# rsync --daemon” to run rsync daemon
then use following command to view rsync daemon status
[root@nfs_service ~]# ps -ef|grep rsync
root 7700 1 0 15:43 ? 00:00:00 rsync --daemon
root 7702 1472 0 15:43 pts/0 00:00:00 egrep --color=auto rsync
Above status indicate you rsync daemon is successfully running
now you can use COMMAND in client to transfer data
2.3.3 Test rsync daemon in client
In client, we need to set a password file for rsync COMMAND
echo “oldboy” >>/etc/rsync.password
the “oldboy” is same as the password in service
The USE is “/etc/rsyncd.conf” auth users = rsync_backup and the shared directory is the path = /oldboy/ by oldboy to build connection from auth_users to shared directory
The common usage is following:
Access via rsync daemon:
Pull: rsync [OPTION...] [USER@]HOST::SRC... [DEST]
example:
rsync -avz rsync_backup@192.168.98.132::oldboy /rysnc --password-file=/etc/rsync.password
这里的oldboy不是目录，是服务器rsync配置文件的一个模块，通过模块建立共享目录和虚拟用户直接的联系
rsync [OPTION...] rsync://[USER@]HOST[:PORT]/SRC... [DEST]
examples:
rsync -avz rsync://rsync_backup@192.168.98.132/oldboy /rsync --password-file=/etc/rsync.password
Push: rsync [OPTION...] SRC... [USER@]HOST::DEST
examples:
rsync -avz /tmp rsync_backup@192.168.98.132::oldboy --password-file=/etc/rsync.password
rsync [OPTION...] SRC... rsync://[USER@]HOST[:PORT]/DEST
examples:
rsync -avz /etc/ rsync://rsync_backup@192.168.98.132/oldboy --password-file=/etc/rsync.password
2.3.4 some other method
exclude copy：exclude some file you want not to copy
--exclude=PATTERN exclude files matching PATTERN
--exclude-from=FILE read exclude patterns from FILE
Examples:
one file exclude:
rsync -avz --exclude=a /rsync/ rsync_backup@192.168.98.132::oldboy --password-file=/etc/rsync.password
more files exclude:
rsync -avz --exclude={a,b} /rsync/ rsync_backup@192.168.98.132::oldboy --password-file=/etc/rsync.password
more files exclude sortly:
rsync -avz --exclude={a..g} /rsync/ rsync_backup@192.168.98.132::oldboy --password-file=/etc/rsync.password
rsync -avz --exclude={1..10} /rsync/ rsync_backup@192.168.98.132::oldboy --password-file=/etc/rsync.password
more files names are put in exclude.log
seq 10 > exclude.log
cat exclude.log
rsync -avz --exclude-from=/rsync/exclude.log /rsync/ rsync_backup@192.168.98.132::oldboy --password-file=/etc/
the same copy keep client and service is same, the file is more in service will be delete：
rsync -avz --delete --exclude=a /rsync/ rsync_backup@192.168.98.132::oldboy --password-file=/etc/rsync.password
2.4 set more shared directory
If you want to create more shared directory, just set in “/etc/rsyncd.conf”
just like:
#rsync configuration
#user and group
uid = rsync
gid = rsync
use chroot = no
max connections = 200
timeout = 300
#the file put pid of rsync
pid file = /var/run/rsyncd.pid
#the file put lock information of rsync
lock file = /var/run/rsync.lock
#the logs file
log file = /var/log/rsyncd.log
ignore errors
read only = false
list = false
hosts allow = 255.255.255.0/24
hosts deny = 0.0.0.0/32
#the vitual user
auth users = rsync_backup
##the password is put in this
secrets file = /etc/rsync.password
#exclude file
#exclude= a b
[oldboy]
path = /oldboy/
[ygh]
path = /ygh/
#self setting
read only = true
[backup]
path = /backup/
If the set information is public, you can set it at top
if you need special setting at your mode, you can setting in self mode
3 Example
某公司有一台Web服务器，里面数据很重要，但是如果硬盘坏了，数据就会丢失，
现在领导要求你对数据做一下备份这样的Web服务器数据丢失就可以恢复，要求如下：
每天晚上00:00在web服务A上打包备份系统配置文件，网站程序目录以及访问日志并通过rsync命令推送到B
服务器上进行备份（备份的思路是:现在本地按日期打包，然后在推到B服务器上）
具体要求如下：
1）web服务器A和备份服务器B的备份目录都为/backup
2)系统的配置文件包括，但是不限于如下
a.定时任务服务的配置文件(/var/spool/cron/root)
b.开机自启的配置文件(/ect/rc.local)
c.日常的脚本目录(/server/scripts)
d.防火墙iptables的配置文件(/etc/sysconfig/iptables)
3)Web服务器的站点目的假定为(/var/html/www)
4)Web服务器A的访问日志的假定路径为(/app/log)
5)Web服务器保留打包7天的备份数据即可(本地留存不能多余7天，因为太多硬盘会满，准备服务器上要保留6个月以上的数据副本)
6)备份服务器上要按照备份数据服务的IP为目录保存，打包的文件按照时间名字保存
we solve it by a shell script and a crontab in every you can use this scripts
The shell scripts is following:
#!/bin/sh
#backup scripts
dir=/backup
ip=$(ifconfig|awk -F '[ :]+' 'NR==2{print $4}')
#if [ ! -d $dir/$ip ]
#then
mkdir -p /backup/$ip
#fi
cd /backup && cd $ip
tar -zcf backup_$(date +%F).tar.gz /var/www/html/ /app/log/ /server/scripts/ /var/spool/cron/ /etc/
rsync -avz /backup/ rsync_backup@192.168.98.132::backup --password-file=/etc/rsync.password && touch ${ip}-flag-$(date +%F)
rsync -avz /backup/ rsync_backup@192.168.98.132::backup --password-file=/etc/rsync.password
find /backup/ -type f -name "backup*.tar.gz" -mtime +7 |xargs rm -f
then set it in crontab
#backup data by rsync
00 00 * * * /bin/sh /server/scripts/backup/backup.sh > /dev/null 2>&1
Inotify
1 inotify instruction
inotify is a monitoring file system events, In a directory, if a file or directory is created or updated or deleted, it will tell you what file is change, then you can use rsync to deal with this file or directory.
Default, out machine don’t install it, so we need to install it at client
The steps of install is following:
install inotify:
1、dowmload inotify-tools-3.14.tar.gz
2、tar -zxvf inotify-tools-3.14.tar.gz
ll
3、cd inotify-tools-3.14
ll
4、./configure --prefix=/usr/local/inotify-tools-3.14
5、make && make install
6、cd ../
7、ln -s /usr/local/inotify-tools-3.14/ /usr/local/inotify
then, you can use this command “”
2 Testing and Using at Client
view inotify help
1、cd /usr/local/inotify
2、./bin/inotifywait --help
inotifywait 3.14
Wait for a particular event on a file or set of files.
Usage: inotifywait [ options ] file1 [ file2 ] [ file3 ] [ ... ]
Options:
-h|--help Show this help text.
@<file> Exclude the specified file from being watched.
--exclude <pattern>
Exclude all events on files matching the
extended regular expression <pattern>.
--excludei <pattern>
Like --exclude but case insensitive.
-m|--monitor Keep listening for events forever. Without
this option, inotifywait will exit after one
event is received.
-d|--daemon Same as --monitor, except run in the background
logging events to a file specified by --outfile.
Implies --syslog.
-r|--recursive Watch directories recursively.
--fromfile <file>
Read files to watch from <file> or `-' for stdin.
-o|--outfile <file>
Print events to <file> rather than stdout.
-s|--syslog Send errors to syslog rather than stderr.
-q|--quiet Print less (only print events).
-qq Print nothing (not even events).
--format <fmt> Print using a specified printf-like format
string; read the man page for more details.
--timefmt <fmt> strftime-compatible format string for use with
%T in --format string.
-c|--csv Print events in CSV format.
-t|--timeout <seconds>
When listening for a single event, time out after
waiting for an event for <seconds> seconds.
If <seconds> is 0, inotifywait will never time out.
-e|--event <event1> [ -e|--event <event2> ... ]
Listen for specific event(s). If omitted, all events are
listened for.
Exit status:
0 - An event you asked to watch for was received.
1 - An event you did not ask to watch for was received
(usually delete_self or unmount), or some error occurred.
2 - The --timeout option was given and no events occurred
in the specified interval of time.
Events:
access file or directory contents were read
modify file or directory contents were written
attrib file or directory attributes changed
close_write file or directory closed, after being opened in
writeable mode
close_nowrite file or directory closed, after being opened in
read-only mode
close file or directory closed, regardless of read/write mode
open file or directory opened
moved_to file or directory moved to watched directory
moved_from file or directory moved from watched directory
move file or directory moved to or from watched directory
create file or directory created within watched directory
delete file or directory deleted within watched directory
delete_self file or directory was deleted
unmount file system containing file or directory unmounted
some example:
Number 1
inotify listening command:
/usr/local/inotify/bin/inotifywait -mrq --timefmt '%d-%m-%y %H:%M' --format '%T %w%f' -e create /backup
-m -r and -q and -e you can retrieve above
execute following commands:
touch b.txt
listening result:
17-01-17 09:43 /backup/b.txt
Number 2
inotify listening command:
/usr/local/inotify/bin/inotifywait -mrq --timefmt '%d-%m-%y %H:%M' --format '%T %w%f' -e create,delete /backup
if want to listen more evenet, you can use "," as the seperator,%T is to print time %w is to print directory name
%f is to print filename
execute following commands:
mkdir t1
touch t2
rm t2
rm -rf t1
the listening results:
17-01-17 09:46 /backup/t1
17-01-17 09:46 /backup/t2
17-01-17 09:48 /backup/t2
17-01-17 09:48 /backup/t1
Number 3:
inotify listening command:
/usr/local/inotify/bin/inotifywait -mrq --timefmt '%d-%m-%y %H:%M' --format '%T %w%f' -e create,delete,close_write /backup
execute following commands:
echo "sdsada
touch c.txt
the listening results:
17-01-17 09:54 /backup/a.txt
17-01-17 09:55 /backup/c.txt
17-01-17 09:55 /backup/c.txt
the touch.txt will be listened by create and close_write
3 Examples
In some important situation, you should to backup data constantly, so you will use
inotify and rsync
you can use following shell scripts to solve it.
#!/bin/sh
#use inotify and rsync to backup data all time
host=192.168.98.132
src=/backup
dst=oldboy
user=rsync_backup
rsync_passfile=/etc/rsync.password
inotify_home=/usr/local/inotify-tools/
${inotify_home}bin/inotifywait -mrq --timefmt '%d-%m-%y %H:%M' --format '%w%f' -e close_write,delete,create,attrib $src
|while read line
do
rsync -avz $line $user@$host::$dst --password-file=${rsync_passfile} > /dev/null 2>&1
done
exit 0
then you can run this script at behind
/bin/sh /server/scripts/inotify/inotify1.sh &