Use Rsync for Daily, Weekly and Full Monthly Backups (Automated SSH Login For Remote Backups) Scripts

Bunch of references using rsync in different scenarios backing up automatically on local or remote server.

SSH login reference 1:

Don’t use a password. Generate a passphraseless SSH key and push it to your VM.

If you already have an SSH key, you can skip this step… Just hit Enter for the key and both passphrases:

[/crayon]
Copy your keys to the target server:

[/crayon]
Now try logging into the machine, with 

, and check in:

[/crayon]
to make sure we haven’t added extra keys that you weren’t expecting.

Finally check logging in…

[/crayon]
You may also want to look into using 

 if you want to try keeping your keys protected with a passphrase.

Easy answer in three easy steps

Generate a rsa keypair:

[/crayon]
then copy it on the server with one simple command:

[/crayon]
you can now log in without password:

[/crayon]

 

Rsync Reference 2:

Today, we will be using rsync to make daily, weekly, incremental backups and then a full compressed/archived backup once a month. We will then use cron to automate the process. Lets face it us humans get lazy sometimes and most backup systems loose complete effectiveness if they are not completely automated.

What is rsync?

Rsync is a sweet utility for syncing files/folders. Many times it is used for producing incremental backups since it is capable of detecting what files are added and changed to a folder. It usually does this by timestamps but it can be set to determine file changes wih a more precise (but slow) method using md5 hashes. However, generating md5 hashes for detecting file changes is usually not required.

Syncing vs Full Backup

Before geting into the details I think it is worth explaining the difference between full backups and incremental backups. With incremental backups I like to think of them as syncing. You are making the two sets of data match. For example, if one data set contains one extra file the incremental backup will only add that one file to the backup. As opposed to re-copying the other files. This is useful for maintaining frequent backups without the added bandwidth or processing overhead.

Installing rsync for Debian / Ubuntu

If rsync is not installed on your Debian / Ubuntu system you can install it with your preferred method or use aptitude:

[/crayon]
Thats it! Did you expect more?

Installing rsync for FreeBSD

It is prefered to compile applications in FreeBSD rather than binary packages. So we will install it with the following commands:

[/crayon]
If you are prompted with make instructions the defaults are fine.

Syncing Two Folders for Daily Backup

For our daily backup we will use the incremental method since it will be very frequent. We also don’t want to waste disk space, use unnecessary write operations, and CPU cycles by doing a full backup.

To sync one folder to the next we use the following command:

[/crayon]
The ‘a’ flag tells rsync to go into archive mode. The ‘v’ option is just verbose (show details).

Now this should copy all of the files from one folder to the other. If the files already exist and they have not been modified since the last time you run this command it will simply copy those files. So basically when you run this command it will backup the files that have changed since the day before.

Adding the –delete flag. By default rsync does not delete files in the backup if they were deleted in the source. This is rsync’s way of protecting you from deleting the entire backup directly on accident. To force rsync to delete files we do this:

[/crayon]
This will ensure that we are not backing up deleted files.

Warning: When using the –delete flag be sure to check your command twice. If you reverse the source with the destination you will sync your data with an empty folder. You will be left with two empty folders!

Weekly Sync

For our weekly sync will just sync with the latest daily folder.

[/crayon]
We run this command once a week to maintain our weekly incremental backup. Now if we accidentally deleted something last Tuesday and just noticed it on Friday we will have a backup.

Full Monthly Backup

Since we are going to keep full monthly backups and they won’t be accessed frequently we can compress them with bzip.

[/crayon]
Now since we are archiving our full backups monthly we want to be sure not to over write an existing monthly backup. We will do this by naming each one with the date. Instead of the command above use this one, to add the date to each filename.

[/crayon]
Now you should have all the commands to set up a rotating backup daily, weekly and a full monthly backup. The only thing now is to execute those commands every day, week, and end of the month.

Automate the Process with Cron

Usually we don’t want to have to remember to type in a command daily, weekly, and at the end of each month so, we will automate it with cron.

[/crayon]
Now add the following lines of code:

[/crayon]
This example cron setup will backup daily at 5:30PM. Backup every Friday at 6:00PM. Do the full backup on the first of each month at 6:00AM

For more information on cron see, Learning Cron by Example.

Now you will need to tailor this to the usage patterns of you or your users. You should also allow enough time for the daily backup to finish before doing the weekly. In this example on Fridays I allowed 59 minutes for the daily backup to finish. If you are worried about the sync time running into each other you can schedule your daily backup in the morning and your Friday weekly backup at night.

If you want to find out how long it is currently taking to do a backup add ‘time’ command to the beginning of each command.

Tell Cron to be Quiet

Cron by default sends emails with the output of the command. If you don’t want to get emails you can pipe the cron comands to /dev/null. Add this to the end of each cron line:

[/crayon]
 

Rsync Reference 3:

The basics of rsync

 is a very versatile copying and backup tool that is included by default in almost every Linux distribution. It can be used as an advanced copying tool, allowing us to copy files both locally and remotely. It can also be used as a backup tool. It supports the creation of incremental backups.

 counts with a famous delta-transfer algorithm that allows us to transfer new files as well as recent changes to existent files, while ignoring unchanged files. Additionally to this, the behavior of 

 can be throughly customized, helping us to automatize backups, it can also be run as a daemon to turn the computer into a host and allow 

 clients connect to it.

Besides the copying of local files and folders, 

 allow us to copy over SSH (Secure Shell), RSH (Remote Shell) and it can be run as a daemon in a computer and allow other computers to connect to it, when 

 is run as a daemon it listens to the port TCP 873.

When we use 

 as a daemon or when we use RSH, the data that is send between computers travels unencrypted, so, if you are transferring files between two computers in the same local network, this is useful, but this shouldn’t be used to transfer files over insecure networks, such as the Internet. For this purpose SSH is the way to go.

This is the main reason why I favor the use of SSH for my transfers, besides, since SSH is secure, many servers have the SSH daemon available. But the use of 

 as a daemon for transfers over fast connections, as is usually the case in a local network, is useful. I don’t have the RSH daemon running in my computers so you may find me a bit biased about SSH in the examples. The examples covering the transfer of files between two computers use SSH as the medium of transport, but in a separate post I cover the use of rsync as a daemon.

Copying local files and folders

To copy the contents of one local folder into another, replacing the files in the destination folder, we use:

In the source_folder notice that I added a slash at the end, doing this prevents a new folder from being created, if we don’t add the slash, a new folder named as the source folder will be created in the destination folder. So, if you want to copy the contents of a folder called Pictures into an existent folder which is also called Pictures but in a different location, you need to add the trailing slash, otherwise, a folder called Pictures is created inside the Pictures folder that we specify as destination.

A graphical representation of the results of rsync with a trailing slash in the source folder.

A graphical representation of the results of rsync without a trailing slash in the source folder.

The parameter 

 means recursive, this is, it will copy the contents of the source folder, as well as the contents of every folder inside it.

The parameter 

 makes rsync preserve the modification times of the files that it copies from the source folder.

The parameter 

 means verbose, this parameter will print information about the execution of the command, such as the files that are successfully transferred, so we can use this as a way to keep track of the progress of rsync.

This are the parameters that I frequently use, as I am usually backing up personal files and this doesn’t contain things such as symlinks, but another very useful parameter to use rsync with is the parameter -a.

The parameter 

 also makes the copy recursive and preserve the modification times, but additionally it copies the symlinks that it encounters as symlinks, preserve the permissions, preserve the owner and group information, and preserve device and special files. This is useful if you are copying the entire home folder of a user, or if you are copying system folders somewhere else.

Dealing with whitespace and rare characters

We can escape spaces and rare characters just as in bash, by the use of \ before any whitespace and rare character. Additionally, we can use single quotes to enclose the string:

Update the contents of a folder

In order to save bandwidth and time, we can avoid copying the files that we already have in the destination folder that have not been modified in the source folder. To do this, we can add the parameter 

 to 

, this will synchronize the destination folder with the source folder, this is where the delta-transfer algorithm enters. To synchronize two folders like this we use:

By default, 

 will take into consideration the date of modification of the file and the size of the file to decide whether the file or part of it needs to be transferred or if the file can be left alone, but we can instead use a hash to decide whether the file is different or not. To do this we need to use the 

 parameter, which will perform a checksum in the files to be transferred. This will skip any file where the checksum coincides.

Synchronizing two folders with rsync

To keep two folders in synchrony, not only do we need to add the new files in the source folder to the destination folder, as in the past topics, we also need to remove the files that are deleted in the source folder from the destination folder. 

 allow us to do this with the parameter 

, this used in conjunction with the previously explained parameter 

 that updates modified files allow us to keep two directories in synchrony while saving bandwidth.

The deletion process can take place in different moments of the transfer by adding some additional parameters:

  • rsync can look for missing files and delete them before it does the transfer process, this is the default behavior and can be set with 
  • rsync can look for missing files after the transfer is completed, with the parameter 
  • rsync can delete the files done during the transfer, when a file is found to be missing, it is deleted at that moment, we enable this behavior with 
  • rsync can do the transfer and find the missing files during this process, but instead of delete the files during this process, waits until it is finished and delete the files it found afterwards, this can be accomplished with 

e.g.:

Compressing the files while transferring them

To save some bandwidth, and usually it can save some time as well, we can compress the information being transfer, to accomplish this we need to add the parameter 

 to rsync.

Note, however, that if we are transferring a large number of small files over a fast connection, rsync may be slower with the parameter -z than without it, as it will take longer to compress every file before transfer it than just transferring over the files. Use this parameter if you have a a connection with limited speed between two computers, or if you need to save bandwidth.

Transferring files between two remote systems

 can copy files and synchronize a local folder with a remote folder in a system running the SSH daemon, the RSH daemon, or the rsync daemon. The examples here use SSH for the file transfers, but the same principles apply if you want to do this with 

 as a daemon in the host computer, read Running rsync as a daemon when ssh is not available further below for more information about this.

To transfer files between the local computer and a remote computer, we need to specify the address of the remote system, it may be a domain name, an IP address, or a the name of a server that we have already saved in our SSH config file (information about how to do this can be found in Defining SSH servers), followed by a colon, and the folder we want to use for the transfer. Note that rsync can not transfer files between two remote systems, only a local folder or a remote folder can be used in conjunction with a local folder. To do this we use:

Local folder to remote folder, using a domain, an IP address and a server defined in the SSH configuration file:

Remote folder to local folder, using a domain, an IP address and a server defined in the SSH configuration file:

Excluding files and directories

There are many cases in which we need to exclude certain files and directories from rsync, a common case is when we synchronize a local project with a remote repository or even with the live site, in this case we may want to exclude some development directories and some hidden files from being transfered over to the live site. Excluding files can be done with 

 followed by the directory or the file that we want to exclude. The source folder or the destination folder can be a local folder or a remote folder as explained in the previous section.

The paths are relative to the folder from which we are calling rsync unless it starts with a slash, in which case the path would be absolute.

Another way to do this is to create a file with the list of both files and directories to exclude from rsync, as well as patterns (all files that would match the pattern would be excluded, *.txt would exclude any file with that extension), one per line, and call this file with 

 followed by the file that we want to use for the exclusion of files. First, we create and edit this file in our favorite text editor, in this example I use gedit, but you may use kate, Vim, nano, or any other text editor:

In this file we can add the following:

And then we call rsync:

In addition to delete files that have been removed from the source folder, as explained in Synchronizing two folders with rsync

 can delete files that are excluded from the transfer, we do this with the parameter 

, e.g.:

This command would make 

 recursive, preserve the modification times from the source folder, increase verbosity, exclude all the files that match the patterns in the file 

, and delete all of this files that match the patternif they exist in the destination folder.

Running rsync as a daemon when ssh is not available

This was moved to it’s own section, Running rsync as a daemon.

Some additional rsync parameters

 Preserves the modification times of the files that are being transferred.

 Suppress any non-error message, this is the contrary to -v which increases the verbosity.

 Transfer a directory without recursing, this is, only the files that are in the folder are transferred.

 Copy the symlinks as symlinks.

 Copy the file that a symlink is pointing to whenever it finds a symlink.

 Copy whole files. When we are using the delta-transfer algorithm we only copy the part of the archive that was updated, sometimes this is not desired.

 Shows the progress of the files that are being transferred.

 Shows the information that rsync provides us in a human readable format, the amounts are given in K’s, M’s, G’s and so on.

Footnotes

The amount of options that 

 provide us is immense, we can define exactly which files we want to transfer, what specific files we want to compress, what files we want to delete in the destination folder if this files exists, and we can deal with system files as well, for more information we can use 

 and 

I leave the information concerning backups out of this post, as this will be covered, together with the automation of the backups, in an incoming post.

It is possible to run rsync on Windows with the use of cygwin, however I don’t have a Windows box available at the moment (nor do I plan to aquire one in the foreseeable future), so even thought I have done it I can’t post about this. If you run 

 as a service in Windows tho, you need to add the line “

” in 

 under the modules area, this will prevent 

 from checking the permissions in the secrets file and thus failing because they are not properly set (as they don’t work the same as in Linux).

This post may be updated if there is something to correct or to add a little more information if I see it necessary.

Rsync (Remote Sync): 10 Practical Examples of Rsync Command in Linux

Rsync (Remote Sync) is a most commonly used command for copying and synchronizing files and directories remotely as well as locally in Linux/Unix systems. With the help of rsync command you can copy and synchronize your data remotely and locally across directories, across disks and networks, perform data backups and mirroring between two Linux machines.

This article explains 10 basic and advanced usage of the rsync command to transfer your files remotely and locally in Linux based machines. You don’t need to be root user to run rsync command.

Some advantages and features of Rsync command
  1. It efficiently copies and sync files to or from a remote system.
  2. Supports copying links, devices, owners, groups and permissions.
  3. It’s faster than scp (Secure Copy) because rsync uses remote-update protocol which allows to transfer just the differences between two sets of files. First time, it copies the whole content of a file or a directory from source to destination but from next time, it copies only the changed blocks and bytes to the destination.
  4. Rsync consumes less bandwidth as it uses compression and decompression method while sending and receiving data both ends.
Basic syntax of rsync command

Some common options used with rsync commands
  1. -v : verbose
  2. -r : copies data recursively (but don’t preserve timestamps and permission while transferring data
  3. -a : archive mode, archive mode allows copying files recursively and it also preserves symbolic links, file permissions, user & group ownerships and timestamps
  4. -z : compress file data
  5. -h : human-readable, output numbers in a human-readable format
Install rsync in your Linux machine

We can install rsync package with the help of following command.

1. Copy/Sync Files and Directory Locally

Copy/Sync a File on a Local Computer

This following command will sync a single file on a local machine from one location to another location. Here in this example, a file name backup.tar needs to be copied or synced to /tmp/backups/ folder.

In above example, you can see that if the destination is not already exists rsync will create a directory automatically for destination.

Copy/Sync a Directory on Local Computer

The following command will transfer or sync all the files of from one directory to a different directory in the same machine. Here in this example, /root/rpmpkgs contains some rpm package files and you want that directory to be copied inside /tmp/backups/ folder.

2. Copy/Sync Files and Directory to or From a Server

Copy a Directory from Local Server to a Remote Server

This command will sync a directory from a local machine to a remote machine. For example: There is a folder in your local computer “rpmpkgs” which contains some RPM packages and you want that local directory’s content send to a remote server, you can use following command.

Copy/Sync a Remote Directory to a Local Machine

This command will help you sync a remote directory to a local directory. Here in this example, a directory /home/tarunika/rpmpkgs which is on a remote server is being copied in your local computer in /tmp/myrpms.

3. Rsync Over SSH

With rsync, we can use SSH (Secure Shell) for data transfer, using SSH protocol while transferring our data you can be ensured that your data is being transferred in a secured connection with encryption so that nobody can read your data while it is being transferred over the wire on the internet.

Also when we use rsync we need to provide the user/root password to accomplish that particular task, so using SSH option will send your logins in an encrypted manner so that your password will be safe.

Copy a File from a Remote Server to a Local Server with SSH

To specify a protocol with rsync you need to give “-e” option with protocol name you want to use. Here in this example, We will be using “ssh” with “-e” option and perform data transfer.

Copy a File from a Local Server to a Remote Server with SSH

4. Show Progress While Transferring Data with rsync

To show the progress while transferring the data from one machine to a different machine, we can use ‘–progress’ option for it. It displays the files and the time remaining to complete the transfer.

5. Use of –include and –exclude Options

These two options allows us to include and exclude files by specifying parameters with these option helps us to specify those files or directories which you want to include in your sync and exclude files and folders with you don’t want to be transferred.

Here in this example, rsync command will include those files and directory only which starts with ‘R’ and exclude all other files and directory.

6. Use of –delete Option

If a file or directory not exist at the source, but already exists at the destination, you might want to delete that existing file/directory at the target while syncing .

We can use ‘–delete‘ option to delete files that are not there in source directory.

Source and target are in sync. Now creating new file test.txt at the target.

Target has the new file called test.txt, when synchronize with the source with ‘–delete‘ option, it removed the file test.txt.

7. Set the Max Size of Files to be Transferred

You can specify the Max file size to be transferred or sync. You can do it with “–max-size” option. Here in this example, Max file size is 200k, so this command will transfer only those files which are equal or smaller than 200k.

8. Automatically Delete source Files after successful Transfer

Now, suppose you have a main web server and a data backup server, you created a daily backup and synced it with your backup server, now you don’t want to keep that local copy of backup in your web server.

So, will you wait for transfer to complete and then delete those local backup file manually? Of Course NO. This automatic deletion can be done using ‘–remove-source-files‘ option.

9. Do a Dry Run with rsync

If you are a newbie and using rsync and don’t know what exactly your command going do. Rsync could really mess up the things in your destination folder and then doing an undo can be a tedious job.

Use of this option will not make any changes only do a dry run of the command and shows the output of the command, if the output shows exactly same you want to do then you can remove ‘–dry-run‘ option from your command and run on the terminal.

10. Set Bandwidth Limit and Transfer File

You can set the bandwidth limit while transferring data from one machine to another machine with the the help of ‘–bwlimit‘ option. This options helps us to limit I/O bandwidth.

Also, by default rsync syncs changed blocks and bytes only, if you want explicitly want to sync whole file then you use ‘-W‘ option with it.

That’s all with rsync now, you can see man pages for more options. Stay connected with Tecmint for more exciting and interesting tutorials in future. Do leave your comments and suggestions.

Rsync Useful Scripts:

 

https://github.com/cripperz/rsync-incremental-backup

https://github.com/cripperz/snapshot-backup

 

References

http://www.jveweb.net/en/archives/2010/11/synchronizing-folders-with-rsync.html

https://www.marksanborn.net/howto/use-rsync-for-daily-weekly-and-full-monthly-backups/

Automatic backup plan for Linux servers using rsync and crontab

https://serverfault.com/questions/241588/how-to-automate-ssh-login-with-password

https://github.com/cripperz/snapshot-backuphttps://github.com/cripperz/rsync-incremental-backup



Comments

comments

Leave a Reply