Saltycrane logo

SaltyCrane Blog

Notes on Python, Django, and web development on Ubuntu Linux

    

Backup on Linux: rsnapshot vs. rdiff-backup (vs. Time Machine)

Date: 2008-02-07  |   Tags: linux, rsync, softwaretools, ubuntu   |   23 Comments

Apple's release of Leopard and the included backup utility, Time Machine, has generated a lot of talk about backups recently. I will admit Time Machine is pretty cool and believe that it is a bit more than a glorified GUI on top of an existing *nix tool as some have claimed. However, the core functionality is very similar to the command-line tool, rsnapshot, which is itself based on a rsync script by Mike Rubel. Time Machine added a couple of features and a GUI to make it easy to use. Since I prefer the command line over GUIs most of the time anyways, rsnapshot seemed perfect for me.

To be complete, I researched a number of other backup utilities for Linux. Dirvish and flyback were out because I prefer the command line and they didn't seem to offer anything more than rsnapshot. Scripting rsync wouldn't get me anything more than rsnapshot either, plus it would be more work. In the end, I eliminated all but rsnapshot and another command line tool called rdiff-backup. Rdiff-backup has some advantages over rsnapshot (and Time Machine) because it stores compressed deltas of each version of a file instead of a complete copy of the file each time it changes. This is not a big deal for small files, but for large files that change often, it makes a significant difference. However, the big disadvantage of rdiff-backup, for me, was the inablility to perform different levels of backup, such as hourly, daily, weekly, monthly, etc. Depending on the needs of the user, this could negate the space saving advantage by needing to keep a large number of snapshots.

I ended up choosing rsnapshot over rdiff-backup because of this last reason. It seems rdiff-backup is closer to a version control tool and rsnapshot closer to a traditional backup solution. It would be great to create a hybrid of the two tools to gain the advantages of each. I started to dig into the source of rdiff-backup (python source makes me happy), but I didn't want to get too sidetracked with another project. For now, I am using Mercurial to version control my /etc and /home/.* config files and rsnapshot as a broader, general purpose backup tool.


Here is my comparison of rsnapshot and rdiff-backup:

Similar Features
  • both use an rsync-like algorithm to transfer data (rsnapshot actually uses rsync; rdiff-backup uses the python librsync library)
  • both can be used over ssh (though rsnapshot cannot push over ssh without some extra scripting)
  • both use a simple copy of the source for the current backup
Written in rsnapshot is written in Perl; rdiff-backup is written in Python and C
Size rdiff-backup stores previous versions as compressed deltas to the current version similar to a version control system. rsnapshot uses actual files and hardlinks to save space. For small files, storage size is similar. For large files that change often, such as logfiles, databases, etc., rdiff-backup requires significantly less space for a given number of versions.
Speed rdiff-backup is slower than rsnapshot
Metadata rdiff-backup stores file metadata, such as ownership, permissions, and dates, separately.
Transparency For rsnapshot, all versions of the backup are accessible as plain files. For rdiff-backup, only the current backup is accessible as plain files. Previous versions are stored as rdiff deltas.
Backup levels rsnapshot supports multiple levels of backup such as monthly, weekly, and daily. rdiff-backup can only delete snapshots earlier than a given date; it cannot delete snapshots in between two dates.
Community Based on the number of responses to my post on the mailing lists (rsnapshot: 6, rdiff-backup: 0), rsnapshot has a more active community.

Reference: Chapter 7 in Backup & Recovery O'Reilly 2007

23 Comments — feed icon Comments feed for this post


#1 yungchin commented on 2008-02-27:

Hi, nice post. I'd be interested to read more about how you've set up version control on your home dir (could make a nice blog post? :)).

I haven't figured out a manageable way yet (using bzr) - there are always lots of new files appearing that have to go to your ignore-list, or something like that...

Thanks! YC


#2 Eliot commented on 2008-03-26:

yungchin,
thanks. i am still refining the process of how to use the version control. i plan to post about it sometime soon. please let me know what solution you go with as well.


#3 manuel quinta commented on 2008-10-17:

I use rdiff-backup since years to make backups over ssh, and i'am happy with it.

we used rsnapshot on our server, to make a backup over ssh. but it took a lot of cpu? also we did a crazy missconfiguration, as we wanted to stop the hourly backup, because of the cpu load. wenn you suspend the hourly backup, it will not make a daily... it just stoped to make any backup!

we did only see it by incident a half a year later, but we did not lose any data! luckely!


#4 Eliot commented on 2008-10-17:

manuel, Thanks for the notes on your experiences.


#5 Chris Wilper commented on 2008-12-01:

Note on rdiff-backup, for transparency, there is a FUSE implementation called archfs that allows you to mount all snapshots in a read-only filesystem. It's a little rough around the edges, but I've been impressed now that I've got it to work.

It occurs to me that it should be possible to run rdiff-backup against the fuse-exposed snapshots, in order, skipping the ones you don't want. Haven't tried it, but it would be an interesting experiment...


#6 Eliot commented on 2008-12-08:

Chris, thanks a lot for the information. Please feel free to leave a link to your notes if you've posted any.


#7 johnP commented on 2009-06-18:

We've been using rdiff-backup for a few months. Our requirements are for a cross platform, cheap, solution, so anything that required hard-link support couldn't be considered. Personally, I'd wipe all the MS-windows server machines, but our customers would be mad. We're mostly *NIX for production systems, with a few Windows dev VMs.

In that time, we've recovered from "oops" moments twice. Once was during an email system upgrade that couldn't be completed and the other was while screwing with an Alfresco system permissions model that failed in the end. Backups rock, but folks here already know that.

Has anyone solved the remote Windows backup issue in a good free way that can actually be restored without loss of user and group permissions?

rdiff-backup is mostly good, but there are a few problems. Large file differencing doesn't work in our experience. If it doesn't crash, you'll get a completely new copy. We tried backing up complete Xen image files this way. We got around this issue by mounting the IMG files on the host and pushing the rdiff-backup to another system on the same LAN. Complete system backups are 2-3 minutes now.

That works for Xen, but not VirtualBox with a Vista-64 host. Any ideas?


#8 Eliot commented on 2009-06-22:

johnP, Thank you for adding your experience with rdiff-backup. I'm sorry I can't be of any to help to you. Maybe another visitor will have ideas.


#9 Jakob Malm commented on 2009-07-07:

I've been using rdiff-backup for my home system since March 2008 and am overall happy with it. I've had to restore single files a few times, and though that procedure is not very straightforward, it's been easy enough to figure out how to do it the few times I've needed it. The FUSE file system seems interesting.

My biggest gripe with rdiff-backup is that backing up large files like VirtualBox disk images takes very long. Since there is no progress indication, and I do backups manually (when I see a need for it), this can be a bit frustrating. But I haven't experienced any problems with it like johnP writes, and the resulting increment files appear to be reasonably small.


#10 Eliot commented on 2009-07-08:

Jakob, thanks for adding your experience also. Good to hear rdiff-backup makes you happy.


#11 koolnyze commented on 2009-10-09:

I have been using rdiff-backup for a long time and it seems to be working well.


#12 Maccy commented on 2010-01-11:

I have been using rdiff-backup for a couple of years now with just a few restore operations and so far i am lucky with it.

Recently i did a small trial on changing meta data of media files such as comments on jpeg's or ID3-tags on mp3 or OGG files. Since rdiff-backup is storing only diffs even on binary files, i expected the diffs to be tiny. However, the diffs i got so far are roughly halve the size of the original file when i changed a few characters.

The background for this is the question whether media data should have a rsync or a rdiff backup. Currently I'm doing rsync on my pictures (jpg) and audio files (ogg). My plan was to change this to rdiff, basically to be save against deletion of the files. This would be a perfect solution if rdiff-backup would find the real differences in the file, e.g. the changed matadata. Otherwise it will mess up my backup system with a lot of unnecessary data. So for the time beeing i will keep rsync for media data and rdiff-backup for all the other files.


#13 Leho Kraav commented on 2010-03-27:

hi thanks for the write-up. i think i'll be going rsnapshot for gentoo server backup strategy. is it possible to get an rss feed for the comments of this post?


#14 Eliot commented on 2010-03-27:

Leho Kraav: thanks, a comments feed is a good idea. i will try to add it this weekend.

UPDATE: I've created the comments feed. See above for the feed link.


#15 Leho Kraav commented on 2010-03-29:

is there a better way to test the feed than giving them kudos to the author!? i thinks not!


#16 Eliot commented on 2010-03-30:

To Leho or anyone else who subscribed to the Atom feed: Sorry about the spam comment. I forgot to filter the spam from the feed. It should be fixed now.


#17 Ben Maden commented on 2010-05-09:

I'd just finished setting up rsnapshot and then read about some glowing recommendations for rdiff-backup.

Thank you for your clear break down.. much time saved :) on what would have been a wild-goose-chase considering I just need a traditional backup set-up.


#18 Richard commented on 2010-05-20:

http://old.nabble.com/Differences-between-rsnapshot-and-rdiff-backup-td15282022.html

Did you write this too? This guy is referencing to the O'Reilly "Backup & Recovery" book.


#19 Eliot commented on 2010-05-21:

Richard: yes, that was my message to the mailing list, before I posted this on my blog. Yeah I forgot to include that I used that book as a reference. I updated the post above to include the book reference and links to the discussions on the two mailing lists.


#20 Harland commented on 2010-06-17:

I have been using rdiff-backup for over 3yrs. It works well for etc, var, and home and I have restored home files because of user error a number of times; it works well but dont get an average user to do it.

I came up with and use a hybrid solution with rsnapshot for these PITA cases. I use rdiff-backup to take daily snapshots on the same host to a different disk, which are then rsync'd to a server (I know rdiff-backup can do this, but trial and sweat have chosen).
I then use rsnapshot for hourly and daily snapshots and let the user have at it. When 'they' do something and lose their 'most needed' file, they just look into the rsnapshot 'copies' and pullout the one they need. If they have to go back further than a week (which almost never happens), then I can dig it out of rdiff-backup 'copies' and dump it on their host.
Yes it takes a little extra space to have the extra 'copy' but for ease of use, this saves (my) time/effort. And just for fun, I still do quarterly tarballs incase doom sets in =).

I would love an hybrid of these two excellent programs, but can python and perl really be friends... Whats really needed is filemanager plugin to display the snapshots' on a right click context menu for a given file or directory.


#21 Howard commented on 2010-06-28:

Thanks for the excellent writeup. Whilst habitually following bu solutions, I find it pretty confusing re: pro/con of the multiple solutions.

Pretty well settled on rdiff-backup as I run five o/s on the same box and chrono backup is not very meaningful for me.

I would note that newbies to command line backup, probably want to prefix their command with 'time' to obtain elapsed data. Also, you really need to figure out the right parms for backup AND (test it !) RESTORE, then incorporate them into a script. Else, a great chance you will forget the parms ! One other recommendation: do a bare metal backup with something like clonezilla. You do not need to change it unless you modify your partitions, but great peace of mind having redundancy to recover from a disaster !

Howard addl argument for CLI vs GUI: I have had good luck with ctl-c to KILL cli backups, not so good luck with gui (Lucky Backup one of the exceptions).


#22 Travis H. commented on 2010-07-17:

Hey there,

I'm working on a disk-to-disk backup utility, called hdb (hard drive backup):

http://www.subspacefield.org/security/hdb/

I link to a variety of related tools, including duplicity, which may interest you.

I'm a security weenie, and you can tell from my choice of technologies (SHA-512 hashes and encrypted backup media).

If you're interested at all in high-speed disk-to-disk backups, then you might want to check out the tool and mlist. It's in alpha right now - so you can easily make suggestions that will affect its fate - but progressing rapidly.

Written 100% in pure ruby... no modules, nothing to install, just plop the script in ~/bin and you're done.


#23 ændrük commented on 2010-08-12:

Thanks for the explanation. I've heard several recommendations for both of these and never was clear on their differences.

Post a comment
Required
Required, but not displayed
Optional

Format using Markdown. (No HTML.)
  • Code blocks: prefix each line by at least 4 spaces or 1 tab (and a blank line before and after)
  • Code span: surround with backticks
  • Blockquotes: prefix lines to be quoted with >
  • Links: <URL>
  • Links w/ description: [description](URL)
Created with Django | Hosted by Slicehost