Saltycrane logo

SaltyCrane Blog

Notes on Python, Django, and web development on Ubuntu Linux

    

Backup on Linux: rsnapshot vs. rdiff-backup (vs. Time Machine)

Date: 2008-02-07  |   Tags: linux, rsync, softwaretools, ubuntu   |   33 Comments

Apple's release of Leopard and the included backup utility, Time Machine, has generated a lot of talk about backups recently. I will admit Time Machine is pretty cool and believe that it is a bit more than a glorified GUI on top of an existing *nix tool as some have claimed. However, the core functionality is very similar to the command-line tool, rsnapshot, which is itself based on a rsync script by Mike Rubel. Time Machine added a couple of features and a GUI to make it easy to use. Since I prefer the command line over GUIs most of the time anyways, rsnapshot seemed perfect for me.

To be complete, I researched a number of other backup utilities for Linux. Dirvish and flyback were out because I prefer the command line and they didn't seem to offer anything more than rsnapshot. Scripting rsync wouldn't get me anything more than rsnapshot either, plus it would be more work. In the end, I eliminated all but rsnapshot and another command line tool called rdiff-backup. Rdiff-backup has some advantages over rsnapshot (and Time Machine) because it stores compressed deltas of each version of a file instead of a complete copy of the file each time it changes. This is not a big deal for small files, but for large files that change often, it makes a significant difference. However, the big disadvantage of rdiff-backup, for me, was the inablility to perform different levels of backup, such as hourly, daily, weekly, monthly, etc. Depending on the needs of the user, this could negate the space saving advantage by needing to keep a large number of snapshots.

I ended up choosing rsnapshot over rdiff-backup because of this last reason. It seems rdiff-backup is closer to a version control tool and rsnapshot closer to a traditional backup solution. It would be great to create a hybrid of the two tools to gain the advantages of each. I started to dig into the source of rdiff-backup (python source makes me happy), but I didn't want to get too sidetracked with another project. For now, I am using Mercurial to version control my /etc and /home/.* config files and rsnapshot as a broader, general purpose backup tool.


Here is my comparison of rsnapshot and rdiff-backup:

Similar Features
  • both use an rsync-like algorithm to transfer data (rsnapshot actually uses rsync; rdiff-backup uses the python librsync library)
  • both can be used over ssh (though rsnapshot cannot push over ssh without some extra scripting)
  • both use a simple copy of the source for the current backup
Written in rsnapshot is written in Perl; rdiff-backup is written in Python and C
Size rdiff-backup stores previous versions as compressed deltas to the current version similar to a version control system. rsnapshot uses actual files and hardlinks to save space. For small files, storage size is similar. For large files that change often, such as logfiles, databases, etc., rdiff-backup requires significantly less space for a given number of versions.
Speed rdiff-backup is slower than rsnapshot
Metadata rdiff-backup stores file metadata, such as ownership, permissions, and dates, separately.
Transparency For rsnapshot, all versions of the backup are accessible as plain files. For rdiff-backup, only the current backup is accessible as plain files. Previous versions are stored as rdiff deltas.
Backup levels rsnapshot supports multiple levels of backup such as monthly, weekly, and daily. rdiff-backup can only delete snapshots earlier than a given date; it cannot delete snapshots in between two dates.
Community Based on the number of responses to my post on the mailing lists (rsnapshot: 6, rdiff-backup: 0), rsnapshot has a more active community.

Reference: Chapter 7 in Backup & Recovery O'Reilly 2007

33 Comments — feed icon Comments feed for this post


#1 yungchin commented on 2008-02-27:

Hi, nice post. I'd be interested to read more about how you've set up version control on your home dir (could make a nice blog post? :)).

I haven't figured out a manageable way yet (using bzr) - there are always lots of new files appearing that have to go to your ignore-list, or something like that...

Thanks! YC


#2 Eliot commented on 2008-03-26:

yungchin,
thanks. i am still refining the process of how to use the version control. i plan to post about it sometime soon. please let me know what solution you go with as well.


#3 manuel quinta commented on 2008-10-17:

I use rdiff-backup since years to make backups over ssh, and i'am happy with it.

we used rsnapshot on our server, to make a backup over ssh. but it took a lot of cpu? also we did a crazy missconfiguration, as we wanted to stop the hourly backup, because of the cpu load. wenn you suspend the hourly backup, it will not make a daily... it just stoped to make any backup!

we did only see it by incident a half a year later, but we did not lose any data! luckely!


#4 Eliot commented on 2008-10-17:

manuel, Thanks for the notes on your experiences.


#5 Chris Wilper commented on 2008-12-01:

Note on rdiff-backup, for transparency, there is a FUSE implementation called archfs that allows you to mount all snapshots in a read-only filesystem. It's a little rough around the edges, but I've been impressed now that I've got it to work.

It occurs to me that it should be possible to run rdiff-backup against the fuse-exposed snapshots, in order, skipping the ones you don't want. Haven't tried it, but it would be an interesting experiment...


#6 Eliot commented on 2008-12-08:

Chris, thanks a lot for the information. Please feel free to leave a link to your notes if you've posted any.


#7 johnP commented on 2009-06-18:

We've been using rdiff-backup for a few months. Our requirements are for a cross platform, cheap, solution, so anything that required hard-link support couldn't be considered. Personally, I'd wipe all the MS-windows server machines, but our customers would be mad. We're mostly *NIX for production systems, with a few Windows dev VMs.

In that time, we've recovered from "oops" moments twice. Once was during an email system upgrade that couldn't be completed and the other was while screwing with an Alfresco system permissions model that failed in the end. Backups rock, but folks here already know that.

Has anyone solved the remote Windows backup issue in a good free way that can actually be restored without loss of user and group permissions?

rdiff-backup is mostly good, but there are a few problems. Large file differencing doesn't work in our experience. If it doesn't crash, you'll get a completely new copy. We tried backing up complete Xen image files this way. We got around this issue by mounting the IMG files on the host and pushing the rdiff-backup to another system on the same LAN. Complete system backups are 2-3 minutes now.

That works for Xen, but not VirtualBox with a Vista-64 host. Any ideas?


#8 Eliot commented on 2009-06-22:

johnP, Thank you for adding your experience with rdiff-backup. I'm sorry I can't be of any to help to you. Maybe another visitor will have ideas.


#9 Jakob Malm commented on 2009-07-07:

I've been using rdiff-backup for my home system since March 2008 and am overall happy with it. I've had to restore single files a few times, and though that procedure is not very straightforward, it's been easy enough to figure out how to do it the few times I've needed it. The FUSE file system seems interesting.

My biggest gripe with rdiff-backup is that backing up large files like VirtualBox disk images takes very long. Since there is no progress indication, and I do backups manually (when I see a need for it), this can be a bit frustrating. But I haven't experienced any problems with it like johnP writes, and the resulting increment files appear to be reasonably small.


#10 Eliot commented on 2009-07-08:

Jakob, thanks for adding your experience also. Good to hear rdiff-backup makes you happy.


#11 koolnyze commented on 2009-10-09:

I have been using rdiff-backup for a long time and it seems to be working well.


#12 Maccy commented on 2010-01-11:

I have been using rdiff-backup for a couple of years now with just a few restore operations and so far i am lucky with it.

Recently i did a small trial on changing meta data of media files such as comments on jpeg's or ID3-tags on mp3 or OGG files. Since rdiff-backup is storing only diffs even on binary files, i expected the diffs to be tiny. However, the diffs i got so far are roughly halve the size of the original file when i changed a few characters.

The background for this is the question whether media data should have a rsync or a rdiff backup. Currently I'm doing rsync on my pictures (jpg) and audio files (ogg). My plan was to change this to rdiff, basically to be save against deletion of the files. This would be a perfect solution if rdiff-backup would find the real differences in the file, e.g. the changed matadata. Otherwise it will mess up my backup system with a lot of unnecessary data. So for the time beeing i will keep rsync for media data and rdiff-backup for all the other files.


#13 Leho Kraav commented on 2010-03-27:

hi thanks for the write-up. i think i'll be going rsnapshot for gentoo server backup strategy. is it possible to get an rss feed for the comments of this post?


#14 Eliot commented on 2010-03-27:

Leho Kraav: thanks, a comments feed is a good idea. i will try to add it this weekend.

UPDATE: I've created the comments feed. See above for the feed link.


#15 Leho Kraav commented on 2010-03-29:

is there a better way to test the feed than giving them kudos to the author!? i thinks not!


#16 Eliot commented on 2010-03-30:

To Leho or anyone else who subscribed to the Atom feed: Sorry about the spam comment. I forgot to filter the spam from the feed. It should be fixed now.


#17 Ben Maden commented on 2010-05-09:

I'd just finished setting up rsnapshot and then read about some glowing recommendations for rdiff-backup.

Thank you for your clear break down.. much time saved :) on what would have been a wild-goose-chase considering I just need a traditional backup set-up.


#18 Richard commented on 2010-05-20:

http://old.nabble.com/Differences-between-rsnapshot-and-rdiff-backup-td15282022.html

Did you write this too? This guy is referencing to the O'Reilly "Backup & Recovery" book.


#19 Eliot commented on 2010-05-21:

Richard: yes, that was my message to the mailing list, before I posted this on my blog. Yeah I forgot to include that I used that book as a reference. I updated the post above to include the book reference and links to the discussions on the two mailing lists.


#20 Harland commented on 2010-06-17:

I have been using rdiff-backup for over 3yrs. It works well for etc, var, and home and I have restored home files because of user error a number of times; it works well but dont get an average user to do it.

I came up with and use a hybrid solution with rsnapshot for these PITA cases. I use rdiff-backup to take daily snapshots on the same host to a different disk, which are then rsync'd to a server (I know rdiff-backup can do this, but trial and sweat have chosen).
I then use rsnapshot for hourly and daily snapshots and let the user have at it. When 'they' do something and lose their 'most needed' file, they just look into the rsnapshot 'copies' and pullout the one they need. If they have to go back further than a week (which almost never happens), then I can dig it out of rdiff-backup 'copies' and dump it on their host.
Yes it takes a little extra space to have the extra 'copy' but for ease of use, this saves (my) time/effort. And just for fun, I still do quarterly tarballs incase doom sets in =).

I would love an hybrid of these two excellent programs, but can python and perl really be friends... Whats really needed is filemanager plugin to display the snapshots' on a right click context menu for a given file or directory.


#21 Howard commented on 2010-06-28:

Thanks for the excellent writeup. Whilst habitually following bu solutions, I find it pretty confusing re: pro/con of the multiple solutions.

Pretty well settled on rdiff-backup as I run five o/s on the same box and chrono backup is not very meaningful for me.

I would note that newbies to command line backup, probably want to prefix their command with 'time' to obtain elapsed data. Also, you really need to figure out the right parms for backup AND (test it !) RESTORE, then incorporate them into a script. Else, a great chance you will forget the parms ! One other recommendation: do a bare metal backup with something like clonezilla. You do not need to change it unless you modify your partitions, but great peace of mind having redundancy to recover from a disaster !

Howard addl argument for CLI vs GUI: I have had good luck with ctl-c to KILL cli backups, not so good luck with gui (Lucky Backup one of the exceptions).


#22 Travis H. commented on 2010-07-17:

Hey there,

I'm working on a disk-to-disk backup utility, called hdb (hard drive backup):

http://www.subspacefield.org/security/hdb/

I link to a variety of related tools, including duplicity, which may interest you.

I'm a security weenie, and you can tell from my choice of technologies (SHA-512 hashes and encrypted backup media).

If you're interested at all in high-speed disk-to-disk backups, then you might want to check out the tool and mlist. It's in alpha right now - so you can easily make suggestions that will affect its fate - but progressing rapidly.

Written 100% in pure ruby... no modules, nothing to install, just plop the script in ~/bin and you're done.


#23 ændrük commented on 2010-08-12:

Thanks for the explanation. I've heard several recommendations for both of these and never was clear on their differences.


#24 JohnP commented on 2010-10-24:

Has anyone looked at bigsync (C http://code.google.com/p/bigsync/) or BigSync (perl http://stefan.hoefer.ch/projects/bigsync/5-bigsync-an-overview) or xdelta (http://freshmeat.net/projects/xdelta/) for efficient large file backups?

I'm still looking for a cross platform, script-able way to deal with virtual machine image files.

We're still using rdiff-backup here. Complete VM backups only take about 2-3 minutes per day, but changes in our VM technology is forcing a revisit. I'm doing performance testing of some alternate solutions now and will post results to my blog when they are complete.


#25 Steve Robbins commented on 2010-11-06:

Nice comparison. After reading your blog I think I'm much more inclined right now to using rdiff-backup as my preferred backup software


#26 JohnP commented on 2011-01-12:

For folks interested in a GUI, Back-In-Time seems to be rsnapshot with a GUI. Deployed this on Mom's Linux machine and it is working very nicely with hourly, daily, weekly and monthly snapshots. Nice if you are limited to *NIX file systems. The automatic "smart" snapshot management of Back-In-Time is nice.

Duplicati is a GUI for Duplicity. I found Duplicati to be extremely slow for both Full and incremental backups. That's an understatement, actually. 8.5 hours for a 100GB backup seems excessive to me.

The only GUI I've seen for rdif-backup is a web server-based solution. Ew.

And in limited tests, rdiff-backup on Win64 to Win64 (both truecrypted disk partitions) of 18GB virtual machines does only store the differences involved. Truecrypt isn't really important to the tests, since both partitions were mounted to drive letters during the test.

Anyway, I wrote up my experiences with Duplicati vs Rdiff-backup in a blog entry.


#27 Michael commented on 2011-01-15:

I would like the hard link-based implementation of rsnapshot for independent backups that would be simpler to recover. However, I ended up rejecting it because it is based on the seriously flawed include/exclude rules of rsync (sensitivity to trailing '/'s, etc, IIRC) which are confusing, easy to get wrong, and often difficult to get right, even when you think you understand it. For compatibility reasons, this design flaw in rsync will never be fixed.

In contrast, the author of rdiff-backup, using librsync rather than rsync itself, was free to implement a different, more straightforward approach to specifying files to be included or excluded, and did so.

This was for me a deciding factor, and I think it deserves a place in your list.

The only minor gripe I have with rdiff-backup is that it's rather slow, but that's not a major issue if you can leave it running, e.g., at lunchtime.


#28 Michael commented on 2011-01-15:

As an update on the last post, I just noticed that another Mike made the same point about ease of use at http://www.mail-archive.com/rdiff-backup-users@nongnu.org/msg02917.html

"Unlike rsync, rdiff-backup was written originally for backups. It has sensible defaults (so no need for the -av –-delete -e > ssh options) and fewer quirks (for instance, no distinction between <destination>, <destination>/, and <destination>/.)."

A subsequent commenter pointed out the same approach I ended up using to cope with the fact that intermediate rdiff-backups could not be deleted (i.e. you have to keep all the incremental backups back to the oldest one you need to keep): use separate repositories for short- and long-term backups.

I would still prefer an rdiff-backup implemented using hard links, but would not use rbackup and its crufty, embrangled rsync interface.


#29 Aaron Boyd commented on 2011-12-21:

Great summary -- thank you. Had I found your writeup earlier, I would have made my choice faster. I came to the same conclusion for the same reasons and am liking rsnapshot. rdiff-backup seemed better in most ways, but the inability to preserve my oldest backups (when space becomes tight) was a deal breaker.


#30 leobard commented on 2012-02-26:

hey, I just evaluate buying a new linux fileserver for home. I am using dirvish since years and it works fine for me. Your observations about dirvish ring a bell - its a useful and well-done tool and easygoing but a bit awkward to setup. The new hardware options I evaluate (synology ds212j or netgear readynas duo) seem to slightly play better with rsnapshot.

conclusion: your post showed up in google on firstpage for "dirvish rsnapshot" and your summary was an excellent thinking I can build upon! thanks!

I will go for rsnapshot.

(and for home server hardware: __synology. there are cookbooks to get rsnapshot running, but as its just a perl script piggybacking rsync, it should run on anything linux. I was looking for a NAS and backup box for less than 200$ and it has 15Watt power drain with a fairly decent linux on top, so synology floats my boat)


#31 nogi commented on 2012-08-07:

and "storebackup"? anyone use it?


#32 Pykler commented on 2013-01-19:

I use rsnapshot too and I love it. I do like the fact that rdiff-backup makes deltas of files and I am thinking in the near future I might implement both (since I like the daily, monthly, etc ...).

My idea is:

rdiff-backup to pull in the files, and do an rsnapshot on the output of the rdiff-backup directory.

deploying these backups back out in the case of disaster recovery might be a small pain ... but not too hard for a small shell script wrapper to handle.

Thanks for the cool review.


#33 Not a name commented on 2013-01-31:

I ran rdiff-backup for years, but the requirement that both the server and the client have to run compatible versions of rdiff-backup proved to be just too frustrating.

I have moved to rsnapshot because it has no such requirement. I have also come to prefer the very transparent hard-link approach that rsnapshot uses.

Post a comment
Required
Required, but not displayed
Optional

Format using Markdown. (No HTML.)
  • Code blocks: prefix each line by at least 4 spaces or 1 tab (and a blank line before and after)
  • Code span: surround with backticks
  • Blockquotes: prefix lines to be quoted with >
  • Links: <URL>
  • Links w/ description: [description](URL)
Created with Django | Hosted by Linode