SaltyCrane: rsynchttps://www.saltycrane.com/blog/2008-11-20T15:01:35-08:00Creating remote server nicknames with .ssh/config
2008-11-20T15:01:35-08:00https://www.saltycrane.com/blog/2008/11/creating-remote-server-nicknames-sshconfig/<p>Using the <code>~/.ssh/config</code> file is an easy way to give your
remote machines nicknames and reduce the number of keystrokes needed
to login with <code>ssh</code>, <code>rsync</code>,
<code>hg push</code>/<code>pull</code>/<code>clone</code>, access
files via <a href="http://www.gnu.org/software/tramp/">Emacs Tramp</a>
(Transparent Remote (file) Access, Multiple Protocol),
or use any other SSH-based tool. You can also set other ssh options such
as <code>IdentityFile</code>, <code>Port</code>, or <code>CompressionLevel</code>.
For more information and a full list of options, check out the man page for <code>ssh_config</code> or
<a href="http://kimmo.suominen.com/docs/ssh/#config">this article</a>
by Kimmo Suominen.
</p>
<p>Here is part of my <code>~/.ssh/config</code> file. It defines the
nicknames turk, tyran, tuna, and tally for some EC2 servers I've been
working with.</p>
<pre>Host turk
User root
HostName ec2-67-202-21-122.compute-1.amazonaws.com
Host tuna
User root
HostName ec2-75-101-178-62.compute-1.amazonaws.com
Host tyran
User root
HostName ec2-67-202-43-207.compute-1.amazonaws.com
Host tally
User root
HostName ec2-67-202-59-207.compute-1.amazonaws.com</pre>
<p>Now, wherever I would normally have typed
<code>root@ec2-67-202-21-122.compute-1.amazonaws.com</code>,
I can just type <code>turk</code>. Here are some examples.</p>
<h4>SSH login</h4>
<p>Old way:</p>
<pre>ssh root@ec2-67-202-21-122.compute-1.amazonaws.com</pre>
<p>New way:</p>
<pre>ssh turk</pre>
<h4>rsync</h4>
<p>Old way:</p>
<pre>rsync -avz myproject root@ec2-67-202-21-122.compute-1.amazonaws.com:/srv</pre>
<p>New way:</p>
<pre>rsync -avz myproject turk:/srv</pre>
<h4>Mercurial</h4>
<p>Old way:</p>
<pre>hg push ssh://root@ec2-67-202-21-122.compute-1.amazonaws.com//srv/myproject</pre>
<p>New way:</p>
<pre>hg push ssh://turk//srv/myproject</pre>
<h4>Emacs Tramp</h4>
<p>To use your <code>~/.ssh/config</code> with Emacs Tramp, you will need
something like the following in your <code>.emacs</code>:</p>
<pre>(tramp-set-completion-function "ssh"
'((tramp-parse-sconfig "/etc/ssh_config")
(tramp-parse-sconfig "~/.ssh/config")))</pre>
<p>Old way:</p>
<pre>C-x C-f /root@ec2-67-202-21-122.compute-1.amazonaws.com:/srv/myproject/myfile.py</pre>
<p>New way:</p>
<pre>C-x C-f /turk:/srv/myproject/myfile.py</pre>
<h4>scp</h4>
<p>Old way:</p>
<pre>scp etc/.screenrc root@ec2-67-202-21-122.compute-1.amazonaws.com:/root</pre>
<p>New way:</p>
<pre>scp etc/.screenrc turk:/root</pre>
Backup on Linux: rsnapshot vs. rdiff-backup (vs. Time Machine)
2008-02-07T13:22:00-08:00https://www.saltycrane.com/blog/2008/02/backup-on-linux-rsnapshot-vs-rdiff/<p>Apple's release of Leopard and the included backup utility,
<a href="http://www.apple.com/macosx/features/timemachine.html">
Time Machine</a>, has generated a lot of talk about backups recently. I will
admit Time Machine is pretty cool and believe that it is a bit more than
a glorified GUI on top of an existing *nix tool as some have claimed.
However, the core functionality is very similar to the command-line tool,
<a href="http://www.rsnapshot.org/">rsnapshot</a>, which is itself
based on a <a href="http://samba.org/rsync/">rsync</a> script by Mike
Rubel. Time Machine added a couple of features and a GUI to make it
easy to use. Since I prefer the command line over GUIs most of the time
anyways, rsnapshot seemed perfect for me.</p>
<p>To be complete, I researched a number of other backup utilities for Linux.
<a href="http://www.dirvish.org/">
Dirvish</a> and <a href="http://code.google.com/p/flyback/">flyback</a>
were out because I prefer the command line and they didn't seem to offer
anything more than rsnapshot. Scripting rsync wouldn't get me anything
more than rsnapshot either, plus it would be more work. In the end,
I eliminated all but rsnapshot and another command line tool called
<a href="http://www.nongnu.org/rdiff-backup/">
rdiff-backup</a>. Rdiff-backup has some advantages over rsnapshot (and Time
Machine) because it stores compressed deltas of each version of a file
instead of a complete copy of the file each time it changes. This is not
a big deal for small files, but for large files that change often, it
makes a significant
difference. However, the big disadvantage of rdiff-backup, for me, was
the inablility to perform different levels of backup, such as hourly,
daily, weekly, monthly, etc. Depending on the needs of the user, this
could negate the space saving advantage by needing to keep a large number
of snapshots.</p>
<p>I ended up choosing rsnapshot over rdiff-backup because of this last
reason. It seems rdiff-backup is closer to a version control tool and
rsnapshot closer to a traditional backup solution. It would be great to
create a hybrid of the two tools to gain
the advantages of each. I started to dig into the source of rdiff-backup
(python source makes me happy), but I didn't want to get too sidetracked
with another project. For now, I am using <a href="http://www.selenic.com/mercurial/">
Mercurial</a> to version control
my <code>/etc</code> and <code>/home/.*</code> config files and rsnapshot
as a broader, general purpose backup tool.</p>
<br /><p>Here is my comparison of rsnapshot and rdiff-backup:</p>
<table>
<tr>
<td valign="top">Similar Features</td>
<td valign="top">
<ul>
<li>both use an rsync-like algorithm to transfer data (rsnapshot
actually uses rsync; rdiff-backup uses the python librsync library)
</li>
<li>both can be used over ssh (though rsnapshot cannot push over ssh without some extra scripting)</li>
<li>both use a simple copy of the source for the current backup</li>
</ul>
</td>
</tr>
<tr>
<td valign="top">Written in</td>
<td valign="top">rsnapshot is written in Perl; rdiff-backup is written in Python and C</td>
</tr>
<tr>
<td valign="top">Size</td>
<td valign="top">rdiff-backup stores previous versions
as compressed deltas to the current version similar to a version
control system. rsnapshot uses actual files and hardlinks to save
space. For small files, storage size is similar. For large files
that change often, such as logfiles, databases, etc., rdiff-backup
requires significantly less space for a given number of versions.</td>
</tr>
<tr>
<td valign="top">Speed</td>
<td valign="top">rdiff-backup is slower than rsnapshot</td>
</tr>
<tr>
<td valign="top">Metadata</td>
<td valign="top">rdiff-backup stores file metadata, such as ownership, permissions, and
dates, separately.</td>
</tr>
<tr>
<td valign="top">Transparency</td>
<td valign="top">For rsnapshot, all versions of the backup are accessible as plain files.
For rdiff-backup, only the current backup is accessible as plain files.
Previous versions are stored as rdiff deltas.</td>
</tr>
<tr>
<td valign="top">Backup levels</td>
<td valign="top">rsnapshot supports multiple levels of backup such as monthly, weekly,
and daily. rdiff-backup can only delete snapshots earlier than a given
date; it cannot delete snapshots in between two dates.</td>
</tr>
<tr>
<td valign="top">Community</td>
<td valign="top">
Based on the number of responses to my post on the mailing lists
(<a href="https://sourceforge.net/p/rsnapshot/mailman/rsnapshot-discuss/thread/592a1f150802041321y61b3a2a3j567f1e62e4cca594%40mail.gmail.com/#msg18526014">rsnapshot: 6</a>,
<a href="http://lists.nongnu.org/archive/html/rdiff-backup-users/2008-02/msg00008.html">
rdiff-backup: 0</a>), rsnapshot has a more active
community.
</td>
</tr>
</table>
<p>Reference: <a href="http://search.safaribooksonline.com/0596102461/I11707__ChapterStart__Chapter_7">
Chapter 7 in <em>Backup & Recovery</em> O'Reilly 2007</a></p>
backing up with rsync
2007-08-31T13:55:00-07:00https://www.saltycrane.com/blog/2007/08/backing-up-with-rsync/<p>
Here is a python script using rsync to backup my Users directory in
Vista to an external hard drive.</p>
<pre class="python">import os
cmd = "rsync -avz --exclude '/AppData/'" + \
"/cygdrive/c/Users/saltycrane" + \
"/cygdrive/f/backup/Users"
os.system(cmd)</pre>
<h4 id="why-not-proprietary-backup">Why not to use the backup program that comes with your external hard drive:</h4>
<blockquote>Do not, whatever you do, feed your valuable data to a program that is going to save it in a file format that can only be read by that program, or by
that kind of computer. Because when the program can’t or the computer can’t,
<em>you’re out of options</em>.
--<a href="">Tim Bray</a>, <a href="http://www.tbray.org/ongoing/When/200x/2006/01/31/Data-Protection#p-5"><em>Protecting Your Data</em></a>
</blockquote>
<h4 id="why-rsync">Why to use rsync:</h4>
<p><kbd>rsync -essh -rtpvz</kbd> rocks.
--<a href="http://diveintomark.org/">Mark Pilgrim</a>,
<a href="http://diveintomark.org/archives/2004/05/01/essentials"><em>Essentials</em></a>
</p>
<h4 id="trailing-slashes">On trailing slashes</h4>
<p>from the rsync man page:
<blockquote> A trailing slash on the source changes this behavior to avoid creating an addi‐
tional directory level at the destination. You can think of a trailing / on a
source as meaning “copy the contents of this directory” as opposed to “copy the
directory by name”, but in both cases the attributes of the containing directory
are transferred to the containing directory on the destination. In other words,
each of the following commands copies the files in the same way, including their
setting of the attributes of /dest/foo:<br>
<br>
rsync -av /src/foo /dest<br>
rsync -av /src/foo/ /dest/foo<br>
</blockquote></p>
<h4 id="rsync-sudo">Use rsync with sudo on the remote host <em>(added 2011-09-07)</em></h4>
<p>Use the <code>--rsync-path</code> option. e.g.</p>
<pre class="console">$ rsync -avz --delete --rsync-path="sudo rsync" /tmp/something myhost:/some/path </pre>