Nikolay pointed me at beanstalkd, a lightweight, message queue service partially inspired by the popular caching system, memcached. It features a blocking "reserve" call so workers don't need to poll for new jobs. However, some might miss the data persistence since the queue is stored in memory. beanstalkd has client libraries in Erlang, Perl, PHP, Python, and Ruby. Of course, I will use the Python version, pybeanstalk. Simon Willison also found beanstalkd interesting. (Credit to him for some of the words above.)

Update 2008-10-13: For a helpful example on using beanstalkd and pybeanstalk see Parand's beanstalkd tutorial.

Here is how I installed the beanstalkd server and pybeanstalk client on Ubuntu. I found no Ubuntu package for beanstalkd, so I installed from source.

Install beanstalkd

• Install prerequistes

  $ sudo apt-get install libevent1 libevent-dev

• Download

  $ cd ~/lib
  $ wget http://xph.us/software/beanstalkd/rel/beanstalkd-1.0.tar.gz

• Unpack

  $ tar zxvf beanstalkd-1.0.tar.gz

• Make

  $ cd beanstalkd-1.0
  $ make

• Print help

  $ ./beanstalkd -h

Install PyYAML

• Download

  $ wget http://pyyaml.org/download/pyyaml/PyYAML-3.06.tar.gz

• Unpack

  $ tar zxvf PyYAML-3.06.tar.gz

• Put PyYAML-3.06/lib/yaml somewhere on your Python path or run python setup.py.

Install pybeanstalk

• Download

  $ wget http://pybeanstalk.googlecode.com/files/pybeanstalk-0.11.1.tar.gz

• Unpack

  $ tar zxvf pybeanstalk-0.11.1.tar.gz

• Put pybeanstalk-0.11.1/beanstalk somewhere on your Python path or run python setup.py

Run beanstalkd server

• $ ~/lib/beanstalkd-1.0/beanstalkd -d -l 127.0.0.5 -p 11300

Run test client

• Create a file and run it:

  from beanstalk import serverconn
  from beanstalk import job
  
  SERVER = '127.0.0.5'
  PORT = 11300
  
  # setup connection
  connection = serverconn.ServerConn(SERVER, PORT)
  connection.job = job.Job
  
  # produce data
  for i in range(5):
      print 'put data: %d' % i
      data = job.Job(data=str(i), conn=connection)
      data.Queue()
  
  # consume data
  while True:
      j = connection.reserve()
      print 'got data: %s' % j.data
      j.Finish()
  

  Results:

  put data: 0
  put data: 1
  put data: 2
  put data: 3
  put data: 4
  got data: 0
  got data: 1
  got data: 2
  got data: 3
  got data: 4

Python 2.6 was released yesterday! This version aims to smooth the transition from Python 2.5 to Python 3.0 which is planned for release soon (currently available as a release candidate). Python 3.0 will be break backwards compatibility with the 2.x series. Python 2.6 is backwards compatible with 2.5. All the backwards compatible features of 3.0 have been backported to 2.6.

One of the new 2.6 features I'm particularly intersted in is the new multiprocessing module which has a similar interface to the threading module, but it uses processes instead of threads. This avoids the limitations imposed by the Global Interpreter Lock in a multi-threaded Python program. Yet it still has the nice communications and management features like Pipe()s, Queues, Pools, etc. I didn't plan to focus so much on the multiprocessing module in this post-- I just want to document my install notes on Ubuntu Linux. For all the new features in 2.6 see What's New in Python 2.6. (It is a long list).

• Download the Python 2.6 compressed source tarball

  $ cd incoming
  $ wget http://www.python.org/ftp/python/2.6/Python-2.6.tgz

• Unpack

  $ tar zxvf Python-2.6.tgz

• Read the README at ~/incoming/Python-2.6/README
  
  
• Install prerequisites (Disclaimer: I know nothing about libraries, packages, dependencies, etc. This is what I did-- I am not sure if I grabbed the correct prerequisites or not.)

  $ sudo apt-get install build-essential
  $ sudo apt-get install libncursesw5-dev
  $ sudo apt-get install libreadline5-dev
  $ sudo apt-get install libssl-dev
  $ sudo apt-get install libgdbm-dev
  $ sudo apt-get install libbz2-dev
  $ sudo apt-get install libc6-dev
  $ sudo apt-get install libsqlite3-dev
  $ sudo apt-get install tk-dev

• Configure. I am installing to ~/lib/python2.6.

  $ cd Python-2.6
  $ ./configure --prefix=/home/sofeng/lib/python2.6

• Make

  $ make

  Note I got the following message:

  Failed to find the necessary bits to build these modules:
  bsddb185           sunaudiodev   
  To find the necessary bits, look in setup.py in detect_modules() for the module's name.

  I was not able to find the Ubuntu packages for these.
  
  
• Try it out (Optional)

  $ ./python

  Python 2.6 (r26:66714, Oct  2 2008, 15:32:46) 
  [GCC 4.2.3 (Ubuntu 4.2.3-2ubuntu7)] on linux2
  Type "help", "copyright", "credits" or "license" for more information.
  >>> 

• Test (Optional)

  $ make test

  I got the following status:

  327 tests OK.
  33 tests skipped:
      test_aepack test_al test_applesingle test_bsddb185 test_bsddb3
      test_cd test_cl test_codecmaps_cn test_codecmaps_hk
      test_codecmaps_jp test_codecmaps_kr test_codecmaps_tw test_curses
      test_gl test_imgfile test_kqueue test_linuxaudiodev test_macos
      test_macostools test_normalization test_ossaudiodev test_pep277
      test_py3kwarn test_scriptpackages test_socketserver test_startfile
      test_sunaudiodev test_timeout test_urllib2net test_urllibnet
      test_winreg test_winsound test_zipfile64
  Those skips are all expected on linux2.

• Install

  $ make install

• Finally, I added ~/lib/python/bin/python2.6 to my PATH

Notes

The first time I ran, I got this message:

Failed to find the necessary bits to build these modules:
_hashlib           _ssl               bsddb185
bz2                gdbm               readline
sunaudiodev
To find the necessary bits, look in setup.py in detect_modules() for the module's name.

I think I found Ubuntu packages for some of them:

• _hashlib and _ssl in libssl-dev
• bz2 in libbz2-dev
• gdbm in libgdbm-dev
• readline in libreadline5-dev

I have been using the emacs-snapshot package in Ubuntu Hardy. However, when I tried to use Tramp, I got an error message: Variable binding depth exceeds max-specpdl-size. I couldn't find out how to fix this, and I didn't want to use Emacs 22.3 because it doesn't have Xft (anti-aliased fonts), so I decided to live on the bleeding edge and install Emacs 23 from CVS. Besides the INSTALL and INSTALL.CVS files, I also used theBlackDragon's article for reference.

Update 2008-10-09: I just found that Romain Francoise maintains an emacs-snapshot Debian package and it has been adapted for Ubuntu. This is an alternative to installing from source.

Update 2010-01-09: Since Emacs has switched from CVS to Bazaar, I've updated these instructions to use Bazaar. Also, I'm now running on Ubuntu 9.10 Karmic Koala instead of Ubuntu Hardy.

Update 2012-09-19: Here are some additional Ubuntu 12.04 packages I needed to install Emacs 24.2.

$ sudo apt-get install libgif-dev
$ sudo apt-get install libtiff4-dev
$ sudo apt-get install xaw3dg-dev
$ sudo apt-get install librsvg2-dev
$ sudo apt-get install libmagick++-dev
$ sudo apt-get install libgpm-dev
$ sudo apt-get install libgconf2-dev
$ sudo apt-get install libselinux1-dev
$ sudo apt-get install libm17n-dev
$ sudo apt-get install libotf-dev 

• Install Bazaar

  $ sudo apt-get install bzr
  

• Get the source code from the Bazaar repository

  $ cd ~/incoming
  $ bzr branch http://bzr.savannah.gnu.org/r/emacs/trunk emacs_trunk
  

• Read the INSTALL and INSTALL.BZR files in ~/incoming/emacs_trunk

• Install prerequisites:

  $ sudo apt-get install build-essential
  $ sudo apt-get build-dep emacs23
  

  To see what is installed by build-dep, see the emacs23 karmic package page

• Configure. The argument --prefix=/home/saltycrane/lib/emacs-bzr-20100210 means I am installing Emacs in /home/saltycrane/lib/emacs-bzr-20100210.

  $ cd ~/incoming/emacs_trunk
  $ ./configure --prefix=/home/saltycrane/lib/emacs-bzr-20100210
  

• Per the INSTALL.BZR file, I needed to do a make bootstrap instead of make because some files, such as byte-compiled lisp files are not stored in Bazaar. Note, this takes a long time (over 10 min for me).

  $ make bootstrap
  

• Make (Optional)

  $ make
  

• Test it (Optional)

  $ src/emacs -q
  

• Install

  $ make install
  

• Create symlinks (~/bin is already on my PATH)

  $ ln -s ~/lib/emacs-bzr-20100210 ~/lib/emacs
  $ cd ~/bin
  $ ln -s ../lib/emacs/bin/* .
  

  Alternatively, I could add ~/lib/emacs-bzr-20100210/bin to my PATH.

Now I have "Pretty Emacs" with working Tramp for remote file access. It also has multi-tty support which is supposed to be very cool (but I haven't tried it yet) is very cool (e.g. for displaying my running desktop emacs process on my Android phone.)

Error messages

• configure: error: You do not seem to have makeinfo >= 4.6, and your
  source tree does not seem to have pre-built manuals in the `info' directory.
  Either install a suitable version of makeinfo, or re-run configure
  with the `--without-makeinfo' option to build without the manuals.
  

  Solution:

  $ sudo apt-get install texinfo
  

• configure: error: The following required libraries were not found:
       libjpeg libgif/libungif libtiff
  Maybe some development libraries/packages are missing?
  If you don't want to link with them give
       --with-jpeg=no --with-gif=no --with-tiff=no
  as options to configure
  

  Solution:

  $ sudo apt-get install libjpeg-dev libgif-dev libtiff4-dev
  

• Warning: arch-dependent data dir (/home/saltycrane/lib/emacs-bzr/libexec/emacs/23.1.92/i686-pc-linux-gnu/) does not exist.
  Warning: Lisp directory `/home/saltycrane/lib/emacs-bzr/share/emacs/23.1.92/site-lisp' does not exist.
  Warning: Lisp directory `/home/saltycrane/lib/emacs-bzr/share/emacs/site-lisp' does not exist.
  Warning: Lisp directory `/home/saltycrane/lib/emacs-bzr/share/emacs/23.1.92/lisp' does not exist.
  Warning: Lisp directory `/home/saltycrane/lib/emacs-bzr/share/emacs/23.1.92/leim' does not exist.
  Warning: Could not find simple.el nor simple.elc
  

  This happened because I originally used ./configure --prefix=/home/saltycrane/lib/emacs-bzr and then renamed ~/lib/emacs-bzr to ~/lib/emacs-bzr-20100210. Solution: don't rename the directory.

I found out about Fabric via Armin Ronacher's article Deploying Python Web Applications. Fabric is a Capistrano inspired deployment tool for the Python community. It is very simple to use. There are 4 main commands: local is almost like os.system because it runs a command on the local machine, run and sudo run a command on a remote machine as either a normal user or as root, and put transfers a file to a remote machine.

Here is a sample setup which displays information about the Apache processes on my remote EC2 instance.

• Install Easy Install
• Install Fabric

  $ sudo easy_install Fabric

• Create a file called fabfile.py located at ~/myproject

  def ec2():
      set(fab_hosts = ['ec2-65-234-55-183.compute-1.amazonaws.com'],
          fab_user = 'sofeng',
          fab_password = 'mypassword',)
  
  def ps_apache():
      run("ps -e -O rss,pcpu | grep apache")
  

  Note: for security reasons, you can remove the password from the fabfile and Fabric will prompt for it interactively. Per the documentation, Fabric also supports key-based authentication.
  
  
• Run it

  $ cd ~/myproject
  $ fab ec2 ps_apache

  Results:

     Fabric v. 0.0.9, Copyright (C) 2008 Christian Vest Hansen.
     Fabric comes with ABSOLUTELY NO WARRANTY; for details type `fab warranty'.
     This is free software, and you are welcome to redistribute it
     under certain conditions; type `fab license' for details.
  
  Running ec2...
  Running ps_apache...
  Logging into the following hosts as sofeng:
      ec2-65-234-55-183.compute-1.amazonaws.com
  [ec2-65-234-55-183.compute-1.amazonaws.com] run: ps -e -O rss,pcpu | grep apache
  [ec2-65-234-55-183.compute-1.amazonaws.com] out:  2163  5504  0.0 S ?        00:00:00 /usr/sbin/apache2 -k start
  [ec2-65-234-55-183.compute-1.amazonaws.com] out:  2520 15812  0.0 S ?        00:00:00 /usr/sbin/apache2 -k start
  [ec2-65-234-55-183.compute-1.amazonaws.com] out:  2521  3664  0.0 S ?        00:00:00 /usr/sbin/apache2 -k start
  [ec2-65-234-55-183.compute-1.amazonaws.com] out:  2522  3664  0.0 S ?        00:00:00 /usr/sbin/apache2 -k start
  [ec2-65-234-55-183.compute-1.amazonaws.com] out:  2523  3664  0.0 S ?        00:00:00 /usr/sbin/apache2 -k start
  [ec2-65-234-55-183.compute-1.amazonaws.com] out:  2524  3664  0.0 S ?        00:00:00 /usr/sbin/apache2 -k start
  [ec2-65-234-55-183.compute-1.amazonaws.com] out:  2619  3664  0.0 S ?        00:00:00 /usr/sbin/apache2 -k start
  [ec2-65-234-55-183.compute-1.amazonaws.com] out:  2629  1204  0.0 R ?        00:00:00 /bin/bash -l -c ps -e -O rss,pcpu | grep apache
  Done.

Create AWS accounts

• Create an AWS account at http://aws.amazon.com/.
• Create an AWS EC2 account at http://aws.amazon.com/ec2/. (You will need to enter a credit card number.)

Create a X.509 Certificate

Note: A X.509 Certificate is one type of Access Identifier. Access Identifiers are used to "identify yourself as the sender of a request to an AWS web service". There are two types of access identifiers: AWS Access Key Identifiers and X.509 Certificates. AWS Access Key Identifiers are supported by all Amazon Web Services and X.509 Certificates are supported only by Amazon's EC2 and SQS services (see here for the chart). However, for some reason, the popular Java command line tools for EC2 only support X.509 Certificates (and not AWS Access Key Identifiers).

• From Your Account page, select Access Identifiers.
• In the "X.509 Certificate" section, click "Create New".
• Download both the "Private Key" file and the "X.509 Certificate" file to the directory, ~/.ec2. (The private key file will be named something like pk-XXXXXXXXXXXXXXXXXXXXXX.pem and the X.509 Certificate file will be named something like cert-XXXXXXXXXXXXXXXXXXXXXX.pem.)

Install Java

The command line tools require Java version 5 or later. Only the JRE is required.

• $ sudo apt-get install sun-java6-jre

Download Java Command-line Tools

• Go to the Amazon EC2 Command-Line Tools library page, and Download the Amazon EC2 Command-Line Tools.
• Unzip the tools to ~/lib

  $ unzip ec2-api-tools.zip
  mv ec2-api-tools-1.3-24159 ~/lib

Define environment variables

• Add the following lines to your ~/.bashrc (or wherever you set your environment variables).

  export EC2_HOME=$HOME/lib/ec2-api-tools-1.3-24159
  export JAVA_HOME=/usr
  export EC2_PRIVATE_KEY=$HOME/.ec2/pk-XXXXXXXXXXXXXXXXXXXX.pem
  export EC2_CERT=$HOME/.ec2/cert-XXXXXXXXXXXXXXXXXXXX.pem
  export PATH=$PATH:$EC2_HOME/bin
  

• Source your .bashrc or whichever file you used

  $ source ~/.bashrc

Test the command-line tools

• Run the ec2-describe-images command to verify everything is working. It should list all the Ubuntu 8.xx images from Alestic.

  $ ec2-describe-images -a | grep alestic/ubuntu-8

  Results:

  IMAGE   ami-3a7c9953    alestic/ubuntu-8.04-hardy-base-20080419.manifest.xml    063491364108    available       public          i386    machine aki-a71cf9ce    ari-a51cf9cc
  IMAGE   ami-75789d1c    alestic/ubuntu-8.04-hardy-base-20080424.manifest.xml    063491364108    available       public          i386    machine aki-a71cf9ce    ari-a51cf9cc
  IMAGE   ami-ce44a1a7    alestic/ubuntu-8.04-hardy-base-20080430.manifest.xml    063491364108    available       public          i386    machine aki-a71cf9ce    ari-a51cf9cc
  IMAGE   ami-2048ad49    alestic/ubuntu-8.04-hardy-base-20080514.manifest.xml    063491364108    available       public          i386    machine aki-a71cf9ce    ari-a51cf9cc
  IMAGE   ami-6a57b203    alestic/ubuntu-8.04-hardy-base-20080517.manifest.xml    063491364108    available       public          i386    machine aki-a71cf9ce    ari-a51cf9cc
  IMAGE   ami-26bc584f    alestic/ubuntu-8.04-hardy-base-20080628.manifest.xml    063491364108    available       public          i386    machine aki-a71cf9ce    ari-a51cf9cc
  IMAGE   ami-179e7a7e    alestic/ubuntu-8.04-hardy-base-20080803.manifest.xml    063491364108    available       public          i386    machine aki-a71cf9ce    ari-a51cf9cc
  IMAGE   ami-c0fa1ea9    alestic/ubuntu-8.04-hardy-base-20080905.manifest.xml    063491364108    available       public          i386    machine aki-a71cf9ce    ari-a51cf9cc
  IMAGE   ami-38d43051    alestic/ubuntu-8.04-hardy-base-20080922.manifest.xml    063491364108    available       public          i386    machine aki-a71cf9ce    ari-a51cf9cc
  IMAGE   ami-1cd73375    alestic/ubuntu-8.04-hardy-base-20080924.manifest.xml    063491364108    available       public          i386    machine aki-a71cf9ce    ari-a51cf9cc
  IMAGE   ami-337c995a    alestic/ubuntu-8.04-hardy-desktop-20080419.manifest.xml 063491364108    available       public          i386    machine aki-a71cf9ce    ari-a51cf9cc
  IMAGE   ami-4f789d26    alestic/ubuntu-8.04-hardy-desktop-20080424.manifest.xml 063491364108    available       public          i386    machine aki-a71cf9ce    ari-a51cf9cc
  IMAGE   ami-f744a19e    alestic/ubuntu-8.04-hardy-desktop-20080430.manifest.xml 063491364108    available       public          i386    machine aki-a71cf9ce    ari-a51cf9cc
  IMAGE   ami-1f4bae76    alestic/ubuntu-8.04-hardy-desktop-20080514.manifest.xml 063491364108    available       public          i386    machine aki-a71cf9ce    ari-a51cf9cc
  IMAGE   ami-0e57b267    alestic/ubuntu-8.04-hardy-desktop-20080517.manifest.xml 063491364108    available       public          i386    machine aki-a71cf9ce    ari-a51cf9cc
  IMAGE   ami-b5bc58dc    alestic/ubuntu-8.04-hardy-desktop-20080628.manifest.xml 063491364108    available       public          i386    machine aki-a71cf9ce    ari-a51cf9cc
  IMAGE   ami-f39e7a9a    alestic/ubuntu-8.04-hardy-desktop-20080803.manifest.xml 063491364108    available       public          i386    machine aki-a71cf9ce    ari-a51cf9cc
  IMAGE   ami-44c4202d    alestic/ubuntu-8.04-hardy-desktop-20080905.manifest.xml 063491364108    available       public          i386    machine aki-a71cf9ce    ari-a51cf9cc
  IMAGE   ami-f7d4309e    alestic/ubuntu-8.04-hardy-desktop-20080922.manifest.xml 063491364108    available       public          i386    machine aki-a71cf9ce    ari-a51cf9cc
  IMAGE   ami-88d733e1    alestic/ubuntu-8.04-hardy-desktop-20080924.manifest.xml 063491364108    available       public          i386    machine aki-a71cf9ce    ari-a51cf9cc
  IMAGE   ami-bcbe5ad5    alestic/ubuntu-8.04-hardy-rightscale-20080701.manifest.xml      063491364108    available       public          i386    machine aki-a71cf9ce    ari-a51cf9cc
  IMAGE   ami-27b95d4e    alestic/ubuntu-8.04-hardy-rightscale-20080703.manifest.xml      063491364108    available       public          i386    machine aki-a71cf9ce    ari-a51cf9cc
  IMAGE   ami-b1ea0ed8    alestic/ubuntu-8.04-hardy-rightscale-20080824.manifest.xml      063491364108    available       public          i386    machine aki-a71cf9ce    ari-a51cf9cc
  IMAGE   ami-47c4202e    alestic/ubuntu-8.04-hardy-rightscale-20080905.manifest.xml      063491364108    available       public          i386    machine aki-a71cf9ce    ari-a51cf9cc
  IMAGE   ami-f4d4309d    alestic/ubuntu-8.04-hardy-rightscale-20080922.manifest.xml      063491364108    available       public          i386    machine aki-a71cf9ce    ari-a51cf9cc
  IMAGE   ami-89d733e0    alestic/ubuntu-8.04-hardy-rightscale-20080924.manifest.xml      063491364108    available       public          i386    machine aki-a71cf9ce    ari-a51cf9cc
  IMAGE   ami-dcbc58b5    alestic/ubuntu-8.10-intrepid-base-20080628.manifest.xml 063491364108    available       public          i386    machine aki-a71cf9ce    ari-a51cf9cc
  IMAGE   ami-db9e7ab2    alestic/ubuntu-8.10-intrepid-base-20080804.manifest.xml 063491364108    available       public          i386    machine aki-a71cf9ce    ari-a51cf9cc
  IMAGE   ami-9de105f4    alestic/ubuntu-8.10-intrepid-base-20080814.manifest.xml 063491364108    available       public          i386    machine aki-a71cf9ce    ari-a51cf9cc
  IMAGE   ami-c3fa1eaa    alestic/ubuntu-8.10-intrepid-base-20080905.manifest.xml 063491364108    available       public          i386    machine aki-a71cf9ce    ari-a51cf9cc
  IMAGE   ami-3bd43052    alestic/ubuntu-8.10-intrepid-base-20080922.manifest.xml 063491364108    available       public          i386    machine aki-a71cf9ce    ari-a51cf9cc
  IMAGE   ami-1ad73373    alestic/ubuntu-8.10-intrepid-base-20080924.manifest.xml 063491364108    available       public          i386    machine aki-a71cf9ce    ari-a51cf9cc
  IMAGE   ami-b6bc58df    alestic/ubuntu-8.10-intrepid-desktop-20080628.manifest.xml      063491364108    available       public          i386    machine aki-a71cf9ce    ari-a51cf9cc
  IMAGE   ami-d69e7abf    alestic/ubuntu-8.10-intrepid-desktop-20080804.manifest.xml      063491364108    available       public          i386    machine aki-a71cf9ce    ari-a51cf9cc
  IMAGE   ami-d4e206bd    alestic/ubuntu-8.10-intrepid-desktop-20080815.manifest.xml      063491364108    available       public          i386    machine aki-a71cf9ce    ari-a51cf9cc
  IMAGE   ami-7dc22614    alestic/ubuntu-8.10-intrepid-desktop-20080908.manifest.xml      063491364108    available       public          i386    machine aki-a71cf9ce    ari-a51cf9cc
  IMAGE   ami-f5d4309c    alestic/ubuntu-8.10-intrepid-desktop-20080922.manifest.xml      063491364108    available       public          i386    machine aki-a71cf9ce    ari-a51cf9cc
  IMAGE   ami-b6d733df    alestic/ubuntu-8.10-intrepid-desktop-20080924.manifest.xml      063491364108    available       public          i386    machine aki-a71cf9ce    ari-a51cf9cc

Generate a keypair

In the second step, I generated a keypair as my X.509 Certificate. That was used to identifiy myself to Amazon Web Services. Now I need to create another keypair which is used to log into a running EC2 instance. (Note, there is exactly one X.509 Certificate per user (i.e. AWS account), but a user can have many keypairs used for logging into various EC2 instances.) See also the Generating a keypair section in the Getting Started Guide.

• Generate the keypair. I named the keypair, disco-keypair because I will use this keypair with EC2 instances used to try out Disco.

  $ ec2-add-keypair disco-keypair > ~/.ec2/id_rsa-disco-keypair
      

• Set the permissions on the private key

  chmod 600 ~/.ec2/id_rsa-disco-keypair

Run an EC2 instance

• Select an image to run. I used the alestic/ubuntu-8.04-hardy-base-20080924 image with image ID ami-1cd73375.
• Run the instance

  $ ec2-run-instances -k disco-keypair ami-1cd73375

  It should return something like:

  RESERVATION     r-568f5d3f      719606167433    default
  INSTANCE        i-339f3c5a      ami-1cd73375                    pending disco-keypair       0               m1.small        2008-09-28T00:50:35+0000        us-east-1c aki-a71cf9ce     ari-a51cf9cc

• Check the status of the running instance:

  $ ec2-describe-instances

  After a short period of time, it should return something like:

  RESERVATION     r-568f5d3f      719606167433    default
  INSTANCE        i-339f3c5a      ami-1cd73375    ec2-75-101-200-13.compute-1.amazonaws.com       ip-10-251-30-10.ec2.internal     running disco-keypair   0               m1.small        2008-09-28T00:50:35+0000us-east-1c       aki-a71cf9ce    ari-a51cf9cc

  Note the address ec2-75-101-200-13.compute-1.amazonaws.com. This is the external address used to connect to the instance. Also note the instance ID i-339f3c5a. This is needed to terminate the instance.
• Authorize access to the instance through ports 22 (ssh) and 80 (http)

  $ ec2-authorize default -p 22
        GROUP           default
  PERMISSION              default ALLOWS  tcp     22      22      FROM    CIDR    0.0.0.0/0

  $ ec2-authorize default -p 80
  GROUP           default
  PERMISSION              default ALLOWS  tcp     80      80      FROM    CIDR    0.0.0.0/0

SSH into instance

• Use the address from the previous step to SSH into your instance:

  $ ssh -i ~/.ec2/id_rsa-disco-keypair -l root ec2-75-101-200-13.compute-1.amazonaws.com

Terminate the instance

• $ ec2-terminate-instance i-339f3c5a

  which returns:

  INSTANCE        i-339f3c5a      running shutting-down

• Running ec2-describe-instances shows that the instance is terminated.

  $ ec2-describe-instances 
  RESERVATION     r-568f5d3f      719606167433    default
  INSTANCE        i-339f3c5a      ami-1cd73375                    terminated      disco-keypair       0               m1.small        2008-09-28T00:50:35+0000           aki-a71cf9ce     ari-a51cf9cc

Here is an example of how to parse a URL using Python's urlparse module. See the urlparse module documentation for more information.

from urlparse import urlparse

url = 'http://www.gurlge.com:80/path/file.html;params?a=1#fragment'
o = urlparse(url)
print o.scheme
print o.netloc
print o.hostname
print o.port
print o.path
print o.params
print o.query
print o.fragment
print o.username
print o.password

Results:

http
www.gurlge.com:80
www.gurlge.com
80
/path/file.html
params
a=1
fragment
None
None

I wrote previously about how to get stdout and stderr using os.popen4. However, per the Python documentation, using the subprocess module is preferred:

The subprocess module allows you to spawn new processes, connect to their input/output/error pipes, and obtain their return codes. This module intends to replace several other, older modules and functions, such as:

os.system
os.spawn*
os.popen*
popen2.*
commands.*

See the subprocess module documentation for more information.

Here is how to get stdout and stderr from a program using the subprocess module:

from subprocess import Popen, PIPE, STDOUT

cmd = 'ls /etc/fstab /etc/non-existent-file'
p = Popen(cmd, shell=True, stdin=PIPE, stdout=PIPE, stderr=STDOUT, close_fds=True)
output = p.stdout.read()
print output

Results:

ls: cannot access /etc/non-existent-file: No such file or directory
/etc/fstab

Monit is a tool that can monitor your Apache web server, MySQL database, or other daemon process. It can restart the service based on configurable conditions such as CPU usage, memory usage, number of children, etc. It can log status to a file, email status, and it has a web interface for monitoring or restarting the service. Here are the steps I took to install and configure the Monit tool on Ubuntu Hardy. It merely monitors the status of my Apache web server and restarts it if it stops. It also checks if the memory used by Apache is greater than 1 MB and logs it in /var/log/monit.log. For more configuration options, see the examples in the default /etc/monit/monitrc file or the configuration examples in the monit documentation. I also found Ubuntu Geek's guide to be very helpful.

• Install monit

  $ sudo apt-get install monit

• Edit the config file

  $ sudo nano /etc/monit/monitrc

  Insert the following:

  # check services every 2 minutes
  set daemon 120
  
  # logging 
  set logfile /var/log/monit.log
  
  # web interface
  set httpd port 2812 and
      use address localhost # only accept connection from localhost
      allow localhost       # allow localhost to connect to the server    
      allow admin:monit     # require user ‘admin’ with password ‘monit’
  
  # monitor apache
  check process apache2 with pidfile /var/run/apache2.pid
      start program = "/etc/init.d/apache2 start"
      if totalmem > 1.0 MB for 2 cycles then alert

• Check the file syntax

  $ sudo monit -t

• Enable the service

  $ sudo nano /etc/default/monit

  Change the following line:

  startup=1

• Start monit

  $ sudo /etc/init.d/monit start

• Point your browser at http://localhost:2812 and log in using the user "admin" and the password "monit".
• Click on "apache2" and you can see information about the Apache process.

Simple cron example (tested on Ubuntu):

• Edit your (user) crontab file

  $ crontab -e

  This will bring up your editor (nano by default in Ubuntu)
  
  
• Enter the following inside. This will append the current date to a log file every minute. The 6 fields of the crontab file are: minute, hour, day of month, month, day of week, command.

  * * * * * /bin/date >> /tmp/cron_output
  

  Be sure to put a blank line at the end of the file.
  (NOTE 1: >> only redirects STDOUT to a file. To redirect both STDOUT and STDERR, use something like /bin/date >> /tmp/cron_output 2>&1)
  (NOTE 2: If output is not redirected, cron will try to email the output to you. To do this, a mail transfer agent such as sendmail or postfix must be installed.)
  (NOTE 3 (added 2015-06-24): When I created my cron script in /etc/cron.d with Emacs using sudo::, cron didn't pick up my script. When I created it with nano, cron picked it up. It seems the cause is the permissions of the cron script. Emacs created the script with 664 permissions while nano created the script with 644 permissions. When I changed the permissions to 644, it started working. I am running Ubuntu 15.04. This Ask Ubuntu answer confirms that a 644 permission is problematic because it is considered insecure. See /var/log/syslog for cron messages. The Ask Ubuntu page has a lot of other good tips: Reasons why crontab does not work)
  
  
• Exit the editor. It should output:

  crontab: installing new crontab

• Check that it is working:

  tail -f /tmp/cron_output

  You should see the date updated every minute on the minute (or close to it):

  Tue Sep 16 23:58:01 PDT 2008
  Tue Sep 16 23:59:01 PDT 2008
  Wed Sep 17 00:00:01 PDT 2008
  Wed Sep 17 00:01:01 PDT 2008
  ...
  

See also my post: Postgres backup with cron

I wanted to insert URL redirects on my old Blogger posts pointing to my new blog articles. A comment on my Migrating Blogger Posts post suggested that I use the (Python) Blogger API. This was a great suggestion. The Blogger API was well documented and easy to use. Here is the script I used to insert the URL redirects on each of my old Blogger posts.

from gdata import service
import re
import gdata
import atom

NEW_HTML = """
<script language="javascript">
  setTimeout('location.href="%s"', 2000);
</script>
<br /><br />
<b>
  </b><p>This is my OLD blog. I've copied this post over to my NEW blog at:</p>
  <p><a href="%s">%s</a></p>
  <p>You should be redirected in 2 seconds.</p>

<br /><br />
"""

# authenticate
blogger_service = service.GDataService('[email protected]', 'mypassword')
blogger_service.service = 'blogger'
blogger_service.account_type = 'GOOGLE'
blogger_service.server = 'www.blogger.com'
blogger_service.ProgrammaticLogin()

# get list of blogs
query = service.Query()
query.feed = '/feeds/default/blogs'
feed = blogger_service.Get(query.ToUri())

# get blog id
blog_id = feed.entry[0].GetSelfLink().href.split("/")[-1]

# get all posts
query = service.Query()
query.feed = '/feeds/%s/posts/default' % blog_id
query.published_min = '2000-01-01'
query.published_max = '2009-01-01'
query.max_results = 1000
feed = blogger_service.Get(query.ToUri())
print feed.title.text

for entry in feed.entry:
    # create link to article on new blog
    new_link = re.sub(r'http://iwiwdsmi\.blogspot\.com/(.*)\.html',
                      r'http://www.saltycrane.com/blog/\1/',
                      entry.link[0].href)
    print new_link

    # update post
    to_add = NEW_HTML % (new_link, new_link, new_link)
    entry.content.text = to_add + entry.content.text
    blogger_service.Put(entry, entry.GetEditLink().href)