SaltyCrane Blog — Notes on JavaScript and web development

Installing beanstalkd and pybeanstalk on Ubuntu

Nikolay pointed me at beanstalkd, a lightweight, message queue service partially inspired by the popular caching system, memcached. It features a blocking "reserve" call so workers don't need to poll for new jobs. However, some might miss the data persistence since the queue is stored in memory. beanstalkd has client libraries in Erlang, Perl, PHP, Python, and Ruby. Of course, I will use the Python version, pybeanstalk. Simon Willison also found beanstalkd interesting. (Credit to him for some of the words above.)

Update 2008-10-13: For a helpful example on using beanstalkd and pybeanstalk see Parand's beanstalkd tutorial.

Here is how I installed the beanstalkd server and pybeanstalk client on Ubuntu. I found no Ubuntu package for beanstalkd, so I installed from source.

Install beanstalkd
  • Install prerequistes
    $ sudo apt-get install libevent1 libevent-dev
  • Download
    $ cd ~/lib
    $ wget
  • Unpack
    $ tar zxvf beanstalkd-1.0.tar.gz
  • Make
    $ cd beanstalkd-1.0
    $ make
  • Print help
    $ ./beanstalkd -h
Install PyYAML
  • Download
    $ wget
  • Unpack
    $ tar zxvf PyYAML-3.06.tar.gz
  • Put PyYAML-3.06/lib/yaml somewhere on your Python path or run python
Install pybeanstalk
  • Download
    $ wget
  • Unpack
    $ tar zxvf pybeanstalk-0.11.1.tar.gz
  • Put pybeanstalk-0.11.1/beanstalk somewhere on your Python path or run python
Run beanstalkd server
  • $ ~/lib/beanstalkd-1.0/beanstalkd -d -l -p 11300
Run test client
  • Create a file and run it:
    from beanstalk import serverconn
    from beanstalk import job
    SERVER = ''
    PORT = 11300
    # setup connection
    connection = serverconn.ServerConn(SERVER, PORT)
    connection.job = job.Job
    # produce data
    for i in range(5):
        print 'put data: %d' % i
        data = job.Job(data=str(i), conn=connection)
    # consume data
    while True:
        j = connection.reserve()
        print 'got data: %s' %
    put data: 0
    put data: 1
    put data: 2
    put data: 3
    put data: 4
    got data: 0
    got data: 1
    got data: 2
    got data: 3
    got data: 4