Saltycrane logo

SaltyCrane Blog

Notes on Python, Django, and web development on Ubuntu Linux

    

Installing beanstalkd and pybeanstalk on Ubuntu

Date: 2008-10-10  |   Modified: 2008-10-20  |   Tags: concurrency, linux, python, ubuntu

Nikolay pointed me at beanstalkd, a lightweight, message queue service partially inspired by the popular caching system, memcached. It features a blocking "reserve" call so workers don't need to poll for new jobs. However, some might miss the data persistence since the queue is stored in memory. beanstalkd has client libraries in Erlang, Perl, PHP, Python, and Ruby. Of course, I will use the Python version, pybeanstalk. Simon Willison also found beanstalkd interesting. (Credit to him for some of the words above.)

Update 2008-10-13: For a helpful example on using beanstalkd and pybeanstalk see Parand's beanstalkd tutorial.

Here is how I installed the beanstalkd server and pybeanstalk client on Ubuntu. I found no Ubuntu package for beanstalkd, so I installed from source.

Install beanstalkd
  • Install prerequistes 
    $ sudo apt-get install libevent1 libevent-dev
  • Download 
    $ cd ~/lib
$ wget http://xph.us/software/beanstalkd/rel/beanstalkd-1.0.tar.gz
  • Unpack 
    $ tar zxvf beanstalkd-1.0.tar.gz
  • Make 
    $ cd beanstalkd-1.0
$ make
  • Print help 
    $ ./beanstalkd -h
Install PyYAML
  • Download 
    $ wget http://pyyaml.org/download/pyyaml/PyYAML-3.06.tar.gz
  • Unpack 
    $ tar zxvf PyYAML-3.06.tar.gz
  • Put PyYAML-3.06/lib/yaml somewhere on your Python path or run python setup.py.
Install pybeanstalk
  • Download 
    $ wget http://pybeanstalk.googlecode.com/files/pybeanstalk-0.11.1.tar.gz
  • Unpack 
    $ tar zxvf pybeanstalk-0.11.1.tar.gz
  • Put pybeanstalk-0.11.1/beanstalk somewhere on your Python path or run python setup.py
Run beanstalkd server
  • $ ~/lib/beanstalkd-1.0/beanstalkd -d -l 127.0.0.5 -p 11300
Run test client
  • Create a file and run it: 
    from beanstalk import serverconn
from beanstalk import job

SERVER = '127.0.0.5'
PORT = 11300

# setup connection
connection = serverconn.ServerConn(SERVER, PORT)
connection.job = job.Job

# produce data
for i in range(5):
    print 'put data: %d' % i
    data = job.Job(data=str(i), conn=connection)
    data.Queue()

# consume data
while True:
    j = connection.reserve()
    print 'got data: %s' % j.data
    j.Finish()
    Results: 
    put data: 0
put data: 1
put data: 2
put data: 3
put data: 4
got data: 0
got data: 1
got data: 2
got data: 3
got data: 4

Post a comment
Required
Required, but not displayed
Optional

Format using Markdown. (No HTML.)
  • Code blocks: prefix each line by at least 4 spaces or 1 tab (and a blank line before and after)
  • Code span: surround with backticks
  • Blockquotes: prefix lines to be quoted with >
  • Links: <URL>
  • Links w/ description: [description](URL)
Created with Django | Hosted by Slicehost