Notes on using Gearman with Python

We recently looked at some lightweight message queue options for a S3 uploader tool. One of the options we tried was Gearman. Gearman was originally developed by Brad Fitzpatrick (author of memcached). Gearman seems to be mature and actively developed. The original Perl version has been rewritten in C for improved performance and I found it easy to use. Here are my notes for getting started on Ubuntu Karmic.

See also¶

• Advice on Python/Django and message queues « Stack Overflow
• Choosing a message queue for Python on Ubuntu on a VPS « Graham King
• Gearman Build Your Own Distributed Platform in 3 Hours « OSCON 2009 Presentation

Install and run Gearman server (C version)¶

There are 2 versions of the Gearman server: the new C version and the original Perl version. I chose the C version.

sudo apt-get install gearman-job-server

During the installation process, Ubuntu/Apt starts the Gearman server process. Running ps -ef | grep gearmand shows me:

gearman    497     1  0 15:41 ?        00:00:00 /usr/sbin/gearmand --pid-file=/var/run/gearman gearmand.pid --user=gearman --daemon --log-file=/var/log/gearman-job-server/gearman.log

This shows the log file is at /var/log/gearman-job-server/gearman.log. Also, it listens at address 127.0.0.1 and port 4730 by default. You can change change the address, port, etc. via the command-line options. To see all the options, type gearmand --help.

Install Python Gearman client library¶

• Install pip
• Install gearman library

  sudo pip install gearman

Example¶

This example is taken from Graham's article.

producer.py:

import time
from gearman import GearmanClient, Task

client = GearmanClient(["127.0.0.1"])

for i in range(5):
    client.dispatch_background_task('speak', i)
    print 'Dispatched %d' % i
    time.sleep(1)

consumer.py:

from gearman import GearmanWorker

def speak(job):
    r = 'Hello %s' % job.arg
    print r
    return r

worker = GearmanWorker(["127.0.0.1"])
worker.register_function('speak', speak, timeout=3)
worker.work()

First running python producer.py gives me the following terminal output:

Dispatched 0
Dispatched 1
Dispatched 2
Dispatched 3
Dispatched 4

Then running python consumer.py gives me the following terminal output:

Hello 0
Hello 1
Hello 2
Hello 3
Hello 4