SaltyCrane Blog — Notes on JavaScript and web development

Notes on using Gearman with Python

We recently looked at some lightweight message queue options for a S3 uploader tool. One of the options we tried was Gearman. Gearman was originally developed by Brad Fitzpatrick (author of memcached). Gearman seems to be mature and actively developed. The original Perl version has been rewritten in C for improved performance and I found it easy to use. Here are my notes for getting started on Ubuntu Karmic.

See also

Install and run Gearman server (C version)

There are 2 versions of the Gearman server: the new C version and the original Perl version. I chose the C version.

sudo apt-get install gearman-job-server

During the installation process, Ubuntu/Apt starts the Gearman server process. Running ps -ef | grep gearmand shows me:

gearman    497     1  0 15:41 ?        00:00:00 /usr/sbin/gearmand --pid-file=/var/run/gearman gearmand.pid --user=gearman --daemon --log-file=/var/log/gearman-job-server/gearman.log

This shows the log file is at /var/log/gearman-job-server/gearman.log. Also, it listens at address 127.0.0.1 and port 4730 by default. You can change change the address, port, etc. via the command-line options. To see all the options, type gearmand --help.

Install Python Gearman client library

  • Install pip
  • Install gearman library
    sudo pip install gearman

Example

This example is taken from Graham's article.

producer.py:

import time
from gearman import GearmanClient, Task

client = GearmanClient(["127.0.0.1"])

for i in range(5):
    client.dispatch_background_task('speak', i)
    print 'Dispatched %d' % i
    time.sleep(1)

consumer.py:

from gearman import GearmanWorker

def speak(job):
    r = 'Hello %s' % job.arg
    print r
    return r

worker = GearmanWorker(["127.0.0.1"])
worker.register_function('speak', speak, timeout=3)
worker.work()

First running python producer.py gives me the following terminal output:

Dispatched 0
Dispatched 1
Dispatched 2
Dispatched 3
Dispatched 4

Then running python consumer.py gives me the following terminal output:

Hello 0
Hello 1
Hello 2
Hello 3
Hello 4

Comments


#1 Parand commented on :

Why use gearman instead of beanstalk or even your own redis base queue, out of curiosity?


#2 Eliot commented on :

Hey Parand,
Honestly, I don't know much about the advantages/disadvantages compared to beanstalkd. They both look pretty good to me. We actually tried this before I pinged you on Twitter. There was a delay before I made this blog post. I just wanted to record my notes in case it might be useful for others or myself in the future. (We are currently using the Redis-based solution because it gives us the delay and uniqueness features. Thanks for the idea!)


#4 Ravi commented on :

Hi,

I have tried same code. When i run python producer.py its shows nothing and keep on waiting.

Then when i ran "python consumer.py" in other terminal window, my older terminal command (python producer.py) executed and shows me result :

Dispatched 0 Dispatched 1 Dispatched 2 Dispatched 3 Dispatched 4

but current terminal again keeps on waiting. It does not show me "Hello 0"... etc

Can you please help.