Saltycrane logo

SaltyCrane Blog

Notes on Python, Django, and web development on Ubuntu Linux

    

Notes on starting processes in Python

Date: 2008-09-12  |   Modified: 2008-10-20  |   Tags: concurrency, python
Using os.fork()

Here is an example using os.fork() to spawn 5 processes each running the python function, myfunc. Don't forget the os._exit() at the end. Per the docs, normally, sys.exit() is used, but os._exit() can be used in child processes after a fork. It does not call cleanup handlers, flush stdio buffers, etc. 

import os
import time

def myfunc(i):
    print "sleeping 5 seconds from process %s" % i
    time.sleep(5)
    print "finished sleeping from process %s" % i

for i in range(5):
    pid = os.fork()
    if pid == 0:
        myfunc(i)
        os._exit(0)

Results:

sleeping 5 seconds from process 0
sleeping 5 seconds from process 1
sleeping 5 seconds from process 2
sleeping 5 seconds from process 3
sleeping 5 seconds from process 4

And 5 seconds later...

finished sleeping from process 0
finished sleeping from process 1
finished sleeping from process 2
finished sleeping from process 3
finished sleeping from process 4
Running an external script in subprocesses

Alternatively, if you want to run an external script in multiple processes, you can use the Popen class in the subprocess module. For example, to run the following script, called "myscript.py":

"myscript.py"
import sys
import time

def myfunc(i):
    print "sleeping 5 seconds from process %s" % i
    time.sleep(5)
    print "finished sleeping from process %s" % i

if __name__ == '__main__':
    myfunc(sys.argv[1])

use the following Python code stored in the same directory:

"popen_ex.py"
from subprocess import Popen

for i in range(5):
    Popen(['python', './myscript.py', str(i)])

The screen output is the same as the previous example. What's the differnce? fork() copies the process memory space including open file descriptors to the child process. In the second example, since I am executing a new Python interpreter from scratch, I get a "cleaner" start but probably more overhead as well.

Post a comment
Required
Required, but not displayed
Optional

Format using Markdown. (No HTML.)
  • Code blocks: prefix each line by at least 4 spaces or 1 tab (and a blank line before and after)
  • Code span: surround with backticks
  • Blockquotes: prefix lines to be quoted with >
  • Links: <URL>
  • Links w/ description: [description](URL)
Created with Django | Hosted by Slicehost