Notes on starting processes in Python
Here is an example using
os.fork() to spawn 5 processes each running the python function,
myfunc. Don't forget the
os._exit() at the end. Per the docs, normally,
sys.exit() is used, but
os._exit() can be used in child processes after a fork. It does not call cleanup handlers, flush stdio buffers, etc.
import os import time def myfunc(i): print "sleeping 5 seconds from process %s" % i time.sleep(5) print "finished sleeping from process %s" % i for i in range(5): pid = os.fork() if pid == 0: myfunc(i) os._exit(0)
sleeping 5 seconds from process 0 sleeping 5 seconds from process 1 sleeping 5 seconds from process 2 sleeping 5 seconds from process 3 sleeping 5 seconds from process 4
And 5 seconds later...
finished sleeping from process 0 finished sleeping from process 1 finished sleeping from process 2 finished sleeping from process 3 finished sleeping from process 4
Running an external script in subprocesses
Alternatively, if you want to run an external script in multiple processes, you can use the
Popen class in the
subprocess module. For example, to run the following script, called "myscript.py":
"myscript.py" import sys import time def myfunc(i): print "sleeping 5 seconds from process %s" % i time.sleep(5) print "finished sleeping from process %s" % i if __name__ == '__main__': myfunc(sys.argv)
use the following Python code stored in the same directory:
"popen_ex.py" from subprocess import Popen for i in range(5): Popen(['python', './myscript.py', str(i)])
The screen output is the same as the previous example. What's the differnce?
fork() copies the process memory space including open file descriptors to the child process. In the second example, since I am executing a new Python interpreter from scratch, I get a "cleaner" start but probably more overhead as well.