How to get stdout and stderr using Python's subprocess module

Date: Sept. 23, 2008 | Tags: processes, python | 5 Comments

I wrote previously about how to get stdout and stderr using os.popen4. However, per the Python documentation, using the subprocess module is preferred:

The subprocess module allows you to spawn new processes, connect to their input/output/error pipes, and obtain their return codes. This module intends to replace several other, older modules and functions, such as:

os.system
os.spawn*
os.popen*
popen2.*
commands.*

See the subprocess module documentation for more information.

Here is how to get stdout and stderr from a program using the subprocess module:

from subprocess import Popen, PIPE, STDOUT

cmd = 'ls /etc/fstab /etc/non-existent-file'
p = Popen(cmd, shell=True, stdin=PIPE, stdout=PIPE, stderr=STDOUT, close_fds=True)
output = p.stdout.read()
print output

Results:

ls: cannot access /etc/non-existent-file: No such file or directory
/etc/fstab

How to use the bash shell with Python's subprocess module instead of /bin/sh — posted 2011-04-13
How to capture stdout in real-time with Python — posted 2009-10-12
How to use python and popen4 to capture stdout and stderr from a command — posted 2007-03-12

5 Comments — Comments feed for this post

#1 Ian McCracken commented on 2008-09-23:

Here's a cleaner way:

from subprocess import Popen, PIPE
cmd = 'blah'
p = Popen(cmd, stdout=PIPE, stderr=PIPE)
stdout, stderr = p.communicate()

That way you wait for the output until the process has actually completed.

Also, take a look at the Process objects in the cliutils package. (Disclaimer: I wrote it.) They let you do some cool things.

#2 Eliot commented on 2008-09-23:

Ian,
This looks like an interesting alternative. I just took my example from the Replacing os.popen* section in the docs.

#3 Flemmingbjerke commented on 2010-12-23:

There is error here: p = Popen(cmd, shell=True, stdin=PIPE, stdout=PIPE, stderr=STDOUT, close_fds=True)

Python says: NameError: name 'STDOUT' is not defined

If I use stderr=PIPE, it works.

This one is not good enough either: p = Popen(cmd, stdout=PIPE, stderr=PIPE)

It works fine with ls, rmdir etc. but with wget it seems to make errors whatever you do. But, the following seems to work fine with wget

p = Popen(cmd, shell=True, stdin=PIPE, stdout=PIPE, stderr=PIPE)

Then, the following seems to work fine:

stdout, stderr = p.communicate()

Ian's p.communicate

#4 Ron commented on 2011-08-23:

to Flemmingbjerke:

You got "'STDOUT' is not defined" because you must either: from subprocess import STDOUT or you must refer to STDOUT as subprocess.STDOUT .

I'm surprised you had any success running ls and rmdir without shell=True. ls and rmdir are not programs, they are internal commands within the shell program.

wget, however, is a program. You can use shell=True, but you don't need to. Using shell=False (the default) will save you from having to start a new shell process (which is an expensive operation). When you submit a command with shell=True, you submit the whole command as one string. But when you submit the command with shell=False, you must issue the command as a list of strings, with the command name in the first element of the list, the first argument in the next list element, etc. For example:

wgproc = subprocess.Popen(['wget', '-r', '--tries=10', 'http://fly.srk.fer.hr/', '-o', 'log'], stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
(standardout, junk) = wgproc.communicate()

#5 Pas commented on 2011-12-26:

Hm. ls, rm, rmdir and others are programs. They live under /bin, just do a which ls.

And thanks for the useful post and comments :)


Name:	Required
Email:	Required, but not displayed
Website:	Optional
Comment:	Format using Markdown. (No HTML.) Code blocks: prefix each line by at least 4 spaces or 1 tab (and a blank line before and after) Code span: surround with backticks Blockquotes: prefix lines to be quoted with > Links: <URL> Links w/ description: [description](URL)

SaltyCrane Blog