Saltycrane logo

SaltyCrane Blog

Notes on Python, Django, and web development on Ubuntu Linux

    

Using psycopg2 with virtualenv on Ubuntu JauntyLucid

Date: 2009-07-31  |   Modified: 2010-05-17  |   Tags: install_setup, python, ubuntu   |   15 Comments

Update 2009-11-02: Well I am dumb-- psycopg2 can be installed with pip/easy_install. The reason I got the error before was because I didn't have the required dependencies installed. On Ubuntu, I used apt-get build-dep. So, here's the summary:

Update 2009-11-11: My update doesn't work. See comments #4 and #5 below.

Update 2010-05-17: Here is what finally worked for me using Python 2.6 on Ubuntu 10.04 Lucid Lynx using virtualenv 1.4.8 and pip 0.7.1. Thanks to Daniel for the final piece of the solution.

Install dependencies

$ sudo apt-get build-dep python-psycopg2

Install pyscopg2 in a virtualenv

$ virtualenv --no-site-packages --distribute myenv
$ source myenv/bin/activate
$ pip install psycopg2
$ easy_install -i http://downloads.egenix.com/python/index/ucs4/ egenix-mx-base


*** THE REST OF THIS POST DESCRIBES MY INITIAL OUTDATED SOLUTION. ***

I recently starting using virtualenv for creating isolated Python environments. Virtualenv has made it easy to manage different Python configurations for different websites, but I was slowed down a bit trying to use psycopg2, the Python-PostgreSQL database adapter, with virtualenv after upgrading to Ubuntu 9.04 Jaunty Jackalope.

Currently, virtualenv (1.3.3) doesn't find Ubuntu Jaunty's Python packages. In Ubuntu Jaunty, the default Python version changed from 2.5 to 2.6. More importantly, the site-packages directory was changed to dist-packages. Prior to Ubuntu 9.04, Ubuntu's Python packages, such as python-psycopg2, were installed to /usr/lib/python2.5/site-packages. Now Ubuntu's Python packages are installed to /usr/lib/python2.6/dist-packages. (See this discussion at the virtualenv group for more information.)

As a result of this change, virtualenv (as of 1.3.3) doesn't find Ubuntu's Python packages installed using apt-get. My solution was to create symlinks to the desired packages and egg-info files in site-packages. I'm not sure if this is the proper way to handle this. If there is a better solution, please let me know. One advantage of using this method is that I don't need to clutter my virtualenv with all the packages that have accumulated in my global site-packagesdist-packages.

Install easy_install, pip, and virtualenv

sudo apt-get install python-setuptools python-dev build-essential
sudo easy_install -U pip
sudo pip install -U virtualenv

Install Ubuntu's psycopg2 package

sudo apt-get install python-psycopg2

Symlink the psycopg2 (and mxDateTime) files

sudo mkdir /usr/lib/python2.6/site-packages
sudo ln -s /usr/lib/python2.6/dist-packages/psycopg2 /usr/lib/python2.6/site-packages
sudo ln -s /usr/lib/python2.6/dist-packages/psycopg2-2.0.8.egg-info /usr/lib/python2.6/site-packages
sudo ln -s /usr/lib/python2.6/dist-packages/mx /usr/lib/python2.6/site-packages

Create a virtualenv

virtualenv myenv

Check what's available

pip freeze -E myenv

Results:

psycopg2==2.0.8
wsgiref==0.1.2


Note: you might wonder why I didn't do a pip install -E myenv psycopg2. I tried this, but got an error. Maybe psycopg2 doesn't support pip/easy_install? Here is my error message:

Downloading/unpacking psycopg2
  Downloading psycopg2-2.0.11.tar.gz (255Kb): 255Kb downloaded
  Running setup.py egg_info for package psycopg2
    error: No such file or directory
    Complete output from command python setup.py egg_info:
    running egg_info

creating pip-egg-info/psycopg2.egg-info

writing pip-egg-info/psycopg2.egg-info/PKG-INFO

writing top-level names to pip-egg-info/psycopg2.egg-info/top_level.txt

writing dependency_links to pip-egg-info/psycopg2.egg-info/dependency_links.txt

writing manifest file 'pip-egg-info/psycopg2.egg-info/SOURCES.txt'

warning: manifest_maker: standard file '-c' not found

error: No such file or directory

----------------------------------------
Command python setup.py egg_info failed with error code 1
Storing complete log in ./pip-log.txt
Complete output from command temp/bin/python /usr/local/lib/python2.6/dist-packages/pip-0.4-py2.6.egg/pip.py install -E temp psycopg2 temp ___VENV_RESTART___:

----------------------------------------
Traceback (most recent call last):
  File "/usr/local/bin/pip", line 3, in 
    pip.main()
  File "/usr/local/lib/python2.6/dist-packages/pip-0.4-py2.6.egg/pip.py", line 926, in main
    return command.main(initial_args, args[1:], options)
  File "/usr/local/lib/python2.6/dist-packages/pip-0.4-py2.6.egg/pip.py", line 258, in main
    restart_in_venv(options.venv, site_packages, complete_args)
  File "/usr/local/lib/python2.6/dist-packages/pip-0.4-py2.6.egg/pip.py", line 1009, in restart_in_venv
    call_subprocess([python, file] + args + [base, '___VENV_RESTART___'])
  File "/usr/local/lib/python2.6/dist-packages/pip-0.4-py2.6.egg/pip.py", line 3643, in call_subprocess
    % (command_desc, proc.returncode))
pip.InstallationError: Command temp/bin/python /usr/local/lib/python2.6/dist-packages/pip-0.4-py2.6.egg/pip.py install -E temp psycopg2 temp ___VENV_RESTART___ failed with error code 1

15 Comments — feed icon Comments feed for this post


#1 aw commented on 2009-08-19:

Thanks, this worked for me.

Another way (but I haven't tried this) might be to install via setup.py from within the virtualenv:

(myenv)$ wget http://initd.org/pub/software/psycopg/psycopg2-2.0.11.tar.gz (myenv)$ tar zxf psycopg2-2.0.11.tar.gz (myenv)$ cd psycopg2-2.0.11 (myenv)$ python setup.py install running install running build [...] (myenv)$ python -c 'import psycopg2'


#2 aw commented on 2009-08-19:

ugh, messed up the line breaks:

Another way (but I haven't tried this) might be to install via setup.py from within the virtualenv:

(myenv)$ wget http://initd.org/pub/software/psycopg/psycopg2-2.0.11.tar.gz
(myenv)$ tar zxf psycopg2-2.0.11.tar.gz
(myenv)$ cd psycopg2-2.0.11
(myenv)$ python setup.py install
running install
running build
[lots of messages...]

(myenv)$ python -c 'import psycopg2'


#3 bw commented on 2009-10-06:

(myenv)$ ../bin/python setup.py install

;-)

..bw


#4 Shige Abe commented on 2009-11-06:

This seemed to work fine for me as far as installing, but....

>>> import psycopg2
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/shige/bioaffinitycareers.com/lib/python2.6/site-packages/psycopg2/__init__.py", line 60, in <module>
from _psycopg import BINARY, NUMBER, STRING, DATETIME, ROWID
ImportError: can't import mx.DateTime module

I googled it but nothing that was obvious to me helped.

Do you have any ideas?


#5 Eliot commented on 2009-11-06:

Shige,
You are right. I got this same error. My update was not complete.

I know we need to install the mxDateTime library. But now I can't seem to install this with pip. I tried pip install -E myenv egenix-mx-base but got an error:

Downloading/unpacking egenix-mx-base
  Downloading egenix-mx-base-3.1.2.tar.gz (8.2Mb): 8.2Mb downloaded
  Running setup.py egg_info for package egenix-mx-base
    no build data file 'build/build-py2.6_ucs4.pck' found
Installing collected packages: egenix-mx-base
  Running setup.py install for egenix-mx-base
    usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
       or: -c --help [cmd1 cmd2 ...]
       or: -c --help-commands
       or: -c cmd --help

    error: option --single-version-externally-managed not recognized
    Complete output from command /home/saltycrane/lib/python-environments/myenv/bin/python -c "import setuptools; __file__='/home/saltycrane/lib/python-environments/myenv/build/egenix-mx-base/setup.py'; execfile('/home/saltycrane/lib/python-environments/myenv/build/egenix-mx-base/setup.py')" install --single-version-externally-managed --record /tmp/pip-1VsR7f-record/install-record.txt --install-headers /home/saltycrane/lib/python-environments/myenv/lib/include:
    usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]

   or: -c --help [cmd1 cmd2 ...]

   or: -c --help-commands

   or: -c cmd --help

error: option --single-version-externally-managed not recognized

----------------------------------------
Command /home/saltycrane/lib/python-environments/myenv/bin/python -c "import setuptools; __file__='/home/saltycrane/lib/python-environments/myenv/build/egenix-mx-base/setup.py'; execfile('/home/saltycrane/lib/python-environments/myenv/build/egenix-mx-base/setup.py')" install --single-version-externally-managed --record /tmp/pip-1VsR7f-record/install-record.txt --install-headers /home/saltycrane/lib/python-environments/myenv/lib/include failed with error code 1
Storing complete log in ./pip-log.txt

#6 Tim commented on 2009-11-15:

This sequence of commands seemed to work for me (helped greatly by this post):

$ sudo apt-get build-dep python-psycopg2
$ virtualenv ENV
$ source ENV/bin/activate
$ easy_install psycopg2
$ easy_install egenix-mx-base

For some reason you can't install egenix with pip..


#7 Eliot commented on 2009-11-16:

Tim, Thank you for figuring this out and posting your solution.


#8 Joebie commented on 2009-11-21:

I had the same problem under fedora-12, for me the solution was installing gcc and the postgresql-devel packages.


#9 Ben commented on 2009-11-21:

I had the exact same problem with psycopg2 in CentOS, but was able to solve it the same way Joebie did. I installed the development extras like so:

$ yum install postgres-devel

And everything seems to be working fine so far.

Thanks much for this post. I've just started a new project and resolved myself to try and start using tools like virtualenv this time around. I doubt I could get there without valuable help from leaders such as yourself.

Sincerely,

Ben.


#10 anothertrad commented on 2010-05-11:

Hi folks! In my case, on Ubuntu, running python 2.5, I have created an environment without inherit the native libs of the system. I couldn't import psycopg2. Then I have downloaded the code, and when I try to build (both on my environment or out of it): Warning: Unable to find 'pg_config' filebuilding 'psycopg2._psycopg' extension gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -DPSYCOPG_DEFAULT_PYDATETIME=1 -DPSYCOPG_VERSION="2.0.14 (dt dec ext pq3)" -DPSYCOPG_EXTENSIONS=1 -DPSYCOPG_NEW_BOOLEAN=1 -DHAVE_PQFREEMEM=1 -DHAVE_PQPROTOCOL3=1 -I/usr/include/python2.5 -I. -c psycopg/psycopgmodule.c -o build/temp.linux-i686-2.5/psycopg/psycopgmodule.o In file included from psycopg/psycopgmodule.c:32: ./psycopg/psycopg.h:31:22: error: libpq-fe.h: No such file or directory In file included from psycopg/psycopgmodule.c:33: ./psycopg/connection.h:72: error: expected specifier-qualifier-list before \u2018PGconn\u2019 ./psycopg/connection.h:94: error: expected declaration specifiers or \u2018...\u2019 before \u2018PGconn\u2019 In file included from psycopg/psycopgmodule.c:34: ./psycopg/cursor.h:61: error: expected specifier-qualifier-list before \u2018PGresult\u2019 In file included from psycopg/psycopgmodule.c:35: ./psycopg/lobject.h:31:28: error: libpq/libpq-fs.h: No such file or directory In file included from psycopg/psycopgmodule.c:35: ./psycopg/lobject.h:51: error: expected specifier-qualifier-list before \u2018Oid\u2019 ./psycopg/lobject.h:57: error: expected declaration specifiers or \u2018...\u2019 before \u2018Oid\u2019 ./psycopg/lobject.h:57: error: expected declaration specifiers or \u2018...\u2019 before \u2018Oid\u2019 psycopg/psycopgmodule.c: In function \u2018psyco_register_type\u2019: psycopg/psycopgmodule.c:254: error: \u2018cursorObject\u2019 has no member named \u2018string_types\u2019 psycopg/psycopgmodule.c:257: error: \u2018connectionObject\u2019 has no member named \u2018string_types\u2019 error: command 'gcc' failed with exit status 1

I have the latest build-essential. Any Ideas?


#11 anothertrad commented on 2010-05-11:

Oh, I almost forgot: I already tried to:

0) sudo -s

1) apt-get remove python-psycopg2

2) enter in my virtual env

3) apt-get install python-psycopg2

but I can only import outside my environment, just like before


#12 Daniel commented on 2010-05-13:

I had to do this:

easy_install -i http://downloads.egenix.com/python/index/ucs4/ egenix-mx-base

To get it to work on ubuntu 10.04 + Python 2.6

cheers


#13 Ryan commented on 2010-05-17:

I upgraded to 10.4 lynx and had a similar error, specifically...

error: libpq/libpq-fs.h: No such file or directory

I tried easy_install egenix-mx-base, but this did not work for me.

I had to install the 'libpq-dev' package which contains the Postgres header files required when building psycopg2.

This now works for me, thanks all.


#14 Eliot commented on 2010-05-17:

Daniel: Thank you. Your solution was the one that worked for me on Ubuntu Lucid. I updated the post.

Ryan: I didn't need to do this, but maybe our configurations our different. Thanks for adding your notes.


#15 Mike commented on 2010-06-04:

...succinctly getting me out of a tight spot.

I raise my cup of java in your honor.

Post a comment
Required
Required, but not displayed
Optional

Format using Markdown. (No HTML.)
  • Code blocks: prefix each line by at least 4 spaces or 1 tab (and a blank line before and after)
  • Code span: surround with backticks
  • Blockquotes: prefix lines to be quoted with >
  • Links: <URL>
  • Links w/ description: [description](URL)
Created with Django | Hosted by Slicehost