Saltycrane logo

SaltyCrane Blog

Notes on Python, Django, and web development on Ubuntu Linux

    

Notes on using pip and virtualenv with Django

Date: 2009-05-06  |   Modified: 2010-04-26  |   Tags: django, install_setup, python, ubuntu   |   17 Comments

I have been using a symlinking method to install Python packages up to this point. To better handle dependencies and multiple versions I have wanted to switch over to pip and virtualenv. Pip is a better alternative to Easy Install and virtualenv is a tool to create isolated Python environments. I have wanted to use pip and virtualenv for a long time now. Finally, today, I took my first steps and created an environment with the Python packages required for this blog. My notes are below. (I am running Ubuntu IntrepidKarmic and Python 2.52.6.)

A lot of my notes on virtualenv are taken from Arthur Koziel's excellent tutorial. Also, below are some links to some essential information on pip, virtualenv, and Python packaging (mostly from the creator of pip and virtualenv, Ian Bicking):

Install Easy Install

If you don't already have Easy Install, it can be installed as follows:

$ sudo apt-get install python-setuptools python-dev build-essential

Install pip

$ sudo easy_install -U pip
Searching for pip
Reading http://pypi.python.org/simple/pip/
Reading http://pip.openplans.org
Best match: pip 0.7.1
Downloading http://pypi.python.org/packages/source/p/pip/pip-0.7.1.tar.gz#md5=420c83ad67bdcb542f772eb64392cce6
Processing pip-0.7.1.tar.gz
Running pip-0.7.1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-lEQ9xp/pip-0.7.1/egg-dist-tmp-KOpPI1
warning: no previously-included files matching '*.txt' found under directory 'docs/_build'
no previously-included directories found matching 'docs/_build/_sources'
zip_safe flag not set; analyzing archive contents...
pip.basecommand: module references __file__
pip.runner: module references __file__
pip.venv: module references __file__
pip.vcs.__init__: module references __file__
Adding pip 0.7.1 to easy-install.pth file
Installing pip script to /usr/local/bin

Installed /usr/local/lib/python2.6/dist-packages/pip-0.7.1-py2.6.egg
Processing dependencies for pip
Finished processing dependencies for pip

Install virtualenv

$ sudo pip install -U virtualenv
Downloading/unpacking virtualenv
  Downloading virtualenv-1.4.8.tar.gz (1.5Mb): 1.5Mb downloaded
  Running setup.py egg_info for package virtualenv
    warning: no previously-included files matching '*.*' found under directory 'docs/_templates'
Installing collected packages: virtualenv
  Running setup.py install for virtualenv
    warning: no previously-included files matching '*.*' found under directory 'docs/_templates'
    Installing virtualenv script to /usr/local/bin
Successfully installed virtualenv
Cleaning up...

Create a virtual environment

This command creates the isolated virtual environment. The --no-site-packages option tells virtualenv not to inherit any libraries from your existing Python installation's site-packages directory. For more information, see the virtualenv documentation on --no-site-packages.

Update 2010-04-13: Updated to use distribute because it is the new hotness. 

$ mkdir -p /srv/python-environments
$ cd /srv/python-environments

$ virtualenv --no-site-packages --distribute saltycrane
New python executable in saltycrane/bin/python
A globally installed setuptools was found (in /usr/lib/python2.6/dist-packages)
Use the --no-site-packages option to use distribute in the virtualenv.
Installing distribute..................................................................................................................................................................................done.

Clear the PYTHONPATH variable

I don't know if this is necessary, but I had a problem with the akismet module when this was set. Maybe I did something wrong, but when I cleared PYTHONPATH, it worked. 

$ export PYTHONPATH=

Install a package (Yolk) in the new virtual environment

Use pip's -E option to install a package to the specified virtual environment. (Yolk is a tool that lists Python packages.) (The working directory is /srv/python-environments.) 

$ pip install -E saltycrane yolk
Downloading/unpacking yolk
  Downloading yolk-0.4.1.tar.gz (80Kb): 80Kb downloaded
  Running setup.py egg_info for package yolk
Requirement already satisfied (use --upgrade to upgrade): distribute in ./saltycrane/lib/python2.6/site-packages/distribute-0.6.10-py2.6.egg (from yolk)
Installing collected packages: yolk
  Running setup.py install for yolk
    Installing yolk script to /srv/python-environments/saltycrane/bin
Successfully installed yolk
Cleaning up...

Activate a virtual environment and use it

  • Activate the saltycrane virtual environment (The working directory is /srv/python-environments.) 
    $ source saltycrane/bin/activate
  • Run yolk to list your installed packages: 
    $ yolk -l
Python          - 2.6.4        - active development (/usr/lib/python2.6/lib-dynload)
distribute      - 0.6.10       - active
pip             - 0.7.1        - active
wsgiref         - 0.1.2        - active development (/usr/lib/python2.6)
yolk            - 0.4.1        - active
  • Deactivate the environment 
    $ deactivate
  • Try running yolk 
    $ yolk -l
yolk: command not found

Install Django in the virtual environment

(The working directory is /srv/python-environments.) 
$ pip install -E saltycrane Django
Downloading/unpacking Django
  Downloading Django-1.1.1.tar.gz (5.6Mb): 5.6Mb downloaded
  Running setup.py egg_info for package Django
Installing collected packages: Django
  Running setup.py install for Django
    changing mode of build/scripts-2.6/django-admin.py from 644 to 755
    changing mode of /srv/python-environments/saltycrane/bin/django-admin.py to 755
Successfully installed Django
Cleaning up...

Create a requirements file from existing library versions using pip freeze

The pip freeze command allows you to take a snapshot of the exact versions of all your Python libraries. For more information, see the documentation on pip freeze. (The working directory is /srv/python-environments.) 

$ pip freeze -E saltycrane > requirements.txt

This creates /srv/python-environments/requirements.txt:

Django==1.1.1
distribute==0.6.10
wsgiref==0.1.2
yolk==0.4.1

Install libraries based on a requirements file

Once you have a requirements file, you can use pip to install the exact versions of the libraries specified in your requirements file. For more information, see the documentation on pip requirements files. Here's my requirements file for this blog, /srv/SaltyCrane/pip-requirements.txt:

Django==1.0.4
Markdown==2.0
Pygments==1.3.1
http://www.crummy.com/software/BeautifulSoup/download/3.x/BeautifulSoup-3.0.8.1.tar.gz
-e svn+http://django-tagging.googlecode.com/svn/trunk/@154#egg=django-tagging
-e hg+http://bitbucket.org/ubernostrum/django-contact-form/#egg=django-contact-form
-e hg+http://bitbucket.org/jezdez/akismet/#egg=akismet

Here's the command to install using my reqirements file: (The working directory is /srv/python-environments.) 

$ pip install -E saltycrane -r /srv/SaltyCrane/pip-requirements.txt
Downloading/unpacking http://www.crummy.com/software/BeautifulSoup/download/3.x/BeautifulSoup-3.0.8.1.tar.gz (from -r /srv/SaltyCrane/pip-requirements.txt (line 4))
  Downloading BeautifulSoup-3.0.8.1.tar.gz
  Running setup.py egg_info for package from http://www.crummy.com/software/BeautifulSoup/download/3.x/BeautifulSoup-3.0.8.1.tar.gz
Downloading/unpacking Markdown==2.0 (from -r /srv/SaltyCrane/pip-requirements.txt (line 2))
  Downloading Markdown-2.0.zip (93Kb): 93Kb downloaded
  Running setup.py egg_info for package Markdown
Obtaining django-tagging from svn+http://django-tagging.googlecode.com/svn/trunk/@154#egg=django-tagging (from -r /srv/SaltyCrane/pip-requirements.txt (line 5))
  Checking out http://django-tagging.googlecode.com/svn/trunk/ (to revision 154) to ./saltycrane/src/django-tagging
  Running setup.py egg_info for package django-tagging
Obtaining django-contact-form from hg+http://bitbucket.org/ubernostrum/django-contact-form/#egg=django-contact-form (from -r /srv/SaltyCrane/pip-requirements.txt (line 6))
  Cloning hg http://bitbucket.org/ubernostrum/django-contact-form/ to ./saltycrane/src/django-contact-form
  Running setup.py egg_info for package django-contact-form
Downloading/unpacking Django==1.0.4 (from -r /srv/SaltyCrane/pip-requirements.txt (line 1))
  Downloading Django-1.0.4.tar.gz (5.1Mb): 5.1Mb downloaded
  Running setup.py egg_info for package Django
Downloading/unpacking Pygments==1.3.1 (from -r /srv/SaltyCrane/pip-requirements.txt (line 3))
  Downloading Pygments-1.3.1.tar.gz (1.1Mb): 1.1Mb downloaded
  Running setup.py egg_info for package Pygments
Obtaining akismet from hg+http://bitbucket.org/jezdez/akismet/#egg=akismet (from -r /srv/SaltyCrane/pip-requirements.txt (line 7))
  Cloning hg http://bitbucket.org/jezdez/akismet/ to ./saltycrane/src/akismet
  Running setup.py egg_info for package akismet
Installing collected packages: akismet, BeautifulSoup, Django, django-contact-form, django-tagging, Markdown, Pygments
  Running setup.py develop for akismet
    Creating /srv/python-environments/saltycrane/lib/python2.6/site-packages/akismet.egg-link (link to .)
    Adding akismet 0.1.5 to easy-install.pth file
    
    Installed /srv/python-environments/saltycrane/src/akismet
  Running setup.py install for BeautifulSoup
  Found existing installation: Django 1.1.1
    Uninstalling Django:
      Successfully uninstalled Django
  Running setup.py install for Django
    changing mode of build/scripts-2.6/django-admin.py from 644 to 755
    changing mode of /srv/python-environments/saltycrane/bin/django-admin.py to 755
  Running setup.py develop for django-contact-form
    Creating /srv/python-environments/saltycrane/lib/python2.6/site-packages/django-contact-form.egg-link (link to .)
    Adding django-contact-form 0.3 to easy-install.pth file
    
    Installed /srv/python-environments/saltycrane/src/django-contact-form
  Running setup.py develop for django-tagging
    Creating /srv/python-environments/saltycrane/lib/python2.6/site-packages/tagging.egg-link (link to .)
    Adding tagging 0.3-pre to easy-install.pth file
    
    Installed /srv/python-environments/saltycrane/src/django-tagging
  Running setup.py install for Markdown
    changing mode of build/scripts-2.6/markdown.py from 644 to 755
    changing mode of /srv/python-environments/saltycrane/bin/markdown.py to 755
  Running setup.py install for Pygments
    Installing pygmentize script to /srv/python-environments/saltycrane/bin
Successfully installed akismet BeautifulSoup Django django-contact-form django-tagging Markdown Pygments
Cleaning up...

Use virtualenv with Django and mod_python

Here is how to use a virtualenv with mod_python. This is taken from Django, mod_python and virtualenv. For more information, see the virtualenv documentation on using virtualenv without bin/python.

  • Create a file /srv/SaltyCrane/myvirtualdjango.py: 
    activate_this = "/srv/python-environments/saltycrane/bin/activate_this.py"
execfile(activate_this, dict(__file__=activate_this))

from django.core.handlers.modpython import handler
  • Edit your httpd.conf 
        <Location "/">
        SetHandler python-program
        PythonHandler myvirtualdjango
        SetEnv DJANGO_SETTINGS_MODULE iwiwdsmi.settings
        PythonPath "['/srv/SaltyCrane',] + sys.path"
        PythonDebug Off
    </Location>

Use virtualenv with Django and mod_wsgi

Added 2009-09-27: Here is how I set up my virtualenv with mod_wsgi. To use the the packages in my virtualenv, I used site.addsitedir at the top of my .wsgi application file. You may also want to set the WSGIPythonHome variable in your httpd.conf file (outside of any VirtualHost sections). For detailed information on using mod_wsgi with virtualenv, see the Virtual Environments section of the modwsgi project documentation.

  • /srv/SaltyCrane/saltycrane.wsgi: 
    import os
import sys
import site

site.addsitedir('/srv/python-environments/saltycrane/lib/python2.5/site-packages')

os.environ['DJANGO_SETTINGS_MODULE'] = 'iwiwdsmi.settings'

sys.path.append('/srv/SaltyCrane')

import django.core.handlers.wsgi
application = django.core.handlers.wsgi.WSGIHandler()
  • httpd.conf: 
        WSGIScriptAlias / /srv/SaltyCrane/saltycrane.wsgi

17 Comments — feed icon Comments feed for this post


#1 Alexander Artemenko commented on 2009-05-06:

By the way, if you'll need something more complex, than just installing applications in the virtual environment, I suggest to look at zc.buildout.

I'm already moving in that direction, I use buildout to deploy my django sites.


#2 Eliot commented on 2009-05-07:

Hi Alexander, thanks for the tip. I had noticed Jacob Kaplan-Moss' post about zc.buildout, but didn't read it carefully. I think I had already made up my mind to use pip+virtualenv. I will keep my eye on zc.buildout.


#3 Skylar commented on 2009-12-22:

Nice article, I love virtualenv. I'm looking for a way to install Django==dev from a requirements file but so far no luck. I guess I could point the requirements file at the github mirror or directly at svn trunk. I thought there was an easy way to denote this ... </ramble>


#4 Tim commented on 2009-12-26:

Skylar:

-e svn+http://code.djangoproject.com/svn/django/trunk#egg=Django

That what you're after?


#5 Joe commented on 2010-04-10:

Great and detailed writeup, thank you for posting it. It was nice to be able to quickly read through how PIP created and installed into a new virtualenv.


#6 Lord Daedra commented on 2010-06-04:

Thanks for good article!

Also there is nginx+uwsgi, good for Django deployment..


#7 Eliot commented on 2010-06-04:

Lord Daedra: I was just reading an article on a benchmark of WSGI servers (http://nichol.as/benchmark-of-python-web-servers). Nginx+uWSGI looks pretty good.


#8 David Talbot commented on 2010-09-07:

Really good stuff - thanks so much for putting this all together. I'm putting together a site at the moment and this helped me out no end.


#9 Dana W commented on 2010-10-13:

If you like PIP and VirtualEnv, you should try Virtualenvwrapper, it's the shizz.

http://www.doughellmann.com/docs/virtualenvwrapper/

None of this crap:

source saltycrane/bin/activate

... you can do this instead:

workon saltycrane

Check it out.


#10 Eliot commented on 2010-10-14:

Dana: I started using virtualenvwrapper a few months ago because I needed to switch environments more often and because everyone at work was using it. I agree it's pretty great.


#11 Jose commented on 2010-11-26:

Excellent post, has been very useful for me.

Especially the part about django and virtualenv.

Thanks!


#12 James commented on 2010-12-04:

I'm using django + wsgi, and I'm about to start using virtualenv. I understand that --no-site-packages creates a bare virtualenv to start from, and that you can add packages to that env. But when you create the wsgi, you add the env's site-packages folder to the current path.

Does that mean that packages installed in the system-level default python installation are still available to your wsgi app? So if I install django 1.0 within the system, then create a bare virtualenv with nothing in it, the wsgi app would have access to django 1.0?

Just curious. Great writeup!


#13 Eliot commented on 2010-12-05:

James:
I actually ran into this recently. I think you are correct. See this page on the modwsgi wiki for much more information on this: http://code.google.com/p/modwsgi/wiki/VirtualEnvironments It suggests creating a baseline virtual environment and setting WSGIPythonHome to point to it. Doing this means you will not use the default system python installation. Then you can add your site-specific virtualenv to this baseline. Hope that helps.


#14 John Chase commented on 2011-08-13:

Thumbs up for virtualenvwrapper.sh.


#15 Brandon commented on 2011-10-15:

Hi Eliot,

Just wanted to thank you for pulling these notes together -- they were very invaluable.

Since I don't see it mentioned anywhere else, I might as well point out something that I've just (re)learned (quoting from http://guide.python-distribute.org/pip.html):

"You can install directly from a tarball or zip file, as long as there is a working setup.py file in the root directory of the unzipped contents."

This means that if you are trying to install a file with pip that is giving you trouble due to its eggy nature (for instance, I had been trying to install nltk), there could be another way around it. Assuming your directory structure and the current nltk package:

sudo pip install -E saltycrane http://nltk.googlecode.com/files/nltk-2.0.1rc1.zip

works very nicely (assuming, in this case, that pyyaml has been installed). And of course it needn't be nltk.

Hope this might help another beginner to virtualenv.

Brandon


#16 Fernando Irías Escher commented on 2011-11-02:

Thanks! Pretty useful guide!


#17 Yulia commented on 2011-12-03:

Thank you a lot! Your tutorial is in fact better than Virtualenv documentation.

Post a comment
Required
Required, but not displayed
Optional

Format using Markdown. (No HTML.)
  • Code blocks: prefix each line by at least 4 spaces or 1 tab (and a blank line before and after)
  • Code span: surround with backticks
  • Blockquotes: prefix lines to be quoted with >
  • Links: <URL>
  • Links w/ description: [description](URL)
Created with Django | Hosted by Slicehost