SaltyCrane Blog — Notes on JavaScript and web development

Somewhere on your Python path

Date:  |  Modified:  |  Tags: install_setup, python, ubuntu

As I install new python packages, I sometimes see instructions which say something like "check out the code, and place it somewhere on your Python path". These are very simple instructions, but since it is not automatic like a Windows installer, or Ubuntu's package management system, it causes me to pause. Where on my Python path should I put it? I could put all my packages in random places and update my PYTHONPATH environment variable every time. I also thought about putting new packages in Python's site-packages directory. This is probably a good option. However, I tend to like to have all my important stuff in my home directory so I can easier maintain it across multiple machines. (Also, I forget where the site-packages lives (it is /usr/lib/python2.5/site-packages on Ubuntu Hardy).) So my solution was to create my own python-packages directory in ~/lib. I set the PYTHONPATH in my ~/.bashrc as follows:

export PYTHONPATH=$HOME/lib/python-packages
and then put all my Python packages here.



Update: Some Python packages are distributed with other stuff besides the actual Python package. To handle these cases, I created a dist directory inside my python-packages directory, and created symbolic links from the actual package directory in dist to python-packages. Finally, I made the entire python-packages directory a Mercurial repository so I can finely control my Python environment and easily maintain it across multiple machines. Here's what my ~/lib/python-packages currently looks like:

drwxr-xr-x 9 sofeng sofeng 4096 2008 08/05 21:40 dist
drwxr-xr-x 4 sofeng sofeng 4096 2008 08/05 21:31 django_openidconsumer
lrwxrwxrwx 1 sofeng sofeng   43 2008 08/05 21:31 elementtree -> dist/elementtree-1.2.6-20050316/elementtree
lrwxrwxrwx 1 sofeng sofeng   31 2008 08/05 21:31 openid -> dist/python-openid-1.2.0/openid
lrwxrwxrwx 1 sofeng sofeng   31 2008 08/05 21:31 openid2.2 -> dist/python-openid-2.2.1/openid
lrwxrwxrwx 1 sofeng sofeng   27 2008 08/05 21:41 pygments -> dist/Pygments-0.10/pygments
lrwxrwxrwx 1 sofeng sofeng   29 2008 08/05 21:31 urljr -> dist/python-urljr-1.0.1/urljr
lrwxrwxrwx 1 sofeng sofeng   29 2008 08/05 21:31 yadis -> dist/python-yadis-1.1.0/yadis

Update 2008-09-14: Here is a post on the django-developers mailing list by Kevin Teague which explains the large number of technologies related to Python package management and deployment including PyPi, Distutils, Eggs, Easy Install, VirtualEnv, and Buildout. Kevin admits that package management and deployment is an area in Python where there is room for a great deal of improvemnt. He notes that the symlinking method that I use can work for simple needs, but it fails for more complicated use cases, such as tracking package dependencies. The new Virtualenv and Buildout technologies seem to be interesting-- I will have to check them out when I have time. I found this link via Simon Willison

Update 2008-09-24: Ian Bicking, author of Virtualenv, has just released pyinstall which seems to be an improved easy_install. I have not tried it yet, but I believe Ian Bicking writes good code.

Update 2008-10-24:Glyph Lefkowitz, lead architect of Twisted, suggests using twisted.python.modules for solving Python Path Programming Problems. This looks like something I could use in my current project-- I just wish I understood it.

Update 2008-12-16:

Comments