Interactive sandboxes: using IPython with virtualenv

A very helpful blog post on IPython and virtualenv by Pedro Algarvio inspired this one. The advice found there takes you 90% to where you want. I'll recap on that 90% but explain and give the extra 10%. I am indebted to Pedro for laying down all the hard work.

First of all, if you are unfamiliar with Ian Bicking's virtualenv package, you should know two things about it:

1. virtualenv allows you to develop in sane, aseptic, "sandbox" development environments, switch between them seamlessly, and maintain harmonious order in your Python universe.
2. virtualenv is certifiably awesome. Proceed directly to installing it (especially in combination with pip)! Do not pass Go! Do not collect $200!

Arthur Koziel already wrote a really good tutorial on using virtualenv, and, in fact, you'll probably find working with Doug Hellman's excellent virtualenvwrapper more convenient; in this case, Doug already wrote an excellent virtualenvwrapper tutorial, too. I've mentioned IPython in a previous blog post, so I won't cover that here, either. Instead, let's cut to the chase and get IPython and virtualenv playing well together.

Ordinarily, IPython, commonly installed system-wide by your preferred package management system, remains oblivious of an activated virtualenv environment, and will just mill about importing packages and modules from the system, rather than the sandbox. This gives two obvious solutions: either 1) configure the system installation of IPython to work with virtualenv, or 2) install IPython in each virtualenv environment. Doug Hellman wrote a nice tutorial on doing the latter approach; here, we'll focus on the former, which I prefer, since it means having to only install IPython once.

IPython (being a Python program) can read and execute Python scripts during launch; we'll use this mechanism to modify IPython's launch to hook into the virtualenv environment we're currently in. First, we'll tell IPython that we want to execute some code in a at startup. If we go to the $HOME/.ipython/ directory, we'll find a file called ipy_user_conf.py. Open the file in your editor of choice, locate the function main(), and at the within that function (I suggest at the end), insert the following line:

execf('~/.ipython/virtualenv.py')

Next, we need to create this file. Still in the $HOME/.ipython/ directory, create a new file called virtualenv.py and open it with your editor. Next, add these contents to this file:

import site
from os import environ
from os.path import join
import sys

if 'VIRTUAL_ENV' in environ:
    virtual_env = join(environ.get('VIRTUAL_ENV'),
                       'lib',
                       'python%d.%d' % sys.version_info[:2],
                       'site-packages')

    # Remember original sys.path.
    prev_sys_path = list(sys.path)
    site.addsitedir(virtual_env)

    # Reorder sys.path so new directories at the front.
    new_sys_path = []
    for item in list(sys.path):
        if item not in prev_sys_path:
            new_sys_path.append(item)
            sys.path.remove(item)
    sys.path[1:1] = new_sys_path

    print 'VIRTUAL_ENV ->', virtual_env
    del virtual_env

del site, environ, join, sys

If you took a look at Pedro's version of virtualenv.py, you'll recognize most of his code here. The important difference lies in the trickery we play with sys.path in lines 12 through 22. These lines were inspired by a solution to a problem presented by using site.addsitedir(), which adds new paths only to the end of sys.path.

Adding paths to the end of sys.path has, for our purposes, the undesirable side-effect of allowing system-wide packages and modules to preempt locally installed ones, since Python searches through sys.path for modules and packages in first-to-last order. I have filed a feature request for site.addsitedir() to allow inserting new paths at the beginning of sys.path; in the meantime, we'll use this hack inspired by the modwsgi programmers, which keeps track of the paths before and after the call to site.addsitedir(), then swaps the position of the new paths from the end, to just after the first element, '', which represents the current working directory (which should preempt every other path).

IPython will have access to the contents of the virtualenv sandbox in which you're currently working. For example, if I activate my networkx virtual environment, which has the latest development version of the NetworkX graph library, then fire up IPython, I get the following result (note the line that begins with VIRTUALENV indicating I'm accessing the virtualenv sandbox):

(networkx)$ ipython
VIRTUAL_ENV -> /home/lasher/.virtualenvs/networkx/lib/python2.6/site-packages
Python 2.6.2 (release26-maint, Apr 19 2009, 01:56:41) 
Type "copyright", "credits" or "license" for more information.

IPython 0.9.1 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object'. ?object also works, ?? prints more.

In [1]: import networkx

In [2]: networkx.__version__
Out[2]: '1.1.dev1518'

When I leave the sandbox (e.g., by using virtualenvwapper's deactivate command), I return to accessing the system-wide default install of NetworkX:

$ ipython
/var/lib/python-support/python2.6/IPython/Magic.py:38: DeprecationWarning: the sets module is deprecated
  from sets import Set
Python 2.6.2 (release26-maint, Apr 19 2009, 01:56:41) 
Type "copyright", "credits" or "license" for more information.

IPython 0.9.1 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object'. ?object also works, ?? prints more.

In [1]: import networkx

In [2]: networkx.__version__
Out[2]: '0.36'

So there you have it: one IPython to rule all your virtualenv sandboxes!