Functional testing with Lettuce and Fabric

After the launch of Nebula, from which I think I’m mostly recovered, I’ve been thinking about and working on ways to allow us to accelerate development. Our team has done a fair bit of work on the Openstack Dashboard, an incubating project within OpenStack, and we have a couple of internal http based interfaces for our own pieces.

We have unit testing up and running, and like most projects a few of those tests are more “functional” or “integration” tests than actually unit tests, but really the project’s been pretty darn good about keeping those separate. As it turns out, the Junit framework is truly awesome for fast, quick tests – but when you start to step into functional testing you want just a touch more linearity to the testing. The unit test frameworks that I’ve run into (nosetests, unittest2) don’t have much in their mechanisms that allow you to set up longer, repeated sequences or flows, so it was time to look around for what else might work there. (sidenote: If you know of some mechanisms that do longer, sequential tests and are based around unittest2 or nose, I’d love it hear about it! I’ll freely admit to not having searched every corner there…)

I’ve been watching and reading quite a bit on BDD and the cucumber framework for testing applications, so I thought it was time to try it out. Of course, being more of a python geek than a ruby geek, I’d heard about Lettuce, which is either a straight up port, or incredibly headily inspired by Cucumber.

Getting started with it was incredibly easy. Everything we’re doing is working from a virtualenv (which I highly recommend for development!), so I just added lettuce to the pip-requires list and it was pretty much ready to go.

In setting up lettuce, I probably spent as much time knocking together a quick “functional_tests.py” script as anything – making it responsible for the sequence of testing. The whole point there was to make a single script that could be run, returning an exit code that meant success or failure for the whole set, so it could be plugged into our local instance of Jenkins.

The core of that fabric script is really just simple fabric commands to set up, clean, and push code to a remote node. I ended up adding a script to our local repository to start up services on the far-end for functional testing – testsetup.sh – that runs the various pieces under “screen” – which makes it super easy to see what’s happened when something goes awry.

You’ll notice at the very end of that functional test script, I invoke “lettuce” on the local machine. That takes care of running the actual test sequences. The documentation for lettuce is pretty reasonable, so I recommend walking through that tutorial for a quick understanding. The super-basic gists is that lettuce uses a DSL that you basically create as you go along, and ends up being rather highly customized to your testing setup. You put together files in a “features” directory, and they use “steps.py” in that same directory to do what’s needed. The pattern that I’ve been using (and which seemed to be implied) was to use the “world” object in lettuce as a sort of black-board to get and store data as you walk through the steps in lettuce. There’s deeper integration into Django based projects (my example was using a little Flask based application run with gunicorn).

The key to setting up the functional testing was taking advantage of the file “terrain.py“, which lettuce uses to do global setup, and setup/teardown around steps, scenarios, features, or the whole shebang. Turns out I had several options for how to reach out and walk through the remote application, so I chose to use twill, although I could clearly have used the local test_client() with Flask, or even the web_driver setup from Selenium to drive Firefox.

One of the nicer things that I found while walking through setting this up was that I could write up a feature file and run lettuce, and it gave me really pleasant “this isn’t implemented yet” output that I could use to expand steps.py. For example, I created “example.feature” in the features directory:

and then ran “lettuce” and got the following output:

As you can see, it’s basically giving you the code to add to steps.py to make this work. As I went, I found myself refactoring steps.py to use some of lettuce more advanced mechanisms – regular expressions and grouping steps together in sequences. And to make sure I don’t leave you completely in the dark about setting up and using steps.py, here’s a snippet from my growing file.