Setting up a python CI server with Hudson

Installing Hudson

I start with a basic virtual machine – in this case, I’m using Ubuntu Server 9.10. Once you have a basic machine installed, I recommend you make sure everything’s up to date with patches.

sudo apt-get update
sudo apt-get dist-upgrade


If you haven’t already, consider installing an ssh server for remote login. You don’t need it for hudson, but I find it makes administration easier.

sudo apt-get install openssh-server

To make installing python packages and pieces a bit easier:

sudo apt-get install build-essential python-dev

Start off by getting a copy of hudson.

wget http://hudson-ci.org/latest/hudson.war

Hudson runs fine from that war directly – a darned handy way to make the whole system work. You can also install this in Tomcat, JBoss, or whatever other servlet container you like to run. I find it easiest to run directly from the war file.

If you’re going using Mercurial or Git, you’ll need to install the appropriate version control client onto your machine as well. The plugins use those systems through the command line interface.

sudo apt-get install mercurial
sudo apt-get install git-core
sudo apt-get install subversion


Since hudson is a java application, we need to have java installed… Ubuntu 9.10 comes with OpenJDK which works just fine. You can also download and install the Sun JDK if you want to, but it doesn’t really seem to be required. If you don’t have any JDK already installed, you can generally get OpenJDK using:

sudo apt-get install openjdk-6-jre-headless

To get rolling with Hudson, just fire it up:

java -jar hudson.war

I just run this as the base user – no special privileges. It creates all it’s working directories, workspaces, etc in a “.hudson” directory. You can do the extra work to make this a unix daemon too – I haven’t bothered because it’s been relatively stable. There are some good notes on doing this at http://weblogs.java.net/blog/2009/02/10/hudson-now-good-behaving-unix-daemon and http://wiki.hudson-ci.org/display/HUDSON/Installing+Hudson+as+a+Unix+daemon, so I won’t dive into those details here.

Once you’ve started it, bring up the UI in a browser to make sure it’s all working. My virtual machine is running at 192.168.0.2, so I open up http://192.168.0.2:8080/. I’ll use this same IP address through the course of this walk-through. Of course you can replace it with a hostname.

I generally start off by making sure we have the latest Hudson installed and getting the various client plugins I want to use all installed – click on “Manage Hudson” (http://192.168.0.2:8080/manage)

If there’s a more recent version of Hudson available, it will show at the very top of this page and give you the option to download and install it. You’ll be able to upgrade plugins the same way – Hudson knows to look at it’s own site for the latest plugins and versions, so it’s very easy to maintain. If you’ve been following along with this, the “wget” statement up above will have likely brought down the latest version already, so you won’t see this sort of screen until a new update comes along.

The upgrade automatically mechanism has worked great for me, but a couple of caveats are needed here – the process needs to be running with permissions to the directory in which it’s located to work. The other is once you’ve upgraded the system, you may need to restart the service from the command line to enable future downloads. I’m not sure if I ran into a bug with that, or if that’s expected behavior – but I haven’t run into it often.

Now you should have the latest version – either directly or just upgraded. Lets get to the plugins – because that’s where a lot of the Hudson goodness really lives. You can get to the list of installed plugins so you can see what’s there by default. From the “Manage Hudson” page, click on the link to “Manage Plugins”. The URL for that page is http://192.168.0.2:8080/pluginManager/

There are four tabs here to help you see what’s available – the first tab Updates is the default, and shows any outdated plugins. The second is Available, that shows what’s linked up from the central Hudson plugin list, and Installed, which shows the bits that are already there. Click on the Available tab (url is http://192.168.0.2:8080/pluginManager/available) and you can see the plethora of interesting things ready to use. The ones I like to install for my python projects are:

• Cobertura Plugin
• Git Plugin
• Mercurial Plugin
• Monitoring Plugin
• Python Plugin
• Violations Plugin

All of these plugins will still require you to have whatever local libraries you need to make them work. That’s why we installed Mercurial, Subversion, Git, etc earlier – to make sure those tools were available. Hudson invokes them on the command line on your behalf to make it all work.

Below is a broad set of libraries that cover a lot of pieces – you may want to install more/different libraries to accomplish your aims with Hudson. I’ll be aiming to create some typical build reports with it all – unit test output, coverage, etc.

sudo apt-get install python-setuptools
sudo easy_install coverage
sudo easy_install pylint
sudo easy_install unittest-xml-reporting
sudo easy_install fabric
sudo easy_install nose


Setting up a new Job

Now we just need a project. Let’s start with an open-source project – Pygments is a great project, with lots of diverse things happening within it. And it’s got tests and will be a good general victim for this writeup.

Start out by clicking on “New Job” (http://192.168.0.2:8080/view/All/newJob). A Job in Hudson is a build process. I’m going to make a fairly simple one, but you can cascade multiple jobs together into a far more complex build process if you want to… I’m not going to delve into that here, focusing instead on getting a single python project rolling.

In the new job form:

• set the job name to “pygments”
• choose “Build a free-style software project”
• I also put in a description (it’s not mandatory)
• then choose “Mercurial” from the Source Code management section

• put in “http://dev.pocoo.org/hg/pygments-main” for Repository URL
• put in “trunk” for the Branch
• click the “Save” button down at the bottom

Now you’ve got a job – we can come back and configure this at any time, and we’ll do more tweaking as we go through this. Let’s make sure we have the basics all working first though…

At this point, you should be at the project pygments page in Hudson. If you’re not, you can navigate there from the front page, or go directly to the URL (http://192.168.0.2:8080/view/All/job/pygments/). The page should look something like this:

On the left hand side of the page are the controls for the job, including Configure and Build Now among a list of other options.

• click on “Build Now”

You should see the build start processing with a little rolling progress bar on the left hand side of the window. You can click on that link that looks like #1 Nov 5, 2009 8:35:18 AM’ in the Build History box.

Either immediately, or when the build reports complete, click on that link.

You can see the console output from the build (which is just pulling down the source) with the link Console Output. The output of the build is exactly what happened on the command line. In this case, you’ll see something akin to:

Started by user anonymous
$ hg clone -r trunk http://dev.pocoo.org/hg/pygments-main /home/ubuntu/.hudson/jobs/pygments/workspace
requesting all changes
adding changesets
adding manifests
adding file changes
added 903 changesets with 2247 changes to 269 files
updating working directory
249 files updated, 0 files merged, 0 files removed, 0 files unresolved
[workspace] $ hg log -r . --template {node}
Finished: SUCCESS

Once you’ve done an initial build, the files will also be available for you to look at and download through the Hudson UI. If you click on the link Back to Project or just the project name at the top of the page, you should see a new reference in the project called Workspace. Clicking into that will show you all the files that were downloaded and any other results of the build process. Right now we’re just getting files from source control, but as we add build steps to the process, the results will be available to dig around in at the workspace.

If you want to set up Hudson to check for updates and build when it finds them, you can go back to the project page and click on the Configure link. Once in there, scroll down in the configuration and enable the checkbox labelled “Poll SCM”. As soon as you do, another text field will come available that will let you specify a Cron-like string to identify when the builds should check source control. In the screenshot below, I’ve enabled Hudson to check the mercurial repository every 10 minutes.

…

Any time you change the configuration, go to the bottom of the Configure page and click on “Save” to set the changes.

Adding Build Steps – doing ‘work’

Right now, the build isn’t doing too much – let’s see about getting it doing some more work for us. The Pygments source code is set up with a Makefile to make doing this a bit easier, so we’ll take advantage of that. If you’re not in the Job configuration page – get there. Go to the job an click on the Configure link. Now we’ll add a build step that simple invokes the shell command “make test”.

Scroll down in the configuration page until you see the section labelled Build. Click on the button underneath there labelled Add build step, and choose Execute shell. When the text field for that script appears, put in the text make test.

Scroll to the bottom of the page, save the configuration, and invoke “Build Now” to run the updated build. Looking at the console output for the build will now show the updated console log with the running tests, like the screen shot below.

Should the built in unit tests fail, the system will show a red indicator on the hudson front page and next to the build results link.

Displaying unit test results

What this isn’t doing it making some pretty graphs and charts. Let’s see about getting some of that enabled.

Pygments uses Nose tests for it’s unit tests. That’s pretty nice, because Nose includes a mechanism to output the format of the tests into Java-based JUnit format, and Hudson in turn knows how to make pretty pictures from that format. Since we installed the nose library earlier, we can take advantage of it. Probably the “right” way to do this would be to change the Makefile to invoke the tests that output XML for the unit tests. For right now, we’ll just shim it into place with new build steps.

• Change the content of the Execute shell build step and replace it with
  python tests/run.py --with-xunit
• save the configuration and invoke “Build Now”
• now look in the workspace in you should find a new file called “nosetests.xml” in the tests subdirectory.

Let’s get that graphed…

• Go to the job configuration and enable “Publish JUnit test result report”. It will have a text field so that you can tell it where to find the results. It uses an Ant format for finding filenames in an existing directory structure.
• In this case use **/tests/nosetests.xml
• Save the config
• invoke Build Now

Now you’ll see a new element in the web user interface called “Latest Test Result” which you can dig into. If you invoke “Build Now” again, it will start graphing the results of the tests and making that trend available in the project view. Right now this graph is going to be really darned boring, because there aren’t any changes between builds. Once more code starts rolling in, you’ll see changes with the number of tests being invoked. You can also click on that link and see the tests that were invoked. For python folks, remember that we’re shimming Python unit tests into a JUnit conceptual framework, so there are going to be some “leaky abstractions here”. In particular, test names, classes, and such may not alway match up with expectations. I have more details on how to enable XML output for a Django based test runner that I’ll post in another write up… uh, later.

Add cobertura style test coverage reporting:

Now we get to use Ned Batchelder’s coverage library to see some cool stuff!

(This whole write up is basically a thank you to Ned for writing Coverage)

Go back to the configuration for the job

• change the Execute Shell build step to read:

coverage run tests/run.py --with-xunit
coverage xml


• scroll down and enable  the check box Publish Cobertura Coverage Report
• use the pattern **/coverage.xml

Add pylint reporting

Head back to the job configuration

• add a new Execute shell build step  to read:

#!/bin/bash
pylint --rcfile scripts/pylintrc -f parseable pygments > pylint.txt
echo "pylint complete"


• It’s important to start this with #!/bin/bash, and then end with the echo statement so that the return codes from pylint don’t get interpreted as a build failure (unless you want that… I don’t know about you, but I’m not pythonic enough to have a 100% pylint result.)
• enable the checkbox Report Violations
• under pylint, use XML filename pattern of **/pylint.txt

• save the configuration and invoke “Build Now”

So there you have it – a python project building under hudson with unit tests getting run, cobertura style coverage reports, and pylint style reporting.