I know I’ve been quiet on the Django front for a while. Here’s something to make up for it. Without further ado, let me present to the world the Django Queue Service.
At OSCON 2007, I was hunting around for something to use to deal with background processing initiated from my web application. Not finding anything that I could immediately use and implement, I took it as a challenge to knock out something in the time I was there. The result is the django queue service.
It was later in the week (at the Django BOF) that I learned about TheSchwartz and Gearman, which Brad Fitzpatrick is hacking away it to make a queue mechanism available for fellow Python and Ruby geeks.
At the time, I thought “I wonder if I can hack this out in a week…” Well, yes I could. I modeled the service after the REST API from the Amazon Simple Queue Service, because that seemed both reasonable and effective. And I didn’t mind the idea of having my own Queue that (mostly) matched that API that I didn’t have to pay anything to use.
The project also contains from code from the CherryPy project. Mark Ramm (a really nice fellow, by the way) gave a session at OSCON about WSGI servers that was really intriguing. I’ve been looking for a “pure python” service to run Django with for a while, and after chatting a bit with him, thought I could probably make it happen with the CherryPy WSGI server. So I did. I have a little piece of linkage code that fires up the CherryPy WSGI server code and links it right on through into Django. I didn’t want to use the rest of CherryPy, so I nicked that code and dropped it into this project as a nice reference and a handy way to run the queue.
I suppose this project could easily end here, and of course there was the “but I want to…” list of things that I had in mind to do before I released it into the world. Well – as they saying goes: “Now is better than never.” so out it goes.
If you are interested in getting involved with the project, sending me patches, or whatever – let me know. You can leave a comment here or use that spiffy Google Code Hosting issue tracker. The project is functional, but not brilliant – an not entirely cleaned up in terms of its API. But enough excuses – its there, feel free to use it or not as you like, contribute or not as you like.
Rhonabwy 
Nice! How are you using this?
LikeLike
Right now I’m using it to start longer running background processes scanning machines in a data center to get updated metrics and associated information. The results are tossed into the database (using Django, naturally), along with a last updated time stamp – so I can see what’s happened and when.
LikeLike
I am new to Python, But when I was supposed to run those longer processes using Django, I used threading module…used to make threads for each request. and update the page with the result using JSON.
So using Django Queue Service would I be able to run and enqueue processes??
LikeLike