Post launch

Objectively and now a few days past the launch, it went well. During the time it was happening? Didn’t feel like that so much until Thursday afternoon and Friday.

We ran into a failure during our final validation runs, and it opened up a potentially nasty can of demon worms. The basic functionality of a ticketing system isn’t usually the problem – it’s all the tendrils and automation into other components in the organization that are such a pain. In this case, the system that took in email and created tickets from that showed a rather nasty flaw at the last minute. Nasty because it was so damned inconsistent.

Even though we opened a “Pri 1” bug with our vendor, they didn’t seem capable of getting a developer to look at the code and tell us what the hell was going on. After about 6 hours on the phone with various support technicians, I was personally convinced that they solved the “how to do this problem” in perhaps the most complex and ass-backwards way possible. It spurred us hard to go find all the key elements that were sending in email (quite a trick) to create tickets – which we did, and found that all but one of them were working for a launch. So we did what we probably should have done and created our own mail processor – python and the email library makes that brutally simple, and from there we just push in tickets into the system through their web services (ugh – SOAP) interface. I’ve become reasonably competent at using python suds now as well – it’s a pretty reasonable SOAP library for the basics. It is maybe 3-4 days work all told to get that all in place, extend our REST API interface that we’ve wrapped around their crappy SOAP one, and make sure it passes a battery of half-way sane tests. We’re still finishing that work up now, but it should be all out and rolling early next week.

So we pulled back from launching on Tuesday and ended up launched on Thursday morning. The best part of this launch was getting the deluge of complaints Friday about something not being bolded, the default sort order being wrong for this or that person, and the general user experience model that the application provides (it really likes to use pop-up windows). It absolutely made my day, because that’s all stuff that can be fixed pretty easily – and nothing in it was a major “I can’t do my job now…” kind of thing. My team successfully did an engine transplant on the organization while we kept on running down the road. I expect we’ll falter and hesitate a few days from this – it’s pretty deeply embedded in the org – but we pulled it off.

I think the thing that stands out most profoundly was a tidbit in the kudo’s/Thank you note that my VP sent out about the project:

I talked to the lead Program Manager from ENTERPRISE-VENDOR yesterday about our implementation and he told me that he hadn’t seen a better one in the past 4 years.  Apparently we did in 3 months what most companies take a full year to do!

Not too damn shabby.