Building the high spires – enabling elastic cloud computing

Somewhat related to my previous post – I’ve been thinking a lot about packaging lately. At work, I’ve jumped into a team that’s responsible for a whole bunch of infrastructure – envisioning it, building it, and running it.

Two months ago I thought I’d jump in and start building some really awesome spires. Spires in a metaphorical sense anyway – I guess if I was really sticking to an architectural theme, it should be something else – something more broadly useful but still fantastically impressive. I found myself focusing on the infrastructure instead. The infrastructure wouldn’t support what I wanted to build. It’s missing pieces – pieces at the very base of the whole setup. So I’m mucking around in the foundations, shoring up some systems, adding others – getting the basics nailed down with my team to build some of these cool spires. And thinking. A lot of thinking – thinking about what’s most critical, what provides the most value, and what needs to be done in what order to make this team really successful.

One of these foundational pieces is packaging – application packaging, to be specific. The more I think about it, the more I think I need to do something about it – but outside of the walls of my job. The reason for going outside my job for this? I honestly think the world needs another application packaging standard.

I could build a company standard for application packaging. I’m pretty convinced of it anyway… but I don’t think it would be successful in a long term (i.e. more than 1-2 years). For an application packaging standard to be really effective, it has to have a broad support base, will require a lot of effort, and for an effective standard I philosophically believe it must be an open standard.

The use case that’s driving this thinking is being able to programmatically deploy new servers, drop down applications, and configure them in a very fast time frame (in a manner of minutes). The use case is one of the classic jet-pack and flying cars dreams of elastic/cloud computing.

There are some enterprise tool chains that do this today. Sort of. If you squint real hard. Most of these tools are born from the world of policy based computing, IT control and compliance, and configuration management. They don’t match up with the use cases real well, and the enterprise software solutions in this space and horrifically and tragically expensive. I can’t even blame them – clearly that’s where the money’s been for these tools.

The marketplace is pushing – there’s a need, there’s a HUGE opportunity here. It’s where applications become components that can be, and are, programmatically deployed and controlled. It’s the next step up in the abstraction layer for the systems we use every day.

There is some work towards solutions to this problem today. It’s not like I’m the only guy seeing this problem (if I was, I’d likely be delusional).

  • RightScale has a great product that layers on top of Amazon’s EC2. It’s damned impressive.
  • 3Tera has a tool chain that starts to deal treat virtual machines as packages to be deployed, connected, and replicated at need.
  • Parallels has created, which is really the closest anyone’s come to what I’m aiming at – but even it’s not quite there – not to mention copyrighted and proprietary.
  • VMware is making some aggressive moves into the tools space as well, propagating the OVF standard (with the DMTF) and the concepts that we’ll be deploying VMs, not applications, in the very near future. I’m not sure they’re wrong – but I think they’re missing half the picture.
  • The DMTF has standards that are skipping around the ideas, but not quite nailing them on the head.
  • In the open source world Puppet and CfEngine are the kingpins – programmatically defining infrastructure. They fail to take the next obvious step into application deployment – but I think only because they lack the infrastructure components needed to do it in an effective way.

I think the solution to this problem is an application packaging standard that includes configuration and parameters into its metadata.

Like other packaging standards, the packaging needs to:

  • allow the code to be dropped into place
  • define dependencies that the application requires
  • have a common and open format that can be stored and shared in open repositories

What it needs in addition is:

  • the package should contain parameters (configuration elements) that can be programmatically determined
  • once determined, the application should be able to be deployed with parameterized configuration – details updated at deploy time
  • ideally this will enable the capability to update that same configuration programmatically, idempotently and remotely

What I think it needs to be successful:

  • An open format with at least one, preferably multiple, implementations enabling that standard.
  • A base set of common applications packaged in this format, available for immediate use
  • a tool to create the packages and verify they’re ‘good’
  • a tool to deploy the packages overlaying configurations – at least on a local basis
  • a public repository of common packages to build a dependency library upon
  • a way to copy/mirror that repository cheaply and easily in any organization to reduce cost of uptake
  • a damned impressive example solution that takes advantage of the kit

In short, a crap-ton of work. For a company to want to go do these things, they’ve have to be pretty damn visionary. And I’d expect that they’d want to keep this kind of IP to themselves. I need this kind of solution, the economic times are rough to chase visionary dreams outside of the core competencies of the job, and I think a successful solution would benefit everybody. To me – this whole setup screams “open source project” to make it work.

So now I’ve defined it – put it out there.

Are you interested in making something like this? Contributing to a open solution? Willing to put some sweat equity behind a project that could revolutionize things, but is essentially building bricks for a foundation?

I have some projects on my plate right now that I can’t just drop – so it’ll be a month or two before I start pursuing this one. Let me know if you’re interested… haven’t even decided on the structure it’ll need around it to work, but I’m heading in that direction.

(image used with permission:

Published by heckj

Developer, author, and life-long student. Writes online at

3 thoughts on “Building the high spires – enabling elastic cloud computing

  1. Joe,

    We had a real mess at work with our development teams a few of years back. Everybody was grabbing code from other groups, building it on their own boxes, and shipping their own “custom” builds. It was a real mess.

    In the end we needed a packaging scheme that would work on Linux and Windows so teams could package and release their components for use by other groups. It was decided to use RPM on Linux and INNO Setup on Windows (I would have picked MSI’s, but INNO works quite well.)

    To describe dependencies YML was used. To parse and install the packages using the YML file a Perl script was written to download and install the packages, and a Ruby script was used to create the initial packages, using either RPM or INNO, depending on platform, again driven by a YML file.

    This has made the development process an absolute dream.

    I know it’s not quite as polished as you’re looking for, but it may point you in the right direction, and most of the technologies used are completely open.


  2. Perhaps it’s not packages or a packaging format. Are you actually deploying packages or are you deploying virtualized servers. Why not have you build process output that. Snapshot and deploy.

    Anyway, won’t work if you want to move the packages between servers, unless you want those moves as part of your build process. But you know what I mean, it’s all about the granularity that you care about.


Comments are closed.

%d bloggers like this: