Yesterday, I wrote about fully autonomic services. It’s been on my mind for months now, or more accurately years. To get to the end state of a self-managing service, there’s a ton of knowledge that needs to be encapsulated in some form. As we represent this knowledge, we tend to it in one of two forms: imperative and declarative, and often built up in layers.
Programmers deal with this in their day to day jobs. They’re imparting knowledge, putting it into a form that systems (computer languages, libraries, and frameworks) can use. The expression of that knowledge, and how it relates to what we’re trying to do, is the art and science, the essential craftsmanship of programming.
Imperative and declarative weave back in forth in the expression of programming. While it’s really something of a “chicken and egg” argument, I’d say we often start with imperative forms of expression, especially while we’re exploring all the different ways we could solve the problems, and learning from that. As it starts to become common – as we “commoditize” or standardize on ways of doing this, we create new language (sometimes new computer languages!) to represent that learning – we turn it into a declarative form. Sometimes this is in the form of libraries or frameworks, sometimes it’s in the form of layers of software, encapsulating and simplifying.
To create a fully autonomic systems, we need to capture information about how to do the things we want the service to do. The challenge in this space is two fold: how to capture that knowledge, and coming up with good ways to frame the problems. If you look at Docker Compose, Kubernetes and Marathon, they are presenting a means of thinking about this problem. They define declarative mechanisms to describe the programs/services they’re managing. Marathon calls them ‘applications‘, and Kubernetes calls a similar concept ‘deployments‘, Compose uses ‘docker-compose‘ as that declaration, sort of avoiding a description like ‘application’ or ‘deployment’ altogether. Terraform calls the concept ‘configurations‘, JuJu calls it ‘charms‘. They’re all declarations on what to run and how they relate to each other in order to provide some specific software service.
One of the most notable differences in these systems is that Compose, Marathon and Terraform all stop at the level of declaring the structures, letting plugins or “something else” orchestrate the coordination for the interesting tasks of a red/green upgrade or rolling upgrade, where Kubernetes is taking a stab at including a means of doing just that within it’s domain of responsibility. The implication in the case of Kubernetes is that developers deploying services with Kubernetes will learn (or know) how the system works and develop code within those constraints – in programmers terms, use it like a framework. In the case or Marathon, it expects a developer to tell it what to do – to use it more like a library. Kubernetes is far more opinionated in this respect, and in a large part betting on the knowledge that it’s development community already has for proving out that it got the right level of abstractions nailed down.
A notable difference in Marathon and Kubernetes from Terraform and Compose is that they include an expected responsibility to keep running and “keep an eye” on the ‘applications’/’deployments’ they’re responsible for.
Puppet, Chef, Ansible, and SaltStack are all focused on the world of configuring services within a single virtual (or physical) machine – the “install and configure” and “start it running”. The concepts they were built for has a responsibility stop at the boundary of getting the virtual (or physical) machine set up, and didn’t include the concept of handling a failure of the machine. Keep in mind that these systems were created long before cloud computing was a reality, and the idea of asking for another machine wasn’t a few seconds of work behind an API, but days, weeks, or months of work. In a general sense, what they made declarative was the pieces within a virtual machine, and didn’t expand to the realm of stitching a lot of virtual machines together.
For the container versus virtual machine divide: Yes, it’s possible to use Puppet, Chef, and Ansible to configure containers, but I’d easily argue that while you can also drive a screw into the wall with a hammer, that doesn’t make it a terrifically good idea or use of the relevant tools.
As a side note: SaltStack stands out in this space with the concept of a reactor, so it’s intentionally sticking around and listening to what’s happening from within the VM (or VMs) that it’s ‘responsible for’.
The stand-out challenge that I keep in my mind for this kind of automation is “How can it be used to easily install, configure, run, and keep running an instance of OpenStack on 3 to 50 physical machines?” It doesn’t have to be OpenStack, of course – and I don’t mean “an IaaS service”, but a significantly complex application with different rules and challenges for the different parts needed to run it. I spent two years at the now defunct Nebula doing exactly this, and it’s a challenge that none of these systems can completely solve themselves. It’s why “OpenStack Fuel” exists as a project, encapsulating that knowledge that’s otherwise represented in Puppet declarations and orchestrated externally using something they created called “Railgun“.
Another side note: this particular challenge has been the source of several companies (Nebula, Piston, and Mirantis) and the failing of many enterprise installations of OpenStack, as just getting the bloody thing installed is a right pain in the ass.
Kubernetes, Marathon, Compose, and Terraform wouldn’t stand a chance of the ‘OpenStack’ challenge, primarily because they all expect an IaaS API to exist which can give them a “server” when they need it, or they work at the level containers, where a Container API (Docker, RKT, GCE, etc) can go spin up a container at request. The concept of the challenge is still useful there – but probably needs another form for a real example. Take a look at Netflix’s architecture and their adoption of micro services for another of the seriously complex use case that’s a relevant touch-stone. Every company that’s seriously providing web-based services has these same kinds of complexity and scale issues, and each services architecture is different. Making those architectures a simpler reality is what these systems are after.
UPDATE:
That’ll teach me to write such a review the day before DockerCon. This morning, the Docker responsibilites and capabilities changed 🙂
As of Docker 1.12, it looks like Docker Compose has been combined into the code, and Docker is heading to take on some of the same space as Marathon and Kubernetes, called that feature “Docker Stacks and Distributed Application Bundles“. It’ll take me a bit to review the new material to see where this really lands out, but I’m not at all surprised that Docker is reaching their essential product responsibilities into this area.
Rhonabwy 
Hey Joseph, good article and breakdown of the various orchestration options. I recently went through a similar decision process, weighing the option, their “intended” use, and how I need to use them. I liked different aspects of all the options, but didn’t want to have to commit to one and forego the others. Since we are going to a full containerized micro-service arch, I decided my platform would be Rancher and RancherOS. Rancher will let me spin up hosts using Mesos, K8S, Swarm, or their own orchestration called “Cattle”, this way I can make use of whatever I need, and have it all under 1 management system.
LikeLike
Very cool – I’ve seen Rancher, mostly through the RancherOS, but haven’t worked with their orchestration system to know what it has (or doesn’t have). I’ll take a look and see what I can see to include it in the set here!
LikeLike