Author Archives: heckj

helm

Review of using Helm to package and host applications

The open source project Helm represents itself as a package manager for Kubernetes. It does that pretty darn well, having been attached to the project from the earliest days of Kubernetes, and continues to evolve alongside, and now a bit more separately from, the Kubernetes project.

Looking at Helm version 2, it provides three main values

  • As a project, it coordinates and reinforces a number of “best practices” in how to do that generation by housing a public collection of some of the most common/popular open source projects, packaged and “ready to use” within a Kubernetes cluster.
  • It provides a templating solution to generate the plethora of YAML files that make up the descriptions and definitions of Kubernetes resources.
  • It provides a “single command” tool for deploying one or more projects to an existing Kubernetes cluster.

The first two values have been the most meaningful to me, although with some definite caveats and gotchas. The third I thought would be more valuable. I still use the single-deploy-command regularly, although I’m questioning if it is a crutch that will ultimately be a trouble point.

More in depth for each of these:

Package Repository

The single most powerful aspect of Helm isn’t the code itself, but the result of the community and contributions those community members have wrought. The charts collection is the defacto set of examples of how to organize, structure the inputs, and run software with the resource concepts of Kubernetes.

While it calls itself a package manager, there’s a gap if what you are expecting from it is a single binary package that is installable – the moral equivalent of an .rpm .deb, .apk, or installer exe file. You don’t get a single file – Instead you get a set of configuration values alongside a set of templates. The configuration values are the defaults used with the templates to generate the description of the Kubernetes resources. These default values can also be overridden when you invoke helm, which is a godsend for scripted deployment (such as CI/CD). The resource descriptions generated from the templates expect (and require) the actual software you’re running to be available via one of the public container registries – GCR.io, quay and DockerHub being the three most common referenced. The software you’ll actually be running – the container images – is (intentionally) not included within the helm chart.

If you want to run your Kubernetes cluster where it can’t take advantage of the public internet and those resources, be aware and prepared. I would not be surprised to see caching proxies of those services develop, much like Artifactory and Nexus developed for the Maven build tooling. In fact, I would keep a close eye on Harbor (technically open source, but dominated by the VMware) to see what might develop to help them deploy in more isolated scenarios. It is not all that difficult to use private and local container repositories, just be aware the public packages expect to use the public container repositories.

Pervasively embedded within the templates is a fairly robust and opinionated set of how to take advantage of Kubernetes. The content of the  templates contains a ton of knowledge, but be aware it is not always consistently applied or exposed. Like many projects it has learned from experience, so newer and more recently updated charts will often reflect different views of what is important and useful for deployment. None the less, they provide a huge number of functional and effective patterns for running software.

These patterns are the strongest where the features have existed and been stable within Kubernetes for a while – Pods, ReplicaSets, and the effective use of the side car pattern for buckling on ancillary functionality or concerns. It is weaker (or perhaps viewed differently: various levels and consistencies of workarounds were created) for some of the new features in Kubernetes, including StatefulSets and even Deployments.

In some cases, early workarounds were so heavily embedded that they persisted long after the need existed: the concept of Helm “deploying and managing state” was a filler to the gap of Kubernetes not having a solid Deployment concept earlier, and the whole world of custom resources and extending Kubernetes with operators overlaps with what Helm enabled with hooks. My perception is that both Kubernetes and Helm charts are struggling with how to best deploy these newer structures, which themselves represent often some operational knowledge or intended patterns.

Like the virtual machine focused brethren (Chef and Puppet) before them, Helm added testing and validation for their charts. The chart validation has expanded significantly in the past 18 months. Like any validation, they do not guarantee 100% effectiveness. Even still, I believe it is important to be willing to review what the chart is doing, and how it’s doing it, before using it. The instances of charts failing with newer versions of Kubernetes has decreased significantly, primarily due to the focus of the Helm community on recognizing and working to expose it as a problem and resolve it when it occurs.

Templating

A bit background – Kubernetes resources can be defined in either JSON or YAML, and are a declarative structure: a desired state of what you want to exist. These structures are loosely coupled, “tied together” with the concept of labels and selectors. This is both a blessing and curse, providing a lot of flexibility, but if you typo or mismatch some of those connections, there can be very little – to no – checking and it can be quite difficult to debug.

In creating these resource manifests, you will often find yourself repeating the same information, sometimes many times – or explicitly using repetition to tie pieces together. It is ripe for the solution that developed to this repetition and boilerplate overhead: templating.

Helm uses (and exposes) the Go template library Sprig, to greater and lesser degrees. From using the templating language, my opinion is that it is no better (or worse) than any other templating system. It has many of the same concepts that you might find in other templating systems, so if you are already familiar with a templating language, picking up the one used by Helm may be awkward but really is not too bad.

There are variants in other projects that enable a similar functionality to Helm (KSonnet, and the now mostly ice-boxed Fabric8). Even with competition, the network effects from Helm’s collection of charts makes it very hard to compete. Most solutions in this space have to make a choice of how much of a language to build vs. how simple the templates are to use – a continuum between a fully fledged programming language and simple, targeted replacement of values. Helm’s choice adds in some language structures (concepts of making and using a variable, and transforming values), but holds back from the slippery slope into a fully custom language.

We will see if that holds true with Helm version 3’s development, which will be adding the language Lua into the mix, although it appears more for handling the deployment hooks aspect.

If you are a NodeJS, Ruby, or Python developer and looking at the charts, you may have more confusion around what the resource you’re trying to create should look like rather than any trouble with the templating language itself. The templating does nothing to encapsulate or simplify Kubernetes resources and how they may be used together (or not). Helm itself has two commands that have been lifesavers in learning the templating and using charts:

helm template

and

helm –debug –dry-run

These two commands run the templating engine and dump (with slight variances in what they’re expecting) the rendered results. You still end up seeing (and having to deal with) the “wall of YAML” that is Kubernetes, but these two commands at least make it easy to see the results of the templates after they render.

Deployment

The exciting (yes, I get excited about weird things) capability of Helm to deploy my applications in a single command reinforced the the concept of it being a package manager, but may ultimately be the biggest crutch of the solution.

As mentioned earlier, Helm “grew up” with Kubernetes, and was alongside the project from its earliest days, covering the gaps from the core of Kubernetes to the cold hard reality of “getting your stuff running” in a cluster. Helms’s concept of Tiller may be the earliest seed of an operator, as it installs itself into your cluster and then handles the deployment and management of resources that it manages. This same capability is more recently codified into custom resources and the operator pattern, as well as the simplest and most common use cases being covered by the Deployment resource and the associated controller.

When Kubernetes finally included RBAC as a default, Helm (and how tiller was installed) illuminated a bit of a hole in how many people were using and deploying software. There was a lot of work exposing and thinking about how to properly secure Helm. Helm 3 will be removing Tiller from the concept of Helm, continuing to evolve with Kubernetes features.

You also don’t strictly need to use this capability of Helm, although it is darned alluring. As mentioned in the section on templating, you can render charts (and their templates) locally and use tools such as kubectl to apply the resulting resources to a cluster.

Having a single command that is easy to add into a script has been a godsend for continuous deployment scenarios. It is what powers GitLab’s “AutoDevOps” hosted continuous deployment. I use the deploy-with-a-single-command myself, and plan to continue to do so, but it comes with a price.

Helm likes to own the whole lifecycle of the release and does not expect or accommodate anything else interacting with the stuff it is managing. In many cases, this is completely fine, but in some cases it can be a pain in the butt. The most common scenario usually involves some manner of persistence – where you want to install and get the software running, and need it operational to do further configuration on how you want to use it. This could be anything from linking multiple instances of some software into a cluster, doing a database schema migration, or doing a restore of prior backups.

Helm 2 has the concept of hooks to help with actions that happen repeatedly and consistently with every deployment or upgrade process. Helm 3 will be expanding on these concepts, although I don’t yet know the specifics, with Lua as the scripting language to enable this functionality and potentially more.

I am personally conflicted on the inclusion of Lua and what it implies for Helm. While Lua is a lovely scripting language, and likely the best choice for what the developers decided they wanted to do, I think it may end up being a new barrier to adoption for developers outside of the Helm charts/Kubernetes space. Every developer that sits down to use Kubernetes comes with their own biases and comfort with scripting languages. They are often used to Python, Ruby, Javascript, or any of a number of other languages. If Lua becomes an implicit requirement for them to use Helm to accommodate their own operational needs, I suspect that barrier will be significant. Because of this I am hesitant to be excited about the inclusion and focus on using Lua with Helm. What it will ultimately mean in terms of developer accessibility to using Helm and Kubernetes together is yet to be seen. I hope it won’t be an even larger and steeper learning curve.

For the scenario where you want to do periodic, but not consistent, interactions – such as backing up a database or doing a partial restore or recovery – you need to be very aware of the application and its components in their lifecycles. In these scenarios, I have not found a terrific way of using Helm and its hooks to help solve those problems.

Kubernetes itself is only partially effective in this space, having the concept of jobs for one-off batch style mechanics. However, jobs can be darned awkward to use for things like a backup. While I used jobs and continue to try and make them work, I often revert to using kubectl to directly interact with temporary pods to get the work done consistently.

With Helm, I struggled with creating job resources that utilize the same ConfigMap, Secrets, and variables that are used with the charts. Helm is crappy at doing this if you’re using the deploy-with-a-single-command mechanism. An as I mentioned earlier, Jobs can be an awkward fit with the use cases I am trying to accommodate. These scenarios are more “pets” and “one-off” needs where knowledge of the underlying systems and their current state are critical. It may be that operators will ultimately win out for these use cases, but they have a fair way to go yet.

At its heart, this deployment capability that I use implicitly use many times a day also strikes me as the current edition of Helm’s weakest point, and I wonder if it is a crutch that I will ultimately need to replace.

Lessons from Kubernetes for Developers

It’s been a little over six months since Kubernetes for Developers hit the streets. It has been interesting to see the uptake, and where the holes have been from what I first envisioned.

The metrics I receive on book sales are unfortunately wan – nothing so effective as a daily metric for either sales or general engagement. Well, nothing that is shared with me. I was tracking the site with bookrank.com, but they went defunct and shut down, so about the only useful metrics I can see is the Amazon page rank information on the book’s amazon page.

What I learned more organically is that while some developers appreciated the book a surprising number didn’t have the same base level of knowledge that I thought they might. Some of the feedback I’ve received included questions about DNS, how to use linux command line tools, and general confusion about ports and IP addresses. Most of these questions came from people new to development, folks who “heard about Kubernetes” and wanted to know if or how they could take advantage of it.

The technical content for the book is getting low but consistent traffic at GitHub. The python demo application a bit less traffic than the nodejs demo application. No real questions or queries through GitHub, but I think unless you’re reading the book itself, you wouldn’t be aware of the GitHub project.

One area that I wish I delved (and knew) more about before I finished the book was ingress. Although in hindsight, this area is one of the perennial sore spots in Kubernetes – a beta feature for the past 9 releases with no consolidation progress, but with some actually interesting uptake at the edges of the Kubernetes project itself – with related projects blurring lines into service meshes, or being semi-solid commercial implementations over software or physical load balancers.

If I had written anything in depth, it probably would have been best about the stock Nginx ingress controller, as that seems to be about the most default – and the advances from some of the open source projects since I published – such as Heptio’s Contour – have been pretty amazing and interesting.

Another area that’s been a surprising win has been cert-manager, a wonderful tool for publicly hosted Kubernetes clusters to help deploy and manage signed TLS certificates through LetsEncrypt. Since I didn’t get anything useful written about ingress or this lovely gem, I am working on an open source documentation contribution including a quick-start for cert-manager that will hopefully be done and live shortly.

There is also just a huge learning curve for developers to adopt and engage with Kubernetes. The concepts aren’t impossible, but there are a lot of them and the topics are complex and intertwined. Even with the complexity, gaps, and edge cases that bite I am still a huge fan of the project. I see it as the best choice for “software that helps you keep your software up and running” when you can take advantage of it.

Kubernetes for Developers is published!

It’s been a quiet few months on the blog, as all of my writing attentions have been focused on the book project I started back in September of last year. It is now published and available!

kubernetes_for_developers_cover

If you are so inclined, you can find a copy at PacktPub.com, or on Amazon.

When I started this project, one of the things I really wanted to do was work with some editors to improve my writing. And while they didn’t work with my on this book, I learned a tremendous amount from Jennifer Rondeau and Steve Perry, who are technical writers (and editors) at Heptio and Google respectively, both maintainers of the  Kubernetes open source project documentation.

As a side effect of this project, I got involved with the Kubernetes Docs team, helping out here and there and becoming a maintainer myself in the process. Its a wonderfully diverse (and growing) team of collaborators from a bunch of different companies and backgrounds, and as a whole have been incredibly welcoming and helped bolster my knowledge to be able to write the book.

I’m writing a book: Kubernetes for Developers

Quite a number of years ago, I did piece work and later published a book through MacMillan publishing. Being technical books, they are long gone from the shelves as time  made the content irrelevant. Although Gus sometimes likes to poke me saying he found a copy of my book in a used book bin. Strange to think that was 11 years ago.

While spending this summer sabbatical on a lot of traveling, I reflected on what I enjoy. Even while away from computers, I kept my involvement in the Kubernetes project. I find the project extremely compelling, primarily because it works from first principles and builds from there.

What drove me to do the writing is a “You don’t know it until you can teach it” philosophy. And this project is a space I’d like to know, really know. This is what led to deciding to shop around the idea of a book about Kubernetes, written specifically for developers who would otherwise not really interact with it: folks knocking together code in nodeJS or Python that are now getting told “Hey, you get to run this code as well as develop it”: Kubernetes for Developers.

Much of the kubernetes documentation is written for people who are setting up the cluster or running software with it. Most of the people that I see touching on it are the same folks that use Puppet, Chef, Ansible, or Salt. If I were to put a name to this persona, it might be SRE, operations, or classic system administrators. There is some great work (full disclosure: I’m involved in it) happening within SIG-DOCS to make the documentation a lot more relevant and directed to different personas.

I signed a deal with Packt publishing to author Kubernetes for Developers, aimed to be complete in spring 2018. I suspect the space will be pretty crowded by that point.  I have been keeping track of what’s out there: Kubernetes: Up and Running is clearly leading the pack at NovelRank, and there are four or five other books out there – three others from Packt – but so far none focused on developers.

I’m drawing a lot of the ideas for what content to focus on from StackOverflow questions, questions that occur in the Kubernetes Slack, and bugs reported in the project. If you have some opinions on what would be useful to know about or learn, I’m all ears here as well – please leave me a comment, or reach out to me on twitter or github.

I see a lot of  potential in making the architecture and operations of running software far more accessible to developers who have mostly been divorced from it. I think Kubernetes could finally provide a developer-centric Data Center API that I have been hunting for.

My first chapters are blocked out, outline sketched in place to provide some guide rails. I am still trying to get used to Packt’s publishing tools, I definitely prefer using Scrivener by a large margin.

 

Accustomed to complexity

I have been watching some youtube videos from Alphonso Dunn on techniques for pen and ink sketching. I found a couple of his tutorials online when I was traveling in Europe, getting frustrated at myself while trying to represent trees or (even worse) cobblestones. Let me tell you, cobblestones make up a significant part of landscape themes in smaller European towns!

Alphonso’s videos are so effective because he makes the techniques appear not just simple, but achievable. The real value is that he helps you achieve the same results, and does so by breaking down the techniques into understandable, small steps. The dude is a damned good teacher.

Simplicity is hard.

One of my earliest mentors loved to say “inch by inch, life’s a cinch”. An echo of a pattern I learned in engineering to solve complicated problems – break things down, isolate issues and stumbling blocks, simplify the problem. This provides a way to deal with complexity that’s otherwise overwhelming.

It is easy to expect that if breaking things down into smaller components makes things more tractable, that building things using small simple components will get you an easy to understand system. Unfortunately, the inverse doesn’t work that way – instead you get the joy of emergent complexity.

This phenomenon hits everyone – not just programmers. You can see it in the complexity of insurance, laws, and taxes. Lots of small, simple changes that add up to an almost unbelievably complex beast. It’s also bites most product managers while doing their jobs, or any long running, growing project. Added features and new capabilities interact with all the rest of what you have, sometimes in very unexpected ways. If you’re involved and working with it while it’s growing, this same growth of complexity can be almost invisible. We get so knowledgeable on the specifics and familiar with the details that we become accustomed to the complexity.

The complexity suddenly becomes visible when you step away from the project and come back. It’s can also become apparent when someone new joins a project, or learns about the product for the first time: Usually about the time they’re shouting “What the hell!” These moments are invaluable, as you get a view through different eyes and expectations. This “fresh feedback” is important because we get so used to the weird intricacies and details.

What can you do to work against emergent complexity and keep it simple?

1) work from first principles

The best advice I ever received was to focus on the core, and use that the highlight and refine back down to simplicity. Look at the key problem that you are solving, or trying to solve, or primary action you are enabling. Call it out, highlight it – and keep it forefront in your mind while you review the variations and branches that exist.

Just as important to what you want to solve, take the time to identify what you don’t want to solve. Sometimes this is obvious, but mostly it’s not.

2) maintain clear boundaries of responsibility

If you’re working on a large project, there are probably multiple components to it. This is super common in larger software efforts. Where you have those boundaries, take the time to describe what happens across it. Take the time to write down and make sure there is common agreement on what a component needs to do, and what it’s responsible for. Often adding what appears to be a simple feature will or confuse these responsibilities, and you’ll have corner cases you don’t anticipate. It will evolve, so don’t think once something is agreed upon it’s static. Make the time to review and validate these as you go along, understanding if original needs and assumptions have changed enough to warrant revision – and then update it.

3) search out fresh eyes

Actively work to get the feedback from fresh eyes. Find folks who aren’t familiar with what you’re making to look at what it does, or even how it does it. This is one of the most effective ways to highlight what is confusing, or where complexity resides. This can be formal, but often doesn’t need to be – even informal conversations, or a quick demo and getting reactions can lead to understanding where the complexity resides and what is confusing.

4) prune and grow

Acknowledge that complexity, change, removal and growth are all part of a healthy, living project. Include a periodic retrospective that reviews what all the assumptions, and actively trim or remove the options to hew to the core of what you want your project to do. Make the time, and intentionally trim as well as grow as you go.

I’m writing this for you, I just don’t know who you are…

I really enjoy traveling, and I’d say “and meeting new people”, but I’m a fairly introverted fellow in person, and tend to hang back a bit. None the less, I watch and I listen. I spent the earlier part of the summer in western Europe – where my knowledge of the language is not even rudimentary: dutch, german, french, danish, icelandic… fortunately tone,  body language, and the immense pervasiveness of english as a trade tongue made it possible.

I am now back in the US, doing a bit of wandering and traveling here. Stomping around in the midwest as I write this post, mostly doing the same – listening, watching, learning. There’s as much to learn next door as there is across the ocean.

I started “this blog” – which has moved a few times, and been re-hosted two or three times as well – almost 16 years ago when I moved from the midwest. Originally a way to “keep in touch” with friends, a sort of public journal. Since then personal writing has morphed to technical writing, the me of now self-censors very differently than the “me of 2000”, and the audience has both grown tremendously and shifted around. RSS news readers have given way to cross posting in social media or just sharing content through search engines. I do post a lot more directly in twitter or facebook, even with their faults and strageness, than I do here. I have a micro.blog account, but haven’t yet really engaged in using it to any depth.

I am thinking about the content here, because as I was looking at the “metrics and analytics” I was struck (not for the first time) how much I write for people that I haven’t met. It’s kind of cool – well, “astoundingly cool” really – to write for people I may never even meet.

I expect the majority of people find my content through a search engine, and even a decade ago I was writing with the idea that search engines would index this stuff and make it available. That Google or it’s competitors would add this to the wealth of knowledge and opinions accessible. But planning for it doesn’t really resonate the same way as seeing the actual reach.

This is a snapshot view of just August of this year – what people are reading and viewing of my writing.

aug2017analytics

I never would have expected to have people from germany, india, china and france reading my words. Hopefully getting something useful from it. Other than these analytic trails, it’s hard to see the evidence.

The post that tops even the current views was one I wrote 18 months ago, and didn’t seem to become popular for a good six months after I wrote it. For quite a number of years, the highest viewed post was one I wrote in 2009 – a walk-through example of setting up CI for a python project with Jenkins (well, Jenkins was known as “hudson” at the time) that I created as a “thank you” for the author of coverage – open source project.

My posts here are for me, and for you – I just don’t know who you are. Not yet anyway, and maybe I never will – hard to say. I think it’s a worthy endeavor, and there are a lot of topics I’m interested in learning and sharing. If you do happen across this and want to say “hi” – I’d love to hear from you, find me on micro.blog, github, stackoverflow, twitter, facebook, or whatever. A sort of pen-pal across time I guess – anyway, I like hearing the reflections.

Does the game of chess show us a map to future jobs?

I have been interested in the field of Artificial Intelligence since the 80's – pretty much the heart of the "AI winter". Like many others, I remember when Kasparov lost to the IBM DeepBlue Chess AI in the 1997, and watched in fascination as DeepMind's AlphaGo beat Lee Sodel in Go in 2015.

At this point, the Kasparov defeat is sort of in the history books, as everyone is focused on the AlphaGo wins and the follow technology uses that it employed. Kasparov's interactions are kind of ignored, but they shouldn't be. What happened since that defeat with the evolution of the game of chess, with human and AI opponents, is fascinating – and I think informative.

In the wake of chess AI getting so strong, a style of chess developed called Freestyle chess, and from that – a "Centaur" model appeared. It's a team of human and AI system working together to play the game – and it's immensely powerful. There's an article from 2010 that outlines more of Kasparov's views, and it's worth a read if you're interested. I'm also certainly not the first to see this. There's articles in the Huffington Post about Centaur Chess, the BBC in 2015, and TechCrunch from last year that draw parallels.

The gist of the thesis is that current AI and human cognition aren't strong in the same areas, and each tackle the solution of tasks in very different ways. If you combine these two systems in a manner that exploits the strengths of each, you get massive capability and productivity improvements. It is exactly this pattern that I see as the future for quite a number of jobs.

I don't know exactly what form this will take, but I can extrapolate some obvious pieces – two attributes that will show up more and more.

The first is "explainability". Somewhere tucked in between psychology, cognitive sciences, and art of user experience and design is the place where you get amazing transfer of information between computing systems and humans. If you have an AI system predicting or analyzing something, then sharing that information effectively is critical to having effective coordination. Early examples of this are already prevalent in cars, planes, and military combat vehicles, which have had even more of this kind of investment. Head's up displays (HUDs) are so relevant that they're the common staple for video games today. And with the variety of first-person shooters, we're even training people en-masse to understand the concepts of assistive displays.

But don't get too focused on the visual aspects of explainability – the direction haptics and sound that work with the Apple Watch while navigating are another example – very different from a head's up display, but just as effective. Conversational interfaces like you're seeing with Alexa, Siri, or Google Home are all steps to broadening interactions, and the whole field of affective computing reaches into the world of understanding, and conveying, information at an emotional level.

Because it's so different from how AI has been portrayed in movies and the "public opinion", some folks are trying to highlight that this is completely different calling it Intelligence Augmentation, although that seems to be a corporate marketing push that hasn't gained too much traction.

The second is collective intelligence. This used to be more of a buzzword in the late 90's when some of the early successes really starting appearing. The obvious, now common, forms of this are recommendation engines, but that's really only the edge of where the strengths lie. By collecting and collating multiple opinions, we can basically leverage collective knowledge to help AI systems know where to start looking for solutions. This includes human bias – and depending on the situation, that can be helpful or hurtful to the solution. In a recommendation engine, for example, it's extremely beneficial.

I think there's secondary forms of this concept, although I haven't seen research on it, that could be just as effective: using crowd source knowledge to know when to stop looking doing a path, or even category, of searches. I suspect a lot of that kind of use is happening right now in guiding the measure of effectiveness of current AI systems – categorization, identification, and so forth.

There is a lot of potential in these kinds of systems, and the biggest problem with them is we simply don't know how to best build and where to apply these kinds of systems. There's a lot of work still outstanding to even identify where AI assistance could improve our abilities, and even more work in making it's assistance easy to use and applicable to wide diversity of people and jobs. It's also going to require training, learning, and practice to use any of these tools effectively. This isn't downloading "kung fu" into someone's head, but more like giving Archimedes the lever he wanted to move the world.

Aside

Navigating European small towns with an Apple Watch

A little before I started my recent walk-about in Europe, I invested in a Series 2 Apple Watch. I had not worn a watch in over a decade – I used to carry a pocket watch until smart phones made that extraneous. I thought I'd try it as a replacement to my twice-worn-out-fitbit since it had compelling features as a fitness tracker: The obvious step counter, and also the heart-rate monitoring along with nice integration to encourage activity and exercise. I started "closing the rings" as they say, and along with more frequent access (and visits) to the gym, it's been tremendously successful for me. Most importantly to my earliest concerns, I haven't destroyed it by being a klutz and smashing my wrist into walls and door frames as I was wont to do a decade or two ago.

The surprise win was how amazingly effective it is for helping to navigate with directions from Apple's Maps IOS app. When we were traveling about some of the smaller towns, navigating from the train station to our B&B or hotel was a common need. The hardest days were when we switched countries and were dealing with a different language, different customs for sign placement, and generally trying to learn what to listen for as the languages changed. I kept my cell phone active with a data roaming plan just for this – I knew I'd need help to find my way.

The surprise was in how subtle and effective the combination of haptic feedback and slight sounds emitted from the watch helped with navigation. One of the things that I found myself constantly failing was interpreting reasonable scale from maps (digital or paper) to the physical reality of streets in the smaller European towns. We stayed and wandered through amazing locations, and not only are the streets much smaller, the organic nature of street layout and smaller streets really lead to our confusion. What my US trained senses thought was another 100 yards was maybe only 20 away, and early navigation had frequent back-tracking or circling around as we tried to get used to the differences.

I found when I was walking/navigating through some distance (this really shone while walking in Utrecht), the "make a left" or "make a right" haptic signals that came through the watch were brilliantly timed with my walking pace. The tone of the two "blips" goes up for an upcoming right, down for a left. I still did a periodic double-take, or hunted for the name of the street to make sure I wasn't veering down some strange side path, but it was immensely effective.

The other complete win is that getting direction feedback from the watch is subtle.

I don't mind standing out periodically like a tourist newb, but not all the time. I didn't want to constantly paint a tourist-target sign on myself if I could help it. As I got comfortable with watch providing muted and haptic feedback, I wasn't doing the "spin in a circle with my iPhone" routine to figure out where I was and needed to go. The pattern that I ended up developing was selecting my target destination in my phone, getting the directions active, and then dropping it in my pocket. From there, the watch displayed the "next step" in the directions nicely, and had provisions for scanning backward (or looking ahead) – and while I was walking it was easy to glance at my wrist – as though I were looking at the time – and see a short note like "turn left in 300 feet". Possibly the best thing was the quality of the maps was outstanding and with my phone safely tucked away I spent far more time looking at my surroundings, getting familiar with whatever location, and just generally being more aware of my surroundings than I otherwise would "tracking with a phone".

The only place where Apple's Map IOS app and directions didn't shine was when I needed to use a combination of public transit and walking in some of these smaller towns. More than once, Google Maps gave better transit directions – or any transit directions in some cases – where Apple's Map application just didn't have the detail. Larger cities like Copenhagen, Berlin, and Paris weren't any problem – but some of the bus or trams in the Netherlands or Belgium just didn't seem to get covered consistently.

Stochastic testing and developer tools

Intermittent issues, or bugs with intermittent reproducibility, are some of the most frustrating issues to track down. In past experience, these kinds of bugs are often multi-thread race conditions or a failed assumption about something that’s happening asynchronously that you thought was synchronous (or just expected to happen quicker). Tracking these down is immensely time consuming, and most of the existing test frameworks and infrastructure isn’t really well suited to help.

A common pattern that I’ve seen (and adopted) is just repeating the tests again and again to try and flesh out the error, (hopefully) with some automation help. I don’t know a name for this process, so I call it “stochastic testing”. Fortunately, it’s generally pretty easy to set something like this up with with continuous integration products (i.e. Jenkins), but while that automates the task of repeating the specific tests, it doesn’t really help you deal with the results and data – and that definitely ends up being the challenge.

There’s a continuum. The simplest is just re-running a single test because you’re seeing an error. I expect most people working with a CI system have seen this, even if they haven’t tracked it down or used stochastic testing to help identify the problem. It’s that time when maybe your build cluster is overburdened, and instead of hunting down the bug, someone tacks in a sleep(10) into a test script. As much as it pains me to admit it, sometimes just adding that sleep() call is the right answer – often you’re waiting for a database to be live, or a connection to finish establishing, and there just isn’t completely causal traceability to hook into know for sure when something is ready to be used. A common secondary theme here is seeing people make a pre-test that validates what should be invariants prior to the actual test running to avoid errors. It’s more work, often slower, but it gets the job down – and reliability in tests is a darned worthy goal.

Then there’s the classic multi-threading race/condition bug, or maybe a race condition between multiple processes that are expected to be used in coordination, but perhaps don’t always check. I’m seeing the later more and more with distributed systems being a common way to build services. A quirk of timing may be causing an exception somewhere in a complex interaction or logic chain, and without knowing everything about all the components, the most expedient way to find it is to just re-run the tests – maybe 10, maybe 1000 times, to find and expose the flaw. Here’s where the data about what you’re doing starts to become tricky – your environment is (hopefully) fairly controlled, but you’re liking going to have logs from multiple separate processes, output from the testing process itself, and all of this compounded by the number of times you’re running the test as well as the need to distinguish the tiny bits that are meaningfully different in these various test runs.

This is also about the level that I’ve found the most benefit in just tracking the test results over time – with a given build, repeating the same sequences, and reporting the percentage of failures and the failures over time. Especially when you’re working on a “release” and stabilization, the time to do these tests pay off. You can verify you don’t have memory leaks (or memory retention cycles, as the case may be), or check for reactions in the system that you didn’t anticipate (like it slowing down after running for a while due to naive algorithm choice). This is also where I like to put in micro-benchmarks, or even full benchmarking code – so I can look at relative performance trade offs between builds.

And the continuum extends right out into running in a public beta test, or perhaps production. In these cases, maybe there isn’t tests – it’s just straight up usage, which complicates tracking down the issues even more. And in these scenarios, you often only get a traceback or maybe a crash report with a little more state detailed. By several measures, this is the most common scenario, and the one that has some explicit tooling created to help with the data tracking and analysis. In 2014, the Mozilla test organization posted some details about their mass collection and processing of crash reports. The project called Socorro has evolved since, and there are commercial products that have similar infrastructure for collecting failures and providing a useful view into what would otherwise be overwhelming aggregate data. TestFlight, Sentry, Rollbar, and Crashlytics all come to mind, focused on either web services or mobile device crash reports. Quite a number of vendors also have their own “on failure, phone home” systems with various levels of detail or metrics available to provide support for their products. At the very least, it’s common to find a “generate diagnostic output” for support from a purchased product.

What I haven’t seen outside of some custom work is the tooling to collect the output of repeated testing, logs of processes, and pro-actively do analysis to provide a state/status dashboard for that testing. When I was at Nebula, the engineering infrastructure team built up a large regression testing infrastructure. Running some of the acceptance tests in a stochastic mode – repeatedly and reporting on the error percentages – was a common practice and a good measure of how we were progressing. I do rather wish that was more common (and perhaps easier) across a broader variety of technologies.

As a side note, one of the really useful details I found to highlight race condition bugs was running tests with constrained resources. It started as an a “accident” – I made the cheapest/smallest VM I could so I could have a lot of them, and in using those for the slaves in a CI system, I found far more issues than we ever did with developer machines, which tended to have a fair amount of resources. Constraining the CPU, and sometimes the memory, really highlighted the issues and could often change the sequencing of operations sufficiently to reproduce some of those hard to find race condition bugs.

The fundamental problem is getting information out of your tests and testing system to determine what’s gone wrong so you know how to fix the problem. This often resulted in culling through verbose logging and hand-correlating events. Things like distributed system tracing, profiling of the relevant systems, and monitoring of resource utilization – could have helped significantly.

Doing these processes highlights a trove of other errors that obscure the real issues. In my own experience, I confused myself (and others) by bundling too many things in a single test, or that the testing framework I used (often a unit-testing framework) did not include hooks or tooling to help capture the information needed. And finally, of course, there’s always the issue of hardware failing – drives or RAM failing in one machine, leading to both false positives and negatives in testing. In distributed systems, a temporary network blip had the same effect – often causing a ripple of otherwise unreproducible errors while testing.

Having a consistent means of collecting, analyzing, and exposing this kind of information would be a huge boon in both development and debugging. There are many variations in this space, it’s easy to find any one piece, and quite challenging to assemble a holistic solution. I’ve wondered if some hosted (or open source) data analysis solution would work. A mix of collecting metrics, events, logs, and potentially even tracing data, as well as having some knowledge of the evolution of builds/source control changes and testing that’s happening to help track down issues.

I keep circling back to this, noodling and considering how to tackle this problem. It’s sort of “missing developer tooling” for me. What would make tracking down these issues easier? What would be good metrics and questions to answer that can provide a sense of progress over time? How to reasonably represent the current “state” of quality in the face of constantly changing system? I have lots of questions, and no clearly obvious answers – just musings and theories right now. If you have something you’ve used that helped identify these kinds of problems, I’d love to know about it.

WordPress for IOS

I use WordPress as a blog engine. I started using it over 12 years ago – at first I hosted it myself, and a two years ago I shifted to using wordpress.com's hosted service, with which I've been very happy. I do all the editing through a browser – until now.

I tried out the WordPress IOS app some time ago, but the editing mismatches and inability to provide a consistent writing experience with the browser based interfaces left me immensely frustrated. I deleted it, and just using Safari or Chrome. Not the greatest experience on a mobile device, but better than nothing. When I was walking about the past couple of months, I heard they had a new editor, so I decided to give it a shot.

Long story short, the new editor (code named Aztec) is a tremendous boost over the existing system, and even in beta a far better experience for me. If you're publishing to a wordpress blog, then I'd encourage you to give it a shot. It's become far easier to include photo's, embed markup, and manage the layout of the writing than it's been in ages.

It's not perfect, but I wouldn't expect it to be in beta form. The means of setting up a scheduled post is still awkward as hell, and selecting categories and other ancillary metadata for a post is fairly constrained. But the editing experience is so far and advanced an improvement that I'd recommend using it immediately.