Creating Machine Learning Models with CreateML

I have been following the bare outlines of building, and using, machine learning models in Apple’s software ecosystem for a while. Most of my learning and personal research has been with foundational technologies – following some of the frameworks (TensorFlow, PyTorch, SciKit-Learn) and some of the advances in models and their results. Until this holiday, I had not applied myself to seeing the latest evolution of Apple’s machine learning frameworks and tooling. I scanned through websites last summer during Apple’s WWDC, but didn’t really clue in to the changes that were coming, and that are now available, with Big Sur (macOS 11).

Aside: The web site Papers With Code has been a great consolidated reference point for digging into the academic research behind quite a variety of machine learning advancements over the past couple of years, and includes both the research papers (often linked to ArXiv) and frequently references to the code matching the papers.

First off, the tooling to create simple models with CreateML has received quite a boost. Matched by the documentation, there are a large number of new models that can be easily generated with you “just” providing the data. Models to classify sounds, motion data, and tabular data – including regressors (predictors) as well as classifiers. Prior to macOS 11, they had models for classifying and tagging words and sentences, as well as image classification. Those all still exist, and Apple provides some ready-to-use models built-in to frameworks such as Natural Language.

The goal I chose to explore with was focused on natural language processing – more on that later. Apple has long had a pretty good natural language library available, the earlier stuff being a bit more focused on Latent Semantic Mapping, but recently turning to – and leveraging – quite a bit of the natural language processing advances that have happened using more recent machine learning techniques.

The most interesting win that I immediately saw was the effectiveness of using transfer learning with CreateML. When you’re making a model (at least the word tagger model), you have the option of using a CRF (conditional random field) or applying the data with a transfer learning over existing models that Apple has in place. They don’t really tell you anything about how these models are built, or what goes into choosing the internals, but the bare results from a few simple experiments are positive, and quite obviously so.

I made sample models that mapped language dependency mapping from some publicly available datasets provided by Universal Dependencies. Dependency maps specify how the words relate to each other, and the part that I was specifically interested in was leveraging the ability to identify a few of those specific relationships: nsubj:pass and aux:pass. These are the tell-tales for sentences that use passive voice. I did multiple passes at making and training models, and transfer learning was clearly more effective (for my models) in terms of the reported accuracy and evaluation.

Training a Model

One of my training runs used approximately 5000 labeled data points (from https://github.com/UniversalDependencies/UD_English-GUM). About 4200 of those points I allocated for training, and another separate file with 800 data points for testing. In this case, a “data point” is a full or partial sentence, with each word mapped to an appropriate dependency relationship. The CRF model topped out at 81% accuracy, while the transfer learning model reached up to 89% accuracy.

Conditional Random Field Model: 11 iterations planned, converging “early” at 7 iterations.
Transfer Learning: Dynamic Embedding (25 iterations)

To put the accuracy in perspective, the latest “state of the art” parts-of-speech tagging models are running around 97 to 98% accuracy, and around 96% for dependency parsing.

Timing the Training

I ran these experiments on one of the new M1 MacBooks with 16GB of memory. This training set took just over a minute to train the CRF model, and 7 minutes to train the transfer learning model. A similar run with more data (27,000 data points) took 20 minutes for the CRF model, and 35 minutes for the transfer learning model. Transfer learning takes longer, but – in my cases – resulted in better accuracy for predictions.

Evaluating the Model

Once trained, CreateML provides an evaluation panel that gives you precision and recall values for each value that in your tagging set. This information is ideal for understanding how your data actually played out versus what you were trying to achieve. Using it, you can spot weak points and consider how to resolve them. One way might be gathering more exemplar data for those specific classifications. For example, in the following data table, you can see that the recall for “csubj:pass” was 0%. This showed that I simply didn’t have any examples in the test data to validate it, and possibly only a few samples in my training data. If that was a tag I was interested in making sure I could predict from input text – I could find and improve the input and testing data to improve that accuracy.

Previewing Predictions from your Model

Probably the most fun of using CreateML is the preview mode. Since I was making a word tagging model, it provided a text entry area and then applied the tagger against the data, showing me an example of the predictions against what-ever I typed.

Since I’d done two different models in the same project, I could flip back and forth between them to see how they fared against this sample, showing me the predictions from the model and the associated confidence values of those predictions. (and yes, I have considered locking my brother into a kitchen, he’s a damned fine chef!)

Exporting a CoreML model from CreateML

CreateML includes an Output tab that shows you the tags (also known as class labels) that you trained. It also gives you a preview of the metadata associated with the model, as well as kinds of inputs and outputs that the model supports. This makes it nicely clear on what you’ll need to send, and accept back, when you’re using the model in your own code.

One of the details that I particularly appreciated was including a metadata field explicitly for the license of the data. The data I used to create this model is public, but licensed to by-nc-sa-4.0 (An attribution, non-commercial, share-alike license). Sourcing data, both quality and licensed use, is a major element in making models. I think it’s super important to pass that kind of information along clearly and cleanly, so I’m glad it’s a default metadata attribute on models from CreateML.

Notes on CreateML’s Rough Edges

While CreateML was super easy to use and apply, it definitely has some issues. The first is that the user interface just doesn’t feel very “Mac” like – things I expected to be able to “click on and rename” didn’t smoothly operate as such (although I could control-click and choose rename), the window doesn’t easily or cleanly resize – so the app dominates your screen wether you want it to or not. On top of that, a queueing feature, and related window, for lining up a bunch of training was quite awkward to use and see the results as it was progressing, and after it was done. I had a couple training runs fail due to poor data, but the queuing window didn’t show any sort “well, that didn’t work” information – it just looked like it completed, but there was no training on those models.

The other awkward bit was dealing with messy data through the lens of CreateML. I can’t really say what it did was “wrong”, but it could have been so much better. In one of my experiments, the data I had chosen for training had issues within it: missing labels where the training system expected to see data. That’s cool – but the error was reported by a small bit of red text at the bottom of a screen saying “Entry # 5071 has a problem”. Finding entry #5071 of a structured JSON data set is, bluntly, a complete pain in the ass. When I’d parsed and assembled the data per the online documentation for making a word tagging model, I’d dumped the data into a monster JSON, single-line data structure with no line breaks. That made finding a specific element using a text editor really rough. In the end, I re-did my JSON export to include pretty-printed JSON, and then also used VSCode’s “json outline” functionality (scaled up, since it defaults to 5000 items), to track down the specific item by position in a list. I found the offending data, and then looking around, noticed a bunch of other areas where the same “partially tagged data” existed. In the end I dealt with it by filtered it out if it wasn’t fully tagged up. It would have been much nicer, especially since CreateML clearly already had and could access the data, if it could have shown me the samples that were an issue – and notified me that it wasn’t just one line, but that a number were screwed up.

The evaluation details after you train a model aren’t readily exportable. As far as I can tell, if you want to share that detail with someone else, you’re either stuck making a screenshot or transcribing what’s in the windows. It seems you can’t export the evaluation data into CSV or or an HTML table format, or really even copy text by entry.

The details about the training process, its training and validation accuracy, number of iterations used, are likewise locked into non-copyable values. In fact, most of the text fields feel sort of “UIKit” rather than “AppKit” – in that you can’t select and copy the details, only get an image with a screenshot. This is a “not very Mac-like” experience in my opinion. Hopefully that will get a bit of product-feature-love to encourage sharing and collaboration of the details around CreateML models.

I filed a few feedback notices with Apple for the truly egregious flaws, but I also expect they’re known, given how easy they were to spot with basic usage. They didn’t stop the app from being effective or useful, just unfortunate and kind of awkward against normal “Mac” expectations.

I do wish the details of what CreateML was doing behind the scenes was more transparent – a lot of what I’ve read about in prior research is starting with descriptions of ML models in PyTorch, Keras, or Tensorflow. If I wanted to use those, I’d need to re-create the models myself with the relevant frameworks, and then use the CoreML tools library to convert the trained model into a CoreML model that I could use. By their very nature, the models and details are available if you take that path. It’s hard to know, by comparison, how that compares to the models created with CreateML.

Creating a Model with CreateML Versus a Machine Learning Framework

My early take-away (I’m very definitely still learning) is that CreateML seems to offer a quick path to making models that are very direct, and one-stage only. If you want to make models that flow data through multiple transforms, combine multiple models, or provide more directed feedback to the models, you’ll need to step into the world of PyTorch, Keras, and Tensorflow to build and train your models. Then in the end convert the trained models back to CoreML models for use within Apple platform applications.

Where the raw frameworks expose (require you to define) all the details, they also inflict the “joy” of hyper-parameter tuning to train them effectively. That same tuning/choosing process is (I think) happening auto-magically when you build models with CreateML. CreateML chooses the iterations, while paying attention to convergence, iterations, and epochs of applying data. It also appears to do a good job of segmenting data for training, evaluation, and testing – all of which you’d need to wrangle yourself (and ideally not screw up) while making a raw machine learning model. That’s a pretty darned big win, even if it does end up being more of a “trust me, I know what I’m doing” rather than a “see, here’s what I did for you” kind of interaction.

Final Thoughts on Learning CreateML

I’m still very much in “the early days” of learning how to build and apply machine learning models, but CreateML has already been an immense help in learning both what’s possible, and providing hints as to how I might structure my own thinking about using and applying models within apps.

The first is simply that they provide a lot of good, basic models that are directly available to use and experiment with – assuming you can source the data, and that’s a big assumption. Data management is not an easy undertaking: including correctly managing the diversity of the sourcing, data licensing, and understanding the bias’ inherent within the data. But assuming you get all that, you can get good – mostly interactive – feedback from CreateML. It gives you a strong hint to answer the question “Will this concept work, or not?” pretty quickly.

The second is that showing you the outputs from the model makes the inputs you provide, and what you expect to get out, more clear. I think of it as providing a clear API for the model. Models expose a “here’s what the model thinks it’s likely to be – and how likely” kind of answer rather than a black-and-white answer with complete assurance. Not that the assurance is warranted with any other system, just that exposing the confidence is an important and significant thing.

If I were tackling a significant machine learning project, I’d definitely expect to need to include some data management tooling as a part of that project, either browser/web based or app based. It’s clear that viewing, managing, and diagnosing the data used to train machine learning models is critical.

I’m also looking forward to see what Apple releases in the future, especially considering the more complex ways machine learning is being used within Apple’s existing products. I imagine it to include more tooling, advances to existing tooling, and maybe some interesting visualization assistance to understand model efficacy. I would love for something related to data management tooling – although I suspect it’s nearly impossible to provide since everyone’s data is rather specific and quite different.

Apple’s M1 Chip Changes… Lots of Things

The new Apple Macs with an M1 chip in them is finishing a job that started a few years ago: changing my assumption that commodity hardware would always win. Having worked in the technology/computing field for over 30 years, you’d think I know better by now not to make such a broad assumption, even internally. For years, I thought the juggernaut of Intel/x86 was unstoppable, but now? Now I’m questioning that.

Apple has always prided itself on its deep integration of hardware and software. And sometimes they’ve even made good on it. The iOS (and iPadOS) integration has been impressive for the last fifteen years, and now they brought it to their laptop lineup, spectacularly so. It’s not just Apple doing this – Samsung and some other manufacturers have been down this road for a while, merging in computing silicon with sensors at a deep foundational level, changing what we think of as components within computing systems. Some of the deeply integrated cameras are effectively stand-alone systems in their own right. Lots of phones and digital cameras use that very-not-commodity component.

I still think there’s a notable benefit to leveraging commodity hardware – shoot, that’s what this whole “cloud” computing market is all about. But it’s also pretty clear that the guarantee that the win that commodity gives you won’t necessarily outweighs the commercial benefits of deep hardware/software integration.

One of the interesting things about the M1 system-on-a-chip isn’t the chip itself, but the philosophy that Apple’s embracing in making the chip. That pattern of behavior and thought goes way beyond what you can do with commodity stuff. The vertical integration allows seriously advanced capabilities. Commodity, on the other hand, tends to be sort of “locked down” and very resistant to change, even improvements. Then pile on top of that the tendency for these chip designs to be far more modular. They’re aggressively using coprocessors and investing in the infrastructure of how to make them work together. I think that’s the core behind the unified memory architecture. What Apple has done, or started to do, is invest in the ways to go even more parallel and make it easier for those co-processors to work together. In a commodity (Intel) system, the rough equivalent is the PCIe bus and the motherboard socket you plug cards into. Only that doesn’t solve the “how you share memory” or “who talks to who, and when” problems. In fact, it kind of makes it worse as you get more cards, sockets, or components.

I may be giving Apple’s hardware engineering team more credit than they’re due – but I don’t think so. I think the reason they talked about “Unified Memory Architecture” so much with this chip is that it IS a solution to the “how to get lots of co-processors to work together”, while not exploding the amount of power that a system consumes while doing so. (If you’re not familiar, memory is a “sunuvabitch” when it comes to power consumption – one of the biggest sinks for power in a modern PC. The only thing that’s worse than memory are ethernet network ports, which was a serious eye-opener for me back in the day.)

There are other computing changes that aren’t trumpeted around so much that are going to contribute equally to the future of computing. I was introduced to the world of NVMe (Non-volatile memory) a few years back, when it was just hitting it’s first iteration of commercial introduction. The “holy crap” moment was realizing that the speed at which it operated was equivalent to those power hungry memory chips. Add that into the mix, and you’ve got lots of compute, persistent memory, and far lower power requirements in the future for a heck of lot of computing. It’s the opposite direction that Intel, and nVidia, are charging into – assuming they’ll have power to spare, and can happily burn the watts to provide the benefits. Truth be told, only individual consumers were still really oriented that way – the cloud providers, over a decade ago, had already clued in that the two most expensive things in running cloud services were 1) power and 2) people.

Bringing this back to the M1 chip, it’s tackling the power component of this all very impressively. I’m writing this on an M1 MacBook, and loving the speed and responsiveness. Honestly, I haven’t felt a “jump” in perceived speed and responsiveness to a computer with a generational gap like this in over 15 years. And that’s WITH the massive power reduction while providing it.

I do wish Apple was a little more open and better about providing tooling and controls to help solve the “cost of people” equation of this situation. Yes, I know there’s MDM and such, but comparatively it’s a pain in the butt to use, and that’s the problem. It needs to be simpler, more straightforward, and easier – but I suspect that Apple doesn’t really give much of a crap about that. Maybe I’m wrong here, but the tooling to make it really, really easy to combine sets of systems together hasn’t been their strong point. At the same time, groupings and clusters of compute is just the larger reflection of what’s happening at the chip level with the M1 SOCs (For what it’s worth: SOC stands for “system on a chip”).

I think you can see some hints they might be considering more support here – talking about their AWS business interactions a bit more, and on the software side the focus on “swift on the server” and the number of infrastructural open-source pieces they help lay out over the past year that are specific to capturing logging, metrics, tracing, or helping to dynamically create clusters of systems.

I think there’a s lot to look forward to, and I’m super interested to see how this current pattern plays over of this coming year (and the next few). Like many others, I’ve raised my expectations for what’s coming. And I think those expectations will be met with future technology building on top of these M1 chips.

Thanksgiving 2020

This has been a right 🤬 of a year, and that probably applies to most anyone else on this globe. In the US, the stress of this presidential election was extreme, and acerbated by COVID pandemic that we pretty much failed to get any sort of handle on. I’ve managed to stay tucked down and safe, but my day to day life from a year ago is completely different now. My favorite “office” coffeeshop, El Diablo, is now gone – 20 years after it started. The loss has an oversized impact on my because it was also my social hub connecting me to friends in my community. Like a lot of others, my world is currently collapsed down to a pretty tiny bubble.

The downsides of this year are undeniable and clear, but there have been a number of silver linings, and since we’re heading into Thanksgiving, it seemed a good time to reflect and celebrate the positives for the year. It doesn’t remove the horror and pain, but to me – it helps offset it.

Right as the pandemic was sweeping up, I managed to get a longer term contract gig that’s been really useful for me, working on technical writing. I’ve been programming, and managing programmers, devops, QA, and the whole software lifecycle kit for multiple decades, and one of the skills that I’ve been trying to cultivate has been communication – specifically writing. Last year around this time, I self-published what was primarily a labor of love – Using Combine – a book/reference doc combination on Apple’s Combine framework. The year before I’d published a different book through Packt, Kubernetes for Developers (A google search on that phrase now directs you to more of offline training and certifications, but there’s a book in there too!). I’d been searching for editors to work with, that really dug into the work to help me structure it – not just the fluffy top level stuff that so a number of publishers stick to.

It’s the combination of these things that ends up being what tops my give-thanks list this year. In the past eight months, I’ve had a chance to work closely with a number of truly amazing editors, helping to refine and improve my writing skills. While I’m still crappy at spotting passive voice in my own writing, the feedback I’ve gotten from Chuck, Joni, Susan, Colleen, Mary Kate, and Paul has been amazing. Everything from basic grammar (that I had just never really did well) to structure, narrative flow, complexity. Top that off with some wonderful writers, Ben, Liz, Dave, and Joanna, willing to answer the odd question or just chat about the latest garden recipes on a video call, and really these past months of contracting have been a terrific experience.

I was super fortunate to find a gig when everyone else seemed to losing them. I’m sure the troubles aren’t even close to over – there’s so much to rebuild – but it’s a bit of light against the backdrop of this year.

Combine and Swift Concurrency

Just before last weekend, the folks on the Swift Core Team provided what I took to be a truly wonderful gift: a roadmap and series of proposals that outline a future of embedding concurrency primitives deeper into the swift language itself. If you’re interested in the details of programming language concurrency, it’s a good read. The series of pitches and proposals:

If you have questions, scan through each of the pitches (which all link into the forums), and you’ll see a great deal of conversation there – some of which may have your answer (other parts of which, at least if you’re like me, may just leave you more confused), but most importantly the core team is clearly willing to answer questions and explore the options and choices.

When I first got involved with the swift programming language, I was already using some of these kinds of concurrency constructs in other languages – and it seemed to be a glaring lack of the language that they didn’t specify, instead relying on the Objective-C runtime and in particular the dispatch libraries in that runtime. The dispatch stuff, however, is darned solid – battle honed as it were. You can still abuse it into poor performance, but it worked solidly, so while it kind of rankled, it made sense with the thinking of “do the things you need to do now, and pick your fights carefully” in order to make progress. Since then time (and the language) has advanced significantly, refined out quite a bit, and I’m very pleased to see formal concurrency concepts getting added to the language.

A lot of folks using Combine have reached for it, looking for the closest thing they can find to Futures and the pattern of linking futures together, explicitly managing the flow of asynchronous updates. While it is something Combine does, Combine is quite a bit more – and a much higher level library, than the low level concurrency constructs that are being proposed.

For those of you unfamiliar, Combine provides a Future publisher, and Using Combine details how to use it a bit, and Donny Wals has a nice article detailing it that’s way more “tutorial like”.

What does this imply for Combine?

First up, pitches and proposals such as these are often made when there’s something at least partially working, that an individual or three have been experimenting with. But they aren’t fully baked, nor are they going to magically appear in the language tomorrow. Whatever comes of the proposals, you should expect it’ll be six months minimum before they appear in any seriously usable form, and quite possibly longer. This is tricky, detailed stuff – and the team is excellent with it, but there’s still a lot of moving parts to manage with this.

Second, where there’s some high level conceptual overlap, they’re very different things. Combine, being a higher level library and abstraction, I expect will take advantage of the the lower-level constructs with updates, very likely ones that we’ll never see exposed in their API, to make the operators more efficient. The capabilities that are being pitched for language-level actors (don’t confuse that with a higher level actor or distributed-actor library – such as Akka or Orleans) may offer some really interesting capabilities for Combine to deal with it’s queue/runloop hopping mechanisms more securely and clearly.

Finally, I think when this does come into full existence, I hope the existing promise libraries that are used in Swift (PromiseKit, Google’s promises, or Khanlou’s promise) start to leverage the async constructs into their API structure – giving a clear path to use for people wanting a single result processed through a series of asynchronous functions. You can use Combine for that, but it is really aimed at being a library that deals with a whole series or stream of values, rather than a single value, transformed over time.

tl;dr

The async and concurrency proposals are goodness, not replacement, for Combine – likely to provide new layers that Combine can integrate and build upon to make itself more efficient, and easier to use.

The range operator and SwiftUI’s layout engine

This post is specific to Swift the programming language and SwiftUI, Apple’s newest multi-platform UI framework. If you’re not interested in both, probably best to skip past this…

I was working on visualization code that leverages SwiftUI to see how that might work, and ran into a few interesting tidbits: playgrounds with SwiftUI works brilliantly, right up until it doesn’t and then you’re in a deep, dark hole of WTF?!, and the SwiftUI layout engine, especially with nested and semi-complex container views, do some sort of iterative solver technique. The less interesting tidbit is a classic aphorism: It’s the things you think you know that aren’t true that bite you.

When you’re fiddling with adding in your own constraints in SwiftUI, you might be tempted to use the range operator – at least my thinking was “Oh hey, this looks like a super convenient way to check to make sure this value is within an expected range”. It works stunningly well for me, as long as I’m careful about creating it. I started creating ranges on the fly from variable values, and that’s where playgrounds, and my bad assumptions, bit me.

If you create a range that’s ludicrous, Swift can throw an exception at runtime. So if you’re working with ranges, you’ve got the possibility that passing in a blatantly incorrect value will give you a non-sensical range, and that will result in a crashing exception. When you stumble into this using Playgrounds, you get a crash that doesn’t really tell you much of anything. When you kick that same thing up in Xcode, it still crashes (of course), but at least the debugger will drop into place and show you what you did that was wrong. I love using SwiftUI with Playgrounds, but the lack of runtime feedback when I hit an exception – about what I screwed up – makes it significantly less useful to me.

And debugging this in Xcode was where I learned that closures you provide within SwiftUI layout, such as alignmentGuide or a method of your own creation working with a GeometryReader don’t get called just once. Sometimes they’re called one, but other times they are called repeatedly, and with pretty strange values for the view’s dimension. I think underneath the covers, there’s an iterative layout solver that’s trying out a variety of layout options for the view that’s being created. Sometimes those closures would be invoked once, other times repeatedly – and in some cases repeatedly with the same values. Interestingly, sometimes those values included a ViewDimension or GeometryProxy with a size.width of 0. The bad assumption I made was that it would be sized quite a bit larger, never zero. Because of that, I attempted to build an incorrect range – effectively ClosedRange(x ... x-1) – which caused the exception.

Even with my own assumptions biting me, I like the use of range and I’m trying to use it in an experimental API surface. Lord knows what’ll come of the experiment, but the basics are bearing some fruit. I have a bit of code where I’ve been porting some of the concepts, such as scale and tick, from D3 to use within SwiftUI.

The current experimental code looks like:

// axis view w/ linear scale - simple/short
HStack {
    VerticalTickDisplayView(
        scale: LinearScale(
            domain: 0 ... 5.0,
            isClamped: false)
    )
    VerticalAxisView(
        scale: LinearScale(
            domain: 0 ... 5.0,
            isClamped: false)
    )
}
.frame(width: 60, height: 200, alignment: .center)
.padding()

// axis view w/ log scale variant - manual ticks
HStack {
    VerticalTickDisplayView(
        scale: LogScale(
            domain: 0.1 ... 100.0,
            isClamped: false),
        values: [0.1, 1.0, 10.0, 100.0]
    )
    VerticalAxisView(
        scale: LogScale(
            domain: 0.1 ... 100.0,
            isClamped: false)
    )
}
.frame(width: 60, height: 200, alignment: .center)
.padding()

And results in fairly nice horizontal and vertical tick axis that I can use around a chart area:

NaNoWriMo Beat Sheet Scrivener Template

I participated in NaNoWriMo once before, in 2016 – having watched my partner get more deeply involved in previous years. I haven’t really been back, but this year with all the … yeah, I decided to give it a shot again. I’ve got my profile set up, at least the basics, and now I’m stumbling around trying to figure out what my story is going to encompass.

Last time through, I made a great start, but didn’t have much of a plan, so it fizzled out pretty hard about half-way through. So this time, I thought I’d try something a bit different and do a little planning. Last weekend, the local NaNoWriMo group hosted a session for planning and plotting, which I’d not tried previously. I’m going to try out using a beat sheet this time, to provide some constraints and structure for the overall story arc.

I grabbed a basic beat sheet from the NaNoWriMo writer’s prep resources, and since I have a copy the amazing writing app Scrivener, I decided to go ahead and make a NaNoWriMo template based on that.

If you are so inclined, you’re absolutely welcome to use it – it’s based on 50,000 words, deadline of 11/30, and the beat sheet google doc linked from the resources.

Open apps with SwiftUI

Earlier this week, James Dempsey asked on twitter about who else was actively trying to build macOS apps using SwiftUI. I’m super interested in SwiftUI. A year ago, it spawned my own side-project into writing my own reference docs on Combine. Originally I had a vision of writing about Combine as well as SwiftUI. Combine alone was hugely, so I stopped there with the notes, especially as SwiftUI is still massively maturing. Some pretty amazing updates came just earlier this year. While clearly not finished, likely not even close to finished, it’s now far enough along in it’s maturity that you can at least consider using it for full apps. Or parts of your app if you like – macOS, iOS, watchOS, or tvOS.

While I’m been keeping track of the framework, I’ve also been keeping track of people who are using it, writing about it, struggling with it, etc. There’s two implementations of full applications, all open source (and hence completely visible), that I’ve been following as super interesting examples of using SwiftUI: NetNewsWire and ControlRoom.

NetNewsWire

I’ve contributed a bit to NetNewsWire, only a tiny amount (and mostly around continuous integration), but I’ve been using it since the earliest days and from it’s original inception and through multiple owners to it’s current state as an open-source project that Brent Simmons is leading. The code is available online, ever evolving, at https://github.com/Ranchero-Software/NetNewsWire. The recent work to embrace SwiftUI is on it’s main branch with a lot of the SwiftUI code under the directory multiplatform/shared. Take a deep dive and dig around – there’s some gems and interesting questions, and you can see some really fascinating examples of integrating SwiftUI and UIKit or AppKit where SwiftUI isn’t quite up to some of the tasks desired by the project.

ControlRoom

The other app I’ve been watching is ControlRoom, an app that Paul Hudson referenced in a demo capture on twitter. ControlRoom’s code is on Github at https://github.com/twostraws/ControlRoom, released earlier in SwiftUI’s lifecycle, and showing an integration not of the new SwiftUI app architecture pieces, but more of “classic” macOS AppKit integration. Like NetNewsWire, I found a number of really great gems within the code, often having “light-bulb” moments when I understood how the app accomplished some of its goals.

Others…

There are easily other apps out there, that I’m unaware of – but not too many folks are openly sharing their development like the two projects above. I did find a list of open-source IOS apps on GitHub that includes a subset listing SwiftUI, that might be interesting.

I have a few of my own experiments, but nothing as polished and effective as these two, and I don’t think I solve any problems in novel ways that they haven’t. In a bit of test and benchmarking code, I was creating SwiftUI interfaces across macOS, iOS, and tvOS – which turns out to be a right pain in the butt, even for the simplest displays.

I hope, but don’t expect, more apps to become available – or to be more visible down the road. Having the open sharing of how they solved problems is invaluable to me for learning, and even more so for sharing. Apple has their sample code, well – some of it anyway – but seeing folks outside of Apple use the framework “in the wild” really shows it’s working (or where it isn’t).

Learning to Write, Writing to Learn

Writing, while I love it, doesn’t come naturally to me. I suspect it doesn’t come naturally to any writer. The process of getting something written, really tightly focused and right on target, is a heroic act of understanding, simplification and embracing constraints. It’s a skill for which I don’t have a good analogue in the other kinds of work I’ve done. Maybe the closest is the finish work (sanding and polishing) with jewelry making or metal-work, but that doesn’t quite fit. I suspect there are good analogues, I’m just not spotting them.

A few weeks ago I wrote about my challenges with the learning to write process, complicated by COVID and the overall lockdown. While I broke through that plateau, there are always more challenges to work through. This last week, I hit another slog-swamp – this time it’s more about mental framing than my actual writing and feedback loops.

Part of why I’m doing technical writing is that writing is an act of sharing that has an incredibly long reach. It’s a really big-damn-lever in the move the world kind of metaphor. That makes it, to me, a supremely worthy endeavor.

To really do it well, you need to know the subject you’re writing about: backwards and forwards, from a couple different angles, maybe even coming in from a different dimension or two. I embraced the idea of “If you can explain something simply, then you may understand it.” That’s the “writing to learn” part of this – what I’m doing with the writing is forcing myself to learn, to think about the subject matter from different angles.

The hardest part of that learning is figuring out the angles that don’t come naturally, or from which I don’t have a lot of background. I’m generally fairly empathetic, so I’ve got at least a slight leg up; I can often at least visualize things from another point of view, even if I don’t fully understand it.

The flip side of it, learning to write, happened this week. I completed a draft revision, and when I reviewed with some folks, I realized it fell way off the mark. There were a few things that I thought were true (that weren’t) that knocked it awry. More than that the feedback I got was about taking things to a more concrete example to help reinforce what I was trying to say. Really, it was super-positive feedback, but the kind of feedback that has me thinking I might need to gut the structure pretty heavily and rebuild it back up. As the weekend closes out, I suspect an act of creative destruction might be coming in order to reset this and get it closer to where it needs to be.

I’ve been noodling on that feedback most of this weekend – it has been percolating in my rear-brain for quite a while. Aside from the “things you thought were true that weren’t” (the evil bits that get you, no matter what the topic area) that I got straight pretty quickly, the key from this feedback is that while I was correct in what I was touching on in the writing, it was too abstracted and too easily mis-undertstood. Especially in the world of technical writing and programming topics, it’s SUPER easy to get “too abstract”. And then there’s the death knell of what should be good technical writing – too abstract AND indirect.

Embracing the constraint of “making it more concrete” is some next-level thinking. It’s forcing me to be less abstract (a good thing) and it’s taking a while to really see how to make that work for the topic I’m writing about. I mean, really, I’m still working on the “nuke all the passive voice” thing. While I’m getting better, it takes me something like 3 or 4 passes, reading sentence by sentence, to spot them in my writing.

For what it’s worth, I’m loving the “nuke passive voice” constraint. I love that it forces the writing to be specific about what takes action and what the results are – so much hidden stuff becomes clear, and it’s a great forcing function to see if you also really understand how it’s working.

For now I’ll continue to noodle on how to make my example more concrete, and get back to my baking and kitchen cleaning for the afternoon. Come tomorrow, I’ll take another pass at that article and see what I can do for the concrete-ness aspect.

Feeling alone and outside of your comfort zone

Month five of the COVID lockdown, and when it started I picked up a new bit of work. It is something I’d wanted to do and from which I get enjoyment: technical writing. I am definitely stepping outside of my comfort zone. Although I’ve written extensively and am a published author with several titles, the skills of grammar, spelling, and word choice aren’t what I’d describe as my super powers.

The past few weeks have been a bit harder that usual, as I’ve been pushing past the basics, and breaking through the plateau where I felt comfortable and knew what I was doing. It is doubly-hard with COVID lockdown and remote-only work, as none of the avenues I’ve used in the past to vet ideas or check thinking and progress has been easily available to me. At it’s heart, writing is about communicating – and the technical writing I’m doing is aimed at being precise, accurate, concise, and easy to understand. Sometimes I’ve got some gems and it works well. Other times I stare at a single paragraph for the better part of 3 hours, tearing apart the sentences word by word and setting them into the form that I think may work – because I certainly don’t think or speak with that level of simple, direct, and accurate conciseness.

I’m confident I’ll get this, eventually. It will take longer than I’d like to get through the “Man, I suck at this” stage – as it always does when you’re learning something. It feels terrible at the moment. I often feel lost, sometimes confused, and – not surprisingly – frustrated. The constraints are a gift, but I rail at them just the same. It’s awkward and painful, and the assistance I have been able to find from coworkers or my fellow coffee-house peers in bouncing ideas around is gone or greatly reduced.

I’m determined to make something better in this whole mess. One of the few things I can control is what I’m working on, and I have that luxury – so I’m using the current time and constraints to improve myself.

If you’re doing the same, remember that it’s worth acknowledging that this shit ain’t easy, whatever your skill or task may be. If it’s worth improving, then it won’t be – almost by the very nature of it. Doesn’t matter if it’s hand-eye timing coordination and mastery for a video game, learning to paint in a new style, strategic puzzle solving of board games, or learning to set a perfect weld. Keep at it and keep moving, even if it doesn’t always feel like forward motion.

post-WWDC – more device collaboration?

It’s been two weeks since WWDC as I’m writing this. I certainly haven’t caught all the content from this year’s event, or even fully processed what I have learned. I see several patterns evolving, hear and read the various rumors, and can’t help but wonder. Maybe it’s wishful thinking, or I’m reading tea leaves incorrectly, but a few ideas keep popping into my head. One of them is that I think we would benefit from — and have the technology seeds for — more and better device collaboration between Apple devices.

One of the features that SwiftUI released this year was leveraging a pattern of pre-computed visualizations for use as complications on the Apple Watch, and more generally as Widgets. The notion of a timeline is ingenious and a great way to solve the constrained compute capability of compact devices. This feature heavily leverages SwiftUI’s declarative views; that is, views derived from data.

Apple devices have frequently worked with each other – Phone and Watch pairing, the Sidecar mechanism that’s available with a Mac laptop and an iPad, and of course AirPlay. They offload data, and in some cases they offload computation. There’s creative caching involved, but even the new App Clips feature seems like it’s a variation on this theme – with code that can be trusted and run for small, very focused purposes.

Apple has made some really extraordinary wearable computing devices – the watch and AirPods, leveraging Siri – in a very different take than the smart speakers of Google Home and Amazon’s Alexa. This year’s update to Siri, for example, supports for on-device translation as well as dictation.

Now extrapolate out just a bit farther…

My house has a lot of Apple devices in it – laptop, watch, AirPods, several iPads, and that’s just the stuff I use. My wife has a similar set. The wearable bits are far more constrained and with me all the time – but also not always able to do everything themselves. And sometimes, they just conflict with each other – especially when it comes to Siri. (Go ahead – say “Hey Siri” in my house, and hear the chorus of responses)

So what about collaboration and communication between these devices? It would be fantastic if they could share sufficient context to support making the interactions even more seamless. A way to leverage the capabilities of a remote device (my phone, tablet, or even laptop) from the AirPods and a Siri request. They could potentially even hand-off background tasks (like tracking a timer) or knowing which device has been used most recently to better infer context for a request. For example, I want a timer while cooking often on my watch, not my phone – but “hey Siri” is not at all guaranteed to get it there.

That they could also know about the various devices capabilities and share those capabilities would make the whole set even smarter and more effective, and depending on which rumors you are excited by – they may be able to do some heavier computation off the devices that are more power constrained (wearables) by near-by but not physically connected (and power efficient) microprocessors. That could be generating visuals like Widgets, or perhaps the inverse – running a machine learning model against transmitted Lidar updates to identify independent objects and their traits from a point cloud or computed 3D mesh.

It’ll be interesting to see where this goes – I hope that distributed coordination, a means of allowing it (as a user), and a means developing for it is somewhere in the near future.

%d bloggers like this: