Nested Observable Objects in SwiftUI

This one often starts with the phrase:

Hey, why isn’t my view updating? It shows the initial data, but it doesn’t update when that data gets changed!

… more than one person, including me …

When you get into seeing the code for the view, how it’s formed, and what the models look like, you see a pattern appear: nested objects that conform to ObservableObject with a reference from one to another, a top level object passed into the view, and view elements that follow the dot-notation chain to create the display.

Let me show you some code, a very simplified representation of this pattern:

class MainThing : ObservableObject {
    @Published var element : SomeElement
    init(element : SomeElement) {
        self.element = element
    }
}

class SomeElement : ObservableObject {
    @Published var value : String
    init(value : String) {
        self.value = value
    }
}

And a view that displays it:

struct MainThingView: View {
    @ObservedObject var model : MainThing
    var body: some View {
        HStack {
            Text("Detail:")
            Text(model.element.value)
        }
    }
}

At first blush, this looks fine – the view displays, and property within the nested view is shown as you’d expect; so what’s the problem? The issue is when you update that nested element’s property, even though it’s listed as @Published, the change doesn’t propagate to the view.

I’ve seen this pattern described as “nested observable objects”, and it’s a subtle quirk of SwiftUI and how the Combine ObservableObject protocol works that can be surprising. You can work around this, and get your view updating with some tweaks to the top level object, but I’m not sure that I’d suggest this as a good practice. When you hit this pattern, it’s a good time to step back and look at the bigger picture. What are you setting up with your views and models, and can you make them a more aligned to a direct representation. Let me explain what’s happening, how you can work around it, and you can judge for yourself.

This pattern is at the intersection of Combine and SwiftUI, and specific to classes. There are two parts to it, the first of which is the ObservableObject protocol, and the second part is one or more properties on that object that have the @Published property wrapper. An object that conforms to the observable object protocol has a publisher on it with a specific name: objectWillChange. You can dig around a bit, and you’ll find what it publishes may not be what you expect: it isn’t publishing the values changing, just that something will change. The type aliases for this publisher point to a type signature of Publisher<Void, Never>. Not what I expected when I first uncovered it, and it made me scratch my head.

The idea, as I understand it, is that the publisher is specifically meant to provide a signal that something has changed – but not the details of what changed. From there, the code within the SwiftUI framework (which uses it), invalidates any relevant view, and looks up what it needs from the referenced object to display a new view.

When most folks use this protocol, they’re not creating the publisher – they’re letting the swift compiler do the heavy lifting, which synthesizes the code to create it, and with the @Published property wrapper, to hook up and watch the properties that should trigger it.

So here’s the kicker to what’s happening: The @Published property wrapper watches for the properties to have changed. We’re dealing with a class here, so we’re in the world of reference semantics. When you update something in a class, you’re not updating the reference to the class – the reference stays the same. That’s the benefit (and trouble) with reference semantics – it’s not entirely obvious that something down below that reference was updated, but as a benefit – you’re not having to copy around the world of what’s in there. If you had replaced the element property with a new instance of SomeElement, then it would trigger the publisher.

So what can you do to work around this? If you’re really tied to this nested class object structure, then you change the objects a bit to “manually” support notifying that publisher chain when the property within a nested object has updated, and you want that reflected in a SwiftUI view. One way to tackle this is to add your own connections from the synthesized publishers on internal @Published property wrappers to the synthesized subject that ObservableObject provides.

The @Published property wrapper synthesizes a publisher for you, referenced with a $ preceding the property name. The compiler synthesizes a Subject for ObservableObject. The subject has a send() method on it that you can invoke. Invoking send() doesn’t require any arguments – it’s not sending any specific data – instead it’s a trigger to say “publish the fact that something is changing and views should be invalidated and redisplayed”. The code below explicitly connects the publisher synthesized within SomeElement to the subject in MainThing.

class MainThing : ObservableObject {
    @Published var element : SomeElement
    var cancellable : AnyCancellable?
    init(element : SomeElement) {
        self.element = element
        self.cancellable = self.element.$value.sink(
            receiveValue: { [weak self] _ in
                self?.objectWillChange.send()
            }
        )
    }
}

There’s a whole world of problems with this setup: from breaking the idea of encapsulation across objects to the fact that it’s incredibly fragile. If you change the element property within MainThing to a new instance, you also need to re-establish the publisher chain to the objectWillChange subject. You can manually create your own subject, such as a PassthroughSubject<Void, Never> to your top-level class and manage the connections to invoke send() on it.

This whole pattern seems to be a “code smell“. If you follow this primrose path, you’ll find yourself triggering the “invalidate and redisplay” at a high level, perhaps more often than you want. A large set of those additional connections, especially with a model changing regularly, pretty quickly leads to performance issues, as the effects invalidate larger swaths of view hierarchy for potentially small changes. One or two connections like this won’t hurt much, but more certainly can.

What seems to be better advice is to look closely at your views, and revise them to make more, and more targeted views. Structure your views so that each view displays a single level of the object structure, matching views to the classes that conform to ObservableObject. In the case above, you could make a view for displaying SomeElement (or even several views) that display’s the property from it that you want shown. Pass the property element to that view, and let it track the publisher chain for you.

struct FocusedView: View {
    @ObservedObject var element : SomeElement
    var body: some View {
        Text(element.value)
    }
}

struct MainThingView: View {
    @ObservedObject var model : MainThing
    var body: some View {
        HStack {
            Text("Detail:")
            FocusedView(element: model.element)
        }
    }
}

This pattern implies making more, smaller, and focused views, and lets the engine inside SwiftUI do the relevant tracking. Then you don’t have to deal with the book keeping, and your views potentially get quite a bit simpler as well.

Integrating SwiftUI Bindings and Combine

A misconception I had when first learning SwiftUI and Combine was that SwiftUI relied on Combine alone for updating data. There was a throw-away comment in one of the 2019 WWDC presentations (Data Flow through SwiftUI) relating the two, and I over-interpreted it to mean that SwiftUI solely used Combine. To be very clear – it doesn’t. SwiftUI nicely integrates with Combine, and the components you use to expose external reference models into SwiftUI (such as @ObservedObject, @EnvironmentObject, @StateObject, and @Published) use it. But quite a lot of the interaction with user interface elements, such as Text, Toggle, or the selection in List operate using a different tool: Bindings.

I think the most important distinction between the two is the direction of data flow. With bindings, the data travels in BOTH directions, and in Combine it travels in a SINGLE direction. You could describe Combine pipelines as being a “one way street for data”, and Bindings as a “two way street”.

Bindings are also not set up to manipulate the timing or transform the data; and the operate on individual values – not streams of data or values over time. That is entirely by design. A core philosophy of the design of SwiftUI is having a single source of truth. A binding exposes that single source of truth, and allows user interface elements to both reflect it and change it. All the while, it keeps the source of truth clear and easily understood. It is designed to support high-speed, low-overheard updates of single values within a view.

So when you start to think something like “Hey, I’d really like to tweak the timing on this…”, then you’re in the realm of Combine. An example might be debouncing text field updates and using the result to trigger a more expensive computation, such as a network call to get information. Unfortunately, there’s not a direct path to create or get a publisher from an existing binding, such as a @State variable within a view.

So how do you get these things to work together? You can rearrange your code to pass in publishers, or otherwise externalize the data from your view with on of the ObservableObject types. If you want to work on data that the view should own, such as view state, then the best option are a couple of view modifiers: onReceive(_:perform:) and onChange(:of:perform:).

Publishing to a Binding

Linking a publisher to a binding, such as a view’s state variable, is in my mind the easier path. How you arrange it depends on where you create the publisher. If the publisher exists outside the view, pass it into your view on initialization, and use it with the onReceive(_:perform:) view modifier.

The view to which its attached becomes subscriber. Remember that in Combine, the subscriber “drives all the action”, so the view is now driving the publisher, for as long as it exists. When the onReceive modifier connects to the publisher, it requests unlimited demand. When SwiftUI invalidates the view and recreates it, the pipeline will get set up again, and requests additional demand from the publisher. If your publisher does heavier work, such as a network request, you might find this a bit surprising.

If you do run into this, it’s a good idea to move that publisher, and the work, into an external object that isn’t owned by the view, but used by it. You can use the publisher from multiple views, and it’s a perfect place for the Combine share operator to use a single publisher from multiple subscribers without needing to replicate the network requests for each one.

Publishing from a Binding

Prior to the SwiftUI updates in iOS 14, publishing from a binding was pretty much squashed. Local state declarations, such as a private state variable, don’t support tacking on your own custom getters and setters, where you could put in a closure to trigger a side effect.

Fortunately, with iOS 14, SwiftUI added the onChange(of:perform:) view modifier, perfectly suited for invoking your own closure when a state variable changes. Specify the binding you to which you want to watch, and do your work within a closure you provide. To get from a closure, an imperative style of code, to a Combine publisher, a declarative style of code, means you need to work with something that crosses that boundary. The most common way is using a Combine subject, designed for imperative code to publish values. I tend to use passthroughSubject, especially within a view, since it’s lightweight and doesn’t require references to other values. This makes it easy to use in a declaration, outside of an initializer.

Again, remember that when using Combine, the subscriber is what drives the action. So be wary of thinking of this flow as pushing a value into a publisher chain; that can lead you to making some incorrect assumptions about how things will react. What’s typically happening is something, somewhere, is already subscribed to the publisher – and it did so with an unlimited demand – meaning any values added in get propagated almost immediately.

Sample Code

To illustrate how these work, I created a simple view, with both a binding sending a value into a publisher, and another getting updated from a publisher chain. The example includes two state variables: one that is the immediate binding for a TextField that updates as you type, the second is a holding spot for values out of a publisher.

A PassthroughSubject provides the imperative to declarative connection by using send() to publish values, called from within an onChange view modifier on the TextField. To debounce the input – in this case delay updating until you’ve stopped typing for a second – the publisher pipeline in the initializer pulls from the subject, uses the debounce operator, and connects to the state variable delayed using the onReceive view modifier.

import SwiftUI
import Combine

struct PublisherBindingExampleView: View {
    
    @State private var filterText = ""
    @State private var delayed = ""
    
    private var relay = PassthroughSubject<String,Never>()
    private var debouncedPublisher: AnyPublisher<String, Never>
    
    init() {
        self.debouncedPublisher = relay
            .debounce(for: 1, scheduler: RunLoop.main)
            .eraseToAnyPublisher()
    }
    
    var body: some View {
        VStack {
            TextField("filter", text: $filterText)
                .onChange(of: filterText, perform: { value in
                    relay.send(value)
                })
            Text("Delayed result: \(delayed)")
                .onReceive(debouncedPublisher, perform: { value in
                    delayed = value
                })
        }
    }
}

You might be tempted to put the combine pipeline directly on the subject, but if you chained any operators, then you would be changing the type of the subject. That would effectively hide it and make it impossible to get a reference to the send() method.

Likewise, you might want to move the publisher chain up to be a declaration, but there’s a quirk here: initialization order. Since you need to reference another local variable (relay in this case), you need to set up that chain from inside a convenience initializer so that the other variables are already initialized and available to use. If you tried to just set it up in the declaration, the compiler would hand you an error such as:

Cannot use instance member 'relay' within property initializer; property initializers run before 'self' is available

You might say, “Hey, but what about that neat trick with lazy initialization?”. In this case, it doesn’t conveniently work. When you use a variable declared with lazy within the onReceive view modifier, you get the error:

Cannot use mutating getter on immutable value: 'self' is immutable

A lazy property mutates an object on its first invocation, so it’s always considered mutating, which doesn’t work with SwiftUI views, which are immutable.

As a final note, keep what you set up and process within a view’s initializer to an absolute minimum. SwiftUI views are intended to be light and ephemeral – and SwiftUI invalidates and recreates them, somethings quite often, so minimize any work you do within them. If, for example, you find yourself reaching to set up a publisher that makes a network request to get some data, seriously consider refactoring your code to externalize that request into a helper object that provides a publisher, rather than have the view repeatedly set it up each time.

The evolution of “safe” and “unsafe” in the Swift programming language

There’s been a lot of motion in the last four months of the evolution of the Swift programming language that I’ve been wanting, waiting, and hoping for. The language maintainers are tackling concurrency as a first-class construct in the language. I’m following along with the language evolution proposals in the forums, and so far have mostly been able to keep up with the details – although I’m sure I’m missing a lot of the fine-grained implications. Reading the forum pitches where people are talking them through has been fascinating.

One of the interesting take-aways is that the terms “safe” and “unsafe”, or at least the specific implications of when they’re used in the swift language, are broadening what they cover with the upcoming changes. You could start to see it as early as last October when the Swift Concurrency Roadmap was published, but the wording wasn’t fully in place, more of just conceptual frameworks. The details of the broadening of the definition didn’t hit home for me until I caught up with the recent discussion on the pitch for task local values.

Prior to these language extensions, “safe” implied something pretty narrow and specific: memory safety. It started with some swift-language specific guarantees about variables being guaranteed to be initialized before use. There’s a Swift.org blog post from back in 2015 that calls this out, and far more detail in The Swift Programming Language book’s chapter on Memory Safety. There’s an even better explanation and detail in the presentation Unsafe Swift from WWDC 20. The heart of it being that the “safe” APIs have more preconditions and guarantees wrapped around them.

Across the recent pitches and proposals, some of the language terms that use safe are now being used to imply concurrency safety, somewhat independently of memory safety. The goal looks to be to provide APIs that have some guarantees about thread-safe access and updates. And along with the safe versions, there are some potential “unsafe” variants to use when you need the escape hatch and are willing to take on the thread safety guarantees yourself.

There is likely going to be some detail about what it means to be safe in the upcoming 3rd pitch for Structured Concurrency (both the task local values pitch and the 2nd structured concurrency pitch mention another round of updates and details for the structured concurrency proposal. As I’m writing this, it’s clear there’s still a lot of active and careful work going on there.

M1 arm64 native OpenSSL with vcpkg

This article isn’t a how-to so much as a debugging/dev diary entry for future-me, and any other soul who stumbles into the same (or similar) issues.

Let me provide the backdrop for this story:

I’m working on a private C++ language based project, previously written to be cross platform (Windows, Linux, and Mac). It has a number of C++ library dependencies, which it’s managing with vcpkg, a fairly nice library package manager solution for C++ projects. It happens to align well with this project, which uses CMake as its build system. One of the dependencies that this project uses is grpc, which in turn has a transitive dependency on OpenSSL.

With the M1 series of laptops available from Apple, we wanted to compile and use this same code as an M1 arm64 native binary. Sure – makes sense, should be easy. The good news is, it mostly has been. Both vcpkg and openssl recently had updates to resolve compilation issues with M1/arm based Macs, most of which revolved around a (common) built-in assumption that macOS meant you were building for an x86_64 architecture, or maybe, just maybe, cross-compiling for iOS. The part that hasn’t been so smooth is there’s an odd complication with vcpkg, openssl, and the M1 Macs that ends up with a linker error when the build system tries to integrate and link the binaries created and managed by vcpkg. It boils down to this:

ld: in /Users/heckj/bin/vcpkg/installed/arm64-osx/debug/lib/libcrypto.a(a_strex.o), building for macOS, but linking in object file built for iOS

For anyone else hitting this sort of thing, there’s two macOS specific command-line tools that you should know about to investigate this kind of thing: lipo and otool.

lipo

lipo does a number of things – mostly around merging various archives into fat libraries, but the key part I’ve been using it for is to determine what architecture a library was built to support. The command lipo -info /path/to/library.a, tells you the architecture for that library. I stashed a copy of vcpkg in ~/bin on my laptop, so using the command lipo -info ~/bin/vcpkg/installed/arm64-osx/lib/libcrypto.a reports the following:

Non-fat file: /Users/heckj/bin/vcpkg/installed/arm64-osx/lib/libcrypto.a is architecture: arm64

Prior to OpenSSL version 1.1.1i, that reported an x86_64 binary.

otool

As I’ve learned, architecture alone isn’t sufficient for C++ code when linking the library (at least on macOS). When the libraries are created (compiled), the libraries are also marked with information about what platform they were built for. This is a little harder to dig out, and where the command line tool otool comes into play. I found some great detail on Apple’s Developer forum in the thread at https://developer.apple.com/forums/thread/662611, which describes using otool, but not quite all the detail. Here’s the quick summary:

You can view the platform that is embedded into the library code directly using otool -lv. Now this generates a lot of output, and you’re looking for some specific patterns. For example, the command

otool -lv ~/bin/vcpkg/installed/arm64-osx/debug/lib/libcrypto.a  | grep -A5 LC_

includes this stanza in the (copious) output:

      cmd LC_VERSION_MIN_IPHONEOS
  cmdsize 16
  version 5.0
      sdk n/a
Load command 2

And as far as I’ve been able to discern, if you see LC_VERSION_MIN_IPHONEOS in the output, it means the library was built for an iOS platform, and you’ll get the linker error I listed above when you try to link it to code built for macOS.

Another library which did get compiled “correctly” shows the following stanza within its output:

      cmd LC_BUILD_VERSION
  cmdsize 24
 platform MACOS
    minos 11.0
      sdk 11.1
   ntools 0

Spotting LC_BUILD_VERSION and then the details following it shows the library can be linked against macOS code build for version 11.0 or later.

Debugging

After a number of false starts and deeper digging, I found that OpenSSL 1.1.1i included a patch that enabled arm64 macOS compilation. The patch https://github.com/openssl/openssl/pull/12369 specifically enables a new platform code: darwin64-arm64-cc.

The vcpkg codebase grabbed this update recently, with patch https://github.com/microsoft/vcpkg/pull/15298, but even with this patch in place, the build was failing with the error above.

I grabbed OpenSSL from its source, and started poking around. A lot of that poking and learning the innards of how OpenSSL does its builds wasn’t entirely useful, so I won’t detail all the dead ends I attempted to follow. What I did learn, in the end, was that the terminal – being native arm64 or rosetta2 emulated x86_64 – can make a huge difference. For convince, I made a copy of iTerm and ran it under rosetta so that I could easily install homebrew and use all those various tools, even though it wasn’t arm64 native.

What I found is that if I want to make a macOS native version of OpenSSL, I need to run the compilation from a native arm64 terminal session. Something is inferring the target platform – not sure what – from the shell in which it runs. I manually configured and compiled OpenSSL with an arm64 native terminal, and was able to get an arm64 library, with the internal markers for macOS.

From there, I thought that perhaps this was an issue of how OpenSSL was configured and built. That’s something that vcpkg controls, with these little snippets of cmake under the covers. vcpkg does a nice job of keeping logs, so I went through the compilation for openssl (using --triplet arm64-osx in case you want to follow along), and grabbed all the configuration and compilation steps. I grabbed those logs and put them into their own text file, and edited them so I could run them step by step in my own terminal window and see the status of the output as it went. I then adapted the steps to reference a clean git repository checkout from openssl and the tag OpenSSL_1_1_1i, and updated the prefix to a separate directory (/Users/heckj/openssl_arm64-osx/debug) so I could inspect things without stepping into the vcpkg spaces.

cd /Users/heckj/src/openssl
git reset --hard OpenSSL_1_1_1i
export CC=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc
export AR=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ar
export LD=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld
export RANLIB=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib
export MAKE=/usr/bin/make
export MAKEDEPPROG=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc
export PATH=$PATH:/Users/heckj/bin/vcpkg/downloads/tools/ninja-1.10.1-osx
/usr/bin/perl Configure no-shared enable-static-engine no-zlib no-ssl2 no-idea no-bf no-cast no-seed no-md2 no-tests darwin64-arm64-cc --prefix=/Users/heckj/openssl_arm64-osx/debug --openssldir=/etc/ssl -fPIC "--sysroot=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.1.sdk"
/usr/local/Cellar/cmake/3.19.3/bin/cmake -DDIR=/Users/heckj/src/openssl -P /Users/heckj/bin/vcpkg/ports/openssl/unix/remove-deps.cmake
# [2/3]
cd /Users/heckj/src/openssl
export PATH=$PATH:/Users/heckj/bin/vcpkg/downloads/tools/ninja-1.10.1-osx
/usr/local/Cellar/cmake/3.19.3/bin/cmake -E touch /Users/heckj/src/openssl/krb5.h
/usr/bin/make build_libs

What I found was the if I ran this code under a terminal running under rosetta, I’d get the results that indicated the code was built against an iOS platform. And when I ran it under a native arm64 terminal, it would correctly report macOS as the platform.

This is a major insight, but I haven’t yet figured out how to apply it…

I originally installed and compiled vcpkg using the rosetta terminal, and it was running as an x86_64 binary, so I thought perhaps that was the issue. Unfortunately, not. After I installed vcpkg with the arm64 native (and verified the binary was arm64 with the lipo -info command), I made another run at installing openssl, but ended up with the same iOS linked binary.

Prior to getting this far, I opened issue 13854 as a question on the OpenSSL repository, which details some of this story. However, I now longer think that’s an issue, as I was able to get an arm64 native binary when I manually compiled things. There might be something OpenSSL could do to make this easier/better, but its build setup is incredibly complex and I get lost pretty darn quickly within it.

So to date, I’ve trailed this back to some interaction that vcpkg, and x86_64 emulated binaries, are having on the build – but that’s it.

The story ends here, as I don’t have a solution. I have filed issue 15741 with the vcpkg project, with a summary of these details.

For anyone reading until the bitter end, I’d love any suggestions on how to fully resolve this. I hope that someone stumbles across the issue at some point with more knowledge than I and has a solution in the future. In the meantime, this blog post will hopefully record the error and how you can diagnose the architecture that a library is compiled for, even if it doesn’t solve the end problem of getting you to a final resolution.

Priming and Pedagogy

A new year’s wander through a machine learning research paper led me down a really interesting rabbit role. Somewhere down the hole, I found a reference and have since been reading Thinking Fast and Slow, by Daniel Kahneman. The book was referenced by several AI/ML researchers, which is what got me started there.

Chapter 4, titled The Associated Machine, is an interesting example of this book. It talks about the psychological effect of priming, and then goes on to illustrate it in the book with simple examples you can practically do while you’re reading the chapter. That kind of example brings the concept home, makes it super concrete and far easier to understand. That’s a delightful mechanism to find in a book where you’re reading to learn.

That chapter, both its content and how it presented the topic, spurred a wacky question in my head:

Are there priming techniques that could be used in technical documentation, either subtly or directly, that makes it easier for the reader to read and retain the content?

I don’t know if there is such a thing. From the examples in the book, I can see where there’s definitely priming that could be done that could work against taking up details, but I’m not coming up an inverse. The closest I could imagine was a relatively simple predictive puzzle of some sort, just something to get you in a puzzle-solving frame of mind, and then tackling the learning. Of course, if you’re one that hates puzzles that’s is likely going to “crash and burn” as a technique.

I’ve no grand conclusions, only more questions at this stage. Although I do highly recommend the book if you’re interested in the processes of cognition, associating, and how they relate to our everyday brain capabilities.

Creating Machine Learning Models with CreateML

I have been following the bare outlines of building, and using, machine learning models in Apple’s software ecosystem for a while. Most of my learning and personal research has been with foundational technologies – following some of the frameworks (TensorFlow, PyTorch, SciKit-Learn) and some of the advances in models and their results. Until this holiday, I had not applied myself to seeing the latest evolution of Apple’s machine learning frameworks and tooling. I scanned through websites last summer during Apple’s WWDC, but didn’t really clue in to the changes that were coming, and that are now available, with Big Sur (macOS 11).

Aside: The web site Papers With Code has been a great consolidated reference point for digging into the academic research behind quite a variety of machine learning advancements over the past couple of years, and includes both the research papers (often linked to ArXiv) and frequently references to the code matching the papers.

First off, the tooling to create simple models with CreateML has received quite a boost. Matched by the documentation, there are a large number of new models that can be easily generated with you “just” providing the data. Models to classify sounds, motion data, and tabular data – including regressors (predictors) as well as classifiers. Prior to macOS 11, they had models for classifying and tagging words and sentences, as well as image classification. Those all still exist, and Apple provides some ready-to-use models built-in to frameworks such as Natural Language.

The goal I chose to explore with was focused on natural language processing – more on that later. Apple has long had a pretty good natural language library available, the earlier stuff being a bit more focused on Latent Semantic Mapping, but recently turning to – and leveraging – quite a bit of the natural language processing advances that have happened using more recent machine learning techniques.

The most interesting win that I immediately saw was the effectiveness of using transfer learning with CreateML. When you’re making a model (at least the word tagger model), you have the option of using a CRF (conditional random field) or applying the data with a transfer learning over existing models that Apple has in place. They don’t really tell you anything about how these models are built, or what goes into choosing the internals, but the bare results from a few simple experiments are positive, and quite obviously so.

I made sample models that mapped language dependency mapping from some publicly available datasets provided by Universal Dependencies. Dependency maps specify how the words relate to each other, and the part that I was specifically interested in was leveraging the ability to identify a few of those specific relationships: nsubj:pass and aux:pass. These are the tell-tales for sentences that use passive voice. I did multiple passes at making and training models, and transfer learning was clearly more effective (for my models) in terms of the reported accuracy and evaluation.

Training a Model

One of my training runs used approximately 5000 labeled data points (from https://github.com/UniversalDependencies/UD_English-GUM). About 4200 of those points I allocated for training, and another separate file with 800 data points for testing. In this case, a “data point” is a full or partial sentence, with each word mapped to an appropriate dependency relationship. The CRF model topped out at 81% accuracy, while the transfer learning model reached up to 89% accuracy.

Conditional Random Field Model: 11 iterations planned, converging “early” at 7 iterations.
Transfer Learning: Dynamic Embedding (25 iterations)

To put the accuracy in perspective, the latest “state of the art” parts-of-speech tagging models are running around 97 to 98% accuracy, and around 96% for dependency parsing.

Timing the Training

I ran these experiments on one of the new M1 MacBooks with 16GB of memory. This training set took just over a minute to train the CRF model, and 7 minutes to train the transfer learning model. A similar run with more data (27,000 data points) took 20 minutes for the CRF model, and 35 minutes for the transfer learning model. Transfer learning takes longer, but – in my cases – resulted in better accuracy for predictions.

Evaluating the Model

Once trained, CreateML provides an evaluation panel that gives you precision and recall values for each value that in your tagging set. This information is ideal for understanding how your data actually played out versus what you were trying to achieve. Using it, you can spot weak points and consider how to resolve them. One way might be gathering more exemplar data for those specific classifications. For example, in the following data table, you can see that the recall for “csubj:pass” was 0%. This showed that I simply didn’t have any examples in the test data to validate it, and possibly only a few samples in my training data. If that was a tag I was interested in making sure I could predict from input text – I could find and improve the input and testing data to improve that accuracy.

Previewing Predictions from your Model

Probably the most fun of using CreateML is the preview mode. Since I was making a word tagging model, it provided a text entry area and then applied the tagger against the data, showing me an example of the predictions against what-ever I typed.

Since I’d done two different models in the same project, I could flip back and forth between them to see how they fared against this sample, showing me the predictions from the model and the associated confidence values of those predictions. (and yes, I have considered locking my brother into a kitchen, he’s a damned fine chef!)

Exporting a CoreML model from CreateML

CreateML includes an Output tab that shows you the tags (also known as class labels) that you trained. It also gives you a preview of the metadata associated with the model, as well as kinds of inputs and outputs that the model supports. This makes it nicely clear on what you’ll need to send, and accept back, when you’re using the model in your own code.

One of the details that I particularly appreciated was including a metadata field explicitly for the license of the data. The data I used to create this model is public, but licensed to by-nc-sa-4.0 (An attribution, non-commercial, share-alike license). Sourcing data, both quality and licensed use, is a major element in making models. I think it’s super important to pass that kind of information along clearly and cleanly, so I’m glad it’s a default metadata attribute on models from CreateML.

Notes on CreateML’s Rough Edges

While CreateML was super easy to use and apply, it definitely has some issues. The first is that the user interface just doesn’t feel very “Mac” like – things I expected to be able to “click on and rename” didn’t smoothly operate as such (although I could control-click and choose rename), the window doesn’t easily or cleanly resize – so the app dominates your screen wether you want it to or not. On top of that, a queueing feature, and related window, for lining up a bunch of training was quite awkward to use and see the results as it was progressing, and after it was done. I had a couple training runs fail due to poor data, but the queuing window didn’t show any sort “well, that didn’t work” information – it just looked like it completed, but there was no training on those models.

The other awkward bit was dealing with messy data through the lens of CreateML. I can’t really say what it did was “wrong”, but it could have been so much better. In one of my experiments, the data I had chosen for training had issues within it: missing labels where the training system expected to see data. That’s cool – but the error was reported by a small bit of red text at the bottom of a screen saying “Entry # 5071 has a problem”. Finding entry #5071 of a structured JSON data set is, bluntly, a complete pain in the ass. When I’d parsed and assembled the data per the online documentation for making a word tagging model, I’d dumped the data into a monster JSON, single-line data structure with no line breaks. That made finding a specific element using a text editor really rough. In the end, I re-did my JSON export to include pretty-printed JSON, and then also used VSCode’s “json outline” functionality (scaled up, since it defaults to 5000 items), to track down the specific item by position in a list. I found the offending data, and then looking around, noticed a bunch of other areas where the same “partially tagged data” existed. In the end I dealt with it by filtered it out if it wasn’t fully tagged up. It would have been much nicer, especially since CreateML clearly already had and could access the data, if it could have shown me the samples that were an issue – and notified me that it wasn’t just one line, but that a number were screwed up.

The evaluation details after you train a model aren’t readily exportable. As far as I can tell, if you want to share that detail with someone else, you’re either stuck making a screenshot or transcribing what’s in the windows. It seems you can’t export the evaluation data into CSV or or an HTML table format, or really even copy text by entry.

The details about the training process, its training and validation accuracy, number of iterations used, are likewise locked into non-copyable values. In fact, most of the text fields feel sort of “UIKit” rather than “AppKit” – in that you can’t select and copy the details, only get an image with a screenshot. This is a “not very Mac-like” experience in my opinion. Hopefully that will get a bit of product-feature-love to encourage sharing and collaboration of the details around CreateML models.

I filed a few feedback notices with Apple for the truly egregious flaws, but I also expect they’re known, given how easy they were to spot with basic usage. They didn’t stop the app from being effective or useful, just unfortunate and kind of awkward against normal “Mac” expectations.

I do wish the details of what CreateML was doing behind the scenes was more transparent – a lot of what I’ve read about in prior research is starting with descriptions of ML models in PyTorch, Keras, or Tensorflow. If I wanted to use those, I’d need to re-create the models myself with the relevant frameworks, and then use the CoreML tools library to convert the trained model into a CoreML model that I could use. By their very nature, the models and details are available if you take that path. It’s hard to know, by comparison, how that compares to the models created with CreateML.

Creating a Model with CreateML Versus a Machine Learning Framework

My early take-away (I’m very definitely still learning) is that CreateML seems to offer a quick path to making models that are very direct, and one-stage only. If you want to make models that flow data through multiple transforms, combine multiple models, or provide more directed feedback to the models, you’ll need to step into the world of PyTorch, Keras, and Tensorflow to build and train your models. Then in the end convert the trained models back to CoreML models for use within Apple platform applications.

Where the raw frameworks expose (require you to define) all the details, they also inflict the “joy” of hyper-parameter tuning to train them effectively. That same tuning/choosing process is (I think) happening auto-magically when you build models with CreateML. CreateML chooses the iterations, while paying attention to convergence, iterations, and epochs of applying data. It also appears to do a good job of segmenting data for training, evaluation, and testing – all of which you’d need to wrangle yourself (and ideally not screw up) while making a raw machine learning model. That’s a pretty darned big win, even if it does end up being more of a “trust me, I know what I’m doing” rather than a “see, here’s what I did for you” kind of interaction.

Final Thoughts on Learning CreateML

I’m still very much in “the early days” of learning how to build and apply machine learning models, but CreateML has already been an immense help in learning both what’s possible, and providing hints as to how I might structure my own thinking about using and applying models within apps.

The first is simply that they provide a lot of good, basic models that are directly available to use and experiment with – assuming you can source the data, and that’s a big assumption. Data management is not an easy undertaking: including correctly managing the diversity of the sourcing, data licensing, and understanding the bias’ inherent within the data. But assuming you get all that, you can get good – mostly interactive – feedback from CreateML. It gives you a strong hint to answer the question “Will this concept work, or not?” pretty quickly.

The second is that showing you the outputs from the model makes the inputs you provide, and what you expect to get out, more clear. I think of it as providing a clear API for the model. Models expose a “here’s what the model thinks it’s likely to be – and how likely” kind of answer rather than a black-and-white answer with complete assurance. Not that the assurance is warranted with any other system, just that exposing the confidence is an important and significant thing.

If I were tackling a significant machine learning project, I’d definitely expect to need to include some data management tooling as a part of that project, either browser/web based or app based. It’s clear that viewing, managing, and diagnosing the data used to train machine learning models is critical.

I’m also looking forward to see what Apple releases in the future, especially considering the more complex ways machine learning is being used within Apple’s existing products. I imagine it to include more tooling, advances to existing tooling, and maybe some interesting visualization assistance to understand model efficacy. I would love for something related to data management tooling – although I suspect it’s nearly impossible to provide since everyone’s data is rather specific and quite different.

Apple’s M1 Chip Changes… Lots of Things

The new Apple Macs with an M1 chip in them is finishing a job that started a few years ago: changing my assumption that commodity hardware would always win. Having worked in the technology/computing field for over 30 years, you’d think I know better by now not to make such a broad assumption, even internally. For years, I thought the juggernaut of Intel/x86 was unstoppable, but now? Now I’m questioning that.

Apple has always prided itself on its deep integration of hardware and software. And sometimes they’ve even made good on it. The iOS (and iPadOS) integration has been impressive for the last fifteen years, and now they brought it to their laptop lineup, spectacularly so. It’s not just Apple doing this – Samsung and some other manufacturers have been down this road for a while, merging in computing silicon with sensors at a deep foundational level, changing what we think of as components within computing systems. Some of the deeply integrated cameras are effectively stand-alone systems in their own right. Lots of phones and digital cameras use that very-not-commodity component.

I still think there’s a notable benefit to leveraging commodity hardware – shoot, that’s what this whole “cloud” computing market is all about. But it’s also pretty clear that the guarantee that the win that commodity gives you won’t necessarily outweighs the commercial benefits of deep hardware/software integration.

One of the interesting things about the M1 system-on-a-chip isn’t the chip itself, but the philosophy that Apple’s embracing in making the chip. That pattern of behavior and thought goes way beyond what you can do with commodity stuff. The vertical integration allows seriously advanced capabilities. Commodity, on the other hand, tends to be sort of “locked down” and very resistant to change, even improvements. Then pile on top of that the tendency for these chip designs to be far more modular. They’re aggressively using coprocessors and investing in the infrastructure of how to make them work together. I think that’s the core behind the unified memory architecture. What Apple has done, or started to do, is invest in the ways to go even more parallel and make it easier for those co-processors to work together. In a commodity (Intel) system, the rough equivalent is the PCIe bus and the motherboard socket you plug cards into. Only that doesn’t solve the “how you share memory” or “who talks to who, and when” problems. In fact, it kind of makes it worse as you get more cards, sockets, or components.

I may be giving Apple’s hardware engineering team more credit than they’re due – but I don’t think so. I think the reason they talked about “Unified Memory Architecture” so much with this chip is that it IS a solution to the “how to get lots of co-processors to work together”, while not exploding the amount of power that a system consumes while doing so. (If you’re not familiar, memory is a “sunuvabitch” when it comes to power consumption – one of the biggest sinks for power in a modern PC. The only thing that’s worse than memory are ethernet network ports, which was a serious eye-opener for me back in the day.)

There are other computing changes that aren’t trumpeted around so much that are going to contribute equally to the future of computing. I was introduced to the world of NVMe (Non-volatile memory) a few years back, when it was just hitting it’s first iteration of commercial introduction. The “holy crap” moment was realizing that the speed at which it operated was equivalent to those power hungry memory chips. Add that into the mix, and you’ve got lots of compute, persistent memory, and far lower power requirements in the future for a heck of lot of computing. It’s the opposite direction that Intel, and nVidia, are charging into – assuming they’ll have power to spare, and can happily burn the watts to provide the benefits. Truth be told, only individual consumers were still really oriented that way – the cloud providers, over a decade ago, had already clued in that the two most expensive things in running cloud services were 1) power and 2) people.

Bringing this back to the M1 chip, it’s tackling the power component of this all very impressively. I’m writing this on an M1 MacBook, and loving the speed and responsiveness. Honestly, I haven’t felt a “jump” in perceived speed and responsiveness to a computer with a generational gap like this in over 15 years. And that’s WITH the massive power reduction while providing it.

I do wish Apple was a little more open and better about providing tooling and controls to help solve the “cost of people” equation of this situation. Yes, I know there’s MDM and such, but comparatively it’s a pain in the butt to use, and that’s the problem. It needs to be simpler, more straightforward, and easier – but I suspect that Apple doesn’t really give much of a crap about that. Maybe I’m wrong here, but the tooling to make it really, really easy to combine sets of systems together hasn’t been their strong point. At the same time, groupings and clusters of compute is just the larger reflection of what’s happening at the chip level with the M1 SOCs (For what it’s worth: SOC stands for “system on a chip”).

I think you can see some hints they might be considering more support here – talking about their AWS business interactions a bit more, and on the software side the focus on “swift on the server” and the number of infrastructural open-source pieces they help lay out over the past year that are specific to capturing logging, metrics, tracing, or helping to dynamically create clusters of systems.

I think there’a s lot to look forward to, and I’m super interested to see how this current pattern plays over of this coming year (and the next few). Like many others, I’ve raised my expectations for what’s coming. And I think those expectations will be met with future technology building on top of these M1 chips.

Thanksgiving 2020

This has been a right 🤬 of a year, and that probably applies to most anyone else on this globe. In the US, the stress of this presidential election was extreme, and acerbated by COVID pandemic that we pretty much failed to get any sort of handle on. I’ve managed to stay tucked down and safe, but my day to day life from a year ago is completely different now. My favorite “office” coffeeshop, El Diablo, is now gone – 20 years after it started. The loss has an oversized impact on my because it was also my social hub connecting me to friends in my community. Like a lot of others, my world is currently collapsed down to a pretty tiny bubble.

The downsides of this year are undeniable and clear, but there have been a number of silver linings, and since we’re heading into Thanksgiving, it seemed a good time to reflect and celebrate the positives for the year. It doesn’t remove the horror and pain, but to me – it helps offset it.

Right as the pandemic was sweeping up, I managed to get a longer term contract gig that’s been really useful for me, working on technical writing. I’ve been programming, and managing programmers, devops, QA, and the whole software lifecycle kit for multiple decades, and one of the skills that I’ve been trying to cultivate has been communication – specifically writing. Last year around this time, I self-published what was primarily a labor of love – Using Combine – a book/reference doc combination on Apple’s Combine framework. The year before I’d published a different book through Packt, Kubernetes for Developers (A google search on that phrase now directs you to more of offline training and certifications, but there’s a book in there too!). I’d been searching for editors to work with, that really dug into the work to help me structure it – not just the fluffy top level stuff that so a number of publishers stick to.

It’s the combination of these things that ends up being what tops my give-thanks list this year. In the past eight months, I’ve had a chance to work closely with a number of truly amazing editors, helping to refine and improve my writing skills. While I’m still crappy at spotting passive voice in my own writing, the feedback I’ve gotten from Chuck, Joni, Susan, Colleen, Mary Kate, and Paul has been amazing. Everything from basic grammar (that I had just never really did well) to structure, narrative flow, complexity. Top that off with some wonderful writers, Ben, Liz, Dave, and Joanna, willing to answer the odd question or just chat about the latest garden recipes on a video call, and really these past months of contracting have been a terrific experience.

I was super fortunate to find a gig when everyone else seemed to losing them. I’m sure the troubles aren’t even close to over – there’s so much to rebuild – but it’s a bit of light against the backdrop of this year.

Combine and Swift Concurrency

Just before last weekend, the folks on the Swift Core Team provided what I took to be a truly wonderful gift: a roadmap and series of proposals that outline a future of embedding concurrency primitives deeper into the swift language itself. If you’re interested in the details of programming language concurrency, it’s a good read. The series of pitches and proposals:

If you have questions, scan through each of the pitches (which all link into the forums), and you’ll see a great deal of conversation there – some of which may have your answer (other parts of which, at least if you’re like me, may just leave you more confused), but most importantly the core team is clearly willing to answer questions and explore the options and choices.

When I first got involved with the swift programming language, I was already using some of these kinds of concurrency constructs in other languages – and it seemed to be a glaring lack of the language that they didn’t specify, instead relying on the Objective-C runtime and in particular the dispatch libraries in that runtime. The dispatch stuff, however, is darned solid – battle honed as it were. You can still abuse it into poor performance, but it worked solidly, so while it kind of rankled, it made sense with the thinking of “do the things you need to do now, and pick your fights carefully” in order to make progress. Since then time (and the language) has advanced significantly, refined out quite a bit, and I’m very pleased to see formal concurrency concepts getting added to the language.

A lot of folks using Combine have reached for it, looking for the closest thing they can find to Futures and the pattern of linking futures together, explicitly managing the flow of asynchronous updates. While it is something Combine does, Combine is quite a bit more – and a much higher level library, than the low level concurrency constructs that are being proposed.

For those of you unfamiliar, Combine provides a Future publisher, and Using Combine details how to use it a bit, and Donny Wals has a nice article detailing it that’s way more “tutorial like”.

What does this imply for Combine?

First up, pitches and proposals such as these are often made when there’s something at least partially working, that an individual or three have been experimenting with. But they aren’t fully baked, nor are they going to magically appear in the language tomorrow. Whatever comes of the proposals, you should expect it’ll be six months minimum before they appear in any seriously usable form, and quite possibly longer. This is tricky, detailed stuff – and the team is excellent with it, but there’s still a lot of moving parts to manage with this.

Second, where there’s some high level conceptual overlap, they’re very different things. Combine, being a higher level library and abstraction, I expect will take advantage of the the lower-level constructs with updates, very likely ones that we’ll never see exposed in their API, to make the operators more efficient. The capabilities that are being pitched for language-level actors (don’t confuse that with a higher level actor or distributed-actor library – such as Akka or Orleans) may offer some really interesting capabilities for Combine to deal with it’s queue/runloop hopping mechanisms more securely and clearly.

Finally, I think when this does come into full existence, I hope the existing promise libraries that are used in Swift (PromiseKit, Google’s promises, or Khanlou’s promise) start to leverage the async constructs into their API structure – giving a clear path to use for people wanting a single result processed through a series of asynchronous functions. You can use Combine for that, but it is really aimed at being a library that deals with a whole series or stream of values, rather than a single value, transformed over time.

tl;dr

The async and concurrency proposals are goodness, not replacement, for Combine – likely to provide new layers that Combine can integrate and build upon to make itself more efficient, and easier to use.

The range operator and SwiftUI’s layout engine

This post is specific to Swift the programming language and SwiftUI, Apple’s newest multi-platform UI framework. If you’re not interested in both, probably best to skip past this…

I was working on visualization code that leverages SwiftUI to see how that might work, and ran into a few interesting tidbits: playgrounds with SwiftUI works brilliantly, right up until it doesn’t and then you’re in a deep, dark hole of WTF?!, and the SwiftUI layout engine, especially with nested and semi-complex container views, do some sort of iterative solver technique. The less interesting tidbit is a classic aphorism: It’s the things you think you know that aren’t true that bite you.

When you’re fiddling with adding in your own constraints in SwiftUI, you might be tempted to use the range operator – at least my thinking was “Oh hey, this looks like a super convenient way to check to make sure this value is within an expected range”. It works stunningly well for me, as long as I’m careful about creating it. I started creating ranges on the fly from variable values, and that’s where playgrounds, and my bad assumptions, bit me.

If you create a range that’s ludicrous, Swift can throw an exception at runtime. So if you’re working with ranges, you’ve got the possibility that passing in a blatantly incorrect value will give you a non-sensical range, and that will result in a crashing exception. When you stumble into this using Playgrounds, you get a crash that doesn’t really tell you much of anything. When you kick that same thing up in Xcode, it still crashes (of course), but at least the debugger will drop into place and show you what you did that was wrong. I love using SwiftUI with Playgrounds, but the lack of runtime feedback when I hit an exception – about what I screwed up – makes it significantly less useful to me.

And debugging this in Xcode was where I learned that closures you provide within SwiftUI layout, such as alignmentGuide or a method of your own creation working with a GeometryReader don’t get called just once. Sometimes they’re called one, but other times they are called repeatedly, and with pretty strange values for the view’s dimension. I think underneath the covers, there’s an iterative layout solver that’s trying out a variety of layout options for the view that’s being created. Sometimes those closures would be invoked once, other times repeatedly – and in some cases repeatedly with the same values. Interestingly, sometimes those values included a ViewDimension or GeometryProxy with a size.width of 0. The bad assumption I made was that it would be sized quite a bit larger, never zero. Because of that, I attempted to build an incorrect range – effectively ClosedRange(x ... x-1) – which caused the exception.

Even with my own assumptions biting me, I like the use of range and I’m trying to use it in an experimental API surface. Lord knows what’ll come of the experiment, but the basics are bearing some fruit. I have a bit of code where I’ve been porting some of the concepts, such as scale and tick, from D3 to use within SwiftUI.

The current experimental code looks like:

// axis view w/ linear scale - simple/short
HStack {
    VerticalTickDisplayView(
        scale: LinearScale(
            domain: 0 ... 5.0,
            isClamped: false)
    )
    VerticalAxisView(
        scale: LinearScale(
            domain: 0 ... 5.0,
            isClamped: false)
    )
}
.frame(width: 60, height: 200, alignment: .center)
.padding()

// axis view w/ log scale variant - manual ticks
HStack {
    VerticalTickDisplayView(
        scale: LogScale(
            domain: 0.1 ... 100.0,
            isClamped: false),
        values: [0.1, 1.0, 10.0, 100.0]
    )
    VerticalAxisView(
        scale: LogScale(
            domain: 0.1 ... 100.0,
            isClamped: false)
    )
}
.frame(width: 60, height: 200, alignment: .center)
.padding()

And results in fairly nice horizontal and vertical tick axis that I can use around a chart area:

%d bloggers like this: