This isn’t something I’ve solved, more of something I’m working on, but I thought there were some interesting things to share with anyone else “walking this path”.
The Swift programming language is a static, strongly-defined language with a huge emphasis on leveraging types to help provide programmatic safety. It’s not always something I remember, as I spent the better part of two decades solving programming problems primarily with dynamic languages (C, Python, JavaScript, Objective-C, etc). The challenge that I’m faced with is interoperability with these dynamic languages. In this case, it’s through the lens of data – and more specifically cross-platform CRDT libraries.
Both Automerge and Y-CRDT started out with implementations in JavaScript, and have both recently rewritten their cores in the Rust language, with an eye towards using that high performance core as an underpinning for a variety of both platforms and languages. I’ve been contributing to both libraries and their Swift language bindings as these core progress. In the APIs that these libraries expose, types that represent lists and dictionaries don’t have the same constraints that Swift does – and mapping a potentially very dynamic data structure into Swift isn’t a straightforward task. At least it’s not straightforward to both provide a mapping that’s developer-driven and also results in a fairly ergonomic, “swift idiomatic” result.
I found two patterns, that are pretty consistent within the Swift ecosystem, for handling this sort of problem space. The two patterns end up layering together, depending on how far you need to go.
The first pattern is to provide a type that wraps that fully describes all the potential variations that could exist. It’s surprisingly powerful, and does a great job of mapping simple data types into the strongly-typed world of Swift. For a simple, one to one mapping, you can do this with a enum – and I’ve spotted exactly that used for encoding and decoding JSON in the swift-extras-json repository (JSONValue.swift). The other part that makes this convenient is a matching protocol that has functions for converting into, and out of, this type. Then for any type, you can pretty readily provide functions that do the conversion mapping, even potentially throwing an error if something doesn’t work out. The limitations of this technique is that it’s a one-to-one mapping, and it doesn’t really compose – at least not by itself. It is most effective for the simplest data structures or data that you deal with as an effectively atomic unit.
The downside of using a single wrapper type is that it doesn’t easily compose. As soon as you want to group multiple, different things together – it kind of falls down.
The second pattern is far more complex, but well established in Swift – the Codable protocol. This protocol setup is a pretty genius idea, but understanding how Codable works under the covers is not at all straightforward.
My brain immediately went to asking questions like “how can I self-inspect types to be able to encode or decode instances?” with the idea that “Surely, the swift language has already has this…”, mostly thinking of Mirror and type reflection. I use the codable protocol myself, but never knew how it worked. So I took to digging, thinking “there’s got to be something that provides the introspection details for codable to work”. I was right – there is – but it wasn’t at all where I thought it would be. Turns out the answer is “Either you provide it, or the Swift compiler – in some cases – can provide the default code”.
The reference that explained it for me is The Flight School Guide to Swift Codable. (Side note: ALL of the flight school guides are excellent resources – if you don’t have these in your library, take the time to get them and thumb through them all. I guarantee you’ll learn something.) The “magic” that looks at the type, knows about its stored properties, is the bit of Swift compiler goodness that synthesizes Codable conformance.
In hindsight, I thought “Huh. Maybe it should have been clear to me that you’d want to provide a way for a developer to provide their own control over encoding and decoding their types.” I have to admit I was disappointed the blunt-force hammer I was hoping to find didn’t exist. The effort of investigating how it worked got me to change up how I was thinking about the problem, which is definitely good.
After I understood a bit more about Codable works with encoders and decoders, I found a thread on the Swift Forums from 2019 asking about how the compiler provides default conformance. Interesting to me – in the thread Itai Ferber talks about how “ideally this kind of functionality could be exposed in the standard library using Macros” – but at the time macros didn’t exist within Swift. As I’m writing this, the review for SE-0397 Freestanding Declaration Macros is just starting, which I suspect has the capabilities that Itai referred to in that Forums post. The new macro capability is something I haven’t even started to grasp, but I’m looking forward to doing so. Since it isn’t available currently (with Swift 5.8), using macros to solve this challenge was outside of the pool of possible solutions to investigate.
I like that nail down a specific goal for myself when I’m trying to solve these kinds of issues – a specific problem that exemplifies what I’d like to be able to do. In this case, it is reading and writing from a CRDT backing store into “something that I define” that provides a schema, and which I can in turn use with SwiftUI. My specific use case is a data model that has a list of composed objects – each object having an image and text, with the text leveraging the collaborative capabilities so that multiple people could work together to provide descriptions for images that were added. This particular use case hits a several corners at once – it expects nested CRDTs (the text inside a list), and provides an interesting challenge of expose a list (or list-like thing) of a concrete type extracted from the model that houses the CRDTs that also allows for accessing the various text pieces through a SwiftUI TextField (which expects a Binding<String> to be able to update).
In the newer Automerge bindings, Alex Good cobbled some nifty property wrappers that work really nicely for exposing wrapper properties that map to simpler scalar values. I’ve been working on refining those a smidge, but also trying to come up with how to represent or reflect a array of something – maybe struct, maybe class – that is mapped from a List CRDT type and the data in it. I suspect I need to leverage a custom decoder to get there, reading from the CRDT data store and decoding into some form that can in turn provide those wrapper properties that enable read-only values as well as bindings that are ever-so-useful with SwiftUI. I spent a bunch of time painting myself into corners with various class structures that acted like collections, and while I got some basic things working, the end result was that creating them – or more specifically defining the structure of what you’d get back, felt super-awkward. If you want to follow my trials ( and mistakes) as I learn them, I’m working in a public repository (AMTravelNotes) on GitHub.
At this point, I think the path forward involves leveraging Codable; likely making a custom Encoder/Decoder that’s specific to reading and writing from Automerge documents. Alex built a very similar thing called AutoSurgeon that implements the same rough pattern in Rust using its rough equivalent of Codable called Serde. Fortunately, I’ve found a number of examples of custom encoder and decoder libraries. The Flight School Codable book includes an example that reads and writes to MsgPack. While digging around for other implementations, I found the library PotentCodables, which has a whole slew of interesting implementations, including CBOR and even ASN.1. Now that it’s been released, I think I’d like to take some time to read through and try to understand the code for JSON codable that’s included in the new Swift Foundation repository. It looks to be a notable step up on complexity, but with that complexity comes some impressive performance.
The current iteration of the Y-CRDT swift language bindings doesn’t yet support nested CRDTs, so I can’t quite yet attempt the same scenario with those bindings.
Rhonabwy 