Native support for USD tools on an M1 Mac

I saw USD release 22.08 drop a few weeks ago, and notably within its release notes is the sentence: “Added support for native builds on Apple Silicon.”

I struggled quite bit with getting USD both installed and operational because, as it turns out, there’s a bit of quirk to Python that made things more difficult. macOS has stopped including Python3 in its default install, although it appears that if you install Xcode (or developer tools on the CLI), you’ll get a version of Python 3 (3.89) installed on the OS. Since I wasn’t sure what the path for Python3 support was looking like, and it took a while get a M1 native build of python in the first place, I’d switched over to installing Python using the packaging tool `conda`, which has been fairly popular and prevalent for ML folks to use.

To get started, I’d installed miniforge, downloading and then running it to install and update:

sh ~/Downloads/
source ~/miniforge3/bin/activate
conda update conda -y

With conda installed (and activated), I started installing the various dependencies:

conda install pyopengl
pip install pyside6

And then grabbed the open source distribution of USD from Github and set it to building, with the results installed at /opt/local/USD:

git clone
cd USD
git checkout release
git pull
python build_scripts/ /opt/local/USD

It compiled and installed fine, but any tool I attempted to use crashed immediately with the following error message:

python crashed. FATAL ERROR: Failed axiom: ' Py_IsInitialized() '
in operator() at line 149 of /Users/heckj/src/USD/pxr/base/tf/pyTracing.cpp
writing crash report to [ Sparrow.local:/var/folders/8t/k6nw7pyx2qq77g8qq_g429080000gn/T//st_python.29962 ] ... done.

I opened an issue on USD’s GitHub project, and quickly got some super helpful support from their team. It turns out that an additional build configuration option is needed, specifically for python installed through Conda, because of the way it links to python. The same issue happens regardless of architecture – so it’s quirk for x86 as much as aarch64 architectures.

The key to getting it working is passing adding a build configuration detail:


I originally though that meant just exporting it as environment variables, but that doesn’t do the trick with USD’s build and installation process. Sunya Boonyatera provided the critical knowledge on how to add it in to get it to work:

python build_scripts/ /opt/local/USD --build-args USD,"-DPXR_PY_UNDEFINED_DYNAMIC_LOOKUP=ON"

Nick Porcino was kind enough to jump in as well and provide an explanation:

Conda distributed Pythons are statically linked as a way to reduce unexpected system and installation coupling, but Python isn’t, to my knowledge, built in such a way that an external build system can robustly discover whether you are targeting a static or dynamically linked Python so we are forced to deal with it manually.

Getting to this point wasn’t at all straightforward for me, so I’m hoping that by pushing this detail out there with my blog, Google will have it included in its index for the next individual, whether they’re searching on the error message or searching for how to install USD on an M1 Mac.

Side note – once its all installed, you still need to activate some paths on the CLI to get things available:

export PATH=/opt/local/USD/bin:$PATH
export PYTHONPATH=$PYTHONPATH:/opt/local/USD/lib/python

SwiftUI Field Notes: DocumentGroup

When you build a SwiftUI-lifecycle-based app for macOS or iOS, you’re expected to define a Scene – the first of which generally defines what (and how) that App displays when it is launched.

DocumentGroup is one of the types of Scene (for macOS and iOS platforms) that is focused on an App lifecycle built around viewing, creating, and editing documents. The common alternative, WindowGroup, doesn’t prevent you from reading, editing, or saving documents, but does change what happens when an App launches. The difference in “how it works” is when you launch an app using DocumentGroup, the app is expecting that the first thing you’ll want to do is either open a document or create a new one. It does this by kicking off an “open” dialog on macOS, or displaying a DocumentBrowser on iOS, configured based on the types your app says it knows how to create (and/or open). WindowGroup, by comparison, just tosses up whatever view structure you define when the app starts – not expecting you to want to start from the concept of “I want to open this document” or “I want to create an XXX document”.

You can define more than one DocumentGroup within the SceneBuilder for your app. You might do this, for example, to support opening and viewer or editing more than one file type. Only the first DocumentGroup that you define (and its associated file type) is linked to the “create new document” of the system provided Document Browser (or the File > New... menu within a macOS app). You’re responsible for exposing and/or providing an interface to create any new file types of additional file types.

Document Model Choices

When you define a DocumentGroup, provide it with a view that’s either the editor or viewer for a document type. That view is expected to conform to one of two protocols: FileDocument or ReferenceFileDocument. The gist of these protocols being that your editor (or viewer) view gets passed a configuration object that represents the document as a wrapper around your data model, and has with it a binding or reference to an ObservableObject, depending on which protocol you choose. Use FileDocument when you have a struct based data model, and ReferenceFileDocument when your data model is based on a class (and conforms to ObservableObject).

Apple provides a sample code entitled Building a Document-Based App with SwiftUI that uses the ReferenceFileDocument, and Paul Hudson has the article How to create a document-based app using FileDocument and DocumentGroup that shows using FileDocument.

The Default Call To Action

Be aware when designing and building an App using DocumentGroup that there’s no easy place to tie into the life cycle to provide a clear call to action before the system attempts to open a file as your app launches. The place where your code (and any control you have) kicks off is after the app has presented a file open/new view construct – either a DocumentBrowser (iOS) or the system Open dialog box (macOS). I find that experience can be confusing for anyone new to using the app, as there’s no easy way to introduce what the app is, what’s new, or provide instruction on how to use the app until after the person has chosen a document to open or created a new document.

If you need (or want) to provide a call to action, or to get some critical information established that you need for creating a new document, you need to capture that detail after the new document is passed to you, but potentially before you display any representation of the document. That in turn implies that a “new” document model that requires explicit parameters won’t work for conforming to FileDocument or ReferenceDocument. You are responsible for making sure any critical details get established (and presumably filled in) after the new document is created.

For example, if a new document requires some value (or values), add a function or computed property that returns a Boolean that indicates if the values exist or not. Then in your editor view, conditionally return a view that 1) captures and stores those initial values into the new document when they don’t yet exist or 2) displays the document if those values are already defined.

NavigationView (or not)

An interesting side effect of DocumentGroup is that the views you provide to it get very different environments based on platform. iOS (and Mac catalyst based apps) display the view you provide enclosed within a NavigationView that the document browser provides. With macOS, there is no default NavigationView. If you want iOS and macOS to behave with a similar design (for example, a two-column style navigational layout), you need to provide your own NavigationView for the macOS version.

I find NavigationView to be a quirky, somewhat confusing setup (I’m glad its deprecated with iOS 16 and macOS 13 in favor of the lovely newer navigation types). When you’re working with iOS, it may not be obvious that you can create a tuple-view to provide the two separate views that you need to enclose within NavigationView to get that two-column view layout. You can do this by providing a view that returns two separate views without an enclosing grouping or container view. For example, the following view declaration provides a two-tuple view (with SidebarView and ContentView) that the NavigationView provided by the built-in document browser will use when displaying your document.

var body: some View {

If you provide your own NavigationView inside the view that the DocumentBrowser vends, you get a rather unfortunate looking “double navigation” view setup. Instead, provide a tuple of views based on the navigation style you’d like your editor (or viewer) to display.

If you’re used to NavigationView, but ready to target these later platform versions, definitely take the time to read Migrating to New Navigation Types and watch related WWDC22 talks.

I wish contextual SwiftUI was more predictable

I’m not certain how to phrase this. It’s either that I wish I was better at predicting what a SwiftUI view would look like, or that I wish SwiftUI was more predictable at how views render in different contexts. I recently built a multi-platform SwiftUI utility app, and was struck by how often I ran into unexpected results.

Contextual rendering of a view is a feature, perhaps a key feature, of the SwiftUI framework. The most obvious example is how views render differently by platform. A default button renders differently on macOS as compared to iOS. But this “different contextual representation” goes much deeper. Some views render differently based on if they are enclosed by another view, such as Form or NavigationView.

If you take the example in Apple’s reference documentation for Form, and replace the Form with a VStack, the result is notably different. For the following two screenshots illustrating the differences, I stripped out the enclosing Section views. The view on the left shows the view enclosed within Form, the view on the right shows the content enclosed within a VStack:

Example code at

I feel obliged to point out that these effects, at least in the case of Form, are documented within the reference documentation for Form. That said, I’m fairly certain that not all view variations are explicitly documented.

In general, the combined effects of views get complicated — and hard to predict — as you compose them, primarily within container views. I do wonder if the struggle to predict how a view will render (and that they render contextually) is responsible for some of the feelings I’ve heard expressed by developers as “SwiftUI isn’t ready for production” or “It still feels beta to me”.

The productivity enhancement of being able to preview a view “live within Xcode” has gone a long way to mitigating this complexity for me, especially when constructing views that I want to operate across multiple platforms (macOS, iOS, iPadOS). Using Xcode 14 beta, the automatic live rendering is notably reducing the time for each iteration loop (barring a few odd bugs here and there – live rendering for macOS seems to come with notably more crashes while attempting to generate a the preview).

Still, I wish it were easier to predict – or barring that: look up – how a view renders in different contexts. The web/javascript development world has Storybook (an amazing resource, by the way), and the closest SwiftUI equivalent to that is the A Companion for SwiftUI app, which provides an interactive visual catalog. If you’re developing with SwiftUI, it’s a worthy purchase and investment. Be aware that even this app doesn’t always help with understanding the contextual variations for a platform or when composed.

The best advice I have to counter this difficulty in prediction is to lean into the previewing – be it for macOS, iOS, tvOS, or watchOS. Use Xcode’s previewing capability, and get to your combined views as quickly as possible. While it can be surprising that an enclosing view can have such an impact on how something renders, seeing the combination so easily while developing at least lets you verify and react when things don’t go as you expect.

What Apple might do with distributed computing

I’m excited to see not only async-await making it thoroughly into the language (in Swift 5.6), but also the extensions that enable actors and distributed actors with this general sweep of the Swift language embracing concurrency. It’s been several years in the making, and the past year has been building much of these base pieces through the open-source side of Swift, which has been fantastic to watch. The next version of Swift (version 5.7 I presume) is likely coming next week, but even that doesn’t yet contain all of the outlined goals for concurrency within it. There’s clearly more to come – so where will it go from here? And how will it get used?

I have some background with distributed computing, back from the early days when OpenStack was starting to become a thing, several years before Kubernetes rose up to provide (in my opinion) even better abstractions, even if it stopped short in just the right way to bring with it a hell-scape of complexity. I’ve been programming using the Swift language recently, primarily to be delivered on Apple platforms, but I keep my fingers in the linux distribution, and a close eye on “server-side swift” in general. I see a huge amount of opportunity in distributed computing and the capabilities that it could provide.

With more hope than any real knowledge, I’d love to see a replacement API structure that’s swift focused and distributed across devices as a successor for XPC. Expanding the cross-process communication capabilities to cross-device, and the supporting infrastructure from Apple operating systems to identify and broadcast to each other, could enable some interesting scenarios. XPC has long been critical in terms of sandboxing processes, and generally enabling cross-process communications. It’s used all over the place in macOS and iOS, in the background where we don’t see it. Enable that across devices… and you’ve made the cluster of Apple devices that I have in the house more powerful because they could leverage each other and what they can individually do.

To me, the obvious starting point is Siri interactions. I have a HomePod mini in my kitchen, and usually more than one other device nearby. When I say “Hey Siri”, everything lights up and tries to respond, and one of them usually runs with the request – but sometimes several make the same attempt. I’ll ask Siri on my phone to activate a shortcut, but the HomePod grabs it – and because it doesn’t have the installed apps or intents, it says it can’t fulfill my request. Wouldn’t it be nice if Siri could know all the intents of all the active devices (where you have permissions, naturally) in the area where the request was made? That kind of thing is only possible when you have a robust and dynamic clustering solution that can let devices come and go, and share information about what’s nearby and what their capabilities are. In short: distributed computing.

Another place that would be useful is data collection and presentation with HomeKit. It feels dumb to me that each device and some iOS app has to manage the history of data for the in-house sensors I have installed. It’s making a silo of every sensor, and restricting my ability to even see, let alone use, the information together. The most effective solution to this today is cloud service integration, but the “advertising freemium” model that has dominance is primarily focused on extracting value from me, not providing it – so no thank you! Instead, I’d love to have a device (such as a HomePod with a bit of storage) stashed in the house that just collected the data from those sensors and made it available. A little, local persistent cache that all the HomeKit things could send to, and iOS apps could read from. A collection center akin to what Apple developed for health, and that doesn’t need to be stashed in a secure enclave. Imagine that as a HomeKit update!

Those two ideas are just based on what’s available today with a bit of evolution. My wildest dreams go full 2001: A Space Odyssey and imagine a rack where you can plug in (or remove) additional modules to add compute, storage, and memory capability. Make that available to other devices with something distributed, authenticated, and XPC like. It could expand the capability of small devices by offering progressively larger amounts of offloaded compute and storage available to devices that are getting lighter, smaller, and more invisible. In short, it’s a place where you could do the offload/tether that used to exist for iPhones to Macs, or watches to iPhones. If the current equivalent of those little white boxes in 2001 had the internals of the low-end current iPhones, it’s a truly amazing amount of compute that you could made available.

Yeah, I know that last one is getting pretty out there, but I love the idea of it anyway. I know it’s possible – maybe not practical – but definitely possible. I helped create something akin to that idea, based on the lessons from NASA’s Nebula Compute (but with a LOT less integration of hardware and software about a decade ago – we were trying to ride it all over commodity gear). That company failed, but the similar vision was replicated later by CoreOS, and a variation included within Canonical’s Ubuntu. I haven’t seen it really come to fruition, but maybe…

In any case, I’m looking forward to WWDC starting next week, the announcements, and mostly seeing the work a number of friends have been quietly working on for a while.

What being an Open Source Developer means to me

I’ve periodically described myself as an “open source developer” in those pithy biography blocks that you sometimes get (or have) to create. Developing and providing open source software means something specific to me, and I’d like to share how I think about it.

At it’s core, it’s only a little about solving problems with code – it’s as much about the community for me. When I provide open source software, it means I am both willing and want to contribute to knowledge that others can use, with few constraints. I’m not seeking payment, nor am I seeking to become a servant. I build and share open source software because I enjoy having, being apart of, and building a community that helps each other. The software solution itself is useful, but almost secondary. I enjoy, and search for, the benefits of having a community with diverse skills. People that I can go to with questions to get answers, or at least opinions on directions to search, and warnings about steep cliffs or ravines that may lie along the path – at least if they know of them.

I’ll happily host software on Github or Gitlab, take the time to make it into a package that others can use, and submit it to a relevant package index. I’m often do that as a side effect of solving a problem for myself. Sometimes I like to share how I tackled it for anyone else coming along with questions. I get a huge amount of value from reading other people’s work solving problems, so I try to share my thinking as well. The solutions may not be optimal, or even correct in all cases, but it’s shared – and that’s the main thing. Because it isn’t easy to find solutions just looking at source (or reading Github discussions), I also write about some of my searching for solutions in this blog. It helps to get at least some of the details in the global search indices. A blog isn’t a great place to share code, but it’s a decent place to talk about a problem or solution – and point to the code.

I’ve run into a number of folks who don’t share my opinion of what’s suitable to open source. I’ve heard them reference their reasons for not publishing as open source, as if they’re feeling guilty for not doing so. I’m not sure that’s what they’re feeling, but I kind of read that into some of what’s said. I’d really like to tell them there no guilt or shame in not contributing, or in keeping your learning and explorations to yourself.

I think some of that guilt thing has evolved from technical recruiters using (and sometimes expecting) a person’s public GitHub repository to be a reflection of their capabilities or a kind of portfolio or work. I’ve looked myself when I’ve hired people, if there’s something there – in the past, I thought it could show interests that can be useful to know about or just a starting point for a conversation. The downside of this practice is that people end up thinking that whatever they might “open source” has to speak with perfection about their cleverness, clarity, or robustness of a solution. I get it, that concern for what others might think based on what I’ve done. While I personally value the sharing, I respect the desire to not want to be put in that position.

There’s also plenty of people who legally can’t share. I think it sucks that corporate policies with these legal constrains exist, and from a corporate policy perspective I think it’s brutally short sighted. That said, I’m also someone who believes that ideas are nearly free – that it is execution that matters in building and running a company. Still it’s common, especially for technology companies, to legally constrain their employees from participating in open source without their explicit, prior approval – or in some cases to deny it all together, shitty as that can be.

I like to learn and I like to teach – or more specifically, I like to share what I’ve learned and learn from others in the process. (I’ve never learned so much as when I taught.) I get a lot of value out of being a part of a community – more so one that shares, sympathizes during the rough spots, and celebrates the successes. I’ve been repeatedly fortunate to find and be involved in great communities, and I’ve learned to seek them out – or just help foster and build them. I’ve also run from “so-called” communities that had individuals that delighted in mocking others, or attacked an opinion because it was different from theirs (I’m looking at you, orange hell-site).

While I love sharing, I’ve learned (the hard way) to set boundaries. Just because I like and want to share doesn’t mean I’m beholden to someone else’s opinion of what I should do, how I solve a problem, or that I have any obligation to them beyond what I choose to give. I’ve had people demand support from me, or from a team I was a part of collaboratively developing some open source software. I’ve even had a few experiences where I was targeted in personal, verbal attacks. It’s unfortunate that this comes with the territory of sharing with the world, but it’s also been (fortunately) rare. Most of the time the people being abusive, or selfish jerks, aren’t part of the community group that that I value. In my experience, they often stumbled onto code I helped create while looking for a solution to some problem they have, and thought because I shared, they were owed something. Sometimes they’re just people having a bad day and relieving their frustrations in the wrong place. There was a time that I’d offer to help those folks who’d been abbrasive – but only for money as a contract or consulting fees. These days I often shunt such ravings into a trash can, shut the lid, and don’t look back. It’s not worth the time, or the money, to deal with assholes.

The most valuable thing is the people you can connect with. These may be folks you’ve known for years, or they might be people you’ve never met or heard from. One of the best parts of “being an open source developer” is putting solutions out there that someone can pick up and use, and along the side infer “hey – there’s a place for you here. Drop in, have a seat, and chat with us.” They might stop in for a few minutes, a couple of days, or maybe longer. I expect everyone moves on at some point, and my hope is that they take away a spark of being a part of positive community. A seed of how it could work that they can grow.

Back at the beginning of this, I said open source meant something specific to me – sharing knowledge with “few constraints”. That’s not the same as no constraints. The formal mechanism in software around these constraints is licensing. I watched the legal battles from the 80’s and 90s, felt (and dealt with) their fallout, and avidly followed the evolution of a formal definition of open source and the blooming of various licenses that provided a legal backing. I get the licensing and I take it seriously. While I understand the foundational desires behind the free-software movement and its various licenses – such as GPL and LGPL – that’s not for me; I’m not a fan. I’m not going to, and don’t want to, take that hard a line and impose those additional legal constraints for the product that represents the knowledge I’m sharing. About the only constraint I want to put on something is attribution. The gist of which is “don’t be a jerk and take credit for my work”, but really it boils down “Recognize that you, like I, stand on the shoulders of our peers and those that came before us.”

Language and framing

Quite a few years ago, I was fortunate to sit in a presentation where Yukihiro Matsumoto talked about his inspirations when making the Ruby programming language. One of the references he cited was the 1966 science fiction novel from Samuel Delaney Babel-17. I was in Portland at the time, and was able to get a copy of the novel from Powell’s books, and spent an evening reading it. The book explores how language structures our thinking, which I thought was a super interesting insight when it comes to making your own programming language. I think about that insight frequently as I watch (and sometimes interact) with the evolution of the Swift programming language.

One of the topics I struggle with is how to frame a programming task to usefully solve it – be it with protocols and generic programming, or simpler techniques. It’s a shift in thinking for me, as I’m bringing a background built up using a lot of dynamic languages, and swift is NOT that. Sometimes, it feels like you start to think that way – what’s possible with some of the limited protocol types (also known as “existential types”), but as I’m learning (and have learned the hard way), it is definitely not the same. The strongly and strictly typed features of the language can make it a real challenge for me to navigate. In hindsight, I’ve found myself reaching for what are fundamentally features of dynamic languages to solve problems, only to run afoul. I’ve found it tricky to know when to use which features of the language to solve the problem at hand.

I’m currently working on taking another stab at trying to use the advanced swift language feature of result builders to build some declarative content that layers into SwiftUI. In terms of Swift, it builds in the deep end of the advanced features of Swift – generics, protocols with associated types, and (I think) the need for at least some type erasure. It’s proving to be a very back and forth learning process, and it feels like maybe the light-bulb is starting to come on about design techniques that leverage these pieces, but I’m still working through all the moving parts.

If you find yourself wanting to learn about result builders, I highly recommend watching, taking some time away, and then going back and re-watching Becca’s WWDC’21 presentation Write a DSL in Swift using result builders.

One kind of documentation that Apple used to do — that I dearly miss and is hard to write well — is a programming guide. I think it’s a huge loss in the broader Apple ecosystem. Blogs and some books get close to it. In years past, the Big Nerd Ranch guides were one of my go to books – and more recently that’s expanded to include the longer-form work from Paul Hudson, also an excellent teacher, and I build on the sweep of individuals that write about Swift, how to use features of it, or APIs within it. When I started learning to program on the Mac, the Cocoa Programming Guide (now retired, but still 100% relevant) introduced me to a lot of the core concepts that the Apple frameworks used – structured delegates, the MVC design pattern, target-action linkages, and the responder chain. Most importantly, these guides showed how to “work with the framework” and how to consider framing a programming task or problem so that the framework could more easily help you achieve you goals. There is still information in those guides that I’ve not found elsewhere – up to and including the reference documentation.

While I’d love to have such a guide for more in-depth examples and framing for how to build applications that use SwiftUI, there are other topics that might be even more valuable. Anything to do with Metal, and quite a bit about RealityKit, could use the help. The topics are just deeply complex, and the piece-meal, short-form content that is more common and popular these days makes it easy to learn a small piece, but (for me, at least) much harder to tie the whole process together. The best resource I’ve found for Metal is Metal Programming Guide by Janie Clayton – and no, that’s not written by Apple. The RealityKit framework makes heavy use of an architectural pattern that’s wholly different from those used in AppKit, UIKit, or SwiftUI – the Entity Component System (ECS). It is familiar to folks who have done game programming, but it’s a whole different way of looking at things for folks used to MVC and Cocoa, or the functional styles commonly seen in javascript programming these days.

The lack of programming guides for the expanding surface area of Apple platforms feels like a huge gap today – and not one I think that will soon close. Having written technical books myself, the longer works — such as these guides — are very hard to justify in terms of the time to write it. The lifespan of a book about technology, and how to use it, is brutally short. Maybe there’s something shorter form than a comprehensive bible style guide that would be useful. I’m still searching to figure out what that might be. Fortunately that’s something great about the broader swift community, there’s a lot to learn from others who are taking the time to share.


I’m starting, or more specifically re-starting, a project that I envisioned a couple years ago. In some of the apps I’ve created, I’ve found it useful – sometimes critical to what I want – to provide small charts (visualizations) of data from the app. One example of this is a series of histograms that show the network latencies to each step in a route from my machine to a location on the internet. There’s some great CLI tools that display the raw data in a terminal window (mtr, if you’re curious), but I think it’s easier to understand viewing the data as a series histograms. In the case of mtr, you can see where network traffic slows down or doesn’t get past.

work in progress – histogram sequence from traceroute data

Looking at the example above, you can see a fairly consistent latency through to the end point, with the majority of the variability at that first step (in this example, it is my wifi router hub). The down side is that the code to display just that – not even getting to labels, axis, etc – was pretty heavy.

A year ago, I tried coming up with drawing this kind of graph in SwiftUI, including displaying axis and labels. Turns out that with SwiftUI’s constraints, and the “every expanding” side effect of using GeometryReader in a SwiftUI layout, it was darned tricky to get it working. I got something working, on which I could overlay SwiftUI shape objects with explicit offsets. The downside was that it wasn’t flexible and small changes could break the layout dramatically. Last year, Apple updated SwiftUI with Canvas, which I used when generating the series of histograms above. That really opened the doors for me. Now I’d like to do the same using a SwiftUI like syntax. It seems like it would be a natural fit, and it would make creating those little sequences of visualizations a hell of a lot easier.

I know there’s other charting libraries out there – and good ones. The “ole standby” that many people use is Daniel Gindi’s Charts, but even a quick search on Swift Package Index shows other folks offering their own takes on some of these libraries, several of which are interesting for the number of stars they’ve collected. (Majid’s SwiftUICharts in particular looks interesting to me, fits in well with SwiftUI, and appears to be stable).

I’ve cobbled previous charts and plots with anything from Excel to using the D3.js library in javascript (which is a really amazing library). That background had me following a trend towards describing charts more declaratively, which fits in well with how SwiftUI works. I dug into Observable’s Plot and the Vega-lite project, and the research papers behind them. Both of have a very declarative style of describing a chart, the style of which fits in wonderfully well with the kind of declarative syntax that SwiftUI uses. Seeing what I can do to enable that kind of functionality, but using Swift, is what I’m after, and what I’d really like to be able to use.

Part of my goal in setting this up is to learn the “ins and outs” of what it takes to enable this kind of thing using result builders. I don’t feel like have a solid understanding right now, and the way I learn best is by doing – so I’m aiming to learn and do in public. I’d like to work out how to tell someone else how they could solve this kind of problem, and right now I’m far from it. I’m guessing the solution likely includes not only result builders, but careful use and design with generics and protocols as well. I’m planning on writing about my development process (and my learning!) as I tackle this. Rather than waiting for when it’s all “good and done” (assuming I get there), I have set this up to develop completely in the open. I do expect it’ll get messy, and it may never come to anything, but I figured it was worth a shot.

If you think tackling something like this might be fun, or just want to poke your head in to watch how I tackle it, you’re welcome to join in. I would love to have others to bounce ideas off of, or help with the coding if you’re so inclined. If you’re interested in following along, I’m starting with some of Github’s community tools on a repository. I set up an organization called SwiftViz, and made the repository Chart to house this project. I’ve enabled Chart Discussions – no idea how well or crappy that will work out – as a starting point. I do expect to evolve as it goes. I thought about just writing about the efforts on this blog, but a blog doesn’t really end up being very interactive – can’t have much of a discussion on it – and I’d like to at least entertain the possibility that someone else might want to collaborate with me on this.

It’s early days, and it will undoubtably get messy, but if you’re interested, drop into the repository discussions and say hello, or reach out to me on twitter if that’s easier.

RealityKit on macOS

Guessing which frameworks are going to be updated, and which aren’t, is — I think — a part of developing software on Apple platforms. Sometimes it’s clear based on what’s been updated over the past three or fours, as is the case with RealityKit. I started my experiments with SceneKit, another lovely high-level API to 3D graphics, but wanted to utilize more of RealityKit for some of my procedural graphics work (Lindenmayer trees).

RealityKit has a huge API surface, only some of it directly related to 3D rendering. Other parts include physics simulations, including rigid body collisions, an ECS, and — as of WWDC’21 — methods for procedural geometry/mesh creation. RealityKit is fundamentally an API that’s meant to mix of 3D rendered content into live images from the real world. Because of that, the camera for rendering 3D content is tightly controlled to match the physical position and location of the actual camera on a device. So it isn’t surprising that you might think, “Hmmm.. how’s that going to work on macOS – which only has a forward-facing camera, and few to none of the fancy motion and depth sensors an iOS device has.”

The good news is that RealityKit, with macOS 10.15, added the capability to run on macOS. It includes an ARView for macOS (even though there’s no ARKit) inside RealityKit. It’s more constrained than the UIKit version, but it’s there. After a comment from James Thomson on a podcast last year talking about getting that view working on macOS, I wanted to do the same. It’s relatively easy to get an ARView showing up – but figuring out what do to with it from there is not as obvious. It turns out that the macOS-only ARView doesn’t respond to mouse, touch, or trackpad input – so user interactions don’t influence the camera position. You can’t appear to move around or look at different things without code to explicitly move the camera. If you use the UIKit ARView on macOS built with mac Catalyst, it’s a different story. This bit is about mixing AppKit and RealityKit together without using MacCatalyst and the UIKit Apis.

Once I’d convinced myself this wasn’t entirely a fools errand, I wrangled up the code to enable user interaction to move the camera. I made my own subclass of ARView and added in keyboard, trackpad, and mouse support to move the camera. It isn’t identical to how SceneKit works, but it’s sufficient to display content and look around the environment. I wanted something where I could load up and view content using RealityKit, but running on macOS. In particular, I wanted to load up a USD file and get sufficient screen captures to render the set of images into an animated gif. As a side note, there’s not an easy and consistently clean way to describe a 3D object in documentation – but a lot of the HTML based mechanisms (include DocC, which I’ve been writing about lately) have no trouble displaying an animated gif.

The end result took a bit, but I got there (the fish in question is from WWDC’21 sample code):

While trying to get this working, I learned that the ARView implementation for macOS that comes with RealityKit appears to swallow up mouse events, which meant there wasn’t an easy way to control it from SwiftUI views using gesture recognizers. I switched to the older mouse and keyboard interaction apis that AppKit supports, making a subclass to add in the various overrides to provide mouse and keyboard event handling. After I got it all working, I extracted the SwiftUI wrapper. Finally, I put the subclass into a it’s own swift package, available for anyone to use:

CameraControlARView (API Docs).

If you’re interested in using RealityKit on macOS, you’re welcome to use it – or to pull it apart to see how it works and use that to do something you prefer. My goal in making this tiny open-source package is to help the next soul who’s stumbling through figuring out how to get started with RealityKit and wanting to experiment a bit on macOS to see how things work.

Design decisions and technical choices

I made a (perhaps) odd choice when I assembled the SwiftUI container for it – I set it up so that you create the instance of my subclass either in a SwiftUI view, or hold it outside that and pass it through. The point was to have a reference to the ARView (and its properties) available from the declarative space of SwiftUI so that you could interact with the 3D scene and entities within it from the SwiftUI control declarations. I didn’t see another straight-forward path to creating the scene (such as a SceneKit Scene) externally and passing it down and in, and in order to do on-the-fly file loading (such as when you drop a USDZ file into the view) I needed to have access to that scene.

The other design choice I made was to make the camera motion support two different modes of motion. The first mode is what’s called “arc-ball” – it orbits and looking at a specific point in 3D space from a variety of rotations. You can imagine it as following arcs around a sphere, centered on a specific point. I also wanted to have a mode where I could move freely move about, which I called “first person” mode. In first person mode, you can move the camera forward, back, side to side, as well as turn to look elsewhere. There are a couple other popular 3D movement modes out there (turn-table being one I considered, but decided to not attempt), but I stuck to these two modes.

The rest is all pretty straight-forward. I’d fortunately become more comfortable with the gems in the Accelerate/SIMD framework, which provide lovely quick tools for creating quaternions and matrices. I still ended up embedding a few “build this rotation matrix from this angle around that axis” kinds of methods – stuff that’s under the covers in SceneKit, but I’m not aware of a “central” implementation in the platform tooling otherwise. It’s not “hard” per se, just tricky to make sure you get correct.

Generating Animated GIF from 3D models

I went ahead and put the macOS App project up publicly on Github as well: Film3D. It’s far from a complete app, it doesn’t even have a placeholder icon (as I’m writing this) and the user interface is a ludicrous “just get it working” sort of thing, but it works to load a USDZ file, set the camera, and capture images into an animated gif.

There’s more I’d like to do with it, but I’m not planning on selling it – I just wanted something that does what it does so that I can use it to create images for documentation.

DocC plugin PSA

In my last two posts about using DocC, I’ve been implicitly encouraging the use of the new docc-plugin for the Swift package manager. I did so knowing that it is in beta and the underlying APIs are evolving – and in my example scripts I include how to do the same thing with commands that directly invoke the swift compiler and DocC to generate documentation.

With the release of Xcode 13.3 beta 3 today, the API for package manager command plugins (of which the swift-docc-plugin is a lovely example) has changed slightly – and as such, what’s currently on the main branch isn’t compatible. There’s a pending pull request which I believe will resolve and update things back to working, but if you’re using the plugin, you should be aware that there’s an issue.


The pull request was merged on March 1st, which resolves using the plugin with Xcode 13.3 beta 3, and Ethan posted a bit of nice detail on the Swift Forums showing what’s changed when you use the plugin after the Swift Package Manager API updates. The big piece is that the --target option moved from in front of generate-documentation to after it, and there’s additional options for specifying a specific product, useful if you have a swift package that generates multiple products.

In the meantime, using docc on the command line directly does the job well, and hopefully soon we will have a fix added to the plugin for the updated API. I also hope that a release will be available before too long, but that will only happen when all the moving parts have stabilized. It is a lovely tool, and we’re in a beta period – so I’m not surprised that there are a few bumps along the road. I’m sure the end result will be better for the iteration.

In case you need the earlier steps laid out, here’s a script for generating documentation into a directory for static hosting:


set -e  # exit on a non-zero return code from a command
set -x  # print a trace of commands as they execute

rm -rf .build
mkdir -p .build/symbol-graphs

$(xcrun --find swift) build --target RSTree \
    -Xswiftc -emit-symbol-graph \
    -Xswiftc -emit-symbol-graph-dir -Xswiftc .build/symbol-graphs

# Enables deterministic output
# - useful when you're committing the results to host on github pages

$(xcrun --find docc) convert Sources/RSTree/Documentation.docc \
    --output-path ./docs \
    --fallback-display-name RSTree \
    --fallback-bundle-identifier com.github.heckj.RSTree \
    --fallback-bundle-version 0.1.0 \
    --additional-symbol-graph-dir .build/symbol-graphs \
    --emit-digest \
    --transform-for-static-hosting \
    --hosting-base-path 'RSTree'

Tips for getting the most out of DocC

1 – Start by adding doc comments to your types.

The starting point is adding a single short summary sentence as comment (using the ///) for each public type in your library or app. Feel free to add more: if you add a “blank line” (meaning include another line with /// but nothing else in it) followed by additional content, that content appears as the “Discussion” or “Overview” for the type. This is a great place to drop in a code snippet showing how to use that type alongside the discussion of what it is and how to use it.

The reason I picked this as the starting point is that everything you add immediately becomes available through Xcode’s quick-help. You get maximum immediate benefit, as it provides content to the panel that opens when you option-click on a type.

The initial single line doc comment is referred to as the abstract. When a symbol (whether it’s a class, struct, protocol, property, or method) doesn’t have an abstract, DocC generates one for you. And that is where the much-commented-on No Overview Available comes from.

2 – Add doc comments to your public methods and properties.

The important bit here isn’t a default initializers or functions that enable conformance to common standard library protocols. This is about the the elements on your type (struct, class, enum, or whatever) that make it unique and special. Aim to get to the core of what the symbol does – and most importantly, why you’d use it and what you need to use it. Sometimes its simple – and a single line abstract describing what it represents is more than sufficient. For those times, a single line summary (the first line with ///) does wonders. If it’s more complex, add that after a break (a line with /// and nothing else as a doc comment), then talk to the details on how to use it, or what to expect.

If you’re writing the doc comments using Xcode, it provides a handy “generate a doc comment” command (command-option-/ being the default key mapping to that Xcode gem) that creates a stub. If you do this for a function or method, it includes the parameters formatted correctly for DocC and placeholders for you to fill in.

If you stop right after doing this, you have the basic, effective documentation that is available to anyone loading your framework or package. But there is still a lot you can do to make it easier to use your library.

3 – Add a documentation catalog and a library overview.

Adding a documentation catalog is the 3rd thing on this list – that’s not a mistake. Everything above provides immediate benefit while you (and any collaborators) work on, or with, the source code. I think of the Documentation Catalog as being akin to the idea that you should include a README file with a project. It provides an introduction and framing to your library, and is the heart of where to learn more.

The documentation catalog is a directory with a collection of goodies. It houses the top-level overview of your library, and any assets (images, articles, and more — should you go that far). When you add a catalog and then use Product > Build Documentation within Xcode, those docs appear in the Xcode documentation window. You can export an archive for others to use from that window. There is also terminal commands to generate documentation that’s suitable for static hosting. I created an earlier post (Hosting Your Swift Library Docs on Github Pages) that goes into detail on the steps of doing just that.

When you create a documentation catalog using Xcode, its template includes a markdown file with a loose structure for what to add. It is intentionally, and specifically, structured for DocC. I recommend writing the content under the Overview heading first, and then going back and writing the summary later. The bottom section of the template (this text that starts with ## Topics and has a placeholder 3rd level heading with a generic symbol underneath) is the placeholder for you to organize the symbols of your project – the next tip.

4 – Add the top-level curation for your library.

In the world of DocC, curation is a term that means to create organization for a set of symbols. When I suggest adding top-level curation, I’m referring to adding organization to that top-page in your library, that includes the overview. This is the text beneath the overview that starts with ## Topics, followed by one or more 3rd level headers (###) with various grouping names.

DocC started with collating only public types, methods, protocols, and so on for libraries and frameworks. As it’s evolving, the project is starting to add in the capability for documenting apps as well. In the latest Xcode 13.3 beta, when you add a catalog and build docs for an app, it includes internal types as well as public.

How you organize the symbols is subtle, but important. We have a tendency to scan down from the top – so organize the most important, or frequently used, elements near the top. Arrange symbols that have less importance beneath. DocC always builds a structure for you, even if you don’t provide one. When you provide some structure, it tries to take that into account first, and fills in any missing gaps. Docc places the symbols you don’t include in your structure beneath what you provide using a generic organization, grouped based on the type of symbol.

That generic structure isn’t too bad. It’s certainly understandable for many developers, so if you don’t get to this point, you still have useful docs due to the abstracts and overviews — that’s a win!

5 – Add a walk-through article.

Add an article that provides a walk-through of the most common, basic usage of library. The Documentation article template from Xcode looks suspiciously like the top-level organization page, but the intention of an article is quite different. It still has a structural pattern of a one-line abstract at the top (the placeholder that reads summary in the template). Beneath that a ## Overview heading, and then a placeholder for a summary of the article. When I write an article, I leave that all alone and jump down into the content.

A single ### heading after the overview section with content underneath it is where to start. Group what you’re showing, frequently I’ve seen this listed by task or step, by the 3rd level headings, and include the detail underneath. Intermix text and code snippets – and images if you can make the time and it makes sense. The point of what you’re making is to provide a quick overview of how to use the library, or at least one way of getting started with it. You may find there’s a lot you could say – too much for one article. That’s fine — in fact, that’s wonderful — just make more articles and add them in.

As you add an article, curate it into your documentation. By which I mean go back to that top-level document and add a link to your article. The article links look like <doc:Your_Article's_File_Name>. When you build the documentation, the rendered content that’s dropped in place there is the top-level heading in your article, and it makes a link over to your content. An article is only useful when you get easily get to it.

I debated if this should be above the curation suggestion or not. It’s more work, certainly, but there is a ton of value. If you can’t, or don’t want, to spend a lot of time on this – focus on getting out something that shows someone wanting to use your library how to do it. There’s so much more that can be said about articles and organization, and opinions on what’s good and bad – but focus first on showing the key elements of your library, and you can build from there.

6 – Add strategic code snippets.

Hopefully your article had useful, direct code snippets in it. The next step to making that kind of directly usable content easier to find is to add code snippets into the overviews for some of your symbols. Take the time to pick the most important, or perhaps the most common, ways your library is used. I like to scribble that down on the side, and build my own little stack-rank of what I think would be most interesting to someone wanting to use my library. Then go through and add code snippets showing how to use that type, method, or function.

A pattern that I’m using at the moment is to create a unit test function that doesn’t actually test anything other than the code I just typed compiles correctly. When I’m making code snippets — both for articles and overviews in the reference section — I add a test case and put the code snippet in there. It keeps me honest with parameters and explicit formatting, and most importantly I’ve found – when the library evolves and something changes, you’ve got an immediate notice in the tests that something in the documentation needs an update there as well. I often put a comment in those “unit tests” that indicate where I used the snippet – for those times when a test fails and then I need to sort of what needs a tweak or update.

7 – Add curation to the rest of your types.

That same organization you provided for the top level of your library? Yeah – this is doing it for pieces below that top level. If you have a large class or struct, or types that include other types, this is hitting at the heart of making the list of symbols within them readily understandable.

While you can recursively do this through all your types, and a person with completion-is tendencies (which very much describes me) might want to, focus on the top level items and biggest collections. It is most useful to get all of your library’s types that include other symbols. When I’m doing this – I work down the types in the same order I added them into top-level structure. That’s assuming you ordered it most-important-first, so as long as you got that pretty close, you’re getting the most valuable pieces done first.

For types with a small number of sub-symbols (properties, enumeration cases, etc), I’ll include the organization of that symbol directly in the source code. But for a symbol with more than 4 or 5 symbols beneath it, I find it easier to manage by making an extension page in the Documentation catalog and including the curation/organization content there.

The process of adding an extension page is straightforward: in your documentation catalog (the directory ending in .docc) add a directory named Extensions, and then add markdown files into it. Xcode includes an extension template file type you can use. I name each of the files for the symbol that it provides organization for. Inside the extension file, include the curation. The content will look something like the following (example borrowed from in-the-works documentation updates for Nick Lockwood’s Euclid library):

# ``Euclid/Angle``

## Topics

### Creating Angles

- ``Angle/init(degrees:)``
- ``Angle/init(radians:)``
- ``Angle/degrees(_:)``
- ``Angle/radians(_:)``

### Inspecting Angles

- ``Angle/degrees``
- ``Angle/radians``

### Common Angles

- ``Angle/zero``
- ``Angle/halfPi``
- ``Angle/pi``

The reason I like using an extension file is that it doesn’t consume a lot of vertical space in the source code. I think it’s useful to have the abstract and overview as doc comments in the source, next to what they describe. I also think it’s less valuable for the organization to be there, and there is a cost to the readability of the source code. The organization isn’t the kind of thing you need to update when you’re tweaking a method’s signature or internals and how it works.

What are the symbols for my project?

When I’m tackling this curation process in adding documentation, I find it super-helpful to know the available symbols and their links. A symbol link is one of those items that represent a symbol that in text start and end with double back-ticks. For example, “init(_:)“ is a symbol link that references to an initializer. When you have two functions that have the same signature, but different types, then DocC adds a little hash value on to the end to disambiguate them.

Unfortunately, getting a complete list is kind of pain right now. Xcode auto-completion does a pretty good job of auto-completing symbols, so you can explore with auto-completion, but I recently ran into some issues where it missed a few.

One way you can find the values is by running xcrun docc preview on the command line, and navigating to through the rendered docs to the symbol you’re trying to find. The URL of that page has the symbol details at the end of the URL. If the symbol has a disambiguating hash code, it’s also is reflected in the URL. But that’s a lot of clicking around to get to a single value.

Another way to get the values is generating the documentation with the --emit-digest option, and extracting what you want from there. I learned this from the DocC team members on the Swift Forums. The emit digest option adds a file to the generated content named docs/linkable-entities.json. It’s a JSON file that includes a documentation reference for every symbol published in your docs as a documentation link. And a substring of the documentation link is what DocC uses as symbol links.

I knocked together a shell script snippet that uses the command line tools jq and other shell commands to generate a sorted list of all the possible symbols:

cat docs/linkable-entities.json \
  | jq '.[].referenceURL' -r > all_identifiers.txt

sort all_identifiers.txt \
  | sed -e 's/doc:\/\/SceneKitDebugTools\/documentation\///g' \
  | sed -e 's/^/- ``/g' \
  | sed -e 's/$/``/g' > all_symbols.txt

The sed commands are regular-expression replacing the lines that convert the referenceURLs into symbols. It strips off the doc://YourModuleName/documentation (the example above being pulled from a tiny library I have) from the list of URLs and adds into the double back-ticks.

I’ve started using the DocC plugin for generating documentation suitable for hosting on Github pages, but you don’t need to use that for getting this file generated. Running docc convert with the --emit-digest will get the same file generated for you. It’s placed in the output directory you specify for the conversion. If you’re lost on how to do all that, take look at one of the scripts that I’m using: docbuild.bash. I started doing this before the DocC plugin tool became available, so it includes all the steps from building the source, to converting the symbol graphs using DocC.

%d bloggers like this: