spelunking Swift Package Manager

In the slowly building sequence of my swift dev diaries, I wrote about how to set up a swift development environment, and noted some details I gleaned from the SwiftPM slack channel about how to make a swift 3.0/3.1 binary “portable”. This likely will not be an issue in another year, as the plans for SwiftPM under swift 4 include being able to make statically compiled binaries.

Anyway, the focal point for all this was an excuse to learn and start seriously using swift, and since I’ve been a lot more server-side than not for the past years, I came at it from that direction. With the help of the guys writing swift package manager, I picked a few bugs and started working on them.

Turns out Swift Package Manager is a fairly complex beast. Like any other project, a seemingly simple bug leads to a lot of tendrils through the code. So this weekend, I decided I would dive in deep and share what I learned while spelunking.

I started from the bug SR-3275 (which was semi-resolved by the time I got there), but I decided to try and “make it better”. SwiftPM uses git and a number of other command line tools. Before it uses it, it would be nice to check to make sure they’re there, and if they’re not – to fail gracefully, or at least informatively. My first cut at this  improved the error output and exposed me to some of the complexity under the covers. Git is used in several parts of SwiftPM (as are other tools). It has definitely grown organically with the code, so it’s not entirely consistent. SwiftPM includes multiple binary command-line tools, and several of them use Git to validate dependencies, check them out if needed, and so forth.

The package structure of SwiftPM itself is fairly complex, with multiple separate internal packages and dependencies. I made a map because it’s easier for me to understand things when I scrawl them out.

swiftpm is the PDF version (probably more readable) which I generated by taking the output of swift package --dump-package and converting it to a graphviz digraph, rendering it with my old friend OmniGraffle.

The command line tools (swift build, swift package, etc) all use a common package Commands, which in turn relies and the various underlying pieces. Where I started was nicely encapsulate in the package SourceControl, but I soon realized that what I was fiddling with was a cross-cutting concern. The actual usage of external command line tools happens in Utility, the relevant component being Process.swift.

Utility has a close neighbor: Basic, and the two seem significantly overlapping to me. From what I’ve gathered, Utility is the “we’re not sure if it’s in the right place” grouping, and Basic contains the more stabilized, structured code. Process seems to be in the grey area between those two groupings, and is getting some additional love to stabilize it even as I’m writing this.

All the CLI tools use a base class called SwiftTool, which sets a common structure. SwiftTool has an init method that sets up CLI arguments and processes the ones provided into an Options object that can then get passed down in the relevant code that does the lifting. ArgumentParser and ArgumentBinder do most of that lifting. Each subclass of SwiftTool has its own runImpl method, using relevant underlying packages and dependencies. run invokes the relevant code in runImpl. If any errors get thrown, it deals with them (regardless of the CLI invoking)  with a method called handle in Error. It is mostly printing some (hopefully useful) error message and then exiting with an error code.

My goal is to check for required files – that they exist and are executable – and report consistently and clearly if they’re not. From what I’ve learned so far that seems to be best implemented by creating a relevant subclass (or enum declaration) of Swift.Error and throwing when things are awry, letting the _handle implementation in Commands.Error take care of printing things consistently.

So on to the file checking parts!

The “file exists” seemed to have two implementations, one in Basic.Filesystem and another in Basic.Pathshims.

After a bit more reading and chatting with Ankit, it became clear that Filesystem was the desired common way of doing things, and Pathshims the original “let’s just get this working” stuff. It is slowly migrating and stabilizing over to Filesystem. There are a few pieces of the Filesystem implementation that directly use Pathshims, so I expect a possible follow-up will be to collapse those together, and finally get rid of Pathshims entirely.

The “file is executable check” didn’t yet exist, although there were a couple of notes that it would be good to have. Similar checking was in UserToolchain, including searching environment paths for the executable. It seemed like it needed to be pulled out from there and moved to a more central location, so it could be used in any of the subclasses of SwiftTool.

At this point, I’ve put up a pull request to add in the executable check to Filesystem, and another to migrate the “search paths” logic into Utility. I need to tweak those and finish up adding the tests and comments, and then I’ll head back to adding in the checks to the various subclasses of SwiftTool that could use the error checking.