Using Combine – reference content complete!

I’m thrilled to be announcing that an updated version of Using Combine is now available!

It has taken me nearly 6 months to draft it all, reverse engineering and writing tests for all the various publishers, operators, and pieces in between – and documenting what I found. The end result is 182 pages (in US PDF format) of reference documentation the way I’d generally like to have it.

While the live site is updated automatically, updated PDF and ePub versions are now available on Gumroad. If you purchased a copy previously, you can go to Gumroad and get an updated, DRM free, content in either PDF or ePub formats.

This updates finishes the largest swath of reference updates, creating tests to verify all the various operators and writing the documentation reference sections for the following:

There was also an update for Xcode 11.3 and associated iOS 13.3 and macOS 10.15.2, which included some subtle changes to the throttle operator behavior, which I recently wrote about in some detail.

With this update, the majority of the core content is now complete, but the work is by no means finished!

The next steps for the book are review and editing. On the to-do list are refining the descriptions of the reference sections, reviewing all the patterns now that we have had Combine for a few months, and seeing the updates as the API changes and refines. There are some diagrams now, but more are likely needed in some sections – both in the patterns and reference sections.

As before, this continues as a labor of love and for the community. Meaning that the content will continue to be free, available on the live site, with updates being made available as I make them. The work has been financially supported by 116 people as I’m writing this, as well as a number of people providing pull requests to fix typos and grammar flaws.

If you want a DRM-free digital copy in either PDF or ePub format, please consider supporting this work by purchasing a copy at gumroad.

Combine: throttle and debounce

Combine was announced and released this past summer with iOS 13. And with this recent iOS 13 update, it is still definitely settling into place. While writing Using Combine, I wrote a number of tests to verify and generally double-check my understanding of how Combine was working. With the update to iOS 13.3, the tests showed me that a few behaviors changed once again.

The operator that changed and trigged my tests was throttle. Throttle is meant to act on values being received from an upstream publisher over a sliding window. It collapses the values flowing through a pipeline over time, choosing a representative value from the set that appeared within a given time window. It also turns out that throttle has slightly different behavior when you’re working with a publisher that starts out sending down an initial value (such as a @Published property).

While I was poking at throttle to understand how it changed, I also realized that debounce was acting differently than I had originally understood, so I took some time to write some additional tests and make a more explicit timing diagram for it as well. Since debounce didn’t change behavior between 13.2 and 13.3 (that I spotted anyway), I’ll describe it first.

debounce

When you set up a debounce operator, you specify a time-window for it to react within. The operator collects all values that come in from the publisher, and most notably it resets the starting point for that sliding time window when it receives a value. It won’t send any values on until that entire window has expired without any other values appearing. In effect, it’s waiting for the value to settle. The marble diagrams show this really well.

Without a break that lets the sliding window expire, a single value is returned.
With a break that lets the sliding window expire, two values are propagated.

throttle

Throttle acts similarly to debounce, in that it collects multiple results over time and sends out a single result – but it does so with fixed time windows. Where debounce will reset the start of that window, throttle does not – so it doesn’t collapse the values entirely, but sort of “slows them down” (and that matches the name of the operator pretty well).

Which value from the set that arrive that’s chosen to be propagated is influenced by the parameter latest, which is set when you create the operator. In general, latest being set to true results in the last value appearing getting chosen, and latest being set to false results in the first value that appears. This is one of those items that is a lot easier to understand by looking at a marble diagram:

Throttle (latest=false), under iOS 13.2.2
throttle (latest=true) under iOS 13.2.2

The notable behavior change is how it handles initial values. Initial value seems to be a bit “flexible” in what is specifically initial though. When I first wrote my tests, I was using a class with a @Published variable, which sends a value upon subscription, and then updates when it is changed. To illustrate the 13.3 behavior change better, I re-wrote and expanded those tests to use a PassthroughSubject, so there wasn’t an automatic initial value.

throttle (latest=false) under iOS 13.3
throttle (latest=true) under iOS 13.3

So under iOS 13.3, the initial value (which is sent out roughly 100ms after the creation of the pipeline in my test), is always propagated, and then the sliding window effect begins immediately after that.

If you want to see the underlying tests that illustrate this, check out the following bits of code from the Using Combine project:

Using Combine (v0.8) update available!

A new version of Using Combine (v0.8) is now available.

The live HTML site for Using Combine is updated automatically, and the PDF and ePub versions are now available on Gumroad.

This version has a number of additional notes and changes, primarily from reader feedback, and some references to Combine’s changes with the release of IOS 13.2. A few more issues have been noted in the book, along with references to feedback reports sent to Apple where they may represent bugs or unexpected behavior in the current implementation.

This release also includes SVG based diagrams – so the original ASCII art diagrams are now gone, which should make that content far more accessible in the ePub format.

In addition, I added a section on marble diagrams, specifically in how to read them and how they apply to the code and examples illustrated in this book. I originally planned on generating all the marble diagrams, but after repeated efforts at that I backed off that idea and am creating them by hand, with the help of OmniGroup‘s wonderful tool OmniGraffle. And yes, the source for this is also in the github repository.

For the next release, I am planning on getting back to detailing out the as-yet-unwritten section on a number of operators. 

The project board at https://github.com/heckj/swiftui-notes/projects/1 also reflects all the various updates still remaining to be written.

A huge thank you to all who have supported this book and my efforts to provide it!

It is OK to test the framework

When I started to write the book Using Combine, I was learning the Combine framework as I went. There was a lot I was unsure about, and especially given that it was released with the beta of the operating system, the implementation was changing between beta releases as it firmed up. I chose to use a technique that I picked up years ago from Mike Clark – write unit tests against the framework to verify my understanding of it – while writing the book. (yes, I’m still working on it – it’s a very lengthy process)

While listening to a few episodes of the Under the Radar podcast, I heard a number of references to the idea of “make sure you’re not testing the framework”. It is generally good advice, in the vein of “make sure you’re testing your code first and foremost”, but as a snippet out of context and taken as a rule – I think it’s faulty. Don’t confuse what you are testing, but reliably testing underlying frameworks or libraries, especially while learning them or they evolve, can easily be worth the effort.

I have received a huge amount of value from testing frameworks – first in verifying that I understand what the library is doing and how it works. More over, it has been a very clear signal when regressions do happen, or intentional functionality changes.

If you do add tests of a framework or library into your codebase, I recommend you break them out into their own set of tests. If something does change in the library, it will be far more clear that it is a change from the library and not a cascading side effect in your code.

Most recently, this effort paid off when I stumbled across a regression in the Combine framework functionality with the GM release of Xcode 11.2. While I’ve been coming up to speed with the various operators, I’ve written unit tests that work the operators. In this case, the throttle operator – which has an option parameter latest – changed in how it operates with this release.

Throttle is very similar to the debounce operator, and in fact it operates the same if you use the option latest=true. They both take in values over time and return a single value for a specific time window. If you want the first value that’s sent within the timeframe, theoretically you should use latest=false with the throttle operator. This worked in earlier releases of Combine and Xcode – but in the latest release, it’s now disregarding that path and sending only the latest value.

You can see the tests I wrote to verify the functionality at https://github.com/heckj/swiftui-notes/blob/master/UsingCombineTests/DebounceAndRemoveDuplicatesPublisherTests.swift, and right now I’m working on a pull request to merge in the change reflecting the current release and illustrating the regression. And before you ask, yes – I have submitted this as a bug to Apple (FB7424221). If you are relying on the specific functionality of throttle with latest=false, be aware that the latest release of Xcode & Combine is likely going to mess with it.

If you are more curious about all the other tests that were created to support Using Combine, then feel free to check out the github repository heckj/swiftui-notes – the tests are in the UsingCombineTests directory, and set up as they’re own test target in the Xcode project. There are more to write, as I drive down into the various operators, so I do expect more will appear. I won’t assert that they’re all amazing, well constructed tests – but they’re getting the job done in terms of helping me understand how they work – and how they don’t work.

Sharp Knives

After writing extensively with the Swift programming language, I recently spent time writing code in C++11. The experience has been very instructional, and I know have a much deeper appreciation for the guard rails that the language Swift has constructed. To my surprise, it has also left me with a renewed appreciation of the compiler tooling – LLVM, specifically the relative clarity of error messages.

C++ as a language was significantly more approachable to me after working on code using Swift for a while. There are a lot of parallels and similarities, which was particularly nice since I was porting code (re-writing the algorithms in another programming language). I think porting between programming languages is the rough equivalent of translating between human languages. In really comparing to allow you to move between the two, you become far more aware of the idiomatic conveniences and conventions.

One of the surprisingly pieces is my realization that I am far more attached to the swift concept of protocols than I had realized. C++ has a concept of interfaces, which is close – but definitely not the same in it’s breadth.

When combined with generic algorithms, the swift language feels far more capable and specific to me. In the swift programming language, you can constrain the implementation of your generic algorithms to only apply to specific protocols, which appears to be something you can’t easily do in C++ – or at least in C++11, which was the version I was working within. I found that programming with generics in C++ is far more reliant on convention, or possibly using subclasses – examples were a bit hard to come by for me.

My limited experience with C++ also leads me to think that the conventions followed between different groups of programmers is more diverse than Swift. The idiomatic patterns I found while reading other groups code had dramatically different patterns included within them. So much so that it was often hard to identify those patterns, and understand what was a team’s convention.

My time with the C++ components also makes me appreciate all the more the tendency of languages these days (to borrow from Python) to come “batteries included” with standard libraries. C++ standard library is more like tools to make tools, and some of the things commonly included with other languages (python, swift, etc) just have to be found individually and assembled.

While I have a bit more to do with C++ before I’ll be done with my project, I relish shifting back to the guard rails of swift. (I must admit, I’m now significantly more curious about the language Rust as well).

A Using Combine update now available!

A new version of Using Combine (v0.7) is now available! 

The free HTML site of Using Combine has been updated automatically, and the PDF and ePub versions are available on Gumroad.

This version has relatively few updates, primarily focused on some of the missing publishers and resolving the some of the egregious flaws in ePub rendering. No significant changes have come with the later Xcode and IOS betas, and with Xcode 11 now in GM release, it was a good time for another update to be made available.

For the next release, I am focusing on fleshing out a number of the not-yet-written reference sections on operators, most of which are more specialized than the more generally used ones that have already been covered.

The project board at https://github.com/heckj/swiftui-notes/projects/1 also reflects all the various updates still remaining to be written.

Using Combine (v0.6) available!

design by Michael Critz

A new version of Using Combine is available! The free/online version of Using Combine is updated automatically as I merge changes, and the PDF and ePub versions are released periodically and available on Gumroad.

https://gumroad.com/js/gumroad.js Purchase Using Combine

The book now has some amazing cover art, designed and provided by Michael Critz, and has benefited from updates provided by a number of people, now in the acknowledgements section.

The updates also include a section broken out focusing on developing with Combine, as well as a number of general improvements and corrections.

For the next release, I am going to focus on fleshing out a number of the not-yet-written reference sections:

the publishers I haven’t touched on yet

– starting into a number of the operators, most of which are more specialized

I reviewed the content prior to this release to see what was remaining to be done, and updated the project planning board with the various updates still remaining to be written.

I do expect we’ll see beta6 from Apple before too long, although exactly when is unknown. I thought it might appear last week, in which case I was planning on trying to accommodate any updates in this release. Xcode 11 beta6 hasn’t hit the streets and I wanted to get an update regardless of its inclusion.

navigating Swift Combine, tuples, and XCTest

What started out as a Github repository to poke at SwiftUI changed course a few weeks ago and became a documentation/book project on Combine, Apple’s provided framework for handling asynchronous event streams, not unlike ReactiveX. My latest writing project (available for free online at https://heckj.github.io/swiftui-notes/) has been a foil for me to really dig into and learn Combine, how it works, how to use it, etc. Apple’s beta documentation is unfortunately highly minimal.

One of the ways I’ve been working out how its all operating is writing a copious amount of unit tests, more or less “poking the beast” of the code and seeing how it’s operating. This has been quite successful, and I’ve submitted what I suspect are a couple of bugs to Apple’s janky FeedbackAssistant along with tests illustrating the results. As I’m doing the writing, I’m generating sample code and examples, and then often writing tests to help illuminate my understanding of how various Combine operators are functioning.

In the writing, I’ve worked my way through the sections to where I’m tackling some of the operators that merge streams of data. CombineLatest is where I started, and it testing it highlighted some of the more awkward (to me) pieces of testing swift code.

The heart of the issue revolves around asserting equality with XCTest, Apple’s unit testing framework, and the side effect that Combine takes advantage of tuples as lightweight types in operators like CombineLatest. In the test I created to validate how it was operating, I collected the results of the data into an ordered list. The originated streams had simple, equatable types – one String, the other Int. The resulting collection, however, was a tuple of <(String, Int)>.

To use XCTAssertEquals, the underlying types that you are validating need to conform to Equatable protocol. In the case of checking a collection type, it drops down and relies on the equatable conformance of its underlying type. And that is where it breaks down – tuples in swift aren’t allowed to conform to protocols – so they can’t declare (or implement conformance with) equatable.

I’m certainly not the first to hit this issue – there’s a StackOverflow question from 2016 that highlights the issue, Michael Tsai highlights the same on his blog (also from 2016). A slightly later, but very useful StackOverflow Q&A entitled XCTest’ing a tuple was super helpful, with nearly identical advice to Paul Hudson’s fantastic swift tips in Hacking With Swift: how to compare equality on tuples. I don’t know that my solution is a good one – it really feels like a bit of a hack, but I’m at least confident that it’s correct.

The actual collection of results that I’m testing is based on Tristan’s Combine helper library: Entwine and EntwineTest. Entwine provides a virtual time scheduler, allowing me to validate the timing of results from operators as well as the values themselves. I ended up writing a one-off function in the test itself that did two things:

  • It leveraged an idea I saw in how to test equality of Errors (which is also a pain in the tuckus) – by converting them to Strings using debugDescription or localizedDescription. This let me take the tuple and consistently dump it into a string format, which was much easier to compare.
  • Secondarily, I also wrote the function so that the resulting tests were easy to read in how they described the timing and the results that were expected for a relatively complex operator like combineLatest.

If you’re hitting something similar and want to see how I tackled it, the code is public UsingCombineTests/MergingPipelineTests.swift. No promises that this is the best way to solve the problem, but it’s getting the job done:

func testSequenceMatch(
    sequenceItem: (VirtualTime, Signal<(String, Int), Never>),
    time: VirtualTime,
    inputvalues: (String, Int)) -> Bool {

    if sequenceItem.0 != time {
        return false
    }
    if sequenceItem.1.debugDescription != Signal<(String, Int),
       Never>.input(inputvalues).debugDescription {
        return false
    }
    return true
}

XCTAssertTrue(
    testSequenceMatch(sequenceItem: outputSignals[0], 
                      time: 300, inputvalues: ("a", 1))
)

XCTAssertTrue(
    testSequenceMatch(sequenceItem: outputSignals[1], 
                      time: 400, inputvalues: ("b", 1))
)

Tristan, the author of Entwine and EntwineTest, provided me with some great insight into how this could be improved in the future. The heart of it being that while swift tuples don’t/can’t have conformance to protocols like Hashable and Equatable, structs within Swift do. It’s probably not sane to make every possible combinatorial set in your code, but it’s perfectly reasonable to make an interim struct, map the tuples into it, and then use that struct to do the testing.

Tristan also pointed out that a different struct would be needed for the arity of the tuple – for example, a tuple of <String, Int> and <String, String, Int> would need different structs. The example implementation that Tristan showed:

struct Tuple2 {
  let t0: T0
  let t1: T1
  init(_ tuple: (T0, T1)) {
    self.t0 = tuple.0
    self.t1 = tuple.1
  }
  var raw: (T0, T1) {
    (t0, t1)
  }
 }

extension Tuple2: Equatable where T0: Equatable, T1: Equatable {}
extension Tuple2: Hashable where T0: Hashable, T1: Hashable {}

A huge thank you to Tristan for taking the time to explain the tuple <-> struct game in swift!

After having spent quite a few years with dynamic languages, this feels like jumping through a lot of hoops to get the desired result, but I’m pleased that at least it also makes sense to me, so maybe I’m not entirely lost to the dynamic languages.

SceneKit interaction handling – Experiment439

A staple of science fiction movies has been 3D holographic visualizations and controls. Most efforts I’ve seen at taking real visualization data and putting them into a 3D context haven’t been terribly successful. At the same time, the advance of AR and VR makes me suspect that we should be able to take advantage of the additional dimension in displaying and visualizing data.

I started a project, named Experiment439, to go through the process of creating and building a few visualizations and seeing what I can do with them, and what it might be refined out into a library that can be re-used.

I wanted to take a shot at this leveraging Apple’s SceneKit 3D abstraction and see how far I could get.

The SceneKit abstraction and organization for scenes is a nice setup, although it’s weak in one area – delegating interaction controls.

The pattern I’m most familiar with is the View Controller setup (and it’s many variants depending on how you display data). Within SceneKit, an SCNNode can encapsulate other nodes (and controls overall placement in the view), so it makes a fairly close analogue to the embedding of views within each other that I’m familiar with from IOS and MacOS development. Coming up with something that encapsulates and controls a SCNNode (or set of SCNNodes) seems like it’s a pretty doable (and useful) abstraction.

The part that gets complicated quickly is handling interaction. User-invoked events in SceneKit today are limited to projecting hit-tests from the perspective of the camera that’s rendering the scene. In the case of AR apps on IOS for example, the camera can be navigating the 3D space, but when you want to select, move, or otherwise interact you’re fairly constrained to mapping touch events projected through the camera.

I’ve seen a few IOS AR apps that use the camera’s positioning as a “control input” – painting or placing objects where the IOS camera is positioned as you move about an AR environment.

You can still navigate a 3D space and scene, and see projected data – both 2D and 3D very effectively, but coming up with equivalent interactions to what you get on Mac and IOS apps – control interactions – has been significantly trickier.

A single button that gets toggled on/off isn’t too bad, but as soon as you step into the world of trying to move a 3D object through the perspective of the camera – shifting a slider or indicating a range – it gets hellishly complex.

With Apple’s WWDC 2019 around the corner (tomorrow as I publish this) and the rumors of significant updates to AR libraries and technologies, I’m hoping that there may be something to advance this space and make this experiment a bit easier, and even more to expand on the capabilities of interacting with the displayed environment.

IOS AR apps today are a glorified window into a 3D space – amazing and wonderful, but heavily constrained. It allows me to navigate around visualization spaces more naturally than anything pinned to a desktop monitor, but at the cost of physically holding the device that you would also use to interact with the environment. I can’t help but feel a bit of jealousy for the VR controllers that track in space, most recently the glowing reviews of the Valve Index VR controllers.

Better interaction capabilities of some kind will be key to taking AR beyond nifty-to-see but-not-entirely-useful games and windows on data. I’m hoping to see hints of what might be available or coming with the Apple ecosystem in the next few days.

Meanwhile, there still a tremendous amount that to be done to make visualizations and display them usefully in 3D. A lot of the inspiration of the current structure of my experiment has been from Mike Bostock‘s amazing D3.js library, which has been so successful in helping to create effective data visualization and exploration tools.

IOS 12 DevNote: Embedded Swift Frameworks and bitcode

A side project for the barista’s at my favorite haunt has been a fun “getting back into it” programming exercise for IOS 12. It’s a silly simple app that checks the status of the network and if the local WIFI router is accessible, and provides some basic diagnostic and suggestions for the gang behind the counter.

It really boils down to two options:

    Yep, probably a good idea to restart that WIFI router
    Nope, you’re screwed – the internet problem is upstream and there’s nothing much you can do but wait (or call the Internet service provider)

It was a good excuse to try out the new Network.framework and specifically NWPathMonitor. In addition to the overall availability, I wanted to report on if a few specific sites were responding that the shop often uses, and on top of that I wanted to do some poking at the local WIFI router itself to make sure we could “get to it” and then made recommendations from there.

As I dug into things, I ended up deciding to use a swift framework BlueSocket, with the idea that if I could open a socket to the wifi router, then I could reasonably assume it was accessible. I could have used Carthage or CocoaPods, but I wanted to specifically try using git submodules for the dependencies, just to see how it could work.

With XCode 10, the general mechanism of dragging in a sub-project and binding it in works extremely easily and well, and the issues I had really didn’t hit until I tried to get something up to the IOS App Store for TestFlight.

The first thing I encountered was the sub-projects had a variable for CFVersionBundle: $(CURRENT_PROJECT_VERSION) that apparently wasn’t getting interpolated and set when it was built as a subproject. I ended up making a fork of the project and hard-coding the Info.plist with the specific version. Not ideal, but something that’s at least tractable. I’m really hoping that this coming WWDC shows some specific Xcode/IOS integration improvements when it comes to Swift Package Manager. Sometimes the Xcode build stuff can be very “black box”, and it would be really nice to have a more clear integration point for external dependencies.

The second issue was a real stumper – even though everything was validating locally for a locally built archive, the app store was denying it. The message that was coming back:

Invalid Bundle – One or more dynamic libraries that are referenced by your app are not present in the dylib search path.

Invalid Bundle – The app uses Swift, but one of the binaries could not link to it because it wasn’t found. Check that the app bundles correctly embed Swift standard libraries using the “Always Embed Swift Standard Libraries” build setting, and that each binary which uses Swift has correct search paths to the embedded Swift standard libraries using the “Runpath Search Paths” build setting.

I dug through all the linkages with otool, and everything was looking fine – and finally google trawled across a question in StackOverflow. Near the bottom there was a suggestion to disable bitcode (which is on by default when you upload an IOS archive). I gave that a shot, and it all flowed through brilliantly.

I can only guess that when you’re doing something with compiled-from-swift dylib’s, the bitcode process does something that the app store really doesn’t like. No probably without the frameworks (all the code in the project directly), but with the frameworks in my project, bitcode needed to be turned off.

Made it through all that, and now it’s out being tested with TestFlight!

El Diablo Network Advisor