This article isn’t a how-to so much as a debugging/dev diary entry for future-me, and any other soul who stumbles into the same (or similar) issues.
Let me provide the backdrop for this story:
I’m working on a private C++ language based project, previously written to be cross platform (Windows, Linux, and Mac). It has a number of C++ library dependencies, which it’s managing with vcpkg, a fairly nice library package manager solution for C++ projects. It happens to align well with this project, which uses CMake as its build system. One of the dependencies that this project uses is grpc, which in turn has a transitive dependency on OpenSSL.
With the M1 series of laptops available from Apple, we wanted to compile and use this same code as an M1
arm64 native binary. Sure – makes sense, should be easy. The good news is, it mostly has been. Both vcpkg and openssl recently had updates to resolve compilation issues with M1/arm based Macs, most of which revolved around a (common) built-in assumption that macOS meant you were building for an
x86_64 architecture, or maybe, just maybe, cross-compiling for iOS. The part that hasn’t been so smooth is there’s an odd complication with vcpkg, openssl, and the M1 Macs that ends up with a linker error when the build system tries to integrate and link the binaries created and managed by vcpkg. It boils down to this:
ld: in /Users/heckj/bin/vcpkg/installed/arm64-osx/debug/lib/libcrypto.a(a_strex.o), building for macOS, but linking in object file built for iOS
For anyone else hitting this sort of thing, there’s two macOS specific command-line tools that you should know about to investigate this kind of thing:
lipo does a number of things – mostly around merging various archives into fat libraries, but the key part I’ve been using it for is to determine what architecture a library was built to support. The command
lipo -info /path/to/library.a, tells you the architecture for that library. I stashed a copy of vcpkg in
~/bin on my laptop, so using the command
lipo -info ~/bin/vcpkg/installed/arm64-osx/lib/libcrypto.a reports the following:
Non-fat file: /Users/heckj/bin/vcpkg/installed/arm64-osx/lib/libcrypto.a is architecture: arm64
Prior to OpenSSL version 1.1.1i, that reported an
As I’ve learned, architecture alone isn’t sufficient for C++ code when linking the library (at least on macOS). When the libraries are created (compiled), the libraries are also marked with information about what platform they were built for. This is a little harder to dig out, and where the command line tool
otool comes into play. I found some great detail on Apple’s Developer forum in the thread at https://developer.apple.com/forums/thread/662611, which describes using
otool, but not quite all the detail. Here’s the quick summary:
You can view the platform that is embedded into the library code directly using
otool -lv. Now this generates a lot of output, and you’re looking for some specific patterns. For example, the command
otool -lv ~/bin/vcpkg/installed/arm64-osx/debug/lib/libcrypto.a | grep -A5 LC_
includes this stanza in the (copious) output:
cmd LC_VERSION_MIN_IPHONEOS cmdsize 16 version 5.0 sdk n/a Load command 2
And as far as I’ve been able to discern, if you see
LC_VERSION_MIN_IPHONEOS in the output, it means the library was built for an iOS platform, and you’ll get the linker error I listed above when you try to link it to code built for macOS.
Another library which did get compiled “correctly” shows the following stanza within its output:
cmd LC_BUILD_VERSION cmdsize 24 platform MACOS minos 11.0 sdk 11.1 ntools 0
LC_BUILD_VERSION and then the details following it shows the library can be linked against macOS code build for version 11.0 or later.
After a number of false starts and deeper digging, I found that OpenSSL 1.1.1i included a patch that enabled arm64 macOS compilation. The patch https://github.com/openssl/openssl/pull/12369 specifically enables a new platform code:
I grabbed OpenSSL from its source, and started poking around. A lot of that poking and learning the innards of how OpenSSL does its builds wasn’t entirely useful, so I won’t detail all the dead ends I attempted to follow. What I did learn, in the end, was that the terminal – being native
arm64 or rosetta2 emulated
x86_64 – can make a huge difference. For convince, I made a copy of iTerm and ran it under rosetta so that I could easily install homebrew and use all those various tools, even though it wasn’t arm64 native.
What I found is that if I want to make a macOS native version of OpenSSL, I need to run the compilation from a native
arm64 terminal session. Something is inferring the target platform – not sure what – from the shell in which it runs. I manually configured and compiled OpenSSL with an
arm64 native terminal, and was able to get an arm64 library, with the internal markers for macOS.
From there, I thought that perhaps this was an issue of how OpenSSL was configured and built. That’s something that vcpkg controls, with these little snippets of cmake under the covers. vcpkg does a nice job of keeping logs, so I went through the compilation for openssl (using
--triplet arm64-osx in case you want to follow along), and grabbed all the configuration and compilation steps. I grabbed those logs and put them into their own text file, and edited them so I could run them step by step in my own terminal window and see the status of the output as it went. I then adapted the steps to reference a clean git repository checkout from openssl and the tag
OpenSSL_1_1_1i, and updated the prefix to a separate directory (
/Users/heckj/openssl_arm64-osx/debug) so I could inspect things without stepping into the vcpkg spaces.
cd /Users/heckj/src/openssl git reset --hard OpenSSL_1_1_1i export CC=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc export AR=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ar export LD=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld export RANLIB=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib export MAKE=/usr/bin/make export MAKEDEPPROG=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc export PATH=$PATH:/Users/heckj/bin/vcpkg/downloads/tools/ninja-1.10.1-osx /usr/bin/perl Configure no-shared enable-static-engine no-zlib no-ssl2 no-idea no-bf no-cast no-seed no-md2 no-tests darwin64-arm64-cc --prefix=/Users/heckj/openssl_arm64-osx/debug --openssldir=/etc/ssl -fPIC "--sysroot=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.1.sdk" /usr/local/Cellar/cmake/3.19.3/bin/cmake -DDIR=/Users/heckj/src/openssl -P /Users/heckj/bin/vcpkg/ports/openssl/unix/remove-deps.cmake # [2/3] cd /Users/heckj/src/openssl export PATH=$PATH:/Users/heckj/bin/vcpkg/downloads/tools/ninja-1.10.1-osx /usr/local/Cellar/cmake/3.19.3/bin/cmake -E touch /Users/heckj/src/openssl/krb5.h /usr/bin/make build_libs
What I found was the if I ran this code under a terminal running under rosetta, I’d get the results that indicated the code was built against an iOS platform. And when I ran it under a nativeThis is a major insight, but I haven’t yet figured out how to apply it…
arm64terminal, it would correctly report macOS as the platform.
I originally installed and compiled vcpkg using the rosetta terminal, and it was running as an
x86_64 binary, so I thought perhaps that was the issue. Unfortunately, not. After I installed vcpkg with the
arm64 native (and verified the binary was
arm64 with the
lipo -info command), I made another run at installing openssl, but ended up with the same iOS linked binary.
Prior to getting this far, I opened issue 13854 as a question on the OpenSSL repository, which details some of this story. However, I now longer think that’s an issue, as I was able to get an arm64 native binary when I manually compiled things. There might be something OpenSSL could do to make this easier/better, but its build setup is incredibly complex and I get lost pretty darn quickly within it.
So to date, I’ve trailed this back to some interaction that vcpkg, and
x86_64 emulated binaries, are having on the build – but that’s it.
The story ends here, as I don’t have a solution. I have filed issue 15741 with the vcpkg project, with a summary of these details.
For anyone reading until the bitter end, I’d love any suggestions on how to fully resolve this. I hope that someone stumbles across the issue at some point with more knowledge than I and has a solution in the future. In the meantime, this blog post will hopefully record the error and how you can diagnose the architecture that a library is compiled for, even if it doesn’t solve the end problem of getting you to a final resolution.