This article isn’t a how-to so much as a debugging/dev diary entry for future-me, and any other soul who stumbles into the same (or similar) issues.
Let me provide the backdrop for this story:
I’m working on a private C++ language based project, previously written to be cross platform (Windows, Linux, and Mac). It has a number of C++ library dependencies, which it’s managing with vcpkg, a fairly nice library package manager solution for C++ projects. It happens to align well with this project, which uses CMake as its build system. One of the dependencies that this project uses is grpc, which in turn has a transitive dependency on OpenSSL.
With the M1 series of laptops available from Apple, we wanted to compile and use this same code as an M1 arm64 native binary. Sure – makes sense, should be easy. The good news is, it mostly has been. Both vcpkg and openssl recently had updates to resolve compilation issues with M1/arm based Macs, most of which revolved around a (common) built-in assumption that macOS meant you were building for an x86_64 architecture, or maybe, just maybe, cross-compiling for iOS. The part that hasn’t been so smooth is there’s an odd complication with vcpkg, openssl, and the M1 Macs that ends up with a linker error when the build system tries to integrate and link the binaries created and managed by vcpkg. It boils down to this:
ld: in /Users/heckj/bin/vcpkg/installed/arm64-osx/debug/lib/libcrypto.a(a_strex.o), building for macOS, but linking in object file built for iOS
For anyone else hitting this sort of thing, there’s two macOS specific command-line tools that you should know about to investigate this kind of thing: lipo and otool.
lipo
lipo does a number of things – mostly around merging various archives into fat libraries, but the key part I’ve been using it for is to determine what architecture a library was built to support. The command lipo -info /path/to/library.a, tells you the architecture for that library. I stashed a copy of vcpkg in ~/bin on my laptop, so using the command lipo -info ~/bin/vcpkg/installed/arm64-osx/lib/libcrypto.a reports the following:
Non-fat file: /Users/heckj/bin/vcpkg/installed/arm64-osx/lib/libcrypto.a is architecture: arm64Prior to OpenSSL version 1.1.1i, that reported an x86_64 binary.
otool
As I’ve learned, architecture alone isn’t sufficient for C++ code when linking the library (at least on macOS). When the libraries are created (compiled), the libraries are also marked with information about what platform they were built for. This is a little harder to dig out, and where the command line tool otool comes into play. I found some great detail on Apple’s Developer forum in the thread at https://developer.apple.com/forums/thread/662611, which describes using otool, but not quite all the detail. Here’s the quick summary:
You can view the platform that is embedded into the library code directly using otool -lv. Now this generates a lot of output, and you’re looking for some specific patterns. For example, the command
otool -lv ~/bin/vcpkg/installed/arm64-osx/debug/lib/libcrypto.a | grep -A5 LC_includes this stanza in the (copious) output:
cmd LC_VERSION_MIN_IPHONEOS
cmdsize 16
version 5.0
sdk n/a
Load command 2And as far as I’ve been able to discern, if you see LC_VERSION_MIN_IPHONEOS in the output, it means the library was built for an iOS platform, and you’ll get the linker error I listed above when you try to link it to code built for macOS.
Another library which did get compiled “correctly” shows the following stanza within its output:
cmd LC_BUILD_VERSION
cmdsize 24
platform MACOS
minos 11.0
sdk 11.1
ntools 0Spotting LC_BUILD_VERSION and then the details following it shows the library can be linked against macOS code build for version 11.0 or later.
Debugging
After a number of false starts and deeper digging, I found that OpenSSL 1.1.1i included a patch that enabled arm64 macOS compilation. The patch https://github.com/openssl/openssl/pull/12369 specifically enables a new platform code: darwin64-arm64-cc.
The vcpkg codebase grabbed this update recently, with patch https://github.com/microsoft/vcpkg/pull/15298, but even with this patch in place, the build was failing with the error above.
I grabbed OpenSSL from its source, and started poking around. A lot of that poking and learning the innards of how OpenSSL does its builds wasn’t entirely useful, so I won’t detail all the dead ends I attempted to follow. What I did learn, in the end, was that the terminal – being native arm64 or rosetta2 emulated x86_64 – can make a huge difference. For convince, I made a copy of iTerm and ran it under rosetta so that I could easily install homebrew and use all those various tools, even though it wasn’t arm64 native.
What I found is that if I want to make a macOS native version of OpenSSL, I need to run the compilation from a native arm64 terminal session. Something is inferring the target platform – not sure what – from the shell in which it runs. I manually configured and compiled OpenSSL with an arm64 native terminal, and was able to get an arm64 library, with the internal markers for macOS.
From there, I thought that perhaps this was an issue of how OpenSSL was configured and built. That’s something that vcpkg controls, with these little snippets of cmake under the covers. vcpkg does a nice job of keeping logs, so I went through the compilation for openssl (using --triplet arm64-osx in case you want to follow along), and grabbed all the configuration and compilation steps. I grabbed those logs and put them into their own text file, and edited them so I could run them step by step in my own terminal window and see the status of the output as it went. I then adapted the steps to reference a clean git repository checkout from openssl and the tag OpenSSL_1_1_1i, and updated the prefix to a separate directory (/Users/heckj/openssl_arm64-osx/debug) so I could inspect things without stepping into the vcpkg spaces.
cd /Users/heckj/src/openssl
git reset --hard OpenSSL_1_1_1i
export CC=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc
export AR=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ar
export LD=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld
export RANLIB=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib
export MAKE=/usr/bin/make
export MAKEDEPPROG=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc
export PATH=$PATH:/Users/heckj/bin/vcpkg/downloads/tools/ninja-1.10.1-osx
/usr/bin/perl Configure no-shared enable-static-engine no-zlib no-ssl2 no-idea no-bf no-cast no-seed no-md2 no-tests darwin64-arm64-cc --prefix=/Users/heckj/openssl_arm64-osx/debug --openssldir=/etc/ssl -fPIC "--sysroot=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.1.sdk"
/usr/local/Cellar/cmake/3.19.3/bin/cmake -DDIR=/Users/heckj/src/openssl -P /Users/heckj/bin/vcpkg/ports/openssl/unix/remove-deps.cmake
# [2/3]
cd /Users/heckj/src/openssl
export PATH=$PATH:/Users/heckj/bin/vcpkg/downloads/tools/ninja-1.10.1-osx
/usr/local/Cellar/cmake/3.19.3/bin/cmake -E touch /Users/heckj/src/openssl/krb5.h
/usr/bin/make build_libsWhat I found was the if I ran this code under a terminal running under rosetta, I’d get the results that indicated the code was built against an iOS platform. And when I ran it under a native
This is a major insight, but I haven’t yet figured out how to apply it…arm64terminal, it would correctly report macOS as the platform.
I originally installed and compiled vcpkg using the rosetta terminal, and it was running as an x86_64 binary, so I thought perhaps that was the issue. Unfortunately, not. After I installed vcpkg with the arm64 native (and verified the binary was arm64 with the lipo -info command), I made another run at installing openssl, but ended up with the same iOS linked binary.
Prior to getting this far, I opened issue 13854 as a question on the OpenSSL repository, which details some of this story. However, I now longer think that’s an issue, as I was able to get an arm64 native binary when I manually compiled things. There might be something OpenSSL could do to make this easier/better, but its build setup is incredibly complex and I get lost pretty darn quickly within it.
So to date, I’ve trailed this back to some interaction that vcpkg, and x86_64 emulated binaries, are having on the build – but that’s it.
The story ends here, as I don’t have a solution. I have filed issue 15741 with the vcpkg project, with a summary of these details.
For anyone reading until the bitter end, I’d love any suggestions on how to fully resolve this. I hope that someone stumbles across the issue at some point with more knowledge than I and has a solution in the future. In the meantime, this blog post will hopefully record the error and how you can diagnose the architecture that a library is compiled for, even if it doesn’t solve the end problem of getting you to a final resolution.
Rhonabwy 