The WebRTC Bitcode Soap Opera (Saúl Ibarra Corretgé)

WebRTC has made its way into nearly every kind of device platform there is out there. No stranger to new device types, Apple now requires the use of bitcode, its intermediary format, for iOS and tvOS apps. As he shows in the engrossing drama below, Saúl Ibarra Corretgé found out the hard way that getting WebRTC to build with bitcode so it can run on these devices isn’t so easy.

Saúl is a long-time open source contributor, a frequent speaker at RTC-related events, and a respected WebRTC expert. He is one of the core contributors to Jitsi, react-native-webrtc, and libuv among many others.

Saúl’s efforts to get WebRTC building with bitcode turned into a daily soap opera. The story – only available on webrtcHacks – gives some background on bitcode, walks through numerous bugs, and finishes with a working build you can reference. Browse for the learnings, stay for the drama!

{“editor”, “chad hart“}

For the past 5 years, I’ve been making the WebRTC builds as part of maintaining the React Native WebRTC library (and prior to that I was building them for the Cordova plugin for a few years). It usually goes smoothly, but every once in a while there is some mystery wrapped in an enigma which makes the builds interesting. Today I’d like to share one of those stories that is happening right now: building WebRTC with bitcode.

What is bitcode?

First, let’s talk about bitcode. What is it and why do you want/need it?

Apple introduced bitcode together with iOS9, in 2015. It became a requirement for watchOS and tvOS applications. Why does Apple require this? All apps require a compiler to convert code into machine code that can be used on a given device. Compilers are always getting better, but to take advantage of compiler improvements you need to recompile. If you wanted to take advantage of Apple’s latest improvements, you would need to recompile and reload your app all the time and many developers wouldn’t do this in a timely fashion. Having Apple load and compile all your code isn’t practical for many reasons, so they needed an Intermediate Representation (IR) – something that the developer considers complete but that Apple can recompile. Bitcode is that IR.

In a nutshell, when building an application with bitcode we are also embedding the intermediate representation (IR) in the binary. Apple can then use this IR to generate the actual machine code for the target device. This is how Apple managed to switch from a 32bit processor in the first few Apple Watch series (0-3) to a 64bit one starting with series 4, without requiring developers to recompile their apps. It’s a pretty clever idea.

Not everybody is on board with it though. Some people have concerns about bitcode’s security since the binaries that arrive at the user’s devices are no longer the exact ones the developers built.

Bitcode and WebRTC

Our project – Jitsi Meet – has a small feature that lets you manage meetings with your Apple Watch. Unfortunately, if your project has a watchOS or tvOS target, the entire project must contain bitcode, there is no way around it. So here we are, we need to build WebRTC with bitcode.

Drama time

Previously on WebRTC bitcode…

Building WebRTC bitcode was possible and worked just fine until Chromium started a change in their compiler toolchain. Let’s take another tangent to talk about that 🙂

Google has embraced the clang / LLVM toolchain for building Chrome on pretty much any platform now. There was one holdout, however, WebRTC itself. Since WebRTC uses the Chromium compilation machinery there was a flag in the Chromium build files that allowed building WebRTC (not the entire Chromium mind you) on iOS using the system Xcode instead of the Chromium toolchain’s clang. This was necessary for bitcode support.

Once clang gained bitcode support and since one can use a framework built with clang and link it in an app built with Xcode just fine this flag no longer needed to exist and it was axed in several commits, linking back to this bug.

November 2021. This is where our soap opera episode begins.

A dubious introduction

The very first problem was is that due to a compiler flag conflict the build will error out. Oops – we have a Chromium bug.

Regular WebRTC contributor Daniel Lee pointed me at a CL (CL=Change List, a request to change Chromium) fixes the compilation, but alas hasn’t landed yet (as of April 2022).

Very suspicious

So, feeling like a happy camper I fired up the compilers and got a build! Except that the size was way too small to contain bitcode. Bitcode builds are huge, over 1GB. If you search on the Internet on how to tell if a binary has bitcode or not it’s likely all the answers are wrong. They focus on checking if there is an LLVM segment that would indicate the presence of bitcode. The problem is, in our case the section is there, but empty:

Oops – Chromium at it again, another bug.

After lots of head-scratching, I saw enlightenment. I was way out of my depth there, so my ignorance likely helped. See, we really have 2 things going on here: a compiler and a linker. We do pass the right bitcode flags to the compiler, and clang does support bitcode… So what if it’s the linker? I recompiled the framework with the use_lld=false GN flag set and… bingo! We got ourselves a build with bitcode!

I quickly filed a new bug and submitted a CL with a fix for the upstream iOS build script. It felt great, I felt like a wizard. Of course, Daniel Lee pointed me to the LLVM issue which explained what happened here: lld (LLVM’s linker) does not know (yet) how to add a bundle section with bitcode, it just adds placeholders!

With the sweet taste of victory still fresh in my mouth I released react-native-webrtc 1.98.0 and submitted our build for apple review.

Binary insanity

Unfortunately, people started to get their apps rejected after updating to this new version. Oops – another issue. This has happened before and it was usually a problem in user code so I dismissed it. Until it happened to us. Oops.

I was still in denial and opened a support case with Apple but didn’t get any good answers. I was convinced it was a false positive. I was wrong.

After receiving a bunch of binary rejections from Apple my eyes caught something in the rejection email that had gone unnoticed until then:

ITMS-90338: Non-public API usage – The app references non-public selectors in Frameworks/JitsiMeetSDK.framework/JitsiMeetSDK: addStream:, initWithBuffer:rotation:timeStampNs:, initWithURLStrings:, isBinary, isPassthrough, localDescription, onSuccess:, removeValuesForKeys:completion:, sdp, sendData:, setCategoryOptions:, setChannelId:, setConfiguration:error:, showLoading, streamId, stringForType:, typeForString:, videoSource,The app contains one or more corrupted binaries. Rebuild the app and resubmit.. If method names in your source code match the private Apple APIs listed above, altering your method names will help prevent this app from being flagged in future submissions. In addition, note that one or more of the above APIs may be located in a static library that was included with your app. If so, they must be removed. For further information, visit the Technical Support Information at http://developer.apple.com/support/technical/

Corrupted binaries? The only binary that had changed was WebRTC so it must be that…

Sweet, sweet victory

I still felt defeated, but I had a lead! My gut told me the only thing (to my knowledge) that could explain that sentence was the toolchain change that had happened in November. I didn’t know why or how, but I suspected that was it. I haphazardly reverted that CL locally, banged my head against the desk until it compiled, then submitted a build to TestFlight, and… success! The build passed where it failed before!

The sweet taste of victory was back! I quickly filed a new issue with all the details so I didn’t forget. Daniel Lee once again, found an explanation: The LLVM bitcode is backward compatible, but not forward compatible. As a result, Apple’s servers can reject bitcodes from newer LLVMs than Apple’s. This checks out because the Chromium toolchain is usually very close to the “tip of tree” (ToT). I’m no expert in the subject matter, but I’m going to call bingo here.

I’m looking forward to helping get this fixed upstream, and if you need a WebRTC build with bitcode in the meantime, we got you.

{“author”: “Saúl Ibarra Corretgé” }