This is the second post in my series on my experiences at BMW Car IT GmbH. If you’ve missed the first post, you can go back and read it here.
In this post, I’m covering missing binary compatibility and its effects on development processes.
The road to a recompile is paved with ABI incompatibilities
To exchange data between processes in its infotainment systems, BMW uses CommonAPI. CommonAPI was originally developed by the GENIVI alliance, which rebranded itself as the Connected Vehicle Systems Alliance COVESA in 2021.
CommonAPI allows specification of interfaces in the Franca Interface Definition Language FIDL independent of the message bus used to send and receive messages. It achieves this by generating C++ bindings from the interface description. These bindings then use a transport backend (CommonAPI supports DBus and Some/IP) that’s also compiled from generated code.
The CommonAPI project does not provide any guarantees on the stability of the application binary interface of its generated code. That means that programs compiled against header files generated with an older version of CommonAPI’s generators can not use bindings and transport backend libraries generated by a newer version. Because CommonAPI’s generated code heavily uses C++ templates, this meant that for every change in CommonAPI, recompilation of all custom software was necessary.
Recompiling on its own would already be a four to six hour ordeal on the 16 core 40 GiB RAM virtual machines used by the CI system. Caching reduced these build times for normal builds, but when a component as low-level as the inter-process communication framework changes, the cache would not contain any of the required results. To make matters worse, for intellectual property and build time reasons, some development teams delivered precompiled binaries, which would require a rebuild by somebody with access to the source code against the new CommonAPI version. This ended up being a major coordination issue across development teams in the entire project every four to six weeks.
Contrary to the ABI stability guarantees, which CommonAPI does not give, it did guarantee that the wire protocol, that is, the messages sent via DBus or Some/IP, would not change. As a consequence, old binaries would theoretically keep working if they still had access to the old bindings and transport libraries. In practice, we could not use this, because the filenames of the bindings and transport libraries would be the same when generated with the old and the new version. Statically linking could have solved this, but disk space was precious and static binaries tend to be larger than their dynamically linked counterparts, so we discarded this approach.
How does the open source world do it?
This problem is not new, and our solution of synchronizing a merge point across the entire project is only an option if there is a central entity to coordinate that. My colleagues and I looked to the open source world for a solution, where such a central entity does not exist.
Whenever the ABI of a library changes, open source projects increase a version number in the SONAME field of the library. By convention, the filename of a library contains the SONAME, so a changed SONAME also resolves the filename conflict we were facing. This gave maintainers of binary components the chance to recompile against the new updated CommonAPI libraries in their own time as long as we continued shipping the old libraries.
With symbolic links to pkg-config files and a bit of magic in CMake files which we generated for the CommonAPI interface libraries, this process became almost transparent for developers: they’d get the current version from the software development kit without changing their source code.
Hey, this may actually be nice for other reasons, too!
As a positive side effect, changes in interface definitions now also caused the library SONAME to change, so teams could write code that worked with both the old and the new versions of interfaces during a transition period, which allowed them to decouple their development process from the large “big bang” synchronization points that everybody else had to respect when CommonAPI interface updates arrived. Some teams even linked against both old and new versions of the interface libraries to provide both services at the same time to allow their users to update to the newer API when they were ready.
We implemented large parts of this mechanism in CMake in the build system for the CommonAPI binding and transport libraries. This build system also generated the required pkg-config .pc files and CMake config files for the libraries it built, with the necessary version information and the magic to make this transparent for users. This work has not been open sourced yet—a recurring theme for some of the work I did. I don’t think there were any business concerns against doing so. It’s rather a matter of putting the work in to do it, and I regret not prioritizing that.