When hand-written bindings drift

In the previous exercise you wrote an extern "C" block binding to C code. It worked great, but can you imagine yourself doing that for hundreds, maybe thousands, of functions and types? Of course not! There’s a subtler problem, too. Hand-written bindings will inevitably drift out of sync with the C header, especially at scale.

Neither the C nor Rust compiler can detect this because each operate in their own small universe called a compilation unit. A compilation unit is the single chunk of work that flows through the various stages of the compiler. In C/C++ this would typically be a .c/.cpp file and in Rust that is typically a single crate. The compiler will parse, typecheck, optimize the compilation unit and produce an intermediate object file. Once all the compilation units that make up your project are built, the intermediate object files are gathered and passed to the linker, which produces the final executable or library.

┌─────────────────────┐          ┌─────────────────────┐
│   Rust source       │          │   C source          │
└──────────┬──────────┘          └──────────┬──────────┘
           │ rustc                          │ cc
           ▼                                ▼
┌───────────────────────┐        ┌───────────────────────┐
│ compile               │        │ compile               │
│ (parse → typecheck →  │        │ (parse → typecheck →  │
│  optimize  → codegen) │        │  optimize  → codegen) │
└──────────┬────────────┘        └──────────┬────────────┘
           │ object file                    │ object file
           └───────────────┬────────────────┘
                           ▼
                ┌─────────────────────┐
                │   link              │
                │   (ld / lld)        │
                └──────────┬──────────┘
                           ▼
                ┌─────────────────────┐
                │   final binary      │
                └─────────────────────┘

This model is great for compilation performance because we can process many of these compilation units in parallel! There is a catch, though: compilation units must not share information since that would destroy our ability to process them in parallel! Each compilation must be its own self-contained universe.

And even if we had a mechanism to share information between compilation units, we would need to make that language agnostic so that a C compilation unit and a Rust compilation unit can interoperate. How would that even work with wildly different type systems? ¹

So compilers have resorted to manual escape hatches like the extern "C" block you wrote. You as the programmer promise to the compiler that a function with given name and given signature will exist at link-time and the compiler takes your word for it.

Possible Consequences

The consequences of drift between the two versions can be dire: If C expects int but Rust expects float the integer bit patterns are being reinterpreted as floats. The result is almost always nonsensical.

It can be worse though: passing too many or too few arguments can result in either overwriting important information on the stack or reading garbage from the stack!

Here is an especially insidious case; the function as we have established is the following:

int bm_add(int a, int b);

but now in the Rust code, we expect it to return 64-bit integers instead of 32-bit integers:

unsafe extern "C" {
    fn bm_add(a: c_longlong, b: c_longlong) -> c_longlong;
}

How will this example fail? If you try this yourself you will notice: It doesn’t! The reason is that modern CPUs pass function arguments in registers and these registers are always word-sized, meaning 64-bit on 64-bit machines. This means that internally even 32-bit ints are passed as 64-bit integers instead.

You can see this in Compiler Explorer here: The correct version passes a and b in the a0 and a1 registers (the li a0, 1 and li a1, 2 lines do the loading) and passes the return value in the a0 register. Those registers are 64-bit registers though and so you can see the incorrect c_longlong-expecting version just works because the registers were 64-bit anyway.

Now, if your 64-bit numbers stay below the 32-bit max value, everything just happens to work. But it is very fragile. As you can see here, when we compile to 32-bit target instead, everything breaks and we end up adding 0 to a instead.

I find this example particularly scary, because for 99% of inputs and deployment configurations this mistake will be virtually consequence-free. The addition will continue to work as expected. But as soon as the input is unusual, the deployment target is different, or you add code that will make the compiler change the generated code even a bit this will be a bug that takes you weeks to troubleshoot in the worst case.

Head to the exercise

Head to the exercise and play with the different “drifted” C implementations of bm_add, see what happens on your machine. Feel free to also play around a bit with the Compiler Explorer playgrounds to see how different miscompilations manifest in the generated assembly.

Exercise

The exercise for this section is located in 01_intro/02_drift

1. Yes the LLVM bitcode embedded by toolchains for LTO (Rust’s -Clto=thin -Cembed-bitcode=yes and Clang’s -flto=thin) does carry the information to catch problems like this at link-time and it is cross-language, but linkers do not generally validate it (the wasm-ld linker does, but only for Wasm!). The reasons are manifold but most importantly because it would break existing compiler optimizations. You could write custom LLVM-bitcode parsing tooling to check this if you wanted. ↩