Rewriting Every Syscall in a Linux Binary at Load Time

Rewriting Every Syscall in a Linux Binary at Load Time

The problem that started this

There’s something odd about the way we run software today. Most containers — the dominant unit of deployment in production — run a single process. One Python script, one Node.js server, one Go binary. But that single process sits on top of a full Linux kernel — roughly 450 system calls, most of which it will never use. The kernel knows about devices, schedulers, multi-process coordination, signal routing, dozens of filesystem types, and hundreds of other things that a single-process workload doesn’t care about. The process doesn’t care how the machine is laid out or what hardware is available. It needs CPU, memory, and I/O. That’s it.

Think about what this gap means. We’re running on a platform that provides a vast surface of features these workloads will never touch. And increasingly, the code running inside these containers is code that isn’t fully trusted — third-party libraries, generated code, autonomous agents making decisions at runtime. That 450-syscall interface becomes difficult to reason about and even harder to secure.

This is not a unique observation, and it’s not the first time people have tried to address it. Stripping a kernel of features you don’t need is an old idea — sometimes motivated by security, sometimes by resource constraints. It’s been common in the embedded world for decades, where you can’t afford a full kernel on a device with 256KB of RAM. In the server world, the same instinct shows up as hardened kernel configs with hundreds of options disabled, custom builds with subsystems removed, and unikernels that compile app and kernel together. And people have shown real success with this approach — substantial gains in memory footprint and measurably better security postures. But they still always end up with more than they want, because of how deeply things are entangled in the kernel. Subsystems have deep interdependencies. Removing the scheduler breaks assumptions in the memory manager. Disabling networking pulls threads that unravel into the VFS layer. You end up with hacks — #ifdefs around code paths that might still be reachable, stub functions that return success without doing anything, and a constant fear that some corner case will hit a code path you thought you’d removed.

Unikernels have tried to address the entire kernel surface from the other direction — building up from nothing. But the problem turns out to be vast. Once you need to support processes that talk to real devices, care about hardware topology, or depend on specific OS features, the scope explodes. You end up rebuilding large chunks of what you were trying to avoid.

But what if you don’t need any of that? What if the process doesn’t care about devices, doesn’t need hardware access, and only uses a small slice of the OS interface? Instead of subtracting from a full kernel or rebuilding one from scratch, what if you started from zero and only implemented what the process actually calls?

strace a Python process — a script that reads data, makes HTTPS calls, and writes output. It uses about 40 distinct syscalls. read, write, open, close, socket, connect, send, recv, brk, mmap, clock_gettime, exit — and a couple dozen more for memory management and file metadata. The other 410 syscalls? Multi-process coordination, device management, signal handling, things a single-process workload never touches.

So implement those ~40 syscalls as a library. A “library kernel” — just the syscalls the process needs, written from scratch, no Linux baggage.

The idea isn’t new. Unikernels tried this. So did various library OS projects. But they all hit the same wall: how do you get the process to call your library instead of the real kernel?

The standard approaches are:

Compiler integration: Modify the toolchain to emit calls to your library instead of syscall instructions. This works, but now you need to support every compiler, every language runtime, every version. GCC, Clang, rustc, Go’s compiler, CPython’s build system, Node.js’s V8 — each with its own way of emitting syscalls. The maintenance surface is enormous.

LD_PRELOAD / libc interposition: Intercept at the C library level by overriding libc functions like write() and open(). But not everything goes through libc. Go makes syscalls directly. So does musl in some paths. JIT compilers emit raw syscall instructions. Anything that bypasses libc bypasses your interposition. You’re playing whack-a-mole with an ever-growing list of exceptions.

Custom libc: Build a libc that routes to your implementation. Similar problem — you need the process to link against your libc, which means controlling the build. And statically-linked binaries ignore your libc entirely.

Every approach that works at the source level, the compiler level, or the library level has the same fundamental problem: there are too many paths to a syscall instruction, and you need to cover all of them. Miss one, and the process escapes your control.

And containers aren’t secure enough as-is. A container shares the host kernel. Every one of those 450 syscalls is a potential attack surface — the process can probe them, exploit bugs in their implementation, or use them in unintended combinations. The kernel’s syscall interface is the largest attack surface in the system, and containers expose all of it.

But eventually you realize something: every one of these approaches — compiler-generated code, libc wrappers, JIT output, hand-written assembly, Go’s raw syscalls, musl’s internal paths — they all converge on the same point. Whatever language, whatever toolchain, whatever runtime, the process eventually executes the same two-byte instruction: 0F 05. The syscall opcode is the single most consistent hook point across the entire software stack. It doesn’t matter how the code got there. It always arrives at the same place.

Work at that level — below the language, below the compiler, below libc — and you only have one thing to catch.

The syscall interface is just an ABI — a contract. A process puts a number in rax, arguments in rdi through r9, and executes syscall. It gets a result back in rax. The process doesn’t care who honored that contract. If you implement those ~40 syscalls yourself — returning the same values, honoring the same error codes — the process can’t tell the difference. You control its entire view of the world. What happens with the other 410 syscalls — how you handle the ones you don’t implement, what to do when a process needs something outside your set — is a design question I’ll get into in later posts. For now, the foundational problem:

The answer: rewrite the binary at load time. Replace every syscall instruction with a trap that redirects to your own implementation.

Why not ptrace, seccomp, or eBPF?

There are established ways to intercept system calls on Linux. Each one has a limitation that matters when your goal is to enforce policy on untrusted code — not just observe it.

ptrace (strace, gdb):

The kernel stops the process, notifies the tracer, the tracer inspects and resumes. That’s two context switches per syscall — roughly 10-20µs of overhead each time. For a process making thousands of syscalls per second, ptrace adds double-digit milliseconds of delay. More fundamentally, ptrace is designed for debugging, not enforcement. The API is awkward for building a policy engine.

seccomp-bpf:

Seccomp lets you install a BPF filter that the kernel evaluates on every syscall. It’s fast — the filter runs in-kernel. But the actions are coarse: allow, kill the process, return an error, or trap to a user-space handler (via SECCOMP_RET_TRACE, which brings back ptrace overhead). You can’t inspect pointer arguments — the BPF filter only sees register values, not the memory they point to. You can’t read the filename being