Lab 2: Seccomp Filters

Prerequisite reading: Security — Intensive (boundary B and the four-layer model), Seccomp Filtering (deep dive) (the three categories, the SyscallRule model, the build-time bake-in), and Lab 1: The jailer in depth (the process sandbox seccomp complements).

Background

The jailer confines the VMM at the process level: filesystem, namespaces, privileges. Seccomp-BPF confines it at the kernel-interface level. A Berkeley Packet Filter program, installed on every thread, inspects each syscall the VMM attempts and allows it, denies it, or kills the thread. This is the last line of defense in boundary B: a guest that finds a device-emulation bug and turns it into arbitrary VMM code execution still cannot execve, cannot ptrace, cannot open an arbitrary file — because those syscalls are not on the allow-list, and the kernel kills the offending thread the instant it tries.

This lab is trace-it and build-it. You will read the shipped filters in resources/seccomp/<arch>.json — the three thread categories, the SyscallRule model with its argument operators — and understand how JSON becomes BPF baked into the binary at build time. Then you will make it real and visible: compile a filter with seccompiler-bin, run Firecracker with a custom --seccomp-filter that denies a syscall it actually needs, and watch the VMM die with SIGSYS — observed three ways (the FC log, strace, and dmesg/auditd). You will run with --no-seccomp and reason precisely about why that is dangerous. Finally you will add a syscall to a filter and rebuild, the exact workflow you follow when you extend the device model.

Why this matters for contributors

• The single most common stumbling block when extending Firecracker is "I added a code path that makes a new syscall, and the VMM dies with SIGSYS the first time it runs." After this lab that failure is instantly legible and you know the one-line fix.
• Seccomp PRs are security-critical and heavily reviewed. A filter that is one syscall too loose is a hole; one too tight is a crash. You must be able to read a SyscallRule — including the args conditions that pin ioctl to specific request numbers — and judge whether it is exactly as tight as it should be.
• The seccompiler crate is also a real, separately-shipped rust-vmm component (rust-vmm seccompiler) — contributing to it, or to Firecracker's filters, is a credible specialization.

Prerequisites

• Completed Lab 1 and read the seccomp deep dive.
• A Firecracker build, plus the ability to boot a microVM by hand (Level 1, Lab 1.3).
• strace, and ideally auditd/ausearch or access to dmesg, on a disposable host you own.

ARCH=$(uname -m)
TARGET=build/cargo_target/${ARCH}-unknown-linux-musl/release
ls $TARGET/firecracker
# The seccomp JSON source for your arch and the compiler binary:
find resources/seccomp -name "*.json"
ls $TARGET/seccompiler-bin 2>/dev/null || cargo build --release -p seccompiler

Step-by-step tasks

Step 1 — Read the three filter categories

Firecracker ships three filters, not one — a per-thread-class filter is tighter than a single global one, because each thread class makes a different set of syscalls.

ARCH=$(uname -m)
JSON=$(find resources/seccomp -name "${ARCH}*.json" | head -1)
echo "filter file: $JSON"

# The three top-level categories and each one's default action.
rg -n '"vmm"|"api"|"vcpu"|default_action|filter_action' "$JSON"

# Pretty-print the vcpu filter (the tightest) and count its syscalls.
python3 -c "import json,sys; d=json.load(open('$JSON')); print(json.dumps(d['vcpu'], indent=2))" | head -60
rg -n '"syscall"' "$JSON" | wc -l

The JSON top level is a map from category name → a filter object with a default_action (what to do for unlisted syscalls — typically trap or kill_thread) and a filter array of per-syscall rules.

• State which category gets the tightest filter and why a per-thread filter beats one global filter. (The vCPU thread makes the fewest syscalls, so its allow-list is shortest.)

Step 2 — Read the SyscallRule model and the argument operators

A rule is not "allow syscall N." Seccomp-BPF can inspect a syscall's register arguments, and Firecracker uses this to allow ioctl only with specific request numbers — KVM_RUN, KVM_SET_USER_MEMORY_REGION, the TUN ioctls — and nothing else. Opening the whole ioctl multiplexer would defeat the point.

# Find an ioctl rule with an args condition (the KVM_RUN / KVM restriction).
rg -n -A8 '"syscall": "ioctl"' "$JSON" | head -50

# The operator/condition types in the compiler.
rg -n "SyscallRule|SeccompRule|SeccompCondition|enum .*Op|masked_eq|MaskedEq" src/seccompiler/src/

A rule with an argument condition looks like:

{
  "syscall": "ioctl",
  "args": [
    { "index": 1, "type": "dword", "op": "eq", "val": 44672 }
  ]
}

index is the argument position (0–5), type its width, op the comparison, val the value.

A syscall entry with no args is allowed unconditionally. Multiple args on one rule are AND-ed; multiple rules for the same syscall are OR-ed. That is how ioctl is pinned to a small set of request codes.

• Find one ioctl rule whose val you can identify (e.g. cross-reference against KVM_RUN's ioctl number for your arch). Explain in your notes what would happen to the filter's safety if you deleted its args array.

Step 3 — Compile the filters with seccompiler-bin

The shipped binary already has the filters baked in (Step 7 covers that), but to experiment you compile JSON → BPF yourself and load it with --seccomp-filter.

ARCH=$(uname -m)
TARGET=$(pwd)/build/cargo_target/${ARCH}-unknown-linux-musl/release
JSON=$(find resources/seccomp -name "${ARCH}*.json" | head -1)

# Compile the shipped JSON to a BPF blob.
$TARGET/seccompiler-bin \
  --target-arch ${ARCH} \
  --input-file "$JSON" \
  --output-file /tmp/seccomp.bpf

ls -l /tmp/seccomp.bpf   # a binary BPF blob, one program per category

# Run Firecracker with the EXPLICIT compiled filter (functionally identical to the default).
sudo $TARGET/firecracker --api-sock /tmp/fc.sock --seccomp-filter /tmp/seccomp.bpf &
# Drive the API to boot a microVM (Lab 1.3 shape) — it should work normally.

Note: --seccomp-filter <path> loads a custom compiled blob instead of the built-in one; seccompiler-bin is the compiler that produces it. The blob must match your architecture — a filter compiled for the wrong arch is rejected at startup, because syscall numbers differ. Verify the exact flag names with --help; CLI surface drifts (verify on your branch).

Step 4 — Deny a syscall on purpose and watch the VMM die with SIGSYS

Now make the boundary visible. Take a syscall the VMM genuinely needs and remove it (or its ioctl arg) from a copy of the JSON, recompile, and run. The VMM will die with SIGSYS the instant it hits that path.

ARCH=$(uname -m)
JSON=$(find resources/seccomp -name "${ARCH}*.json" | head -1)
cp "$JSON" /tmp/broken.json

# Delete an ioctl the VMM needs — e.g. narrow the ioctl rule so KVM_RUN is no longer allowed,
# OR remove a whole-syscall allow like 'epoll_wait' / 'read' from the "vmm" category.
# (Use jq or an editor. Example: drop the FIRST ioctl rule from the vmm filter.)
python3 - "$JSON" /tmp/broken.json <<'PY'
import json, sys
src, dst = sys.argv[1], sys.argv[2]
d = json.load(open(src))
# Remove every ioctl rule from the vmm category to guarantee a denial on the KVM path.
d["vmm"]["filter"] = [r for r in d["vmm"]["filter"] if r.get("syscall") != "ioctl"]
json.dump(d, open(dst, "w"), indent=2)
print("wrote broken filter:", dst)
PY

TARGET=$(pwd)/build/cargo_target/$(uname -m)-unknown-linux-musl/release
$TARGET/seccompiler-bin --target-arch ${ARCH} --input-file /tmp/broken.json --output-file /tmp/broken.bpf

# Run with the broken filter and try to boot. The VMM will SIGSYS as soon as it issues a KVM ioctl.
sudo $TARGET/firecracker --api-sock /tmp/fc.sock --seccomp-filter /tmp/broken.bpf 2>/tmp/fc.stderr &
# ...drive the boot sequence; the process dies almost immediately.

Observe the death three ways:

(a) Firecracker's own log. Firecracker installs a SIGSYS handler that logs the offending syscall number and the faulting thread before exiting:

tail -n 20 /tmp/fc.stderr      # or the configured log file inside the jail
# Look for: "Shutting down because of bad syscall N" / SIGSYS / seccomp violation.
rg -n "SIGSYS|si_syscall|bad syscall|seccomp" src/vmm/src/signal_handler.rs

(b) strace. Run the same broken config under strace and watch the kill arrive:

sudo strace -f -e trace=ioctl,read,write \
  $TARGET/firecracker --api-sock /tmp/fc.sock --seccomp-filter /tmp/broken.bpf 2>&1 | tail -30
# You'll see the syscall that triggers it, then: +++ killed by SIGSYS +++

(c) dmesg / auditd. The kernel logs seccomp violations; with auditd, ausearch shows the exact syscall:

sudo dmesg | grep -i "seccomp\|audit" | tail
# Or, if auditd is running:
sudo ausearch -m SECCOMP -ts recent 2>/dev/null | tail -20
# Each record names the pid, the syscall number, and the action (SECCOMP_RET_TRAP/KILL).

The full chain, signal by signal:

VMM issues a denied syscall (e.g. ioctl(KVM_RUN))
        │
        ▼  BPF: syscall not in allow-list → SECCOMP_RET_TRAP
        ▼  kernel delivers SIGSYS to that thread
        ▼  Firecracker's SIGSYS handler logs "bad syscall N" → process exits
        ▼  the host is never compromised; the microVM is gone (fail-closed)

• Capture all three views (FC log, strace, dmesg/auditd) and note that they agree on the same syscall number. This is the payoff of the whole design: even total VMM code execution dead-ends here.

Step 5 — Run with --no-seccomp, and reason about why it's dangerous

To feel the contrast, run the same broken-filter scenario but with the filter disabled entirely:

# With --no-seccomp there is NO filter at all — the denied syscall now succeeds.
sudo $TARGET/firecracker --api-sock /tmp/fc.sock --no-seccomp &
# The microVM boots fine. That's the danger: every syscall is allowed.

Warning: --no-seccomp removes the entire syscall sandbox. With it, a guest that achieves VMM code execution can execve, ptrace, open arbitrary host files — the last boundary-B defense is gone. It exists because seccomp traps ptrace/process_vm_* and so breaks debuggers; use it for a gdb session and never, ever in production. The maintainers and docs/prod-host-setup.md reject any production config that disables seccomp. This is the security analogue of running without the jailer from Lab 1: convenient for dev, catastrophic in prod.

• Articulate, in one sentence each: what --no-seccomp removes, what attack it re-enables, and the one legitimate use (debugging).

Step 6 — Add a syscall to a filter and rebuild

This is the real contributor workflow. Suppose you add a code path that legitimately needs a syscall not currently allowed — the fix is to add it to the JSON and rebuild. Practice it: add an innocuous syscall to the vmm category, recompile, and confirm it loads.

ARCH=$(uname -m)
JSON=$(find resources/seccomp -name "${ARCH}*.json" | head -1)
cp "$JSON" /tmp/extended.json

# Add a new whole-syscall allow to the vmm filter (example: getrandom, by NAME not number).
python3 - "$JSON" /tmp/extended.json <<'PY'
import json, sys
d = json.load(open(sys.argv[1]))
names = {r.get("syscall") for r in d["vmm"]["filter"]}
if "getrandom" not in names:
    d["vmm"]["filter"].append({"syscall": "getrandom"})
json.dump(d, open(sys.argv[2], "w"), indent=2)
print("added getrandom to vmm filter")
PY

TARGET=$(pwd)/build/cargo_target/${ARCH}-unknown-linux-musl/release
$TARGET/seccompiler-bin --target-arch ${ARCH} --input-file /tmp/extended.json --output-file /tmp/extended.bpf
sudo $TARGET/firecracker --api-sock /tmp/fc.sock --seccomp-filter /tmp/extended.bpf &
# Boots normally; the new syscall is now permitted on the vmm thread.

Warning: Add the syscall by name, never by number. Syscall numbers differ between x86_64 and aarch64, and there is a separate JSON file per architecture — if you add a number for one arch you silently break the other. This is why CI runs the filter tests on both arches.

• In a real PR you would edit both arch JSON files, add the syscall with the tightest args condition you can justify, and add an integration test that exercises the new path under the default (baked-in) filter so the suite catches a missing allow. Note what you'd change.

Step 7 — Understand the build-time bake-in

In the default configuration there is no JSON to ship, parse, or tamper with at runtime: the seccompiler runs at build time and embeds the compiled BPF into the firecracker binary. Each thread loads its category as it starts.

# The compiler crate (donated upstream to rust-vmm, maintained in-tree).
find src/seccompiler -name "*.rs" | sort
rg -n "fn compile|to_bpf|BpfProgram|sock_filter|SECCOMP_RET|fn main" src/seccompiler/src/

# Where the build wires the JSON into the binary (build script / include).
rg -n "seccomp|seccompiler|include_bytes|build.rs|\.json" src/vmm/build.rs src/firecracker/build.rs 2>/dev/null
rg -n "seccomp" $(find src -name build.rs)

# Where Firecracker loads the embedded filters per thread at runtime.
rg -n "install_filter|apply_filter|get_filters|BpfThreadMap|SeccompFilter|seccomp" src/vmm/src/seccomp.rs

flowchart LR
    JSON["resources/seccomp/<arch>.json"] --> SC["seccompiler (build time)"]
    SC --> BPF["BPF program per category"]
    BPF --> BIN["baked into the firecracker binary"]
    BIN --> Load["each thread: install_filter() on start"]
    Load --> Kernel["kernel attaches BPF to the thread"]
    Kernel --> Check["every syscall checked against the filter"]

This is why, when you add a syscall to the JSON, you must rebuild — the running binary carries its own copy of the filter. --seccomp-filter (Steps 3–6) is the override that lets you load a different blob without rebuilding, which is exactly why it is handy for experimentation but is not how production runs.

• Confirm from the build script (rg -n "seccomp" $(find src -name build.rs)) that the JSON is consumed at compile time, and explain why that means a tampered runtime JSON cannot weaken a default-configured Firecracker.

Deliverables

• The three category names and the per-arch JSON file path, with a one-line statement of why the vcpu filter is tightest.
• One ioctl rule quoted from the JSON, with its args condition explained, plus what removing the args would do to safety.
• A compiled /tmp/seccomp.bpf from seccompiler-bin, and a note that running with it behaves like the default.
• The SIGSYS death captured three ways (FC log, strace, dmesg/auditd), all agreeing on the same syscall number.
• A one-sentence-each statement of what --no-seccomp removes, what it re-enables, and its only legitimate use.
• An extended filter (getrandom added) compiled and loaded, plus the note about editing both arch files and adding by name not number.
• Evidence from build.rs that the filter is baked in at build time.

Troubleshooting

seccompiler-bin not found

Build it: cargo build --release -p seccompiler (or tools/devtool build --release then look in build/cargo_target/<arch>-unknown-linux-musl/release/). The crate is seccompiler in the workspace; the binary is seccompiler-bin. Confirm flag names with --help — they drift (verify on your branch).

Custom filter rejected at startup

The compiled blob's architecture must match the running binary's. Recompile with --target-arch $(uname -m). A blob built for x86_64 will not load on aarch64 and vice versa — syscall numbers differ.

The VMM dies with SIGSYS and you didn't intend to deny anything

You removed or narrowed a syscall the VMM needs. Read the logged syscall number, find it in the JSON, and restore the allow (or its args value). This is exactly the failure new device code produces when you forget to extend the filter — the intended lesson of Step 4.

strace/gdb itself triggers SIGSYS

The default filter traps ptrace/process_vm_*, which strace and gdb use. For a debugging session, run with --no-seccomp (dev box only) so the debugger can attach — this is the one legitimate use of the flag.

Tests pass on your arch, fail in CI on the other

You added a syscall to only one arch's JSON, or hard-coded a syscall number. Edit both resources/seccomp/<arch>.json files and use syscall names.

Expected output

• rg '"vmm"|"api"|"vcpu"' lists exactly three categories in the per-arch JSON.
• Running with a faithfully compiled filter boots a microVM normally; running with the broken filter kills Firecracker with SIGSYS immediately on the first denied syscall, visible identically in the FC log, strace output, and the kernel audit/dmesg trail.
• --no-seccomp boots the same broken scenario fine — demonstrating, by contrast, exactly what the filter was stopping.

Stretch goals

1. Diff the three filters. Extract each category's syscall list and compute the set differences: what does vmm allow that vcpu does not, and vice versa? Explain each difference in terms of what that thread does (e.g. only vmm needs TAP read/write; only api needs socket syscalls).
2. Tighten an existing rule. Find a syscall currently allowed unconditionally that could carry an args condition (e.g. mmap flags via masked_eq) and propose the tighter rule. Reason about whether it would break a legitimate path.
3. Reproduce the real contributor failure. Add a tiny code path to the VMM that makes a syscall not in the filter (e.g. getrandom before you add it), build without changing the JSON, and confirm the SIGSYS. Then fix it by adding the allow. This is the entire "I added a device and it dies" debugging loop, compressed.
4. Read the seccompiler BPF emission. Trace to_bpf / sock_filter generation in src/seccompiler/src/ and sketch how one SyscallRule with an args condition becomes a sequence of BPF instructions (load arg, compare, jump). See rust-vmm seccompiler.

Validation / self-check

Answer without notes; these gate completion.

• Name the three filter categories, the thread each protects, and why a per-thread filter is tighter than one global filter.
• Describe the SyscallRule model: syscall name, args conditions, the operator set, and how multiple conditions and multiple rules combine (AND vs OR).
• Take an ioctl rule with an args condition: what does it restrict, and why is restricting ioctl specifically so important for a VMM?
• Trace, signal by signal, what happens from a denied syscall to process exit — name the BPF return action and the signal, and the three places you can observe it.
• Explain when JSON becomes BPF (build time vs runtime), where the BPF lives in the default config, and why that makes a tampered runtime JSON harmless to a default build.
• State what --no-seccomp and --seccomp-filter each do and exactly when (if ever) each is appropriate.
• You add a device that calls a new syscall and it dies on first run. What did you forget, what is the precise fix, and what must you do for both architectures?

Next: Lab 3: A threat-model audit — step back from the individual mechanisms and audit the whole attack surface, anchored on a real virtio-PCI vulnerability (CVE-2026-5747), thinking like both attacker and defender.