Lab 9.1: Seccomp Filters and the Jailer

Lab type: Trace-it / Run-it (security model, hands-on) Estimated time: 4–6 hours

Background

Firecracker's security argument is defense in depth. The guest — including the guest kernel — is untrusted, and the VMM is privileged host code that the guest will try to exploit. A single boundary is not enough, so Firecracker stacks four that all have to hold: the KVM hardware-virtualization boundary, the jailer (chroot, cgroups, namespaces, privilege drop), a seccomp-BPF syscall whitelist installed on every thread, and Rust memory safety. This lab is where you stop reading about that model and start operating it: you will run Firecracker under the jailer from a chroot you build by hand, read the actual default seccomp filter that ships in the binary, recompile a modified one with seccompiler-bin, and watch a syscall get denied.

Two facts about the relationship between these two tools matter before you start, because almost everyone gets them backwards. First: the jailer and the seccomp filter are independent layers. The jailer is a separate binary that sets up isolation and then execs firecracker; it does not install seccomp. Firecracker installs its own seccomp filter, whether or not it was launched by the jailer. Second: seccomp is per-thread and per-category. Firecracker compiles three filters — one for the VMM thread, one for the API thread, one for vCPU threads — because those threads legitimately make different syscalls, and a tighter per-thread whitelist is a smaller surface than one union filter.

Why This Lab Matters for Contributors

  • A maintainer reviewing any PR that adds a syscall, a device, or an ioctl is asking did this widen the attack surface? You cannot answer that without having read the seccomp filter and understood the jailer's job. This lab gives you that literacy.
  • Real Firecracker bugs and CVEs live here: a missing seccomp constraint, an over-broad jailer mount, a device that exposed host state. The jailer deep dive and the seccomp deep dive explain the mechanisms; this lab makes you exercise them.
  • The production hardening guidance (docs/prod-host-setup.md) only makes sense once you've seen what the jailer does and does not do. The security masterclass — especially the jailer-deep lab and the threat-model audit — goes deeper; this lab is the on-ramp.

Prerequisites

  • Completed Level 8: you can reproduce an issue and read an execution path across the vmm/firecracker/jailer crates.
  • You can build the binaries and boot a microVM by hand (Lab 1.3).
  • You have a kernel (vmlinux-*) and a rootfs (*.ext4) from Lab 1.1.
  • You understand the three-thread model (threading deep dive).
  • Read docs/jailer.md and docs/seccomp.md in full.
# Verify your toolchain produced all four binaries this lab needs.
tools/devtool build --release
ARCH=$(uname -m)
BIN=build/cargo_target/${ARCH}-unknown-linux-musl/release
ls -l $BIN/firecracker $BIN/jailer $BIN/seccompiler-bin
# If seccompiler-bin is missing, build it explicitly:
#   cargo build --release -p seccompiler --bin seccompiler-bin

The Two Layers at a Glance

            ┌──────────────────────────────────────────────────────────────┐
  root  ──► │  jailer  (runs as root)                                       │
            │   1. parse --id --exec-file --uid --gid --chroot-base-dir ... │
            │   2. mkdir <chroot>/root ; pivot_root / chroot into it        │
            │   3. cgroups: write --cgroup / --resource-limit values        │
            │   4. namespaces: unshare(mount) ; setns(--netns) ; opt PID ns │
            │   5. mknod  root/dev/kvm   root/dev/net/tun                    │
            │   6. setgid(gid) ; setuid(uid)        ◄── PRIVILEGE DROP       │
            │   7. exec ./firecracker --id <id> [--api-sock ...]            │
            └───────────────────────────────────────────────────────────────┘
                                  │  exec, now unprivileged, inside chroot
                                  ▼
            ┌───────────────────────────────────────────────────────────────┐
unpriv ──►  │  firecracker                                                   │
            │   • installs the COMPILED-IN seccomp BPF, per thread:          │
            │       VMM thread  ← vmm filter                                 │
            │       API thread  ← api filter                                 │
            │       vCPU thread ← vcpu filter   (one per vCPU)               │
            │   • (override: --seccomp-filter <bpf>  | disable: --no-seccomp)│
            └───────────────────────────────────────────────────────────────┘

Note: Step 6 is the whole point of the jailer being a separate root-owned binary: it does the things that need root (mknod, cgroup setup, setns) and then drops to an unprivileged uid/gid before handing control to the much larger firecracker codebase. Firecracker itself never needs to run as root.

Step 1: Read the jailer's job in the source

Before you run it, see what it actually does. Locate the jailer's configuration struct and the ordered steps it runs — do not trust line numbers, run the search.

# The parsed jailer environment: every flag and the work it triggers.
rg -n "struct Env" src/jailer/src/
rg -n "pivot_root|chroot|mknod|setuid|setgid|unshare|setns|cgroup" src/jailer/src/

# The exec at the end — where it hands off to firecracker.
rg -n "exec|execv|Command::new" src/jailer/src/

Answer for yourself, from the code: in what order does the jailer drop privileges relative to mknod and the cgroup/namespace setup? (It must do the privileged operations first; once it setuids away from root it cannot mknod anymore.)

Tip: The --resource-limit and --cgroup flags map onto rlimits and cgroup files. rg -n "RLIMIT|resource_limit|cgroup_conf|cgroup_v" src/jailer/src/ shows you how the values are applied, and whether the host is cgroup v1 or v2 (the code branches on it — verify which your host uses with mount | grep cgroup).

Step 2: Build the jailer chroot and run a microVM under it

The jailer expects a chroot under <chroot-base-dir>/<exec-file-name>/<id>/root. The default --chroot-base-dir is /srv/jailer. You will assemble that tree, drop the kernel and rootfs into it (the jailer cannot reach files outside the chroot), and launch.

ARCH=$(uname -m)
BIN=$(pwd)/build/cargo_target/${ARCH}-unknown-linux-musl/release
ID=lab91
UID_FC=1234        # an unprivileged uid the microVM will run as
GID_FC=1234

# 1. The jailer derives the chroot path from the exec-file name. Stage the binary.
sudo mkdir -p /srv/jailer
sudo cp "$BIN/firecracker" /srv/jailer/firecracker   # not strictly required, but keep names stable

CHROOT=/srv/jailer/firecracker/$ID/root              # where the jailer will land us

# 2. Set up a host network tap for the guest, in a netns we will hand to the jailer.
sudo ip netns add fcnet$ID 2>/dev/null || true
sudo ip netns exec fcnet$ID ip tuntap add dev tap0 mode tap 2>/dev/null || true
sudo ip netns exec fcnet$ID ip addr add 172.16.0.1/24 dev tap0
sudo ip netns exec fcnet$ID ip link set tap0 up

# 3. Launch firecracker UNDER the jailer. The jailer creates the chroot, mknods
#    /dev/kvm and /dev/net/tun inside it, enters fcnet$ID, drops to uid/gid, execs FC.
sudo "$BIN/jailer" \
  --id "$ID" \
  --exec-file "$BIN/firecracker" \
  --uid "$UID_FC" --gid "$GID_FC" \
  --netns "/var/run/netns/fcnet$ID" \
  --cgroup "cpu.max=50000 100000" \
  -- \
  --api-sock /run/firecracker.socket &

Now confirm the jailer did its work. Everything below should be visible inside the chroot:

# The chroot was created and the device nodes were mknod'd inside it.
sudo ls -l "$CHROOT" "$CHROOT/dev/kvm" "$CHROOT/dev/net/tun"

# The firecracker process is running as the unprivileged uid, not root.
ps -o pid,uid,gid,comm -C firecracker

# The API socket is INSIDE the chroot (paths are relative to the new root).
sudo ls -l "$CHROOT/run/firecracker.socket"

To drive the API you talk to the socket inside the chroot, and you must place the kernel and rootfs inside the chroot too (hard-link or copy them in, then reference them by their in-chroot path):

# Stage guest artifacts inside the chroot so the jailed firecracker can open them.
sudo cp ./vmlinux-* "$CHROOT/vmlinux"
sudo cp ./ubuntu-*.ext4 "$CHROOT/rootfs.ext4"
sudo chown $UID_FC:$GID_FC "$CHROOT/vmlinux" "$CHROOT/rootfs.ext4"

API="$CHROOT/run/firecracker.socket"
sudo curl -X PUT --unix-socket "$API" --data \
  '{"kernel_image_path":"/vmlinux","boot_args":"console=ttyS0 reboot=k panic=1 pci=off"}' \
  http://localhost/boot-source
sudo curl -X PUT --unix-socket "$API" --data \
  '{"drive_id":"rootfs","path_on_host":"/rootfs.ext4","is_root_device":true,"is_read_only":false}' \
  http://localhost/drives/rootfs
sudo curl -X PUT --unix-socket "$API" --data \
  '{"iface_id":"net1","guest_mac":"06:00:AC:10:00:02","host_dev_name":"tap0"}' \
  http://localhost/network-interfaces/net1
sudo curl -X PUT --unix-socket "$API" --data \
  '{"action_type":"InstanceStart"}' http://localhost/actions

Warning: Paths in the API are now relative to the chroot (/vmlinux, not ./vmlinux-6.1.x). This trips up everyone the first time. If you get an ENOENT on boot, you almost certainly referenced a host path that does not exist inside the chroot.

In production you would normally also pass --daemonize and let an orchestrator manage the socket; here we keep it in the foreground so you can watch it. The jailer-in-production integration lab covers the production shape.

Step 3: Find and read the default seccomp filter

The seccomp filter is compiled into the binary at build time from JSON in resources/seccomp/. Find the file for your architecture and read its structure.

ARCH=$(uname -m)
find resources/seccomp -name '*.json'
# e.g. resources/seccomp/x86_64-unknown-linux-musl.json  (name/layout: verify on your branch)

FILTER=$(find resources/seccomp -name "*${ARCH}*.json" | head -1)

# Top-level keys are the THREAD CATEGORIES. Expect: vmm, api, vcpu.
python3 -c "import json,sys; d=json.load(open('$FILTER')); print(list(d.keys()))"

# For each category: its default_action and how many syscall rules it allows.
python3 - "$FILTER" <<'PY'
import json, sys
d = json.load(open(sys.argv[1]))
for cat, spec in d.items():
    rules = spec.get("filter", [])
    print(f"{cat:5} default={spec.get('default_action')!s:30} rules={len(rules)}")
PY

Now inspect a single rule to see the SyscallRule shape — a syscall name plus, optionally, args that constrain which invocations of that syscall are allowed:

# Show the rules that carry argument constraints (the interesting, tight ones).
python3 - "$FILTER" <<'PY'
import json, sys
d = json.load(open(sys.argv[1]))
for cat, spec in d.items():
    for r in spec.get("filter", []):
        if r.get("args"):
            print(cat, r["syscall"], "->", r["args"])
PY

A rule's args is a list of conditions, each naming an argument index, a type (e.g. dword/qword), an operator (eq, ge, gt, le, lt, ne, masked_eq), and a value. masked_eq is how flag/permission bits are constrained — for example, allowing mmap only with a specific protection mask. This is the difference between "allow ioctl" (broad) and "allow ioctl only when arg1 == KVM_RUN" (tight).

JSON fieldMeaning
default_actionWhat happens to a syscall with no matching rule (typically trap or kill_process — verify)
filter[]The list of allowed SyscallRules
syscallThe syscall name this rule governs
args[]Optional per-argument conditions; if present, all must match for the rule to apply
args[].indexWhich syscall argument (0–5)
args[].opeq / ge / gt / le / lt / ne / masked_eq
commentHuman note (why this syscall is needed) — read these; they document the threat reasoning
# See how the JSON is parsed into rules — the Rust side of the SyscallRule shape.
rg -n "SyscallRule|SeccompRule|SeccompCondition|masked_eq|SeccompCmpOp" src/seccompiler/

Step 4: Compile a filter with seccompiler-bin and install it

seccompiler-bin turns the JSON into a binary BPF blob; firecracker can load that blob at runtime with --seccomp-filter. This is exactly what the build does internally, but doing it by hand lets you ship a modified filter without rebuilding firecracker.

ARCH=$(uname -m)
BIN=build/cargo_target/${ARCH}-unknown-linux-musl/release
FILTER=$(find resources/seccomp -name "*${ARCH}*.json" | head -1)

# Compile the stock filter to BPF.
"$BIN/seccompiler-bin" \
  --target-arch "$ARCH" \
  --input-file "$FILTER" \
  --output-file /tmp/seccomp.bpf

ls -l /tmp/seccomp.bpf   # the binary filter firecracker will mmap and install

# Run firecracker with the externally-compiled filter (functionally identical to default).
sudo "$BIN/firecracker" --api-sock /tmp/fc.sock --seccomp-filter /tmp/seccomp.bpf &

Confirm firecracker is filtered. The kernel records seccomp mode in /proc:

# Seccomp 2 = filter mode is active on the threads.
for t in /proc/$(pgrep -n firecracker)/task/*; do
  printf "%s seccomp=%s\n" "$(basename $t)" "$(grep -i ^Seccomp: $t/status | awk '{print $2}')"
done

Warning: --no-seccomp disables the entire whitelist. It exists for development and for narrowing down whether a bug is a seccomp denial. It must never be used in production — without it, a compromised VMM has the host's full syscall surface. Treat any PR or runbook that reaches for --no-seccomp outside a debugging context as a red flag.

Step 5: Make a syscall get denied (and observe it)

Now prove the filter is real by removing a syscall firecracker needs and watching it die. Copy the JSON, delete a rule the running VMM relies on (a good candidate is a syscall used during normal operation but not during the earliest startup — so firecracker starts and then dies when it hits the denied call), recompile, and run.

ARCH=$(uname -m)
BIN=build/cargo_target/${ARCH}-unknown-linux-musl/release
FILTER=$(find resources/seccomp -name "*${ARCH}*.json" | head -1)

cp "$FILTER" /tmp/broken.json

# Remove ONE syscall from the vmm category to provoke a denial. Pick a call the VMM
# makes during steady-state device work (inspect the list from Step 3 and choose).
# Example: drop "timerfd_settime" or "ppoll" — verify it's present first, then remove.
python3 - <<'PY'
import json
d = json.load(open("/tmp/broken.json"))
victim = "timerfd_settime"   # change to a syscall your filter actually lists
before = [r["syscall"] for r in d["vmm"]["filter"]]
d["vmm"]["filter"] = [r for r in d["vmm"]["filter"] if r["syscall"] != victim]
after  = [r["syscall"] for r in d["vmm"]["filter"]]
print("removed:", victim, "->", len(before), "to", len(after), "rules")
json.dump(d, open("/tmp/broken.json","w"))
PY

"$BIN/seccompiler-bin" --target-arch "$ARCH" \
  --input-file /tmp/broken.json --output-file /tmp/broken.bpf

# Run under strace so you SEE the kernel deliver the seccomp action.
sudo strace -f -e trace=timerfd_settime,seccomp -o /tmp/fc.strace \
  "$BIN/firecracker" --api-sock /tmp/fc.sock --seccomp-filter /tmp/broken.bpf &

Drive a boot as in Step 2 (without the jailer is fine here) and watch it die when the VMM thread first invokes the removed syscall. Then inspect the evidence:

# strace shows the syscall and the seccomp action (SIGSYS / killed).
grep -E "timerfd_settime|SIGSYS|seccomp|killed" /tmp/fc.strace | tail

# The kernel audit log records the denial with the syscall number.
sudo dmesg | grep -i seccomp | tail
# or, if auditd is running:
sudo ausearch -m SECCOMP -ts recent 2>/dev/null | tail

You should see firecracker terminated by SIGSYS (or killed, depending on the filter's default_action) at the exact moment it tried the syscall you removed. That is the whitelist doing its job: a syscall not on the list does not execute.

Tip: Map the syscall number the audit log reports back to a name with ausyscall <n> or grep <n> /usr/include/asm/unistd_64.h. This is precisely the loop you run when a real PR adds code that needs a new syscall and CI's security test fails: read the denial, identify the syscall, add the tightest rule that admits it.

Step 6: Tie it back to the threat model

You have now operated all four layers' configuration except KVM and Rust (which are structural, not per-run). Write — in your own words, in your lab notes — how each layer contributes, and crucially what each layer does not protect against, so the next layer's necessity is clear.

KVM boundary   : isolates guest CPU/memory from host. Does NOT protect the VMM process
                 itself — a guest that finds a VMM bug (bad device emulation) is now
                 running host code as the VMM.  ⇒ need to confine the VMM.
jailer         : chroot + cgroups + namespaces + drop root. Limits what a compromised
                 VMM can REACH (files, other processes, network) and what privileges it
                 holds. Does NOT limit which SYSCALLS it can make.  ⇒ need seccomp.
seccomp        : whitelists the ~tens of syscalls each thread legitimately needs. Shrinks
                 the kernel attack surface a compromised VMM can use. Does NOT prevent the
                 VMM from being compromised in the first place.  ⇒ need memory safety.
Rust           : eliminates whole classes of memory-corruption bugs in the VMM itself,
                 reducing the chance any of the above is ever breached.

This is the reasoning a maintainer applies to every surface-touching PR. The threat-model audit lab makes you do it adversarially against the real device model.

Implementation Requirements / Deliverables

  • A microVM booted under the jailer from a /srv/jailer/.../root chroot, with evidence (ps showing the unprivileged uid; ls showing dev/kvm/dev/net/tun mknod'd inside the chroot; the API socket inside the chroot).
  • A short written summary of the jailer's ordered steps and why privilege drop comes last, citing the function you found in src/jailer/src/.
  • The default seccomp JSON dissected: the three categories, each default_action, and at least two rules with args operators explained.
  • A BPF filter you compiled yourself with seccompiler-bin and loaded via --seccomp-filter, with /proc/.../status showing Seccomp: 2.
  • A reproduced denied syscall: the modified JSON, the recompiled BPF, and the strace/dmesg/auditd evidence of the SIGSYS/kill.
  • The four-layer defense-in-depth note (Step 6) in your own words.

Troubleshooting

The jailer exits immediately with a chroot or mknod error

You likely lack root, or --chroot-base-dir points somewhere unwritable, or /dev/kvm//dev/net/tun don't exist on the host to mknod from. Confirm ls -l /dev/kvm /dev/net/tun on the host, run the jailer with sudo, and check the exact failing step in the jailer output. rg -n "mknod|MkNod|create_dev" src/jailer/src/ shows where it happens.

ENOENT on PUT /boot-source under the jailer

Your kernel/rootfs path is a host path, not an in-chroot path. The jailed firecracker's filesystem root is the chroot. Copy the artifacts into the chroot and reference them by their in-chroot path (/vmlinux, not /home/you/vmlinux-6.1.x).

Firecracker dies on boot even with the stock filter

You may have compiled the filter for the wrong --target-arch, or your kernel/host needs a syscall the stock filter omits on your distro. Re-run under strace to find the actual denied syscall; confirm --target-arch matches uname -m. As a diagnostic only, run once with --no-seccomp — if it boots, the issue is a missing seccomp rule.

Seccomp: 0 in /proc/.../status

Seccomp is not installed. Either you passed --no-seccomp, the --seccomp-filter path was wrong (firecracker may have errored), or you checked the wrong PID/thread. Check every thread under /proc/<pid>/task/*/status, not just the main thread.

auditd shows nothing

auditd may not be running, or the filter's default_action killed without auditing on your distro. Fall back to dmesg | grep -i seccomp and to strace, which always shows the SIGSYS.

Expected Output

  • The jailed firecracker process running as uid 1234, with dev/kvm and dev/net/tun present inside the chroot and absent from the jailer's own view of /dev mounts you didn't set up.
  • The seccomp JSON parsed into exactly three categories, each with a default_action and a list of allowed syscalls, some carrying args constraints.
  • /proc/<pid>/task/*/status showing Seccomp: 2 for every firecracker thread when a filter is installed.
  • A clean reproduction of a SIGSYS/kill the instant a removed syscall is invoked, visible in strace and the kernel log.

Stretch Goals

  1. Tighten a rule. Take a syscall in the filter that has no args and add a masked_eq or eq constraint that still admits firecracker's real usage but narrows it. Verify firecracker still boots, then verify a deliberately wrong constraint breaks it. Document the surface reduction.
  2. Diff filters across thread categories. Programmatically compute which syscalls the vmm filter allows that the vcpu filter does not, and explain (from the threading model) why a vCPU thread needs fewer.
  3. Per-instance UID/GID under cgroups. Run two jailed microVMs with distinct uid/gid/cgroup, and show with cat /sys/fs/cgroup/.../cpu.max (or v1 equivalent) that the --cgroup / --resource-limit limits are actually applied. Cross-check against docs/prod-host-setup.md.
  4. Read the security integration tests. Find the pytest suite that validates the shipped filter and the jailer (rg -l "seccomp|jailer" tests/integration_tests/), run it, and explain what invariant each test protects.

Validation / Self-check

Answer these without notes. They gate completion.

  1. Why is the jailer a separate, root-owned binary instead of code inside firecracker? What does it do that requires root, and what does it do after dropping root? (Trick: it does nothing after dropping root except exec — explain.)
  2. The jailer does not install seccomp. Which process does, and at what point in startup, and why are there three filters instead of one?
  3. In a seccomp SyscallRule, what does masked_eq express that eq cannot, and give a concrete example of a syscall where you'd want it.
  4. You add code to firecracker that calls a new syscall and CI's security test fails. Walk the exact steps from "test red" to "tightest rule that admits the call merged."
  5. Name one thing the jailer protects against that seccomp does not, and one thing seccomp protects against that the jailer does not.
  6. Why is --no-seccomp acceptable as a debugging tool but unshippable in production? What would an attacker gain if a production host ran with it?
  7. A reviewer sees a PR that adds ioctl to a filter with no args constraint, to support one new KVM ioctl. What change do you request, and why is the broad allow worse than the narrow one?

Next: Lab 9.2 — Snapshot and Restore, where the security boundary you just learned meets the persistence boundary — and where a careless field reorder becomes an unrestorable customer snapshot.