Stage 11 — Security and Seccomp

What class of issue this is

Stage 11 is the highest-bar non-release stage: changes to Firecracker's isolation machinery — the seccomp-BPF syscall filters, the jailer's containment, and any fix made through an attack-surface lens. Firecracker's entire reason to exist is being a credible isolation boundary for hostile, multi-tenant guests; the guest (including the guest kernel) is untrusted and the job is to protect the host. Defense in depth means the KVM boundary, plus the jailer (chroot, cgroups, namespaces, privilege drop), plus a seccomp filter that whitelists the ~40 syscalls Firecracker is allowed to make, plus Rust's memory safety. A Stage 11 change touches one of those layers, so every such change is scrutinized for whether it widens the attack surface.

Concretely, a Stage 11 PR is one of:

• A seccomp filter change: a syscall that a legitimate new code path needs added to resources/seccomp/<arch>.json (with an argument filter where possible), or an over-broad rule tightened.
• A jailer hardening fix: a containment gap (a resource not isolated, a privilege not dropped, a device node created too permissively), or a --cgroup/--resource-limit/--netns handling bug.
• An attack-surface-aware fix elsewhere: closing a path where guest-controlled input reaches privileged host code (often overlapping Stage 7), or removing unnecessary surface.

Why it's at this difficulty

A mistake here is a sandbox escape or a privilege escalation — the failure mode the entire project exists to prevent. The work requires the threat model in your head, knowledge of exactly which syscalls each thread category legitimately makes, and the judgment to prefer the narrowest change. It also has a hard procedural rule: if you discover an exploitable vulnerability, you report it privately to AWS Security, never as a public issue or PR (SECURITY.md). Maps to Level 9; the jailer and seccomp filtering deep dives are required reading, and the compatibility and maintainer mindsets apply.

What you must already understand

• The seccomp model. Per-thread-category filters — vmm, api, vcpu — each a default_action plus a list of SyscallRules (a syscall number, optional args with operators eq/ge/gt/lt/ne/masked_eq). Compiled to BPF by seccompiler and baked into the binary at build time:

ls resources/seccomp/
rg -n "default_action|filter|syscall|args|comment" resources/seccomp/x86_64.json | head
rg -n "seccompiler|SyscallRule|apply_filter|install.*seccomp|BpfProgram" \
  src/vmm/src/seccomp.rs src/seccompiler/ | head

• Overrides and the rule that they're not for production. --seccomp-filter <path> swaps the filter; --no-seccomp disables it (testing only). Know how to test with a custom filter without shipping a weakened default.

• The jailer. Runs as root, builds the barrier, then setuid/setgid drops to unprivileged and execs firecracker. It does pivot_root/chroot, cgroups, namespaces (unshare/--netns/ --new-pid-ns), mknod for /dev/kvm and /dev/net/tun, optional --daemonize. The jailer does not apply seccomp — firecracker does:

ls src/jailer/src/
rg -n "pivot_root|chroot|setuid|setgid|unshare|cgroup|mknod|namespaces|drop" src/jailer/src/ | head

• The threat model (docs/prod-host-setup.md, the threat-model docs): the guest is untrusted; side channels (SMT/KSM), Rowhammer, and egress to the metadata IP are part of the picture.

Representative tasks

How to approach one — worked example: a seccomp rule for a new path

Illustrative of the pattern. The filters are per-arch JSON; both x86_64 and aarch64 must stay consistent.

Symptom: a legitimate code change adds a syscall (say a new *at variant, or io_uring-related calls for the async block engine) that the current seccomp filter does not allow, so Firecracker is killed by the kernel (SIGSYS) when it reaches the new path. The fix is to add the narrowest rule that permits exactly what's needed — never a blanket allow.

Step 1 — confirm the syscall and the thread category

Identify which thread makes the call (vmm, api, or vcpu) and the exact syscall. A SIGSYS / audit log or strace pins it:

strace -f -e trace=all ./firecracker --config-file boot.json 2>&1 | tail -40   # find the blocked call
rg -n "io_uring|io_uring_enter|io_uring_setup" resources/seccomp/ src/vmm/src/devices/virtio/block/ | head

Step 2 — open the discussion (mandatory here), then add a minimal, argument-filtered rule

Seccomp changes always go through a maintainer — widening the allow-list is exactly what the project is conservative about. State why the syscall is needed, by which thread, and whether you can constrain its arguments. Then add it to the right category in each arch file, narrowed with an arg filter when the syscall takes a constrainable argument:

--- a/resources/seccomp/x86_64.json
+++ b/resources/seccomp/x86_64.json
@@  "vmm": {
       "default_action": "trap",
       "filter_action": "allow",
       "filter": [
+        {
+          "syscall": "io_uring_enter",
+          "comment": "Async block engine: submit/complete I/O on the io_uring fd."
+        },

When the syscall has an argument worth constraining (a fixed flag, a specific fd kind, a request type), add an args filter rather than allowing every invocation:

{
  "syscall": "fcntl",
  "args": [{ "index": 1, "type": "dword", "op": "eq", "val": 3, "comment": "F_GETFL only" }]
}

Mirror the change in resources/seccomp/aarch64.json (syscall names are portable in the JSON, but the available syscalls and any numeric assumptions differ — verify each).

Step 3 — test that the path works with the filter on, and that nothing else opened up

The decisive test exercises the new path under the real (rebuilt) filter — the filter is baked at build time, so rebuild, then run the functional test that hits the path:

tools/devtool build
tools/devtool test -- -k "block or async or io_uring"     # the path that needed the syscall

Confirm you did not broaden anything else: diff the compiled filter or review that you added exactly one rule, argument-constrained where possible. Firecracker has seccomp-specific tests — run them:

rg -n "def test_.*seccomp|seccomp" tests/integration_tests/ | head
tools/devtool test -- -k seccomp

Warning: Two hard rules. (1) Narrowest possible. An argument-filtered rule beats a bare syscall; a bare syscall beats relaxing default_action. Never disable the filter to "make it work." (2) If the issue is an actual exploitable hole — a missing isolation that a guest could use to reach the host — it is a vulnerability: report it privately to AWS Security per SECURITY.md and do not open a public PR or issue describing the exploit.

How to approach a jailer hardening fix

Illustrative.

Symptom: the jailer creates a device node or leaves a resource accessible more permissively than needed, or a --cgroup limit isn't actually applied. Find the relevant step and tighten it:

rg -n "mknod|chmod|0o666|cgroup|write_cgroup|setuid|setgid" src/jailer/src/ | head

The fix narrows permissions / fixes the limit application, with an integration test that asserts the containment actually holds (the node has the expected mode, the cgroup limit is enforced). The jailer has its own integration tests — extend them.

What a good PR looks like

• The narrowest possible change. For seccomp: argument-filtered rule > bare syscall allow > touching default_action (almost never). For the jailer: least privilege, smallest hole closed.
• Both architectures kept consistent (x86_64.json and aarch64.json), verified per-arch.
• Tested with the real filter / real jailer, proving the legitimate path works and nothing else opened up.
• A maintainer-led discussion preceded it. Security changes are never surprise PRs.
• Exploitable vulnerabilities go to AWS Security privately — the public PR only ever contains the hardening, never a working exploit, and only after coordination.
• CHANGELOG entry under ### Security when appropriate; seccomp/jailer tests pass.

Graduation criteria — ready for the final stage when

• You have one merged security PR — a narrowly-scoped seccomp rule (argument-filtered where possible) or a jailer hardening fix — reviewed under the security lens, tested with the real filter/jailer.
• You can explain the three seccomp thread categories, the rule structure, and why the filter is baked at build time; and you can describe the jailer's containment steps in order.
• You reach for the narrowest mechanism by reflex and never disable a protection to make a path work.
• You know the disclosure rule cold: exploitable issues go to AWS Security privately, never public.

The last stage is where all of this converges. Stage 12 is about the issues that can hold a release — regressions, CVE-class fixes — and the judgment to call one.

Next: Stage 12 — Release-Blocking Issues.