Level 9: Advanced Maintainer — Security, Snapshots, Performance

This is the last level before the capstone, and it is the one that changes how you read every diff. Through Level 8 you were a contributor: you found an issue, reproduced it, fixed it, wrote an integration test, and opened a PR with a DCO sign-off. A Firecracker maintainer carries a heavier burden. The maintainer is the person who blocks a green, well-tested, genuinely useful PR — because it adds a syscall to the seccomp filter without justifying it in the threat model, because it changes a serialized snapshot field without a versioned migration, or because it adds 8 ms to boot or 200 KiB to the per-microVM memory floor on a host that runs thousands of microVMs. The contributor optimizes for "does my change work." The maintainer optimizes for "is the host still safe, can every snapshot still restore, and is the fleet still dense — after my change merges and a billion Lambda invocations run through it."

Firecracker's reason for existing is multi-tenant isolation at density: VM-grade security on the same host that packs containers' worth of workloads. Three properties encode that mission directly into the codebase, and they are the three you must learn to defend in this level: the security model (the jailer, the seccomp filter, and the threat model that justifies both), snapshot/restore (and the brutal compatibility and versioning constraints that come with persisting a live machine's state), and performance and density (boot time, the < 5 MiB memory overhead, > 20x oversubscription, and the I/O engines that make block I/O cheap). These are the concerns that separate "I can add a feature" from "I can be trusted not to break the fleet."

This curriculum will not hold your hand here. By Level 9 you can read the vCPU run loop, trace an API request to a VmmAction, instrument a virtio-block I/O down to a host pread, and reproduce a GitHub issue. What you build now is judgment — the reflexes a maintainer applies to every PR they review, including their own: does this widen the attack surface, can this snapshot still be loaded by the version that wrote it (and the next one), and did I measure the density cost?

Learning Objectives

By the end of Level 9 you must be able to:

1. Run Firecracker under the jailer from scratch — chroot/pivot_root, cgroups, namespaces, mknod for /dev/kvm and /dev/net/tun, --netns, and the setuid/setgid privilege drop — and explain why the jailer is a separate binary that does not itself apply seccomp.
2. Read and modify a seccomp filter: locate the per-thread-category JSON in resources/seccomp/<arch>.json, understand SyscallRule with args operators, compile it with seccompiler-bin, and reason about what --seccomp-filter and --no-seccomp do to the security boundary.
3. Articulate the threat model and defense in depth — the untrusted guest, the privileged VMM as part of the attack surface, and the four overlapping layers (KVM boundary, jailer, seccomp, Rust memory safety) — well enough to evaluate whether a PR widens the surface.
4. Execute the full snapshot/restore workflow (PATCH /vm Paused → PUT /snapshot/create → PATCH /vm Resumed → load on a fresh process), inspect both the state file and the memory file with snapshot-editor, and explain the Persist trait that every device implements.
5. Reason about snapshot backward compatibility as a maintainer concern: why adding a field to persisted device state is a versioning event, what Full vs Diff snapshots cost, and why a careless Persist change is unshippable.
6. Measure boot time and memory overhead, run the performance test harness with A/B comparison (tools/devtool test --performance / --ab), profile a microVM, and attribute a regression to a specific commit by bisection.
7. Connect a change to the density and oversubscription goals — and to how performance is guarded in CI so that a regression is caught before it ships.
8. Review a security-, snapshot-, or performance-touching PR the way a maintainer does: surface-area first, compatibility second, measured cost third — not just "the diff is clean and CI is green."

The Three Constant Maintainer Concerns

Strip away everything else and a Firecracker maintainer's job, at this tier, reduces to defending three invariants on every change. Internalize these; they are the lens for the whole level.

flowchart TB
    PR[Incoming PR / your own change] --> SEC{Does it touch the<br/>attack surface?<br/>seccomp, jailer,<br/>a device, an ioctl}
    PR --> SNAP{Does it change<br/>persisted state,<br/>a Persist impl,<br/>or the snapshot format?}
    PR --> PERF{Does it touch a<br/>hot path? boot,<br/>per-microVM memory,<br/>the virtio fast path}
    SEC -->|yes| G1[Justify in the threat model,<br/>minimize syscalls/devices,<br/>review seccomp diff]
    SNAP -->|yes| G2[Version-gate it, prove<br/>old→new and new→old restore,<br/>add a snapshot compat test]
    PERF -->|yes| G3[Measure with --ab,<br/>profile, check boot time<br/>and memory floor]
    SEC -->|no| OK1[Standard review]
    SNAP -->|no| OK2[Standard review]
    PERF -->|no| OK3[Standard review]

Concern 1 — Security (the host stays protected while the VMM changes)

Firecracker's threat model is explicit and harsh: the guest, including the guest kernel, is untrusted, and the VMM is privileged host code that is part of the attack surface. A guest that compromises the VMM escapes onto the host and into other tenants. Defense in depth is therefore not optional decoration; it is four overlapping boundaries — the KVM hardware-virtualization boundary, the jailer (chroot/cgroups/namespaces/privilege-drop), the seccomp-BPF syscall whitelist, and Rust memory safety — each of which must hold even if another is bypassed. Every PR that adds a syscall to a filter, emulates a new device, or adds an ioctl widens the surface and must justify itself against this model. The mechanisms live in the jailer deep dive and the seccomp deep dive; the auditing discipline is drilled in the security masterclass. Re-read those before this level's first lab — the lab assumes them.

Concern 2 — Snapshot compatibility (the state stays restorable while it changes)

A snapshot is a serialized copy of a running machine: KVM state, every device's registers and queues, and the guest's RAM. The moment you persist that state, its layout becomes a contract. Firecracker supports loading a snapshot taken by a range of versions; that promise breaks the instant someone reorders a field in a device's Persist state, changes a default, or removes a register from the saved set without a versioned migration. The failure is not a red unit test — it is a customer's saved microVM that will not resume after they upgrade Firecracker, which is among the worst bugs this project can ship. The full mechanism — the Persist trait, the state file vs memory file split, Full vs Diff, and the UFFD restore path — is in the snapshotting deep dive and drilled in the snapshotting masterclass.

Concern 3 — Performance and density (the fleet stays dense while the code changes)

Firecracker exists so AWS can run thousands of mutually-untrusting microVMs per host: boot to app code in < 125 ms, < 5 MiB of memory overhead per microVM, tested oversubscription beyond 20x. Those are not marketing numbers; they are the budget. A path that runs once per boot, once per microVM, or on the virtio fast path is hot: a few hundred KiB added to the per-microVM floor, or a few milliseconds added to boot, multiplies across the fleet into stranded capacity and slower cold starts. Maintainers do not accept "it should be about the same" — they accept "here is the --ab run, here is the boot-time and memory delta, here is why it is within budget." The discipline is taught in oversubscription and density and drilled in this level's third lab and the performance-density masterclass.

These three concerns are why maintainer review of a Level 9 PR feels slow. It is not gatekeeping for its own sake; it is the cost of invariants that cannot be un-shipped once a release goes out to a fleet measured in millions of microVMs per second.

Required Reading

These are the documents and source areas you must read before the labs. Confirm they exist on your checkout — paths drift, so run the commands rather than trusting this table.

# One sweep to confirm the security/snapshot docs are where this level expects them.
ls docs/seccomp.md docs/jailer.md docs/prod-host-setup.md
ls docs/snapshotting/
ls resources/seccomp/

Source Code Areas to Inspect

You will live in these directories this level. Locate them with find/rg — do not trust the paths blindly, the big crate refactor moved several of these into vmm.

# Locate the security, snapshot, and performance surfaces on YOUR branch.
find src/jailer -name '*.rs' | head
rg -n "seccomp" src/vmm/src/seccomp.rs | head
fd -e json . resources/seccomp/ 2>/dev/null || find resources/seccomp -name '*.json'
rg -n "trait Persist" src/vmm/src/                 # the persistence contract
rg -ln "impl .*Persist" src/vmm/src/devices/       # every device that persists state
find tests/integration_tests/performance -type f

Key Types Quick Reference

Do not memorize these; run the rg and read them. They are the types the labs touch.

GitHub Issue Categories for Level 9

These are the labels and issue shapes a Level 9 graduate can credibly take on. Find them with gh:

gh issue list -R firecracker-microvm/firecracker \
  --label "Type: Enhancement" --search "snapshot OR seccomp OR jailer OR performance" --limit 40
gh issue list -R firecracker-microvm/firecracker --label "Kani" --limit 20   # formal-verification work

Deliverables

Demonstrate all of the following before attempting the capstone:

• Completed Lab 9.1: booted a microVM under the jailer from a hand-built /srv/jailer chroot, inspected the default seccomp JSON, recompiled a modified filter with seccompiler-bin, and observed a denied syscall under your custom filter.
• Completed Lab 9.2: created a Full snapshot, restored it on a fresh Firecracker process, inspected both files with snapshot-editor, and explained the Persist trait and a snapshot-compat hazard.
• Completed Lab 9.3: measured boot time and memory overhead, ran an A/B (--ab) comparison, and attributed a representative regression to a commit by bisection.
• A written review (GitHub-review style) of one real Firecracker PR that touches seccomp, the jailer, a Persist impl, or a performance path — explicitly assessing attack surface, snapshot compatibility, and measured cost.
• From memory: explain the four defense-in-depth layers, the difference between Full and Diff snapshots, and one failure mode plus one mitigation for each of the three maintainer concerns.

Common Mistakes

How to Verify Success

# 1. You can build the binaries this level uses.
tools/devtool build --release
ls build/cargo_target/$(uname -m)-unknown-linux-musl/release/{firecracker,jailer,seccompiler-bin,snapshot-editor}

# 2. You can locate every Level 9 surface without notes.
rg -n "trait Persist" src/vmm/src/
rg -n "SeccompFilter\|apply_filter" src/vmm/src/seccomp.rs
find src/jailer -name '*.rs' | head
find tests/integration_tests/performance -type f

# 3. You can run the security and performance suites.
tools/devtool test -- integration_tests/security/   2>/dev/null || \
  rg -l "seccomp\|jailer" tests/integration_tests/security/
tools/devtool test --performance --ab 2>/dev/null || echo "review the --ab harness in Lab 9.3"

You have succeeded when you can, from memory, state the threat model in two sentences, name the three files that make a snapshot a compatibility contract, and quote the boot- time and memory-overhead budgets — and when the rg commands above land on real code on your branch.

Maintainer Profile: Level 9 Graduate

You are now ready for the capstone — an end-to-end contribution where you select a real issue, reproduce it, find the root cause, implement and test the fix (with security, snapshot, and performance discipline), open the PR with DCO sign-off and a CHANGELOG entry, and write it up. Start at the Capstone Overview. The three reflexes you build here — does this widen the attack surface, can this snapshot still restore, did I measure the density cost? — are the ones the capstone evaluates hardest.

Note: Most engineers never reach this tier on an open-source project, not because they can't, but because they stop at "my PR is green." The distance from contributor to maintainer is the distance from "it works" to "the host is still safe, every snapshot still restores, and the fleet is still dense — for everyone who upgrades into it." That is the entire content of Level 9.

Next: Lab 9.1 — Seccomp Filters and the Jailer.