Engineering at Scale: Real Design Problems
Everything before this section taught you to operate inside Firecracker: build it, trace a request, read the run loop, write a device, prepare a PR. That is the work of a contributor. This section is about the jump to the work of a designer — the engineer who looks at Firecracker not as a codebase to navigate but as an artifact shaped by a small number of brutally hard constraints, and who can therefore reason about where the project is going and where a serious contribution can move the needle.
Firecracker is not a general-purpose VMM that happens to be small. It is a machine purpose-built to satisfy three requirements that are individually achievable and jointly in tension:
- VM-grade isolation for mutually-untrusting, multi-tenant code — the guest, including the guest kernel, is hostile, and the host must survive it.
- Container-grade density — thousands of microVMs per host, with < 5 MiB of memory overhead each and oversubscription tested past 20×.
- Container-grade startup — application code running in < 125 ms, and, via snapshots, in single-digit milliseconds.
Pick any two and the third gets easy. Real VMs (QEMU) give you isolation and
compatibility but cost you density and boot time. Containers (runc) give you
density and startup but share a kernel, so they fail the isolation test against a
hostile tenant. gVisor splits the difference by emulating the kernel in userspace
and pays in compatibility and syscall-path overhead. Firecracker's entire design
— the minimal device model, the direct kernel boot, the snapshot machinery, the
soft-allocation bet, the seccomp-and-jailer cage — is a series of answers to "how
do we get all three at once?" The
NSDI 2020 paper
("Firecracker: Lightweight Virtualization for Serverless Applications") is the
canonical statement of the problem, and you should read it before this section if
you have not.
Note: This is a demanding section. It assumes you have done the levels and the deep dives. It will not re-explain virtqueues, the vCPU run loop, or the jailer — it assumes you can read those and instead asks the harder question: given how this works, what is the design tension, what does it cost, and what would you change? If a chapter references a mechanism you are shaky on, the linked deep dive is the prerequisite. Go read it, then come back.
Firecracker as an engineering artifact
Every interesting design decision in Firecracker is downstream of the three-way constraint above. Internalize this table — it is the lens for the whole section.
| NSDI design goal | What it forces in the code | Where the tension lives |
|---|---|---|
| Isolation (untrusted guest, untrusted guest kernel) | KVM boundary + jailer + seccomp + Rust; minimal device model | every feature is host attack surface — see minimal device model |
| Low overhead (< 5 MiB/microVM) | no BIOS, no PCI enumeration, no ACPI tables you don't need, tiny resident set | overhead vs. features (hotplug, PCI, richer devices) |
| Performance (boot, I/O latency) | direct kernel load, virtio-MMIO, io_uring engine, huge pages | latency vs. throughput vs. CPU vs. host kernel surface — see io-engines |
| Compatibility (run unmodified Linux) | real KVM, real virtio, real boot protocol | minimalism vs. "the guest expects X" |
| Fast switching (millisecond cold-starts) | snapshot/restore, COW memory, UFFD lazy loading | restore speed vs. versioning burden vs. security of untrusted state — see snapshotting at scale |
| Soft allocation (oversubscribe CPU + memory) | balloon device, on-demand paging, no pre-reservation | the statistical bet vs. what breaks at density — see oversubscription and density |
A general-purpose VMM optimizes the features column. Firecracker optimizes the tension column — it is willing to ship fewer features, less compatibility, and a narrower hardware story in exchange for staying on the right side of every one of those trade-offs at once. When you propose a change, the maintainers will evaluate it against this table, not against "is this a nice feature." That is the single most important cultural fact about contributing here, and it is the subject of the minimal device model philosophy chapter.
Where the open hard problems are
The roadmap is not a secret. Firecracker develops in the open, and you can read its direction directly from the repository and the GitHub project. The trick is knowing where to look — design intent is scattered across four sources, and the high-leverage contributors read all four.
1. The roadmap labels on GitHub. Firecracker uses an explicit label
vocabulary. The two that matter most for "where is this going" are
Roadmap: Tracked and Roadmap: New Request. Run this on a real checkout (you
need gh auth login once):
# Items the maintainers have committed to the roadmap project.
gh issue list --repo firecracker-microvm/firecracker \
--label "Roadmap: Tracked" --state open --limit 40
# Feature requests not yet accepted — where you can argue for direction.
gh issue list --repo firecracker-microvm/firecracker \
--label "Roadmap: New Request" --state open --limit 40
2. The docs/ design notes. The most reliable signal of near-future
direction is a design doc that ships before the feature is GA. As of this writing
the repo carries design notes for features that are partly landed and partly
in-flight — each is a map of an open problem:
cd ~/src/firecracker
ls docs/ # then read the ones describing not-yet-default features
# pvh.md memory-hotplug.md pmem.md device-hotplug.md hugepages.md ...
docs/memory-hotplug.md (virtio-mem), docs/pvh.md (PVH direct boot),
docs/pmem.md (virtio-pmem), and the --enable-pci transport are all examples of
features that exist in some form but are still maturing — exactly the kind of area
a serious contributor can own.
3. The CHANGELOG. CHANGELOG.md is the project's narrated direction. Read the
Unreleased section and the last few releases together and a trajectory appears:
which devices are being added, which fields are being deprecated, what is moving
from "developer preview" to GA. A field rename like
enable_diff_snapshots → track_dirty_pages tells you the snapshot subsystem is
being hardened; a new endpoint tells you a feature is graduating.
4. The labels on Type: and Status:. Type: Enhancement,
Type: Performance, and Good first issue partition the work by kind;
Status: Awaiting author, Status: Blocked, and Status: Parked tell you which
threads are live. The full practical playbook for turning these into a contribution
is the next chapter.
flowchart LR
Paper["NSDI '20 paper:\nthe constraints"] --> Docs["docs/ design notes:\nnear-future features"]
Docs --> Labels["gh labels:\nRoadmap / Enhancement / Perf"]
Labels --> Change["CHANGELOG Unreleased:\nthe trajectory"]
Change --> Own["pick an area\nto own"]
Own --> PR["sustained PRs\nthat respect the constraints"]
The chapters
This section is seven essays, each taking one real Firecracker design problem from the constraint that creates it, down to the code that implements it, out to the open work where you can contribute. Read the roadmap guide first; the order of the rest is a rough difficulty gradient but they stand alone.
| # | Chapter | The hard problem |
|---|---|---|
| 1 | Real Issues & Roadmap | How to find and engage genuinely impactful work — labels, design notes, the CHANGELOG, picking an area to own. Read this first. |
| 2 | Snapshotting at Scale | Millisecond cold-starts from saved state: the two-file model, COW + UFFD lazy loading, diff snapshots, the versioning burden, restoring untrusted state. |
| 3 | Oversubscription & Density | Thousands of microVMs per host: soft allocation, the balloon, why KSM is off, the statistical bet, what breaks at density. |
| 4 | The Minimal Device Model Philosophy | Why fewer devices = smaller attack surface; the no-BIOS/PCI/USB decision; the bar a new device must clear. The cultural essay. |
| 5 | I/O Engines | The synchronous engine vs. the async io_uring engine: how each drives host file I/O from the virtqueue, and the latency/throughput/CPU trade-off. |
| 6 | Huge Pages & Memory Performance | Backing guest RAM with 2 MiB pages: TLB pressure, boot and restore impact, how guest memory is mmap'd, NUMA. |
| 7 | Boot-Time Optimization | The < 125 ms boot: the critical path, how snapshots bypass it entirely, the measurement harness, and how regressions are caught. |
How to read this section
These are not labs. There are no step-by-step instructions and no checkboxes. Each
chapter gives you a thesis, the code that grounds it (via rg/gh commands you
run yourself — never trust a line number printed in a book), the real trade-off
analysis, and a pointer to where the open work is. The intended outcome is not
that you did something but that you can now reason about a class of Firecracker
design problems well enough to argue about it on a GitHub issue with a maintainer
and be taken seriously.
The capstone and the performance-density masterclass turn this reasoning into hands-on work — boot-time benchmarking, oversubscription experiments, io-engine micro-benchmarks. Treat this section as the theory and those as the lab.
Begin with Real Issues & Roadmap: Finding Work That Matters.