Performance & Density — Intensive

Firecracker exists because of three numbers: it boots to application code in < 125 ms, it adds < 5 MiB of memory overhead per microVM, and it lets a host oversubscribe past 20× and pack thousands of mutually-untrusting microVMs. Those are not marketing figures — they are the load-bearing requirements of the serverless business model, and every one of them is measured, defended, and regression-gated in the project's own test suite. This intensive is about turning those headline numbers from claims you've read into numbers you've produced, broken, and explained on your own host.

The discipline that separates a serious performance contributor from a beginner is not knowing that Firecracker is fast — it's being able to measure the right thing, break it down into a critical path, detect a regression against a baseline, and explain a number that disagrees with expectation. A benchmark you cannot explain is not a result. So this intensive is built around three measurable questions a contributor actually answers:

1. How fast does it boot, and where does the time go? — the BootTimer device, the performance test harness, the kernel-load / device-setup / guest-init breakdown, regression detection.
2. How many fit on one host, and what breaks at density? — many-microVM launching, aggregate memory and CPU overhead, the balloon for reclaim, the statistical oversubscription bet.
3. Which I/O engine wins, and when? — Sync vs Async/io_uring under fio, latency vs throughput vs CPU, and how the choice is guarded.

   one microVM                          one host, thousands of microVMs
 ┌──────────────────────┐             ┌───────────────────────────────────────────┐
 │ boot path (Lab 1)    │  ×N         │  aggregate overhead, oversubscription      │
 │ • kernel load        │ ─────────►  │  (Lab 2): the >20× statistical bet,        │
 │ • device setup       │             │  the balloon for reclaim, what breaks      │
 │ • guest init         │             │  at scale                                  │
 │ I/O engine (Lab 3)   │             │                                            │
 │ • Sync vs io_uring   │             │                                            │
 └──────────────────────┘             └───────────────────────────────────────────┘
   per-microVM cost  ───────────────────────►  bounds total density

Note: The two halves are one argument. Per-microVM cost (boot time, memory overhead, per-request I/O CPU) is multiplied by the number of microVMs on a host, so a millisecond of boot or a megabyte of overhead is a density lever, not just an efficiency nicety. Hold both scales in view: Lab 1 and Lab 3 measure the per-microVM cost; Lab 2 measures what happens when you stack thousands of them.

What you will be able to do

By the end of this intensive you will be able to:

1. Measure true boot time with the in-VMM BootTimer pseudo-device (the guest writes a magic byte; Firecracker logs Guest-boot-time), via the test_boottime.py performance harness, and from first principles.
2. Break the boot critical path into VMM setup (kernel load, vCPU init, device attach), guest kernel init, and userspace, and attribute time to each phase.
3. Experiment with boot-time levers — kernel config, fewer devices, the cmdline tokens, and snapshots that bypass boot entirely — and quantify each.
4. Detect a boot-time regression against a baseline, distinguishing real regressions from run-to-run noise (CPU frequency scaling, cold page cache, NUMA).
5. Launch many microVMs on one host (a loop / the firecracker-demo pattern) and measure aggregate memory overhead (RSS vs configured) and CPU.
6. Reclaim memory with the balloon — set a target, watch madvise(MADV_DONTNEED) shrink the resident set, read /balloon/statistics — and reason about deflate_on_oom and the "Out of puff!" failure.
7. Reason about the >20× oversubscription bet as statistical multiplexing — why it works, when it loses (memory overcommit, CPU correlation), and what second-order effects break at scale.
8. Benchmark the Sync vs Async/io_uring block engines under fio across queue depths, interpret latency/throughput/CPU, and connect the result to which engine is appropriate when and how the perf is guarded.

The three labs

Do them in order: Lab 1 establishes how the project measures anything (the harness, baselines, noise); Lab 2 scales one microVM to thousands; Lab 3 drills into the one per-request cost (block I/O) that most affects density under load.

• Lab 1: Measure and optimize boot time — measure-it / trace-it. Use the performance test harness and the BootTimer device (magic byte 123 — verify) to measure boot to userspace, break down the critical path, experiment with kernel config / fewer devices / snapshots, and detect a regression with real numbers.

• Lab 2: Oversubscription and density — measure-it / reason-it. Launch many microVMs on one host, measure aggregate memory overhead and CPU, use the balloon to reclaim memory, observe behavior at high density, and reason about the >20× statistical oversubscription bet from the NSDI paper.

• Lab 3: Benchmark the I/O engines — measure-it. Compare the Sync vs Async/io_uring block engines under fio (latency, throughput, CPU), configure each via io_engine, interpret the results, connect to when each is appropriate, and find how the perf is guarded in the test suite. Read the io_engine code under devices/virtio/block/.

Prerequisites

This is a masterclass, not an introduction. You must have completed Level 9, Lab 9.3: Performance Regression — that lab gives you the first guided pass over the performance test suite and the discipline of a baseline; this intensive assumes it and goes deeper. You should also have read the four engineering essays this intensive turns into hands-on work:

You need a working Firecracker build, the ability to boot a microVM by hand (Level 1, Lab 1.3), the pytest performance harness runnable via tools/devtool test, and a guest rootfs with fio for Lab 3. Confirm now:

# 1. The firecracker binary and a kernel + rootfs.
ls build/cargo_target/$(uname -m)-unknown-linux-musl/release/firecracker
ls vmlinux-* *.ext4

# 2. The performance test suite (the regression gate) exists.
ls tests/integration_tests/performance/
rg -ln "boottime|boot_time|Guest-boot-time" tests/integration_tests/performance/

# 3. The BootTimer pseudo-device and the I/O engines exist in the tree.
rg -n "BootTimer|MAGIC_VALUE_SIGNAL_GUEST_BOOT_COMPLETE" src/vmm/src/devices/pseudo/
ls src/vmm/src/devices/virtio/block/virtio/io/    # sync_io.rs async_io.rs (verify path)

# 4. Tools you'll measure with on the host.
which fio iperf3 2>/dev/null; echo "(fio also needed INSIDE the guest for Lab 3)"

Warning: Performance benchmarks are noisy. CPU frequency scaling, a cold page cache, a busy host, NUMA placement, and thermal throttling can each swamp the signal you're after. Run these labs on a quiet, dedicated host you own (ideally bare metal — a *.metal instance, not a shared VM), pin CPUs where you can, warm caches, run multiple iterations, and report a distribution, not a single number. A "result" you can't reproduce twice is not a result.

The measurement discipline (read before any lab)

Every lab here lives or dies by methodology. Internalize these rules — they are what reviewers hold your numbers to:

flowchart LR
    B["baseline (same host)"] --> C["change one variable"]
    C --> M["measure: N iterations, wall + CPU"]
    M --> D{"delta > noise?"}
    D -->|no| Noise["not a regression — report the spread"]
    D -->|yes| Explain["explain the cause<br/>(then it's a result)"]

Common mistakes contributors make in this area

How to verify you are ready to start

# Answer each from the engineering essays BEFORE Lab 1.
# 1. What magic byte does the guest write to signal boot-complete, and where is it handled?
rg -n "MAGIC_VALUE_SIGNAL_GUEST_BOOT_COMPLETE|BootTimer|Guest-boot-time" src/vmm/src/devices/pseudo/
# 2. Which mmap flag makes guest RAM lazily allocated (not reserved)?
rg -n "MAP_NORESERVE|MAP_PRIVATE|MAP_ANONYMOUS" src/vmm/src/vstate/memory.rs
# 3. How does the balloon physically reclaim host memory?
rg -n "madvise|MADV_DONTNEED|inflate|deflate" src/vmm/src/devices/virtio/balloon/
# 4. What selects a drive's I/O engine, and what are the two variants?
rg -n "file_engine_type|FileEngineType|io_engine|enum FileEngine" src/vmm/src/devices/virtio/block/

If you cannot say, from memory, that boot time is measured by a magic-byte MMIO write, that guest RAM is faulted in on demand (so resident ≪ configured), and that the block device dispatches to one of two I/O engines, re-read the three engineering essays above before continuing. The labs assume all three.

Next: Lab 1: Measure and optimize boot time — drive the BootTimer device and the performance harness to measure boot to userspace, break down the critical path, and detect a regression with real numbers.