Project 6: A Boot-Time / Density Benchmark Harness

Firecracker's headline numbers — boots to application code in under 125 ms, adds less than 5 MiB of memory overhead per microVM, sustains thousands of microVMs per host with oversubscription tested past 20x — are the reason it exists. They are also the numbers that quietly regress. A change that adds a few milliseconds to the boot path, or a few hundred kilobytes of per-microVM overhead, is invisible in a unit test and invisible in a code review, but at Lambda/Fargate scale it is a capacity and latency disaster. The defense is measurement: a reproducible harness that produces those numbers the same way every time, so a regression shows up as a number moving instead of a customer incident.

This project asks you to build that harness to a quality the maintainers would actually use: a one-command, reproducible benchmark that measures boot-to-userspace latency, per-microVM memory overhead, and many-microVM density, characterizes its own noise, and detects regressions against a baseline. The primary deliverable is the harness plus a findings write-up; because Firecracker already ships performance tooling and the maintainers care deeply about these numbers, a clean harness or a methodology improvement is a genuinely mergeable contribution.

Note: Read the performance & density masterclass in full — especially Lab 1: boot time and Lab 2: oversubscription — and the engineering essays boot-time optimization, oversubscription and density, and hugepages and memory performance. This brief assumes you can boot a microVM by hand (Lab 1.3), drive the API socket, and read /proc memory accounting. It will not re-teach the boot path; it teaches you to measure it without lying to yourself.

Problem & motivation

Performance claims rot silently, and Firecracker's claims are load-bearing:

• Boot time is the product. Serverless cold-start latency is boot-to-userspace latency plus snapshot-restore latency. A 10 ms regression in the boot path is a 10 ms regression in every cold start, across billions of invocations. There is no unit test for "the boot got slower"; there has to be a benchmark.
• Memory overhead sets the density ceiling. Per-microVM overhead — the VMM's own RSS beyond the guest's configured RAM — directly bounds how many microVMs fit on a host. Firecracker claims < 5 MiB; a change that quietly doubles it halves a capacity-planning assumption. Measuring overhead correctly (separating VMM overhead from guest RAM from shared/COW pages) is subtle, and subtlety is exactly where regressions hide.
• Density behaves non-linearly. Booting one microVM tells you almost nothing about booting a thousand. Contention on /dev/kvm, the host scheduler, page-cache pressure, and memory oversubscription all emerge only at scale. The interesting failures — tail-latency collapse, OOM under oversubscription, ref-count contention — are density phenomena.

The motivation is leverage and credibility: a harness that produces these numbers reproducibly is the only way to defend a performance claim or catch a regression before it ships. The maintainers run performance CI precisely because manual measurement is unreliable; a contributor who can build a rigorous harness is contributing to the part of the project that protects its reason to exist.

What you'll build

A benchmark harness — scripted, one command to run, reproducible on a stated host — that measures three dimensions and detects regressions:

Plus the parts that make it a harness and not a script:

1. Noise characterization — repeat each measurement ≥20 times, report median and p99, and state the run-to-run variance, so a real change is distinguishable from jitter.
2. A baseline + regression detector — store a baseline result, compare a new run against it, and flag a regression beyond a stated threshold.
3. A findings write-up — what you measured, on what host, and what the numbers say (including any regression you found and bisected).

Prerequisites

• Level 1 (build + boot) and Level 9 Lab 3: perf regression.
• The performance & density masterclass, all three labs.
• The debugging & profiling masterclass, especially Lab 3: reproduce and bisect — regression detection is bisection with numbers.
• Engineering essays boot-time optimization and oversubscription and density.
• The boot sequence deep dive and guest memory management deep dive — you must understand what you're measuring.
• Hardware reality check: density and overhead numbers only mean something on bare-metal with a real /dev/kvm (a *.metal instance or a physical host). Nested virtualization distorts boot time and memory accounting badly. State your host.

Phased plan

Phase 0 — Build, find the existing perf tooling, and measure one boot (1–2 days)

Do not start from zero — Firecracker already ships performance tests and tooling. Find them first, so you extend the methodology instead of inventing a worse one.

tools/devtool build --release
# The in-tree performance tests and tooling (paths move — find them):
rg -rn "boot.?time|boottime|MicrovmStarted|InstanceStart|guest_boot|def test_.*perf|performance" tests/ | head -40
find tests -ipath '*performance*' -o -iname '*boot*' | head
# How the integration framework already times a boot (the Microvm test fixture):
rg -n "class Microvm\b|def start\b|InstanceStart|wait.*boot|serial|console" tests/framework/ | head
# How FC itself logs boot timing (if a metric exists):
rg -n "boot|MicrovmStarted|start_time|TimestampUs|metric" src/vmm/src/ | rg -i "boot|start" | head

Boot one microVM and time it crudely, to anchor the harness:

API=/tmp/fc.sock
sudo ./firecracker --api-sock "$API" &
# ... PUT /boot-source, /drives/rootfs, /machine-config ...
# Measure from InstanceStart to a known guest-userspace marker on the serial console:
time curl -X PUT --unix-socket "$API" --data '{"action_type":"InstanceStart"}' http://localhost/actions

Anti-staleness: the performance test layout (tests/integration_tests/performance/, a framework/ fixture, a tools/ script — paths vary), the boot-timing metric names, and whether FC emits a structured boot-time metric all move between releases. Find the current shape with rg/find, read CHANGELOG.md for performance-tooling changes, and check gh issue list --repo firecracker-microvm/firecracker --search "boot time OR performance OR density" for live work. Do not hard-code a path you didn't just confirm.

Produce capstone-work/measure-baseline.md: how FC and the test framework currently define and measure boot, what memory accounting exists, and where your harness will plug in.

Phase 1 — A correct, reproducible single-boot timer (the foundation)

Define "boot-to-userspace" precisely and measure it the same way every time. The definition is the hard part: it must be a guest-observable event that means user code is running, not just "the kernel started."

Pick one, justify it, and make the measurement deterministic: same kernel, same rootfs, same machine-config, same cmdline, the VMM pinned to a CPU, the rootfs cache state controlled.

Milestone 1: a single command produces a boot-to-userspace number, run 20 times, with a median and p99 and a stated variance under (say) a few percent. If your variance is huge, your methodology is wrong — fix it before measuring anything else.

Phase 2 — Memory-overhead measurement (the subtle one)

Per-microVM overhead is not the VMM's RSS — that includes guest RAM. You want the VMM's own footprint beyond the configured guest memory, and you must account for COW and shared pages honestly.

# Find the VMM pid, then read smaps_rollup for PSS (proportional set size — COW-aware):
pid=$(pgrep -f "firecracker --api-sock $API")
grep -E "Pss|Rss|Private|Shared" /proc/$pid/smaps_rollup
# Overhead ≈ VMM PSS − configured guest RAM (with the guest RAM region identified in smaps).

The discipline:

• Use PSS, not RSS, so shared/COW pages are attributed proportionally — otherwise N microVMs sharing the binary's pages over-count overhead N times.
• Measure at a defined moment (just-booted, idle) so you compare like with like.
• Control page-cache state. A warm rootfs in the host page cache changes the picture; state cold vs. warm.
• Subtract guest RAM correctly — identify the guest-memory mapping in /proc/$pid/smaps and exclude it, leaving true VMM overhead.

Milestone 2: a per-microVM overhead number with the methodology stated, near FC's < 5 MiB claim (or, if it isn't, a clear explanation of why your accounting differs).

Phase 3 — Density (the non-linear one)

Boot N microVMs and watch how boot latency and host pressure scale. This is where the interesting behavior lives.

   N microVMs        boot p50 (ms)   boot p99 (ms)   total VMM PSS (MiB)   host free (MiB)
       1                <a>             <b>               <c>                 <d>
      50                ...             ...               ...                 ...
     200                ...             ...               ...                 ...
     500                ...             ...               ...                 ...

Sweep N, plot boot p50/p99 against N, track total overhead and host free memory, and find where it falls over (scheduler contention, page-cache pressure, oversubscription limits). For oversubscription, deliberately configure total guest RAM beyond host RAM and show the balloon/COW behavior holds (or where it doesn't) — this is the oversubscription claim under test.

Warning: Density runs can OOM or wedge the host. Bound the sweep, watch host free memory, and have a kill-all teardown. A harness that takes down the machine it's measuring is not reproducible.

Milestone 3: a density curve (boot latency vs. N) and an overhead-vs-N curve, with the host stated and the failure point identified.

Phase 4 — Noise characterization, baselining, and regression detection

This is what turns three measurements into a harness:

• Noise model: for each metric, report median, p99, and run-to-run variance over ≥20 iterations. State what's noise so a reader can tell a real change from jitter.
• Baseline: serialize a run's results (JSON) as a baseline keyed by FC git rev + host.
• Regression detector: a command that runs the suite, compares to the baseline, and exits non-zero (with a clear diff) if a metric regressed beyond a stated threshold — threshold chosen above the measured noise floor, or it'll cry wolf.

# The shape of the deliverable — one command to run, one to compare:
./harness/run.sh --out results-$(git rev-parse --short HEAD).json
./harness/compare.sh --baseline baseline.json --candidate results-*.json   # exits 1 on regression

If you find a real regression, bisect it (the reproduce-and-bisect skill) to the commit, and that bisection is a contribution in itself.

Key code areas

Design considerations & trade-offs

• The definition of "done" is the experiment. "Boot time" is meaningless until you fix exactly which guest-observable event ends the clock. Most bad boot benchmarks measure the wrong endpoint (kernel start, not userspace) and are quietly comparing apples to oranges. Pin the definition and defend it.
• Noise is the enemy of regression detection. A harness whose run-to-run variance exceeds the regression it's meant to catch is worse than useless — it generates false alarms and trains people to ignore it. Characterize noise first; set thresholds above it.
• PSS vs. RSS is a correctness decision, not a style one. Reporting RSS for density over-counts shared pages and makes overhead look N× worse than it is. Use PSS and say so.
• Cold vs. warm page cache changes everything. A warm rootfs makes every boot look fast. Decide which you measure (cold is the honest worst case; warm is the steady state), control it explicitly (drop_caches), and state it.
• Reproducibility beats absolute numbers. A benchmark that gives a different absolute number on a different host but the same delta for a given change is doing its job. The harness's value is detecting change, not producing a marketing figure — though it should be able to reproduce FC's published claims on comparable hardware.
• Bare metal or it didn't happen. Nested virtualization distorts boot timing and memory accounting enough to make the numbers meaningless. State the host; prefer *.metal.

How to test & validate

• Self-consistency: run the harness twice on the same rev/host; the medians must agree within the stated noise. If they don't, the harness is the bug.
• Sensitivity: introduce a known slowdown (e.g. a deliberate sleep in a non-fast path, or a larger initrd) and confirm the harness detects it. A regression detector that can't catch a planted regression is unproven.
• Cross-check the overhead number against FC's published < 5 MiB claim and explain any gap with your accounting method.
• Integration discipline: if you wire this into the pytest performance suite, follow its conventions (rg -n "def test_\|pytest.mark" tests/integration_tests/performance/) so it could run in CI.
• The harness is the deliverable — it must run with one command on a freshly cloned fork, on a stated host, and produce the same shape of output every time.

Stretch goals

• CI-shaped regression gate: package the regression detector so it could run as a CI check on a PR, failing on a boot-time or overhead regression beyond threshold — the form the maintainers actually want.
• Snapshot-restore latency as a fourth dimension (restore-to-responsive), connecting this harness to Project 2 (UFFD handler).
• Per-phase boot breakdown: instrument the boot path to attribute the boot time to phases (kernel load, device setup, guest init) so a regression points at where it slowed.
• A real regression found and bisected: sweep recent FC revisions, find a real boot-time or overhead regression (or confirm there is none), bisect it, and write it up.
• An io-engine bench: extend the harness to compare Sync vs. io_uring block I/O (io-engines).

What a strong deliverable looks like

A strong deliverable is a one-command, reproducible harness that measures boot-to-userspace, memory overhead, and density; characterizes its own noise; baselines a run and detects regressions above a stated threshold; and comes with a findings write-up that states the host, the methodology, the numbers, and (ideally) a regression found and bisected.

The upstreaming path is among the cleanest in the portfolio, because tooling is mergeable:

1. Performance tooling is wanted. FC runs performance CI; a rigorous harness, a methodology improvement, or a new dimension in the existing suite is mergeable on its own merits. Find the live state: gh issue list --repo firecracker-microvm/firecracker --search "boot time OR performance OR density OR benchmark" and read the existing tests/integration_tests/performance/ suite.
2. Methodology is the contribution. The numbers are host-specific; the way you measure (the boot-done definition, PSS accounting, noise characterization, the regression threshold) is the durable, mergeable part. Write it down clearly.
3. A real regression is the strongest result. If your harness finds and bisects an actual boot-time or overhead regression, that is a high-value contribution by itself — exactly the kind of thing performance CI exists to catch.

Even if nothing lands upstream, a harness plus a findings write-up is a portfolio-grade artifact: it proves you can measure a systems claim honestly, which is rarer than it sounds. A finished version at 90+ on the rubric is maintainer-grade performance-engineering work.

Next: Project 7 — a seccomp filter audit & diff tool to turn the same measurement rigor on the security boundary, or back to the portfolio overview.