Stage 10 — Performance Improvements

What class of issue this is

Stage 10 is making Firecracker measurably faster or lighter without changing what it does — and proving it with numbers the project's performance CI will defend. Firecracker's headline claims are its product: boot to application code in < 125 ms, < 5 MiB memory overhead per microVM, oversubscription > 20×, thousands of microVMs per host. Those numbers are guarded by a benchmark suite, so a perf change is only real if it moves a tracked metric and a regression in any other tracked metric fails CI. The bugs/opportunities are: boot-time (kernel load, device setup, the path to the guest's first instruction), memory overhead (allocations per microVM, page handling, the VMM's own footprint), and I/O throughput/latency (the block io_engine, the net data path, the rate limiter's hot path).

Concretely, a Stage 10 PR is one of:

• A boot-time reduction (less work before KVM_RUN, a cheaper kernel-load or device-init path).
• A memory-overhead reduction (fewer/smaller allocations, better page handling, a leaner struct on a per-microVM path).
• An I/O improvement (a tighter block/net data path, an io_engine win, less per-request overhead).

Why it's at this difficulty

Performance work is measurement-bound, not idea-bound. The hard part is not the optimization — it is building a trustworthy before/after measurement on a stable baseline, isolating your change from noise, and being certain you traded nothing away (a boot-time win that costs memory, a throughput win that adds tail latency). You need Stage 9's determinism as the floor and deep subsystem knowledge to know where the time/memory actually goes. Maps to Level 9; the engineering chapters boot-time optimization, io engines, and hugepages & memory performance are required context, plus guest memory management.

What you must already understand

• Where the performance tests live and what they assert. Firecracker has a performance test suite with tracked baselines; find it before you measure anything:

ls tests/integration_tests/performance/
rg -n "def test_.*boot|boottime|def test_.*memory|def test_.*throughput|baseline|A/B|ab_test" \
  tests/integration_tests/performance/ | head
rg -n "boottime|first.byte|to_userspace|console=ttyS0" tests/ src/ | head

• The A/B test mechanism. Firecracker's perf CI typically compares your branch against the baseline (an "A/B" run) so noise is controlled by running both on the same host. Learn it:

rg -n "ab_test|a_b|baseline|--baseline|git_ab" tests/ tools/ | head

• The hot paths. Boot path through the builder; the block io_engine (Sync vs io_uring); the net TAP RX/TX; guest memory setup:

rg -n "build_microvm_for_boot|fn boot|load_kernel|configure_system" src/vmm/src/builder.rs | head
rg -n "FileEngine|io_uring|Async|Sync|submit|complete" src/vmm/src/devices/virtio/block/ | head

• Profiling tools (perf, flamegraphs, /proc/<pid>/smaps for RSS, strace -c for syscall counts) — to find where the cost is before changing anything.

Representative tasks

How to approach one — worked example: a boot-time improvement

Illustrative of the pattern. Profile your branch; never optimize from intuition.

Symptom: you suspect (or an issue reports) that a step in the boot path does redundant work, adding milliseconds before the guest's first instruction.

Step 1 — establish a stable baseline and a measurement

You cannot improve what you cannot measure repeatably. Run the existing boot-time test on a quiet host, several times, and confirm it's stable (Stage 9 skill) before you touch anything:

rg -n "def test_.*boottime|boottime|to_userspace" tests/integration_tests/performance/ | head
tools/devtool test -- -k boottime          # baseline; run a few times, confirm low variance

If variance is high, fix that first (pin the host, disable SMT/turbo as the perf-CI host does — see prod-host-setup) — a noisy baseline makes any "win" meaningless.

Step 2 — profile to find the actual cost

# syscall counts during boot (cheap, revealing):
strace -f -c ./firecracker --no-api --config-file boot.json 2>strace.out
# CPU time:
perf record -g -- ./firecracker --no-api --config-file boot.json ; perf report

Let the profile, not your guess, point at the hot step. The most common real wins are removing redundant work (a thing computed twice), avoiding an allocation on a per-boot path, or deferring work that isn't needed before first instruction.

Step 3 — make the change, then A/B it

Open the discussion with your measured hypothesis first. Then make the minimal change and run the A/B comparison so your branch and the baseline are measured on the same host in the same run:

rg -n "ab_test|--baseline|git_ab" tests/ tools/ | head
# Run the project's A/B harness comparing your branch to the baseline ref; capture both numbers.

Report both numbers and the delta with a confidence interval, e.g.:

boot time (p50): baseline 118.4 ms -> change 109.7 ms  (-8.7 ms, -7.3%), n=50, non-overlapping CIs
memory overhead: baseline 4.9 MiB -> change 4.9 MiB    (unchanged)
block throughput: baseline ... -> change ...           (unchanged)

The unchanged metrics matter as much as the improved one: a perf PR must show it traded nothing away. Firecracker's perf CI will fail your PR if any tracked metric regresses, so check them all.

Warning: Never widen a perf-test threshold to make your change "pass." If your improvement moves a baseline, the baseline is updated by the maintainers as part of accepting the win — you do not get to loosen the bound yourself. A PR that quietly raises a regression threshold is the perf equivalent of hiding a flake.

What a good PR looks like

• Numbers, on a stable baseline, both directions. The improved metric and every other tracked metric, with sample size and confidence — measured A/B on one host, not eyeballed.
• The change is minimal and behaviour-preserving. Same outputs, same correctness tests pass; you optimized how, not what.
• A profile justifies it. "I profiled and this step cost X" beats "this looked slow." Attach or describe the perf/strace/RSS evidence.
• No regression in another metric, and no threshold loosening. If the win shifts a baseline, that is a maintainer action, called out explicitly.
• The discussion preceded the work with your measured hypothesis — perf maintainers will tell you if your target is already optimal or if the noise floor will swallow the change.
• CHANGELOG entry; correctness tests unchanged and passing.

Graduation criteria — ready to move on when

• You have one merged perf PR with before/after A/B numbers (improved metric + unchanged others), measured on a stabilized baseline.
• You can stand up a low-variance benchmark, profile with perf/strace/RSS, and read the perf-CI A/B output.
• You instinctively check all tracked metrics for regressions, and you never loosen a threshold to pass.
• You can explain where boot time, memory overhead, and I/O cost actually go in Firecracker, with an rg for each hot path.

Performance is one cross-cutting maintainer skill; security is the other. Stage 11 treats every change as an attack-surface change — seccomp filters, the jailer, and the threat model — at an even higher bar.

Next: Stage 11 — Security and Seccomp.