Lab 9.3: Analyze a Performance Regression

Lab type: Research & Benchmark (measurement discipline, hands-on) Estimated time: 5–7 hours

Background

Firecracker's reason for existing is density: boot to application code in < 125 ms, add less than 5 MiB of memory overhead per microVM, and pack thousands of mutually-untrusting microVMs onto one host at oversubscription ratios beyond 20x. Those are not aspirations — they are the budget the product is sold on, and they translate directly into AWS Lambda cold-start latency and the number of functions a host can hold. A change that adds 8 ms to boot or 300 KiB to the per-microVM memory floor looks free in a single run and is enormously expensive across a fleet measured in millions of microVMs per second.

That is why a Firecracker maintainer treats performance on hot paths as a correctness property. "It should be about the same" is not an acceptable claim on a path that runs once per boot, once per microVM, or on the virtio fast path. The acceptable claim is "here is the A/B run, here is the boot-time delta and the memory-overhead delta, here is why it is within budget — and here is the bisection that proves this commit caused the change we saw." This lab teaches that measurement loop end to end: measure boot time and memory overhead with the real test harness, run an A/B comparison, profile a microVM to find where time and memory go, and attribute a regression to a single commit by bisection.

Why This Lab Matters for Contributors

• Performance patches — and performance regressions — are held to a higher review bar than correctness patches, because "faster" or "no slower" is unfalsifiable without numbers, and the cost of shipping a regression is paid by every user. You need to produce and read those numbers.
• The CI already guards these budgets with A/B performance tests; a maintainer needs to understand what those tests measure so they can interpret a CI failure and decide whether a delta is real or noise. The mechanism is in oversubscription and density and boot-time optimization.
• The I/O engines (Sync vs io_uring/Async block I/O) are a live performance lever; the trade-offs are in I/O engines and drilled in the I/O-engines bench lab.
• Bisection over a real regression is the skill that turns "something got slower" into a one-line attribution a maintainer can act on. The reproduce-and-bisect lab and the performance-density masterclass go further; this lab is the foundation.

Prerequisites

• Completed Lab 9.2; you can build, boot, and snapshot a microVM by hand.
• You understand the boot sequence (boot sequence deep dive) — boot time is the sum of the phases it describes.
• You understand guest memory (guest memory deep dive) — memory overhead is the host memory cost beyond the guest RAM allocation.
• You can run the pytest integration harness (Lab 5.1).
• perf is available on your host (perf --version); you have permission to profile (sudo, or kernel.perf_event_paranoid lowered).

tools/devtool build --release
find tests/integration_tests/performance -type f      # the perf suite you'll run
rg -ln "boot.?time|memory.?overhead|throughput" tests/integration_tests/performance/

The Measurement Loop

flowchart LR
    M[Measure baseline<br/>boot time + mem overhead] --> AB[A/B: run the same<br/>tests on two revisions]
    AB --> D{Delta beyond<br/>noise?}
    D -->|no| STOP[Not a regression —<br/>say so, with numbers]
    D -->|yes| P[Profile: where did<br/>the time/memory go?]
    P --> B[Bisect: which commit<br/>introduced it?]
    B --> R[Attribute + report:<br/>one commit, one number]

Two definitions you must keep straight, because they are measured differently:

Step 1: Measure boot time and memory overhead with the harness

The performance suite already encodes "boot time" and "memory overhead" as tests. Run them and read what they assert.

# Run the performance tests through devtool (they need a real /dev/kvm host).
tools/devtool test --performance -- integration_tests/performance/ 2>/dev/null || \
  tools/devtool test -- integration_tests/performance/

# Read exactly what "boot time" and "memory overhead" mean in the harness.
rg -n "boot.?time|guest.?boot|marker|magic" tests/integration_tests/performance/
rg -n "memory.?overhead|rss|RSS|/proc.*status|VmRSS" tests/integration_tests/performance/

Open the boot-time test and the memory-overhead test and answer, from the code: what marks "the guest reached app code," and what exactly is subtracted to isolate overhead from the guest's own RAM? You cannot interpret a regression you can't define.

Tip: Boot time has several components — kernel load + boot_params setup, vCPU creation, the guest kernel's own boot, and userspace init. A regression in one is invisible if you only watch the total. The boot sequence deep dive breaks the phases down; keep it open while you read the test.

Step 2: A/B compare two revisions

The whole point of --ab is to remove your machine's noise from the comparison: it runs the same tests on two revisions back to back on the same host and reports the delta with a significance judgment, so you are not comparing a number from today against a number you wrote down last week on a busier machine.

# Discover the A/B harness and its arguments on your branch — flags vary.
tools/devtool test --help 2>&1 | rg -i "ab|a/b|performance|baseline|revision"
rg -n "def .*ab|--ab|A/B|baseline|git_ab|run_ab" tools/ tests/ | rg -i "ab|baseline" | head

# Typical shape: compare the working tree (or a revision) against a baseline revision.
tools/devtool test --performance --ab main 2>/dev/null || \
  echo "use the flags the help text above prints; the idea is: same tests, two revisions, same host"

The A/B output gives you, per metric, the baseline value, the candidate value, the delta, and whether it is statistically significant. Three outcomes, three responses:

Warning: A single slow run is not a regression. Performance numbers are noisy — CPU frequency scaling, neighbor load, ASLR, and cache state all move the needle. Always compare on the same host, back to back, with enough iterations that the A/B harness can call significance. A maintainer who reports a regression from one run loses credibility fast.

Step 3: Profile a microVM

When the A/B says a metric moved, profiling tells you where. Three complementary tools: perf for CPU time, the metrics for what firecracker itself counted, and tracing for the boot timeline.

ARCH=$(uname -m)
BIN=build/cargo_target/${ARCH}-unknown-linux-musl/release

# perf-record a boot. Capture call stacks so you can see WHERE time goes.
sudo perf record -g -- "$BIN/firecracker" --api-sock /tmp/fc.sock &
# ...drive a boot via curl as in earlier labs, let the guest reach app code, then stop...
sudo perf report --stdio | head -40         # the hot functions during boot

Firecracker emits structured metrics — counters and timings it records itself, which are far more reliable than guessing from stack samples:

# Find the metrics the VMM records (timings, per-device counters, etc.).
rg -n "struct .*Metrics|IncMetric|StoreMetric|process_time|boot" src/vmm/src/logger/
# Enable metrics output to a file and read the boot-relevant fields.
sudo curl -X PUT --unix-socket /tmp/fc.sock --data \
  '{"metrics_path":"/tmp/fc-metrics.json"}' http://localhost/metrics

And there is in-tree tracing instrumentation for the boot path:

# The log-instrument tooling adds tracing spans across functions — see how it's used.
rg -n "log_instrument|instrument|trace_span|tracing" src/vmm/src/ | head

Tip: For memory overhead specifically, sample the firecracker process RSS right after boot and subtract mem_size_mib: grep VmRSS /proc/$(pgrep -n firecracker)/status. A memory regression often hides in an allocation that scales per-device or per-vCPU rather than a one-time cost — profile with that in mind. The hugepages and memory performance chapter covers how page size interacts with the overhead number.

Step 4: Attribute a regression by bisection

You have a metric that moved and a profile that points at a region. Now nail it to a single commit with git bisect, driven by a script that returns pass/fail based on the metric crossing a threshold.

# A good/bad test script: boots a microVM, measures the metric, exits 0 (good) / 1 (bad).
cat > /tmp/perf-probe.sh <<'SH'
#!/usr/bin/env bash
set -e
tools/devtool build --release >/dev/null 2>&1
# Run the single performance test that captures the regressed metric, parse its number,
# and compare against THRESHOLD. Return 0 if good (fast/light), 1 if bad (slow/heavy).
RESULT=$(tools/devtool test -- integration_tests/performance/ -k boot_time 2>/dev/null \
         | rg -o 'boot_time[^0-9]*([0-9.]+)' -r '$1' | head -1)
THRESHOLD=125     # ms — set from your known-good baseline
awk -v r="$RESULT" -v t="$THRESHOLD" 'BEGIN{exit !(r<=t)}'
SH
chmod +x /tmp/perf-probe.sh

# Bisect between a known-good and known-bad revision.
git bisect start
git bisect bad  HEAD                 # the slow revision
git bisect good v1.15.0              # a revision you know was within budget (verify a real tag)
git bisect run /tmp/perf-probe.sh    # git walks the history; the script judges each commit
git bisect reset

git bisect run converges in log2(N) builds to the first commit where the script flips from good to bad. That commit is your attribution — the thing to read, the author to ask, the change to scrutinize for the per-boot or per-microVM cost it added.

Warning: Bisection is only as good as your good/bad test. If the metric is noisy near the threshold, bisect will give you a plausible but wrong commit. Average several iterations per build, choose a threshold comfortably between the good and bad values (not at the edge of noise), and verify the final commit by hand with an A/B run.

Step 5: Walk a representative regression hunt end to end

Tie it together on one realistic story. A reviewer notices CI's boot-time A/B test failed on a PR that "only" added a field to the machine config and a small device-init step.

1. CONFIRM     A/B on the PR vs main, same host, 20 iterations each.
               Boot time: main 112 ms, PR 119 ms, delta +7 ms, flagged significant.
               -> real, not noise. Investigate.
2. DEFINE      Which phase? Re-run with the boot-time test that splits phases (or add
               tracing spans). The +7 ms is entirely in device setup, not kernel load.
3. PROFILE     perf report + metrics show the new device-init step doing synchronous work
               on the boot path that could be deferred or done once, not per-vCPU.
4. ATTRIBUTE   git bisect run over the PR's commits pins it to the device-init commit, not
               the config-field commit. Now you know the exact change.
5. REPORT      Comment with: the A/B numbers, the phase, the profile snippet, the commit.
               Request: move the work off the boot path or make it one-time. Re-A/B to
               confirm the fix lands the metric back within budget.
6. GUARD       Confirm the CI perf test would have caught this (it did) and that the fix
               does not regress memory overhead while fixing boot time — measure BOTH.

This is the shape of every performance review at this tier: confirm it's real, localize the phase, profile the cause, attribute the commit, report with numbers, and verify the fix against all the budgets — not just the one that broke.

Implementation Requirements / Deliverables

• Baseline boot-time and memory-overhead numbers from the performance harness, with a one-line definition of each (what marks "booted," what is subtracted for "overhead").
• An A/B comparison of two revisions on the same host, reporting per-metric deltas and whether each is significant.
• A profile of a single boot (perf report and/or firecracker metrics) identifying where boot time or memory goes.
• A git bisect run over a real or constructed regression that attributes it to a single commit, plus the good/bad probe script you used.
• A written walkthrough of one regression hunt (Step 5 shape) with your own numbers.
• A statement connecting your numbers to the density budget: what +X ms boot or +Y KiB overhead costs across a fleet, and why CI guards it.

Troubleshooting

Performance numbers swing wildly between runs

Pin CPU frequency (cpupower frequency-set -g performance), disable turbo, quiesce the host (no other heavy processes), and increase iteration count so the A/B harness can average. Run both sides of a comparison back to back on the same host — never compare against an old number from a different machine state.

perf record produces no useful stacks

You need frame pointers or DWARF unwinding and permission to profile. Build with frame pointers if available, use perf record --call-graph dwarf, and lower kernel.perf_event_paranoid (or run under sudo). Confirm perf --version works first.

The boot-time test can't find the guest's "booted" marker

The harness depends on a specific guest image/marker. Use the test kernel/rootfs the harness expects (the CI artifacts), not an arbitrary image. rg -n "marker|magic|boot" tests/integration_tests/performance/ shows what it looks for.

git bisect run blames a commit that obviously isn't the cause

Your probe is noisy at the threshold, or a middle commit didn't build (a non-zero exit from a build failure is treated as "bad"). Use git bisect skip semantics for unbuildable commits (exit 125), widen the good/bad gap from the threshold, and average more iterations per build.

Memory overhead looks fine but boot time regressed (or vice versa)

They are independent budgets. A change can trade one for the other. Always measure both; a fix that helps boot time by allocating eagerly may push memory overhead over budget.

Expected Output

• Boot-time and memory-overhead baselines consistent with Firecracker's budgets (boot well under 125 ms for a minimal config; overhead in the low single-digit MiB).
• An A/B report with a clear significant/not-significant verdict per metric.
• A perf report or metrics dump localizing the cost to a boot phase or a per-device allocation.
• A bisection that converges to a single commit and a probe script that judges it.

Stretch Goals

1. I/O-engine A/B. Configure a block device with the Sync engine and again with the io_uring/Async engine (rg -n "io_engine|Async|io_uring|Sync" src/vmm/src/devices/), benchmark I/O throughput/latency for each, and explain the trade-off. Cross-link your findings to I/O engines.
2. Memory overhead vs vCPU/device count. Measure overhead as you scale vcpu_count and the number of devices. Is the per-microVM floor constant, or does it grow per-vCPU? What does that imply for density?
3. Reproduce a real merged perf PR. gh pr list -R firecracker-microvm/firecracker --search "boot time performance regression" --state merged, check out before/after, and reproduce the reported delta with your A/B harness.
4. Add a tracing span. Instrument one boot-path function with the in-tree log-instrument tooling and read its timing in the trace output. Explain what phase it isolates.
5. Hugepages overhead. Boot with huge_pages enabled in /machine-config and measure the effect on memory overhead and boot time. Tie it to hugepages and memory performance.

Validation / Self-check

Answer without notes. They gate completion.

1. Define "boot time" and "memory overhead" as the harness measures them. What marks the guest as booted, and what is subtracted to isolate overhead from guest RAM?
2. Why is a single slow run not a regression? What does --ab do that comparing against a number you wrote down yesterday does not?
3. You see boot time regress by 7 ms. Walk the steps from "A/B flagged it" to "this exact commit, this exact phase, here's the fix request."
4. Why is git bisect run only as trustworthy as the good/bad probe? Name two ways a noisy probe gives you the wrong commit and how you'd guard against each.
5. A change improves boot time but you suspect it cost memory. Why must you measure both budgets, and what's the fleet consequence of trading one for the other unknowingly?
6. Connect a +300 KiB per-microVM overhead regression to the density goal: roughly what does it cost a host that runs thousands of microVMs, and why does CI guard this automatically?
7. How does the Sync vs io_uring/Async block I/O engine choice affect performance, and on what axis (throughput, latency, host CPU) would you A/B them?

You have completed Level 9. You can now operate Firecracker's security model, snapshot and restore a microVM while reasoning about cross-version compatibility, and measure, profile, and attribute a performance regression against the density budget. These are the three reflexes a maintainer applies to every surface-, snapshot-, or hot-path-touching change.

Next: return to the Level 9 overview to confirm every deliverable is checked, then begin the Capstone — one full real contribution cycle, judged hardest on exactly the security, compatibility, and performance discipline you built here.