Step 4: Root-Cause Identification

You have localized the bug to a component and seen the value turn wrong. Now you answer the harder question: why does it turn wrong, and where is the right place to fix it? These are two different questions, and confusing them is the difference between a fix a maintainer merges and a patch a maintainer rejects with "this just hides the symptom."

The output of this step is a root-cause statement: a short, precise paragraph that names the cause, distinguishes the fix site from the symptom site, and is concrete enough that the implementation in Step 5 is almost mechanical.

Goal

Write capstone-work/root-cause.md (150–400 words): the precise mechanism of the bug, the introducing change if you can find it, the fix site (where the code should change) distinguished from the symptom site (where it currently breaks), and the reason that fix site is correct rather than a band-aid.

Symptom Site vs. Fix Site

The most common failure mode in a Capstone is fixing where the program crashes instead of where it goes wrong. They are usually different lines, sometimes different files, occasionally different threads.

The guiding principle is Firecracker's defense-in-depth and minimal-surface philosophy: reject bad input as early as possible, at the boundary, with a clear error, so that internal code can assume well-formed state. A panic deep in the builder is a missed validation at the API edge far more often than it is a bug in the builder. See the API server and error messages.

But the inverse trap exists too: do not push validation to the edge when the real cause is an internal logic error. If a virtqueue handler miscomputes a length, the fix is in the handler's arithmetic, not a new check in the API. Localization (Step 3) tells you where the value is correct and where it isn't; root cause is deciding which boundary between those two is the right one to change.

Find the Introducing Change

If the bug is a regression (it worked in an older version — you may have noticed this in Step 2 when checking the version range), find the commit that introduced it. This both confirms your root cause and gives you the strongest possible evidence for the PR.

# If you know a good tag and a bad tag, bisect between them.
git bisect start
git bisect bad main
git bisect good v1.X.0
# git hands you commits to test; run your repro at each and mark good/bad:
#   ./repro.sh && git bisect good   # or  git bisect bad
git bisect reset    # when it names the first bad commit

Or, for a localized line, read its history directly:

# Blame the fix-site line(s) to see who last touched the logic and why.
git log -L :the_buggy_fn:src/vmm/src/path/to/file.rs    # function history
git log --oneline -p -- src/vmm/src/path/to/file.rs | rg -n "<keyword>" -B2 -A6

When git blame/bisect points at a commit, read its PR. An optimization, a refactor, or a "small cleanup" that didn't consider your trigger's parameterization is a textbook root cause — and citing the introducing PR (#NNNN, commit SHA) in your write-up is what separates contributor-grade from maintainer-grade analysis.

Tip: Not every bug is a regression — some are original behavior that was always wrong, or a gap that no input previously exercised. If bisect says "good" all the way back, say so in the root-cause doc; "this was never handled" is a legitimate root cause and changes how you frame the fix (no regression blame, but a clearer "why was this never caught" question for the test).

Confirm the Mechanism, Don't Assume It

You have a hypothesis. Before you write it down as fact, disprove the alternatives the way you would in a write-up's investigation log. A root cause you only confirmed positively (it matches!) is weaker than one where you also killed two competing explanations.

A worked example of the discipline:

Hypothesis A: the value is mangled in the API parse. Disproved: a probe in update_machine_config shows the value arrives correct and is stored correct; nothing mangles it.

Hypothesis B: the builder mishandles a valid value. Disproved: with vcpu_count == 1 the builder is correct; only 0 fails, and 0 is never a valid machine config. The builder is right to assume >= 1.

Hypothesis C (confirmed): validation never enforces the documented vcpu_count lower bound. The swagger spec (src/firecracker/swagger/firecracker.yaml, located by rg -n "vcpu_count" src/firecracker/swagger/) declares a minimum; the Rust path never checks it. The fix site is the config-validation boundary.

Each "disproved" line is a one-command experiment you ran. That is what makes the root cause defensible under review.

Consider the Firecracker-Specific Dimensions

Before settling the fix site, ask the four questions a Firecracker maintainer will ask. Each can move where the correct fix lives:

A guest-reachable virtio bug is a different animal from an operator-only API gap: the former demands you treat every malformed descriptor as hostile (the guest is untrusted), so the root cause is usually "trusted guest-supplied data without bounds-checking it." Frame it that way.

Write the Root-Cause Statement

Now write the paragraph. It should read like the "Root Cause" section of a postmortem (you will reuse it almost verbatim in Step 10):

Root cause. VmResources::update_machine_config (src/vmm/src/resources.rs, located by rg -n "fn update_machine_config") accepts any vcpu_count, including 0, even though firecracker.yaml documents a minimum of 1. The value flows unchecked into build_microvm_for_boot (src/vmm/src/builder.rs), which assumes >= 1 and fails when creating zero vCPUs — the symptom site. The builder's assumption is correct; the defect is the missing lower-bound check at the configuration boundary. This is an original gap, not a regression (git bisect is good back to v1.A.0). Fix site: add the lower-bound validation in update_machine_config, returning the existing MachineConfigError variant, so the bad value is rejected at the API edge with a 400 and never reaches the builder.

Notice it states the cause, the mechanism, the regression status, the fix site, the symptom site, and why the fix site is correct (the builder's assumption is sound). That is a complete root cause.

Deliverable for Step 4

• capstone-work/root-cause.md (150–400 words): mechanism, fix site vs. symptom site, why the fix site is correct.
• The introducing commit/PR identified (or an explicit "original gap, not a regression" with the bisect evidence).
• At least two competing hypotheses recorded as disproved, each with the one-command experiment that killed it.
• The four Firecracker dimensions (API/snapshot/surface/perf) explicitly considered, with a note on which apply.

Rubric Hooks

This step is the spine of Problem articulation in the write-up and a gate on Fix quality: a fix at the symptom site, however clever, caps Implementation quality because reviewers read it as a band-aid. The disproved-hypotheses log is the raw material for the write-up's Investigation Log. A root-cause doc that distinguishes fix site from symptom site and cites the introducing change scores high; "I found where it crashes and added a check there" scores low. See the evaluation rubric.

Validation / Self-check

Before advancing to Step 5:

1. You can state the cause in one sentence that names a mechanism, not a location.
2. You can point to the fix site and the symptom site and explain why they differ.
3. You ruled out at least two alternative explanations with experiments, not argument.
4. You know whether it's a regression, and if so, which PR introduced it.
5. You have considered API contract, snapshot compat, attack surface, and performance, and noted which constrain the fix.
6. You can already imagine the diff — if you cannot, your root cause is not yet sharp enough; keep going.

Then go to Step 5: Implementation.