Lab 3: Reproduce and Bisect

Background

This is the lab the other two feed into, and it is the single most important debugging skill there is. A debugger (Lab 1) and observability (Lab 2) help you understand a failure you can already trigger. But most real bug reports arrive as "it worked in 1.13 and it's broken in 1.16" or "boot got 10 ms slower somewhere this quarter" — and you cannot understand, let alone fix, what you cannot reproduce on demand. The discipline is two moves, in order: make it fail deterministically, then binary-search the history with that determinism as the test.

git bisect run is the engine. Given a known-good commit, a known-bad commit, and a script that exits 0 for good and 1 for bad, it checks out the midpoint, runs your script, and uses the verdict to halve the search space — converging on the first bad commit in O(log n) steps. Across a thousand commits that is about ten test runs. The payoff is not just the commit: it is the PR that commit came from — its description, its review discussion, and its linked issue usually tell you why the change was made, which is half of how you fix the regression without re-breaking the original intent.

This lab makes you do the whole cycle for both a behaviour regression and a performance regression, because the scripted test differs (a boolean assertion vs. a threshold on a noisy number), and the second is where beginners hand git bisect a flaky test and get a confident wrong answer.

Note: This lab uses the pytest integration harness as the bisect test, and the BootTimer (from Lab 2) as the performance metric. Have both working first.

Why This Lab Matters for Contributors

• "Regression introduced in <sha>, here is the deterministic repro and the minimal fix" is the highest-signal bug-fix PR you can open. It does the maintainers' hardest work for them.
• A deterministic reproducer is, by itself, a valuable contribution — maintainers can act on a clean repro even before a fix exists, and it becomes the regression test that guards the fix.
• Bisecting teaches you the codebase's history: you read the diffs and PRs at the boundary, which is how you learn why the code is the way it is.
• This is exactly the discipline the Capstone demands — issue → reproduce → execution-path → root cause. This lab is the rehearsal.

Prerequisites

• Lab 1.1: Build from source and Lab 1.2: Run the tests — you can build and test any commit.
• Level 5 — you can write and run a pytest integration test.
• Lab 2 of this intensive — you can produce a Guest-boot-time number.
• git, a clean working tree, and a checkout with full history.

cd ~/src/firecracker
git status                     # must be clean before you bisect
git log --oneline -5
git tag | rg "^v1\." | tail    # the release tags you'll use as good/bad anchors

Step 1: Establish the regression and pick the anchors

A regression has a good end and a bad end. Find a commit/tag where the behaviour is correct and one where it is broken. Release tags are the natural anchors because they are coarse and trustworthy.

# Confirm the two ends. (Use the real tags on your branch; these are illustrative.)
git checkout v1.13.0   # suspected good
# ... build, run the repro, confirm it PASSES ...
git checkout main      # or v1.16.0 — suspected bad
# ... build, run the repro, confirm it FAILS ...
git checkout main

If you cannot make the good end pass and the bad end fail by hand, stop — you do not yet have a reproducer, and bisecting now will give you a meaningless commit. Steps 2–3 build the reproducer first, for exactly this reason.

Warning: Verify both ends before bisecting. A common failure is assuming the old tag is good without testing it — if the bug was always present (not a regression), bisect will march to a nonsense commit. "Good" must be proven good.

Step 2: Write a deterministic reproducer (behaviour regression)

The goal is a single command that exits 0 when the behaviour is correct and non-zero when it is not, every time. For a behaviour regression, the gold standard is a pytest integration test, because it builds in CI and git bisect run can drive it directly. The shape:

# tests/integration_tests/functional/test_repro.py  (your reproducer)
def test_block_read_returns_correct_bytes(uvm_plain):
    """Deterministic repro: a known write must read back identical bytes.
    PASS on good commits, FAIL once the regression is present."""
    vm = uvm_plain
    vm.spawn()
    vm.basic_config(vcpu_count=1, mem_size_mib=256)
    vm.add_net_iface()
    vm.start()
    # Drive the exact code path the bug lives on. Make EVERY input fixed:
    # same kernel, same rootfs, same offsets, no randomness, no timing dependence.
    _, stdout, _ = vm.ssh.run("dd if=/dev/vdb bs=512 count=1 2>/dev/null | sha256sum")
    assert stdout.strip().split()[0] == "EXPECTED_KNOWN_HASH"

Determinism rules — violate any one and your bisect lies:

Run it at both ends and confirm the verdict flips:

git checkout v1.13.0 && ./tools/devtool build --release
./tools/devtool test -- integration_tests/functional/test_repro.py     # expect PASS
git checkout main     && ./tools/devtool build --release
./tools/devtool test -- integration_tests/functional/test_repro.py     # expect FAIL

Tip: The test file lives in your working tree, but bisect will check out old source. Keep the reproducer out of the bisected history — stash it and re-apply, or keep it in a file git ignores, or pass it on the command line — so checking out an old commit doesn't delete your test. The cleanest trick: commit the test on a branch and git stash/git checkout -- carefully, or run the test from a path outside the repo. See Step 4's wrapper.

Step 3: Make the reproducer a pass/fail script

git bisect run needs a command whose exit code is the verdict: 0 = good, 1–124 (except 125) = bad, 125 = skip/untestable. Wrap build + test so any build failure becomes a skip (not a false "bad"), and the test result becomes the verdict:

cat > /tmp/repro.sh <<'EOF'
#!/usr/bin/env bash
set -u
cd ~/src/firecracker

# Build the commit under test. If it doesn't even build, this commit is untestable -> skip (125).
if ! ./tools/devtool build --release >/tmp/build.log 2>&1; then
    echo "BUILD FAILED -> skip"; exit 125
fi

# Run the deterministic reproducer (kept outside bisected history; see Step 4).
if ./tools/devtool test -- integration_tests/functional/test_repro.py >/tmp/test.log 2>&1; then
    echo "PASS -> good"; exit 0
else
    echo "FAIL -> bad"; exit 1
fi
EOF
chmod +x /tmp/repro.sh

The 125-for-build-failure rule matters: across a release window some commits won't build with your toolchain, and you must not let a build break masquerade as the regression.

Step 4: Run git bisect run

Now drive it. Keep the reproducer test file from being clobbered by old checkouts — the simplest robust approach is to copy it into the source tree inside the wrapper each run, from a stable location outside the repo:

# Stash the test outside the bisected tree:
cp tests/integration_tests/functional/test_repro.py /tmp/test_repro.py

# Make the wrapper re-plant it before testing (add near the top of /tmp/repro.sh):
#   mkdir -p tests/integration_tests/functional
#   cp /tmp/test_repro.py tests/integration_tests/functional/test_repro.py

cd ~/src/firecracker
git bisect start
git bisect bad  main        # proven broken (Step 1)
git bisect good v1.13.0     # proven good   (Step 1)
# git checks out the midpoint; let the script render each verdict automatically:
git bisect run /tmp/repro.sh

git bisect prints the convergence as it goes and finishes with:

<sha> is the first bad commit
commit <sha>
Author: ...
Date:   ...
    <subject line of the offending commit>
...
bisect found first bad commit

Always reset when done so you return to a normal HEAD:

git bisect reset

flowchart LR
    Good["git bisect good v1.13.0"] --> BS["bisect picks midpoint"]
    Bad["git bisect bad main"] --> BS
    BS --> Run["git bisect run /tmp/repro.sh"]
    Run -->|"exit 0"| G["mark good, search newer half"]
    Run -->|"exit 1"| B["mark bad, search older half"]
    Run -->|"exit 125"| S["skip, pick neighbour"]
    G --> BS
    B --> BS
    S --> BS
    BS -->|"converged"| First["first bad commit + its PR"]

Step 5: Bisect a performance regression (the harder case)

A behaviour bug gives a crisp boolean. A performance regression — "boot got slower" — gives a noisy number, and a naive threshold turns run-to-run noise into random good/bad verdicts that send bisect to the wrong commit. The fix is to make the measurement robust before you bisect.

Use the BootTimer number, measured carefully, with a threshold set clear of the noise band:

cat > /tmp/repro_perf.sh <<'EOF'
#!/usr/bin/env bash
set -u
cd ~/src/firecracker
./tools/devtool build --release >/tmp/build.log 2>&1 || { echo skip; exit 125; }

# Measure boot time N times; take the MEDIAN (robust to outliers), not a single run.
# Pin CPUs / warm cache to shrink the noise band (see the boot-time lab).
RUNS=7
vals=()
for i in $(seq "$RUNS"); do
    # boot with --boot-timer, parse the Guest-boot-time ms field from the log:
    ms=$( ./tools/devtool test -- integration_tests/performance/test_boottime.py \
            2>&1 | rg -o "Guest-boot-time =\s+\d+ us\s+(\d+) ms" -r '$1' | head -1 )
    vals+=("${ms:-99999}")
done
median=$(printf '%s\n' "${vals[@]}" | sort -n | awk '{a[NR]=$1} END{print a[int((NR+1)/2)]}')
echo "median boot ms = $median"

# Threshold set ABOVE the good-commit noise band, BELOW the regressed value.
THRESHOLD_MS=95
if [ "$median" -le "$THRESHOLD_MS" ]; then echo "good"; exit 0; else echo "bad"; exit 1; fi
EOF
chmod +x /tmp/repro_perf.sh

The discipline that makes this trustworthy:

git bisect start
git bisect bad  main
git bisect good v1.13.0
git bisect run /tmp/repro_perf.sh
git bisect reset

Warning: If your good and bad ends' boot-time distributions overlap, you do not have a measurable regression to bisect — you have noise. Tighten the measurement (more runs, more CPU pinning) until the distributions separate, or the bisect is meaningless. Report a distribution, not a number.

Step 6: Read the diff and root-cause

The first bad commit is the start of the analysis, not the end. Open it and the PR it came from:

git show <sha> --stat        # what files it touched
git show <sha>               # the actual change
# Find the PR and the discussion behind the commit:
gh pr list --repo firecracker-microvm/firecracker --search "<sha>" --state merged
gh api "repos/firecracker-microvm/firecracker/commits/<sha>/pulls" --jq '.[].html_url'

Now reason: why did this change cause the regression? Read the PR description and review comments for the change's intent — the fix must preserve that intent while removing the regression. For a behaviour bug, the diff often shows the exact line; for a perf regression, it shows the added work (a new device probe, an extra copy, a synchronous call on the boot path). Cross-link the relevant deep dive (e.g. boot sequence, virtio-block) to confirm your mechanism. That is the root cause — and it is what your PR's description will state.

Implementation Requirements / Deliverables

• A deterministic reproducer that passes on a proven-good commit and fails on a proven-bad one, by hand, before any bisect.
• A git bisect run wrapper script with correct exit-code semantics (0 / 1 / 125).
• A completed behaviour bisect that names the first bad commit.
• A completed performance bisect using a median-of-N BootTimer measurement, with the threshold justified by the good/bad distributions.
• A short root-cause write-up: the offending commit, its PR/intent, the mechanism, and the deep dive that confirms it.

Troubleshooting

Bisect converged on an obviously-unrelated commit

Your reproducer was nondeterministic, or "good" was never actually good. Re-verify both ends by hand, remove all randomness/timing from the repro, and rerun. A bisect is only as honest as its test.

Old commits don't build with my toolchain

Expected across a release window. The wrapper must exit 125 (skip) on build failure — confirm it does. If many commits won't build, narrow the good/bad window to a range that shares a toolchain.

My reproducer test file vanished mid-bisect

Old checkouts overwrite the working tree. Keep the test outside the bisected history and re-plant it in the wrapper (Step 4), or pass an absolute path to a test that lives outside the repo.

Performance bisect gives different first-bad commits on reruns

The measurement is noisier than the regression is large. Increase RUNS, pin CPUs, idle the host, and confirm the good/bad distributions separate (Step 5). If they can't be separated, there is no bisectable regression — say so.

git bisect run says "running ... " then nothing

The script is hanging (a microVM that never exits, a FIFO with no reader). Add timeouts to the build/test in the wrapper and ensure every spawned firecracker is reaped.

Expected Output

$ git bisect run /tmp/repro.sh
running /tmp/repro.sh
PASS -> good
...
FAIL -> bad
...
<sha> is the first bad commit
 N files changed, ...
bisect found first bad commit

Stretch Goals

1. Automate the whole report. Extend the wrapper so that on convergence it prints the first-bad git show --stat, the linked PR URL, and the good/bad boot-time medians — a one-command regression report you could paste into an issue.
2. Bisect across a refactor boundary. Pick a window that spans the big crate-merge refactor (where modules moved into vmm). Notice how git bisect skip and the 125 exit code carry you across commits where paths changed and your test won't apply — and how to keep the test path-agnostic.
3. Turn the repro into a permanent guard. Clean up your deterministic reproducer into a proper integration test, with a DCO sign-off, that would have caught the regression. That is a mergeable PR on its own.

Validation / Self-check

1. Why must you prove both the good and the bad end by hand before running git bisect?
2. What do exit codes 0, 1, and 125 mean to git bisect run, and why does build failure map to 125?
3. How do you keep your reproducer test file from being clobbered when bisect checks out old source?
4. Why is a performance regression harder to bisect than a behaviour one, and what makes the measurement trustworthy?
5. Where do you set the threshold for a perf bisect, and how do the good/bad distributions tell you whether a bisect is even possible?
6. After bisect names the first bad commit, what two things do you read from its PR, and why does the intent of the change matter for the fix?
7. Why is a deterministic reproducer, even without a fix, a contribution maintainers value?

You have completed the Debugging and Profiling intensive: you can attach to either side of the KVM boundary (Lab 1), localize a fault from counters and logs without a debugger (Lab 2), and pin a regression to a commit and root-cause it (this lab). Take these into the Performance & Density intensive, where the BootTimer and bisect skills become the tools for defending Firecracker's headline numbers, and ultimately into the Capstone, where reproduce-and-root-cause is the whole job.