Lab 8.1: Reproduce an Existing GitHub Issue

Background

A bug you cannot reproduce on demand is a bug you cannot fix with confidence. In a VMM this is doubly true: the failures live in places — a virtqueue, a VM exit, a snapshot restore, a seccomp deny — where "it worked on my machine" hides a real defect behind a missing condition. The reproducer is the single most valuable artifact in a Firecracker contribution. It makes the root cause provable, makes your fix verifiable, and is the first thing a maintainer looks for in your PR. Before you write one line of a fix, you must have something that fails the same way every time on your build of main.

This lab takes a real open issue from firecracker-microvm/firecracker and turns its prose bug report into a deterministic reproducer — ideally a failing pytest integration test (for behavioral bugs on a real microVM) or a cargo test unit test (for pure logic), at minimum a recorded config-file or curl --unix-socket sequence with captured output. The governing rule, echoed from the Level 8 index: it must fail on main without your change, every run, and pass after the fix. If it does not fail on main, you are reproducing the wrong thing.

Why This Lab Matters for Contributors

• Maintainers triage by reproducibility. "I can't reproduce" closes more Firecracker issues than any fix merges.
• A reproducer separates symptom (what the reporter saw) from trigger conditions (the minimal inputs that actually provoke it). That separation is the root-cause work, started early — it feeds directly into the execution-path walk in Lab 8.2.
• A reproducer promoted to a pytest integration test or a cargo test becomes your regression guard for free, and it is exactly the artifact CONTRIBUTING requires you to ship for new behavior.
• It is the skill the Capstone grades in Step 2: Reproduction. This lab is the warm-up.

Prerequisites

• A built checkout on main from Lab 1.1, and the ability to boot a microVM by hand from Lab 1.3.

• The pytest harness running from Lab 5.1.

• You can read the API → VMM path (Level 3) and the virtio path (Level 7) — most reproducible bugs live in one of those.

• Record exactly what you are building against:

  git rev-parse --short HEAD
  git log -1 --format='%h %ci %s'
  rg -n "^version" Cargo.toml | head -1     # the workspace version you are reproducing against
  

• The GitHub CLI authenticated (gh auth login) so you can read issues and PRs from the terminal.

Step-by-Step Tasks

Step 1 — Pick and triage an issue (15 min)

Open the issue tracker from the terminal and filter to things you can actually take on:

# Maintainer-vetted, small, self-contained.
gh issue list --repo firecracker-microvm/firecracker --state open --label "good first issue"

# Active bugs, most-recently-updated first (more likely still valid).
gh issue list --repo firecracker-microvm/firecracker --state open --label "Type: Bug" \
  --limit 40

Note: Verify the exact label strings on the tracker for your point in time — Firecracker uses colon-namespaced labels (Type: Bug, Status: Awaiting review, Priority: High) and these occasionally get renamed. If --label "Type: Bug" returns nothing, list labels with gh label list --repo firecracker-microvm/firecracker | rg -i bug.

Pick something that is (a) in a subsystem you can read (config validation, an API status code, a virtio device edge case — not a snapshot-format change or the seccomp boundary), (b) has a concrete symptom, and (c) is small. Then read the entire thread and write a triage note:

Issue:   #NNNNN — <title>
Symptom (reporter's words): <quote the failing behavior verbatim>
Repro given by reporter?    <yes/no; paste their commands if any>
Suspected subsystem:        <e.g. machine-config validation / virtio-block / boot-source parsing>
Assignee / linked PR?       <none — check the sidebar and `gh issue view --comments`>
My plan:                    reproduce via <config-file | curl | pytest | cargo test>, then root-cause.

Check for a live PR before you commit a day:

gh issue view NNNNN --repo firecracker-microvm/firecracker --comments
gh pr list --repo firecracker-microvm/firecracker --search "NNNNN in:body"

Warning: Do not start work on an issue someone is actively working. If there is a recent "I'll take this" comment or a linked open PR, pick another. Then comment that you are investigating — announce intent, then work.

Note (no clean issue right now?): This lab works equally well on a representative, evergreen bug class, which we use as the worked example below: the API accepts an out-of-range or malformed configuration value that should be rejected, and the bad value surfaces later as a confusing failure instead of a clean 400 at configuration time. That is the literal shape of many real Firecracker validation issues. We make it concrete with machine-config.

Step 2 — Reproduce it manually first, to see it (20 min)

Always start manual. You must watch the bug happen before you encode it. Boot the VMM and drive the reported scenario over the socket exactly as in Lab 1.3:

API=/tmp/fc-repro.socket
rm -f "$API"
sudo ./build/cargo_target/x86_64-unknown-linux-musl/debug/firecracker --api-sock "$API" &
FC_PID=$!

Now drive the suspicious request. For the worked example — does machine-config reject an absurd mem_size_mib, or does it accept it and fail confusingly at boot?

# Configure a kernel + rootfs first (so the only variable is the bad value).
curl -s -X PUT --unix-socket "$API" --data \
  '{"kernel_image_path":"./vmlinux-6.1.x","boot_args":"console=ttyS0 reboot=k panic=1"}' \
  http://localhost/boot-source
curl -s -X PUT --unix-socket "$API" --data \
  '{"drive_id":"rootfs","path_on_host":"./ubuntu-24.04.ext4","is_root_device":true,"is_read_only":false}' \
  http://localhost/drives/rootfs

# The suspicious request: an out-of-range machine config. Capture status AND body.
curl -s -i -X PUT --unix-socket "$API" --data \
  '{"vcpu_count":0,"mem_size_mib":0}' \
  http://localhost/machine-config

# And, separately, the path the reporter says is broken — e.g. it is accepted, then InstanceStart fails.
curl -s -i -X PUT --unix-socket "$API" --data '{"action_type":"InstanceStart"}' \
  http://localhost/actions

Record exactly what you see: the HTTP status, the JSON fault_message, or the wrong behavior. That recorded output is your symptom baseline — paste it into the triage note. Tear down:

kill "$FC_PID" 2>/dev/null; rm -f "$API"

Tip: If the symptom is a log line or a panic message rather than an HTTP response, run the VMM with --log-path /tmp/fc.log --level Debug and watch the log. Then grep the codebase for a fixed substring of that message — it is the fastest jump to the responsible code: rg -n "a unique phrase from the message" src/.

Step 3 — Separate symptom from trigger conditions (20 min)

A bug report says "Firecracker crashed." Your job is to find the minimal conditions that provoke it. Vary one factor at a time and record which ones matter:

The set of factors that must be true to reproduce is your trigger condition; everything else is noise to strip out of the test. This is the heart of the lab: a vague report becomes a precise, minimal statement — for the example, "PUT /machine-config with vcpu_count == 0 or mem_size_mib == 0 is accepted instead of returning 400."

Note: Try the config-file boot path too (--config-file config.json --no-api). The API and config-file paths can validate differently, and a bug present in one may be absent in the other. That difference is itself a finding worth noting in the issue.

Step 4 — Pin version, build, and (if randomized) seed (5 min)

Reproducibility means the same inputs every run:

• Commit: the git rev-parse --short HEAD from prerequisites — state it in the issue/PR.
• Build flavor: debug vs release (tools/devtool build defaults to musl). Some bugs only show in one; say which you used.
• Kernel/rootfs: which vmlinux-X.Y.Z and which rootfs (a guest-kernel bug can masquerade as a Firecracker bug). Record the artifact names.
• Host: KVM is hardware-specific — a CPUID/MSR or KVM-feature bug may be host-dependent. Note the CPU and kernel: uname -r; rg -m1 "model name" /proc/cpuinfo.

Step 5 — Promote the manual repro to a deterministic test (30 min)

Manual curl is for seeing the bug. Ship code. Choose the lowest-cost harness that reliably reproduces it:

For the worked example the bug is in config validation — pure logic that has a unit-test home, and a behavior worth an integration test too. First locate the validation (never trust a line number):

# Where machine config is parsed and where MachineConfig lives.
rg -n "struct MachineConfig\|fn .*machine.?config" src/vmm/src/vmm_config/
rg -n "vcpu_count\|mem_size_mib" src/vmm/src/vmm_config/machine_config.rs

# Where the validation would (or should) reject bad values, and the error enum.
rg -n "enum .*Error\|InvalidVcpuCount\|InvalidMemorySize\|too small\|must be" \
  src/vmm/src/vmm_config/machine_config.rs

Then write a unit test that asserts the desired behavior so it fails on main (where the validation is missing or wrong). Put it in the module's test block — find the existing one to match style:

rg -n "#\[cfg\(test\)\]\|mod tests" src/vmm/src/vmm_config/machine_config.rs

#![allow(unused)]
fn main() {
// In src/vmm/src/vmm_config/machine_config.rs, inside `mod tests`.
// On `main` this is RED if the validation is missing; GREEN after the Lab 8.2 fix.
#[test]
fn test_machine_config_rejects_zero_vcpus() {
    // Build the config the way the parser does (match the real struct fields on your branch).
    let json = r#"{ "vcpu_count": 0, "mem_size_mib": 1024 }"#;
    let parsed: Result<MachineConfig, _> = serde_json::from_str(json);
    // Parsing may succeed; the *validation* must reject it. Adjust to the real validation entry point.
    let cfg = parsed.expect("valid JSON");
    let err = cfg.validate().expect_err("vcpu_count == 0 must be rejected");
    assert!(
        format!("{err}").contains("vcpu"),
        "error should name the offending field, got: {err}"
    );
}

#[test]
fn test_machine_config_rejects_zero_memory() {
    let cfg = MachineConfig { vcpu_count: 1, mem_size_mib: 0, ..Default::default() };
    let err = cfg.validate().expect_err("mem_size_mib == 0 must be rejected");
    assert!(format!("{err}").contains("mem"), "error should name memory, got: {err}");
}
}

Warning: The struct fields, the validate() entry point, and the error type are version sensitive — the exact name of the validation function and error enum drift between branches (verify on your branch). Use the rg commands above to find the real names and adapt the test. The shape — assert the rejection, assert the message names the field — is what matters.

Run it and confirm it is red on main:

cargo test -p vmm test_machine_config_rejects_zero
# or through the dev container, matching the project's flow:
tools/devtool test -- --no-pytest 2>/dev/null || cargo test -p vmm test_machine_config_rejects_zero

If the test passes on main, the validation already exists — you reproduced the wrong thing, or the issue is already fixed. Go back to Step 3 and re-derive the trigger conditions.

Step 6 — (For behavioral bugs) a pytest integration reproducer (25 min)

If the bug only manifests on a running microVM — a boot failure, a device hang, a wrong API status over the real socket — encode it as a pytest integration test, which is what CONTRIBUTING wants you to ship for new behavior. Find an existing test to copy the fixtures and style:

# The integration suite and its conftest fixtures.
find tests -name "test_*.py" | rg -i "api\|machine\|config\|drive" | head
rg -n "def test_\|microvm\|test_microvm\|api\." tests/integration_tests/functional/ | head -20

A reproducer asserting a clean 400 from the real API (sketch — adapt to the harness's fixtures on your branch; verify the fixture and client names with the rg above):

# tests/integration_tests/functional/test_machine_config_repro.py
def test_zero_vcpu_count_is_rejected(uvm_plain):
    """PUT /machine-config with vcpu_count == 0 must return 400, not be accepted."""
    vm = uvm_plain
    vm.spawn()
    response = vm.api.machine_config.put(vcpu_count=0, mem_size_mib=1024)
    # On `main` (the bug) this may be 204/accepted; after the fix it must be a clean 400.
    assert response.status_code == 400, response.text
    assert "vcpu" in response.text.lower()

Run only your reproducer and confirm it is red on main:

tools/devtool test -- integration_tests/functional/test_machine_config_repro.py

Tip: Keep the integration reproducer as small as possible — no drives, no network, no boot if the bug shows pre-boot. The fewer moving parts, the faster it runs in CI and the easier it is to review.

Step 7 — Document the reproducer (10 min)

Write the repro up so a maintainer (and future-you) can run it in one command. This goes in the issue comment and later in the Lab 8.2 PR:

Reproduced on <commit hash>, debug build (tools/devtool build), kernel vmlinux-6.1.x.

Minimal trigger: PUT /machine-config with vcpu_count == 0 (or mem_size_mib == 0).

Failing test (red on main):
  cargo test -p vmm test_machine_config_rejects_zero
  tools/devtool test -- integration_tests/functional/test_machine_config_repro.py

Observed on main: the value is accepted; the failure surfaces later/confusingly (paste exact output).
Expected: a 400 at config time with a message naming the offending field.

Implementation Requirements

Your deliverable is a reproducer package:

• A triage note (issue link, symptom verbatim, subsystem, assignee/PR check, plan).
• A minimal trigger statement — the stripped-down conditions from Step 3.
• A pinned commit hash, build flavor, and (if relevant) kernel/host details.
• A reproducer that fails on main: a cargo test unit test or a pytest integration test or a recorded config/curl sequence with captured output.
• A one-command way to run it.

Expected Output

A red test on main. For the unit-test reproducer:

running 2 tests
test vmm_config::machine_config::tests::test_machine_config_rejects_zero_memory ... FAILED
test vmm_config::machine_config::tests::test_machine_config_rejects_zero_vcpus ... FAILED

failures:
---- ...test_machine_config_rejects_zero_vcpus ----
called `Result::expect_err` ... : vcpu_count == 0 must be rejected

For the pytest reproducer:

tests/integration_tests/functional/test_machine_config_repro.py::test_zero_vcpu_count_is_rejected FAILED
AssertionError: assert 204 == 400

Plus a documented repro block ready to paste into the issue and the PR.

Troubleshooting

Test passes on main

The bug is already fixed, or you asserted the current (buggy) behavior instead of the desired behavior. Re-read the issue; re-derive the trigger conditions in Step 3. A reproducer that does not fail on main is not a reproducer.

tools/devtool test can't find your pytest file

You may be pointing at the wrong path or missing a fixture. Confirm the file is under tests/integration_tests/ and that the fixture you used (uvm_plain or similar) exists on your branch: rg -n "def uvm_plain\|@pytest.fixture" tests/. Fixture names are version-sensitive.

The struct fields / validate() / error enum don't exist by those names

They drifted. Re-run the rg commands in Step 5 to find the real names on your branch and adapt the test. Name code by role, not by a remembered identifier.

Manual curl shows the bug but the unit test doesn't

The bug is in the API/transport layer, not the pure-logic struct you tested. Move up a harness level to the pytest integration test (Step 6), which drives the real socket.

The VMM won't start

Stale socket or a leftover process: rm -f /tmp/fc-repro.socket; pkill -f firecracker. Re-check ls -l /dev/kvm and your kvm-group membership.

The bug only reproduces sometimes

It is timing- or host-dependent (an EventManager race, a KVM-feature difference). Pin the host details, make the test fail often by tightening the loop or removing sleeps, and say so. An intermittent bug is still a real bug — say it is intermittent and give the reproduce rate.

Stretch Goals

1. Reduce the reproducer to its absolute minimum: fewest fields, no drive, no boot if the bug is pre-boot. A maintainer should read it in 20 seconds.
2. Reproduce at two levels (unit + pytest integration) and decide which you would ship with the fix. Justify it against what CONTRIBUTING requires for new behavior.
3. Reproduce the same bug on the config-file path (--config-file ... --no-api) and note whether the two paths diverge — a divergence is a finding.
4. Bisect to find roughly when the behavior was introduced (or whether it was always present):

   git log -S "the symbol you grepped" -- src/vmm/src/vmm_config/ | head
   

5. Read a recently merged bug-fix PR and find its reproducer. Did the contributor ship the test as part of the PR? gh pr list --repo firecracker-microvm/firecracker --state merged --label "Type: Bug".

Validation / Self-check

Answer without notes. These gate completion:

1. Does your reproducer fail on main? Show the red output and the pinned commit.
2. State the minimal trigger conditions — what must be true, and what is irrelevant?
3. Which harness did you choose (cargo test, pytest, or recorded) and why was it the lowest-cost one that reliably reproduces?
4. What build flavor, kernel, and host did you pin, and why might any of them matter for this bug?
5. How will this exact artifact prove your fix works in Lab 8.2?
6. Could a maintainer reproduce the bug from your write-up alone, in one command?
7. Did you check for an assignee or a linked PR, and announce your intent before working?

Next: Lab 8.2 — Implement the Fix, Write the Test, Open the PR, where you turn this red reproducer into a green, merge-quality PR. For the graded version of reproduction, see Capstone Step 2; for how to read your way to the cause, see reading the codebase.