Step 2: Reproduction

You cannot fix what you cannot make fail on demand. A reproduction you can run in one command, that fails the same way every time, is the foundation for everything downstream: it tells the execution-path analysis where to put its breakpoints, it becomes the basis of your regression test, and it is the first thing a maintainer will try when reviewing your PR. A Capstone that reaches Step 5 with no reliable repro is a Capstone built on sand.

This step turns "the issue says it's broken" into "here is the exact build, the exact input, and the exact wrong output — every time."

Goal

Produce a deterministic, minimal reproduction against the exact version the issue was reported on, captured in capstone-work/repro.md precisely enough that another engineer (or your reviewer) can reproduce the bug by copy-paste. Confirm the current main is also affected, or note the version range where it is.

Build the Exact Version First

Firecracker bugs are version-sensitive. The boot path, device set, and snapshot format all evolve, and a bug "fixed" between the reported version and main will waste your day. Pin the version before you do anything else.

# What version did the reporter run? Read it from the issue (gh issue view N), then:
git fetch --tags
git tag | rg "^v1\."            # see available release tags
git checkout v1.X.0            # the reported version (verify the exact tag)

# Build it the way the project builds it — Docker-based devtool, musl by default.
tools/devtool build            # add --release only if the issue is perf/timing-shaped

# The binary lands here (verify arch/libc on your branch):
ls -l build/cargo_target/x86_64-unknown-linux-musl/debug/firecracker

Note: Always confirm the toolchain pin matches the checkout — cat rust-toolchain.toml on that tag (it may differ from main). tools/devtool uses the dev container public.ecr.aws/firecracker/fcuvm:vNN; the tag increments over time (verify), and an old checkout may expect an older container.

Once you reproduce on the reported version, switch to main and reproduce again — your fix lands on main, so the bug must exist there. If main already behaves correctly, the bug was fixed; find the fixing PR with git log --oneline v1.X.0..main -- <suspected/path> and close the loop on the issue rather than re-fixing it.

Choose the Reproduction Mechanism

Firecracker gives you three reproduction surfaces. Pick the smallest one that exhibits the bug — minimality is what makes a repro deterministic and a reviewer trust it.

1. Config-file reproduction

The most self-contained option, and the one reviewers love because it is one file. Build a minimal config.json with only the sections the bug needs (kebab-case; verify field names against src/firecracker/swagger/firecracker.yaml for your branch):

{
  "boot-source": {
    "kernel_image_path": "./vmlinux-6.1.x",
    "boot_args": "console=ttyS0 reboot=k panic=1"
  },
  "drives": [
    { "drive_id": "rootfs", "path_on_host": "./ubuntu-24.04.ext4",
      "is_root_device": true, "is_read_only": false }
  ],
  "machine-config": { "vcpu_count": 2, "mem_size_mib": 1024 }
}

API=/tmp/fc.sock
sudo ./firecracker --no-api --config-file ./config.json
# Observe the failure: a startup error, a wrong serial-console line, a bad exit code.

Strip the config to the minimum that still fails. If the bug reproduces without the network interface, delete the network-interfaces section. Every section you can remove and still see the bug is one fewer variable for your reviewer.

2. curl reproduction

For API-shaped bugs, capture the exact request that misbehaves. The discipline is to send only the requests needed to trigger it, and to record the full response — status code and body — not just "it failed":

API=/tmp/fc.sock
sudo ./firecracker --api-sock $API &

# The single request the issue is about. Capture status + body.
curl -sS -w '\nHTTP %{http_code}\n' -X PUT --unix-socket $API \
  --data '{"vcpu_count":0,"mem_size_mib":1024}' \
  http://localhost/machine-config
# Expected per the issue: a clear 400 with a descriptive fault.
# Actual (the bug): e.g. HTTP 204, then a panic/odd failure at InstanceStart.

Record the observed status and body verbatim in repro.md. The gap between the documented behavior (from firecracker.yaml / docs/) and the observed behavior is the bug statement.

3. pytest reproduction

For internals, the integration harness is the natural home — and a failing test here is half of your Step 6 deliverable already. Find the closest existing test and adapt it; do not write from scratch what the harness already scaffolds:

# Locate tests touching the subsystem (block I/O, machine config, etc.).
rg -n "machine_config|vcpu_count" tests/integration_tests/
# Run one test to confirm the harness works on your box, then add a failing case.
tools/devtool test -- integration_tests/functional/test_api.py -k machine_config

Use the framework's microvm fixtures (the harness builds and boots a microVM for you — see Lab 5.1). Write the assertion that should hold and watch it fail on the buggy build. That failing assertion is the test you will keep.

For a function that is unit-reachable (a virtqueue helper, a config validator, a rate-limiter calculation), prefer a cargo test unit test — it is faster and pins the bug to one function:

rg -n "fn the_buggy_fn" src/
# Add a #[test] in the module's #[cfg(test)] block asserting the correct result.
tools/devtool test -- ...   # or run cargo test for that crate directly

Make It Deterministic

"Fails sometimes" is not reproduced. Before you move on, eliminate the variables.

For a flaky/timing bug, quantify the rate — a repro that fails 2/100 needs a tight loop to be useful, and the rate itself is data for your root-cause doc:

fails=0
for i in $(seq 1 50); do
  rm -f /tmp/fc.sock
  if ! ./repro.sh >/dev/null 2>&1; then fails=$((fails+1)); fi
done
echo "failed $fails/50"

If you can drive the race deterministically (a specific request ordering, a forced scheduling point via log-instrument tracing), do that instead — a deterministic repro of a race is worth ten loops. See Stage 9 and Level 5 Lab 4.

Capture the Repro

Write capstone-work/repro.md so a stranger can reproduce in one pass. It must contain:

• Version: the exact tag/commit SHA you reproduced on, and confirmation that main is affected (or the version range that is).
• Environment: arch, host CPU if relevant, kernel + rootfs filenames, devtool container tag.
• Steps: the minimal config/curl/pytest, copy-pasteable.
• Expected: the documented/intended behavior, with the firecracker.yaml or docs/ citation that defines it.
• Actual: the observed wrong behavior, verbatim (status code, body, serial-log line, panic message, or the failing assertion).
• Failure rate: 100% for a deterministic bug; the measured fraction for a race.

Tip: Save the actual reproduction as an executable artifact too — a repro.sh or the failing test file. You will run it dozens of times across Steps 4–7, and you will paste it (or a link) into the issue and PR so reviewers can confirm independently.

Distinguish Symptom from Trigger (a preview of Step 4)

While reproducing, you are already gathering the raw material for root cause. Keep two things separate in your notes — conflating them is the most common Capstone error:

• The symptom: what you observe (empty buckets, a panic, a wrong byte on the wire, a 204 where you expected a 400).
• The trigger: the precise input conditions that produce it (vcpu_count: 0; a zero-length descriptor; min < kernel-load address; a PATCH after boot).

You do not yet know the cause — that is Step 4. But a sharp record of symptom + trigger is exactly what makes Step 3's trace efficient: you know what to grep for (the trigger value) and what to watch for (the symptom).

Deliverable for Step 2

• capstone-work/repro.md with version, environment, steps, expected, actual, failure rate.
• An executable repro artifact (repro.sh, a config file, or a failing test).
• Confirmation that main is affected (or the documented version range).
• Symptom and trigger stated separately, in one sentence each.

Rubric Hooks

This step drives the Reproduction dimension directly (12 pts): deterministic, minimal, version-pinned, copy-pasteable scores high; "fails sometimes, I think" scores low. It also seeds Tests — a pytest repro is most of an integration test — and Problem articulation in the write-up, which is just repro.md written for humans. See the evaluation rubric.

Validation / Self-check

Before advancing to Step 3:

1. You can reproduce the bug from a clean state in one command, and it fails the same way every time (or at a measured, documented rate).
2. You reproduced on both the reported version and current main.
3. Your repro is minimal — you removed every config section / request / setting that wasn't required to trigger it.
4. You have the actual wrong output captured verbatim, and the expected output with a citation to where it's specified.
5. You can state symptom and trigger as two separate sentences.
6. A repro.sh (or failing test) exists and you can hand it to someone else.

Then go to Step 3: Execution-Path Analysis.