Lab I5: Reproducing Integration Bugs Across Components

Background

In Lab I4 you learned to attribute a cross-component failure — to name which layer (Firecracker / guest kernel / KVM / host / orchestrator) owns it. Attribution tells you where the bug lives. This lab is about the next, harder step: turning that attributed failure into a minimal reproduction a maintainer can actually run — one that crosses the FC/guest-kernel/KVM/host/orchestrator boundaries you've been studying, pins the exact versions of every component, and reproduces the symptom on a stranger's machine without a paragraph of "oh, you also need…".

This is where most integration bug reports die. An operator hits a real failure, files "snapshot restore crashes the guest" or "container won't start under firecracker-containerd", and attaches their whole production stack — a custom guest kernel, an internal rootfs builder, a host they don't fully describe, an orchestrator at an unstated commit. A maintainer cannot run any of it. The bug is real, but it is unactionable, and it sits in the tracker until it's closed as stale. The skill this lab builds is the opposite: a repro so small and so precisely pinned that a maintainer clones it, runs one script, and sees the failure in five minutes — which is the difference between an issue that gets fixed and one that gets bounced.

A cross-component repro is harder than a single-repo one for a specific reason: the bug only appears at a boundary, so you cannot delete the components — you can only pin and minimize them. You will take an attributed bug, reduce each component to the smallest version that still reproduces, pin all of them explicitly, and package the result as something runnable.

This is a build-it lab.

Why This Lab Matters for Contributors

• A maintainer on a single-vendor project with a high bar and two-approval merges spends their scarcest resource — attention — deciding which issues are actionable. A runnable, version-pinned repro moves your issue to the front of that queue. An un-runnable one moves it to the back, regardless of how real the bug is.
• Cross-component repros are exactly the bugs the maintainers cannot easily produce themselves, because they don't run your orchestrator, your kernel, or your host. A contributor who can hand them a minimal cross-boundary reproduction is doing work they structurally cannot do — which is how cross-component contributors become trusted.
• The discipline — pin everything, minimize each component, prove it reproduces with the suspect isolated — is the same one that turns into a clean PR and a regression test in Level 8 and the capstone. This is that skill applied across repos.

Prerequisites

cd ~/firecracker
B=build/cargo_target/$(uname -m)-unknown-linux-musl/release
# The exact FC version/commit is the FIRST thing a repro must pin.
$B/firecracker --version
git rev-parse HEAD
git describe --tags 2>/dev/null
uname -r ; ls -l /dev/kvm     # host kernel + KVM access — also pinned facts

Step-by-Step Tasks

Step 1: Start from an attributed bug and capture the exact failing configuration

You cannot minimize what you haven't attributed. Begin with a bug for which I4 already told you the owning layer, and capture the full configuration that currently reproduces it — every component, every version — before you cut anything.

# A repro manifest: the exact state of every component. Fill in real values.
cat > repro-manifest.txt <<'EOF'
=== Firecracker ===
version:        $(firecracker --version)
commit:         $(git rev-parse HEAD)
build:          release, musl

=== Guest kernel ===
vmlinux:        vmlinux-6.1.x  (source: CI bucket / your build)
config:         resources/guest_configs/microvm-kernel-...config  (or your config)

=== Rootfs ===
image:          ubuntu-24.04.ext4  (how built: ...)

=== Host ===
host kernel:    $(uname -r)
arch:           $(uname -m)
KVM:            $(ls -l /dev/kvm)
SMT/KSM:        $(cat /sys/devices/system/cpu/smt/active 2>/dev/null), KSM: $(cat /sys/kernel/mm/ksm/run 2>/dev/null)

=== Orchestrator (if any) ===
firecracker-containerd / go-sdk / firectl:  <repo@commit>
EOF
sed 's/\$([^)]*)/<fill-in>/g' repro-manifest.txt   # remind yourself to fill real values

This manifest is the deliverable's backbone. Every line is a variable you will try to pin to a fetchable, named version, and every line is something a maintainer needs to reproduce your result. A repro without this is "works on my machine."

Step 2: Pin every component to an exact, fetchable version

"Pinned" means a maintainer can obtain the identical artifact. Replace every "my custom X" with either a named public artifact or an exact, scripted build. The pinning rules per component:

# Prefer the project's own pinned artifacts — a maintainer already has access to them.
gh release list --repo firecracker-microvm/firecracker --limit 10
# CI kernels/rootfs are referenced by the test suite — find the exact names it pins:
rg -rn "vmlinux-|\.ext4|\.squashfs|artifact|S3|ci-artifacts|spec.ccfc.min" tests/ | head

Tip: The single highest-leverage move is to reproduce with the project's own CI kernel and CI rootfs instead of yours. If the bug survives on stock CI artifacts, the maintainer can run your repro with zero custom artifacts — and you've also proven (per I4) that your kernel/rootfs isn't the cause. If the bug only appears with your custom kernel, then your kernel is part of the repro and you must pin it as a buildable thing, not "my internal image."

Step 3: Minimize each component around the owning layer

Now shrink. For every component that is not the attributed owner, reduce it to the smallest version that still reproduces — fewer drives, no network if irrelevant, the smallest rootfs, the fewest vCPUs, default everything you can. You are not removing components (the bug is at a boundary), you are reducing each to its minimal reproducing form.

# Minimization is bisection on configuration. For each non-owning component, ask:
#   "Does the bug still reproduce if I simplify this?" — and keep the simplest YES.
# Examples:
#   - Drop the network interface       → still repros? remove it from the repro.
#   - 1 vCPU instead of 4              → still repros? pin 1.
#   - 128 MiB instead of 1024          → still repros? pin the smaller.
#   - Stock CI rootfs instead of yours → still repros? use CI rootfs (huge win).
#   - Raw curl instead of the SDK      → still repros? drop the orchestrator (unless it owns it).

The discipline mirrors I4's "reproduce with the suspect removed," but in service of a smaller repro rather than attribution. Keep a log: for each thing you simplified, note whether the bug survived. That log is itself evidence — it shows the maintainer exactly which components are load-bearing for the repro and which were incidental.

Warning: Do not over-minimize past the boundary. If the bug is a snapshot-compat failure that only appears across two FC versions, you cannot minimize to one version — the two-version delta is the bug. Minimize everything else; preserve the boundary that produces the failure. Removing the thing that causes the bug is not minimization, it's losing the repro.

Step 4: Package the repro as a single runnable script

A maintainer should run one thing. Package the pinned, minimized repro as a script (or a short README with copy-paste blocks) that fetches the exact artifacts, configures the microVM verbatim, triggers the symptom, and prints whether it reproduced.

cat > repro.sh <<'SCRIPT'
#!/usr/bin/env bash
set -euo pipefail
# === Repro for: <symptom>, attributed to <layer>, FC <version/commit> ===
# Pinned artifacts (a maintainer must be able to fetch these exactly):
FC_VERSION="v1.16.0"            # or a commit; how to build is in the README
KERNEL="vmlinux-6.1.x"         # CI artifact name
ROOTFS="ubuntu-24.04.ext4"     # CI artifact name or scripted build
# ... fetch/verify artifacts (checksums!) ...

API=/tmp/fc-repro.sock; rm -f "$API"
sudo ./firecracker --api-sock "$API" &
for i in $(seq 1 100); do [ -S "$API" ] && break; done

# Verbatim configuration that reproduces — NOT paraphrased.
curl -sX PUT --unix-socket "$API" --data \
 '{"kernel_image_path":"'"$KERNEL"'","boot_args":"console=ttyS0 reboot=k panic=1 nomodule"}' \
 http://localhost/boot-source
curl -sX PUT --unix-socket "$API" --data \
 '{"drive_id":"rootfs","path_on_host":"'"$ROOTFS"'","is_root_device":true,"is_read_only":false}' \
 http://localhost/drives/rootfs
curl -sX PUT --unix-socket "$API" --data '{"vcpu_count":1,"mem_size_mib":128}' http://localhost/machine-config
curl -sX PUT --unix-socket "$API" --data '{"action_type":"InstanceStart"}' http://localhost/actions

# === Trigger the symptom and report ===
# ... the specific action that triggers the bug (snapshot/load, ctr run, etc.) ...
# ... capture the evidence (serial console, fc.log, the error) ...
echo "EXPECTED: <what should happen>"
echo "ACTUAL:   <the failure> — see fc.log / console capture"
SCRIPT
chmod +x repro.sh

The script must be self-verifying: it states what should happen, triggers the bug, and surfaces the failure. A maintainer running it sees the discrepancy without interpreting your prose.

Step 5: Prove the repro is portable — run it somewhere else

A repro that only works on your host hasn't been pinned, it's been assumed. Run repro.sh on a second machine (a fresh cloud *.metal, a colleague's box, a clean container where applicable) and confirm the symptom appears. Whatever breaks on the second machine is an un-pinned dependency you missed — fix it and re-run.

# On a second, clean host:
#   1. fetch the exact FC build (tag or commit)
#   2. fetch the exact CI kernel + rootfs by name (with checksums)
#   3. run ./repro.sh
#   4. confirm ACTUAL == the failure
# If it DOESN'T reproduce, something on host #1 was load-bearing and unstated:
#     host kernel version? a sysctl? SMT/KSM? a leftover file? Find and pin it.

This step is the actual test of your work. The whole value of a repro is that it travels; if it doesn't, you've documented your machine, not the bug.

Step 6: Assemble the issue a maintainer can act on

Finally, write the issue around the repro. The structure that gets picked up:

# File it in the RIGHT repo (I4's table): firecracker vs firecracker-containerd vs
# your kernel vs your runbook. Search first to avoid duplicates.
gh issue list --repo firecracker-microvm/firecracker --search "<keywords> in:title,body" --state all

The "what I already ruled out" section is what marks you as someone who did the work: it shows the maintainer you attributed, minimized, and pinned — so they can trust the repro and go straight to the fix.

Implementation Requirements / Deliverables

• A repro manifest pinning every component (FC version/commit, guest kernel + config, rootfs, host kernel/arch/KVM, orchestrator commit) to exact, obtainable versions.
• Each non-owning component minimized, with a log of what was simplified and whether the bug survived each simplification.
• A single runnable repro.sh (or verbatim copy-paste steps) that fetches the artifacts, configures the microVM verbatim, triggers the symptom, and reports expected vs actual.
• The repro run successfully on a second host, with any newly-discovered un-pinned dependency identified and pinned.
• A drafted issue with symptom, attribution + evidence, pinned environment, the repro, expected vs actual, and a "what I ruled out" section — filed in the right repo.

Troubleshooting

The bug reproduces on host #1 but not host #2

You missed a pinned dependency — that's the point of Step 5, not a failure of it. The usual culprits: the host kernel version (KVM behavior differs), a sysctl (vm.unprivileged_userfaultfd, vm.max_map_count), SMT/KSM state, a CPU-feature difference, or a leftover file from a previous run. Diff the two hosts' environments and pin whatever differs.

You can't reproduce with the stock CI kernel/rootfs, only with yours

Then your kernel or rootfs is part of the repro (and, per I4, may be the actual owner). Pin it as a buildable artifact — a kernel .config + source ref + build steps, or a scripted rootfs build — not "my internal image." A maintainer must be able to recreate it.

The repro needs the full orchestrator and won't minimize to raw curl

If the bug genuinely only appears under firecracker-containerd, the orchestrator is part of the boundary (per I4 it may own it). Pin the orchestrator at an exact commit and include the minimal ctr run/config that triggers it — and consider whether the issue belongs on the firecracker-containerd repo, not firecracker.

The symptom is intermittent

A flaky cross-component bug still needs a repro — make repro.sh loop until it fires and report the hit rate ("reproduces ~1 in 20 runs"). An intermittent-but-pinned repro is far more actionable than a deterministic-but-unpinned one. See the flaky-test discipline in Level 5.

The artifacts are too large to attach

Don't attach gigabytes — name them. Reference CI artifacts by their exact names and provide checksums and fetch commands. A maintainer with repo access can fetch the same named CI kernel/rootfs you used.

Expected Output

A repository (or gist) containing: repro-manifest.txt with every component pinned; repro.sh that a maintainer runs unchanged; a minimization log; and a drafted issue. The proof of quality is Step 5 — the same repro.sh, run on a second clean host, producing the same failure.

$ ./repro.sh        # on a fresh *.metal host, after fetching the pinned artifacts
... boot ...
... trigger snapshot/load with backend_type Uffd ...
EXPECTED: guest resumes, marker present
ACTUAL:   guest panics on resume — see fc.log:  "<exact error>"
REPRODUCED on host-2 (kernel 6.1.x, FC v1.16.0 commit abc123).

Stretch Goals

1. Containerize the repro. Wrap repro.sh in a container/devcontainer that pins even the host-side tooling, so "clone and run" is literally one command. Note the limits — /dev/kvm and privilege still have to come from the host.
2. Bisect the FC version. If the bug appeared after an upgrade, git bisect across FC commits with repro.sh as the test, and pin the exact commit that introduced it. That commit SHA is gold in the issue.
3. Turn the repro into a regression test. Convert the minimal repro into a pytest integration test (Level 5) that the project could merge — the bug becomes permanently guarded once fixed.
4. Repro across all five layers. Build one repro that deliberately exercises every boundary (orchestrator → jailer → FC → KVM → guest) and document which layer each step pins — a teaching artifact for the next contributor.
5. Take a real stale issue and re-repro it. Find an open, under-specified cross-component issue (gh issue list --repo firecracker-microvm/firecracker --search "no repro OR cannot reproduce in:comments state:open") and produce the pinned, runnable repro it was missing.

Validation / Self-check

Answer without notes; these gate completion:

1. Why is a cross-component repro harder to minimize than a single-repo one, and what do you minimize around?
2. What does "pinned" mean operationally for each of the five components?
3. Why is reproducing with the project's CI kernel + rootfs the highest-leverage single move?
4. When must you not minimize a component away, and why?
5. What is Step 5 (running on a second host) actually testing, and what does a failure there tell you?
6. What does the "what I already ruled out" section prove to a maintainer?
7. The bug is intermittent. What must your repro still provide to be actionable?

When you can take an attributed cross-component bug, pin every component to a fetchable version, minimize around the owning layer, and produce a repro.sh that fires on a second host, you've completed Lab I5 — you can now hand a maintainer something they can run.

Next: Lab I6: Writing Diagnostics for Integration Bugs — instead of reproducing failures by hand, improve the signal Firecracker emits at its boundaries so the next cross-component failure explains itself.