Lab I4: Bug Attribution — Firecracker vs. Guest Kernel vs. KVM vs. Host vs. Orchestrator

Background

This is the keystone cross-component skill, and the most valuable single thing this entire section teaches. A microVM misbehaves — it hangs at boot, panics, runs slow, loses its network, refuses to snapshot. Before anyone writes a line of code, someone must decide which layer owns the failure:

File it on the wrong layer and the maintainer spends a week bouncing it before anyone touches the real bug. The new contributor's reflex is "the VM is broken, therefore Firecracker is broken." It is usually wrong: Firecracker is a small, well-tested, memory-safe program, while the guest kernel, the host config, and the orchestrator are large, variable, and operator-controlled. The prior probability is against Firecracker. Your job is to turn that prior into evidence.

This lab gives you the layered diagnostic method (which instrument reads which boundary), a decision tree, the "reproduce-with-the-suspect-removed" bisection, and four worked examples — one per non-Firecracker layer — so you learn what a real Firecracker bug looks like by contrast. It depends on everything before it: I1 (the orchestrator boundary), I2 (the host/jailer boundary), I3 (the SDK boundary), and the debugging skills from Level 8 and the debugging masterclass.

Why This Lab Matters for Contributors

Attribution is where cross-component contributors earn their reputation. A maintainer who consistently attributes correctly — and attaches a minimal repro for the right layer — is trusted, and their issues get picked up fast. The bisection technique here ("reproduce the symptom with the suspect removed or replaced") is your most powerful tool, because it converts an opinion ("I think it's Firecracker") into a proof ("here is the same failure with a stock kernel on a stock host with no jailer and raw curl — Firecracker is the only thing left"). On a project where the maintainers are a single AWS team with a high bar and two-approval merges, a habit of crisp attribution is how you get taken seriously.

Prerequisites

The layered diagnostic method: which instrument reads which boundary

The core discipline: each boundary has its own instrument. Reaching for the wrong one wastes hours. Internalize this table — it is the lab.

   ┌─ serial console ──────► guest kernel / userspace messages
   │
   ├─ guest dmesg ─────────► guest-side device & driver state
   │
   ├─ FC log + metrics ────► what the VMM did and the errors IT raised
   │                          (the boundary BETWEEN guest and host)
   │
   ├─ strace -f on FC ─────► the host syscall that actually failed
   │
   ├─ kvm:* trace events ──► what KVM did with each ioctl / vCPU exit
   │
   └─ host dmesg / cgroup ─► the host's own view (perms, OOM, throttle)

The mistake to never make: reading the guest's dmesg to diagnose a host problem, or the FC log to diagnose a guest-kernel problem. A guest kernel panic is in the guest — it shows on the serial console, and Firecracker's log will say almost nothing because, from the VMM's perspective, the vCPU is just running guest code that chose to halt. Match the instrument to the layer.

Enable every instrument before you need it

Set these up at the start of any investigation so all five views are available at once:

# 1. Serial console — guest output. Ensure console=ttyS0 in boot args; FC writes it to stdout.
#    (Capture FC's stdout/stderr to a file.)

# 2. Firecracker log + metrics — the VMM's own view. Configure BEFORE InstanceStart:
API=/tmp/fc.sock
curl -X PUT --unix-socket $API --data \
  '{"log_path":"fc.log","level":"Debug","show_level":true,"show_log_origin":true}' \
  http://localhost/logger
curl -X PUT --unix-socket $API --data '{"metrics_path":"fc-metrics.json"}' \
  http://localhost/metrics

# 3. strace, attached to the FC process (or launch FC under strace -f).
FC_PID=$(pgrep -n firecracker); sudo strace -f -p "$FC_PID" -e trace=ioctl,openat,mmap 2>strace.log &

# 4. KVM trace events (host kernel). Names vary by host kernel — list them first.
sudo cat /sys/kernel/debug/tracing/available_events | rg '^kvm:' | head
sudo trace-cmd record -e 'kvm:kvm_entry' -e 'kvm:kvm_exit' -e 'kvm:kvm_mmio' &  # then trace-cmd report

# 5. Host view.
ls -l /dev/kvm; dmesg | tail            # host dmesg, not guest

Note: kvm:kvm_exit with its reason field is gold for vCPU-exit attribution — it tells you why the guest left the CPU back to KVM. Cross-reference exit reasons against the vCPU run loop deep dive and the KVM ioctl cheat-sheet.

The decision tree

flowchart TD
  S[microVM misbehaves]
  S --> Q1{Does firecracker even start<br/>and create the VM?}
  Q1 -->|No: error before InstanceStart| Q2{strace shows a failed<br/>host syscall?}
  Q2 -->|open/ioctl on /dev/kvm EACCES/ENOENT| HOST[HOST/ORCH: perms or jailer chroot<br/>missing /dev/kvm]
  Q2 -->|KVM_* ioctl returns errno| KVM[KVM/HOST: unsupported feature<br/>or wrong kernel version]
  Q2 -->|API/config rejected with a clean error| FC1[FIRECRACKER: API validation<br/>- or your bad request]
  Q1 -->|Yes: VM starts, vCPUs run| Q3{Anything on the<br/>serial console?}
  Q3 -->|Guest kernel panic / no init / driver error| GK[GUEST KERNEL: config, rootfs,<br/>missing virtio driver]
  Q3 -->|Nothing - guest never printed| Q4{Did the kernel even load?<br/>check FC log for load errors}
  Q4 -->|FC log: ELF/load/cmdline error| FC2[FIRECRACKER: boot config<br/>- or wrong kernel format]
  Q4 -->|Kernel loaded, vCPU spins, no output| Q5{kvm:kvm_exit reason?}
  Q5 -->|FAIL_ENTRY / INTERNAL_ERROR / SHUTDOWN| KVM2[KVM/HARDWARE: bad VM state<br/>or unsupported guest]
  Q5 -->|HLT then nothing, no panic| GK2[GUEST: hung in early boot<br/>console or driver]
  Q1 -->|Yes, runs fine, but...| Q6{Reproduces with raw curl,<br/>stock kernel, no jailer?}
  Q6 -->|Yes, all suspects removed| FC3[FIRECRACKER owns it]
  Q6 -->|No - needs the orchestrator/kernel/host| OTHER[That removed component owns it]

The tree encodes one idea, the same as the OpenSearch attribution lab's: remove or replace suspects one at a time and see whether the symptom survives. Each branch eliminates a layer.

The bisection technique: reproduce with the suspect removed

The single most valuable move. For each suspect layer, there is a way to take it out of the picture.

Tip: The fastest single bisection for "is it my kernel?" is to swap in a stock Firecracker CI kernel + the stock CI rootfs and boot with raw curl. If a clean, well-known kernel+rootfs boots and yours doesn't, the fault is in your guest kernel/rootfs — full stop, no Firecracker involvement. That one swap resolves a huge fraction of "Firecracker won't boot my VM" reports.

Worked Example 1 — Boot hangs: guest kernel vs. Firecracker

Symptom: InstanceStart returns 204, but the guest never reaches a login prompt; the serial console is blank or stops mid-boot.

Bisect:

# 1. Did the kernel even load? (Firecracker's view.)
rg -i "error|fail|cmdline|elf|load" fc.log
#    Clean load logged, vCPUs running → FC did its job; the guest is hung. Go to the console.

# 2. The serial console is the guest's voice.
#    - Kernel panic "VFS: Unable to mount root fs"  → rootfs/boot-args problem (GUEST/rootfs)
#    - Hang after "Run /sbin/init"                  → init/rootfs problem (GUEST/rootfs)
#    - NOTHING printed at all                        → console misconfig or kernel didn't start

# 3. The decisive swap: boot the SAME FC build with the stock CI kernel + rootfs.
#    Boots fine → your kernel/rootfs owns it. Still hangs → now suspect FC.

• If the stock kernel boots and yours doesn't, it's a guest-kernel/rootfs bug — wrong config (missing CONFIG_VIRTIO_*, wrong console), wrong boot_args, or a broken rootfs. Not Firecracker. The fix is in your kernel config, and there's no FC issue to file.
• If nothing prints even with the stock kernel and the FC log shows a load/cmdline error, now you have a candidate Firecracker boot-path bug — confirm against the boot sequence deep dive and check the FC log's exact message before filing.

Attribution most often: guest kernel / rootfs. A blank console is almost never Firecracker. This is the single most misfiled class of issue.

Worked Example 2 — InstanceStart fails immediately: host/KVM vs. Firecracker

Symptom: the VM won't even start; an error comes back before any guest code runs.

Bisect with strace — it names the failing syscall, which names the layer:

sudo strace -f -e trace=openat,ioctl ./firecracker --api-sock /tmp/fc.sock 2>strace.log
# ... drive the API, InstanceStart ...
rg "kvm|KVM|EACCES|ENOENT|EINVAL|ENODEV" strace.log

• openat("/dev/kvm") = -1 EACCES or ENOENT → host/orchestrator: you're not in the kvm group, or (jailed) the jailer didn't mknod /dev/kvm into the chroot (I2). Not Firecracker.
• ioctl(…, KVM_CREATE_VM) = -1 EINVAL/ENODEV → KVM/host: nested virt not enabled, or a host-kernel/KVM limitation. Reproduce on a *.metal / bare-metal host to confirm it's host-specific.
• ioctl(…, KVM_SET_CPUID2 …) failing, or a CPUID/MSR the guest needs being rejected → KVM/host CPU features, often solved by a CPU template; not an FC bug unless FC built a malformed CPUID.
• A clean, structured API error from Firecracker (e.g. "machine config: mem_size_mib must be…") → that's Firecracker's validation working correctly — usually your bad request, occasionally a genuine validation bug.

Attribution most often: host/orchestrator (permissions, chroot, nested virt). A failed /dev/kvm open is a host problem in disguise.

Worked Example 3 — Snapshot restore fails only on another host: KVM/CPU vs. Firecracker

Symptom: a microVM snapshots fine and restores on the source host, but restoring on a different host fails or the guest crashes after resume.

Bisect:

# Compare the two hosts' CPU features — the usual culprit.
diff <(ssh host-a 'cat /proc/cpuinfo | rg flags | head -1') \
     <(ssh host-b 'cat /proc/cpuinfo | rg flags | head -1')
# Was a CPU template applied to normalize features across the fleet?
rg -i "cpu_template|cpu-config" fc.log

• If the destination host lacks a CPU feature the guest saw at snapshot time, the guest resumes into a CPU that no longer offers an instruction it relied on → crash. That is a CPU/KVM/host-heterogeneity problem whose correct fix is a CPU template applied at boot — an operator/config responsibility, documented behavior, not a Firecracker bug.
• If both hosts have identical CPUs and the same FC version still fails to restore, now you have a candidate Firecracker snapshot-compat bug — check the FC version on both ends (snapshots are only guaranteed compatible within the documented window; see the snapshotting deep dive and Stage 8) before filing.

Attribution most often: host CPU heterogeneity (fixable with a CPU template), or a version skew between the snapshotting and restoring Firecracker builds. A true FC snapshot bug requires same-CPU, same-version, and still-broken.

Worked Example 4 — Container won't start, but the microVM boots: orchestrator vs. Firecracker

Symptom (the I1 classic): under firecracker-containerd, ctr run produces a booted microVM but the container never starts.

Bisect by removing the orchestrator:

# Boot the SAME kernel + rootfs by hand with raw curl (no containerd, no shim, no agent path).
#   Boots to a guest shell → Firecracker is fine; the fault is on the agent/vsock plane.
# Then check, in order:
unsquashfs -l rootfs.img | rg agent     # does the rootfs even contain the agent?
rg -i "vsock|metrics" fc-metrics.json   # did FC's vsock device error, or is it idle?

• MicroVM boots by hand, and FC's vsock metrics show no errors → Firecracker transported the channel fine; the agent or runc failed. That's a firecracker-containerd bug (agent plane), filed on that repo, not firecracker.
• FC's vsock metrics show the device dropped or rejected the stream → now Firecracker's vsock device is a candidate; confirm against the vsock deep dive and file with the metrics as evidence.

Attribution most often: orchestrator (the agent/vsock plane), because the boot plane demonstrably worked. The whole value of "boot it by hand" is that it removes the orchestrator in one step.

What a real Firecracker bug looks like

By contrast — so you recognize the genuine article. A real Firecracker bug typically:

• Reproduces with raw firecracker + curl (no orchestrator, no SDK).
• Reproduces with a stock CI kernel and rootfs (the guest is exonerated).
• Reproduces on multiple hosts / kernel versions (the host is exonerated).
• Lives in something Firecracker actually owns: device emulation, the boot/memory setup, the API/action channel, snapshot serialization, rate limiting, MMDS.
• Has a clean repro that pins the exact FC version/commit.

When all the other suspects are removed and the symptom survives, then it's Firecracker — and you can say so with proof, not opinion.

Filing in the right place

The discipline of excluding the other layers from the repro is what proves your attribution. A Firecracker bug report that only reproduces under firecracker-containerd with your custom kernel on your one host hasn't been attributed — it's been punted.

Implementation Requirements / Deliverables

• All five instruments enabled simultaneously on one investigation (serial, FC log+metrics, strace, kvm:* trace, host view).
• The decision tree applied, in writing, to one symptom you actually reproduce.
• For two of the four worked examples, the real bisection run on your host with the verdict and the deciding evidence.
• One "stock kernel + raw curl" swap performed, with its result, for a boot symptom.
• A one-paragraph statement of what a real Firecracker bug would have to show that your symptom does (or doesn't).

Troubleshooting

Nothing on the serial console at all

Check console=ttyS0 is in boot_args and that you captured FC's stdout. No console output is itself a clue (console misconfig or the kernel never executed), not a dead end.

strace shows thousands of ioctl(KVM_RUN) and nothing else

That's a healthy vCPU loop — the guest is running. Your problem is in the guest, not at the FC↔host syscall boundary. Move to the serial console and guest dmesg.

KVM trace events aren't present

The event names differ by host kernel, or tracing isn't mounted. List available_events | rg kvm: first and enable debugfs/tracefs.

You can't tell guest dmesg from host dmesg

That confusion is the bug in your method. Guest dmesg comes from inside the VM (serial or a guest shell); host dmesg is on the host. They describe different layers.

Expected Output

A written attribution of one real symptom you reproduced: the decision-tree path you took, the instrument you read at each step, the suspect you removed to bisect, and a verdict that names exactly one layer — backed by the "reproduce with the suspect removed" evidence rather than a guess.

Stretch Goals

1. Take a real open Firecracker issue and decide, from its description alone, whether it's truly an FC bug or a misfiled guest-kernel/host/orchestrator problem — and what one test would settle it.
2. Build the layered-instrument setup (Step "Enable every instrument") into a single diagnose.sh you could hand to a triager.
3. Deliberately plant one fault per layer (bad /dev/kvm perms, a broken kernel config, a missing agent rootfs) and confirm your decision tree routes each to the right owner.
4. Correlate a single kvm:kvm_exit reason with the matching VM-exit handling in Firecracker's vCPU run loop.

Validation / Self-check

1. State the one-sentence rule behind the entire decision tree.
2. For each of the five layers, name the primary instrument and what it reveals.
3. The serial console is blank and the FC log shows a clean kernel load. Which layer, and what's the deciding swap?
4. openat("/dev/kvm") = -1 EACCES. Which layer, and the two most likely causes?
5. A snapshot restores on host A but crashes the guest on host B with the same FC version. Most likely owner, the fix, and how to confirm it's not an FC bug.
6. A container won't start but the microVM boots. Which plane, which repo, and the one-step bisection that proves it?
7. List the four things a symptom must demonstrate before you're entitled to call it a real Firecracker bug.

Next: Lab I5: Reproducing Integration Bugs — turn an attributed bug into a minimal, version-pinned repro a maintainer can run.