Lab 4.4: Fix a vCPU/KVM State Edge Case

This is a fix-it lab in src/vmm/src/vstate/vcpu/. You will diagnose a representative class of vCPU bug — a KVM ioctl error that surfaces with no context — reproduce it, find the code with rg, fix it with proper error handling, add a unit test in vstate, and trace how a vCPU error propagates from the vCPU thread, through the VcpuResponse channels, to the VMM thread and the user. The class of bug is the most common one at this layer: a Result from a KVM ioctl that loses its meaning on the way up.

The skill is precise and unglamorous: when KVM_RUN or KVM_SET_CPUID2 returns an errno, the user must learn which ioctl failed, on which vCPU, with what value, and the maintainer must be able to find the call site from the message alone. A bare Error: errno 22 is useless. Getting the error context right — and pinning it with a test — is exactly the kind of small, mergeable PR a Level 4 graduate opens.

Background

Firecracker's vCPU code is a stack of Results. A KVM ioctl can fail (the kernel rejects an MSR, a CPUID leaf is malformed, KVM_RUN returns an unexpected exit). The kvm-ioctls crate returns Result<_, kvm_ioctls::Error> (an errno wrapper). Firecracker wraps those in its own error enums and propagates them up:

  KVM ioctl (errno)
        │  kvm-ioctls returns Err(errno)
        ▼
  KvmVcpu method  ──► wraps in a typed VcpuError (with context?)
        │
        ▼
  Vcpu run loop  ──► turns it into a VcpuResponse / a faulted state
        │  over the VcpuResponse channel
        ▼
  VMM thread     ──► logs it, increments a metric, tears the microVM down
        │
        ▼
  the user       ──► sees a log line / an API error / a non-zero exit

The bug class: somewhere in that chain, the error is flattened — a map_err(|_| SomethingError) that throws away the errno, an unhandled VcpuExit variant logged as a generic "unexpected exit," or a ? that propagates a type so broad the user can't tell SET_CPUID2 from SET_MSRS. The fix is to add context: name the ioctl, the vCPU index, and the offending value, and keep the underlying errno.

This lab teaches the motion: reproduce → locate → add context → test → confirm propagation. We use one concrete, realistic scenario, but the technique transfers to every vCPU/KVM error in the file.

Companion reading: vCPU run loop & VM exits, KVM fundamentals, kvm-ioctls & kvm-bindings, CPU templates & CPUID.

Why This Lab Matters for Contributors

"Firecracker died with errno 22 and I have no idea why" is a real, frequent issue shape. The maintainers' first move is to read vstate/vcpu/ and ask "which ioctl, which value?" — and if the error swallowed that, the bug is twice as hard. Improving error context is one of the highest-value, most-mergeable kinds of PR at this layer (it's a whole stage in the issue roadmap and a recurring good first issue shape). This lab is also the bridge from reading the run loop (Lab 4.1) to changing it under test.

Prerequisites

  • Firecracker builds and unit tests run: tools/devtool test -- src/vmm (or cargo test -p vmm).
  • You completed Lab 4.1 — you can find the run loop, the VcpuExit match, and the VcpuResponse channel.
  • You read Lab 4.3 — you know where CPUID/MSRs are set (a common failure site).
mkdir -p ~/fc-notes ; : > ~/fc-notes/reading-log-4.4.md

Note: This lab is written around a teaching regression you plant yourself, so you can verify the fix against the real, correct code. The technique (add ioctl + value context, test it) is exactly what you'd do on a genuine issue. Names and exact error enums are version-sensitive — every step gives you the rg to find the real ones on your branch.

Step 1 (15 min) — Read the error chain end to end

Find the vCPU error types and how a KVM ioctl failure becomes one:

# The error enum(s) for the vCPU.
rg -n "enum .*Error" src/vmm/src/vstate/vcpu/
# Where KVM ioctls are called and their Results wrapped.
rg -n "set_cpuid2|set_msrs|set_regs|set_sregs|\.run\(\)|map_err|\.map_err\(" \
  src/vmm/src/vstate/vcpu/x86_64.rs
# How an error leaves the run loop toward the VMM thread.
rg -n "VcpuResponse|VcpuEmulation|Error|fault|exit" src/vmm/src/vstate/vcpu/mod.rs

In your reading log, write the chain for one concrete ioctl — say set_cpuid2:

  1. The KvmVcpu method that calls vcpu_fd.set_cpuid2(...) and what it does with the Err.
  2. The VcpuError variant it produces (does it keep the errno? the leaf? the vCPU index?).
  3. How that error reaches Vcpu's run loop and becomes a VcpuResponse (or a faulted state).
  4. What the VMM thread does with that VcpuResponse (logs, metric, teardown).

This is the artifact: a four-hop chain from errno to the user, with the function name at each hop.

Step 2 (10 min) — The bug: a flattened error

Here is the planted regression — a realistic "simplification" that throws away the context. Pick the set_cpuid2 (or set_msrs) call site you found and degrade it like this:

--- a/src/vmm/src/vstate/vcpu/x86_64.rs
+++ b/src/vmm/src/vstate/vcpu/x86_64.rs
@@ impl KvmVcpu {
     fn configure_cpuid(&self, cpuid: &CpuId) -> Result<(), VcpuError> {
-        self.fd
-            .set_cpuid2(cpuid)
-            .map_err(|err| VcpuError::SetCpuid2 { source: err, vcpu_index: self.index })?;
+        // "Simplified" — drops the ioctl identity and the vCPU index.
+        self.fd.set_cpuid2(cpuid).map_err(|_| VcpuError::Generic)?;
         Ok(())
     }

Two defects, both realistic and both common in real PRs:

  1. Lost errno + ioctl identity. map_err(|_| VcpuError::Generic) discards which ioctl failed and the underlying errno. A user sees a generic error; a maintainer can't tell SET_CPUID2 from SET_MSRS.
  2. Lost vCPU index. With multiple vCPUs, "which one?" is the first question. Dropping self.index makes a multi-vCPU failure ambiguous.

Note: This is a teaching regression. The real code keeps the source error and context. You're practicing the fix-it motion on a class whose correct behavior you can verify against the actual source on your branch — read what the real map_err does before you plant the bug, so your fix restores it faithfully.

Step 3 (15 min) — Reproduce: make the ioctl fail

You need the ioctl to actually fail so you can see the bad message. The cleanest reproduction is a unit test that feeds an invalid CPUID/MSR set (more reliable than provoking it at runtime), but first confirm the runtime symptom so you understand the user-facing impact.

A wrong-vendor CPU template (from Lab 4.3) makes KVM_SET_CPUID2 or downstream configuration reject the set on some hosts. Boot with a mismatched template and watch the log:

# On an AMD host, deliberately apply an Intel template (or vice versa):
curl -X PUT --unix-socket /tmp/fc.sock \
  --data '{"vcpu_count":2,"mem_size_mib":256,"cpu_template":"T2"}' http://localhost/machine-config
# ... boot-source, drives ...
curl -X PUT --unix-socket /tmp/fc.sock --data '{"action_type":"InstanceStart"}' http://localhost/actions
grep -iE "cpuid|errno|vcpu|set_cpuid|Generic|Error" /tmp/fc.log | tail

With the buggy code, the log says something like VcpuError::Generic / a bare error — no errno, no ioctl name, no vCPU index. With the fixed code, it names the ioctl and the vCPU. Record both messages in your log; the difference is the entire value of the fix.

Tip: If you can't easily provoke the runtime failure on your host (vendor/template alignment), don't force it — the unit test in Step 5 is the real proof. Reproduce conceptually, then pin it in a test.

Step 4 (10 min) — The fix: restore the context

@@ impl KvmVcpu {
     fn configure_cpuid(&self, cpuid: &CpuId) -> Result<(), VcpuError> {
-        self.fd.set_cpuid2(cpuid).map_err(|_| VcpuError::Generic)?;
+        self.fd
+            .set_cpuid2(cpuid)
+            .map_err(|err| VcpuError::SetCpuid2 { source: err, vcpu_index: self.index })?;
         Ok(())
     }

The fix restores three things:

  • The ioctl identity — a dedicated SetCpuid2 variant (so the message names KVM_SET_CPUID2).
  • The underlying errno — source: err keeps the kvm_ioctls::Error, so EINVAL vs ENOMEM is visible.
  • The vCPU index — vcpu_index: self.index, so a multi-vCPU failure is unambiguous.

If the SetCpuid2 variant doesn't already exist on your branch, add it to the VcpuError enum with a #[error(...)] (Firecracker uses thiserror) message that includes the index and source:

#![allow(unused)]
fn main() {
#[error("Failed to set CPUID (KVM_SET_CPUID2) on vCPU {vcpu_index}: {source}")]
SetCpuid2 { source: kvm_ioctls::Error, vcpu_index: u8 },
}

Pitfall — error messages are a contract. Log lines and API error strings are consumed by humans, log scrapers, and runbooks. Name the ioctl the way the kernel does (KVM_SET_CPUID2), keep the errno, and don't churn the wording casually. When you improve one, note it in CHANGELOG.md under Changed/Fixed.

Step 5 (20 min) — The unit test that pins the context

This is the heart of the lab. Find the existing vCPU tests to mirror their setup (they construct a KvmVcpu against a real /dev/kvm in CI):

rg -n "#\[cfg\(test\)\]|mod tests|fn test_|KvmVcpu::new|setup_vcpu|Kvm::new" src/vmm/src/vstate/vcpu/

Write a test that forces the ioctl to fail and asserts the error carries context. The reliable way to force KVM_SET_CPUID2 to fail is to hand it a malformed CPUID set (e.g. too many entries, or an empty/invalid leaf set) — read how the existing tests build a CpuId and invert it:

#![allow(unused)]
fn main() {
#[cfg(test)]
mod tests {
    use super::*;

    // Helper from the existing test module; adapt to your branch.
    fn test_vcpu() -> KvmVcpu {
        // ... existing helper builds a Kvm + Vm + a vCPU index 0 ...
        setup_vcpu(0)
    }

    #[test]
    fn test_set_cpuid2_error_has_context() {
        let vcpu = test_vcpu();

        // Build a deliberately-invalid CPUID set so KVM_SET_CPUID2 returns an errno.
        // (Too many entries is a common, reliable rejection. Verify the cap on your branch.)
        let bad_cpuid = make_oversized_cpuid();

        let err = vcpu
            .configure_cpuid(&bad_cpuid)
            .expect_err("KVM_SET_CPUID2 should reject an oversized CPUID set");

        // The fix's contract: the error names the ioctl AND the vCPU index AND keeps the source.
        match err {
            VcpuError::SetCpuid2 { vcpu_index, .. } => {
                assert_eq!(vcpu_index, 0, "error must carry the vCPU index");
            }
            other => panic!("expected SetCpuid2 with context, got {other:?}"),
        }

        // And the rendered message must be actionable.
        let msg = err.to_string();
        assert!(msg.contains("CPUID"), "message names the ioctl: {msg}");
        assert!(msg.contains("vCPU 0"), "message names the vCPU: {msg}");
    }
}
}

The assertions are the whole point:

  • The buggy code returns VcpuError::Generic → the match panics (the test fails).
  • The fixed code returns VcpuError::SetCpuid2 { vcpu_index: 0, source } → the test passes, and the message assertions pin that the ioctl name and vCPU index can't silently regress.

Run it:

tools/devtool test -- src/vmm/src/vstate/vcpu/ -- --nocapture
# or directly:
cargo test -p vmm --lib vstate::vcpu -- test_set_cpuid2_error_has_context --nocapture

A failing run on the buggy code, then a passing run after the fix, is your proof.

Note: Some vstate tests require /dev/kvm and are gated (they only run on a KVM-capable host / in the dev container). Run them in tools/devtool (inside the container, on a KVM host), not on a machine without /dev/kvm. If make_oversized_cpuid/setup_vcpu don't exist verbatim, read the real test module — it has equivalents; reuse them rather than constructing a KvmVcpu by hand.

Step 6 (10 min) — Confirm propagation to the VMM thread and user

The test pins the type; now confirm the journey. Trace, from the test or by reading, how a VcpuError from configure_cpuid reaches the user:

# Where the run loop / builder consumes the vCPU error.
rg -n "VcpuResponse|VcpuConfigureError|configure|StartMicrovm|build_microvm" src/vmm/src/ | grep -i vcpu
# Where the VMM thread logs/metrics a vCPU failure.
rg -n "METRICS\.vcpu|error!|fatal|exit_code|FcExitCode" src/vmm/src/

In your log, complete the propagation: configure_cpuid → the builder/Vcpu that called it → the VmmAction result / a VcpuResponse / a process exit code → what the user sees (an API error body on a pre-boot failure, or a log line + non-zero exit on a runtime one). The fix's payoff is that every hop now carries the ioctl name and vCPU index instead of Generic.

Implementation Requirements / Deliverables

  • A reading-log error chain: the four hops from errno to the user for set_cpuid2 (or your chosen ioctl), with the function name at each hop.
  • The fix restores the ioctl-specific VcpuError variant, keeps the source errno, and carries the vcpu_index.
  • A unit test in vstate/vcpu/ that forces the ioctl to fail and asserts the error is the context-bearing variant and that its message names the ioctl and vCPU. Fails on the bug, passes on the fix.
  • The before/after log messages recorded, showing the difference (generic vs ioctl + vCPU + errno).
  • A note on the full propagation path to the user, and a CHANGELOG.md entry under Fixed if this were a real PR.

Troubleshooting

The test panics with "no such device" / can't open /dev/kvm

You're running outside a KVM-capable environment. Run it in tools/devtool on a host with /dev/kvm (the dev container passes it through). vstate tests that touch real ioctls are gated for exactly this reason.

KVM_SET_CPUID2 accepts my "invalid" CPUID set

KVM is more permissive than you assumed. Make it clearly invalid: exceed the entry cap (rg -n "KVM_MAX_CPUID_ENTRIES|MAX_CPUID" src/vmm/ ~/.cargo to find the limit), or read how the existing tests provoke a rejection and copy that. The goal is a reliable Err, not a specific errno.

My VcpuError::SetCpuid2 variant doesn't compile

You likely need the #[error(...)] attribute (Firecracker uses thiserror) and the right field types. rg -n "thiserror|#\[error" src/vmm/src/vstate/vcpu/ shows the existing variants' exact shape — mirror one.

The error reaches the user but still looks generic

You fixed one call site; another on the path re-wraps it lossily. Grep the whole chain for map_err(|_| and From impls that flatten — rg -n "map_err\(\|_\|" src/vmm/src/vstate/. A good error survives every hop.

Expected Output

> Task test (vstate::vcpu)
test vstate::vcpu::x86_64::tests::test_set_cpuid2_error_has_context ... ok

# Runtime log, AFTER the fix (context restored):
ERROR Failed to set CPUID (KVM_SET_CPUID2) on vCPU 0: Invalid argument (os error 22)

# Runtime log, BEFORE the fix (the regression you removed):
ERROR VcpuError::Generic

Stretch Goals

  1. Do it for set_msrs. Apply the same motion to KVM_SET_MSRS: a rejected MSR should name the MSR index, the vCPU, and the errno. MSR failures are a classic snapshot-restore bug (Level 9) — restoring an MSR the target host rejects.
  2. Harden an unhandled exit. Find the catch-all arm of the VcpuExit match (Lab 4.1). If an unexpected variant is logged as a generic "unexpected exit," turn it into a typed error that names the variant. Add a test (you can construct a VcpuExit value in a unit test) asserting the message includes the variant.
  3. Metric the failure. Find the METRICS counter for vCPU errors (rg -n "vcpu.*fail|VcpuMetrics" src/vmm/) and confirm your error path increments the right one. A good fix improves both the message and the metric.
  4. Trace a real issue. Find a closed Firecracker issue about an opaque vCPU/KVM error (gh issue list --repo firecracker-microvm/firecracker --search "errno vcpu KVM_RUN" --state closed) and read the PR that fixed it. Compare its approach to yours.

Validation / Self-check

Answer without notes. These gate completion.

  1. Trace the four hops a KVM ioctl errno takes from kvm-ioctls to the user. Name what each hop adds (or should add) to the error.
  2. What three pieces of context must a good vCPU-ioctl error carry, and why is each one the first thing a maintainer asks for?
  3. Why is map_err(|_| VcpuError::Generic) a bug even though the program still "works"? Who pays the cost, and when?
  4. How did you force KVM_SET_CPUID2 to fail in a unit test reliably? Why is a unit test a better proof than a runtime reproduction here?
  5. Your test asserts two things: the error variant and the error message. Why pin both? What regression does each assertion catch?
  6. After your fix, what exactly does the user see for a pre-boot CPUID failure vs a runtime one, and why do they differ?
  7. If this were a real PR, what goes in CHANGELOG.md, under which heading, and why is the wording of the error message itself something you must not churn casually?

When your test fails on the bug and passes on the fix, your before/after log messages show the restored context, and you can recite the propagation chain, you've completed Lab 4.4 — and Level 4. Continue to Level 5: Testing and Debugging.