Stage 6 — vCPU and KVM Issues

What class of issue this is

Stage 6 goes beneath every device to the layer that actually runs guest code: the vCPU run loop and the KVM ioctls it is built on. Each vCPU is a thread that mmaps the kvm_run shared page and loops ioctl(vcpufd, KVM_RUN); every time the loop returns, it reads run->exit_reason and dispatches a VM exit — KVM_EXIT_IO (PIO), KVM_EXIT_MMIO, KVM_EXIT_HLT, KVM_EXIT_SHUTDOWN, KVM_EXIT_FAIL_ENTRY, KVM_EXIT_INTERNAL_ERROR. The bugs here are: an unhandled VcpuExit variant that should be handled (or should fail cleanly instead of panicking), an ioctl error swallowed or mishandled, and CPUID/MSR/CPU-template edge cases where the normalization Firecracker applies across heterogeneous hosts is wrong on some hardware.

Concretely, a Stage 6 PR is one of:

• Handling (or cleanly rejecting, instead of unwrap/panic) a VcpuExit variant the run loop doesn't currently cover.
• Fixing error handling around a KVM ioctl (KVM_RUN, KVM_SET_CPUID2, KVM_GET/SET_MSRS, KVM_SET_USER_MEMORY_REGION, …) so a failure surfaces with context instead of aborting.
• A CPUID-leaf or MSR edge case in a CPU template — a leaf masked wrong, an MSR not handled on a given vendor, a topology bit (SMT, core count) computed incorrectly.

Why it's at this difficulty

This is the correctness-critical core. A wrong CPUID leaf can make a guest kernel mis-detect its topology; a mishandled exit can hang or crash a vCPU thread; a swallowed ioctl error can leave the VM in an inconsistent state. You cannot fix these without the KVM model in your head and the ability to reproduce on the relevant hardware (x86_64 and aarch64 differ significantly). Maps to Level 4; read the KVM fundamentals, vCPU run loop & VM exits, and CPU templates & CPUID deep dives first — they are not optional here.

What you must already understand

• The run loop. Find it and read it until you can draw it:

rg -n "fn run|KVM_RUN|VcpuExit|match .*exit|fn run_emulation" src/vmm/src/vstate/vcpu/ | head
rg -n "enum VcpuExit|VcpuExit::" src/vmm/src/vstate/vcpu/  # variants the loop dispatches

• Firecracker uses KVM via rust-vmm kvm-ioctls/kvm-bindings — Kvm, VmFd, VcpuFd, VcpuExit. The VcpuExit enum comes from kvm-ioctls; the run loop matches on it:

rg -n "kvm_ioctls|kvm_bindings|VcpuFd|VmFd" src/vmm/src/vstate/ | head

• CPU config / templates. CPUID and MSRs are normalized in cpu_config/ and per-arch under arch/{x86_64,aarch64}/:

ls src/vmm/src/cpu_config/ src/vmm/src/arch/x86_64/ src/vmm/src/arch/aarch64/
rg -n "KVM_GET_SUPPORTED_CPUID|KVM_SET_CPUID2|cpuid|leaf|msr|KVM_SET_MSRS" src/vmm/src/cpu_config/ src/vmm/src/arch/ | head

• The Kani label. Some vstate logic is model-checked with the Kani formal-verification harness. If your change touches a Kani-proved region, the proof may need updating — check on the tracker.

Representative tasks

How to approach one — worked example: an unhandled VM exit

Illustrative of the pattern. The rg finds the real dispatch match; do not trust line numbers — this is exactly the code that moves between branches.

Symptom: an issue reports that under a particular guest workload the vCPU thread aborts with a panic, and the trace points at the run loop's match on VcpuExit. The guest triggered an exit variant the loop falls through to unreachable!() / a panic, instead of either handling it or shutting the microVM down cleanly.

Step 1 — read the dispatch and reproduce

rg -n "match .*\.run\(\)|VcpuExit::|Ok\(VcpuExit|Err\(e\)" src/vmm/src/vstate/vcpu/mod.rs
git log --oneline -n 8 -- src/vmm/src/vstate/vcpu/

The schematic of the dispatch:

#![allow(unused)]
fn main() {
match self.fd.run() {
    Ok(VcpuExit::IoIn(addr, data))  => { /* PIO read  */ }
    Ok(VcpuExit::IoOut(addr, data)) => { /* PIO write */ }
    Ok(VcpuExit::MmioRead(addr, data))  => { /* MMIO read  */ }
    Ok(VcpuExit::MmioWrite(addr, data)) => { /* MMIO write */ }
    Ok(VcpuExit::Hlt)      => { /* graceful stop  */ }
    Ok(VcpuExit::Shutdown) => { /* reset / shutdown */ }
    Ok(unexpected)         => { panic!("unexpected exit: {:?}", unexpected) }  // <-- the bug
    Err(e)                 => { /* KVM_RUN error handling */ }
}
}

Step 2 — decide the right behaviour, on the issue, before coding

This is the heart of the discussion you open first. For an exit Firecracker genuinely does not emulate (e.g. an MSR access it doesn't model, or an internal error), the correct behaviour is almost never panic! — it is to log with context, bump a metric, and fail the microVM cleanly (a vCPU exit signalling the VMM thread to shut down), so one guest cannot abort the process in a way an orchestrator can't observe. Confirm the intended policy with a maintainer; the threat model (untrusted guest) makes "crash the process" an unacceptable response to guest-triggered exits.

--- a/src/vmm/src/vstate/vcpu/mod.rs
+++ b/src/vmm/src/vstate/vcpu/mod.rs
@@
-            Ok(unexpected) => panic!("unexpected exit: {:?}", unexpected),
+            Ok(unexpected) => {
+                // An exit we do not emulate must not abort the process — a guest could
+                // trigger it deliberately. Record it, then stop this microVM cleanly.
+                METRICS.vcpu.failures.inc();
+                error!("vcpu: unhandled KVM exit {unexpected:?}, stopping microVM");
+                Err(VcpuError::UnhandledKvmExit(format!("{unexpected:?}")))
+            }
@@  pub enum VcpuError {
+    #[error("Unexpected KVM exit received: {0}")]
+    UnhandledKvmExit(String),

Step 3 — handle the KVM error path with equal care

KVM_RUN can also return EINTR (a signal — expected, retry the loop) vs a real error. A common Stage 6 bug is treating EINTR as fatal. Check the existing handling:

rg -n "EINTR|Errno|ENOSYS|retry|continue" src/vmm/src/vstate/vcpu/ | head

Step 4 — test it

vCPU-level behaviour is hard to unit-test directly, so Firecracker has both targeted unit tests and integration tests that drive a guest. Find the existing pattern and extend it:

rg -n "#\[test\].*vcpu|fn test_.*exit|VcpuExit" src/vmm/src/vstate/vcpu/ | head
rg -n "def test_.*vcpu|def test_.*reboot|def test_.*shutdown" tests/integration_tests/ | head
tools/devtool test -- -k vcpu

For a CPUID/MSR template fix, the test asserts the normalized leaf/MSR value — Firecracker has CPUID/MSR snapshot/golden tests; copy that mechanism rather than asserting against live hardware.

Warning: Build and test on both architectures when you can. arch/x86_64 and arch/aarch64 diverge sharply (zero page + CPUID + MSRs on x86 vs FDT + sysregs on ARM). A fix that compiles on x86 may not even build on aarch64 — tools/devtool checkbuild --all covers both.

What a good PR looks like

• Guest-triggered conditions never abort the process. An unhandled exit logs, bumps a metric, and stops the microVM cleanly — never panic! on a path a guest can reach. This is a threat-model requirement, not a style preference.
• EINTR/signal handling is correct in the KVM_RUN error arm — signals are normal, not fatal.
• CPUID/MSR changes are justified against the spec/hardware, not guessed, and pinned by a golden/snapshot test. A wrong leaf can break a whole class of guests.
• Both architectures build and (where relevant) are tested. checkbuild --all.
• The Kani proof is updated if the change touches a model-checked region.
• A discussion preceded the diff; CHANGELOG entry; integration test for new behaviour.

Graduation criteria — ready for Stage 7 when

• You have one merged vCPU/KVM PR — an unhandled exit handled cleanly, an ioctl error path fixed, or a CPUID/MSR/template edge case corrected — with a test that pins the behaviour.
• You can draw the run loop from memory: mmap kvm_run, loop KVM_RUN, dispatch on exit_reason, and name each exit variant's handler with an rg.
• You can explain why a guest-triggered exit must stop the microVM rather than panic the process, in terms of the threat model.
• You can build and reason about both x86_64 and aarch64 paths and know which constructs are arch-specific.

The vCPU runs the guest; the devices serve it. Stage 7 goes into the virtio devices that ride on top of the MMIO exits you just learned to dispatch.

Next: Stage 7 — Virtio Device Issues.