Level 4: KVM, vCPUs, and the Run Loop

This is the level where Firecracker stops being a clever HTTP service and becomes a virtual machine monitor. Everything below the API thread you traced in Level 3 — the VmmAction channel, the EventManager epoll loop, the device managers — exists to set up and feed one thing: a set of vCPU threads, each spinning in a KVM_RUN loop, executing guest code directly on the physical CPU until the hardware exits back into Firecracker to ask for help. This level is about that boundary: the KVM ioctl API, how Firecracker drives it through the rust-vmm kvm-ioctls crate, the VM exit taxonomy, the in-kernel interrupt controller, and how a vCPU's CPUID and MSR state is configured (and normalized with CPU templates).

You already wrote a tiny version of this in Lab 1.4: ~70 lines of Rust that opened /dev/kvm, created a VM and a vCPU, mapped a page of guest memory, loaded a few bytes of machine code, and looped on KVM_RUN printing each exit. That toy is the skeleton of everything in src/vmm/src/vstate/. The difference between your toy and Firecracker is not the KVM API — it is the same dozen ioctls. The difference is everything Firecracker layers on top: a real guest kernel instead of three instructions, a Bus that routes MMIO/PIO exits to virtio and legacy devices, an in-kernel IRQ chip so the guest can take interrupts without exiting, CPUID masking for snapshot portability, and a control protocol so the VMM thread can pause and resume vCPUs.

Warning: Do not start Level 4 until Lab 1.4 is behind you and you can trace API → VMM without a guide (Level 3). This level assumes you know what KVM_RUN, a VM exit, and mmap of the kvm_run shared page are. If those words are fuzzy, re-read the KVM fundamentals deep dive first.

Learning Objectives

By the end of Level 4 you must be able to:

1. Name the three KVM fd levels (system → VM → vCPU) and the ioctls that create each, and explain why guest memory is registered with KVM_SET_USER_MEMORY_REGION rather than passed at KVM_RUN time.
2. Read Firecracker's vCPU run loop in src/vmm/src/vstate/vcpu/ and explain, without a guide, how it calls VcpuFd::run, matches on VcpuExit, and dispatches PIO/MMIO exits to the device buses.
3. Distinguish a KVM_EXIT_IO (PIO) from a KVM_EXIT_MMIO exit: what hardware causes each, how the data is conveyed (io.data_offset vs mmio.phys_addr/data/len/is_write), and which Firecracker bus handles each.
4. Explain how the in-kernel irqchip (KVM_CREATE_IRQCHIP + LAPIC/IOAPIC/PIT) lets the guest take interrupts without a VM exit, and how KVM_IRQFD and KVM_IOEVENTFD form the virtio fast path.
5. Explain why a vCPU's CPUID (KVM_SET_CPUID2) and MSRs must be configured before first run, and why CPU templates exist (consistent guest CPU across heterogeneous hosts; snapshot portability).
6. Trace how a vCPU error (a failed KVM ioctl, an unhandled exit) propagates from the vCPU thread, through the VcpuResponse/event channels, to the VMM thread and the user.
7. Map the rust-vmm kvm-ioctls types (Kvm, VmFd, VcpuFd, VcpuExit) and kvm-bindings structs onto the raw ioctl() calls and struct kvm_run fields they wrap.

The Layer Beneath Everything

KVM (the Kernel-based Virtual Machine) is a Linux kernel module that turns the CPU's hardware virtualization extensions (Intel VT-x, AMD-V, ARM EL2) into a file-descriptor-and-ioctl API exposed at /dev/kvm. Firecracker is a userspace program; it cannot run guest code itself. It asks KVM to do it. The entire relationship is three nested file descriptors and a handful of ioctls:

   userspace (Firecracker)                       kernel (KVM)                hardware
 ┌──────────────────────────┐
 │ open("/dev/kvm")         │ ── system fd ───►  KVM module
 │   Kvm                    │
 │     │ KVM_CREATE_VM      │ ── VM fd ───────►  per-VM state (memory slots, irqchip)
 │     ▼                    │
 │   VmFd                   │
 │     │ KVM_SET_USER_MEMORY_REGION ─────────►  guest RAM = host mmap registered as a slot
 │     │ KVM_CREATE_IRQCHIP / KVM_IRQFD ─────►  in-kernel LAPIC/IOAPIC/PIT
 │     │ KVM_CREATE_VCPU    │ ── vCPU fd ─────►  per-vCPU state (VMCS/VMCB)
 │     ▼                    │
 │   VcpuFd                 │
 │     │ KVM_SET_CPUID2     │ ──────────────►   masked CPUID leaves
 │     │ KVM_SET_REGS/SREGS │ ──────────────►   initial register state
 │     │ KVM_RUN  ──────────┼──────────────────────────────────────────►  runs guest on VT-x/AMD-V
 │     │   (blocks)         │                                              │
 │     ◄── exit_reason ◄────┼──────────────  VM EXIT (IO/MMIO/HLT/...) ◄───┘
 │   read kvm_run shared page, handle exit, loop
 └──────────────────────────┘

Each KVM_RUN is a trip into the guest: KVM loads the vCPU's saved register state into the physical CPU, switches into guest mode (VMX non-root / SVM guest), and lets the guest execute at native speed. It runs until something the hardware can't (or won't) handle by itself — an I/O instruction to an emulated device, an access to an unmapped MMIO region, a HLT, a triple fault. That event is a VM exit: the CPU switches back to host mode, KVM fills in struct kvm_run (a page shared between KVM and Firecracker via mmap), and the KVM_RUN ioctl returns. Firecracker reads kvm_run.exit_reason, does whatever the guest needed (emulate the device write, return the read data), and calls KVM_RUN again. That loop — run, exit, handle, run — is the heartbeat of the microVM, and it is the spine of this entire level. The definitive reference is the kernel's own Documentation/virt/kvm/api.rst; read it alongside the vCPU run loop deep dive.

Note: Firecracker never calls ioctl() directly for these. It uses the rust-vmm kvm-ioctls crate, which wraps the fds as Kvm/VmFd/VcpuFd and returns the exit as a typed VcpuExit enum, and kvm-bindings for the struct kvm_* definitions. Whenever this level names an ioctl, picture the kvm-ioctls method that wraps it (vm.set_user_memory_region(...), vcpu.run(), vcpu.set_cpuid2(...)).

Where Firecracker Keeps vCPU and VM State

Everything in this level lives under one module tree. Find it — don't trust a path you read here:

# The vstate module: VM, vCPU, and guest-memory state.
find src/vmm/src/vstate -name "*.rs" | sort

# The vCPU run loop and the KvmVcpu/Vcpu types.
rg -n "fn run|VcpuExit|fn run_emulation|KVM_RUN" src/vmm/src/vstate/vcpu/

Tip: The split between Vcpu (in mod.rs) and KvmVcpu (in the arch file) is the single most important structural fact of this level. Vcpu is the thread and control plane — it owns the VcpuEvent/VcpuResponse channels, decides when to run, pause, or exit. KvmVcpu is the KVM plane — it owns the VcpuFd, runs KVM_RUN, and turns one VcpuExit into one device dispatch. Keep that boundary in your head; the labs lean on it.

Required Reading

Confirm each exists on your checkout before you start (paths drift between branches):

# Firecracker source you will read this level.
find src/vmm/src/vstate -type f
ls src/vmm/src/cpu_config/ src/vmm/src/arch/x86_64/ 2>/dev/null

# The rust-vmm crate that wraps the ioctls (version pinned in Cargo).
rg -n 'kvm-ioctls|kvm-bindings' Cargo.toml src/vmm/Cargo.toml

Source Code Areas to Inspect

Key Types Quick Reference

Locate each by role, not line number. Run the command, read what it finds.

GitHub Issue Categories for Level 4

The kinds of real issues a Level 4 graduate can credibly take on (find them by label):

gh issue list --repo firecracker-microvm/firecracker --label "Type: Bug" --search "vcpu OR kvm OR cpuid OR MSR OR exit" --state all
gh issue list --repo firecracker-microvm/firecracker --search "CPU template" --state all

Deliverables

Demonstrate all of these before advancing to Level 5:

• A read-through of src/vmm/src/vstate/vcpu/ with a mermaid/ASCII map of one run-loop iteration, and the VcpuExit variants matched, with file citations (Lab 4.1).
• A boot trace that counts and categorizes VM exits (PIO vs MMIO vs HLT), with PIO exits tied to 0x3f8/0x60-0x64 and MMIO exits tied to virtio-MMIO register offsets (Lab 4.2).
• A dump of the host CPUID/MSR config with cpu-template-helper, a static template applied via /machine-config, and the guest's masked CPUID observed from inside (Lab 4.3).
• A reproduced vCPU/KVM edge-case bug, a fix with proper error context, and a unit test in vstate that pins it (Lab 4.4).
• From memory: the three fd levels, the four most common exit reasons, why interrupts mostly don't exit, and why CPU templates exist.

Common Mistakes

How to Verify Success

# 1. You can locate the run loop and the exit match by grep, not memory.
rg -n "fn run|VcpuExit::(IoIn|IoOut|MmioRead|MmioWrite|Hlt|Shutdown)" src/vmm/src/vstate/vcpu/

# 2. You can find where guest memory is registered and the irqchip created.
rg -n "set_user_memory_region|create_irqchip|KVM_CREATE_IRQCHIP" src/vmm/src/vstate/

# 3. You can find where CPUID is pushed into the vCPU and where templates apply.
rg -n "set_cpuid2|SET_CPUID2|CpuConfiguration|apply.*[Tt]emplate" src/vmm/src/

# 4. You can dump a CPU template from the host with the tooling.
ls cpu-template-helper 2>/dev/null || find . -path "*cpu-template-helper*" -name "*.rs" | head

When you can run those four blocks, explain what each line of output means, and answer the validation questions in all four labs without notes, you have Level 4.

PR Profile: Level 4 Graduate

What a graduate of this level can credibly open:

These are the bread-and-butter of vstate contribution — small, sharply-scoped, test-backed changes to the layer where Firecracker meets KVM. The instinct you build here (read the loop, instrument the exits, name the failure precisely) is exactly what Level 8's real-issue contribution demands.

Begin with Lab 4.1: Read the vCPU Run Loop.