KVM & vCPUs — Intensive

This masterclass extends the KVM fundamentals, vCPU run loop & VM exits, and CPU templates & CPUID deep dives, and the work you did in Lab 1.4 (the ~70-line real-mode KVM VMM) and across Level 4. Those gave you the shape of KVM: three fd levels, a memory region, a vCPU, a run loop over VcpuExit. This intensive goes down to the metal: you build a multi-region, long-mode, multi-vCPU VMM by hand; you instrument a real Firecracker boot and categorize every single VM exit by class and by device; and you drive KVM_GET_SUPPORTED_CPUID, KVM_SET_CPUID2, and the MSR ioctls to see exactly what the guest is told about the CPU and how a template rewrites it.

By the end you can: stand up a 64-bit guest on multiple vCPU threads from /dev/kvm up; predict and then measure the exit profile of a boot down to the register offset; and explain — with the bytes in front of you — how Firecracker normalizes CPUID and MSRs so a snapshot taken on one host resumes safely on another.

Note: Firecracker talks to KVM through the rust-vmm kvm-ioctls crate (Kvm / VmFd / VcpuFd / VcpuExit) and kvm-bindings (the raw structs). You will use the exact same crates in this masterclass's standalone labs, so the APIs you learn here are the APIs you read in src/vmm/src/vstate/. The KVM concepts (VMX/SVM, EPT/NPT, the kvm_run shared page) are kernel features; kvm-ioctls is just the safe wrapper around the ioctl()s.

First principles: what KVM gives you and what it doesn't

KVM is a Linux kernel module that exposes /dev/kvm. It does exactly one hard thing for you: it runs guest instructions on the physical CPU using the hardware virtualization extensions (Intel VT-x / VMX with EPT, AMD-V / SVM with NPT, ARM EL2 with stage-2 translation), and it bounces control back to your userspace VMM whenever the guest does something the VMM must handle — touch an unmapped address, execute an I/O instruction, halt. Everything else — what the guest's memory map looks like, what devices exist, what the CPU reports about itself, when and how a vCPU starts — is the VMM's job. KVM runs the guest; the VMM decides what the guest is.

The interface is ioctl()-driven and layered into three file-descriptor levels. You climb them in order; each ioctl is valid only at its level:

  open("/dev/kvm")                         ── the SYSTEM fd  (capabilities, supported CPUID)
        │  KVM_CREATE_VM
        ▼
  the VM fd                                ── the VM fd      (memory regions, irqchip, ioeventfd)
        │  KVM_CREATE_VCPU  (×N)
        ▼
  one vCPU fd per virtual CPU              ── the vCPU fd    (regs, sregs, cpuid, msrs, KVM_RUN)
        │  mmap(vcpu_fd) -> struct kvm_run
        ▼
  the kvm_run shared page                  ── exit_reason + per-exit union, read after each KVM_RUN

Find the three levels in Firecracker — they are not in one file, which is itself worth understanding:

cd ~/firecracker
# The system fd (Kvm) and VM fd (Vm / VmFd):
rg -n "Kvm::new|create_vm|struct Vm\b|VmFd" src/vmm/src/vstate/vm.rs
# The vCPU fd and run loop:
rg -n "create_vcpu|struct KvmVcpu|struct Vcpu\b|fn run\b|VcpuExit" src/vmm/src/vstate/vcpu/
# The kvm_run mmap size, fetched once at the system level:
rg -n "KVM_GET_VCPU_MMAP_SIZE|run_size|kvm_run" src/vmm/src/vstate/

Note: vmm-sys-util (which provides EventFd and the ioctl macros) is now an external rust-vmm dependency, no longer vendored. If an rg for it comes up empty in src/, it's in Cargo.lock, not the tree — rg -n '^name = "vmm-sys-util"' Cargo.lock.

The kvm_run shared page and the exit taxonomy

After KVM_CREATE_VCPU, you mmap the vCPU fd to get a struct kvm_run — a page KVM and the VMM share. ioctl(vcpufd, KVM_RUN) blocks; when it returns, the guest has stopped and run->exit_reason tells you why. The exit reason selects which arm of a union in kvm_run carries the request. kvm-ioctls decodes that union for you into the VcpuExit enum, so the Rust match you write is a direct mirror of the C switch (run->exit_reason).

# See the variants Firecracker's run loop actually handles, per arch:
rg -n "VcpuExit::" src/vmm/src/vstate/vcpu/
# The x86 and aarch64 exit handlers diverge — read both:
rg -n "match.*run\(\)|VcpuExit::IoIn|VcpuExit::MmioWrite|VcpuExit::Hlt|VcpuExit::Shutdown" \
  src/vmm/src/vstate/vcpu/x86_64.rs src/vmm/src/vstate/vcpu/aarch64.rs

The economics of the whole VMM live in this table. Every exit is a round trip: guest → KVM → your userspace handler → back into KVM_RUN. On the hot path (virtio), Firecracker works hard to avoid exits entirely with KVM_IOEVENTFD (a guest write to a magic address wakes an eventfd the VMM thread polls, without the vCPU ever leaving KVM_RUN) and KVM_IRQFD (inject an interrupt by writing an eventfd). You'll count exactly how many exits a boot and a disk write produce in Lab 2.

sequenceDiagram
    participant V as vCPU thread
    participant K as KVM (kernel)
    participant G as Guest code
    V->>K: ioctl(KVM_RUN)
    K->>G: enter guest (VMX/SVM)
    G->>G: run until a trapping event
    G-->>K: VM exit (IO / MMIO / HLT / ...)
    K-->>V: KVM_RUN returns; read run->exit_reason
    Note over V: match VcpuExit { ... } dispatch to a device
    V->>K: ioctl(KVM_RUN)  (resume)

Guest memory: regions, slots, and the MMIO gap

KVM_SET_USER_MEMORY_REGION is the single most important memory call. It tells KVM "host virtual range userspace_addr of size memory_size backs guest physical range guest_phys_addr, in slot slot." A VMM registers several regions — a real microVM is not one flat block. On x86_64 there is a deliberate hole below 4 GiB where MMIO devices live: guest physical addresses in that hole are not backed by any memory region, so a guest access there faults out as KVM_EXIT_MMIO instead of hitting RAM. That hole is how virtio-MMIO devices get their register windows.

x86_64 guest-physical layout (constants in src/vmm/src/arch/x86_64/layout.rs — verify on your branch)

  0x0000_0000  ┌─────────────────────────────┐
               │ low RAM: real-mode IVT, zero │  ZERO_PAGE_START=0x7000, boot stack ~0x8ff0,
               │ page, cmdline, page tables…  │  CMDLINE_START=0x20000
  0x0010_0000  ├─────────────────────────────┤  HIMEM_START = 1 MiB  ── kernel loads here (e_entry)
               │ high low-RAM region          │
               │ (guest RAM up to the gap)    │
  ~0xC000_0000 ├─────────────────────────────┤  MMIO gap below 4 GiB ── NOT backed by a region;
               │ MMIO window (virtio devices) │  accesses here => KVM_EXIT_MMIO
  0x1_0000_0000├─────────────────────────────┤  4 GiB
               │ high RAM (if mem > the gap)  │  a SECOND memory region above 4 GiB
               └─────────────────────────────┘

# The layout constants and where regions get built:
rg -n "HIMEM_START|MMIO_MEM_START|FIRST_ADDR_PAST_32BITS|GUEST_MEM|MMIO_LEN|layout" \
  src/vmm/src/arch/x86_64/
# Region construction (note it returns MULTIPLE regions when RAM straddles the gap):
rg -n "arch_memory_regions|GuestMemoryMmap|GuestRegionMmap|set_user_memory_region" \
  src/vmm/src/vstate/memory.rs src/vmm/src/arch/

In Lab 1 you register multiple regions and a gap yourself, set up the page tables and segment descriptors that put the vCPU in 64-bit long mode, and run a small protected/long-mode payload — the step up from the single real-mode page of Lab 1.4 that makes the jump to "Firecracker loads a real vmlinux" comprehensible.

Long mode: what the VMM must set up before a 64-bit guest runs

A guest powers on in 16-bit real mode. To run 64-bit code (and a real Linux kernel) the VMM must hand the vCPU a fully-formed long-mode state via KVM_SET_SREGS and KVM_SET_REGS: control registers, segment descriptors, and a page-table hierarchy already built in guest memory. This is the part Lab 1.4 explicitly skipped; it's the part you build here.

# How Firecracker sets long mode for the real kernel — your hand-built version mirrors this:
rg -n "EFER_LME|EFER_LMA|X86_CR0_PE|X86_CR0_PG|X86_CR4_PAE|PML4|pdpte|configure_segments_and_sregs|setup_page_tables" \
  src/vmm/src/arch/x86_64/

Identity-mapping (guest-phys == guest-virt) for the low gigabyte is the trick that keeps the bootstrap simple: the kernel's early e_entry code runs at a virtual address equal to its physical address, so the rip you set and the bytes you loaded line up without a relocation step.

CPUID and MSRs: what the guest is told about the CPU

The guest discovers its CPU by executing the CPUID instruction and reading MSRs (model-specific registers). Both are virtualized, and the VMM controls the answers. The flow is:

1. KVM_GET_SUPPORTED_CPUID (a system-fd ioctl) asks KVM for the full set of CPUID leaves this host+KVM can support — the menu of what's possible.
2. The VMM masks and edits that template: clearing features it won't expose, fixing up topology leaves (cores/threads per the configured vcpu_count and SMT setting), patching the vendor/brand string, and applying any CPU template the operator requested.
3. KVM_SET_CPUID2 installs the edited leaves on each vCPU before the first KVM_RUN. From then on the guest's CPUID returns exactly those values.
4. MSRs follow the same pattern with KVM_GET_MSRS / KVM_SET_MSRS (and KVM_GET_MSR_INDEX_LIST for the supported set).

# The CPUID/MSR plumbing and the cpu_config module that owns templates:
rg -n "GET_SUPPORTED_CPUID|get_supported_cpuid|SET_CPUID2|set_cpuid2|KVM_GET_MSRS|KVM_SET_MSRS|set_msrs" \
  src/vmm/src/
rg -n "normalize|cpuid|Leaf|brand_string|topology" src/vmm/src/cpu_config/
# The tooling that creates/inspects/verifies templates:
rg -n "main|get-cpu-config|fingerprint|template" src/cpu-template-helper/

This is not cosmetic. Two reasons it matters deeply, both of which you'll work hands-on in Lab 3:

• Snapshot portability. A snapshot taken on a Skylake host must resume on an Ice Lake host without the guest noticing the CPU changed underneath it. CPU templates normalize the visible CPUID/MSR set to a common baseline so the guest sees the same CPU before and after restore. Get this wrong and the guest executes an instruction the new host lacks, or a userspace program that cached CPUID at startup misbehaves.
• Security / isolation. Exposing a CPUID feature bit or an MSR the guest can use to leak host state or attack a side channel is a real attack surface. Masking is part of the threat model — the guest should see a minimal, controlled CPU, not the host's raw capabilities.

How this maps onto Firecracker's vstate

Everything above lives under src/vmm/src/vstate/ (the VM/vCPU/memory state) and src/vmm/src/arch/ (the per-architecture boot and CPU setup). The mapping you should carry into every lab:

Read vstate/vcpu/ and arch/x86_64/ with this table open. The labs make each row real by having you build the bare-KVM version first, then rg to the hardened Firecracker counterpart.

Common bugs and symptoms

Validation: prove you understand this

1. Name the three KVM fd levels and one ioctl valid at each. Why is KVM_GET_SUPPORTED_CPUID a system-level ioctl but KVM_SET_CPUID2 a vCPU-level one?
2. Draw the x86_64 guest-physical layout with the MMIO gap. Why must the gap be unbacked by a memory region for virtio-MMIO to work?
3. List the four register/MSR conditions that must hold before a vCPU executes 64-bit code, and name the ioctl that installs them.
4. Explain the full VcpuExit taxonomy: which guest instruction causes each of IO, MMIO, and HLT, and where each carries its data in kvm_run.
5. Walk the CPUID flow: GET_SUPPORTED_CPUID → mask/edit → SET_CPUID2. Give one example of an edit done for correctness and one done for security.
6. Why does a snapshot taken on one host model need CPUID/MSR normalization to resume on another? What goes wrong without it?
7. What do KVM_IOEVENTFD and KVM_IRQFD remove from the exit path, and why is that the difference between a usable and an unusable virtio fast path?

Next: Lab 1 — Build a VM From Scratch. Then Lab 2 — VM-Exit Taxonomy and Lab 3 — CPUID and MSRs. This intensive deepens Level 4 and feeds issue-roadmap Stage 6 (vCPU & KVM).