KVM Fundamentals

Everything Firecracker does ultimately reduces to ioctl() calls on file descriptors derived from /dev/kvm. KVM — the Kernel-based Virtual Machine — is the Linux kernel module that exposes the processor's hardware virtualization extensions (Intel VT-x, AMD-V, ARM virtualization) to userspace. It runs guest code at near-native speed and hands control back to the VMM only when the guest does something the VMM must handle. Firecracker is the userspace half: it decides what the guest is (how much memory, which devices, which kernel) while KVM runs the guest. This chapter teaches you the KVM model that sits beneath the rest of the curriculum — the fd hierarchy, the essential ioctls, the hardware behind them — and shows how Firecracker reaches all of it through the rust-vmm kvm-ioctls crate.

After this chapter you will be able to: explain the system/VM/vCPU fd hierarchy and which ioctl creates each; name the ioctls that map memory, create CPUs, and run the guest, and say what each does; sketch what VT-x/EPT (and the AMD/ARM equivalents) provide; and find every place Firecracker calls into kvm-ioctls.

Note: You should not learn KVM only by reading Firecracker. Build a ~70-line KVM "VMM" by hand first — open /dev/kvm, create a VM and a vCPU, map a page of memory, run it, handle one exit. That is Lab 1.4 and rust-vmm Lab r1. Reading Firecracker's vstate/ is far easier once you have felt the raw ioctls in your fingers.

The fd hierarchy

KVM is organized as three levels of file descriptor, each created from the one above it:

   open("/dev/kvm")                ──►  system fd   (the KVM subsystem; capability queries)
        │  ioctl(KVM_CREATE_VM)
        ▼
     VM fd                          ──►  one virtual machine (memory regions, irqchip, devices)
        │  ioctl(KVM_CREATE_VCPU)
        ▼
   vCPU fd × N                      ──►  one virtual CPU each (registers, the KVM_RUN loop)

# Find where Firecracker opens each level. The rust-vmm wrappers are Kvm / VmFd / VcpuFd.
rg -n "Kvm::new|open.*kvm|KVM_CREATE_VM|create_vm|KVM_CREATE_VCPU|create_vcpu" src/vmm/src/vstate/

Firecracker's wrappers around these live in vstate/:

rg -n "struct Vm\b|struct Vcpu\b|struct KvmVcpu|VmFd|VcpuFd|Kvm" src/vmm/src/vstate/vm.rs src/vmm/src/vstate/vcpu/

The essential ioctls

You do not need all of KVM's ~100 ioctls. You need this core set, grouped by what they do. Firecracker issues each through a kvm-ioctls method, so the table also tells you what to rg for.

# See them in Firecracker. These calls are concentrated in vstate/ and arch/.
rg -n "set_user_memory_region|create_irq_chip|register_irqfd|register_ioevent|set_cpuid2|set_sregs|set_regs" src/vmm/src/

Two of these deserve emphasis now because they are why virtio is fast:

• KVM_IOEVENTFD tells KVM: "when the guest writes value X to this MMIO/PIO address, don't bounce out to userspace — just signal this eventfd." That turns a virtqueue "kick" into an eventfd wake-up the VMM thread's epoll loop handles, with no synchronous VM exit on the vCPU.
• KVM_IRQFD is the reverse: "when userspace writes this eventfd, inject this interrupt into the guest." That is how a device signals completion without the vCPU thread doing anything.

Together they keep the hot path off synchronous, expensive userspace round-trips. See interrupts-and-irqchip.md and virtio-transport-mmio.md.

The hardware underneath

KVM is a thin layer over CPU features. You should know the shape, not the silicon:

The model is the same across all three: the CPU has a privileged "host/root" mode and a guest mode. The VMM (via KVM) configures the per-VM control structure to say which guest events trap to the host, then executes a "VM entry" to run guest code. When the guest hits a trapping event — a privileged instruction, an access to an unmapped or device address — the CPU performs a "VM exit" back to KVM, which either handles it in-kernel or returns to userspace (a KVM_RUN return). Second-level paging (EPT/NPT/ stage-2) is what makes guest physical addresses map to host physical addresses in hardware, so a guest's memory accesses don't trap — they're translated by the MMU. That is why the KVM_SET_USER_MEMORY_REGION buffer can be plain host memory.

One consequence you must know for the boot sequence: a freshly created x86 vCPU starts in 16-bit real mode. To boot an uncompressed vmlinux ELF, Firecracker must set up 64-bit long mode itself — page tables, GDT, and the control-register bits (cr0 PE|PG, cr4 PAE, efer LME|LMA) — via KVM_SET_SREGS/KVM_SET_REGS before the first KVM_RUN. KVM gives you the CPU; Firecracker decides what state it boots in.

How Firecracker reaches all this: kvm-ioctls

Firecracker does not issue raw ioctl() syscalls in most places. It uses two rust-vmm crates:

• kvm-ioctls — safe Rust wrappers: Kvm, VmFd, VcpuFd, and the VcpuExit enum returned by vcpu.run(). Each method corresponds to an ioctl (vm.set_user_memory_region(...), vcpu.set_cpuid2(...), vcpu.run()).
• kvm-bindings — the raw struct and constant definitions generated from the kernel headers (kvm_userspace_memory_region, kvm_regs, kvm_sregs, kvm_cpuid2, the KVM_* constants).

# Confirm the dependencies and where they're imported.
rg -n "kvm-ioctls|kvm-bindings" Cargo.lock src/vmm/Cargo.toml
rg -n "use kvm_ioctls|use kvm_bindings|VcpuExit|Kvm::new" src/vmm/src/

flowchart LR
    FC["Firecracker (vstate/, arch/)"] --> KI["kvm-ioctls: Kvm/VmFd/VcpuFd"]
    FC --> KB["kvm-bindings: structs + KVM_* consts"]
    KI -->|ioctl| KVM["/dev/kvm (kernel module)"]
    KVM -->|VM entry / exit| HW["CPU: VT-x / SVM / EL2 + EPT/NPT"]

This indirection matters for two reasons. First, it is shared with Cloud Hypervisor and other VMMs through rust-vmm, so KVM-level bug fixes and new ioctl support land once and benefit everyone. Second, when you read vstate/ you will see method names, not magic numbers — but they map one-to-one onto the ioctls above, and cargo doc --open -p kvm-ioctls (or the kvm-ioctls deep dive) is the way to confirm a method's exact ioctl. (Crate versions are pinned in Cargo.lock — verify on your branch.)

Reading exercise

# 1. The fd hierarchy in Firecracker.
rg -n "Kvm::new|create_vm|create_vcpu|struct Vm\b|struct Vcpu\b" src/vmm/src/vstate/

# 2. Memory registration.
rg -n "set_user_memory_region|KVM_SET_USER_MEMORY_REGION|kvm_userspace_memory_region" src/vmm/src/

# 3. The run + register ioctls.
rg -n "vcpu.run\(|KVM_RUN|set_regs|set_sregs|set_cpuid2" src/vmm/src/vstate/vcpu/

# 4. The fast-path ioctls.
rg -n "register_ioevent|register_irqfd|KVM_IOEVENTFD|KVM_IRQFD" src/vmm/src/

# 5. The rust-vmm crates.
rg -n "kvm-ioctls|kvm-bindings" Cargo.lock ; rg -n "use kvm_ioctls|VcpuExit" src/vmm/src/

# 6. Confirm your host can do any of this.
ls -l /dev/kvm    # you must be able to read+write it

Answer:

1. Name the three fd levels and the exact ioctl that creates each from the one above.
2. What does KVM_SET_USER_MEMORY_REGION register, and which four fields does it take?
3. Explain KVM_IOEVENTFD and KVM_IRQFD and why each keeps the virtio path off a synchronous VM exit.
4. In what CPU mode does an x86 vCPU start, and what must Firecracker do before it can run a vmlinux ELF?
5. What is the difference between kvm-ioctls and kvm-bindings, and why is sharing them via rust-vmm valuable?
6. What is second-level paging (EPT/NPT/stage-2), and why does it let guest RAM be plain host memory?

Common bugs and symptoms

Validation: prove you understand this

1. Draw the system→VM→vCPU fd hierarchy and label the ioctl on each arrow.
2. List the ioctls needed to go from "open /dev/kvm" to "guest code running," in order.
3. Explain the role of KVM_IOEVENTFD and KVM_IRQFD in making virtio fast, in terms of VM exits.
4. Describe the host/guest CPU mode model and what a VM exit is, using VT-x or SVM terms.
5. Why does Firecracker have to configure long mode itself, and which ioctls does it use to do it?
6. Explain how kvm-ioctls/kvm-bindings relate to raw ioctls and to the broader rust-vmm ecosystem.

Next: The vCPU Run Loop and VM Exits — what a vCPU thread actually does between KVM_RUN calls.