KVM ioctl Cheat-Sheet

Everything Firecracker does to run a guest is, at the bottom, an ioctl() on a file descriptor under /dev/kvm. This page is the lookup table for those ioctls: which fd level each one applies to (system, VM, or vCPU), what it does in one line, and the kvm-ioctls wrapper Firecracker calls instead of the raw ioctl. It closes with the VcpuExit taxonomy — the reasons KVM_RUN returns to userspace and the kvm_run fields each carries — because decoding a VM exit is the single most common thing you do reading the vCPU run loop.

Reach for this when you are reading the run loop, writing a minimal KVM VMM by hand in Lab 1.4, or staring at a KVM_EXIT reason you don't recognize.

Warning: ioctl numbers (like KVM_GET_API_VERSION = 12) are stable kernel ABI and safe to cite; the kvm_run struct fields, capability checks, and which exits Firecracker actually handles are version-sensitive across kernel and kvm-ioctls versions. Confirm the wrapper API and exit enum against the crate Firecracker pins, not your memory:

# The exact kvm-ioctls / kvm-bindings version Firecracker pins (verify on your branch):
rg -n 'name = "kvm-ioctls"' -A2 Cargo.lock
# Browse the real wrapper API and the VcpuExit enum:
cargo doc -p kvm-ioctls --open       # Kvm / VmFd / VcpuFd / VcpuExit
cargo doc -p kvm-bindings --open      # raw kvm_run, kvm_regs, kvm_sregs, …
# Where Firecracker drives the run loop and matches on VcpuExit:
rg -n "VcpuExit|fn run|KVM_RUN" src/vmm/src/vstate/vcpu/

For the conceptual model — VT-x/AMD-V, EPT/NPT, why the guest is untrusted — read the KVM fundamentals deep dive. For the Rust-wrapper view of these exact calls, read kvm-ioctls and kvm-bindings.

The three fd levels

KVM is a hierarchy of file descriptors. You open one, use it to create the next, and each ioctl is valid only on the right level. Get this hierarchy wrong and the ioctl returns EINVAL.

  open("/dev/kvm")                 ── SYSTEM fd ──  Kvm        (capabilities, API version, create a VM)
        │  KVM_CREATE_VM
        ▼
  VM fd                            ── VM fd ──────  VmFd       (guest memory, IRQ chip, devices, create vCPUs)
        │  KVM_CREATE_VCPU
        ▼
  vCPU fd                          ── vCPU fd ────  VcpuFd     (registers, CPUID, KVM_RUN — runs guest code)

In Firecracker, Kvm/VmFd are set up by vstate/vm.rs and the vCPU is owned by vstate/vcpu/ (rg -n "struct Vm\b|Kvm::new|create_vm" src/vmm/src/vstate/vm.rs).

System-fd ioctls (the Kvm handle)

Note: KVM_GET_SUPPORTED_CPUID is a system-level query (what the host/KVM can offer), but it is applied per-vCPU with KVM_SET_CPUID2. Firecracker reads the supported set, runs it through a CPU template to normalize features across heterogeneous hosts, then sets the filtered result on each vCPU.

VM-fd ioctls (the VmFd handle)

Tip — the two fast paths. KVM_IOEVENTFD and KVM_IRQFD are the heart of virtio performance. A guest "kick" (write to the virtqueue QueueNotify register) is registered as an ioeventfd, so the notify becomes an eventfd the VMM's EventManager epoll loop wakes on — without a KVM_EXIT_MMIO round-trip to userspace. The device's completion interrupt goes back the other way through an irqfd. If virtio went through plain MMIO exits for every notify and every interrupt, it would be far slower. See Lab 7.2.

vCPU-fd ioctls (the VcpuFd handle)

#![allow(unused)]
fn main() {
// The shape of the run loop these ioctls form (illustrative — rg the real one):
//   rg -n "fn run|match .*VcpuExit" src/vmm/src/vstate/vcpu/
loop {
    match vcpu_fd.run()? {                 // KVM_RUN: blocks until a VM exit
        VcpuExit::IoOut(addr, data) => { /* service a PIO write (e.g. serial) */ }
        VcpuExit::MmioRead(addr, data) => { /* a device register read */ }
        VcpuExit::Hlt => break,            // guest executed HLT
        VcpuExit::Shutdown => break,       // triple fault / reset
        other => { /* log + handle or bail */ }
    }
}
}

Note: Initial register state at boot is set with these ioctls. For an x86 vmlinux, Firecracker uses KVM_SET_SREGS to enter long mode directly (cr0 PE|PG, cr4 PAE, efer LME|LMA, identity page tables) and KVM_SET_REGS to set rip = e_entry and rsi = ZERO_PAGE_START — skipping 16-bit real mode entirely. See the boot sequence and the memory layout cheat-sheet.

The kvm_run shared page

KVM_RUN does not pass data by argument — it shares a page. After KVM_CREATE_VCPU, the VMM mmaps a region of KVM_GET_VCPU_MMAP_SIZE bytes over the vCPU fd; that maps struct kvm_run. On every return from KVM_RUN you read kvm_run.exit_reason, then read the union field that matches it. kvm-ioctls hides this: VcpuFd::run() reads the page for you and returns a typed VcpuExit.

  mmap(kvm_run_mmap_size, vcpu_fd)  ──►  struct kvm_run { exit_reason; union { io; mmio; ... } }
                                              ▲
                            KVM_RUN writes exit_reason + the matching union member here

VcpuExit reasons (the taxonomy you dispatch on)

This is the table you keep open while reading the run loop. Each row is a raw KVM_EXIT_* reason, the kvm-ioctls VcpuExit variant Firecracker matches on, the kvm_run fields that carry the payload, and what the VMM does. PIO = x86 in/out (KVM_EXIT_IO); MMIO = a load/store to a device register window (KVM_EXIT_MMIO).

Tip: The exits you'll see constantly in a healthy microVM are IoOut/IoIn (serial console, legacy PIO), MmioRead/MmioWrite (virtio device config registers — the configuration path, not the data fast path, which goes through ioeventfd), and eventually Hlt/Shutdown at teardown. FailEntry and InternalError are never normal — if you see them you have a setup bug (bad register state, an unmapped page the guest jumped to, an emulation hole). Build the full taxonomy hands-on in Lab 4.2 and ../masterclass/kvm-and-vcpus/lab-02-vm-exit-taxonomy.md.

A minimal VMM in ioctls (the whole arc on one page)

Every box below is one ioctl from the tables above. This is exactly the skeleton you build by hand in Lab 1.4 before you ever open Firecracker's Vcpu.

flowchart TD
    A["open(/dev/kvm) — Kvm"] --> B["KVM_GET_API_VERSION == 12"]
    B --> C["KVM_CREATE_VM — VmFd"]
    C --> D["mmap guest RAM + KVM_SET_USER_MEMORY_REGION"]
    D --> E["KVM_CREATE_VCPU — VcpuFd"]
    E --> F["KVM_GET_VCPU_MMAP_SIZE + mmap kvm_run"]
    F --> G["KVM_SET_SREGS / KVM_SET_REGS — initial state (long mode, rip)"]
    G --> H["KVM_SET_CPUID2 — guest-visible features"]
    H --> I["loop: KVM_RUN"]
    I --> J{"VcpuExit?"}
    J -->|IoOut/IoIn| K["service PIO"]
    J -->|MmioRead/Write| L["dispatch to device"]
    J -->|Hlt/Shutdown| M["stop"]
    K --> I
    L --> I

Reading exercise

From a Firecracker checkout (and with cargo doc -p kvm-ioctls open):

# 1. Where does Firecracker create the Kvm/VmFd and check the API version?
rg -n "Kvm::new|create_vm|check_extension|get_api_version" src/vmm/src/vstate/vm.rs

# 2. Where is the run loop, and which VcpuExit variants does it match?
rg -n "fn run|match .*VcpuExit|VcpuExit::" src/vmm/src/vstate/vcpu/

# 3. Where is guest memory registered with KVM_SET_USER_MEMORY_REGION?
rg -n "set_user_memory_region|GuestMemoryMmap|create_guest_memory" src/vmm/src/vstate/memory.rs

# 4. Where are ioeventfd / irqfd registered (the virtio fast path)?
rg -n "register_ioevent|register_irqfd|IoEventAddress" src/vmm/src/

Questions to answer from what you find:

1. Which fd level does each of KVM_CREATE_VCPU, KVM_SET_USER_MEMORY_REGION, and KVM_RUN require, and what is the kvm-ioctls type that owns each?
2. When a guest writes a virtio device's QueueNotify register, does that produce a KVM_EXIT_MMIO exit or an ioeventfd signal — and why does the answer matter for performance?
3. Which two VcpuExit variants should never appear in a healthy microVM, and what class of bug does each indicate?
4. For a MmioWrite exit, which kvm_run.mmio fields tell the VMM the address, length, and that it is a write?

Related references

• Memory Layout Cheat-Sheet — what the addresses in KVM_SET_USER_MEMORY_REGION and the boot registers point at.
• Key Types by Crate — Vcpu/KvmVcpu/Vm and where the run loop lives.
• API Endpoint Map — InstanceStart is what ultimately drives the vCPU threads into KVM_RUN.
• KVM fundamentals and vCPU run loop & VM exits — the prose behind this table.
• kvm-ioctls and kvm-bindings — the Rust wrapper crate view, with cargo doc.

Next: Memory Layout Cheat-Sheet — where in guest physical memory each of these addresses lives.