kvm-ioctls and kvm-bindings

KVM is an ioctl interface. To create a VM you ioctl(kvm_fd, KVM_CREATE_VM); to run a vCPU you ioctl(vcpu_fd, KVM_RUN); to register guest RAM you fill a struct kvm_userspace_memory_region and ioctl(vm_fd, KVM_SET_USER_MEMORY_REGION, &region). Doing that in Rust means two things: a set of C struct definitions that exactly match the kernel's linux/kvm.h, and a layer of safe wrappers so the rest of the VMM never writes a raw ioctl() by hand. rust-vmm splits those into two crates — kvm-bindings (the raw FFI structs) and kvm-ioctls (the safe RAII wrappers) — and Firecracker builds its entire vstate/ machinery on top of both. This is the foundation of KVM fundamentals and the vCPU run loop; here you learn the crates underneath those chapters.

After this chapter you can: explain what each crate provides and why they are separate; create a VM, register memory, and create + run a vCPU using kvm-ioctls; read the VcpuExit enum and map it to KVM exit reasons; find where Firecracker's vstate/ wraps these types; and explain the serde feature that makes kvm-bindings structs snapshot-serializable.

Note: These crates are the thinnest possible safe layer over the kernel. They add no policy — no memory layout, no boot protocol, no device model. They hand you Kvm/VmFd/VcpuFd and the raw structs, and get out of the way. Everything interesting is what Firecracker builds on top.

Two crates, one boundary

# Confirm both are dependencies and see the pinned versions (verify on your branch):
rg -n "kvm-ioctls|kvm-bindings" Cargo.toml src/vmm/Cargo.toml Cargo.lock
# Read the pinned-version docs locally — better than docs.rs/latest:
cargo doc -p kvm-ioctls -p kvm-bindings --no-deps --open

The split exists because they have different lifecycles. kvm-bindings tracks the kernel uapi — when KVM adds a field or an ioctl, bindgen regenerates. It is mechanical and architecture-specific (kvm-bindings has x86_64 and arm64 module trees). kvm-ioctls tracks the safe API — ergonomic wrappers, error handling, the VcpuExit enum — and changes when the abstraction improves, not when the kernel does. You will touch kvm-bindings when you need a struct the wrapper doesn't expose; you will touch kvm-ioctls for everything routine.

 your VMM code (Firecracker vstate/)
        │  Kvm / VmFd / VcpuFd / VcpuExit         ← safe, owns fds, RAII
        ▼
 ┌─────────────────────┐
 │     kvm-ioctls       │  ioctl_with_ref!(vm_fd, KVM_SET_USER_MEMORY_REGION, &r)
 └─────────────────────┘
        │  raw #[repr(C)] structs + ioctl numbers
        ▼
 ┌─────────────────────┐
 │    kvm-bindings      │  struct kvm_run { ... }, KVM_RUN = _IO(KVMIO, 0x80), ...
 └─────────────────────┘
        │  syscall
        ▼
   /dev/kvm  (Linux KVM)

kvm-ioctls: the RAII fd hierarchy

The three-level KVM fd hierarchy from KVM fundamentals — system → VM → vCPU — maps to three RAII types. Each owns its file descriptor and closes it on drop.

The crown jewel is VcpuExit. VcpuFd::run() calls ioctl(vcpu_fd, KVM_RUN), blocks while the guest executes on the physical CPU, and on return decodes kvm_run.exit_reason into a Rust enum you match on:

# Find the exit variants the crate defines (verify the exact set on your version):
cargo doc -p kvm-ioctls --no-deps --open      # then search VcpuExit
# Or grep the unpacked source:
find ~/.cargo/registry/src -maxdepth 2 -type d -name 'kvm-ioctls-*' \
  -exec rg -n "enum VcpuExit|IoIn|IoOut|MmioRead|MmioWrite|Hlt|Shutdown|FailEntry|InternalError" {} +

This enum is the literal API of the vCPU run loop. When Firecracker's vCPU run loop does match self.fd.run()? { VcpuExit::MmioWrite(addr, data) => ... }, that match arm is a kvm-ioctls variant.

A worked example: a VM and a vCPU from nothing

This is the smallest real program that creates a VM, gives it a page of guest memory containing a tiny bit of code, and runs a vCPU until it exits. It is the exact shape of Lab R1 and of Firecracker's own setup, stripped to the bone.

use kvm_bindings::kvm_userspace_memory_region;
use kvm_ioctls::{Kvm, VcpuExit};

fn main() {
    // 1. System level: open /dev/kvm.
    let kvm = Kvm::new().expect("open /dev/kvm");
    assert_eq!(kvm.get_api_version(), 12); // KVM_GET_API_VERSION is always 12

    // 2. VM level: KVM_CREATE_VM.
    let vm = kvm.create_vm().expect("KVM_CREATE_VM");

    // 3. Guest memory: mmap one page on the host, register it with KVM at GPA 0x1000.
    const MEM_SIZE: usize = 0x4000;
    let load_addr = unsafe {
        libc::mmap(
            std::ptr::null_mut(), MEM_SIZE,
            libc::PROT_READ | libc::PROT_WRITE,
            libc::MAP_ANONYMOUS | libc::MAP_SHARED | libc::MAP_NORESERVE,
            -1, 0,
        ) as *mut u8
    };

    let mem_region = kvm_userspace_memory_region {
        slot: 0,
        guest_phys_addr: 0x1000,
        memory_size: MEM_SIZE as u64,
        userspace_addr: load_addr as u64,
        flags: 0,
    };
    // set_user_memory_region is unsafe: you promise the host mapping is valid.
    unsafe { vm.set_user_memory_region(mem_region).expect("register memory"); }

    // 4. vCPU level: KVM_CREATE_VCPU, then set up registers (omitted: real-mode
    //    code bytes copied into the page, segment/CR setup). See Lab R1.
    let mut vcpu = vm.create_vcpu(0).expect("KVM_CREATE_VCPU");

    // 5. The run loop — this is the heart of every VMM.
    loop {
        match vcpu.run().expect("KVM_RUN") {
            VcpuExit::IoOut(port, data) => {
                println!("guest wrote {:?} to port {:#x}", data, port);
            }
            VcpuExit::Hlt => { println!("guest halted"); break; }
            VcpuExit::MmioWrite(addr, data) => {
                println!("guest MMIO write @ {:#x}: {:?}", addr, data);
            }
            exit => { println!("unhandled exit: {:?}", exit); break; }
        }
    }
}

Note the two unsafe blocks. mmap is unsafe because you are handing KVM a raw host pointer; set_user_memory_region is unsafe because you are promising the mapping outlives the region registration and is valid for memory_size bytes. The rest is safe — create_vm, create_vcpu, run, the VcpuExit match all carry Rust's safety guarantees. That boundary — unsafe only at the memory-registration seam — is the entire value proposition of kvm-ioctls.

Warning: set_user_memory_region being unsafe is not ceremony. If the host mapping is freed while KVM still has the region registered, the guest can make KVM touch freed host memory — a host-side use-after-free. Firecracker's guest memory management exists precisely to make that lifetime correct. vm-memory (vm-memory.md) owns the mappings so this seam stays sound.

How Firecracker wraps these in vstate/

Firecracker never sprinkles raw kvm-ioctls calls through the codebase. It wraps the three fds in its own vstate/ types so the rest of the VMM talks to Firecracker abstractions, and the KVM details stay in one place.

# The KVM/VM/vCPU state lives under vstate/ — find the wrapper types:
rg -n "use kvm_ioctls|Kvm::new|create_vm|create_vcpu|VcpuFd|VmFd" src/vmm/src/vstate/
rg -n "struct Vm\b|struct KvmVcpu|struct Vcpu\b|self\.fd\.run\(\)|VcpuExit" src/vmm/src/vstate/
# Where kvm-bindings structs are filled (regs, sregs, memory region):
rg -n "kvm_userspace_memory_region|kvm_regs|kvm_sregs|set_user_memory_region|set_sregs|set_regs" src/vmm/src/

The pattern is: kvm-ioctls gives the safe fd, Firecracker's vstate/ gives the policy — which memory regions to register, what CPUID to set (CPU templates), how to translate a VcpuExit into a device-bus dispatch, how to serialize the vCPU for a snapshot. That last point connects directly to the next feature.

The serde feature and snapshots

Firecracker's snapshotting has to save and restore the exact KVM state of a paused vCPU: its registers (kvm_regs), special registers (kvm_sregs), CPUID, MSRs, the LAPIC, the FPU, the xsave area — all of which are kvm-bindings structs. To serialize them, kvm-bindings exposes a serde feature that derives Serialize/Deserialize on those #[repr(C)] structs. Firecracker turns it on so its snapshot code can write vCPU state to the microVM-state file and read it back.

# See whether FC enables the serde feature on kvm-bindings:
rg -n "kvm-bindings" Cargo.toml src/vmm/Cargo.toml          # look for features = ["serde", ...] / "fam-wrappers"
# Where vCPU KVM state gets (de)serialized for snapshots:
rg -n "Serialize|Deserialize|kvm_regs|kvm_sregs|VcpuState|save_state|restore_state" src/vmm/src/vstate/vcpu/

Note: This is why a snapshot is only restorable on a compatible host. The serialized blob is literally the kernel's kvm_* structs; restoring on a CPU with different features needs CPUID/MSR normalization — the reason CPU templates exist. The serde feature is the mechanism; CPU templates are the compatibility policy.

Some kvm-bindings structs are FAM structs (flexible-array-member: a header followed by a runtime-length array, like kvm_cpuid2 with nent entries, or kvm_msrs). These are handled via vmm-sys-util's FamStructWrapper (vmm-sys-util.md) — another reason the crates interlock.

Reading exercise

# 1. Confirm both crates and their pinned versions.
rg -n "kvm-ioctls|kvm-bindings" Cargo.toml Cargo.lock

# 2. Read the safe API for the pinned versions.
cargo doc -p kvm-ioctls -p kvm-bindings --no-deps --open

# 3. The VcpuExit variants the run loop matches on.
find ~/.cargo/registry/src -maxdepth 2 -type d -name 'kvm-ioctls-*' \
  -exec rg -n "enum VcpuExit" -A40 {} +

# 4. Firecracker's vstate wrappers over VmFd/VcpuFd.
rg -n "struct Vm\b|struct KvmVcpu|VcpuFd|VmFd|\.run\(\)|VcpuExit::" src/vmm/src/vstate/

# 5. Where kvm-bindings structs get filled (memory region, regs).
rg -n "kvm_userspace_memory_region|set_user_memory_region|kvm_regs|kvm_sregs" src/vmm/src/

# 6. The serde feature path for snapshots.
rg -n "kvm-bindings.*serde|VcpuState|save_state|restore_state" Cargo.toml src/vmm/

Answer:

1. Why are kvm-bindings and kvm-ioctls two crates? What does each track, and when would you edit one versus the other?
2. Name the three RAII fd types, how each is created, and the ioctl behind each creation step.
3. Walk the worked example: which two operations are unsafe, and what unsafe contract does each one carry?
4. Take three VcpuExit variants and say how Firecracker's run loop dispatches each. Where is that match in vstate/?
5. What does the serde feature on kvm-bindings enable, and why does snapshotting need it? How does that connect to CPU templates?
6. What is a FAM struct, give a KVM example, and which crate provides the wrapper for it?

Common bugs and symptoms

Next: vm-memory — the guest-memory model that owns the host mappings you just registered with set_user_memory_region, and lets devices read and write guest RAM safely.