Lab R1: Build a KVM VM with kvm-ioctls

Background

In Level 1 Lab 1.4 you wrote a ~70-line program that opened /dev/kvm, created a VM and a vCPU, mapped a page of guest memory, loaded a hand-assembled real-mode blob, and ran the KVM_RUN loop. That lab's goal was the concept: a VMM is a userspace program driving KVM through ioctl(). This lab has a different goal: the crates. You are going to write a structurally similar program, but this time you treat kvm-ioctls and kvm-bindings as objects of study in their own right — the safe-wrapper boundary, the full VcpuExit taxonomy, the error model, what KvmRunWrapper hides from you, and the exact shape of the types Firecracker's vstate/ builds on. By the end you will read kvm-ioctls' own source as comfortably as your own.

These two crates are the foundation of the entire rust-vmm ecosystem. kvm-bindings is machine-generated bindgen output: the raw C structs and ioctl numbers from <linux/kvm.h>, with no logic. kvm-ioctls is the thin, safe Rust layer on top: it owns the file descriptors as RAII types (Kvm, VmFd, VcpuFd), turns ioctls into methods that return Result, and — crucially — hides the gnarliest piece of KVM, the shared kvm_run mmap page, behind a type called KvmRunWrapper. Firecracker consumes both crates directly; nearly every line of src/vmm/src/vstate/ is a call into them. Understanding them in isolation, in a standalone crate you control, is the fastest way to make vstate/ legible.

Note: This is a standalone Rust program — a crate of your own, not inside the Firecracker tree. You are learning the building blocks on a bench before you read how Firecracker wires them into a production VMM. Every step ends by mapping what you wrote back to a Firecracker file via rg.

Why This Lab Matters for Contributors

• The vCPU run loop is the single most important control flow in Firecracker, and it is a match over VcpuExit. You will exercise far more of that enum here than Lab 1.4 did, so the real run loop reads as familiar.
• The kvm-ioctls & kvm-bindings chapter is the theory of the safe/unsafe boundary; this lab is where you feel it — which calls are unsafe, why set_user_memory_region is unsafe but run is not.
• When you eventually fix a bug in Firecracker's vCPU handling or propose a change to kvm-ioctls upstream, you must know the crate's error model and which exits it surfaces versus swallows. You build that knowledge here.
• Guest memory is "host mmap registered with KVM via KVM_SET_USER_MEMORY_REGION." You will see why that call is unsafe and what invariant you are promising KVM.

Prerequisites

• Lab 1.4 complete and understood. This lab assumes you have already run guest code once and know the three fd levels.
• A Rust toolchain (any recent stable; this standalone crate does not need Firecracker's musl pin) and a readable+writable /dev/kvm on an x86_64 Linux host.
• A Firecracker checkout to rg against (clone it if you have not):

rustc --version
ls -l /dev/kvm                                  # must be rw for you
git clone https://github.com/firecracker-microvm/firecracker.git ~/firecracker 2>/dev/null || true
# Recall the exact versions Firecracker resolves, so your crate API matches what you'll read:
rg -n -A2 '^name = "kvm-ioctls"|^name = "kvm-bindings"' ~/firecracker/Cargo.lock

Tip: Match your crate versions to Firecracker's Cargo.lock where you can. The versions pinned below are current as of mid-2026 — verify on crates.io and bump if the API in Step 3 differs.

Step-by-Step Tasks

Step 1: Create the project and pin the crates

Work outside the Firecracker repo.

cargo new --bin kvm-vm-lab
cd kvm-vm-lab

Cargo.toml — pin kvm-ioctls and kvm-bindings. These pairs move together; a mismatched pair will not compile because kvm-ioctls re-exports types from a specific kvm-bindings:

[package]
name = "kvm-vm-lab"
version = "0.1.0"
edition = "2021"

[dependencies]
# kvm-ioctls 0.25 pairs with kvm-bindings 0.14 (verify on crates.io; bump together).
kvm-ioctls = "0.25"
kvm-bindings = "0.14"
libc = "0.2"

Warning — version skew is the #1 failure here. kvm-ioctls declares its own kvm-bindings dependency. If you pin an incompatible kvm-bindings you get confusing trait/type errors at set_user_memory_region. The safe move: pin kvm-ioctls to a version, run cargo tree -p kvm-bindings to see which kvm-bindings it pulled, and pin that exact one. (verify on crates.io) — both crates are versioned independently and bump often.

cargo add kvm-ioctls@0.25 kvm-bindings@0.14 libc@0.2   # or hand-edit as above
cargo tree -p kvm-bindings                              # confirm the bindings version kvm-ioctls wants

Step 2: Understand the three types and the run page

Before you write code, hold the type hierarchy in your head — it mirrors KVM's fd levels exactly:

 kvm_ioctls::Kvm     ── owns the fd from open("/dev/kvm")        (system level)
        │  .create_vm()  → KVM_CREATE_VM
        ▼
 kvm_ioctls::VmFd    ── owns the VM fd                            (VM level)
        │  .create_vcpu(id)  → KVM_CREATE_VCPU
        ▼
 kvm_ioctls::VcpuFd  ── owns the vCPU fd                          (vCPU level)
        │  .run()  → KVM_RUN  (returns a VcpuExit)
        ▼
 (internally) KvmRunWrapper ── mmap of the shared `struct kvm_run` page

The piece Lab 1.4 glossed over is KvmRunWrapper. When you call vm.create_vcpu(0), kvm-ioctls issues KVM_GET_VCPU_MMAP_SIZE and mmaps the per-vCPU shared struct kvm_run page for you, wrapping it in a private KvmRunWrapper. Every time run() returns, KVM has written exit_reason and a union of exit data into that page; kvm-ioctls reads it and hands you a decoded VcpuExit enum value. The C programmer's ritual of ((char*)run + run->io.data_offset) to find the I/O bytes is done inside the crate — VcpuExit::IoOut(port, data) gives you the slice directly. That single abstraction is the reason Firecracker's run loop is a clean match.

Note: You will never construct or touch a KvmRunWrapper directly — it is crate-private. But knowing it exists explains why VcpuFd::run(&mut self) takes &mut self (it mutates the shared page) and why the returned slices borrow from the vCPU.

Step 3: Write the program

Replace src/main.rs. This is longer than Lab 1.4's version on purpose: it exercises more of the crate surface — multiple VcpuExit arms, explicit error handling instead of blanket .expect(), and a vm-memory-free manual mmap so you see the raw KVM_SET_USER_MEMORY_REGION call (Lab R2 and R3 switch to vm-memory's GuestMemoryMmap, the type Firecracker actually uses).

use kvm_bindings::{kvm_userspace_memory_region, KVM_MEM_LOG_DIRTY_PAGES};
use kvm_ioctls::{Kvm, VcpuExit, VmFd};
use std::io::Write;

const GUEST_PHYS_ADDR: u64 = 0x1000; // place RAM at 4 KiB, not 0 (closer to a real layout)
const MEM_SIZE: usize = 0x4000; // 16 KiB

// Real-mode code: print a NUL-terminated string at offset 0x100 to COM1 (0x3f8),
// then read port 0x3f8 once (to demonstrate an IoIn exit), then `hlt`.
//
//   be 00 01        mov si, 0x0100      ; SI = string offset within this region
//   ba f8 03        mov dx, 0x3f8       ; DX = COM1
// print:
//   ac              lodsb               ; AL = [SI]; SI++
//   3c 00           cmp al, 0
//   74 04           je read_one         ; +4 -> 'in al, dx' at 0x0e
//   ee              out dx, al          ; -> VcpuExit::IoOut
//   eb f8           jmp print           ; -8 -> 'print'
// read_one:
//   ec              in al, dx           ; -> VcpuExit::IoIn (we just demo it)
//   f4              hlt                 ; -> VcpuExit::Hlt
const GUEST_CODE: [u8; 16] = [
    0xbe, 0x00, 0x01, // [0x00] mov si, 0x0100
    0xba, 0xf8, 0x03, // [0x03] mov dx, 0x3f8
    0xac, //             [0x06] lodsb            (print:)
    0x3c, 0x00, //       [0x07] cmp al, 0
    0x74, 0x04, //       [0x09] je read_one  (rel8 from 0x0b; +4 -> 0x0f)
    0xee, //             [0x0b] out dx, al
    0xeb, 0xf8, //       [0x0c] jmp print   (rel8 from 0x0e; -8 -> 0x06)
    0xec, //             [0x0e] in al, dx       (read_one:)
    0xf4, //             [0x0f] hlt
];

fn main() -> Result<(), Box<dyn std::error::Error>> {
    // ── System level ────────────────────────────────────────────────────────
    // Kvm::new() opens /dev/kvm and returns the RAII owner of that fd.
    let kvm = Kvm::new()?;
    // KVM_GET_API_VERSION must be 12 — the stable KVM ABI version.
    assert_eq!(kvm.get_api_version(), 12, "unexpected KVM API version");

    // ── VM level ────────────────────────────────────────────────────────────
    let vm: VmFd = kvm.create_vm()?;

    // ── Guest memory: host mmap, then register with KVM ─────────────────────
    // We mmap MEM_SIZE bytes of anonymous host memory; THIS is the guest's RAM.
    let host_addr = unsafe {
        libc::mmap(
            std::ptr::null_mut(),
            MEM_SIZE,
            libc::PROT_READ | libc::PROT_WRITE,
            libc::MAP_PRIVATE | libc::MAP_ANONYMOUS,
            -1,
            0,
        )
    };
    assert_ne!(host_addr, libc::MAP_FAILED, "mmap failed");
    let guest_mem = unsafe { std::slice::from_raw_parts_mut(host_addr as *mut u8, MEM_SIZE) };

    // Lay out the region: code at offset 0, the string at offset 0x100.
    guest_mem[..GUEST_CODE.len()].copy_from_slice(&GUEST_CODE);
    let msg = b"Hello from kvm-ioctls!\n\0";
    guest_mem[0x100..0x100 + msg.len()].copy_from_slice(msg);

    // KVM_SET_USER_MEMORY_REGION. This call is `unsafe` because we are promising KVM
    // that `userspace_addr .. +memory_size` is a valid, live host mapping for the
    // lifetime of the region. Violating that is a host memory-safety bug.
    let region = kvm_userspace_memory_region {
        slot: 0,
        guest_phys_addr: GUEST_PHYS_ADDR,
        memory_size: MEM_SIZE as u64,
        userspace_addr: host_addr as u64,
        // flags: 0 normally. Set KVM_MEM_LOG_DIRTY_PAGES to enable the dirty-page
        // bitmap that diff snapshots rely on (Stretch Goal 3).
        flags: 0,
    };
    let _ = KVM_MEM_LOG_DIRTY_PAGES; // referenced so the import is meaningful; see Stretch Goal 3
    unsafe { vm.set_user_memory_region(region)? };

    // ── vCPU level ──────────────────────────────────────────────────────────
    let mut vcpu = vm.create_vcpu(0)?;

    // Special registers: keep cs.base = 0 so CS:IP maps linearly. rip starts at the
    // first code byte, which lives at GUEST_PHYS_ADDR in *guest physical* space.
    let mut sregs = vcpu.get_sregs()?;
    sregs.cs.base = 0;
    sregs.cs.selector = 0;
    vcpu.set_sregs(&sregs)?;

    // General registers: rip points at guest-physical GUEST_PHYS_ADDR (= our code).
    // In real mode, linear = (cs.base) + rip; cs.base is 0, so rip must equal the
    // guest-physical address of the code: 0x1000.
    let mut regs = vcpu.get_regs()?;
    regs.rip = GUEST_PHYS_ADDR; // 0x1000
    regs.rflags = 0x2; // bit 1 is reserved-and-must-be-1
    vcpu.set_regs(&regs)?;

    // ── The run loop: a match over the full VcpuExit taxonomy ──────────────
    let stdout = std::io::stdout();
    let mut out = stdout.lock();
    loop {
        let exit = vcpu.run()?; // KVM_RUN; blocks until the guest causes an exit
        match exit {
            VcpuExit::IoOut(port, data) => {
                // PIO write. `data` is already the slice at kvm_run.io.data_offset.
                if port == 0x3f8 {
                    out.write_all(data)?;
                    out.flush()?;
                } else {
                    eprintln!("[IoOut on unexpected port {port:#x}: {data:?}]");
                }
            }
            VcpuExit::IoIn(port, data) => {
                // PIO read. A real device would FILL `data` here; we just observe it.
                eprintln!("[IoIn on port {port:#x}, {} byte(s) requested]", data.len());
                // Returning a value to the guest = write into `data`. We do nothing,
                // so the guest reads whatever KVM left there. That's fine for the demo.
            }
            VcpuExit::MmioRead(addr, data) => {
                // A guest read from an address with no registered memory region.
                eprintln!("[MmioRead @ {addr:#x}, {} byte(s)]", data.len());
            }
            VcpuExit::MmioWrite(addr, data) => {
                eprintln!("[MmioWrite @ {addr:#x}: {data:?}]");
            }
            VcpuExit::Hlt => {
                eprintln!("[VcpuExit::Hlt — guest halted, stopping]");
                break;
            }
            VcpuExit::Shutdown => {
                // Triple fault / reset. On real hardware a reboot; for us, stop.
                eprintln!("[VcpuExit::Shutdown — triple fault or reset]");
                break;
            }
            VcpuExit::FailEntry(reason, cpu) => {
                // KVM refused to enter the guest — almost always bad initial vCPU state.
                return Err(format!("FailEntry: hw_reason={reason:#x} cpu={cpu}").into());
            }
            VcpuExit::InternalError => {
                return Err("KVM_EXIT_INTERNAL_ERROR — KVM hit an internal problem".into());
            }
            other => {
                // The enum has many variants (X86Rdmsr, Hypercall, SystemEvent, ...).
                // For this tiny guest, anything else is unexpected.
                return Err(format!("unexpected VM exit: {other:?}").into());
            }
        }
    }

    // Clean up the host mapping (RAII handles the fds; the raw mmap we own).
    unsafe { libc::munmap(host_addr, MEM_SIZE) };
    Ok(())
}

Step 4: Run it

cargo run

Expected output:

Hello from kvm-ioctls!
[IoIn on port 0x3f8, 1 byte(s) requested]
[VcpuExit::Hlt — guest halted, stopping]

The first line is the string, delivered one byte per VcpuExit::IoOut. The middle line is the single in al, dx your guest executed — proof you can see the read side of PIO, which Lab 1.4 never triggered. The last line is your Hlt arm.

Tip: If /dev/kvm is not accessible: sudo setfacl -m u:$USER:rw /dev/kvm (preferred), or be in the kvm group, or build then sudo ./target/debug/kvm-vm-lab.

Step 5: Compare and contrast with Lab 1.4

Lab 1.4 and this lab use the same two crates. What changed is depth. Hold them side by side:

The two unsafe blocks are the whole lesson of the safe-wrapper boundary. Kvm::new, create_vm, create_vcpu, get/set_regs, and run are all safe — kvm-ioctls can guarantee those ioctls cannot corrupt host memory regardless of arguments. set_user_memory_region is unsafe because you supply a raw host pointer and length, and KVM will dereference that range from the guest; the crate cannot verify your mmap is valid, so it makes you assert it. That is the exact boundary Firecracker's vstate/memory.rs lives on, wrapped one level higher by vm-memory.

Step 6: Map every call to Firecracker's vstate/

Run each rg against your checkout. Do not trust the table — confirm it.

Read vstate/vcpu/ with this table open. Firecracker's loop is recognizably your match, but every arm dispatches to a real device on a bus, runs in its own thread, and is coordinated by the VMM thread over channels. The skeleton is yours; the muscle is theirs.

Implementation Requirements / Deliverables

• kvm-vm-lab compiles and cargo run prints the greeting, the IoIn line, and the Hlt line.
• You can state, for each of the seven crate calls in main (Kvm::new, create_vm, set_user_memory_region, create_vcpu, set_sregs/set_regs, run), the KVM ioctl it issues and whether the call is unsafe and why.
• You can explain what KvmRunWrapper is, where it comes from, and why run() takes &mut self.
• The Step 6 mapping table, each rg run against your Firecracker checkout, with the file you found noted next to it.
• A short written comparison (5–8 sentences) of this lab versus Lab 1.4 — what each unsafe block guards and which VcpuExit variants you added.

Troubleshooting

error[E0308] / trait mismatch at set_user_memory_region

Version skew between kvm-ioctls and kvm-bindings. The kvm_userspace_memory_region type the wrapper expects comes from the kvm-bindings that kvm-ioctls was built against. Fix:

cargo tree -p kvm-bindings        # see which version kvm-ioctls pulled
# then pin THAT exact kvm-bindings in Cargo.toml

(verify both versions on crates.io) — the pair bumps frequently.

VcpuExit::FailEntry immediately

The vCPU's initial state is invalid. Two classic causes: regs.rflags is 0 (bit 1 must be 1; set 0x2), or rip does not point at executable bytes. Recall RAM is at 0x1000, so rip must be 0x1000, not 0x0.

Greeting prints but no IoIn line

The guest halted before reaching in al, dx. Check the je displacement (0x74, 0x04) lands on the in at offset 0x0e, and that the jmp print (0xeb, 0xf8) lands back on lodsb at 0x06.

unexpected VM exit: SystemEvent(...) or similar

Your code ran off into garbage and triggered a fault. Re-check the code blob offsets against the listing — a single wrong jump displacement does this. The catch-all other => arm is there exactly so this surfaces as a clear error rather than a hang.

Operation not permitted opening /dev/kvm

sudo setfacl -m u:${USER}:rw /dev/kvm     # preferred
# or: sudo usermod -aG kvm $USER && re-login

Expected Output

$ cargo run
   Compiling kvm-vm-lab v0.1.0 (.../kvm-vm-lab)
    Finished `dev` profile [unoptimized + debuginfo] target(s)
     Running `target/debug/kvm-vm-lab`
Hello from kvm-ioctls!
[IoIn on port 0x3f8, 1 byte(s) requested]
[VcpuExit::Hlt — guest halted, stopping]

Stretch Goals

1. Return a value on IoIn. In the VcpuExit::IoIn(port, data) arm, write a byte into data (e.g. data[0] = 0x42) and have the guest out it back so you see the round trip. You have now implemented the read side of a device — exactly what Firecracker's serial UART does for the LSR/RBR registers.

2. Trigger a real MMIO exit. Add guest code that writes to an address with no registered region (e.g. mov word [0xd0000000], ax, with cs/ds set up for it). KVM returns VcpuExit::MmioWrite(addr, data) instead of faulting. Your arm already prints it. This is the foundation of the MMIO bus every virtio-MMIO device sits on — and it leads directly into Lab R3.

3. Enable dirty-page tracking. Set flags: KVM_MEM_LOG_DIRTY_PAGES on the region, then after the run call vm.get_dirty_log(0, MEM_SIZE) and print which pages the guest dirtied. You have touched the exact primitive behind diff snapshots — and you can map it to track_dirty_pages in Firecracker's machine-config.

4. Inspect KVM_GET_VCPU_MMAP_SIZE. Call kvm.get_vcpu_mmap_size() and print it. This is the size of the kvm_run page KvmRunWrapper mmaps per vCPU. Note how the crate hides this from you — then rg -n "get_vcpu_mmap_size\|KvmRunWrapper" ~/firecracker and confirm Firecracker never touches it directly either; the crate owns that detail.

5. Read the crate source. cargo doc --open -p kvm-ioctls, then find KvmRunWrapper in the crate source (find ~/.cargo/registry -path "*kvm-ioctls*/src/*"). Read how run() decodes kvm_run.exit_reason into VcpuExit. You will recognize every variant from your match.

Validation / Self-check

You are done when you can answer these without notes:

1. What are the three kvm-ioctls types corresponding to KVM's three fd levels, and which method moves you down each level?
2. What is KvmRunWrapper, what does it wrap, when is it created, and why does run() take &mut self?
3. Which call in your program is unsafe, and what invariant are you promising KVM by calling it? Why are create_vm/create_vcpu/run not unsafe?
4. Name five VcpuExit variants and the guest behavior (or KVM condition) that produces each.
5. Where do the I/O bytes in VcpuExit::IoOut(port, data) come from in the raw KVM ABI, and what bookkeeping did kvm-ioctls do so you did not have to?
6. What is the difference between kvm-bindings and kvm-ioctls, and which one is machine-generated?
7. For each of the seven crate calls, name the Firecracker vstate/ file that wraps it (from your Step 6 table).

When you can drive KVM through kvm-ioctls and explain the safe/unsafe boundary and the exit taxonomy, you are ready to put a real kernel into guest memory. Continue to Lab R2: Load a Kernel with linux-loader.