Lab 1.4: Project — Build a Minimal KVM VMM by Hand

Background

Firecracker is a VMM, and a VMM is, at its irreducible core, a userspace program that drives /dev/kvm through ioctl(). Every grand thing Firecracker does — booting Linux, emulating virtio devices, snapshotting — is built on top of a small set of KVM primitives: open the device, create a VM, give it memory, create a vCPU, load some code, and run it. Until you have done exactly that with your own hands, "the vCPU run loop" and "guest memory" are words. After you have done it, reading Firecracker's Vcpu and vstate/memory.rs is reading a hardened, productionized version of a program you already wrote.

In this build-it project you write a ~70-line standalone Rust program using the same rust-vmm crates Firecracker uses — kvm-ioctls (the safe Kvm/VmFd/VcpuFd wrappers) and kvm-bindings (the raw KVM structs). Your program opens /dev/kvm, creates a VM, mmaps a page of guest memory and registers it with KVM_SET_USER_MEMORY_REGION, loads a tiny hand-assembled real-mode code blob that writes bytes to I/O port 0x3f8 and then halts, creates a vCPU, sets its segment and general registers, and runs the KVM_RUN loop — printing each byte the guest writes (a KVM_EXIT_IO exit) and stopping on KVM_EXIT_HLT. That serial port, 0x3f8, is COM1 — the very port Firecracker's emulated 16550 UART lives on. You are building, in miniature, the thing the serial console deep dive describes.

This is the most important lab in Level 1. Get the code correct and understand every line — the mapping table at the end ties each primitive you wrote to the Firecracker type that wraps it.

Why This Lab Matters for Contributors

• The vCPU run loop is the single most important control flow in Firecracker. You cannot review a change to it that you have not first written yourself, in skeleton form. This lab is that skeleton.
• Guest memory is "host mmap registered with KVM." That sentence is abstract until you call KVM_SET_USER_MEMORY_REGION yourself and see the guest read the bytes you wrote.
• The KVM fundamentals deep dive and the rust-vmm kvm-ioctls chapter are the theory; this lab is the practice that makes them stick. The rust-vmm Lab R1 extends it.

Prerequisites

• Lab 1.1 complete; in Stretch Goal 2 you found the exact kvm-ioctls and kvm-bindings versions Firecracker resolves. Use compatible versions here so the API matches what you will read in the Firecracker tree.
• A Rust toolchain on your PATH (any recent stable works for this standalone program — it does not need Firecracker's musl pin) and a working, accessible /dev/kvm.

rustc --version
ls -l /dev/kvm        # you must be able to read+write this
# Recall the versions Firecracker uses, to pick matching crate versions below:
rg -n -A2 '^name = "kvm-ioctls"|^name = "kvm-bindings"' ~/firecracker/Cargo.lock

Step-by-Step Tasks

Step 1: Create the project

Work outside the Firecracker repo so you do not pollute it.

cargo new --bin mini-vmm
cd mini-vmm

Step 2: Declare the dependencies

Edit Cargo.toml. Pin kvm-ioctls and kvm-bindings to versions compatible with what Firecracker uses (the 0.x lines below are representative — verify against your Cargo.lock from the prerequisite and adjust if the API in Step 4 differs):

[package]
name = "mini-vmm"
version = "0.1.0"
edition = "2021"

[dependencies]
kvm-ioctls = "0.17"
kvm-bindings = "0.9"

Note: kvm-ioctls is the safe wrapper crate (Kvm, VmFd, VcpuFd, VcpuExit); kvm-bindings is the raw FFI crate (kvm_userspace_memory_region, kvm_regs, kvm_sregs). These are the exact two crates Firecracker uses to talk to KVM — confirm: rg -n "kvm_ioctls::|kvm_bindings::" ~/firecracker/src/vmm/src/vstate/ | head.

Step 3: Understand the guest code you will run

The guest runs in 16-bit real mode (the CPU's power-on state; we do not set up long mode the way Firecracker does for a real kernel — that is the whole point of starting small). Our guest is a handful of bytes that read a value, write it to the serial port, increment, and halt. Here is the assembly and the machine code, byte for byte:

                              ; AL is preloaded by us via REGS (rax low byte) = 'H' (0x48)
  ba f8 03        mov dx, 0x3f8     ; DX = COM1 data port
  00 d8           add al, bl        ; AL += BL  (BL preloaded = 0; keeps AL as-is first pass)
  ee              out dx, al        ; write AL to port 0x3f8  -> causes KVM_EXIT_IO (PIO write)
  b0 0a           mov al, 0x0a      ; AL = '\n'
  ee              out dx, al        ; write newline
  f4              hlt               ; halt -> causes KVM_EXIT_HLT

For a slightly more interesting demo that emits a whole word, we instead embed the string in memory and loop. But to keep the program short and the control flow unmistakable, the version below writes a fixed greeting one byte at a time using a tiny loop in machine code. Read the bytes in Step 4's GUEST_CODE and match them to this listing:

0x00  be 10 00     mov si, 0x0010    ; SI = address of the string (offset 0x10 in this page)
0x03  ba f8 03     mov dx, 0x3f8     ; DX = COM1 data port
0x06  loop:
0x06  ac           lodsb             ; AL = [SI]; SI++
0x07  3c 00        cmp al, 0         ; reached the NUL terminator?
0x09  74 03        je  done          ; if zero, jump forward +3 to hlt (at 0x0e)
0x0b  ee           out dx, al        ; else write AL to port 0x3f8  -> KVM_EXIT_IO
0x0c  eb f8        jmp loop          ; jump back -8 to 0x06
0x0e  done:
0x0e  f4           hlt               ; -> KVM_EXIT_HLT
0x10  ; the bytes "Hello from a guest!\n\0"

The two key VM exits you handle: KVM_EXIT_IO (the guest executed out dx, al; KVM hands control back to you with the byte the guest wrote) and KVM_EXIT_HLT (the guest executed hlt; you stop the loop).

Step 4: Write the program

Replace src/main.rs with this complete, runnable source. Read it as you type it — every line is load bearing.

use kvm_bindings::kvm_userspace_memory_region;
use kvm_ioctls::{Kvm, VcpuExit};
use std::io::Write;
use std::ptr::null_mut;
use std::slice;

// One 4 KiB page of guest physical memory, placed at guest physical address 0.
const GUEST_PHYS_ADDR: u64 = 0x0;
const MEM_SIZE: usize = 0x1000; // 4 KiB

// Real-mode machine code: walk a NUL-terminated string at offset 0x10 and `out` each byte
// to port 0x3f8 (COM1), then `hlt`. See the listing in Step 3 for the assembly.
//
//   be 10 00   mov si, 0x0010
//   ba f8 03   mov dx, 0x3f8
// loop:
//   ac         lodsb
//   3c 00      cmp al, 0
//   74 04      je done
//   ee         out dx, al
//   eb f8      jmp loop
// done:
//   f4         hlt
//
// Byte offsets are tracked in the comments so the jump displacements are verifiable:
const GUEST_CODE: [u8; 15] = [
    0xbe, 0x10, 0x00, // [0x00] mov si, 0x0010
    0xba, 0xf8, 0x03, // [0x03] mov dx, 0x3f8
    0xac, //             [0x06] lodsb            (loop:)
    0x3c, 0x00, //       [0x07] cmp al, 0
    0x74, 0x03, //       [0x09] je done  (rel8 measured from 0x0b; +3 -> 0x0e = hlt)
    0xee, //             [0x0b] out dx, al
    0xeb, 0xf8, //       [0x0c] jmp loop (rel8 measured from 0x0e; -8 -> 0x06 = loop)
    0xf4, //             [0x0e] hlt              (done:)
];

fn main() {
    // 1. Open /dev/kvm. This is the "system" fd level.
    let kvm = Kvm::new().expect("failed to open /dev/kvm — check permissions and that KVM is present");
    assert_eq!(kvm.get_api_version(), 12, "unexpected KVM API version");

    // 2. Create a VM. This is the "VM" fd level (KVM_CREATE_VM under the hood).
    let vm = kvm.create_vm().expect("KVM_CREATE_VM failed");

    // 3. Allocate one page of host memory with mmap. This memory IS the guest's RAM.
    let host_addr = unsafe {
        libc_mmap(MEM_SIZE)
    };
    let guest_mem: &mut [u8] =
        unsafe { slice::from_raw_parts_mut(host_addr as *mut u8, MEM_SIZE) };

    // 4. Lay out the guest's memory: code at offset 0, string at offset 0x10.
    guest_mem[..GUEST_CODE.len()].copy_from_slice(&GUEST_CODE);
    let msg = b"Hello from a guest!\n\0";
    guest_mem[0x10..0x10 + msg.len()].copy_from_slice(msg);

    // 5. Register the host memory with KVM as guest physical RAM at address 0.
    //    This is KVM_SET_USER_MEMORY_REGION — the single most important memory call.
    let region = kvm_userspace_memory_region {
        slot: 0,
        guest_phys_addr: GUEST_PHYS_ADDR,
        memory_size: MEM_SIZE as u64,
        userspace_addr: host_addr as u64,
        // flags: 0. Set kvm_bindings::KVM_MEM_LOG_DIRTY_PAGES here to track dirty pages
        // (the primitive behind diff snapshots — see Stretch Goal 4).
        flags: 0,
    };
    unsafe { vm.set_user_memory_region(region).expect("KVM_SET_USER_MEMORY_REGION failed") };

    // 6. Create a vCPU (the "vCPU" fd level, KVM_CREATE_VCPU).
    let mut vcpu = vm.create_vcpu(0).expect("KVM_CREATE_VCPU failed");

    // 7. Set up SREGS so the code segment base is 0 and we execute from physical 0.
    //    In real mode, linear address = (segment << 4) + offset. We keep cs.base = 0 and
    //    cs.selector = 0 so that CS:IP = 0x0000:rip maps to physical `rip`.
    let mut sregs = vcpu.get_sregs().expect("KVM_GET_SREGS failed");
    sregs.cs.base = 0;
    sregs.cs.selector = 0;
    // ds/es/ss left as KVM's defaults (base 0) — fine for our flat, low-memory layout.
    vcpu.set_sregs(&sregs).expect("KVM_SET_SREGS failed");

    // 8. Set up REGS: start executing at the top of our code; AL/BL seed values; the
    //    mandatory-1 bit (bit 1) of RFLAGS must be set or KVM_RUN rejects the state.
    let mut regs = vcpu.get_regs().expect("KVM_GET_REGS failed");
    regs.rip = 0x0; // entry point = guest physical 0 = first byte of GUEST_CODE
    regs.rax = 0;   // AL = 0
    regs.rbx = 0;   // BL = 0
    regs.rflags = 0x2; // bit 1 is reserved-and-must-be-1; everything else clear
    vcpu.set_regs(&regs).expect("KVM_SET_REGS failed");

    // 9. The run loop. Each KVM_RUN either runs guest code until an exit or returns an exit reason.
    let stdout = std::io::stdout();
    let mut out = stdout.lock();
    loop {
        match vcpu.run().expect("KVM_RUN failed") {
            // The guest executed `out dx, al`. `data` is the byte(s) it wrote; port is 0x3f8.
            VcpuExit::IoOut(port, data) => {
                if port == 0x3f8 {
                    out.write_all(data).unwrap();
                    out.flush().unwrap();
                } else {
                    eprintln!("[unexpected IoOut on port {:#x}]", port);
                }
            }
            // The guest could also read a port; we don't expect it, but handle it cleanly.
            VcpuExit::IoIn(port, _data) => {
                eprintln!("[unexpected IoIn on port {:#x}]", port);
            }
            // The guest executed `hlt`. We are done.
            VcpuExit::Hlt => {
                eprintln!("\n[KVM_EXIT_HLT — guest halted, stopping]");
                break;
            }
            // Anything else is a state/setup error for this tiny program.
            other => {
                panic!("unexpected VM exit: {:?}", other);
            }
        }
    }
}

/// Minimal anonymous `mmap` helper. We declare `mmap` ourselves so this program needs no extra
/// crates beyond kvm-ioctls/kvm-bindings; in real code you would use the `libc` or `nix` crate.
unsafe fn libc_mmap(size: usize) -> *mut std::ffi::c_void {
    extern "C" {
        fn mmap(
            addr: *mut std::ffi::c_void,
            length: usize,
            prot: i32,
            flags: i32,
            fd: i32,
            offset: i64,
        ) -> *mut std::ffi::c_void;
    }
    const PROT_READ: i32 = 0x1;
    const PROT_WRITE: i32 = 0x2;
    const MAP_PRIVATE: i32 = 0x2;
    const MAP_ANONYMOUS: i32 = 0x20;
    const MAP_FAILED: *mut std::ffi::c_void = !0usize as *mut std::ffi::c_void;
    let p = mmap(
        null_mut(),
        size,
        PROT_READ | PROT_WRITE,
        MAP_PRIVATE | MAP_ANONYMOUS,
        -1,
        0,
    );
    assert!(p != MAP_FAILED, "mmap failed");
    p
}

Warning — get the code blob right. The je and jmp displacements must match the byte layout exactly. x86 rel8 displacements are measured from the address of the next instruction. In the array above, 0xeb, 0xf8 (jmp -8) is measured from 0x0e and lands on 0x06 (lodsb); the je done (0x74, 0x03) is measured from 0x0b and lands on 0x0e (hlt), skipping the out (1 byte) and the jmp (2 bytes). Verify the offsets against the Step 3 listing and recompute if you change the code. A wrong displacement runs into garbage and you get an unexpected VM exit. This is exactly the kind of low-level precision Firecracker's arch code demands.

Note on data_offset: When you use raw KVM via C you read the I/O byte from (char*)kvm_run + kvm_run->io.data_offset. kvm-ioctls does this bookkeeping for you and hands you a &[u8] directly in VcpuExit::IoOut(port, data) — data is the slice at data_offset. Keep that equivalence in mind; you will see the raw form when you read KVM docs and the wrapped form when you read Firecracker.

Step 5: Run it

cargo run

Expected output:

Hello from a guest!

[KVM_EXIT_HLT — guest halted, stopping]

The first line is bytes the guest wrote to port 0x3f8, each delivered to you as a KVM_EXIT_IO exit and printed by your run loop. The last line is your KVM_EXIT_HLT handler firing. You just ran guest code on the physical CPU, intercepted its I/O, and stopped it cleanly — you wrote a VMM.

Tip: If cargo run fails to open /dev/kvm, run it with access (sudo is the blunt fix; the right fix is setfacl -m u:$USER:rw /dev/kvm or being in the kvm group). Do not move on until the greeting prints.

Step 6: Map your primitives to Firecracker

Every primitive you wrote has a Firecracker counterpart that wraps it with safety, configuration, and production concerns. Verify each row with an rg into your Firecracker checkout — do not take the table on faith.

Read vstate/vcpu/ with this table in hand. Firecracker's run loop is recognizably the same shape as yours — match vcpu.run() over VcpuExit variants — but each arm dispatches to a real device on the MMIO/PIO bus instead of println!, and the whole thing runs in its own thread coordinated by the VMM thread over channels. The skeleton is yours; the muscle is theirs.

Step 7: Reflect — what does Firecracker add on top of this?

Your program is a complete VMM in the literal sense, and it is also a toy. Write down what stands between it and Firecracker. The honest answer is the entire rest of this curriculum, but the major layers are:

• A real guest kernel. You ran 14 bytes of real-mode code. Firecracker loads an uncompressed vmlinux ELF, sets up long mode (cr0 PE|PG, cr4 PAE, efer LME|LMA, identity-mapped page tables), builds a boot_params / zero page with an e820 map and the cmdline, and jumps to the kernel's e_entry. That is the boot sequence.
• Device emulation. Your IO handler is one println!. Firecracker has a real 16550 UART, plus a whole virtio device model — block, net, vsock, rng, balloon — sitting on an MMIO bus, each handling exits and doing real I/O against the host.
• The control plane. You hardcoded everything. Firecracker has a REST API over a Unix socket, a VmmAction channel between threads, and a configuration/validation layer.
• The threading model. Your loop is single-threaded. Firecracker runs the API thread, the VMM thread (an EventManager epoll loop), and one thread per vCPU, coordinated by channels and eventfds.
• The jailer and seccomp. Your program runs with your full privileges. Firecracker is wrapped by the jailer (chroot/cgroups/namespaces/privilege-drop) and confines itself with a per-thread seccomp-BPF filter that whitelists the ~40 syscalls it is allowed to make.

That list is the threat model in action: every layer Firecracker adds is either functionality the guest needs (a kernel, devices, an API) or a wall between a hostile guest and the host (jailer, seccomp, Rust's memory safety, the minimal device surface).

Implementation Requirements / Deliverables

• mini-vmm compiles and cargo run prints Hello from a guest! followed by the KVM_EXIT_HLT line.
• You can explain, line by line, what each of the numbered steps (1–9) in main does and which KVM ioctl it maps to (KVM_CREATE_VM, KVM_SET_USER_MEMORY_REGION, KVM_CREATE_VCPU, KVM_SET_SREGS, KVM_SET_REGS, KVM_RUN).
• The mapping table above, with each row's rg run against your Firecracker checkout and the file you found noted.
• A written reflection (5–8 sentences) answering: what does Firecracker add on top of this bare KVM program, and why is each addition either functionality or a security wall?

Troubleshooting

failed to open /dev/kvm / Operation not permitted

You lack access to the device.

ls -l /dev/kvm
sudo setfacl -m u:${USER}:rw /dev/kvm     # preferred
# or be in the kvm group:  sudo usermod -aG kvm $USER  (then re-login)
# or, bluntly:  cargo build && sudo ./target/debug/mini-vmm

KVM_RUN failed immediately / unexpected VM exit: FailEntry

The vCPU's initial state is invalid. The usual cause is rflags: bit 1 is reserved and must be

1. If you set regs.rflags = 0 instead of 0x2, KVM_RUN rejects the state. Confirm regs.rflags = 0x2.

Garbage output or unexpected VM exit partway through

Your real-mode segments or your code blob displacements are off. Check that cs.base = 0 and cs.selector = 0 so CS:IP maps to physical rip, that rip = 0 points at the first code byte, and that the je/jmp displacements in GUEST_CODE match the Step 3 listing exactly. A single wrong byte sends execution into the string data or off the end of the page.

Nothing prints but the program exits cleanly

The guest halted before writing — likely your string is not where SI points (0x10) or the NUL terminator landed early. Confirm guest_mem[0x10..] holds the message and that the page layout (code at 0, string at 0x10) does not overlap (your code is 14 bytes, comfortably below 0x10).

IoOut on an unexpected port

The guest's DX is not 0x3f8. Confirm the mov dx, 0x3f8 bytes (0xba, 0xf8, 0x03) — note the little-endian 0xf8, 0x03 = 0x03f8.

I read about data_offset in the KVM docs and don't see it here

kvm-ioctls reads kvm_run.io.data_offset for you and gives you the slice as the second tuple element of VcpuExit::IoOut(port, data). The raw-C step is hidden behind the safe wrapper — that is the abstraction Firecracker relies on.

Expected Output

$ cargo run
   Compiling mini-vmm v0.1.0 (.../mini-vmm)
    Finished dev [unoptimized + debuginfo] target(s)
     Running `target/debug/mini-vmm`
Hello from a guest!

[KVM_EXIT_HLT — guest halted, stopping]

Stretch Goals

1. Add a second I/O port. Have the guest also out a byte to port 0x80 (a classic POST/debug port). Add a VcpuExit::IoOut(0x80, ..) arm that logs "[debug port write: {byte}]" instead of printing it. You now have two device-like behaviors keyed off the port — exactly how a real bus dispatches by address.

2. Handle an MMIO exit. Have the guest write to a memory-mapped address that is not in a registered KVM memory region (e.g. mov word [0x2000], ax with code adjusted, while only registering page 0). KVM will return VcpuExit::MmioWrite(addr, data) instead of faulting. Add an arm that prints the address and bytes. This is the foundation of the MMIO bus that virtio-MMIO devices live on — Firecracker's entire device model rides on MmioRead/MmioWrite exits.

3. Map more memory and run a longer program. Bump MEM_SIZE to several pages, write a guest program that counts from 0 to 9 and outs each as an ASCII digit, and watch the loop drive many IoOut exits. Observe how every guest I/O is a round trip through your run loop — the cost model that makes the virtio fast path (ioeventfd) necessary.

4. Enable dirty-page tracking. Set flags: KVM_MEM_LOG_DIRTY_PAGES on the memory region and call vm.get_dirty_log(0, MEM_SIZE) after the run. You have just touched the primitive behind diff snapshots.

Validation / Self-check

You are done when you can answer these without notes:

1. Name the three KVM "fd levels" and the ioctl that creates each (system → VM → vCPU).
2. What does KVM_SET_USER_MEMORY_REGION do, and what are the four fields you filled in (slot, guest physical address, size, userspace address)?
3. Why must rflags have bit 1 set before KVM_RUN, and what happens if it isn't?
4. In real mode, how does CS:IP map to a physical address, and why did setting cs.base = 0 and rip = 0 make execution start at the first byte of your code?
5. What are the two VcpuExit variants your run loop handles, what guest instruction causes each, and where does the I/O byte come from (the data_offset equivalence)?
6. For each primitive you wrote, name the Firecracker type that wraps it (Kvm/VmFd/VcpuFd → vstate; the run loop → Vcpu::run; guest memory → GuestMemoryMmap).
7. Name three things Firecracker adds on top of your program, and classify each as functionality or security wall.

You have now built a VMM from /dev/kvm up, booted a real microVM from the top down, built and tested Firecracker, and mapped the whole stack. That is the foundation. Proceed to Level 2 — Firecracker Contributor Onboarding to turn this understanding into pull requests the maintainers will take seriously.