Lab 1: Build a VM From Scratch (Long Mode, Multiple Regions, Multiple vCPUs)

Background

In Lab 1.4 you wrote a ~70-line VMM that ran a handful of real-mode bytes on one vCPU and printed the bytes the guest out-ed to COM1. That was the irreducible core. This lab is the next rung: a standalone Rust VMM that does three things Lab 1.4 deliberately didn't, all of which Firecracker does for a real kernel:

1. Registers multiple guest-memory regions with an MMIO gap — not one flat page, but a low-RAM region, an (unbacked) MMIO window, and the layout discipline Firecracker uses.
2. Puts the vCPU into 64-bit long mode — builds page tables and segment descriptors in guest memory, sets cr0/cr3/cr4/efer, and runs a small 64-bit payload. This is the exact step that separates "ran some bytes" from "could load a vmlinux."
3. Runs multiple vCPUs, each on its own thread, sharing the VM and its memory, and handles a richer set of VcpuExit variants (IO, MMIO, HLT, Shutdown, and the error variants) cleanly.

This is a build-it lab. By the end you will have written the skeleton of Firecracker's vstate — the part that becomes Vm, Vcpu, GuestMemoryMmap, and the long-mode boot setup in arch/x86_64/. Get the code correct and understand every line; the mapping table in Step 9 ties each piece to the Firecracker type that productionizes it.

Why This Lab Matters for Contributors

• Firecracker's vCPU run loop runs one thread per vCPU, each in its own KVM_RUN loop, coordinated by the VMM thread over channels. You cannot review a change to that threading or to a vCPU's lifecycle that you have not built in skeleton form. This lab is that skeleton.
• Guest memory is "multiple host mmaps registered with KVM, with a deliberate hole for MMIO." That sentence is abstract until you register two regions and a gap yourself and watch a guest MMIO access fault out as KVM_EXIT_MMIO.
• The boot sequence jumps to a 64-bit e_entry. The long-mode setup here is the same cr0/cr3/cr4/efer + page-table
  • segment-descriptor work Firecracker does in arch/x86_64/ before it enters the kernel — minus the kernel. You'll meet the real version in Boot Process Lab 1.

Prerequisites

• Lab 1.4 complete — you have the bare-KVM reflexes (three fd levels, a memory region, a run loop).
• You read the KVM & vCPUs intensive index, especially the long-mode table and the layout diagram.
• A Rust toolchain and an accessible /dev/kvm.

rustc --version
ls -l /dev/kvm        # you must be able to read+write this
nproc                 # you want >= 2 logical CPUs to see the vCPUs run in parallel
# Recall the crate versions Firecracker resolves, to match the API you'll read later:
rg -n -A2 '^name = "kvm-ioctls"|^name = "kvm-bindings"' ~/firecracker/Cargo.lock

Step-by-Step Tasks

Step 1: Create the project and declare dependencies

cargo new --bin mini-vmm-64
cd mini-vmm-64

Edit Cargo.toml. We add libc this time (we need mmap flags and want clean code) — kvm-ioctls/kvm-bindings versions should be compatible with what Firecracker resolves (the lines below are representative; verify against your Cargo.lock and adjust if an API in Step 5 differs):

[package]
name = "mini-vmm-64"
version = "0.1.0"
edition = "2021"

[dependencies]
kvm-ioctls = "0.17"
kvm-bindings = "0.9"
libc = "0.2"

Step 2: Decide the memory layout

We mirror Firecracker's x86_64 layout in miniature. Everything is below 4 GiB to keep one low-RAM region; the MMIO window is a small unbacked hole that a guest MMIO write will fault into.

Within low RAM we place:

  0x0000  page-table hierarchy   (PML4 @ 0x1000, PDPT @ 0x2000, PD @ 0x3000)
  0x4000  the 64-bit guest code  (entry point; rip starts here)
  0x8000  per-vCPU stacks grow down from here

Note: We put the page tables at 0x1000/0x2000/0x3000 and code at 0x4000 so nothing overlaps. Firecracker's real constants (PML4_START, PDPTE_START, PDE_START, etc.) live in src/vmm/src/arch/x86_64/ — rg -n "PML4_START|PDPTE_START|PDE_START|BOOT_GDT" src/vmm/src/arch/x86_64/ (verify on your branch). The idea is identical; only the addresses differ.

Step 3: Understand the 64-bit guest code

The payload runs in long mode. Each vCPU writes its own id and a fixed message to COM1, touches the MMIO gap once (to generate a KVM_EXIT_MMIO), then halts. We hand-assemble it; read the listing and match it to the GUEST_CODE bytes in Step 5.

; On entry: rdi = this vCPU's id (we set it per-vCPU via KVM_SET_REGS).
; DX is the COM1 data port; we walk a NUL-terminated string and `out` each byte.
0x00  b2 f8            mov dl, 0xf8           ; (DX low) — set DX = 0x03f8 in two steps
0x02  66 ba f8 03      mov dx, 0x03f8         ; DX = COM1 data port  (operand-size prefix)
0x06  48 89 f8         mov rax, rdi           ; rax = vcpu id
0x09  04 30            add al, 0x30           ; al = '0' + id  (ASCII digit)
0x0b  ee               out dx, al             ; write the id digit       -> KVM_EXIT_IO
0x0c  48 be 00 50 ...  mov rsi, 0x5000        ; rsi = address of the message string
0x16  ac               lodsb                  ; (loop:) al = [rsi]; rsi++
0x17  3c 00            cmp al, 0
0x19  74 06            je done                ; on NUL, jump to the MMIO poke
0x1b  ee               out dx, al             ; write the char           -> KVM_EXIT_IO
0x1c  eb f8            jmp loop
0x1e  ; done:
0x1e  48 c7 c0 00 ...  mov rax, 0x00200000    ; rax = MMIO gap address (unbacked)
0x28  c6 00 5a         mov byte [rax], 0x5a   ; write to MMIO            -> KVM_EXIT_MMIO
0x2b  f4               hlt                     ;                          -> KVM_EXIT_HLT
0x5000  "Hello from long mode!\n\0"

The three exits to handle: KVM_EXIT_IO (the outs — the id digit and the message), KVM_EXIT_MMIO (the write to 0x200000, which is in the unbacked gap), and KVM_EXIT_HLT (the hlt).

Note: We assemble this by hand for transparency. In real code you'd use an assembler; the point of typing the bytes is that you can never again be confused about what "the guest exits to MMIO" means — you wrote the instruction that does it.

Step 4: Page tables for long mode

Long mode requires paging. We build a 3-level identity map of the first 1 GiB using a single 1 GiB page at the PDPT level (the simplest valid long-mode map): PML4[0] → PDPT[0] with the PS (page-size) bit set, mapping 0x0000_0000 identity. The constants:

cr3 points at the PML4. cr0 gets PE|PG, cr4 gets PAE, efer gets LME|LMA. The cs descriptor gets the long-mode L bit.

Step 5: Write the program

Replace src/main.rs with this complete, runnable source. It is longer than Lab 1.4 because it does more — read it in the numbered blocks.

use kvm_bindings::{kvm_segment, kvm_userspace_memory_region};
use kvm_ioctls::{Kvm, VcpuExit, VmFd};
use std::io::Write;
use std::sync::Arc;
use std::thread;

// ---- Layout (mirrors arch/x86_64/layout.rs in miniature) ----
const LOWMEM_BASE: u64 = 0x0;
const LOWMEM_SIZE: usize = 0x20_0000; // 2 MiB low RAM (slot 0)
const MMIO_BASE: u64 = 0x20_0000; // 64 KiB unbacked MMIO gap (no slot)
const PML4_ADDR: u64 = 0x1000;
const PDPT_ADDR: u64 = 0x2000;
const CODE_ADDR: u64 = 0x4000;
const STACK_TOP: u64 = 0x8000;
const MSG_ADDR: u64 = 0x5000;

// Page-table entry flags.
const PTE_PRESENT: u64 = 1 << 0;
const PTE_WRITABLE: u64 = 1 << 1;
const PTE_PAGE_SIZE: u64 = 1 << 7;

// Control-register / EFER bits for long mode.
const CR0_PE: u64 = 1 << 0;
const CR0_PG: u64 = 1 << 31;
const CR4_PAE: u64 = 1 << 5;
const EFER_LME: u64 = 1 << 8;
const EFER_LMA: u64 = 1 << 10;

// 64-bit payload — see the Step 3 listing. Offsets in comments are guest-phys
// once relocated to CODE_ADDR (we set rip = CODE_ADDR, and rsi/rax use absolute
// addresses, so the code is position-dependent on CODE_ADDR/MSG_ADDR).
const GUEST_CODE: &[u8] = &[
    0x66, 0xba, 0xf8, 0x03, //          mov dx, 0x03f8
    0x48, 0x89, 0xf8, //                mov rax, rdi
    0x04, 0x30, //                      add al, 0x30
    0xee, //                            out dx, al            (id digit)
    0x48, 0xbe, 0x00, 0x50, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, // mov rsi, 0x5000
    0xac, //               (loop:)      lodsb
    0x3c, 0x00, //                      cmp al, 0
    0x74, 0x04, //                      je done   (skip out+jmp = 1+2 bytes -> +4)
    0xee, //                            out dx, al            (message byte)
    0xeb, 0xf8, //                      jmp loop  (-8)
    // done:
    0x48, 0xc7, 0xc0, 0x00, 0x00, 0x20, 0x00, //   mov rax, 0x00200000  (MMIO gap)
    0xc6, 0x00, 0x5a, //                mov byte [rax], 0x5a  (MMIO write)
    0xf4, //                            hlt
];

fn main() {
    // ---- 1. System + VM fd levels ----
    let kvm = Kvm::new().expect("open /dev/kvm");
    assert_eq!(kvm.get_api_version(), 12);
    let vm = kvm.create_vm().expect("KVM_CREATE_VM");

    // ---- 2. Allocate and register the LOW-RAM region (slot 0). The MMIO gap at
    //         MMIO_BASE is intentionally NOT registered, so accesses fault out. ----
    let host_ptr = unsafe {
        libc::mmap(
            std::ptr::null_mut(),
            LOWMEM_SIZE,
            libc::PROT_READ | libc::PROT_WRITE,
            libc::MAP_PRIVATE | libc::MAP_ANONYMOUS | libc::MAP_NORESERVE,
            -1,
            0,
        )
    };
    assert!(host_ptr != libc::MAP_FAILED, "mmap low RAM");
    let guest_mem =
        unsafe { std::slice::from_raw_parts_mut(host_ptr as *mut u8, LOWMEM_SIZE) };

    let region = kvm_userspace_memory_region {
        slot: 0,
        guest_phys_addr: LOWMEM_BASE,
        memory_size: LOWMEM_SIZE as u64,
        userspace_addr: host_ptr as u64,
        flags: 0,
    };
    unsafe { vm.set_user_memory_region(region).expect("KVM_SET_USER_MEMORY_REGION") };

    // ---- 3. Lay out guest memory: page tables, code, message. ----
    // PML4[0] -> PDPT (present, writable)
    write_u64(guest_mem, PML4_ADDR, PDPT_ADDR | PTE_PRESENT | PTE_WRITABLE);
    // PDPT[0] -> identity-map a 1 GiB page at phys 0 (present, writable, PS)
    write_u64(guest_mem, PDPT_ADDR, 0 | PTE_PRESENT | PTE_WRITABLE | PTE_PAGE_SIZE);
    // The code, at CODE_ADDR.
    guest_mem[CODE_ADDR as usize..CODE_ADDR as usize + GUEST_CODE.len()]
        .copy_from_slice(GUEST_CODE);
    // The message string, at MSG_ADDR.
    let msg = b"Hello from long mode!\n\0";
    guest_mem[MSG_ADDR as usize..MSG_ADDR as usize + msg.len()].copy_from_slice(msg);

    // ---- 4. Spawn one thread per vCPU. The VmFd is cloneable/shareable. ----
    let vm = Arc::new(vm);
    let num_vcpus = 2u64;
    let mut handles = Vec::new();
    for id in 0..num_vcpus {
        let vm = Arc::clone(&vm);
        handles.push(thread::spawn(move || run_vcpu(&vm, id)));
    }
    for h in handles {
        h.join().expect("vcpu thread panicked");
    }
    eprintln!("[all vCPUs halted]");
}

/// Set up one vCPU in long mode and run its KVM_RUN loop until HLT.
fn run_vcpu(vm: &VmFd, id: u64) {
    let mut vcpu = vm.create_vcpu(id).expect("KVM_CREATE_VCPU");

    // ---- 5. SREGS: enter long mode. ----
    let mut sregs = vcpu.get_sregs().expect("KVM_GET_SREGS");
    sregs.cr3 = PML4_ADDR;
    sregs.cr4 = CR4_PAE;
    sregs.cr0 = CR0_PE | CR0_PG;
    sregs.efer = EFER_LME | EFER_LMA;
    // A flat 64-bit code segment (L=1) and data segments.
    let code_seg = seg(0, 0xffff_ffff, 0x9b /* present, exec, read */, true);
    let data_seg = seg(0, 0xffff_ffff, 0x93 /* present, data, write */, false);
    sregs.cs = code_seg;
    sregs.ds = data_seg;
    sregs.es = data_seg;
    sregs.fs = data_seg;
    sregs.gs = data_seg;
    sregs.ss = data_seg;
    vcpu.set_sregs(&sregs).expect("KVM_SET_SREGS");

    // ---- 6. REGS: entry point, stack, and the vCPU id in rdi. ----
    let mut regs = vcpu.get_regs().expect("KVM_GET_REGS");
    regs.rip = CODE_ADDR;
    regs.rsp = STACK_TOP - id * 0x400; // give each vCPU its own slice of stack
    regs.rdi = id; // the payload reads its id from rdi
    regs.rflags = 0x2; // bit 1 reserved-must-be-1
    vcpu.set_regs(&regs).expect("KVM_SET_REGS");

    // ---- 7. The run loop: handle a richer exit set than Lab 1.4. ----
    let stdout = std::io::stdout();
    loop {
        match vcpu.run().expect("KVM_RUN") {
            VcpuExit::IoOut(port, data) => {
                if port == 0x3f8 {
                    let mut out = stdout.lock();
                    out.write_all(data).unwrap();
                    out.flush().unwrap();
                } else {
                    eprintln!("[vcpu {id}] unexpected IoOut port={port:#x}");
                }
            }
            VcpuExit::IoIn(port, _) => {
                eprintln!("[vcpu {id}] unexpected IoIn port={port:#x}");
            }
            VcpuExit::MmioWrite(addr, data) => {
                eprintln!("[vcpu {id}] MMIO write addr={addr:#x} data={data:02x?}");
                // A real device would act on this; we just observe it.
            }
            VcpuExit::MmioRead(addr, data) => {
                eprintln!("[vcpu {id}] MMIO read  addr={addr:#x} len={}", data.len());
                // A real device would FILL `data` here; we return zeros (default).
            }
            VcpuExit::Hlt => {
                eprintln!("[vcpu {id}] HLT — halted");
                break;
            }
            VcpuExit::Shutdown => {
                eprintln!("[vcpu {id}] SHUTDOWN (triple fault?) — stopping");
                break;
            }
            VcpuExit::FailEntry(reason, cpu) => {
                panic!("[vcpu {id}] FAIL_ENTRY reason={reason:#x} cpu={cpu}");
            }
            VcpuExit::InternalError => {
                panic!("[vcpu {id}] INTERNAL_ERROR — KVM could not emulate");
            }
            other => {
                panic!("[vcpu {id}] unexpected exit: {other:?}");
            }
        }
    }
}

/// Build a kvm_segment with a given base/limit and access byte.
fn seg(base: u64, limit: u32, access: u8, long_mode: bool) -> kvm_segment {
    kvm_segment {
        base,
        limit,
        selector: if access & 0x08 != 0 { 0x08 } else { 0x10 },
        type_: (access & 0xf) as u8,
        present: 1,
        dpl: 0,
        db: if long_mode { 0 } else { 1 },
        s: 1, // code/data (not system)
        l: if long_mode { 1 } else { 0 }, // long-mode code segment
        g: 1, // limit in 4 KiB pages
        avl: 0,
        unusable: 0,
        padding: 0,
    }
}

/// Write a little-endian u64 into guest memory at a guest-phys offset.
fn write_u64(mem: &mut [u8], addr: u64, val: u64) {
    let a = addr as usize;
    mem[a..a + 8].copy_from_slice(&val.to_le_bytes());
}

Warning — the access byte and segment fields. kvm_segment is finicky: the type_, s, present, db, l, and g fields must be internally consistent or KVM_RUN returns FailEntry. The seg() helper above sets a long-mode (l=1, db=0) flat code segment and db=1 flat data segments. If you see FailEntry, dump sregs and compare field-by-field with what Firecracker sets — rg -n "kvm_segment|configure_segments|BOOT_GDT|code_seg|data_seg" src/vmm/src/arch/x86_64/.

Note on the MMIO read default. When the guest reads unbacked MMIO, kvm-ioctls hands you a &mut [u8] to fill. We leave it as KVM initialized it (zeros). A real device computes the register value and writes it into that slice; that is precisely what MMIODeviceManager does when it dispatches a MmioRead — rg -n "fn read\b|MmioRead|bus.read" src/vmm/src/.

Step 6: Run it

cargo run

You should see each vCPU print its id digit, the shared message, log its MMIO write, and halt. Output ordering interleaves because the two vCPUs run on two host threads in parallel — that interleaving is the point.

Step 7: Prove long mode is actually on

Add a temporary check: after set_sregs, read them back and assert the long-mode bits stuck, and have the payload do something only valid in 64-bit mode (it already does — mov rax, rdi with a REX.W prefix and a 64-bit immediate mov rsi, 0x5000 are 64-bit-only encodings). If you flip efer back to 0 (no LME/LMA) and re-run, the 64-bit instructions decode as garbage and you'll get a Shutdown or InternalError — try it, observe the failure, then restore.

#![allow(unused)]
fn main() {
// Temporary: confirm the bits stuck.
let s = vcpu.get_sregs().unwrap();
assert!(s.efer & (EFER_LME | EFER_LMA) == (EFER_LME | EFER_LMA), "not long mode");
assert!(s.cr0 & CR0_PG != 0 && s.cr4 & CR4_PAE != 0, "paging/PAE off");
}

Step 8: Make the MMIO gap behave like a device (Stretch into the bus idea)

Right now your MMIO arm just logs. Extend it so that a write of 0x5a to 0x200000 is treated as "reset this vCPU" — set a flag and break. You've just written the world's smallest MMIO device: an address whose write has a side effect. That is the seed of Firecracker's MMIO bus, where each virtio device owns an address range and reacts to reads/writes in its window.

Step 9: Map your VMM onto Firecracker's vstate

Verify each row with an rg into your Firecracker checkout — don't take the table on faith.

Read vstate/vcpu/x86_64.rs and arch/x86_64/ with this table open. Firecracker's per-vCPU thread is recognizably your run_vcpu: set sregs/regs, loop on vcpu.run(), dispatch each VcpuExit — except each arm hits a real device on the bus, the long-mode setup loads a real kernel's page tables, and the whole thing is coordinated by the VMM thread over channels and broken out of KVM_RUN by an exit eventfd.

Implementation Requirements / Deliverables

• mini-vmm-64 compiles and cargo run prints, for each of two vCPUs, the id digit, the message, an MMIO-write log line, and a HLT line.
• You can explain, for each numbered block (1–7) in main/run_vcpu, what it does and which KVM ioctl it maps to.
• You can state the four long-mode preconditions (cr0 PE|PG, cr4 PAE, efer LME|LMA, valid page tables via cr3) and point at where you set each.
• The Step 9 mapping table, each row's rg run against your Firecracker checkout, the file noted.
• The Step 8 extension: the MMIO write at 0x200000 triggers a side effect (reset) rather than just logging.

Troubleshooting

KVM_RUN returns FailEntry immediately

The vCPU's initial state is inconsistent. The usual causes, in order: regs.rflags missing bit 1 (0x2); a kvm_segment field combination KVM rejects (check s, present, l/db consistency); efer set to long mode but cr4 missing PAE or cr0 missing PG. Dump sregs/regs and compare with rg -n "configure_segments_and_sregs|setup_regs" src/vmm/src/arch/x86_64/.

Guest immediately Shutdown (triple fault)

Paging is broken: cr3 doesn't point at your PML4, or the PML4/PDPT entries are wrong, or e_entry/CODE_ADDR isn't mapped. The 1 GiB identity page (PDPT entry with PS set) must cover CODE_ADDR. Verify write_u64(PML4_ADDR, PDPT_ADDR | PRESENT | WRITABLE) and the PDPT entry has PTE_PAGE_SIZE set.

Garbage output / InternalError

Long mode isn't actually on (so the 64-bit-encoded payload decodes wrong), or a jump displacement in GUEST_CODE is off. Re-run the Step 7 assertion; recount the je/jmp rel8 displacements against the Step 3 listing.

No MmioWrite exit appears

The MMIO address 0x200000 is inside a backed region — check LOWMEM_SIZE is exactly 0x20_0000 so 0x200000 is the first byte past the region and thus unbacked. If you bumped LOWMEM_SIZE, the write hits RAM and never exits.

Threads hang / cargo run never returns

A vCPU is blocked in KVM_RUN and never reached Hlt. With this payload that means a setup error stalled the guest (see the FailEntry/Shutdown cases). In Firecracker the equivalent break-out mechanism is an exit eventfd the VMM writes to force a KVM_RUN return — rg -n "exit_evt|exit_eventfd|immediate_exit" src/vmm/src/vstate/vcpu/.

Expected Output

$ cargo run
   Compiling mini-vmm-64 v0.1.0
    Finished dev [unoptimized + debuginfo] target(s)
     Running `target/debug/mini-vmm-64`
0Hello from long mode!
[vcpu 0] MMIO write addr=0x200000 data=[5a]
[vcpu 0] HLT — halted
1Hello from long mode!
[vcpu 1] MMIO write addr=0x200000 data=[5a]
[vcpu 1] HLT — halted
[all vCPUs halted]

The two vCPUs' lines may interleave arbitrarily (parallel threads). The id digit (0/1) before each message proves each vCPU read its own rdi; the MMIO line proves the unbacked gap faulted out as expected.

Stretch Goals

1. A real MMIO device. Make MmioRead of 0x200004 return a 4-byte "device id" you choose, and have the payload read it back and out it to the console. You now handle both MMIO directions — the contract every virtio-MMIO register uses. Connect this to Virtio Devices Lab 4.
2. Scale the vCPUs. Bump num_vcpus to nproc and give each a distinct message. Watch the interleaving. Then add a shared AtomicU64 the threads increment on each IoOut — you've just built the seed of a shared device protected by host synchronization, exactly the problem Firecracker's device locking solves.
3. Two memory regions above and below a gap. Add a second backed region at 0x1000_0000 (256 MiB up) and have the payload write a marker there. Confirm that a write between the two regions still MMIO-exits. This is the real Firecracker shape: RAM, gap, more RAM.
4. Break out of KVM_RUN from another thread. Use vcpu.set_kvm_immediate_exit(1) (or the EventFd-based mechanism) from main to force a running vCPU to return without halting — the primitive behind Firecracker's pause/shutdown. Find the real one: rg -n "immediate_exit|set_kvm_immediate_exit" ~/firecracker/src/vmm/src/.

Validation / Self-check

Answer without notes. These gate completion.

1. Name the three KVM fd levels and which one KVM_SET_USER_MEMORY_REGION, KVM_CREATE_VCPU, and KVM_RUN each belong to.
2. List the four conditions for long mode and the register/structure that carries each. What single missing condition produces FailEntry vs Shutdown?
3. Why is the MMIO gap unbacked, and what exit does a guest access there produce? Contrast with an access to a backed region.
4. In a multi-vCPU VMM, what is shared across vCPU threads and what is per-vCPU? (Memory regions and the VM fd vs each vCPU's regs/sregs and run loop.)
5. For a MmioRead, who fills the returned byte slice, and what does that map to in Firecracker's device model?
6. How does your run_vcpu correspond to a Firecracker per-vCPU thread? Name two things Firecracker adds (channels to the VMM thread; an exit eventfd; real device dispatch — any two).
7. Why did setting efer to long mode but leaving cr4 without PAE fail, and where would you confirm Firecracker sets all of them together?

When you can run two vCPUs in long mode, explain every long-mode precondition, and map your VMM onto vstate, you've completed Lab 1. Continue to Lab 2 — VM-Exit Taxonomy, where you stop making exits by hand and start counting every exit a real Firecracker boot produces.