Level 6: The Boot Process and Guest Memory

You have read the vCPU run loop and watched VM exits resolve (Level 4), and you have learned how the project defends itself with tests (Level 5). Now you learn the moment before the run loop spins: how a microVM goes from "a KVM_CREATE_VM fd and some mmap'd host pages" to "a Linux kernel executing its first instruction at e_entry in 64-bit long mode." This is the boot path, and it is where Firecracker's "do almost nothing" philosophy is most visible — there is no BIOS, no bootloader, no firmware, no PCI enumeration. Firecracker hands the kernel a region of memory, a structured set of boot parameters, a command line, and a register file, and lets go.

A kernel does not boot itself. Something has to parse the ELF, copy its segments into guest physical memory, decide where in that memory the RAM is and where the holes are, write a machine-readable description of all of that into a structure the kernel knows how to read, and set the CPU into exactly the state the kernel's entry code assumes. On x86_64 that description is the boot_params / "zero page" with its e820 memory map; on aarch64 it is a flattened device tree (FDT/DTB). Getting any byte of it wrong does not produce a Rust panic — it produces a guest that hangs, triple-faults, or prints garbage to a serial port that may not even be wired up yet. This is unforgiving code, and it is some of the most rewarding in the whole codebase to understand.

By the end of this level you will be able to stand at InstanceStart and narrate, hop by hop, how the builder allocates guest memory, loads the kernel with linux-loader, lays out the boot structures, programs the initial vCPU register and segment state for long mode, and only then starts the vCPU threads — and you will be able to do it for both x86_64 and aarch64.

Learning Objectives

By the end of Level 6 you must be able to:

1. Trace the boot path from build_microvm_for_boot through guest-memory creation, kernel loading, boot-parameter configuration, and initial vCPU register/segment programming, naming the function and file that owns each step (Lab 6.1).
2. Explain how Firecracker loads an uncompressed vmlinux ELF with linux-loader's Elf loader: how PT_LOAD segments are copied into guest memory and how e_entry becomes the guest's first rip.
3. Describe the x86_64 guest memory layout — the sub-1 MiB region, the zero page, the kernel command line, the GDT and page tables, the boot stack, the MMIO gap below 4 GiB, and high RAM above 4 GiB — and state the constant that pins each (Lab 6.2).
4. Explain the boot_params / zero page: the e820 memory map, the command-line pointer, and the initrd pointer, and how linux-loader's LinuxBootConfigurator writes them.
5. State the initial vCPU state for 64-bit long mode: rip = e_entry, rsi = ZERO_PAGE_START, the control registers (cr0 PE|PG, cr4 PAE, efer LME|LMA), the GDT/segment descriptors, and the identity-mapped first GiB of page tables.
6. Explain how guest RAM is allocated on the host (mmap) and registered with KVM via KVM_SET_USER_MEMORY_REGION through vm-memory's GuestMemoryMmap, and why the MMIO gap forces memory to be split into regions.
7. Describe the aarch64 boot path and how it differs: the arm64 PE Image, the FDT passed in x0, DRAM_MEM_START, and the absence of a zero page.
8. Experiment with boot configuration — boot_args, console=ttyS0, reboot=k, panic=1, an initrd, and the virtio_mmio.device=... advertisements — and predict their effect on the guest (Lab 6.3).

The Boot Path, End to End

Booting a microVM is a strictly ordered sequence inside the VMM thread, driven by the builder when the StartMicroVm action fires. Nothing here runs on a vCPU thread — the vCPU threads do not exist yet. The builder constructs the entire machine in memory, programs each vCPU's initial register file, and then unleashes the vCPU threads into their KVM_RUN loops.

flowchart TD
    A["VmmAction::StartMicroVm<br/>(builder.rs)"] --> B["create_vmm_and_vcpus:<br/>open /dev/kvm, KVM_CREATE_VM"]
    B --> C["allocate guest memory<br/>host mmap → GuestMemoryMmap<br/>→ KVM_SET_USER_MEMORY_REGION"]
    C --> D["load_kernel:<br/>linux-loader Elf::load()<br/>copy PT_LOAD segments, get e_entry"]
    D --> E["load_initrd (optional):<br/>copy initrd image high in RAM"]
    E --> F["build the kernel cmdline<br/>(linux-loader Cmdline)"]
    F --> G["configure_system / boot params:<br/>e820 map + zero page + cmdline ptr<br/>+ MPTable/ACPI (x86) or FDT (aarch64)"]
    G --> H["per vCPU: configure regs/sregs<br/>rip=e_entry, rsi=zero page,<br/>long mode, GDT, page tables"]
    H --> I["attach devices<br/>(MMIO bus, serial, virtio)"]
    I --> J["start vCPU threads<br/>each enters KVM_RUN"]
    J --> K["guest executes first instruction<br/>at e_entry"]

Read it as five phases:

1. VM and memory (the substrate): open /dev/kvm, KVM_CREATE_VM, allocate host RAM with mmap, wrap it in a GuestMemoryMmap, and register each region with KVM via KVM_SET_USER_MEMORY_REGION. The guest's physical address space now exists.
2. Kernel load (the payload): linux-loader's Elf loader parses the uncompressed vmlinux, copies each PT_LOAD segment to its guest physical address, and returns the entry point e_entry.
3. Boot structures (the contract): write the structures the kernel's entry code reads — on x86_64 the zero page (boot_params) with the e820 map, the command line at CMDLINE_START, and the initrd pointer; the CPU topology table (MPTable and/or ACPI); on aarch64 the FDT.
4. vCPU programming (the handshake): for each vCPU, set the general-purpose registers (rip, rsi/x0), the special registers (control registers, segment descriptors, GDT base), and install minimal identity-mapped page tables so the kernel starts in long mode with the first GiB mapped.
5. Devices, then go: attach the MMIO/PIO device managers, then spawn the vCPU threads — each enters KVM_RUN and the guest runs.

Note: The order is load-bearing. Boot params must be written after the cmdline and initrd are placed (because they store pointers to them), and the vCPU registers must be programmed after the kernel is loaded (because rip is e_entry, which the ELF parse produces). When you trace this in Lab 6.1, watch the ordering as carefully as the data.

The x86_64 Guest Memory Layout

Guest physical memory is not a flat slab. The kernel, the boot structures, and the holes that devices live in all sit at fixed, agreed-upon addresses. Firecracker pins these as constants. The names below are the roles; confirm the exact identifiers on your branch — they have been renamed before.

# The layout constants live here. Run this; do not trust the names from memory.
rg -n "START|GAP|HIMEM|CMDLINE|ZERO_PAGE|MMIO|MEM_32BIT|FIRST_ADDR" \
  src/vmm/src/arch/x86_64/layout.rs

GUEST PHYSICAL ADDRESS SPACE (x86_64)              (constants — verify on your branch)

0x0000_0000  ┌────────────────────────────────────────────┐
             │ real-mode IVT / BIOS data area (unused)     │
0x0000_7000  │ ZERO_PAGE_START   — boot_params (zero page) │  ← rsi points here
0x0000_8000  │ page tables (PML4/PDPT/PD), boot GDT        │
0x0000_8ff0  │ BOOT_STACK_POINTER (boot stack grows down)  │
0x0002_0000  │ CMDLINE_START     — kernel command line     │
   ...       │ (free low memory)                           │
0x0010_0000  │ HIMEM_START (1 MiB) — vmlinux loaded here   │  ← e_entry near/at here
             │ kernel image (PT_LOAD segments)             │
   ...       │ guest RAM continues upward                  │
             │ ... up to the 32-bit MMIO gap ...           │
0xD000_0000  │ MMIO_MEM_START — virtio-MMIO device windows │  ← (gap, below 4 GiB)
   ...       │ MMIO gap (devices, no RAM here)             │
0x1_0000_0000│ 4 GiB boundary                              │
             │ HIGH RAM (if mem_size pushes past the gap)  │  ← remaining guest RAM
             └────────────────────────────────────────────┘

The two structural facts to internalize:

• The sub-1 MiB region is crowded. The zero page, the boot GDT, the initial page tables, and the boot stack all live below HIMEM_START (1 MiB), in real-mode-addressable territory. The kernel image itself loads at 1 MiB.
• There is a hole below 4 GiB. Guest RAM cannot extend continuously to 4 GiB, because the region just under 4 GiB is reserved as the MMIO gap where memory-mapped devices (virtio-MMIO windows, the APIC, etc.) live. If the configured mem_size_mib is larger than the gap allows, the remainder of RAM is placed above 4 GiB as "high RAM." This split is why guest memory is multiple regions, not one.

# Where the RAM regions and the MMIO gap are computed (function names vary — verify).
rg -n "arch_memory_regions|MMIO_MEM_START|MEM_32BIT_GAP|first_usable|GuestMemoryMmap" \
  src/vmm/src/arch/x86_64/ src/vmm/src/vstate/memory.rs

The full layout cheat-sheet, including the constants for both architectures side by side, lives in the memory-layout cheat-sheet and is explained in depth in the guest memory management deep dive.

How Guest Memory Is Allocated and Registered

Firecracker does not ask KVM for memory. It allocates anonymous host memory with mmap, hands the host virtual address to KVM, and tells KVM "this host range is this guest physical range." That mapping is one KVM_SET_USER_MEMORY_REGION ioctl per region.

   HOST                                            GUEST PHYSICAL
   ┌──────────────────────────┐                    ┌──────────────────────────┐
   │ mmap(...) → userspace VA  │  KVM_SET_USER_     │ guest_phys_addr 0x0...    │
   │ (anonymous, MAP_PRIVATE)  │  MEMORY_REGION     │ memory_size = region len  │
   │ userspace_addr ───────────┼───────────────────►│ slot N                    │
   └──────────────────────────┘                    └──────────────────────────┘

The vm-memory crate's GuestMemoryMmap owns these regions and gives the rest of the codebase a safe API to read/write guest physical addresses (GuestAddress) — bounds-checked, region-aware. Every later subsystem (the kernel loader, the boot-params writer, virtio descriptor handling) goes through this abstraction rather than touching raw pointers. The rust-vmm contract is covered in the vm-memory chapter; the KVM ioctl shape is in the KVM fundamentals deep dive.

# Memory creation + KVM registration. Find where regions are mmap'd and set on the VM.
rg -n "GuestMemoryMmap|from_ranges|mmap|set_user_memory_region|KVM_SET_USER_MEMORY|memory_init|create_guest_memory" \
  src/vmm/src/vstate/memory.rs src/vmm/src/vstate/vm.rs src/vmm/src/builder.rs

Loading the Kernel: linux-loader's Elf Loader

Firecracker boots an uncompressed vmlinux ELF (not a compressed bzImage) on x86_64. The linux-loader crate's Elf (a.k.a. loader::Elf) loader parses the ELF header, walks the program headers, copies every PT_LOAD segment into guest memory at its physical address, and reports back the entry point e_entry plus where the loadable image ended.

vmlinux ELF                      GUEST MEMORY
┌────────────────┐               ┌──────────────────────────┐
│ ELF header     │               │                          │
│  e_entry ──────┼──────────────►│  (becomes initial rip)   │
│ Program headers│               │                          │
│  PT_LOAD #0 ───┼──copy───────► │  segment @ p_paddr       │
│  PT_LOAD #1 ───┼──copy───────► │  segment @ p_paddr       │
│  ...           │               │                          │
└────────────────┘               └──────────────────────────┘

# Where Firecracker calls into linux-loader to load the kernel.
rg -n "Elf|linux_loader|load_kernel|KernelLoader|load\(|e_entry|kernel_load" \
  src/vmm/src/arch/ src/vmm/src/builder.rs

The loader's job ends at "the bytes are in memory and here is e_entry." Everything that makes the kernel able to run — the boot params, the page tables, the register state — is Firecracker's job, done after the load. The linux-loader contract (the KernelLoader trait, Elf vs BzImage vs PE, and the bootparam/Cmdline helpers) is the subject of the linux-loader chapter, and the end-to-end sequence is in the boot sequence deep dive.

Required Reading

Read these before the labs. Each row tells you what to extract — read for that, not cover to cover.

# Confirm these exist on your branch before you start (paths drift after refactors).
ls src/vmm/src/arch/x86_64/ src/vmm/src/arch/aarch64/
rg -l "ZERO_PAGE_START|CMDLINE_START|HIMEM_START" src/vmm/src/arch/
cargo doc -p linux-loader --no-deps   # then open the generated docs for the loaders

Source Code Areas to Inspect

Key Types and Constants Quick Reference

Do not memorize these — run the rg and read the role. Names are stable enough to grep by role even when paths move.

GitHub Issue Categories for Level 6

Boot and memory issues a Level 6 graduate can credibly engage with:

• Boot configuration / cmdline — incorrect or surprising default cmdline behavior, escaping, length limits, virtio_mmio.device= advertisement bugs.
• Memory layout edge cases — large mem_size_mib near or past the MMIO gap, region splitting, alignment, huge-pages interactions.
• Kernel/initrd loading — ELF parsing edge cases, initrd placement, error messages when a kernel is the wrong format (e.g. a bzImage where a vmlinux is expected).
• aarch64 FDT — device-tree node correctness, FDT placement, missing properties.
• e820 / boot params — memory-map correctness, reserved-region reporting, ACPI-vs-MPTable migration.
• Documentation — the layout and boot protocol are under-documented in-tree; precise docs are genuinely welcome.

gh issue list --repo firecracker-microvm/firecracker \
  --search "boot OR e820 OR cmdline OR initrd OR memory layout OR fdt" --state open

Deliverables

You must demonstrate all of the following before advancing to Level 7:

• A written boot-path reading log tracing build_microvm_for_boot through memory creation, kernel load, boot-param configuration, vCPU register setup, and vCPU thread start, naming the function/file at each step (Lab 6.1).
• An annotated memory-layout diagram for x86_64 with each constant pinned by a rg against layout.rs on your branch, plus the aarch64 contrast (Lab 6.2).
• Empirical observation of boot config effects: changing boot_args (console=, reboot=k, panic=1, a custom param), adding an initrd, and reading the resulting cmdline (including the virtio_mmio.device= advertisements) in the boot log (Lab 6.3).
• From memory: the initial long-mode vCPU state (rip, rsi, cr0/cr4/efer, GDT, identity page tables) and why each value; and why guest RAM is split into regions around the MMIO gap.

Common Mistakes

How to Verify Success

# 1) You can locate every boot step from a grep chain.
rg -n "fn build_microvm_for_boot" src/vmm/src/builder.rs
rg -n "load_kernel|Elf|e_entry" src/vmm/src/arch/ src/vmm/src/builder.rs
rg -n "ZERO_PAGE_START|CMDLINE_START|HIMEM_START|MMIO_MEM_START" src/vmm/src/arch/x86_64/layout.rs
rg -n "configure_system|boot_params|e820|FdtWriter" src/vmm/src/arch/

# 2) You can boot and read what the guest actually got, from the serial log.
#    (Boot a microVM per docs/getting-started.md; capture console=ttyS0 output.)
#    Look for: "Command line: ..." and "BIOS-e820: [mem ...]" lines printed by the kernel.

# 3) You can register guest memory by hand (this is rust-vmm Lab r1/r2 territory).
rg -n "set_user_memory_region|KVM_SET_USER_MEMORY" src/vmm/src/vstate/vm.rs

When you can narrate the boot path, draw the memory layout from the constants on your branch, and predict how a boot_args change shows up in the kernel's printed cmdline, you are ready for the device model in Level 7.

PR Profile: Level 6 Graduate

A graduate of this level can credibly open these kinds of PRs:

These are real, mergeable contribution shapes — not toy exercises. The boot path is well-trodden but still produces genuine bugs, and precise documentation of it is perennially welcome.

Begin with Lab 6.1: Trace the Boot Sequence. Lean throughout on the boot sequence deep dive, the guest memory management deep dive, the ACPI and MPTable deep dive, and the rust-vmm chapters on linux-loader and vm-memory.