Lab 6.1: Trace the Boot Sequence

Background

You cannot reason about a boot bug until you can see the boot. In this lab you follow the construction of a microVM from the StartMicroVm action down to the moment the first vCPU thread enters KVM_RUN — and you tie the guest's very first executed instruction to two concrete facts produced earlier in the path: the entry point e_entry returned by the ELF loader (which becomes the initial rip) and the zero-page address (which is handed to the kernel in rsi).

This is a code-reading trace, the same discipline you practiced on the API path in Lab 3.1, now applied to the builder. The boot path is almost entirely sequential — one long, ordered procedure inside the VMM thread — which makes it an ideal trace: there is a single thread of control, no concurrency, and the ordering itself carries meaning. Your output is a boot-path reading log: one row per step, naming the file and function, the data it produces, and the data it consumes from earlier steps.

Note: This lab is timed (Step 0). The point is not to read every line of arch/ — it is to map the spine of the boot path and pin the two or three places where the guest's initial state is decided. Pair it with the boot sequence deep dive and the Level 6 overview diagram.

Why This Lab Matters for Contributors

Every boot failure, memory-layout edge case, and "the guest hangs immediately" report lands somewhere on this path. Knowing it turns "no idea where to start" into "this is a configure_system problem" or "this is a register-setup problem."
Maintainers describe boot bugs by function (build_microvm_for_boot, load_kernel, the regs/sregs setup). You must be able to open the right one and orient in seconds.
The trace reveals the ordering constraints that make the boot correct — and ordering bugs (writing boot params before the cmdline is placed, programming rip before the kernel is loaded) are a real class of regressions.
The two "first-instruction" facts (rip = e_entry, rsi = zero page) are the bridge between what Firecracker set up and what the guest kernel assumes — the x86 boot protocol contract. See the boot sequence deep dive and the linux-loader chapter.

Prerequisites

Completed Level 4 (you can read the vCPU run loop) and the Level 6 overview.
A built Firecracker and the ability to boot a microVM by hand (from Lab 1.3). Verify:
```
ls build/cargo_target/*/release/firecracker || tools/devtool build --release
```
The repo open with rg and "go to definition" working on vmm crate symbols.

# Confirm the builder and arch modules are where you expect before you start.
rg -n "fn build_microvm_for_boot|fn build_and_boot_microvm" src/vmm/src/builder.rs
ls src/vmm/src/arch/x86_64/ src/vmm/src/arch/aarch64/

Step-by-Step Tasks

Step 0: Set a timer and prepare the reading log

Give yourself 75 minutes. Create a scratch file and fill one row per step as you go:

STEP | FILE | fn name             | PRODUCES (data out)        | CONSUMES (from earlier)
-----+------+---------------------+----------------------------+-------------------------

For each step answer only: what does this produce, and which later step needs it? You are drawing a data-flow map, not auditing the code.

Step 1: Find the boot orchestrator

The whole boot path hangs off one function in the builder. Find it and read its top-level structure — the sequence of calls is your skeleton.

rg -n "fn build_microvm_for_boot|fn build_and_boot_microvm|fn create_vmm_and_vcpus" \
  src/vmm/src/builder.rs
# Then read the body of build_microvm_for_boot top to bottom and list the calls in order.
grep -n "load_kernel\|load_initrd\|configure_system\|create_guest_memory\|attach\|vcpu\|cmdline\|boot" \
  src/vmm/src/builder.rs | head -40

Record the ordered list of calls build_microvm_for_boot makes. That list is the lab. Every subsequent step zooms into one of them.

Note: Names drift. If build_microvm_for_boot isn't the exact name on your branch, grep for the action that triggers boot and follow it: rg -n "StartMicroVm" src/vmm/src/rpc_interface.rs src/vmm/src/builder.rs.

Step 2: Step 1 of the path — VM and guest memory

Before anything is loaded, the VM and its memory must exist. Find where /dev/kvm is opened, the VM fd is created, host memory is mmap'd, and the regions are registered with KVM.

rg -n "create_guest_memory|GuestMemoryMmap|from_ranges|mmap" \
  src/vmm/src/vstate/memory.rs src/vmm/src/builder.rs
rg -n "set_user_memory_region|KVM_SET_USER_MEMORY|create_vm|Vm::new" \
  src/vmm/src/vstate/vm.rs

Record: the guest physical address space now exists as a GuestMemoryMmap, split into regions around the MMIO gap (the split itself is computed in arch/x86_64/; see Lab 6.2). Everything downstream writes into this memory.

Step 3: Step 2 of the path — load the kernel

This is where the payload arrives. Find the call into linux-loader and capture what it returns.

rg -n "fn load_kernel|Elf|linux_loader|KernelLoader|\.load\(" \
  src/vmm/src/arch/ src/vmm/src/builder.rs

Record three things:

Firecracker opens the vmlinux ELF and calls linux-loader's Elf loader, which copies the PT_LOAD segments into the GuestMemoryMmap from Step 2.
The loader returns an entry point — e_entry — and the highest loaded address.
This is the source of the guest's initial rip. Mark it: e_entry will reappear in Step 6.

# Pin where e_entry / the kernel entry point is captured and stored for later.
rg -n "e_entry|kernel_load|kernel_entry|entry_addr|KernelLoaderResult" \
  src/vmm/src/arch/ src/vmm/src/builder.rs

Step 4: Step 3 of the path — initrd (optional) and the command line

If an initrd was configured (boot-source.initrd_path), it is copied high in RAM and its address/size recorded. Then the kernel command line is assembled with linux-loader's Cmdline.

rg -n "load_initrd|initrd|InitrdConfig" src/vmm/src/arch/ src/vmm/src/builder.rs
rg -n "Cmdline|cmdline|insert|virtio_mmio.device" src/vmm/src/ | head -30

Record: the cmdline is built up from the user's boot_args plus device advertisements the VMM appends (notably virtio_mmio.device=SIZE@ADDR:IRQ per MMIO device on x86 — you will see these in Lab 6.3). The finished cmdline is written into guest memory at CMDLINE_START, and its address + the initrd address are about to be stored into the zero page.

Step 5: Step 4 of the path — boot parameters (zero page / e820 / FDT)

Now the structures the kernel reads. On x86_64 this is configure_system writing the zero page (boot_params) — the e820 memory map, the pointer to the cmdline, the initrd pointer — plus the CPU topology (MPTable and/or ACPI). On aarch64 it is the FDT.

rg -n "fn configure_system|boot_params|BootParams|e820|add_e820|LinuxBootConfigurator|setup_boot|mptable|ACPI|rsdp|madt" \
  src/vmm/src/arch/x86_64/
rg -n "fn configure_system|FdtWriter|create_fdt|fdt" src/vmm/src/arch/aarch64/

Record, in order:

The e820 map is built from the guest memory regions (Step 2) — it tells the kernel which physical ranges are usable RAM and which are reserved (the MMIO gap, low reserved areas).
The cmdline pointer (from Step 4) and the initrd pointer (from Step 4) are stored into the zero page.
The zero page is written to guest memory at ZERO_PAGE_START. Mark this address — it reappears in Step 6 as rsi.
The CPU topology table (MPTable today, ACPI increasingly — see the ACPI and MPTable deep dive) is written so the kernel sees the configured number of vCPUs.

Warning: This step must run after Steps 3–4, because it stores pointers to the cmdline and initrd. An ordering regression here (boot params before cmdline placement) produces a guest that boots with a garbage or empty command line. This is exactly the kind of bug the reading log exposes.

Step 6: Step 5 of the path — program the initial vCPU state

For each vCPU, the builder sets the registers the kernel's 64-bit entry code assumes. This is where e_entry (Step 3) and ZERO_PAGE_START (Step 5) are consumed.

rg -n "fn configure|set_regs|set_sregs|kvm_regs|kvm_sregs|rip|rsi|cr0|cr3|cr4|efer|gdt|setup_page_tables|configure_segments" \
  src/vmm/src/arch/x86_64/regs.rs src/vmm/src/arch/x86_64/
rg -n "set_one_reg|x0|PSTATE|MPIDR|configure" src/vmm/src/arch/aarch64/

Record the x86_64 long-mode handshake:

Register / structure	Value	Why
`rip`	`e_entry` (Step 3)	The guest's first instruction is the kernel entry point
`rsi`	`ZERO_PAGE_START` (Step 5)	The x86 boot protocol passes `boot_params` in `rsi`
`cr0`	PE \| PG	Protected mode + paging on
`cr4`	PAE	Physical address extension (required for long mode)
`efer`	LME \| LMA	Long Mode Enable + Active
`cr3`	base of boot page tables	Points at the identity-mapped page tables built low in memory
GDT + CS/DS/...	64-bit flat descriptors	Long-mode flat segments; CS marked long (L bit)

# Where the boot GDT and identity page tables are built (function names vary — verify).
rg -n "write_gdt|gdt_entry|BOOT_GDT|setup_page_tables|PML4|PDPTE|identity" \
  src/vmm/src/arch/x86_64/

On aarch64 there is no zero page: the FDT address is placed in x0, the entry is the PE Image load address, and pc is set accordingly. Record the contrast.

Step 7: Step 6 of the path — devices, then start the vCPU threads

Devices (the MMIO bus, the serial console, virtio devices) are attached, and finally the vCPU threads are spawned. Each enters its KVM_RUN loop — the loop you read in Lab 4.1 — and the guest begins executing at e_entry.

rg -n "attach_.*device|MMIODeviceManager|PortIODeviceManager|start_vcpus|vcpu.*spawn|thread::Builder|run\(" \
  src/vmm/src/builder.rs src/vmm/src/lib.rs

Record: the order is devices first, then vCPUs. A device that a vCPU touches must exist on the bus before the vCPU runs, or the first MMIO access faults into nothing.

Step 8: Tie the trace to a real boot

Make the path concrete. Boot a microVM with the serial console enabled and read what the kernel itself prints about the state Firecracker handed it:

API=/tmp/fc.sock
sudo ./firecracker --api-sock $API &
curl -X PUT --unix-socket $API --data \
 '{"kernel_image_path":"./vmlinux-6.1.x","boot_args":"console=ttyS0 reboot=k panic=1"}' \
 http://localhost/boot-source
curl -X PUT --unix-socket $API --data \
 '{"drive_id":"rootfs","path_on_host":"./ubuntu-24.04.ext4","is_root_device":true,"is_read_only":false}' \
 http://localhost/drives/rootfs
curl -X PUT --unix-socket $API --data '{"action_type":"InstanceStart"}' http://localhost/actions

In the serial output, find the lines the kernel prints that echo the structures from your trace:

Command line: console=ttyS0 reboot=k panic=1 ...        ← your cmdline (Step 4), via the zero page
BIOS-provided physical RAM map:
BIOS-e820: [mem 0x0000000000000000-0x...] usable        ← your e820 map (Step 5)
BIOS-e820: [mem 0x...] reserved                          ← the MMIO gap / reserved low memory

Those e820 lines are literally what configure_system wrote in Step 5, read back by the guest kernel. The Command line: line is what you placed at CMDLINE_START in Step 4. You have now seen both ends of the contract: the code that writes the structure and the kernel that consumes it.

Implementation Requirements / Deliverables

A completed boot-path reading log: one row per step (Steps 1–7), each naming the file and function, what it produces, and what earlier output it consumes.
The two first-instruction facts pinned precisely: where e_entry is captured (Step 3) and becomes rip (Step 6), and where ZERO_PAGE_START is written (Step 5) and becomes rsi (Step 6).
The long-mode register table (Step 6) reproduced with the why for each entry.
The ordering constraints stated: why boot params come after cmdline/initrd, and why devices are attached before vCPU threads start.
The serial-log evidence (Step 8): the Command line: and BIOS-e820: lines, mapped back to the functions that produced them.

VmmAction::StartMicroVm
  → build_microvm_for_boot                 (builder.rs)        [orchestrates everything below]
  → create_guest_memory → GuestMemoryMmap  (vstate/memory.rs)  [mmap regions, split at MMIO gap]
       → KVM_SET_USER_MEMORY_REGION         (vstate/vm.rs)      [register each region]
  → load_kernel → linux-loader Elf::load    (arch/, builder.rs) [copy PT_LOAD; returns e_entry]   ●rip
  → load_initrd (optional) + build Cmdline  (arch/, builder.rs) [place initrd; cmdline @ CMDLINE_START]
  → configure_system                        (arch/x86_64/)      [e820 + zero page @ ZERO_PAGE_START]  ●rsi
       → MPTable / ACPI (RSDP/MADT)          (arch/, acpi-tables/)
  → per-vCPU regs/sregs setup               (arch/x86_64/regs.rs) [rip=e_entry, rsi=zero page, long mode]
  → attach devices (MMIO bus, serial, virtio)
  → start vCPU threads → KVM_RUN            (lib.rs, vstate/vcpu/) [guest runs at e_entry]

The two ● marks are the heart of the lab: e_entry flows into rip, ZERO_PAGE_START flows into rsi.

Stretch Goals

Add tracing. Firecracker uses the log-instrument tooling; or add temporary log::debug! / eprintln! lines at the head of each boot step (memory, kernel load, cmdline, configure_system, regs setup, vCPU start), rebuild, boot, and watch the order print. Confirm it matches your log.
```
rg -n "log_enabled|debug!|instrument" src/vmm/src/builder.rs
```
Print the actual e_entry. Add a debug line that logs the entry address returned by the loader and the value written to rip. Confirm they are identical.
Print the e820 map. Log each e820 entry as configure_system adds it, boot, and diff your logged entries against the kernel's printed BIOS-e820: lines. They should match exactly.
Diff aarch64. Repeat Steps 3, 5, 6 against arch/aarch64/. Where x86 writes a zero page, aarch64 writes an FDT; where x86 sets rsi, aarch64 sets x0. Produce a side-by-side table.
Find the snapshot path. build_microvm_from_snapshot skips the kernel load and boot-param setup (the state is restored instead). Read it and note exactly which boot steps it omits and why.
```
rg -n "fn build_microvm_from_snapshot" src/vmm/src/builder.rs
```

Validation / Self-check

Answer without notes; these gate completion:

Name the boot steps in order from StartMicroVm to the first guest instruction, with the owning function at each.
Where does the guest's initial rip come from, and which function produced that value?
Where does rsi point on x86_64, and what structure lives there? Why does the kernel need it?
Why must configure_system (boot params) run after the cmdline and initrd are placed? What breaks if the order is swapped?
List the control-register state for 64-bit long mode (cr0, cr4, efer) and say what each bit enables.
Why are devices attached before the vCPU threads start?
On aarch64, what replaces the zero page, and in which register is its address passed to the kernel?

When your reading log narrates the full boot path without looking it up, proceed to Lab 6.2: Guest Memory Layout.

Open-Source Engineer & Contributor