The Boot Sequence

InstanceStart is the moment a configured-but-dormant Firecracker process becomes a running microVM. One PUT to /actions flows through the action channel to the PrebootApiController, which calls the builder. From there a precise sequence runs: allocate guest memory, load the kernel ELF into it, write the boot parameters the kernel expects (the "zero page" with the e820 map, the command line, the initrd pointer), set each vCPU's registers so it boots straight into 64-bit long mode, create the devices, and finally release the vCPU threads into their KVM_RUN loops. The order is not arbitrary — each step depends on the previous one. This chapter walks that sequence in builder.rs and arch/, explains why each step is where it is, and contrasts the aarch64 path (no zero page; an FDT instead).

After this chapter you will be able to: trace InstanceStart from the action to running guest code; name the artifacts of the boot protocol (zero page, e820, cmdline, initial regs) and where they are written; explain why an x86 vCPU needs long-mode setup before its first KVM_RUN; and describe how aarch64 differs.

Note: Firecracker loads an uncompressed vmlinux ELF and jumps to e_entry in 64-bit long mode — no BIOS, no bootloader, no real-mode trampoline on x86. That is a major reason microVMs boot in tens of milliseconds. The cost is that Firecracker itself must do everything a firmware/bootloader normally does: set up page tables, the GDT, long mode, and the boot params.

The trigger: InstanceStart → the builder

# Find the builder entry points called when the microVM starts.
rg -n "build_microvm_for_boot|build_and_boot_microvm|build_microvm_from_snapshot|StartMicroVm" src/vmm/src/builder.rs src/vmm/src/rpc_interface.rs

PrebootApiController handles StartMicroVm by invoking the builder with the accumulated VmResources. The builder is the choreographer of every step below. Read it top to bottom — it is the single best map of the boot path.

rg -n "fn build_microvm_for_boot" -A 80 src/vmm/src/builder.rs

The sequence (x86_64)

flowchart TD
    A["InstanceStart -> build_microvm_for_boot(VmResources)"] --> B["create guest memory (mmap + KVM_SET_USER_MEMORY_REGION)"]
    B --> C["create VM fd, irqchip (KVM_CREATE_IRQCHIP), vCPUs (KVM_CREATE_VCPU)"]
    C --> D["load vmlinux ELF into guest memory (linux-loader Elf)"]
    D --> E["load initrd if any"]
    E --> F["write boot_params / zero page: e820, cmdline ptr, initrd ptr"]
    F --> G["configure CPUID/MSRs per CPU template (SET_CPUID2)"]
    G --> H["set vCPU regs/sregs: long mode, rip=e_entry, rsi=ZERO_PAGE"]
    H --> I["create + register devices (block/net/vsock/serial/...) on the buses"]
    I --> J["advertise devices to guest (cmdline virtio_mmio.device=... / ACPI)"]
    J --> K["start vCPU threads; each enters KVM_RUN"]

Walk it with the table; locate each step with the rg beside it.

The memory layout and why constants live where they do

x86 Linux boot has a fixed protocol, and Firecracker mirrors it with layout constants:

rg -n "ZERO_PAGE_START|CMDLINE_START|HIMEM_START|BOOT_STACK|GDT|PML4|layout" src/vmm/src/arch/x86_64/layout.rs

The zero page (boot_params) is the data structure the Linux x86 boot protocol mandates. Firecracker fills it via linux-loader's bootparam types and a LinuxBootConfigurator: the e820 memory map (which physical ranges are RAM vs reserved — the MMIO gap is a hole), the pointer to the command line at CMDLINE_START, and the initrd location/size. The kernel reads all of this from the address Firecracker puts in rsi.

rg -n "configure_system|LinuxBootConfigurator|add_e820_entry|E820|setup_header" src/vmm/src/arch/x86_64/

Why long mode before the first KVM_RUN

A fresh x86 vCPU is in 16-bit real mode (see kvm-fundamentals.md). A vmlinux ELF's entry point expects 64-bit long mode with identity-mapped low memory. So before releasing the vCPU, Firecracker sets, via KVM_SET_SREGS/KVM_SET_REGS:

rg -n "setup_regs|setup_sregs|setup_page_tables|setup_gdt|EFER_LME|EFER_LMA|CR0_PE|CR4_PAE" src/vmm/src/arch/x86_64/

Get any of these wrong and the very first KVM_RUN returns KVM_EXIT_FAIL_ENTRY — the CPU could not even enter the guest. This is the single most common "guest never boots" failure for hand-rolled VMMs, which is why Lab 1.4 makes you do it yourself first.

The aarch64 contrast

aarch64 has no zero page and no e820. Instead, Firecracker builds a Flattened Device Tree (FDT/DTB) — a binary blob describing CPUs, memory, the GIC interrupt controller, and the virtio-mmio devices — using rust-vmm's vm-fdt (FdtWriter). It loads an arm64 PE Image kernel, writes the FDT into guest RAM, and passes the FDT's address in x0.

rg -n "FdtWriter|vm-fdt|create_fdt|DRAM_MEM_START|GIC|PE\b|aarch64" src/vmm/src/arch/aarch64/

The two paths converge again at "create devices, advertise them, start the vCPUs." The architecture difference is concentrated in arch/{x86_64,aarch64}/; the builder selects the right one. The default kernel command line is roughly reboot=k panic=1 pci=off nomodule 8250.nr_uarts=0 plus console=ttyS0 (varies — verify).

Reading exercise

# 1. The builder, top to bottom.
rg -n "fn build_microvm_for_boot" -A 100 src/vmm/src/builder.rs

# 2. Kernel load + entry.
rg -n "Elf|load_kernel|kernel_entry|HIMEM_START" src/vmm/src/ src/vmm/src/arch/x86_64/

# 3. Boot params / zero page.
rg -n "configure_system|LinuxBootConfigurator|E820|cmdline|ZERO_PAGE_START" src/vmm/src/arch/x86_64/

# 4. Long-mode register setup.
rg -n "setup_regs|setup_sregs|EFER_LME|CR0_PE|CR4_PAE|e_entry|rsi" src/vmm/src/arch/x86_64/

# 5. aarch64 FDT.
rg -n "FdtWriter|create_fdt|DRAM_MEM_START|x0" src/vmm/src/arch/aarch64/

# 6. Boot a real microVM and watch the kernel come up on the serial console (Lab 1.3).

Answer:

1. List the boot steps in order from InstanceStart to running guest code, and say why memory must come before kernel load and kernel load before boot params.
2. What is the zero page, what three things does Firecracker put in it, and how does the kernel find it?
3. What register state makes an x86 vCPU boot in long mode, and what happens at the first KVM_RUN if it is wrong?
4. Why does loading the kernel require recording e_entry, and where does it end up?
5. How does aarch64 describe hardware to the kernel instead of a zero page, and which register carries it?
6. Where in the tree do the x86 vs aarch64 boot differences live, and what selects between them?

Common bugs and symptoms

Validation: prove you understand this

1. Draw the boot sequence from InstanceStart to KVM_RUN, labelling each step's dependency on the prior.
2. Explain the zero page: its contents, who reads it, and how its address reaches the kernel.
3. State the exact register/control-bit setup for long mode and the failure mode if it's wrong.
4. Explain why Firecracker needs no BIOS/bootloader and what it does instead.
5. Contrast x86_64 and aarch64 boot: kernel format, hardware description, and the register that carries it.
6. Name three things that go wrong if the device-advertisement step is skipped.

Next: CPU Templates and CPUID — step 6 of the boot sequence, in depth.