The Boot Process — Intensive

You have already traced the boot sequence once. In Level 6 you followed InstanceStart from a PUT on /actions through the action channel into the builder, watched guest memory get allocated, the kernel get loaded, the zero page get filled, and the vCPU threads get released into KVM_RUN. The boot-sequence deep dive gave you the map. This masterclass is about the territory — the actual bytes.

Here you stop reading "Firecracker loads the kernel ELF and jumps to e_entry" as a sentence and start treating it as a claim you can falsify. You will open a real vmlinux with readelf, find its PT_LOAD segments and entry point, then follow Firecracker's linux-loader-driven loader and arch/ placement code until you can predict — to the guest-physical byte — where each segment lands and which instruction the vCPU executes first. You will dissect the zero page (boot_params) field by field: the e820 memory map, the command-line pointer, the initrd pointer, and where the kernel reads each of them. And you will cross the architecture boundary into aarch64, where there is no zero page at all — a Flattened Device Tree built with vm-fdt describes the machine instead, and the kernel is a PE Image that gets the FDT's address in x0.

This intensive will not hand you line numbers. Code moves between branches, and a contributor who has memorized layout.rs constants is already wrong on the next release. Every structure named here comes with the rg/find/readelf command that locates it on your checkout, and the version-sensitive facts are flagged so you verify them against your branch and the CHANGELOG. The work is to run everything.

Note: The single fact that makes the whole x86 boot path comprehensible: Firecracker has no BIOS, no bootloader, and no real-mode trampoline. It loads an uncompressed vmlinux ELF, builds the boot params itself, sets the vCPU into 64-bit long mode by hand, and jumps straight to e_entry. Everything a firmware would normally do, Firecracker does in arch/x86_64/ — which is exactly why a microVM boots in tens of milliseconds and exactly why this code is dense.

What you will be able to do

By the end of this masterclass you can:

1. Open an arbitrary vmlinux with readelf -l, read its program headers, and state where each PT_LOAD segment will be copied in guest physical memory and why.
2. Trace Firecracker's kernel loader (linux-loader's Elf loader) from the file bytes on disk to the recorded e_entry, and connect that entry to the rip the first vCPU starts at.
3. Reproduce the x86_64 guest-physical layout — zero page, cmdline, HIMEM_START, the MMIO gap — from memory, and locate each constant in arch/x86_64/layout.rs.
4. Dissect boot_params: name the fields Firecracker fills (the e820 entries, the cmdline pointer, the initrd ramdisk pointer/size, the setup header), and observe the result from inside the guest via dmesg and /sys/firmware/memmap.
5. Read the aarch64 boot path: enumerate the FDT nodes Firecracker builds (memory, cpus, chosen/bootargs, the GIC interrupt controller, the virtio-mmio devices), explain why x0 carries the FDT address, and contrast the whole thing with the x86 zero page.
6. Dump and decompile a real DTB, match it node-for-node to Firecracker's FdtWriter calls, and explain what would break if a node were malformed.

Prerequisites

This is an intensive, not an introduction. You must arrive having done the work below — the labs assume it and will not re-teach it.

# One-shot prerequisite check. All three must succeed.
cd ~/firecracker
test -f build/cargo_target/x86_64-unknown-linux-musl/release/firecracker && echo "firecracker built"
readelf --version | head -1
rg -q "fn build_microvm_for_boot" src/vmm/src/builder.rs && echo "builder present"

Tip: Even if you are on x86_64 hardware, you must complete Lab 3 (aarch64 FDT). You will read the aarch64 code and inspect a dumped DTB rather than boot an arm64 guest — but a Firecracker contributor who only understands one architecture's boot path will break the other one. The boot code is one of the two places (the other is CPU templates) where x86 and aarch64 diverge hardest, and reviewers expect you to reason about both.

The three labs

This intensive is three labs, taken in order. Each is a trace-it lab: you read and instrument a real path, you inspect real binaries, and you tie the two together. The deliverables gate progression.

flowchart LR
    A["Lab 1<br/>kernel ELF → guest RAM<br/>(PT_LOAD, e_entry)"] --> B["Lab 2<br/>boot_params<br/>(e820, cmdline, initrd)"]
    B --> C["Lab 3<br/>aarch64 contrast<br/>(FDT instead of zero page)"]

The arc is deliberate. Lab 1 establishes where the kernel goes; Lab 2 establishes what the kernel is told about its hardware on x86; Lab 3 shows that the second question has a completely different answer on aarch64 even though the first is nearly the same. After all three you can read arch/x86_64/ and arch/aarch64/ side by side and explain, with the bytes in front of you, why a microVM boots the way it does.

How the boot path sits in the tree

Keep this mapping open as you work. Every row is something you will locate yourself with the rg beside it — do not trust the path, run the command, because the crate merge moved many of these and they drift between branches.

Common mistakes this masterclass corrects

How to verify success

You have completed this intensive when, on your own checkout and with no notes, the following all hold:

# 1. You can dump a kernel's load segments and entry, unaided.
readelf -l ~/firecracker/vmlinux-6.1.* | sed -n '1,30p'
readelf -h ~/firecracker/vmlinux-6.1.* | grep "Entry point"

# 2. You can locate every boot artifact in the source by role, not line number.
rg -n "fn build_microvm_for_boot" src/vmm/src/builder.rs
rg -n "Elf|load_kernel|kernel_entry" src/vmm/src/
rg -n "configure_system|LinuxBootConfigurator|E820|add_e820_entry" src/vmm/src/arch/x86_64/
rg -n "FdtWriter|create_fdt|DRAM_MEM_START" src/vmm/src/arch/aarch64/

# 3. You can read the e820 map from inside a booted guest.
#    (in the guest:)  dmesg | grep -i e820 ; cat /sys/firmware/memmap/*/{start,end,type}

And you can answer, cold: Where does the kernel's entry instruction live in guest physical memory, how did it get there, and how did the vCPU come to start at it? Plus the same for the boot params: what does the kernel read at the address in rsi (x86) or x0 (aarch64), and who wrote it?

Where this leads

The boot path is the foundation the rest of the machine stands on. Once the kernel is running it immediately starts probing the virtio devices Firecracker advertised to it (on x86 via the cmdline you'll dissect in Lab 2, on aarch64 via the FDT nodes you'll read in Lab 3) — which is the subject of the virtio-devices masterclass. And the entire boot sequence runs again, differently, when a microVM is restored from a snapshot (build_microvm_from_snapshot instead of build_microvm_for_boot) — the snapshotting masterclass is where you'll see how the boot artifacts you traced here are serialized and reconstructed.

This intensive deepens Level 6 and feeds issue-roadmap Stage 5 (device config) and the boot-time work in engineering/boot-time-optimization.

Next: Lab 1 — Trace the Kernel Load. Then Lab 2 — The Zero Page and the e820 Map and Lab 3 — The aarch64 Boot Path.