Lab 3: The aarch64 Boot Path — The Flattened Device Tree

Background

Everything you traced in Lab 1 and Lab 2 is x86-specific in one way that goes deeper than registers: the zero page. There is no boot_params on aarch64, no e820 map, no rsi-points-at-a-struct convention. arm64 Linux is told about its hardware through a completely different mechanism — a Flattened Device Tree (FDT), a binary blob (a DTB, device-tree blob) describing the machine as a tree of nodes: memory, CPUs, the interrupt controller, the timer, the devices. Firecracker builds this tree at boot with rust-vmm's vm-fdt (FdtWriter), writes it into guest RAM, and passes its address to the kernel in x0. The kernel is not a vmlinux ELF either — it's an arm64 PE Image.

This lab makes you fluent in that path. You will read Firecracker's arch/aarch64/ FDT-building code and enumerate every node it creates; you will dump and decompile a real DTB and match it node-for-node to those FdtWriter calls; and you will write out, precisely, how the aarch64 boot protocol differs from the x86 zero page. The contrast is the point: a contributor who touches shared boot code must keep both protocols in their head, because the builder selects between them and a change that looks innocent on x86 can corrupt the device tree on arm64.

You are almost certainly on x86_64 hardware. That is fine and expected. This is a read-and-inspect lab: you will read the aarch64 source on your existing checkout (the code is there regardless of your host arch) and decompile a DTB you generate or obtain. You will not need an arm64 machine. The Firecracker tree compiles arch/aarch64/ into the source you already have; rg works on it identically.

Why This Lab Matters for Contributors

• The FDT is the boot protocol on aarch64 — it carries everything the x86 zero page, e820 map, cmdline pointer, MPTable/ACPI tables, and virtio advertisement carry, all in one tree. A malformed node (wrong reg, wrong interrupt) hangs the guest at entry with no console output, the hardest class of boot bug to diagnose.
• Boot code is one of exactly two places (CPU templates are the other) where x86 and aarch64 diverge most. Reviewers expect a boot-touching PR to reason about both. "I only tested x86" is not acceptable for a change in shared builder.rs paths.
• vm-fdt is a rust-vmm crate Firecracker consumes (and the device-tree concept recurs across the rust-vmm ecosystem). Understanding FdtWriter here is transferable to Cloud Hypervisor and anywhere arm64 virtualization lives.

Prerequisites

• Lab 1 and Lab 2: you can fully account for the x86 boot path, which is what you'll contrast against.
• The boot-sequence deep dive's aarch64 section.
• dtc (the device tree compiler) installed for decompiling DTBs.

cd ~/firecracker
dtc --version                       # the device tree compiler
rg -q "FdtWriter|create_fdt" src/vmm/src/arch/aarch64/ && echo "aarch64 FDT code present"
rg -n "DRAM_MEM_START|GIC|layout" src/vmm/src/arch/aarch64/

Tip: Install dtc via your package manager (device-tree-compiler on Debian/Ubuntu, dtc on Fedora/Arch/Homebrew). It both compiles .dts → .dtb and decompiles .dtb → readable .dts, which is how you'll inspect the blob.

Step-by-Step Tasks

Step 1: Find the FDT builder

Firecracker builds the device tree in its aarch64 arch module. Locate the entry:

cd ~/firecracker
rg -n "create_fdt|FdtWriter|fn fdt|begin_node|end_node|property" src/vmm/src/arch/aarch64/

You are looking for a function (historically create_fdt) that constructs an FdtWriter, then opens and closes nodes (begin_node/end_node) and sets properties (property_u32, property_string, property_array_u64, …) to describe the machine. Read it top to bottom — like build_microvm_for_boot on x86, it is the single best map of the aarch64 boot configuration.

# Read the whole FDT builder with context:
rg -n "fn create_fdt" -A 120 src/vmm/src/arch/aarch64/

Step 2: Enumerate the nodes Firecracker builds

As you read, list every node. The standard set Firecracker emits (verify on your branch — nodes get added/refined across releases):

The single most important contrast to internalize:

  x86_64                              aarch64
  ───────────────────────────────    ───────────────────────────────
  boot_params (zero page)            a Flattened Device Tree (DTB)
   ├─ e820 map           ──────────►  /memory  reg=<base size>
   ├─ hdr.cmd_line_ptr   ──────────►  /chosen  bootargs="..."
   ├─ hdr.ramdisk_image  ──────────►  /chosen  linux,initrd-start/-end
   ├─ MPTable / ACPI MADT ─────────►  /cpus + /intc (GIC)
   └─ cmdline virtio_mmio.device= ─►  /virtio_mmio@ADDR nodes
  written at ZERO_PAGE_START          written into guest RAM
  address passed in rsi               address passed in x0
  kernel = vmlinux ELF (e_entry)      kernel = arm64 PE Image

Everything the x86 path scatters across the zero page, the cmdline, and the ACPI tables, the aarch64 path concentrates into one tree. That is why the FDT is "the boot protocol" on arm64 in a way nothing single thing is on x86.

Step 3: Find the virtio-mmio nodes — device advertisement

On x86, devices are advertised on the cmdline (the virtio_mmio.device= tokens you met in Lab 2). On aarch64, each device gets an FDT node. Find where Firecracker adds them:

rg -n "virtio_mmio|virtio,mmio|fn .*fdt.*device|interrupts\b|reg\b|MmioDeviceInfo|DeviceInfoForFDT" \
  src/vmm/src/arch/aarch64/ src/vmm/src/device_manager/

Each node carries the device's MMIO window (reg = <addr size>) and its interrupt (interrupts = <...>). This is the aarch64 expression of the same fact you'll trace on x86 in the virtio-devices masterclass: the guest must be told where each device lives, and the transport is virtio-mmio either way — only the advertisement mechanism (cmdline vs FDT node) differs.

Step 4: Find the kernel format and the x0 hand-off

On aarch64 the kernel is a PE Image, not an ELF, and the FDT address goes in x0:

# Kernel format (PE/Image loader) and the register setup:
rg -n "PE\b|Image\b|pe_image|arm64|load_kernel|Loader" src/vmm/src/arch/aarch64/ src/vmm/src/
# The fdt address into x0, the entry, and the DRAM base:
rg -n "x0\b|regs\[0\]|set_one_reg|fdt_addr|DRAM_MEM_START|FDT_MAX_SIZE|entry" src/vmm/src/arch/aarch64/

Note that the x86 e_entry-from-ELF mechanism from Lab 1 has no analog: a PE Image has a fixed entry convention. Read how your branch derives the entry and the load address for the Image.

Step 5: Get a real DTB to inspect

You need a concrete device tree to match against the code. Two routes:

Route A — extract from a running aarch64 guest (if you have arm64 access, e.g. a Graviton/*.metal instance): a booted aarch64 Linux exposes its tree at /sys/firmware/fdt (the raw blob the kernel got in x0):

# On an aarch64 guest, inside it:
cp /sys/firmware/fdt /tmp/guest.dtb
dtc -I dtb -O dts /tmp/guest.dtb -o /tmp/guest.dts
sed -n '1,120p' /tmp/guest.dts

Route B — no arm64 hardware (the common case): obtain a representative arm64 virt DTB. QEMU can emit one for the virt machine, which is close in spirit to Firecracker's tree (memory, cpus, gic, virtio-mmio):

# Generate a reference arm64 'virt' device tree without booting anything:
qemu-system-aarch64 -machine virt,dumpdtb=/tmp/virt.dtb -cpu cortex-a57 -nographic 2>/dev/null
dtc -I dtb -O dts /tmp/virt.dtb -o /tmp/virt.dts
sed -n '1,160p' /tmp/virt.dts

Note: The QEMU virt tree is a reference, not Firecracker's exact tree — Firecracker's is more minimal. Use it to learn the grammar of an FDT (how /memory, /cpus, /intc, /virtio_mmio@... nodes look), then map that grammar onto what create_fdt actually emits. The authoritative tree is the one Firecracker's code builds; the DTB just makes the format legible.

Step 6: Match the decompiled DTB to the FdtWriter calls

This is the core exercise. Put /tmp/virt.dts (or /tmp/guest.dts) beside create_fdt and reconcile them node by node:

# In the DTS, find the nodes:
grep -nE "memory@|cpus \{|cpu@|chosen \{|bootargs|intc|gic|virtio_mmio@|timer" /tmp/virt.dts

# In the source, find the matching FdtWriter calls:
rg -n "begin_node|memory|cpus|cpu@|chosen|bootargs|intc|gic|virtio_mmio|timer|property" \
  src/vmm/src/arch/aarch64/

For each node in the DTS, point at the FdtWriter call in the source that produces it. Build the table yourself:

When every node in the blob has a corresponding begin_node/property call in the source, you have proven you understand how Firecracker describes an arm64 machine — the same standard of proof Lab 1 demanded with entry_addr == e_entry and Lab 2 demanded with the e820 reconciliation.

Step 7: Write the contrast

Produce a short written comparison (the deliverable). For each of the following, name the x86 mechanism and the aarch64 mechanism: (a) how RAM is described, (b) how the cmdline reaches the kernel, (c) how the initrd is located, (d) how CPUs/topology are advertised, (e) how the interrupt controller is described, (f) how virtio devices are advertised, (g) the kernel format, and (h) the register that carries the hardware-description pointer. You built every row of this across the three labs.

Implementation Requirements / Deliverables

• The rg output locating create_fdt (or its successor) and the per-node begin_node/property calls, with the file noted.
• An enumerated list of every FDT node Firecracker builds, each with the kernel hardware fact it conveys.
• A decompiled DTB (.dts) — from a guest or the QEMU virt reference — and a node-by-node table mapping its nodes to FdtWriter calls.
• The located virtio-mmio node code and a sentence contrasting it with the x86 cmdline virtio_mmio.device= advertisement.
• The written x86-vs-aarch64 contrast covering all eight points in Step 7.

Troubleshooting

rg for create_fdt finds nothing

Renamed across refactors. Anchor on vm-fdt's API: rg -n "FdtWriter|begin_node|property_u32|property_string|FdtWriterNode" src/vmm/src/arch/aarch64/. FdtWriter is the rust-vmm type and is stable.

dtc -I dtb -O dts fails: "FDT_ERR_BADMAGIC"

The file isn't a valid DTB — you may have grabbed a .dts already, or the QEMU dumpdtb didn't write (check the path and that qemu-system-aarch64 is installed). file /tmp/virt.dtb should report "Device Tree Blob".

No qemu-system-aarch64 and no arm64 hardware

Use any reference virt.dts you can find in the QEMU or kernel docs/test data — the goal is to learn FDT grammar. As a last resort, hand-read the kernel's Documentation/devicetree/bindings/ for the memory, cpus, and arm,gic bindings, then map those required properties onto create_fdt.

I can't tell which GIC version Firecracker emits

Read the GIC node's compatible string in the source (rg -n "gic-v2\|gic-v3\|arm,gic\|GICv2\|GICv3\|gic" src/vmm/src/arch/aarch64/). The version determines the node's reg regions and compatible. Firecracker selects based on host/KVM support — verify on your branch.

Expected Output

$ dtc -I dtb -O dts /tmp/virt.dtb | sed -n '1,30p'
/dts-v1/;
/ {
    #address-cells = <0x2>;
    #size-cells = <0x2>;
    compatible = "linux,dummy-virt";
    memory@40000000 { device_type = "memory"; reg = <0x0 0x40000000 0x0 0x8000000>; };
    cpus { #address-cells = <0x1>; cpu@0 { device_type = "cpu"; reg = <0x0>; }; };
    intc@8000000 { compatible = "arm,gic-v3"; interrupt-controller; ... };
    virtio_mmio@a000000 { compatible = "virtio,mmio"; reg = <0x0 0xa000000 0x0 0x200>; interrupts = <...>; };
    chosen { bootargs = "console=ttyAMA0 ..."; };
};

$ rg -n "begin_node|memory|cpus|chosen|gic|virtio_mmio" src/vmm/src/arch/aarch64/
... (the FdtWriter calls that produce each of the above) ...

Every node in the blob maps to a call in create_fdt. The mapping is the lab.

Stretch Goals

1. Diff the two architectures' device advertisement. Put the x86 cmdline virtio_mmio.device=SIZE@ADDR:IRQ tokens (Lab 2) beside the aarch64 virtio_mmio@ADDR { reg; interrupts; } nodes for the same device set. They encode the same three facts (size, address, IRQ) two different ways. Write the correspondence explicitly.
2. Trace the FDT size constant. Find FDT_MAX_SIZE (or equivalent) and where the blob is placed in guest RAM relative to DRAM_MEM_START (rg -n "FDT_MAX_SIZE\|fdt_addr\|fdt_offset\|DRAM_MEM_START" src/vmm/src/arch/aarch64/). Why does the FDT's placement matter for the memory layout the same way the zero page's does on x86?
3. Read vm-fdt directly. Skim the rust-vmm vm-fdt crate's FdtWriter API (rg -n '^name = "vm-fdt"' Cargo.lock to get the version, then read its docs). Note how begin_node/end_node enforce a balanced tree and how properties are typed. This is the rust-vmm boundary again.
4. The PSCI/reset node. Find how the guest is told how to reset/power-off (rg -n "psci\|PSCI\|cpu_off\|system_reset" src/vmm/src/arch/aarch64/) and connect it to signals, shutdown and reset. On x86 the i8042 partial controller catches reset; on aarch64 it's PSCI in the FDT.

Validation / Self-check

Answer without notes. These gate completion.

1. What replaces the x86 zero page on aarch64, what form does it take in memory, and which register carries its address to the kernel?
2. List five FDT nodes Firecracker builds and the kernel fact each conveys.
3. How is the kernel command line passed on aarch64, and how does that differ from x86?
4. How are virtio-mmio devices advertised on aarch64 vs x86? What three facts does each mechanism encode?
5. What is the kernel format on aarch64, and how does the entry/load differ from the x86 vmlinux ELF e_entry mechanism?
6. What is DRAM_MEM_START, and what is the aarch64 interrupt controller called?
7. Give the full eight-point x86-vs-aarch64 contrast from Step 7, from memory.

When you can enumerate Firecracker's FDT nodes, map a decompiled DTB onto the FdtWriter calls that produce it, and recite the full x86-vs-aarch64 boot contrast, you've completed Lab 3 and the Boot Process intensive.

Next: take what you learned about device advertisement (the cmdline virtio_mmio tokens and the FDT nodes) into the virtio-devices masterclass, where you follow a real I/O from the guest driver through the device the boot path just announced. Or jump to the snapshotting masterclass, where this entire boot configuration is serialized and rebuilt by build_microvm_from_snapshot. This intensive feeds issue-roadmap Stage 5 (device config) and engineering/boot-time-optimization.