Lab 2: The Zero Page and the e820 Map

Background

In Lab 1 you placed the kernel's bytes and tied e_entry to the vCPU's rip. But a kernel cannot boot on instructions alone — it needs to be told about its hardware: how much RAM there is and where, where its command line is, whether there's an initrd, what the early console looks like. On x86, the Linux boot protocol mandates a single structure that carries all of this: boot_params, universally called the zero page. Firecracker fills it, writes it to ZERO_PAGE_START (0x7000 — verify), and hands its address to the kernel in rsi. The very first thing the kernel does at e_entry is read it.

This lab dissects the zero page field by field. The headline field is the e820 memory map — an array of (addr, size, type) entries that tells the kernel which physical ranges are usable RAM, which are reserved, and (crucially) where the MMIO gap is so the kernel does not treat it as RAM. You will also locate the command-line pointer (hdr.cmd_line_ptr → CMDLINE_START) and the initrd ramdisk pointer/size (hdr.ramdisk_image/ramdisk_size). Then you will boot a real microVM and read the result back from inside the guest — dmesg and /sys/firmware/memmap echo the e820 entries Firecracker wrote, so you can compare "what FC put in" against "what the kernel saw."

This is a trace-it lab: source side with rg, guest side with dmesg and sysfs, and the two reconciled.

Why This Lab Matters for Contributors

• The e820 map is the kernel's ground truth for memory. Get it wrong — overlap RAM with the MMIO gap, miscount high RAM above 4 GiB, forget a reserved range — and the guest silently corrupts its own view of memory, or virtio-mmio devices land on addresses the kernel thinks are RAM. Boot bugs of this class are subtle and exactly the kind reviewers scrutinize.
• The zero page is built through rust-vmm's linux-loader bootparam types and a LinuxBootConfigurator. This is another live crate boundary (linux-loader chapter); contributors who add a boot feature (a new e820 entry, an ACPI RSDP pointer) work right here.
• The memory layout the e820 encodes is also the layout a snapshot must reproduce. The snapshotting masterclass restores a microVM whose memory regions and gap must match what the running kernel believes — a mismatch is a restore-time corruption. The e820 you dissect here is the contract.

Prerequisites

• Lab 1 complete: you can find the loader, the layout constants, and the e_entry → rip hand-off.
• The boot-sequence deep dive and the guest-memory deep dive.
• A microVM you can boot (Lab 1.3) and get a shell on (the CI rootfs has dmesg and /sys).

cd ~/firecracker
rg -q "configure_system" src/vmm/src/arch/x86_64/ && echo "boot-params builder present"
rg -n "ZERO_PAGE_START|CMDLINE_START|HIMEM_START" src/vmm/src/arch/x86_64/layout.rs

Step-by-Step Tasks

Step 1: Find where the zero page is built

Firecracker assembles boot_params in its x86 system-configuration routine. Locate it by role:

cd ~/firecracker
# The function that builds boot_params and writes it to ZERO_PAGE_START:
rg -n "configure_system|LinuxBootConfigurator|boot_params|BootParams|bootparam|setup_header|ZERO_PAGE_START" \
  src/vmm/src/arch/x86_64/

You are looking for a function (historically configure_system) that:

1. constructs a boot_params (the linux-loader bootparam type),
2. fills its setup header (hdr) — magic, version, the cmdline pointer, the ramdisk pointer/size,
3. builds the e820 map by adding entries, and
4. writes the whole struct into guest memory at ZERO_PAGE_START via a LinuxBootConfigurator.

# The setup-header magic/type fields and the writer:
rg -n "boot_flag|header\b|hdr\.|HEADER_MAGIC|KERNEL_LOADER_OTHER|0xeb|type_of_loader|kernel_alignment" \
  src/vmm/src/arch/x86_64/

Step 2: Dissect the e820 entries Firecracker writes

The e820 map is the centerpiece. Find where entries are added:

rg -n "add_e820_entry|E820|e820|E820_RAM|E820RAM|MEMORY|RESERVED|e820_entries|e820_table" \
  src/vmm/src/arch/x86_64/

Each entry is (addr, size, type). The two types that matter:

Firecracker's x86 map is shaped by the layout you saw in Lab 1. Read the code and reconstruct the entries for a microVM with, say, mem_size_mib = 1024:

e820 map Firecracker builds (1 GiB example; shape per arch_memory_regions — verify)

  type  start         end           note
  RAM   0x0000_0000   0x0009_fc00   low RAM below the legacy 640 KiB / EBDA line
  RAM   0x0010_0000   <gap_start>   main low RAM from 1 MiB up to the MMIO gap
  ----  <gap_start>   0x1_0000_0000 MMIO GAP — NOT an e820 RAM entry; the hole
  RAM   0x1_0000_0000 <high_end>    high RAM above 4 GiB (only if mem > the gap)

The decisive observation: the MMIO gap is represented by the absence of a RAM entry. Firecracker does not add an e820 RAM range there, so the kernel never tries to put RAM in the gap, which leaves it free for virtio-mmio device windows. This is the e820 expression of the same gap you registered as an unbacked region in the KVM intensive.

Note: Whether RAM straddles the 4 GiB gap depends on mem_size_mib. Below the gap size, there is no high-RAM entry at all. Confirm how your branch computes the split — rg -n "arch_memory_regions|FIRST_ADDR_PAST_32BITS|MMIO_MEM_START|MEM_32BIT_GAP" src/vmm/src/arch/x86_64/ src/vmm/src/vstate/memory.rs.

Step 3: Find the command-line pointer

The kernel finds its command line through boot_params.hdr.cmd_line_ptr, which Firecracker sets to CMDLINE_START (0x20000 — verify). The cmdline string is written to that address separately; the zero page just holds the pointer.

# The cmdline pointer in the header, and where the string is written:
rg -n "cmd_line_ptr|cmdline_addr|CMDLINE_START|load_cmdline|Cmdline|InitrdConfig" \
  src/vmm/src/arch/x86_64/ src/vmm/src/

So the zero page carries two related but distinct things to track:

The default cmdline is roughly reboot=k panic=1 pci=off nomodule 8250.nr_uarts=0 plus console=ttyS0 (varies — verify). It is the cmdline that, on x86, also advertises the virtio-mmio devices to the guest (virtio_mmio.device=SIZE@ADDR:IRQ) — a thread you'll pull in the virtio-devices masterclass.

Step 4: Find the initrd (ramdisk) pointer and size

If an initrd is configured, its load address and size go into the header so the kernel can find the initramfs:

rg -n "ramdisk_image|ramdisk_size|initrd|load_initrd|InitrdConfig|hdr\.ramdisk" \
  src/vmm/src/arch/x86_64/ src/vmm/src/

If no initrd is configured, both are zero and the kernel boots straight to the root device on the virtio-block disk.

Step 5: Boot a microVM and read the e820 back from the guest

Now reconcile source with reality. Boot a microVM (Lab 1.3) with mem_size_mib = 1024, get a shell, and read the e820 the kernel received:

# Inside the guest:
dmesg | grep -iE "e820|BIOS-provided|usable|reserved" | head -30

[    0.000000] e820: BIOS-provided physical RAM map:
[    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
[    0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000003fffffff] usable
[    0.000000] e820: last_pfn = 0x40000 max_arch_pfn = ...

Then read it structurally from sysfs — /sys/firmware/memmap is the kernel's parsed copy of the firmware memory map:

# Inside the guest:
for d in /sys/firmware/memmap/*/; do
  printf "%-18s %-18s %s\n" \
    "$(cat $d/start)" "$(cat $d/end)" "$(cat $d/type)"
done | sort

0x0                0x9fbff            System RAM
0x100000           0x3fffffff         System RAM

Compare line by line against the entries you reconstructed in Step 2. The usable ranges are your E820_RAM entries; the absence of a System RAM range in the MMIO-gap region is the gap Firecracker deliberately left out. This is the proof: the e820 Firecracker built in arch/x86_64/ is exactly what the guest kernel believes.

Step 6: Read the command line and the boot loader back

Confirm the cmdline pointer landed too:

# Inside the guest:
cat /proc/cmdline
# -> console=ttyS0 reboot=k panic=1 pci=off nomodule 8250.nr_uarts=0 root=/dev/vda ...
dmesg | grep -i "command line" | head -2

/proc/cmdline is the string the kernel read from the address in hdr.cmd_line_ptr — i.e., the bytes Firecracker wrote at CMDLINE_START, reached via the pointer in the zero page. If you see your boot_args, the pointer chain held.

Step 7 (optional): Instrument the e820 build

Make the map observable from the FC side. Add a temporary log where entries are added (names per your branch from Step 2):

#![allow(unused)]
fn main() {
// Temporary — REMOVE before commit. After each add_e820_entry call, or in a loop
// over the entries you built:
log::info!("E820 add: addr={:#x} size={:#x} type={}", addr, size, mem_type);
}

tools/devtool build
# boot with the logger enabled, then:
grep "E820 add" /tmp/fc.log

The FC-side log lines must match the guest-side /sys/firmware/memmap ranges. When they do, you have closed the loop in both directions — what Firecracker wrote and what the kernel read are the same map. Revert the instrumentation.

Implementation Requirements / Deliverables

• The rg output locating: (a) the zero-page builder (configure_system or its successor), (b) the e820 entry-adding code, (c) the cmdline pointer, (d) the ramdisk pointer/size — each with the file noted.
• A reconstructed e820 table for a mem_size_mib = 1024 microVM, with the MMIO gap shown as an absence of a RAM entry, derived from the source.
• The guest-side dmesg//sys/firmware/memmap dump, reconciled line-by-line against your reconstructed table.
• /proc/cmdline from the guest matching the boot_args you configured, with a sentence explaining the pointer chain (hdr.cmd_line_ptr → CMDLINE_START → the string).
• A one-paragraph statement of why the MMIO gap must not appear as an e820 RAM entry.

Troubleshooting

rg for configure_system finds nothing

The function was renamed across refactors. Anchor on the behavior: rg -n "boot_params|LinuxBootConfigurator|add_e820_entry|E820|bootparam" src/vmm/src/arch/x86_64/. The LinuxBootConfigurator (from linux-loader) is a stable landmark — it is what writes the zero page.

Guest dmesg shows no e820 lines

Some kernels/log levels suppress them. Use /sys/firmware/memmap instead — it is the parsed firmware map and does not depend on the log level. If /sys/firmware/memmap is empty, the kernel may have used a different memory-detection path; check that you booted the FC-provided vmlinux, not a foreign kernel.

The guest's RAM total is wrong (too small / too large)

Your e820 RAM entries don't sum to mem_size_mib, or the high-RAM-above-4-GiB entry is mishandled. Re-derive the split from arch_memory_regions (rg -n "arch_memory_regions" src/vmm/src/arch/x86_64/) and confirm the gap boundaries. A common slip is forgetting the high-RAM entry when mem > gap.

/proc/cmdline is empty or truncated

The cmdline string didn't reach CMDLINE_START, or the pointer in hdr.cmd_line_ptr is wrong, or the cmdline exceeded the max length. Check the cmdline-loading path (rg -n "load_cmdline|cmd_line_ptr|CMDLINE_MAX" src/vmm/src/).

Expected Output

# guest:
$ cat /sys/firmware/memmap/0/{start,end,type} | paste - - -
0x0    0x9fbff    System RAM
$ cat /proc/cmdline
console=ttyS0 reboot=k panic=1 pci=off nomodule 8250.nr_uarts=0 root=/dev/vda rw

# FC side after instrumentation:
$ grep "E820 add" /tmp/fc.log
... E820 add: addr=0x0 size=0x9fc00 type=1
... E820 add: addr=0x100000 size=0x3ff00000 type=1

The two memory maps — what Firecracker logged and what the guest read — describe the same physical layout, with the same gap. That reconciliation is the lab.

Stretch Goals

1. ACPI vs MPTable. CPU topology and tables historically came via MPTable; Firecracker has added ACPI (RSDP/MADT) and is deprecating MPTable. Find which your branch builds and where the table pointers reach the guest (rg -n "MPTable\|mptable\|RSDP\|MADT\|acpi\|Madt" src/vmm/src/arch/x86_64/ src/acpi-tables/). Read acpi-and-mptable and confirm from the guest with dmesg | grep -iE "ACPI|MP-table".
2. Two-region high RAM. Boot with mem_size_mib large enough to straddle the 4 GiB gap (e.g. 5120). Re-read /sys/firmware/memmap and find the second System RAM range above 0x1_0000_0000. Map it back to the high-RAM e820 entry in the source.
3. Corrupt the map (in a scratch fork). On a throwaway branch, drop the high-RAM e820 entry, boot, and watch the guest under-report RAM. Reverting teaches you which entry maps to which guest-visible number.
4. Read the raw zero page. With a GDB attach to a paused guest (Debugging masterclass) or a memory snapshot, dump 4 KiB at ZERO_PAGE_START and decode the boot_params header fields (cmd_line_ptr, ramdisk_image, the e820 entry count) against the bootparam layout. You will be reading the exact bytes Firecracker wrote.

Validation / Self-check

Answer without notes. These gate completion.

1. What is the zero page, where is it written, and how does the kernel find it?
2. What is an e820 entry, what are the two types Firecracker uses, and how is the MMIO gap represented?
3. Why must the MMIO gap not appear as an E820_RAM entry? What breaks if it does?
4. Trace the command line: where is the string, where is the pointer, and which guest file lets you read the result?
5. What two header fields carry the initrd, and what are they when no initrd is configured?
6. You boot with mem_size_mib = 5120. How many System RAM ranges do you expect in /sys/firmware/memmap, and why?
7. Which rust-vmm crate provides the boot_params/bootparam types and the configurator that writes the zero page?

When you can reconstruct the e820 map from the source, reconcile it against the guest's /sys/firmware/memmap, and explain the cmdline and initrd pointers, you've completed Lab 2. Continue to Lab 3 — The aarch64 Boot Path, where the answer to "what is the kernel told about its hardware?" changes completely.