Lab 6.2: Guest Memory Layout

Background

A guest's physical address space is a contract. The kernel, the boot structures, the device windows, and the holes between them all sit at addresses that Firecracker and the guest kernel have silently agreed on. Get one constant wrong — place the cmdline where the kernel doesn't look, leave RAM where a device window should be, or split a region across the MMIO gap incorrectly — and the guest fails in ways that no Rust type can catch.

In this lab you build a precise, branch-accurate map of the x86_64 guest physical address space: where the kernel image lives, where the zero page and command line sit, where the GDT and page tables and boot stack go in the crowded sub-1 MiB region, where the MMIO gap carves a hole below 4 GiB, where high RAM spills above 4 GiB, and where the virtio-MMIO device windows are placed inside the gap. You will pin every address to a constant in layout.rs on your branch (never from memory), then observe the layout from three angles: the host's view of the allocation, the API's view of the configuration, and the guest kernel's view in its boot log. Finally you contrast it with the very different aarch64 layout.

Note: This is a trace-and-observe lab, not a build-it lab. Its deliverable is an annotated diagram and a constants table, both grounded in rg output and live observation. Read it alongside the guest memory management deep dive and the memory-layout cheat-sheet.

Why This Lab Matters for Contributors

  • Memory-layout bugs are subtle and high-impact: a region-split error near the MMIO gap can corrupt guest RAM or silently lose memory. Knowing the layout cold is the prerequisite for touching any of it.
  • Large-mem_size_mib configurations, huge-pages, and (newer) memory hotplug all interact with this layout. Contributors working on those features must reason about the gap and the region split.
  • Maintainers reason about placement by constant (HIMEM_START, MMIO_MEM_START). You must be able to state where each thing lives and why it can't go elsewhere.
  • The layout is under-documented in-tree relative to its importance — a precise, verified description is a genuinely welcome documentation PR (a Level 6 graduate PR profile).

Prerequisites

  • Completed Lab 6.1 (you can narrate the boot path).
  • A buildable checkout and the ability to boot a microVM (Lab 1.3).
  • The Level 6 overview layout diagram in front of you for comparison.
# The file you will live in for this lab. Confirm it exists and skim the constants.
rg -n "START|GAP|HIMEM|CMDLINE|ZERO_PAGE|MMIO|STACK|GDT|MEM_32BIT|FIRST_ADDR" \
  src/vmm/src/arch/x86_64/layout.rs

Step-by-Step Tasks

Step 1: Extract every layout constant from your branch

Do not copy the diagram in the overview — generate your own from the source. List every constant in layout.rs, with its value, and write down what each one pins.

# Dump the constants verbatim. These ARE the layout — nothing else is authoritative.
rg -n "pub const" src/vmm/src/arch/x86_64/layout.rs

For each constant, fill a row:

CONST                | VALUE       | PINS WHAT                          | WHY IT'S THERE
---------------------+-------------+-----------------------------------+----------------------------
ZERO_PAGE_START      | 0x0000_7000 | boot_params (the zero page)       | low, real-mode addressable; rsi
CMDLINE_START        | 0x0002_0000 | kernel command line text          | low memory, below the kernel
HIMEM_START          | 0x0010_0000 | kernel image load address (1 MiB) | x86 high memory starts at 1 MiB
... (continue for every const on YOUR branch) ...

Warning: The constant names are version-sensitive — MMIO_MEM_START, the 32-bit gap constant, and the boot-stack/GDT names have changed across refactors. Grep for them; do not assume the names in this lab are exactly what your branch uses. If a name in the table below doesn't resolve, find its sibling: rg -n "MMIO|GAP|32BIT|STACK|GDT" src/vmm/src/arch/x86_64/.

Step 2: Map the crowded sub-1 MiB region

Below HIMEM_START (1 MiB) is real-mode-addressable memory where the boot scaffolding lives. Find each resident and its address.

rg -n "ZERO_PAGE_START|CMDLINE_START|BOOT_STACK|STACK_POINTER|GDT|PAGE_TABLE|PML4|PDPTE|BOOT_GDT" \
  src/vmm/src/arch/x86_64/

Record what shares the low region:

ResidentConstantRole
Zero page (boot_params)ZERO_PAGE_STARTBoot params the kernel reads via rsi
Boot page tables(PML4/PDPT/PD const)Identity-map the first GiB for long mode
Boot GDT(GDT const)64-bit flat segment descriptors
Boot stackBOOT_STACK_POINTER (verify)Stack the kernel entry uses before it sets up its own
Kernel command lineCMDLINE_STARTThe cmdline text the zero page points at

The crowding matters: these structures must not overlap each other or the kernel image. A bug here is an overlap or an off-by-one, and it manifests as a corrupted boot param or a triple fault.

Step 3: Locate the kernel image and the rest of low RAM

The kernel ELF loads at HIMEM_START (1 MiB) and grows upward; ordinary low RAM continues above it.

rg -n "HIMEM_START|kernel_load|load_kernel|first_usable|FIRST_ADDR_PAST" \
  src/vmm/src/arch/x86_64/ src/vmm/src/builder.rs

Record: the kernel image occupies a contiguous span starting at 1 MiB; the loader (Lab 6.1, Step 3) copied the PT_LOAD segments here. Usable RAM extends upward from there until it hits the MMIO gap.

Step 4: Find the MMIO gap and the region split

This is the structural heart of x86_64 layout. Guest RAM cannot run continuously to 4 GiB — a window just below 4 GiB is reserved for memory-mapped I/O. Find the gap constant and the function that splits RAM around it.

rg -n "MMIO_MEM_START|MEM_32BIT_GAP|MEM_32BIT_DEVICES|FIRST_ADDR_PAST_32BITS|arch_memory_regions" \
  src/vmm/src/arch/x86_64/

Read arch_memory_regions (or its equivalent — verify the name) carefully. It is the function that, given a total RAM size, decides the list of regions:

mem_size FITS below the gap:          mem_size EXCEEDS the gap:
┌────────────────────────────┐        ┌────────────────────────────┐
│ region 0: [0, mem_size)    │        │ region 0: [0, gap_start)    │  RAM up to the gap
└────────────────────────────┘        ├────────────────────────────┤
                                       │ MMIO gap (no RAM)           │  devices live here
(one region)                          ├────────────────────────────┤
                                       │ region 1: [4 GiB, 4 GiB +   │  high RAM = remainder
                                       │           (mem_size - gap)) │
                                       └────────────────────────────┘
                                       (two regions)

Record the rule precisely: below what threshold is there a single region, and above it, exactly how is the remainder placed above 4 GiB? Note the gap start address and width. This is the most common source of real layout bugs.

Step 5: Find where virtio-MMIO device windows are placed in the gap

The MMIO gap is not abstract — actual device register windows are allocated inside it. Find the allocator and the per-device window size.

rg -n "MMIO_MEM_START|MMIO_MEM_SIZE|MMIO_LEN|allocate|IdAllocator|AddressAllocator|register_mmio" \
  src/vmm/src/arch/x86_64/ src/vmm/src/device_manager/

Record: each virtio-MMIO device gets a fixed-size register window at a successive address inside the gap, plus an IRQ. The guest is told the address/size/IRQ of each via the cmdline string virtio_mmio.device=SIZE@ADDR:IRQ (you will see these strings in Lab 6.3). The placement and dispatch logic is the subject of the MMIO bus and device manager deep dive and the virtio-MMIO transport deep dive.

Step 6: Observe the layout from the host

Boot a microVM and look at how the host sees the guest RAM allocation. The guest's physical memory is just mmap'd host memory; you can see it in /proc.

# Boot a 1 GiB microVM (see Lab 6.1 Step 8 for the full curl sequence), then:
FCPID=$(pgrep -n firecracker)
# The large anonymous mapping(s) that back guest RAM:
grep -E "rw-p|rw-s" /proc/$FCPID/maps | awk '{print $1, $6}' | sort -u | tail -20
# Resident set / mapped sizes:
grep -E "VmRSS|VmSize" /proc/$FCPID/status
# The mapping list, sorted by size, to spot the guest RAM region:
cat /proc/$FCPID/smaps_rollup 2>/dev/null | head

Record: a ~1 GiB anonymous mapping (or two, if the config crossed the MMIO gap) corresponds to the guest RAM regions you traced in Step 4. With track_dirty_pages off and no faults, RSS may be far smaller than the mapping size — guest RAM is allocated lazily by the host. Note the difference between mapped and resident; this is the basis of oversubscription (see oversubscription and density).

Step 7: Observe the layout from the API and the boot log

Now the guest's own view. Read the configured machine via the API, then read what the kernel printed.

API=/tmp/fc.sock
# What memory the microVM was configured with:
curl -s --unix-socket $API http://localhost/machine-config | jq .
# (newer builds) the full resolved config:
curl -s --unix-socket $API http://localhost/vm/config | jq . 2>/dev/null

Then in the serial console output, find the kernel's reconstruction of your layout:

Command line: console=ttyS0 ... virtio_mmio.device=4K@0x... :5 ...   ← Step 5 device windows
BIOS-provided physical RAM map:
BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable        ← low RAM (Step 2/3)
BIOS-e820: [mem 0x0000000000100000-0x...] usable                     ← RAM from 1 MiB (Step 3)
BIOS-e820: [mem 0x...-0x00000000ffffffff] reserved                   ← the MMIO gap (Step 4)
BIOS-e820: [mem 0x0000000100000000-0x...] usable                     ← high RAM > 4 GiB (Step 4)
Memory: ...K/...K available                                          ← total usable

Record: each BIOS-e820: ... usable line is a RAM region from Step 4; the reserved span is the MMIO gap; the virtio_mmio.device= entries on the Command line: are the device windows from Step 5. You now have the layout confirmed from three independent vantage points: the constants, the host allocation, and the guest's e820/cmdline.

Step 8: Contrast the aarch64 layout

aarch64 is laid out differently and has no zero page. Pull its constants and the FDT placement.

rg -n "pub const|DRAM_MEM_START|DRAM_MEM_MAX|FDT|MMIO|GIC" src/vmm/src/arch/aarch64/layout.rs
rg -n "DRAM_MEM_START|FdtWriter|create_fdt|fdt_addr|x0" src/vmm/src/arch/aarch64/

Record the key differences in a table:

Aspectx86_64aarch64
RAM baselow (0x0), kernel at HIMEM_START (1 MiB)DRAM_MEM_START = 0x8000_0000
Boot paramszero page (boot_params) at ZERO_PAGE_START, e820 mapFDT/DTB, no zero page, no e820
Params address handoffrsi = ZERO_PAGE_STARTx0 = FDT address
Kernel formatuncompressed vmlinux ELFarm64 PE Image
Interrupt controlleri8042/IOAPIC/MMIOGIC (described in the FDT)
FDT placementn/acommonly near the top of DRAM (verify on your branch)

The FDT is the aarch64 analog of "e820 + MPTable/ACPI + cmdline pointer" all rolled into one device-tree blob. See the boot sequence deep dive for the FDT build path.

Implementation Requirements / Deliverables

  • A constants table for x86_64, generated from rg against layout.rs on your branch — every constant, its value, and what it pins.
  • An annotated ASCII diagram of the x86_64 guest physical address space, with each region/hole labeled by its constant, drawn from your branch's values (not copied from the overview).
  • The region-split rule stated precisely: the single-region vs two-region condition around the MMIO gap, with the gap start and width.
  • Three-vantage confirmation: the host /proc/$PID/maps mapping(s), the API machine-config / vm/config, and the guest BIOS-e820: + virtio_mmio.device= lines — all reconciled to the same layout.
  • The x86_64-vs-aarch64 contrast table with the FDT-vs-zero-page distinction.

Troubleshooting

A constant name in this lab doesn't exist on my branch

Expected — they get renamed. Grep by role: rg -n "MMIO|GAP|32BIT|HIMEM|STACK|GDT|CMDLINE|ZERO" src/vmm/src/arch/x86_64/layout.rs. Match the value and purpose, not the literal name.

/proc/$PID/maps shows many mappings, none obviously "guest RAM"

The guest RAM region is the largest anonymous rw-p (or rw-s with file-backed snapshots) mapping; its size matches mem_size_mib. Sort by computed size and look for the ~mem_size entry. Other mappings are the binary, stacks, and the KVM run page.

RSS is much smaller than mem_size_mib

Correct and expected. Guest RAM is allocated lazily by the host kernel; only touched pages are resident. That is the whole point of oversubscription. Touch more guest memory (run a workload) and watch RSS grow.

No high-RAM e820 line appears

Your mem_size_mib fit entirely below the MMIO gap, so there is only one region and no high RAM. Boot with a memory size larger than the gap (multiple GiB) to force the two-region split, then re-check.

jq errors on /vm/config

That endpoint is newer — it may not exist on your branch (the guide flags it). Fall back to /machine-config, which is always present.

Expected Output

A finished map that looks like this (with your branch's exact values), reconciled across all three vantage points:

x86_64 GUEST PHYSICAL LAYOUT (fill values from YOUR layout.rs)
  0x0000_7000  ZERO_PAGE_START   boot_params           ── rsi target ── confirmed by Command line:
  0x0000_8000  page tables + GDT identity map / segs
  0x0000_8ff0  BOOT_STACK        boot stack
  0x0002_0000  CMDLINE_START     cmdline text          ── confirmed by Command line: line
  0x0010_0000  HIMEM_START       kernel image (vmlinux)── BIOS-e820 usable from 1 MiB
     ...       low/usable RAM                          ── BIOS-e820 usable spans
  0x????_????  MMIO_MEM_START    virtio-MMIO windows   ── virtio_mmio.device= on cmdline
     ...       MMIO gap (reserved)                     ── BIOS-e820 reserved span
  0x1_0000_0000 4 GiB            high RAM (if any)     ── BIOS-e820 usable >= 4 GiB

Stretch Goals

  1. Compute the region split yourself. Pick a mem_size_mib that straddles the gap (e.g. several GiB). By hand, from the constants, predict the two regions' base+length. Boot with that size and verify against the BIOS-e820: lines. Off by anything? Re-read arch_memory_regions.
  2. Find the exact gap width. From MMIO_MEM_START (or the 32-bit gap constant) and the 4 GiB boundary, compute the gap size. Cross-check against the reserved e820 span the kernel reports.
  3. Map a virtio device window to its cmdline entry. Boot with a block + a net device, find the two virtio_mmio.device=SIZE@ADDR:IRQ entries, and confirm each ADDR falls inside the MMIO gap and the windows don't overlap.
  4. Huge pages. If your branch supports huge_pages in machine-config, boot with it on and observe how the host mapping changes (AnonHugePages in /proc/$PID/smaps). Note how the layout itself is unchanged — only the backing page size differs. See hugepages and memory performance.
  5. aarch64 in full. If you have aarch64 hardware, boot there and read the FDT the kernel parsed (/sys/firmware/fdt in the guest, or dtc it). Confirm DRAM_MEM_START, the RAM size, and the GIC node match what arch/aarch64/ wrote.

Validation / Self-check

Answer without notes; these gate completion:

  1. List, in address order, what lives below 1 MiB in the x86_64 guest, and the constant that pins each.
  2. Why does the kernel image load at HIMEM_START (1 MiB) and not at 0?
  3. What is the MMIO gap, why does it exist, and what is the exact rule for how RAM is split around it?
  4. Given mem_size_mib larger than the gap, where does the overflow RAM go, and how many memory regions result?
  5. How is a virtio-MMIO device's location communicated to the guest, and where (which address range) must that location fall?
  6. You observed the layout from three vantage points. Name them and say what each one confirms.
  7. How does the aarch64 layout differ structurally from x86_64? What replaces the zero page and the e820 map, and in which register is its address passed?

When your diagram is reconciled across the constants, the host allocation, and the guest's e820/cmdline, proceed to Lab 6.3: Boot Configuration and the Kernel Command Line.