Memory Layout Cheat-Sheet

When you debug a kernel that won't load, place a virtio device on the MMIO bus, or trace the boot sequence, the question is always the same: what lives at this guest physical address? This page is the answer key — the x86_64 and aarch64 guest-memory maps, every layout constant with the file it comes from, and an ASCII memory map you can hold a KVM_EXIT_MMIO address up against to classify it in five seconds.

Reach for this while tracing the boot sequence, working through Level 6, or reasoning about guest memory.

Warning — every constant here is version-sensitive. Layout constants live in per-arch source files that change between branches and across the recent crate-merge refactor. Treat every value below as "the role, and roughly where it sits," not gospel, and verify the literal in your checkout:

# x86_64 constants (the source of truth — read it, don't trust this page):
rg -n "START|GAP|ADDR|_BASE|HIMEM|ZERO_PAGE|CMDLINE" src/vmm/src/arch/x86_64/layout.rs
# aarch64 constants:
rg -n "DRAM_MEM_START|FDT|GIC|MEM_START|MMIO" src/vmm/src/arch/aarch64/layout.rs
# Where memory regions are actually created and registered with KVM:
rg -n "create_guest_memory|arch_memory_regions|MMIO_MEM_START" src/vmm/src/

Each row below is tagged (verify in arch/<arch>/layout.rs on your branch) as a reminder.

x86_64 guest-memory layout

The x86_64 map is two RAM regions split by an MMIO gap just below 4 GiB. Low addresses hold the boot scaffolding the kernel expects (the zero page, the command line, the GDT/page tables, the boot stack); the kernel itself loads at 1 MiB; device MMIO windows and the APIC live in the gap; any RAM beyond what fits below the gap is "high RAM" above 4 GiB.

Note — the e820 map is how the guest learns this. The VMM doesn't just place RAM, it describes it: it writes an e820 memory map into the zero page so the kernel knows which physical ranges are usable RAM versus the reserved MMIO gap. A guest that thinks the gap is RAM (or can't find its high RAM) is an e820 bug, not a placement bug. See ../masterclass/the-boot-process/lab-02-zero-page-and-e820.md.

x86_64 ASCII memory map

 guest physical address
 0xFFFF_FFFF ───────────────────────────────────────────  4 GiB
   ↑  (high RAM continues above 4 GiB if configured mem is large)
 0x1_0000_0000 ═══════════════════════════════════════════  HIGH RAM  ▲ above 4 GiB
                   guest RAM region 2 (only if mem > sub-gap capacity)  │
 0x1_0000_0000 ─────────────────────────────────────────── 4 GiB ──────┘
 0xFEE0_0000  ┌─────────────────────────────┐  Local APIC (LAPIC) MMIO
 0xFEC0_0000  ├─────────────────────────────┤  IOAPIC MMIO
              │   . . . MMIO GAP . . .       │  reserved, below 4 GiB
 ~0xC000_0000 ├─────────────────────────────┤  ← MMIO gap base
              │  virtio-MMIO device windows  │  net / block / vsock / … (fixed block + IRQ each)
              └─────────────────────────────┘
              ╔═════════════════════════════╗  LOW RAM  (guest RAM region 1)
              ║   initrd / allocations       ║
 0x0010_0000  ║─ HIMEM_START (1 MiB) ────────║  ← kernel image loaded here
              ║   sub-1MiB scaffolding:       ║
 0x0002_0000  ║─ CMDLINE_START ──────────────║  kernel command line
 0x0000_8ff0  ║   boot stack                 ║
              ║   GDT / IDT / page tables    ║  (low, set up for long mode)
 0x0000_7000  ║─ ZERO_PAGE_START ────────────║  boot_params (rsi points here)
 0x0000_0000  ╚═════════════════════════════╝

Warning: The MMIO-gap base and the exact low-RAM ceiling are the values most likely to differ on your branch. When you place a device or decode an MMIO exit, rg the gap constant — rg -n "MMIO|GAP|MEM_START" src/vmm/src/arch/x86_64/layout.rs — rather than trusting the ~0xC000_0000 above, which is illustrative.

aarch64 guest-memory layout

aarch64 is a different model. There is no zero page and no e820 — instead the VMM builds a Flattened Device Tree (FDT/DTB) describing memory, CPUs, and devices, and passes its address to the kernel in register x0. The kernel is an arm64 PE Image (not a vmlinux ELF). Guest DRAM starts high (DRAM_MEM_START), and the interrupt controller is the GIC, not an x86 APIC.

aarch64 ASCII memory map

 guest physical address
   ↑   (guest DRAM continues upward for the configured memory size)
              ╔═════════════════════════════╗  GUEST DRAM
              ║   FDT / DTB (addr → x0)       ║  device tree blob, placed in DRAM
              ║   kernel Image (arm64 PE)     ║
 0x8000_0000  ║─ DRAM_MEM_START (2 GiB) ─────║  ← base of guest RAM
              ╚═════════════════════════════╝
              ┌─────────────────────────────┐  DEVICE / MMIO region (below DRAM)
              │  virtio-MMIO device windows  │  net / block / vsock / … (FDT node each)
              │  GIC distributor / redist.   │  the interrupt controller
 0x0000_0000  └─────────────────────────────┘

Note — FDT is the aarch64 analogue of the x86 zero page. Where x86 writes boot_params + e820 and points rsi at it, aarch64 builds an FDT (with vm-fdt's FdtWriter) describing the same facts — RAM ranges, CPU count, each virtio-MMIO device's address/IRQ, the GIC — and points x0 at it. A device the guest can't see on aarch64 is usually a missing or wrong FDT node, the way a missing x86 device is usually a wrong cmdline virtio_mmio.device= entry. See ../masterclass/the-boot-process/lab-03-aarch64-fdt.md.

How a virtio device's address reaches the guest

The layout only matters because the guest has to find each device. The two arches tell it differently — and this is the single most common "my device isn't detected" debugging axis.

# See the per-device MMIO placement and the cmdline/FDT wiring on your branch:
rg -n "virtio_mmio.device|MMIO_MEM_START|register_mmio|allocate" src/vmm/src/device_manager/
rg -n "FdtWriter|/virtio_mmio|interrupts" src/vmm/src/arch/aarch64/

Where each region comes from (the boot pipeline)

flowchart TD
    A["machine-config: mem_size_mib"] --> B["arch_memory_regions() splits RAM<br/>around the MMIO gap (x86) / from DRAM_MEM_START (arm64)"]
    B --> C["create_guest_memory: mmap + KVM_SET_USER_MEMORY_REGION"]
    C --> D["linux-loader: copy kernel image to HIMEM_START (x86) / DRAM_MEM_START (arm64)"]
    D --> E["x86: write boot_params @ ZERO_PAGE_START + e820 + cmdline @ CMDLINE_START<br/>arm64: build FDT, place in DRAM"]
    E --> F["set initial regs: x86 rsi=ZERO_PAGE_START, rip=e_entry<br/>arm64 x0=FDT addr"]
    F --> G["MMIODeviceManager places virtio devices in the gap / device region"]

Every box is rg-able: rg -n "arch_memory_regions|create_guest_memory|ZERO_PAGE_START|FdtWriter" src/vmm/src/. The KVM_SET_USER_MEMORY_REGION step is where these host mmaps become guest RAM — see the KVM ioctl cheat-sheet.

Reading exercise

From a Firecracker checkout:

# 1. Read the literal x86 constants and compare to the table above:
rg -n "ZERO_PAGE_START|CMDLINE_START|HIMEM_START|MMIO|GAP" src/vmm/src/arch/x86_64/layout.rs

# 2. Read the aarch64 base + GIC + FDT placement:
rg -n "DRAM_MEM_START|GIC|FDT|MEM_START" src/vmm/src/arch/aarch64/layout.rs

# 3. Find where RAM is split around the MMIO gap (the two-region logic):
rg -n "arch_memory_regions|MMIO_MEM_START|first_addr_past_32bits|GuestAddress" src/vmm/src/arch/

# 4. Find where the kernel is copied in and the entry point captured:
rg -n "load_kernel|e_entry|Elf|Image|HIMEM_START|DRAM_MEM_START" src/vmm/src/

Questions to answer from what you find:

1. On x86_64, which constant is the kernel load address, and which is the zero-page address the boot register rsi points at?
2. Why does x86_64 split guest RAM into two regions, while aarch64 has one contiguous DRAM region? What sits in the gap on x86?
3. aarch64 has no zero page. What replaces it, in which register is its address passed, and which crate builds it?
4. A guest doesn't detect a virtio-block device. Name the x86 mechanism and the aarch64 mechanism that tell the guest where the device's MMIO window is — and which one you'd check first on each arch.
5. Which value in the x86 table is most likely to differ on your branch, and what rg confirms it?

Related references

• KVM ioctl Cheat-Sheet — KVM_SET_USER_MEMORY_REGION registers these regions; the boot registers (rsi/x0, rip) are set with KVM_SET_REGS.
• API Endpoint Map — /machine-config (mem size), /boot-source (kernel, cmdline) feed this layout.
• Glossary — ZERO_PAGE_START, HIMEM_START, DRAM_MEM_START, MMIO gap, FDT, e820.
• The boot sequence and guest memory management — the prose behind these maps.
• Level 6 and ../masterclass/the-boot-process/index.md — the labs that make you instrument this.

Next: Firecracker vs. Other VMMs — why a VMM this minimal exists at all.