Guest Memory Management

Guest RAM is, at the bottom, just host memory: Firecracker mmaps an anonymous region in its own address space and registers it with KVM so the guest sees it as physical memory. Everything else — the kernel loader copying ELF segments, a virtio-block device writing a disk sector into a guest buffer, dirty-page tracking for snapshots, huge pages, ballooning — is built on top of that one idea. This chapter shows how guest memory is allocated and registered, how the rust-vmm vm-memory crate (GuestMemoryMmap, GuestAddress, GuestRegionMmap) models it, how device code and the loader read and write it safely (checked against guest bounds, never trusting a guest-supplied address), and how dirty-page tracking and huge pages plug in.

After this chapter you will be able to: explain the host-mmap → KVM_SET_USER_MEMORY_REGION pipeline; use the GuestMemory traits to read/write a guest address with bounds checking; explain why a device must never dereference a guest address as a host pointer without translation; and say what dirty-page tracking buys snapshots.

Note: The guest is untrusted, and that includes every address it puts in a virtqueue descriptor. The entire GuestMemory API exists so that a guest-supplied address is checked against registered guest regions before any host access. A device that bypasses this — that treats a guest address as a raw host pointer — is a memory-safety hole, the exact bug class Firecracker's Rust + checked-access design is meant to prevent.

From host mmap to guest physical memory

# Where guest memory is created and registered with KVM.
rg -n "GuestMemoryMmap|mmap|set_user_memory_region|create_guest_memory|memfd|MmapRegion" src/vmm/src/vstate/memory.rs src/vmm/src/builder.rs

The pipeline:

1. Decide the layout: how much RAM, where the MMIO gap is, where high memory starts (the arch layout constants — see the-boot-sequence.md).
2. mmap host-anonymous memory for each guest region (MmapRegion / GuestRegionMmap), optionally backed by a memfd (needed for some snapshot/UFFD modes) and optionally with MAP_HUGETLB for huge pages.
3. For each region, call KVM_SET_USER_MEMORY_REGION with slot, guest_phys_addr, memory_size, and userspace_addr (the host mmap address). This is the moment the host buffer becomes guest physical RAM. See kvm-fundamentals.md.

   Firecracker host address space            guest physical address space
   ┌───────────────────────────┐   KVM_SET_USER_MEMORY_REGION   ┌───────────────────────┐
   │ mmap region @ userspace_addr ──────────────────────────────► │ @ guest_phys_addr     │
   │ (anonymous / memfd / huge) │   slot, memory_size           │ (guest sees as RAM)   │
   └───────────────────────────┘                                 └───────────────────────┘

After registration, the guest's loads and stores to that physical range are translated by the CPU's second-level paging (EPT/NPT/stage-2) straight to the host pages — no VM exit. Firecracker, meanwhile, can still read and write the same bytes through the host mapping, which is exactly what the loader and the devices do.

The vm-memory model: GuestMemoryMmap, GuestAddress, the traits

Firecracker does not hand-roll guest memory access. It uses rust-vmm's vm-memory:

rg -n "vm-memory" Cargo.lock src/vmm/Cargo.toml
rg -n "GuestMemoryMmap|GuestAddress|GuestRegionMmap|GuestMemory\b|GuestMemoryRegion" src/vmm/src/
cargo doc -p vm-memory --open

The key property: a GuestAddress is not a host pointer. To touch the bytes at a GuestAddress you go through GuestMemoryMmap, which finds the region containing that address, checks the access fits inside it, and only then performs the host read/write. An out-of-bounds or unmapped address returns an Err, not undefined behaviour.

#![allow(unused)]
fn main() {
// Shape only. The real calls are throughout devices/ and the loader.
let mem: &GuestMemoryMmap = ...;
let addr = GuestAddress(desc.addr);              // a guest-supplied descriptor address
let header: VirtioHeader = mem.read_obj(addr)?;  // bounds-checked; Err if out of range
mem.write_slice(&disk_sector, GuestAddress(desc.addr))?; // bounds-checked write
}

How devices and the loader touch guest memory safely

Two consumers dominate: the kernel loader (at boot) and the virtio devices (at runtime). Both follow the same rule — never dereference a guest address; always go through GuestMemory.

# The loader copies kernel/initrd into guest memory.
rg -n "GuestMemory|read_to_mem|load_kernel|load_cmdline|Elf::load|write_obj" src/vmm/src/arch/ src/vmm/src/

# Devices read descriptors and move payloads through guest memory.
rg -n "read_obj|write_obj|read_slice|write_slice|get_slice|GuestMemory" src/vmm/src/devices/virtio/

The descriptor-chain case is where the trust boundary bites hardest. A virtqueue descriptor contains a guest physical addr and len supplied by the guest driver. A malicious or buggy guest can put any value there. The device must validate that addr..addr+len lies within registered guest memory before reading or writing — which it gets for free by going through GuestMemory::get_slice/read_slice rather than computing a host pointer. The virtqueues chapter walks this in detail.

Warning: "I already trust the descriptor because the queue logic produced it" is wrong. The queue logic comes from the guest. Every guest-supplied address is hostile until checked against guest memory bounds.

Dirty-page tracking (for snapshots)

A diff snapshot records only the guest pages that changed since the base. To know which pages changed, Firecracker uses KVM's dirty-page logging, enabled per memory region, and reads the dirty bitmap when snapshotting.

rg -n "track_dirty_pages|dirty|KVM_MEM_LOG_DIRTY_PAGES|get_dirty_log|dirty_bitmap" src/vmm/src/ src/vmm/src/vstate/

The field was historically enable_diff_snapshots and is now track_dirty_pages (verify on your branch). Dirty tracking has a runtime cost (KVM must trap or log writes), which is why it is opt-in and only needed for diff snapshots. Full snapshots write all of guest RAM and need no tracking. See snapshotting.md.

Huge pages

Backing guest RAM with 2 MiB huge pages reduces TLB pressure and page-table overhead for large guests. Firecracker exposes this through huge_pages in /machine-config.

rg -n "huge_pages|HugePages|MAP_HUGETLB|hugetlb|HugePageConfig" src/vmm/src/vmm_config/ src/vmm/src/vstate/memory.rs

When enabled, the guest memory mmap uses MAP_HUGETLB (the host must have huge pages configured/reserved). The trade-offs — fewer, larger pages means coarser dirty-page granularity and an all-or-nothing allocation — are real and interact with snapshotting and ballooning; the hugepages and memory performance engineering chapter weighs them. Ballooning (virtio-balloon.md) reclaims guest pages with madvise(MADV_DONTNEED), which also interacts with the page size.

Reading exercise

# 1. Allocation + KVM registration.
rg -n "GuestMemoryMmap|create_guest_memory|set_user_memory_region|MmapRegion|memfd" src/vmm/src/vstate/memory.rs src/vmm/src/builder.rs

# 2. The vm-memory types and traits.
rg -n "GuestAddress|GuestRegionMmap|GuestMemory\b|read_obj|write_obj|get_slice" src/vmm/src/
cargo doc -p vm-memory --open

# 3. Safe access in a device and in the loader.
rg -n "read_obj|write_slice|read_slice|get_slice" src/vmm/src/devices/virtio/ src/vmm/src/arch/

# 4. Dirty-page tracking.
rg -n "track_dirty_pages|get_dirty_log|KVM_MEM_LOG_DIRTY_PAGES|dirty_bitmap" src/vmm/src/

# 5. Huge pages.
rg -n "huge_pages|MAP_HUGETLB|HugePageConfig" src/vmm/src/vmm_config/ src/vmm/src/vstate/memory.rs

# 6. See it live: boot with --data '{"mem_size_mib":1024,...}' then inspect the process maps.
grep -i huge /proc/"$(pgrep -n firecracker)"/smaps 2>/dev/null | head

Answer:

1. Describe the host-mmap → KVM_SET_USER_MEMORY_REGION pipeline and the four fields of the memory region.
2. Why is GuestAddress a newtype and not a *mut u8? What bug class does that prevent?
3. How does a device safely read a virtqueue descriptor's (addr, len) without trusting the guest?
4. What does track_dirty_pages enable at the KVM level, and which snapshot type needs it?
5. What does backing guest RAM with huge pages change, and what is one downside?
6. Where do the loader and the devices get their checked access to guest memory from?

Common bugs and symptoms

Validation: prove you understand this

1. Draw the host-address-space → guest-physical-address-space mapping and name the ioctl that creates it.
2. Explain the GuestMemory trait's role as a trust boundary, with the descriptor-address example.
3. Why must a device never compute a host pointer from a guest descriptor address directly?
4. Explain dirty-page tracking end to end: enable, track, read, use — and which snapshot needs it.
5. State the trade-offs of huge-page-backed guest memory.
6. Name the two main consumers of GuestMemory (loader, devices) and what each does with it.

Next: The Boot Sequence — how this freshly allocated memory gets a kernel loaded into it and the vCPUs set running.