Huge Pages & Memory Performance

Guest memory is the single biggest thing a microVM owns, and the way the host maps it has outsized effects on performance that are invisible until you measure them. The default is 4 KiB pages — the universal Linux page size. But a microVM with gigabytes of RAM addressed in 4 KiB chunks puts enormous pressure on the CPU's TLB (Translation Lookaside Buffer), the small cache that turns virtual addresses into physical ones. Every TLB miss is a page-table walk; with nested paging (EPT/NPT) under virtualization, a miss can mean walking two page-table hierarchies. Huge pages — backing guest RAM with 2 MiB pages instead of 4 KiB — are the lever that relieves that pressure.

This chapter is about that lever: how Firecracker maps guest memory, what huge pages buy you (boot time, runtime, snapshot restore), what they cost, and the subtle ways they interact with dirty-page tracking, the balloon, and UFFD. It is a performance chapter, which means the rule is absolute: measure, do not assume.

Note: Read the guest memory deep dive first for how guest RAM is mapped and registered with KVM. This chapter assumes you know what GuestMemoryMmap and KVM_SET_USER_MEMORY_REGION are and asks the performance question on top.

How guest memory is mapped

Firecracker mmaps guest RAM as a host memory region and registers it with KVM via KVM_SET_USER_MEMORY_REGION. The mapping flags are where the performance and correctness properties come from. Find them — do not trust a line number:

cd ~/src/firecracker
rg -n "MAP_NORESERVE|MAP_PRIVATE|MAP_ANONYMOUS|MAP_HUGETLB|MAP_HUGE|with_mmap_flags|PROT_READ|MmapRegionBuilder" \
  src/vmm/src/vstate/memory.rs

Two flags matter for this chapter:

• MAP_NORESERVE — the kernel does not reserve backing store at mmap time; pages are claimed on demand. This is what makes soft allocation and oversubscription possible, and — as you will see — it is also what makes huge pages capable of failing with SIGBUS if the pool is exhausted.
• The hugetlb flags — when huge_pages is set, the region is backed by hugetlbfs 2 MiB pages instead of ordinary 4 KiB pages.

You enable huge pages per microVM via /machine-config:

curl -X PUT --unix-socket $API --data \
 '{"vcpu_count":2,"mem_size_mib":1024,"huge_pages":"2M"}' \
 http://localhost/machine-config
# huge_pages: "None" (default, 4K) or "2M" (2 MiB hugetlbfs). Verify on your branch.
rg -n "huge_pages|HugePageConfig|2M|hugetlbfs|MAP_HUGETLB" src/vmm/src/vmm_config/machine_config.rs | head
sed -n '1,30p' docs/hugepages.md

Why huge pages help: TLB and EPT

The win is mechanical and worth understanding precisely, because it tells you which workloads benefit.

4 KiB pages:                         2 MiB pages:
512 pages cover 2 MiB of RAM         1 page covers 2 MiB of RAM
→ 512 TLB entries needed             → 1 TLB entry covers the same span
→ frequent TLB misses on big RSS     → far fewer TLB misses
→ each miss: page-table walk         → 512× fewer entries for the same memory
   (and under virtualization, a       → fewer EPT/NPT levels to rebuild after
   GUEST walk + an EPT/NPT walk)         a snapshot restore

A single 2 MiB TLB entry covers what 512 separate 4 KiB entries would, so a workload with a large, actively-touched working set spends far less time in page-table walks. Under hardware virtualization the effect compounds, because a TLB miss can require walking both the guest page tables and the host's nested page tables (EPT on Intel, NPT on AMD). Fewer, larger pages mean fewer of those expensive two-level walks.

The docs call out three concrete payoffs (verify and measure):

# The boot-time impact is in the perf suite. Read it.
rg -rln "huge|hugepage|2M|boottime" tests/integration_tests/performance/ | head
sed -n '1,20p' docs/hugepages.md

The costs and the requirement

Huge pages are not free, and the costs are exactly the kind a contributor must be able to articulate before recommending them.

The pool requirement. hugetlbfs requires the host to pre-allocate a pool of 2 MiB pages. Firecracker maps with MAP_NORESERVE, so it does not reserve pool pages at mmap time — it claims them on demand. If the pool runs dry while the guest is faulting in memory, Firecracker can behave erratically or take a SIGBUS. This is the central operational gotcha:

# Manage the host 2 MiB pool before relying on huge pages.
cat /proc/meminfo | grep -i huge      # HugePages_Total / Free / 2048 kB size
# (operators pre-allocate via vm.nr_hugepages / hugetlbfs mount — see the docs)

Warning: A too-small huge-page pool turns a memory allocation into a SIGBUS-induced crash of the microVM, not a graceful degradation. Pool sizing is a hard operational dependency, not a tuning nicety. This is the tax you pay for the performance.

Internal fragmentation. A 2 MiB page is the minimum granularity. A guest that touches one byte of a 2 MiB region pulls in the whole 2 MiB. For dense oversubscription where resident sets must stay tiny, that coarser granularity can reduce density even as it improves per-guest performance — a direct tension with the density chapter.

The subtle interactions (where the bugs live)

Huge pages do not compose cleanly with three other Firecracker mechanisms, and these interactions are exactly where misconfiguration and real bugs show up. The docs are explicit; read them and internalize them.

sed -n '/Known Limitations/,/FAQ/p' docs/hugepages.md     # dirty-tracking + balloon limits
rg -n "page size|page_size|2M|4K|handshake" docs/snapshotting/handling-page-faults-on-snapshot-resume.md | head

The dirty-tracking interaction is the most counterintuitive and the most important: you cannot have both huge-page performance and diff snapshots at once, because turning on dirty tracking forces KVM back to 4 KiB guest page tables. If you see a huge-page microVM performing no better than a 4 KiB one, the first thing to check is whether dirty tracking is silently on.

NUMA considerations

On a multi-socket host, where a microVM's memory and vCPU threads sit relative to each other matters as much as page size. If a vCPU thread runs on socket 0 but its guest memory was allocated from socket 1's hugetlbfs pool, every memory access crosses the interconnect — a latency penalty that can swamp the TLB win huge pages were supposed to deliver.

Firecracker itself is deliberately NUMA-unaware in its core (one process, one microVM, minimal policy — consistent with the minimal philosophy); NUMA placement is an operator concern, applied from outside:

# Pin a microVM's memory + threads to one NUMA node from the outside.
numactl --cpunodebind=0 --membind=0 ./firecracker --api-sock /tmp/fc.sock
# Allocate the huge-page pool per node so node-local hugetlbfs pages exist.

The lesson for a contributor: when you benchmark huge pages on a NUMA host and the numbers are noisy or worse than expected, suspect cross-node memory before you suspect the feature. Always record numactl --hardware and your pinning alongside the result.

Measuring it honestly

# A clean comparison: same workload, 4K vs 2M, everything else fixed.
# 1. Boot with huge_pages:"None", run a memory-bound benchmark, record.
# 2. Boot with huge_pages:"2M" (host pool pre-allocated), same benchmark, record.

# Inside the guest, watch TLB behavior with perf (memory-bound workload):
perf stat -e dTLB-load-misses,iTLB-load-misses,cycles ./your_workload

# Boot-time delta via the suite's harness.
./tools/devtool test -- integration_tests/performance/test_boottime.py 2>&1 | tail -20

What you should expect to see, and must be able to explain: a memory-bound workload with a large working set shows a meaningful drop in dTLB misses and an improvement in boot/runtime; a workload with a tiny working set shows little or nothing, because it never stressed the TLB in the first place. If you see no difference, the workload is the wrong probe — not the feature failing.

Where to contribute

gh issue list --repo firecracker-microvm/firecracker \
  --search "huge OR hugepage OR memfd OR TLB in:title,body state:open" --limit 40
gh issue list --repo firecracker-microvm/firecracker --label "Type: Performance" --state open

On-ramps: huge-page + UFFD restore edge cases; documenting the dirty-tracking / balloon limitations more sharply; boot-time and TLB benchmarks across page sizes; making the pool-exhaustion SIGBUS failure mode more diagnosable.

Next: huge pages cut boot time as a side effect; the whole boot critical path is the subject of Boot-Time Optimization.