Oversubscription & Density

The serverless business model is a bet on idleness. A host running a thousand Lambda functions is, at any instant, running maybe a few dozen that are actually executing code; the rest are blocked on I/O, between requests, or simply not being called. If you provisioned each microVM its full configured RAM and CPU up front, you would strand almost all of your hardware. Oversubscription — soft allocation of CPU and memory well beyond the physical supply — is what turns that idleness into density, and density into margin.

Firecracker's design document states the goal plainly: microVMs can oversubscribe host CPU and memory, with the degree controlled by the operator who factors in workload correlation. The NSDI paper reports oversubscription tested past 20× and thousands of microVMs per host. This chapter is about how that works, what the bet actually is, and — the part newcomers underestimate — what breaks when you push it.

Note: Oversubscription is mostly an operator discipline (cgroups, host tuning, placement) layered on top of Firecracker mechanisms (low overhead, the balloon, on-demand paging). This chapter covers both, but the security-relevant host configuration lives in docs/prod-host-setup.md — read it alongside.

Soft allocation: what Firecracker actually reserves

The reason oversubscription is even possible is that Firecracker reserves almost nothing eagerly. Two mechanisms make a microVM's real footprint far smaller than its configured footprint:

1. Memory is faulted in on demand. Guest RAM is mmap'd but not populated. A microVM configured with 512 MiB whose workload touches 40 MiB has a resident set near 40 MiB, not 512. The MAP_NORESERVE flag (verify in vstate/memory.rs) tells the kernel not to reserve backing store at mmap time.
2. The VMM overhead itself is tiny. Firecracker targets < 5 MiB of memory overhead per microVM beyond the guest's own usage — the entire point of the minimal device model. Less VMM code and less device emulation means less resident memory multiplied across thousands of processes.

# Guest memory is mapped lazily, not reserved. Confirm the flags.
rg -n "MAP_NORESERVE|MAP_PRIVATE|MAP_ANONYMOUS|with_mmap_flags|PROT_READ" \
  src/vmm/src/vstate/memory.rs | head

CPU is "soft" in the same spirit: each vCPU is a host thread, and the host scheduler (under cgroup limits the operator sets via the jailer) time-slices many more vCPU threads than there are physical cores. A vCPU blocked in KVM_RUN waiting for the guest to do work consumes no core. Oversubscribing CPU is just oversubscribing threads, which operating systems have done forever — the novelty is doing it with a hard isolation boundary in between.

configured:   1000 microVMs × (2 vCPU, 512 MiB) = 2000 vCPU, 500 GiB "promised"
physical:     64 cores, 256 GiB RAM
the bet:      at any instant only a fraction are hot; resident sets are small;
              loads are uncorrelated enough that aggregate demand < physical supply

The statistical bet (and when it loses)

Oversubscription is a statistical multiplexing bet, identical in spirit to an airline overbooking seats. It works because demand is uncorrelated: independent tenants peak at different times, so aggregate demand stays under physical supply with high probability. The bet has two failure modes, and a serious engineer keeps both in view:

The defense is operator discipline, not a Firecracker feature: pick the oversubscription ratio from real workload correlation data, set cgroup limits so a single noisy microVM cannot starve the host, and — critically — disable swap. docs/prod-host-setup.md is explicit that swap is a footgun here: swapping guest memory to disk converts a memory-pressure event into a latency catastrophe and also opens a side channel.

Warning: Memory oversubscription without a reclaim mechanism is a slow-motion OOM. Guest kernels allocate page cache aggressively and rarely hand memory back on their own. That is exactly the problem the balloon solves.

The balloon: reclaiming memory the guest won't give back

A guest Linux kernel, left alone, will fill unused RAM with page cache and never voluntarily return it to the host. From the host's perspective the microVM looks "full" even when the guest is doing nothing. The virtio-balloon device is how the host reclaims that memory.

ls src/vmm/src/devices/virtio/balloon/
rg -n "madvise|MADV_DONTNEED|inflate|deflate|target|actual" \
  src/vmm/src/devices/virtio/balloon/device.rs | head

Mechanism: the host sets a balloon target size; the in-guest balloon driver allocates that many pages inside the guest and reports their physical addresses to the device. The host then madvise(MADV_DONTNEED)s the corresponding host pages — telling the host kernel it can drop them, instantly shrinking the microVM's resident set. To give memory back, the host lowers the target and the guest driver frees the pages ("deflate").

flowchart LR
    Host["host: set balloon target ↑"] --> Drv["guest balloon driver:\nallocate N pages in guest"]
    Drv --> Report["report guest PFNs to device"]
    Report --> Madv["host: madvise(MADV_DONTNEED)\non those host pages"]
    Madv --> Reclaim["resident set shrinks →\nmemory available for other microVMs"]

The balloon also reports statistics (/balloon/statistics, stats_polling_interval_s) — VIRTIO_BALLOON_S_* fields like available/free/cached memory and swap activity — which is the host's window into guest memory pressure, and what an oversubscription controller reads to decide how hard to squeeze.

rg -n "VIRTIO_BALLOON_S_|statistics|stats_polling|deflate_on_oom" \
  src/vmm/src/devices/virtio/balloon/device.rs | head
sed -n '1,40p' docs/ballooning.md

The trade-offs are real:

virtio-mem (docs/memory-hotplug.md) is the newer, more flexible alternative — dynamic memory resizing rather than balloon inflate/deflate — and is one of the maturing areas worth owning.

KSM: the density win Firecracker deliberately refuses

Kernel Samepage Merging (KSM) scans host memory, finds identical pages across processes, and merges them into one shared copy-on-write page. For a host running a thousand microVMs from the same base image — same kernel, same libc, same runtime — KSM looks like free density: enormous numbers of identical pages collapse into one.

Firecracker's production guidance disables KSM anyway. This is one of the clearest examples in the whole project of security beating density:

rg -n "KSM|samepage|Samepage" docs/prod-host-setup.md

The reasoning: KSM merges pages across tenants, which creates a side channel. Whether a write is fast (private page) or slow (triggers a copy-on-write break of a merged page) leaks whether another tenant has an identical page — a memory- deduplication timing attack. In a single-tenant deployment KSM might be fine; in Firecracker's multi-tenant threat model, the guest is hostile and any cross-tenant observable is a leak. So the density win is left on the table on purpose.

Tip: This is the pattern to internalize. Density is desirable, but the moment a density optimization creates a cross-tenant observable, it fails the isolation constraint and the answer is no. KSM, swap, and SMT (also disabled in prod for the same side-channel reason) are all the same decision.

The same logic puts SMT/hyperthreading off in production: sibling hyperthreads share microarchitectural resources, so co-scheduling two tenants on one core's threads is another side channel. The density you would gain from SMT is refused for isolation.

Minimizing per-microVM overhead

Density is bounded by per-microVM overhead as much as by oversubscription ratio. Every megabyte of VMM resident memory and every device you emulate is multiplied by the number of microVMs on the host, so the < 5 MiB overhead target is a density target, not just an efficiency nicety. This is the through-line connecting this chapter to the minimal device model: fewer devices, no BIOS, no PCI enumeration, a memory-safe runtime with a small footprint — all of it exists partly so that the fixed cost of a microVM stays tiny enough that thousands fit.

# Per-microVM overhead is measured in CI. Find the harness.
rg -rln "memory_overhead|overhead|rss|resident" tests/integration_tests/performance/ | head

What breaks at density

The honest part. At a few microVMs everything works; at thousands, second-order effects appear that you will not see in a dev environment:

The pattern: density problems are rarely bugs, they are resource exhaustion and contention that only manifest above a threshold. Reproducing them requires running at scale, which is why the oversubscription masterclass lab and a real multi-microVM harness matter. A contributor who can build a credible density reproduction is doing work the maintainers cannot easily do themselves.

Where to contribute

gh issue list --repo firecracker-microvm/firecracker \
  --search "balloon OR virtio-mem OR oversubscription OR memory in:title state:open" --limit 40
gh issue list --repo firecracker-microvm/firecracker --label "Type: Performance" --state open

On-ramps: balloon statistics and edge cases; virtio-mem hardening; per-microVM overhead regression tests; documenting density-limit tuning; reproducing a resource-exhaustion failure mode and proposing the limit/knob that prevents it.

Next: density is bounded by per-microVM overhead, which is bounded by the device model — The Minimal Device Model Philosophy.