Lab 2: Oversubscription and Density

Background

Lab 1 measured the cost of one microVM. This lab multiplies it by thousands. The serverless business model is a bet on idleness: a host advertises far more CPU and memory than it physically has, wins because most tenants are idle at any instant, and packs density into margin. The NSDI paper reports oversubscription tested past 20× and thousands of microVMs per host. Those are not Firecracker features you switch on — they are an operator bet layered on top of Firecracker mechanisms (lazy memory, tiny VMM overhead, the balloon). This lab makes you run the bet on your own host: launch many microVMs in a loop, measure what they actually cost in aggregate (not what they're configured for), reclaim memory with the balloon, push density until something gives, and reason — with numbers — about when the statistical bet wins and when it loses.

The central discipline carries over from Lab 1: configured is not resident. A microVM configured with 512 MiB whose workload touches 40 MiB has a resident set near 40 MiB, because guest RAM is faulted in on demand and mmap'd with MAP_NORESERVE. Reading configured RAM as a footprint overstates it by an order of magnitude — the single most common density-reasoning error. You will measure the real resident set, watch the balloon shrink it with madvise(MADV_DONTNEED), and find the failure modes (Out of puff!, OOM, file-descriptor and VMA exhaustion) that only appear at scale.

This is a measure-it / reason-it lab.

Why This Lab Matters for Contributors

• Density is the reason Firecracker exists, and the per-microVM overhead target (< 5 MiB) is a density target, not an efficiency nicety — every megabyte is multiplied by the microVM count. A contributor who can measure aggregate overhead and attribute it can credibly touch anything that affects the fixed cost of a microVM.
• The balloon is an active, stateful device with real edge cases (virtio-balloon deep dive): deflate_on_oom, statistics polling, the "Out of puff!" failure. You cannot review balloon or virtio-mem PRs without having driven reclaim and seen it fail.
• The failure modes at scale (FD limits, vm.max_map_count, scheduler contention) are resource-exhaustion problems the maintainers cannot easily reproduce on a laptop. A contributor who can build a credible multi-microVM density reproduction is doing work the team genuinely needs — see engineering/oversubscription-and-density.

Prerequisites

cd ~/firecracker
B=build/cargo_target/$(uname -m)-unknown-linux-musl/release
test -x $B/firecracker && echo "firecracker built"
ls vmlinux-* *.ext4 2>/dev/null

# Confirm the two mechanisms this lab leans on, in the source.
rg -n "MAP_NORESERVE|MAP_PRIVATE|MAP_ANONYMOUS" src/vmm/src/vstate/memory.rs
rg -n "madvise|MADV_DONTNEED|inflate|deflate|target|actual" src/vmm/src/devices/virtio/balloon/device.rs | head

# Host limits you will hit at density — note them now.
ulimit -n ; cat /proc/sys/vm/max_map_count

Warning: Density benchmarks are even noisier than boot-time ones, and they can destabilize a host: pushing memory oversubscription on a machine with swap turns a pressure event into a thrash. Disable swap (swapoff -a on a throwaway host), set conservative limits, and run on a machine you can afford to OOM. Never run this on anything shared or production.

Step-by-Step Tasks

Step 1: Establish single-microVM overhead as the baseline

Density is a multiplication, so you need the multiplicand: the real resident cost of one idle microVM. Boot a single small microVM and measure its RSS — both the Firecracker process and the guest's resident memory.

API=/tmp/fc-1.sock; rm -f $API
sudo $B/firecracker --api-sock $API &
for i in $(seq 1 100); do [ -S $API ] && break; done
curl -sX PUT --unix-socket $API --data \
 '{"kernel_image_path":"./vmlinux-6.1.x","boot_args":"console=ttyS0 reboot=k panic=1 nomodule 8250.nr_uarts=0"}' \
 http://localhost/boot-source
curl -sX PUT --unix-socket $API --data \
 '{"drive_id":"rootfs","path_on_host":"./ubuntu-24.04.ext4","is_root_device":true,"is_read_only":false}' \
 http://localhost/drives/rootfs
curl -sX PUT --unix-socket $API --data '{"vcpu_count":1,"mem_size_mib":256}' http://localhost/machine-config
curl -sX PUT --unix-socket $API --data '{"action_type":"InstanceStart"}' http://localhost/actions
sleep 3

FC=$(pgrep -n firecracker)
echo "configured guest RAM: 256 MiB"
grep -E 'Rss|Pss' /proc/$FC/smaps_rollup     # ACTUAL resident set — far below 256 MiB

Record the resident set of an idle 256 MiB microVM. It should be a small fraction of 256 MiB — that gap is the whole point. Configured RAM is a promise; resident RAM is the bill. This number, times N, is the floor of your density.

Step 2: Launch many microVMs in a loop

Now scale. The firecracker-demo pattern is a launch loop: one socket, one tap (or no network for simplicity first), one rootfs per microVM. Start small (N=10), confirm it's stable, then climb.

N=20
KERNEL=./vmlinux-6.1.x
launch_one() {
  local i=$1
  local api=/tmp/fc-d-$i.sock
  rm -f $api
  sudo $B/firecracker --api-sock $api >/tmp/fc-d-$i.log 2>&1 &
  for t in $(seq 1 200); do [ -S $api ] && break; done
  curl -sX PUT --unix-socket $api --data \
    "{\"kernel_image_path\":\"$KERNEL\",\"boot_args\":\"console=ttyS0 reboot=k panic=1 nomodule 8250.nr_uarts=0\"}" \
    http://localhost/boot-source >/dev/null
  # Each microVM needs its own rootfs (read-only base + per-VM overlay is the real pattern;
  # for measurement, a shared read-only rootfs with is_read_only:true is simplest).
  curl -sX PUT --unix-socket $api --data \
    '{"drive_id":"rootfs","path_on_host":"./ubuntu-24.04.ext4","is_root_device":true,"is_read_only":true}' \
    http://localhost/drives/rootfs >/dev/null
  curl -sX PUT --unix-socket $api --data '{"vcpu_count":1,"mem_size_mib":256}' http://localhost/machine-config >/dev/null
  curl -sX PUT --unix-socket $api --data '{"action_type":"InstanceStart"}' http://localhost/actions >/dev/null
}
for i in $(seq 1 $N); do launch_one $i; done
sleep 5
echo "launched $(pgrep -c firecracker) firecracker processes"

Tip: A shared, read-only base rootfs (is_read_only:true) lets N microVMs share one backing file via the host page cache — the cleanest way to measure Firecracker overhead without N copies of the rootfs dominating. The production pattern is a read-only base plus a per-VM writable overlay; note the difference.

Step 3: Measure aggregate memory overhead and CPU

This is the headline measurement. Sum the resident memory across all Firecracker processes and compare it to both the configured total and N × the single-VM baseline.

# Aggregate resident memory across all firecracker processes (PSS is the fairest —
# it splits shared pages, e.g. the shared read-only rootfs, across sharers).
sum_pss() {
  local total=0
  for pid in $(pgrep firecracker); do
    local pss=$(grep -m1 '^Pss:' /proc/$pid/smaps_rollup 2>/dev/null | awk '{print $2}')
    total=$(( total + ${pss:-0} ))
  done
  echo "aggregate PSS: $(( total / 1024 )) MiB across $(pgrep -c firecracker) microVMs"
}
sum_pss

echo "configured total: $(( N * 256 )) MiB"   # the promise
# Compare: aggregate PSS should be FAR below the configured total (lazy memory),
# and close to N × your Step-1 baseline.

# Aggregate CPU: idle microVMs should burn almost nothing (vCPUs blocked in KVM_RUN).
top -b -n1 | grep -c firecracker
ps -o pid,pcpu,rss,comm -C firecracker | sort -k2 -rn | head

Interpret the three numbers:

The gap between configured and resident is oversubscription headroom you didn't have to pay for. Idle CPU near zero is the same story for cores: a vCPU blocked in KVM_RUN consumes no core.

Step 4: Reclaim memory with the balloon

Idle guests fill RAM with page cache and never hand it back, so resident sets creep up even when guests are doing nothing. The virtio-balloon reclaims it. Configure a balloon (pre-boot), then inflate it and watch the resident set drop.

# A microVM WITH a balloon (configure pre-boot). Re-launch one VM with a balloon device:
API=/tmp/fc-bln.sock; rm -f $API
sudo $B/firecracker --api-sock $API &
for i in $(seq 1 100); do [ -S $API ] && break; done
# ... boot-source, drive, machine-config as before, mem_size_mib:512 ...
curl -sX PUT --unix-socket $API --data \
 '{"amount_mib":0,"deflate_on_oom":true,"stats_polling_interval_s":1}' http://localhost/balloon
# ... InstanceStart ...

# Make the guest touch memory (so there's something to reclaim), then inflate.
FC=$(pgrep -n firecracker)
grep -m1 '^Rss:' /proc/$FC/smaps_rollup           # resident before reclaim

# Inflate the balloon: claim 256 MiB back from the guest.
curl -sX PATCH --unix-socket $API --data '{"amount_mib":256}' http://localhost/balloon
sleep 2
grep -m1 '^Rss:' /proc/$FC/smaps_rollup           # resident AFTER reclaim — should drop

# Read the balloon's view of guest memory pressure:
curl -sX GET --unix-socket $API http://localhost/balloon/statistics

The mechanism, made concrete: you raised the target → the in-guest driver allocated 256 MiB of guest pages and reported their physical addresses → Firecracker madvise(MADV_DONTNEED)'d the corresponding host pages → the resident set shrank. To give memory back, PATCH a lower amount_mib ("deflate"). Confirm the mechanism in the source:

rg -n "madvise|MADV_DONTNEED|inflate|deflate|amount_mib|target|actual|VIRTIO_BALLOON_S_" \
  src/vmm/src/devices/virtio/balloon/device.rs | head -20

Step 5: Push density until something breaks

Now find the edges. Climb N until you hit a real limit, and identify which one. This is the honest part of density work — the failures are resource exhaustion, not bugs, and they only appear at scale.

# Climb N. After each batch, check for the failure signatures:
# 1) Open file descriptors — each microVM consumes several (socket, kvm, drive, tap...).
ls /proc/$(pgrep -n firecracker)/fd | wc -l ; ulimit -n
# 2) mmap regions / VMAs — guest memory + devices per VM.
cat /proc/$(pgrep -n firecracker)/maps | wc -l ; cat /proc/sys/vm/max_map_count
# 3) Memory pressure on the host.
free -m ; dmesg | tail -5 | grep -i 'oom\|kill' || echo "no OOM yet"
# 4) Balloon over-inflation: the guest OOMs if you take too much.
curl -sX GET --unix-socket /tmp/fc-bln.sock http://localhost/balloon/statistics
rg -n "Out of puff|out of puff" src/vmm/src/devices/virtio/balloon/   # the over-inflation log

Record which limit you hit first and at what N. That number — and which resource ran out — is the real, host-specific density ceiling for your configuration.

Step 6: Reason about the >20× statistical bet

Finally, connect the numbers to the argument. Oversubscription is statistical multiplexing — overbooking seats — and it works only because tenant demand is uncorrelated. Write up the bet with your own measurements as evidence:

configured:   N microVMs × (1 vCPU, 256 MiB)  = N vCPU, N×256 MiB "promised"
resident:     aggregate PSS (Step 3)            ≈ a fraction of the promise
the bet:      at any instant only a few are hot; resident sets small; loads
              uncorrelated → aggregate demand < physical supply (with high prob.)
loses when:   (a) too many touch RAM at once → overcommit → OOM (no swap!)
              (b) a correlated event (deploy, traffic wave) makes many hot at once
                  → vCPU threads contend → latency climbs host-wide

The two failure modes are not symmetric with the success case — they are tail events, and the operator's whole job is choosing a ratio from real correlation data so the tail stays below physical supply. Firecracker provides the mechanisms (lazy memory, tiny overhead, the balloon); the ratio is the operator's bet. State, from your run: the oversubscription ratio you sustained, the resource that bounded it, and the failure mode you'd watch for in production.

Note: The density wins Firecracker deliberately refuses — KSM and SMT — are the tell that this is a security project first. Both would raise density and both create cross-tenant side channels, so production guidance disables them (oversubscription engineering essay). When a density optimization creates a cross-tenant observable, the answer is no. Hold that line in any density PR.

Implementation Requirements / Deliverables

• The single-microVM resident baseline (idle, 256 MiB configured) with the real RSS recorded and the configured-vs-resident gap noted.
• A launch loop bringing up N ≥ 10 microVMs simultaneously, stable.
• The three aggregate numbers — configured total, aggregate PSS, N × baseline — with a sentence interpreting the gaps.
• A balloon inflate that visibly shrinks a microVM's resident set, with the before/after RSS and a read of /balloon/statistics.
• Density pushed to a real limit, with the limiting resource identified (FDs, VMAs, memory, or scheduler) and the N at which it appeared.
• A written statement of the oversubscription bet using your numbers: the ratio sustained, the bounding resource, and the failure mode to watch.

Troubleshooting

Aggregate memory looks like N × configured (huge)

You measured the wrong thing — almost certainly configured (or virtual) size, not resident. Use Pss/Rss from /proc/<pid>/smaps_rollup, not mem_size_mib and not VSZ. Guest RAM is faulted in on demand; resident is what counts.

MicroVMs fail to launch past some N with no obvious error

Check the host limits before blaming Firecracker: ulimit -n (FDs), vm.max_map_count (VMAs), free -m (RAM). The per-VM logs (/tmp/fc-d-$i.log) will show the failing syscall. This is resource exhaustion, not a VMM bug — that distinction is the lesson.

Balloon inflate doesn't shrink the resident set

The guest had nothing reclaimable (it wasn't using the memory), or the guest balloon driver isn't present/loaded, or you inflated and the guest re-faulted the pages immediately. Make the guest touch memory first, confirm the balloon device is attached and the guest has virtio_balloon, and read /balloon/statistics to see actual vs target.

The guest OOMs when you inflate the balloon

You took too much — "Out of puff!". The balloon cannot reclaim memory the guest is actively using; inflating past the guest's free memory starves it. Inflate to a target with headroom and set deflate_on_oom:true (which only helps userspace allocations — kernel allocations can still OOM).

Host becomes unresponsive under density

You overcommitted memory with swap on, and it's thrashing. swapoff -a on a throwaway host before density work — swap converts a memory-pressure event into a latency catastrophe, which is exactly why prod guidance forbids it.

Expected Output

# Single idle microVM (256 MiB configured):
$ grep -E 'Rss|Pss' /proc/$FC/smaps_rollup
Rss:   38912 kB        # ~38 MiB resident vs 256 MiB configured — the gap is the point

# 20 microVMs:
aggregate PSS: 612 MiB across 20 microVMs       # vs configured total 5120 MiB
# ~30 MiB/VM resident — well under the 256 MiB promise

# Balloon reclaim:
before inflate:  Rss:  410000 kB
after  inflate:  Rss:  152000 kB    # madvise(MADV_DONTNEED) dropped ~256 MiB

# Pushing density — the limit you hit (illustrative):
launch failed at N=240:  EMFILE   (ulimit -n was 1024)  <-- FD exhaustion, not a bug

Stretch Goals

1. Quantify the COW/page-cache saving. Compare aggregate PSS with N microVMs sharing one read-only rootfs vs N copies. The gap is what the shared base + page cache saves — a density lever in its own right.
2. Correlated-load failure. Make all N guests go hot at once (a stress in each) and watch per-request latency climb. This is failure mode (b) — reproduce it and explain why uncorrelated load is the whole premise.
3. Balloon statistics as a controller input. Poll /balloon/statistics across all VMs and write a toy controller that inflates the balloon on the VMs reporting the most free/cached memory. That is, in miniature, an oversubscription controller.
4. virtio-mem instead of the balloon. If your branch has it (docs/memory-hotplug.md), reclaim with virtio-mem and compare its behavior to the balloon. Where is it better?
5. Find a density issue. gh issue list --repo firecracker-microvm/firecracker --search "balloon OR oversubscription OR memory OR density in:title state:open" — reproduce one at scale and propose the limit/knob that prevents it.

Validation / Self-check

Answer without notes; these gate completion:

1. Why is configured RAM not the same as resident RAM, and which flag makes guest memory lazy? What's the consequence for density reasoning?
2. Describe the balloon mechanism end to end: target, the guest driver, the host madvise, and how memory is given back.
3. You measure aggregate PSS far below the configured total for N microVMs. What two things explain the gap?
4. State the oversubscription bet in one sentence, and name its two failure modes.
5. At scale you can't launch past N. Name three host limits to check before suspecting Firecracker.
6. Why do KSM and SMT — both density wins — get disabled in production?
7. The balloon logs "Out of puff!". What did you do wrong, and what does deflate_on_oom actually protect?

Next: Lab 3: Benchmark the I/O engines — drill into the one per-request cost that most shapes density under load. Compare the Sync vs io_uring block engines under fio and connect the result to when each is appropriate.