The virtio Balloon Device

The balloon is the device that runs backwards. Every other virtio device exists to give the guest something — a disk, a network, randomness. The balloon exists to take memory away from the guest and hand it back to the host. You inflate the balloon inside the guest; the guest kernel surrenders free pages; Firecracker calls madvise(MADV_DONTNEED) on the matching host-memory range; the host reclaims that physical RAM. That is the entire point: the balloon is the lever by which a host that oversubscribed its RAM pulls back what its microVMs are not using.

This chapter traces the inflate path end to end — the PFN array travelling in the inflate queue, the shift from guest page-frame-number to host virtual address, the destructive madvise — then covers deflate, deflate_on_oom, the statistics queue, free-page reporting, and how all of this makes the oversubscription and density story economically real. After it you will be able to read Firecracker's inflate processing, explain why MADV_DONTNEED is safe only if the guest is honest, and reason about how the balloon interacts with snapshots.

Note: The balloon is virtio device type ID 5. It has up to three virtqueues — inflateq (queue 0), deflateq (queue 1), and an optional statsq present only when VIRTIO_BALLOON_F_STATS_VQ is negotiated. Direction is the thing to keep straight: inflate grows the balloon and shrinks the guest's usable memory (host gains RAM); deflate shrinks the balloon and grows the guest's usable memory (host loses RAM). The balloon "size" is memory the guest has agreed not to use. Confirm the type ID and queue layout on your branch with the rg below.

Where the balloon lives

# The device's source tree.
find . -type d -path '*devices/virtio/balloon*'

# The device struct, its type constant, queue indices, and config space.
rg -n "struct Balloon\b|TYPE_BALLOON|INFLATE_INDEX|DEFLATE_INDEX|STATS_INDEX|NUM_QUEUES|VIRTIO_BALLOON" \
  src/vmm/src/devices/virtio/balloon/

# The feature flags the device offers.
rg -n "VIRTIO_BALLOON_F_STATS_VQ|VIRTIO_BALLOON_F_DEFLATE_ON_OOM|VIRTIO_BALLOON_F_REPORTING" \
  src/vmm/src/devices/virtio/balloon/

Confirm three things before reading on: the type constant resolves to 5; there are two always-on queues (inflate, deflate) plus a conditionally-present stats queue; and the device holds a handle to guest memory (it must, to madvise it) plus a config block carrying the target balloon size. The module is typically split into the device struct, the queue-event handlers, and an event_handler that wires it into the EventManager — verify the file split on your branch.

                 ┌──────────────────────── guest ─────────────────────────┐
   API           │  virtio-balloon driver                                  │
 PUT /balloon ──►│  reads target from config space                         │
 amount_mib=256  │  inflate: pick free pages, push their PFNs to inflateq  │
                 │  deflate: pull pages back, push their PFNs to deflateq   │
                 └─────────────────────────┬──────────────────────────────┘
                                           │ QueueNotify (ioeventfd)
                  ┌──────────────────── host (VMM thread) ─────────────────┐
                  │  Balloon::process_inflate_queue                        │
                  │    for each PFN: gpa = pfn << 12                        │
                  │    coalesce into ranges → madvise(MADV_DONTNEED)        │
                  │  Balloon::process_deflate_queue  (mostly bookkeeping)   │
                  │  Balloon::process_stats_queue    (parse guest stats)    │
                  └─────────────────────────────────────────────────────────┘

The inflate path: PFN array → madvise → reclaimed RAM

# The inflate handler, the PFN shift, and the host syscall that reclaims memory.
rg -n "fn process_inflate|VIRTIO_BALLOON_PFN_SHIFT|fn remove_range|MADV_DONTNEED|madvise|compact" \
  src/vmm/src/devices/virtio/balloon/
# How PFNs are read out of the descriptor.
rg -n "fn process_pfns|pfn|read_obj|GuestAddress" src/vmm/src/devices/virtio/balloon/

This is the device's whole reason to exist, so read it slowly. When you raise the balloon target, the guest's virtio-balloon driver selects pages that are genuinely free, removes them from its allocator, and writes their Page Frame Numbers into the inflate queue. A PFN is a 4-byte little-endian integer: the guest physical page index. The descriptor's buffer is therefore a packed array of u32 PFNs — this is the concrete payload that the virtqueue machinery carries. The guest kicks; the ioeventfd fires; the VMM thread runs the inflate handler.

For each PFN the device computes the guest physical address by shifting:

#![allow(unused)]
fn main() {
// page = 4 KiB, so VIRTIO_BALLOON_PFN_SHIFT == 12
let guest_phys_addr = (pfn as u64) << VIRTIO_BALLOON_PFN_SHIFT; // pfn << 12
}

That guest physical address is then translated, through the host mmap that backs guest memory, into a host virtual address. The device coalesces consecutive PFNs into contiguous ranges (one madvise over a 2 MiB run beats 512 syscalls), and on each range calls the syscall that does the actual reclaim:

madvise(host_addr, length, MADV_DONTNEED);

MADV_DONTNEED tells the host kernel: drop the physical pages backing this virtual range; I don't need their contents. The host frees that RAM and returns it to the system. The guest's view of its own physical address space is unchanged — the pages still "exist" — but they are no longer backed by physical memory on the host. If the guest later touches one of those addresses (it should not, having promised they are in the balloon), the page re-faults to zero: a fresh, zeroed physical page is allocated on demand. That re-fault-to-zero is exactly why inflate is safe to reclaim and exactly why it is dangerous if abused.

sequenceDiagram
    participant API as API thread
    participant Cfg as config space
    participant G as guest balloon driver
    participant IQ as inflateq
    participant D as Balloon (VMM thread)
    participant K as host kernel

    API->>Cfg: write target num_pages (256 MiB)
    Cfg-->>G: config-change interrupt
    G->>G: pick free pages, take from allocator
    G->>IQ: descriptor = [pfn0,pfn1,...] (u32 LE array)
    G->>D: QueueNotify (ioeventfd kick)
    loop each PFN
        D->>D: gpa = pfn << 12 ; gva = mmap_base + gpa
    end
    D->>K: madvise(gva, len, MADV_DONTNEED)
    K-->>K: free physical pages (host RAM reclaimed)
    D->>G: add_used + interrupt (descriptor consumed)
    D->>Cfg: bump "actual" toward target

Warning: MADV_DONTNEED is destructive. The page contents are gone the instant the syscall returns. Correctness depends entirely on the guest driver only ever inflating pages it has genuinely freed. The device cannot verify this — it trusts the guest's PFN list. A buggy or malicious guest that inflates a page still in use is corrupting its own memory, not the host's (the host mapping is private per microVM), so this is a guest-integrity contract, not a host escape. But it is the reason the inflate path must never be "optimised" into touching pages outside the supplied ranges.

Deflate, and deflate_on_oom

rg -n "fn process_deflate|VIRTIO_BALLOON_F_DEFLATE_ON_OOM|DEFLATE_ON_OOM|deflate" \
  src/vmm/src/devices/virtio/balloon/

Deflate is the inverse intent but a far simpler implementation. When the balloon target drops, the guest driver pulls pages back out of the balloon and returns them to its own allocator, pushing their PFNs onto the deflateq. Crucially, Firecracker generally does not call madvise on deflate. There is nothing to do on the host side: the host pages were already freed during inflate, and the guest simply needs them back. The moment the guest writes to one of those addresses it re-faults to zero and gets a fresh physical page automatically. So process_deflate is mostly bookkeeping — consume the descriptors, mark them used, and let the page table do the rest on next touch. Read it and confirm it is conspicuously not doing a syscall per PFN.

deflate_on_oom is a feature, not a path: the VIRTIO_BALLOON_F_DEFLATE_ON_OOM flag. When the host sets deflate_on_oom: true in the API and the guest negotiates the feature, the guest is permitted to auto-deflate the balloon when it hits memory pressure, before the OOM killer starts killing processes. The balloon is, in effect, an emergency reserve the guest can reclaim from itself under duress. Without it, an over-inflated balloon plus a memory spike means the guest OOM-kills a workload while RAM it "owns" sits trapped in the balloon. With it, the guest deflates first and tries to survive. The trade-off is obvious: deflate_on_oom makes the guest more robust but lets it defeat your reclaim under pressure — the host's memory is no longer guaranteed back.

Tip: Decide deflate_on_oom per your overcommit posture. If you are densely packed and rely on the balloon staying inflated to make the arithmetic work, leaving it off keeps reclaimed memory reclaimed. If guest availability matters more than density, turn it on so a memory spike degrades gracefully instead of OOM-killing.

The statistics queue

rg -n "fn process_stats|BalloonStats|statistics|VIRTIO_BALLOON_S_|stats_polling_interval|TimerFd" \
  src/vmm/src/devices/virtio/balloon/

The third queue, statsq, exists only when VIRTIO_BALLOON_F_STATS_VQ is negotiated, and it runs on a pull model. Firecracker arms a timer fd with period stats_polling_interval_s; on each tick it makes a stats-request descriptor available, the guest fills it with a tag/value list, and process_stats parses the VIRTIO_BALLOON_S_* entries into a BalloonStats struct that GET /balloon/statistics returns. The fields are guest-reported memory telemetry:

These statistics are how you decide how much to reclaim. AVAIL tells you how much the guest could give up; SWAP_OUT and MAJFLT rising tell you the guest is under pressure and you have inflated too far. Statistics require stats_polling_interval_s > 0; set it to 0 and the timer never arms and GET /balloon/statistics has nothing to report. The polling interval is the knob between freshness and overhead — each poll is a guest round-trip.

flowchart LR
    Timer["TimerFd<br/>(every stats_polling_interval_s)"] -->|tick on EventManager| H["process_stats_queue"]
    H -->|make descriptor available| Guest["guest fills tag/value list"]
    Guest -->|kick| H
    H -->|parse VIRTIO_BALLOON_S_*| Stats["BalloonStats"]
    Stats -->|GET /balloon/statistics| API["API response"]

Free-page reporting

rg -n "VIRTIO_BALLOON_F_REPORTING|free_page|reporting|hinting" \
  src/vmm/src/devices/virtio/balloon/

Free-page reporting (VIRTIO_BALLOON_F_REPORTING, sometimes called free-page hinting) is a newer mechanism — verify it is wired up on your branch before relying on it. The idea: instead of you driving a balloon target and the guest inflating toward it, the guest proactively reports pages it is not using, on its own schedule, so the host can madvise them without a full inflate cycle. It is the same host-side reclaim (MADV_DONTNEED), but driven continuously and autonomously by the guest rather than episodically by your API calls. Where classic ballooning is a deliberate "give me 256 MiB back now," reporting is a steady trickle of "you can drop these while I'm not looking."

Activation and the event loop

rg -n "fn activate|register|inflate.*eventfd|deflate.*eventfd|stats.*eventfd|DRIVER_OK" \
  src/vmm/src/devices/virtio/balloon/
rg -n "impl Persist|fn save|fn restore|BalloonState" src/vmm/src/devices/virtio/balloon/

Nothing happens until the guest driver sets DRIVER_OK. At that moment the virtio-MMIO transport calls the device's activate, which registers the inflate, deflate, and (if negotiated) stats queue eventfds plus the stats timer fd with the EventManager. From then on a guest kick on any queue, or a timer tick, wakes the VMM thread's epoll loop and dispatches into the matching handler. The balloon also implements Persist, so its target, negotiated features, and stats config travel into a snapshot — which is where the next caution comes in.

Warning: The balloon's reclaim interacts with snapshotting and track_dirty_pages. A page that was MADV_DONTNEED-ed is not present; how it appears to the dirty-page tracker and the snapshot writer matters for correctness and for snapshot size. Restoring a snapshot taken with an inflated balloon must reconstruct a consistent balloon state, and a restored guest that deflates will re-fault pages that the snapshot may or may not have persisted. Read snapshotting alongside this chapter; do not assume balloon + snapshot "just works" without checking your branch.

Why the balloon exists: oversubscription

The balloon only earns its complexity at scale. Suppose you run 20 microVMs, each "given" 1 GiB, on a host with 16 GiB. You have promised 20 GiB you do not have — you oversubscribed. This works because guests rarely touch all their RAM at once: with demand paging, a guest only consumes host physical memory for pages it actually faults in. The balloon is the active lever on top of that passive saving — when the host runs short, you inflate balloons in idle microVMs and MADV_DONTNEED pulls their unused RAM back into the host's free pool, to be handed to whichever microVM needs it. Without the balloon you are stuck waiting for guests to never touch memory; with it, you reclaim on demand. This is the whole subject of oversubscription and density, and the balloon is its primary tool.

   Promised:  [VM1 1G][VM2 1G][VM3 1G] ... [VM20 1G]  = 20 GiB
   Host RAM:  ───────────────── 16 GiB ─────────────
   Reclaim:   inflate balloons in idle VMs → madvise(DONTNEED)
              ↳ unused guest pages return to host free pool
              ↳ re-allocate to the VMs that are actually busy

Reading exercise

Run these against your Firecracker checkout (paths assume the repo root):

# 1. Confirm the device type ID and the queue indices.
rg -n "TYPE_BALLOON|INFLATE_INDEX|DEFLATE_INDEX|STATS_INDEX|NUM_QUEUES" \
  src/vmm/src/devices/virtio/balloon/

# 2. Find the PFN shift and the madvise call.
rg -n "VIRTIO_BALLOON_PFN_SHIFT|MADV_DONTNEED|madvise|fn remove_range" \
  src/vmm/src/devices/virtio/balloon/

# 3. Read the inflate handler top to bottom.
rg -n "fn process_inflate|fn process_pfns" src/vmm/src/devices/virtio/balloon/

# 4. Compare it with the deflate handler.
rg -n "fn process_deflate" src/vmm/src/devices/virtio/balloon/

# 5. Find where the API target reaches config space.
rg -n "amount_mib|num_pages|fn write_config|actual|target" src/vmm/src/devices/virtio/balloon/

# 6. Drive a real balloon (microVM must be running).
API=/tmp/firecracker.socket
curl -X PUT --unix-socket $API \
  --data '{"amount_mib":256,"deflate_on_oom":true,"stats_polling_interval_s":1}' \
  http://localhost/balloon
curl --unix-socket $API http://localhost/balloon/statistics

Then answer:

1. What is the concrete byte layout of a single inflate descriptor's buffer, and what does the device do with each 4-byte element?
2. Why is gpa = pfn << 12 and not pfn << some-other-shift? What does 12 encode?
3. Trace one PFN from guest physical address to the exact host virtual address madvise receives. Which mapping performs that translation, and where is it established?
4. Why does the deflate handler not call madvise, while inflate does? What re-establishes a deflated page's backing memory?
5. With stats_polling_interval_s: 0, what does GET /balloon/statistics return, and why?
6. If a guest inflated a page it was still using, who gets corrupted — guest or host — and why is the blast radius bounded to that microVM?

Common bugs and symptoms

Validation: prove you understand this

1. A guest inflates 64 MiB. Walk the data from the guest driver picking free pages to the host's RSS dropping, naming the queue, the descriptor payload, the shift, and the syscall in order.
2. Explain precisely why MADV_DONTNEED is destructive and what re-fault-to-zero means for a page the guest later touches.
3. Distinguish inflate from deflate at the host syscall level — what does the host do on each, and why is deflate "mostly bookkeeping"?
4. State what deflate_on_oom changes, who acts when it fires (host or guest), and the density trade-off it imposes on your overcommit plan.
5. Tie the PFN array to virtqueues: which descriptor field holds the PFNs, are those descriptors device-readable or device-writable, and how does the guest signal a batch is ready?
6. Given 20 microVMs of 1 GiB on a 16 GiB host, describe the moment the host runs short and the exact mechanism by which the balloon returns RAM to the free pool — and what VIRTIO_BALLOON_S_AVAIL tells you about how far you can safely inflate.

Next: The virtio Entropy Device — the smallest virtio device in the tree, and how the host feeds guest randomness through a single queue.