Snapshotting — Intensive

A snapshot is a serialized, resurrectable capture of a running microVM: pause it, write its KVM state and device state to one file and its guest RAM to another, and later — on the same host or a different one, in a fresh Firecracker process — reconstruct the machine and resume it from exactly where it stopped. You have read the snapshotting deep dive and done Lab 9.2, so you know the shape: two files, Full vs Diff, the Persist trait, the Pause→create→Resume create flow, the PUT /snapshot/load restore flow, and memory that comes back lazily via MAP_PRIVATE copy-on-write or userfaultfd.

This masterclass turns that shape into operating skill. You will run the full create/restore workflow until it is muscle memory, inspect both snapshot files with snapshot-editor, measure restore time, and fan many clones out of one snapshot — the actual serverless cold-start pattern that makes Firecracker matter. Then you will go to the hard edge: write a real userfaultfd page-fault handler in Rust that serves guest pages on demand from the memory file, load a snapshot with backend_type: Uffd pointed at its socket, and watch pages fault in one at a time. Finally you will confront the part of snapshotting that gates maintainership — the serialization format as a compatibility surface: the Persist trait, MicrovmState, the serde + bitcode encoding (verify on your branch — versionize is historical), what breaks cross-version restore, and the maintainer discipline that surrounds every change to it.

Note: Snapshotting is not a convenience feature; it is the foundation of serverless density. Restore-and-resume in single-digit milliseconds — versus tens-to-hundreds of milliseconds for a cold boot — is what lets a fleet keep "pre-warmed" function instances and clone them on demand. It is also the single most dangerous area to contribute to, because the snapshot format is a public compatibility surface: break it and you break every operator who relies on cross-version restore. This masterclass takes both facts seriously.

What you will be able to do

By the end of this masterclass you can:

1. Execute the complete Full-snapshot workflow by hand — PATCH /vm {Paused} → PUT /snapshot/create → PATCH /vm {Resumed} — and restore on a fresh firecracker with mem_backend: File.
2. Inspect both snapshot files with snapshot-editor: dump the microVM state file's structure and version, and reason about the memory file's size and layout.
3. Measure restore-and-resume latency and explain where the time goes (state deserialization vs memory faulting).
4. Restore many independent clones from one base snapshot, understand why that is safe (MAP_PRIVATE COW) and where it is not (shared secrets, duplicated entropy/RNG state, MAC/IP collisions), and state the security caveats of restoring shared or untrusted state.
5. Write a working userfaultfd-based memory backend in Rust, modeled on Firecracker's UFFD example, that lazily serves pages from the memory file; load a snapshot against it; and observe on-demand page faults.
6. Explain the snapshot serialization end to end: the Persist trait, MicrovmState aggregation, the serde + bitcode encoding (verify), versioning across releases, exactly what changes break compatibility, how snapshot-editor and rebase-snap help, and the review discipline a snapshot-format PR demands.

Prerequisites

This intensive assumes the foundations below. It will not re-teach them.

# Prerequisite check.
cd ~/firecracker
B=build/cargo_target/x86_64-unknown-linux-musl/release
test -x $B/firecracker && echo "firecracker built"
ls $B/snapshot-editor $B/rebase-snap 2>/dev/null || \
  echo "build the tools: tools/devtool build --release (then check src/snapshot-editor, src/rebase-snap)"
rg -q "trait Persist" src/vmm/src/ && echo "Persist trait present"

Warning: Snapshot restore is privileged and trust-sensitive. Restoring a snapshot reconstructs guest memory and device state from files — if those files are attacker-controlled, restore is a code-execution surface in your VMM. The labs use snapshots you created; the security lab (and Lab 1's caveat section) make the threat model explicit. Never restore a snapshot you did not produce or fully trust.

The three labs

Three labs, in order, climbing from operational fluency to systems-level depth to maintainer judgment.

flowchart LR
    A["Lab 1<br/>create / restore / clone<br/>(File backend)"] --> B["Lab 2<br/>UFFD backend<br/>(lazy, on-demand pages)"]
    B --> C["Lab 3<br/>the format as a<br/>compatibility surface"]

The progression mirrors how a contributor grows into snapshotting: first you operate it (Lab 1), then you understand the memory machinery deeply enough to extend it (Lab 2), then you internalize why the format is sacred and how to change it without breaking the world (Lab 3). The third is the one that separates a contributor from a maintainer.

How snapshotting sits in the tree

Keep this open. Every row is something you locate yourself — paths drift, the crate merge moved several of these, and the serialization backend in particular has changed (verify).

Common mistakes this masterclass corrects

How to verify success

You have completed this intensive when, on your own checkout and with no notes:

# 1. You can run the full create/restore/clone workflow (Lab 1) from memory.
# 2. You can inspect a snapshot:
$B/snapshot-editor info-vmstate version --vmstate-path ./vm.state   # (subcommands vary — verify)
# 3. You can point at the serialization format and the Persist machinery:
rg -n "trait Persist|struct MicrovmState" src/vmm/src/
rg -n "serde|bitcode|Serialize" src/vmm/src/snapshot/
# 4. You have a working UFFD handler (Lab 2) that serves a real restore.
# 5. You can list, cold, three changes that break snapshot compatibility (Lab 3).

And you can answer the question that defines maturity in this area: Given a PR that adds a field to a device's State struct, what must the author do so that existing snapshots still restore and new snapshots restore on older versions where required — and what tests must the PR include?

Where this leads

Snapshotting touches everything you have learned. Restore rebuilds the boot configuration (boot-process masterclass) from serialized state; every virtio device (virtio-devices masterclass) must save/restore its queues and rate limiters correctly or the resumed guest's I/O breaks; the CPU template machinery (CPU templates deep dive) exists partly so a snapshot taken on one host model resumes on another; and the security model (security masterclass) frames restore as a trust boundary. The scale story — UFFD-backed page serving across a fleet — is the subject of engineering/snapshotting-at-scale.

This intensive deepens Level 9 and feeds the dedicated contributor track issue-roadmap Stage 8 (snapshot compatibility).

Next: Lab 1 — Create, Restore, and Clone. Then Lab 2 — Build a UFFD Page-Fault Handler and Lab 3 — Snapshot Compatibility.