Lab 9.2: Snapshot and Restore

Lab type: Run-it / Trace-it (persistence + compatibility, hands-on) Estimated time: 4–6 hours

Background

A snapshot is a serialized photograph of a running microVM: the KVM CPU state, every emulated device's internal state (registers, virtqueue positions, in-flight config), and the guest's entire RAM. Firecracker writes this as two files — a small microVM state file and a large memory file — and can later reconstruct a live, resumable microVM from them on a different Firecracker process, possibly on a different host. This is the feature behind fast cold-start mitigation: boot once, snapshot, then clone the snapshot into thousands of warm microVMs that resume in milliseconds.

The reason this lab sits at Level 9 is not the happy path — creating and restoring a snapshot is a handful of API calls. The reason is the compatibility burden it creates. The moment Firecracker persists device state, the byte layout of that state becomes a contract: a snapshot written by one version must remain loadable by the versions Firecracker promises to support. Every device implements a Persist trait whose serialized State is part of that contract. Reorder a field, change a default, or drop a register without a versioned migration, and you have produced a snapshot that a customer cannot restore after upgrading — a bug that does not show up as a red unit test and that you cannot un-ship. Learning to see that hazard in a diff is the maintainer skill this lab builds.

Why This Lab Matters for Contributors

Snapshot-restore is one of the highest-leverage, highest-risk areas of the codebase. Maintainers review every Persist change for cross-version safety, the way they review every wire change in a clustered system. You need to be able to do that review.
The Persist trait and the state/memory split are the heart of the snapshotting deep dive; this lab makes the abstract trait concrete by having you inspect the bytes it produces with snapshot-editor.
The UFFD restore path — lazy, on-demand page faulting of guest memory from the snapshot — is how restore-at-scale stays cheap. This lab introduces it; the UFFD lab in the snapshotting masterclass builds a handler.
Snapshot density is a fleet economics question covered in snapshotting at scale and oversubscription and density — this lab is where the mechanism behind those economics becomes real to you.

Prerequisites

Completed Lab 9.1 and can boot a microVM by hand.
You understand guest memory layout (guest memory deep dive) — the memory file is the guest RAM mapping.
You understand the device model (MMIO bus / device manager deep dive); every device on that bus has Persist state.
Read docs/snapshotting/ in full, especially the support and compatibility pages.

# Build the binaries this lab needs, including the snapshot inspector.
tools/devtool build --release
ARCH=$(uname -m)
BIN=build/cargo_target/${ARCH}-unknown-linux-musl/release
ls -l $BIN/firecracker $BIN/snapshot-editor
# If snapshot-editor is missing:
#   cargo build --release -p snapshot-editor

The Snapshot/Restore Lifecycle

sequenceDiagram
    participant Op as Operator (curl)
    participant FC1 as firecracker #1 (source)
    participant Disk as state + mem files
    participant FC2 as firecracker #2 (target)

    Note over FC1: microVM is booted and running
    Op->>FC1: PATCH /vm {"state":"Paused"}
    Note over FC1: vCPUs paused, devices quiesced
    Op->>FC1: PUT /snapshot/create {Full, snapshot_path, mem_file_path}
    FC1->>Disk: serialize MicrovmState -> state file
    FC1->>Disk: write guest RAM -> memory file
    Op->>FC1: PATCH /vm {"state":"Resumed"}
    Note over FC1: source keeps running (or is killed)

    Op->>FC2: PUT /snapshot/load {snapshot_path, mem_backend{File}, resume_vm}
    FC2->>Disk: read state file -> rebuild Vmm (build_microvm_from_snapshot)
    FC2->>Disk: mmap memory file MAP_PRIVATE (COW / on-demand)
    Note over FC2: restored microVM resumes from the exact paused instant

Note: Pausing first is mandatory and not cosmetic. A snapshot taken while vCPUs were mutating memory and devices were mid-transaction would be internally inconsistent. Pause stops the vCPUs and quiesces devices so the serialized state is coherent.

Step 1: Boot, then pause and create a Full snapshot

Boot a microVM as usual (no jailer needed for this lab; add it as a stretch goal), then run the create sequence.

ARCH=$(uname -m)
BIN=build/cargo_target/${ARCH}-unknown-linux-musl/release
API=/tmp/fc.sock
sudo "$BIN/firecracker" --api-sock "$API" &

# Standard boot. NOTE: enable dirty-page tracking BEFORE boot if you want Diff snapshots
# later — it cannot be turned on after the machine is running.
sudo curl -X PUT --unix-socket "$API" --data \
  '{"vcpu_count":2,"mem_size_mib":256,"track_dirty_pages":true}' \
  http://localhost/machine-config
sudo curl -X PUT --unix-socket "$API" --data \
  '{"kernel_image_path":"./vmlinux","boot_args":"console=ttyS0 reboot=k panic=1 pci=off"}' \
  http://localhost/boot-source
sudo curl -X PUT --unix-socket "$API" --data \
  '{"drive_id":"rootfs","path_on_host":"./rootfs.ext4","is_root_device":true,"is_read_only":false}' \
  http://localhost/drives/rootfs
sudo curl -X PUT --unix-socket "$API" --data \
  '{"action_type":"InstanceStart"}' http://localhost/actions

# Let it finish booting (watch the serial console), then snapshot.
sudo curl -X PATCH --unix-socket "$API" --data \
  '{"state":"Paused"}' http://localhost/vm

sudo curl -X PUT --unix-socket "$API" --data \
  '{"snapshot_type":"Full","snapshot_path":"/tmp/snap.state","mem_file_path":"/tmp/snap.mem"}' \
  http://localhost/snapshot/create

# Resume the source (or kill it — the snapshot already captured everything).
sudo curl -X PATCH --unix-socket "$API" --data \
  '{"state":"Resumed"}' http://localhost/vm

Inspect what you produced:

ls -l /tmp/snap.state /tmp/snap.mem
# state file: small (KVM + device state).  memory file: ~= mem_size_mib (the guest RAM).

Tip: The memory file is the same size as the guest's RAM allocation because it is the guest RAM, page for page. This is why the memory file dominates snapshot size and why Diff snapshots (only the dirty pages) and UFFD (lazy loading) exist — see Steps 4 and 5.

Step 2: Restore on a fresh Firecracker process

Restore is a single API call on a brand-new firecracker, before any boot configuration. The state file rebuilds the machine; the memory file is mapped back in.

ARCH=$(uname -m)
BIN=build/cargo_target/${ARCH}-unknown-linux-musl/release
API2=/tmp/fc2.sock
sudo "$BIN/firecracker" --api-sock "$API2" &

sudo curl -X PUT --unix-socket "$API2" --data '{
  "snapshot_path": "/tmp/snap.state",
  "mem_backend": { "backend_path": "/tmp/snap.mem", "backend_type": "File" },
  "resume_vm": true
}' http://localhost/snapshot/load

# Confirm the restored machine is alive and at the same instant the source was paused.
sudo curl --unix-socket "$API2" http://localhost/   # state should be Running/Resumed

The restored guest continues from the exact instruction the source was paused on — its process tables, open files, and memory contents are identical. If you logged into the guest before snapshotting and left a shell variable set, it is still set after restore.

Warning: Restoring from a snapshot taken on a different host CPU can fail or misbehave if the guest saw CPU features the target host lacks. Production restore uses CPU templates to normalize CPUID/MSRs so a snapshot is portable across a fleet of heterogeneous hosts. See the CPU templates deep dive; this is a real maintainer concern for any snapshot-format change.

Step 3: Inspect both files with snapshot-editor

snapshot-editor reads the state file's structured format and lets you dump and (carefully) edit it. This is how you see the Persist state the abstract trait produces.

ARCH=$(uname -m)
BIN=build/cargo_target/${ARCH}-unknown-linux-musl/release

# Discover the subcommands — names vary by version, so ask the tool.
"$BIN/snapshot-editor" --help
"$BIN/snapshot-editor" info --help 2>/dev/null || true

# Dump the version the snapshot was written with, and the high-level state contents.
"$BIN/snapshot-editor" info --snapshot-path /tmp/snap.state 2>/dev/null || \
  "$BIN/snapshot-editor" version --snapshot-path /tmp/snap.state 2>/dev/null

Read the format on the Rust side so the dump means something:

# The top-level serialized microVM state — what the state file contains.
rg -n "struct MicrovmState|VmInfo|DeviceStates|vm_state" src/vmm/src/persist.rs

# How the snapshot records the version that wrote it (the compatibility anchor).
rg -n "SNAPSHOT_VERSION|app_version|version" src/vmm/src/snapshot/ src/vmm/src/persist.rs

# The serialization format used for the state file (what snapshot-editor parses).
rg -n "Serialize|serialize|Versionize|bincode|serde" src/vmm/src/snapshot/

Map what snapshot-editor shows you onto these structs. The state file contains, at minimum: a version stamp, the VmInfo/machine config, the KVM VM and per-vCPU state, and each device's serialized Persist::State. The memory file is opaque guest RAM with no internal structure.

File	Contains	Size driver	Inspected with
state file (`snapshot_path`)	version stamp, machine config, KVM VM/vCPU state, every device's `Persist` state	small, fixed-ish	`snapshot-editor`
memory file (`mem_file_path`)	the guest's entire RAM, page for page	`mem_size_mib`	`xxd`/`cmp` (Step 4)

Step 4: Diff snapshots and dirty-page tracking

A Full snapshot writes all of guest RAM. A Diff snapshot writes only the pages that changed since a base snapshot, using dirty-page tracking — which is why you set track_dirty_pages: true before boot in Step 1. Diff snapshots make incremental snapshots cheap, but they are dev-preview (verify the status on your branch / CHANGELOG before relying on them).

# Take a Diff snapshot after some guest activity (requires track_dirty_pages from boot).
sudo curl -X PATCH --unix-socket "$API" --data '{"state":"Paused"}'  http://localhost/vm
sudo curl -X PUT   --unix-socket "$API" --data \
  '{"snapshot_type":"Diff","snapshot_path":"/tmp/diff.state","mem_file_path":"/tmp/diff.mem"}' \
  http://localhost/snapshot/create
sudo curl -X PATCH --unix-socket "$API" --data '{"state":"Resumed"}' http://localhost/vm

# The diff memory file is typically MUCH smaller than the full one — only dirty pages.
ls -l /tmp/snap.mem /tmp/diff.mem

# Diff memory must be rebased onto the base before a plain File restore (or use UFFD).
rg -n "rebase" src/rebase-snap/ 2>/dev/null
"$BIN/../release/rebase-snap" --help 2>/dev/null || \
  echo "rebase-snap merges a diff memory file onto a base — see src/rebase-snap/"

# How track_dirty_pages drives the dirty bitmap on the VMM side.
rg -n "track_dirty_pages|dirty|KVM_GET_DIRTY_LOG|get_dirty_bitmap" src/vmm/src/

Note: Dirty-page tracking has a runtime cost — KVM has to log writes. That is why it is opt-in and off by default, and why a PR that turns it on unconditionally would be a performance regression. The field used to be called enable_diff_snapshots; it was renamed track_dirty_pages (verify on your branch).

Step 5: A first look at the UFFD memory backend

The File backend mmaps the whole memory file MAP_PRIVATE and faults pages in from it on demand. The UFFD (userfaultfd) backend goes further: firecracker registers guest memory with a userspace page-fault handler, and a separate process serves pages on demand — from local storage, from a remote store, or copy-on-write from a shared base. This is how you restore thousands of microVMs from one base snapshot without each copying all of RAM.

# The two memory backends and how load chooses between them.
rg -n "backend_type|MemBackendType|Uffd|File" src/vmm/src/

# The UFFD wiring on the restore side.
rg -n "userfaultfd|uffd|UffdMsg|register_uffd|MemoryRegion" src/vmm/src/

Restore with the UFFD backend points backend_path at a Unix socket where your handler process is listening, rather than a file:

{
  "snapshot_path": "/tmp/snap.state",
  "mem_backend": { "backend_path": "/tmp/uffd.sock", "backend_type": "Uffd" },
  "resume_vm": true
}

You will build such a handler in the UFFD masterclass lab. Here you only need to know it exists and why: it decouples "the microVM is restored and running" from "all of its RAM has been read off disk," which is the whole game at fleet scale.

Step 6: Read the `Persist` trait and reason about compatibility

This is the maintainer payload of the lab. Find the trait and one device's implementation.

# The contract: save() produces a serializable State; restore() rebuilds from it.
rg -n "trait Persist" src/vmm/src/
rg -n "type State|fn save|fn restore|associated" src/vmm/src/    # the trait's shape

# A concrete device implementation — e.g. virtio-block.
rg -ln "impl .*Persist" src/vmm/src/devices/virtio/
rg -n "struct .*State|impl .*Persist" src/vmm/src/devices/virtio/block/persist.rs 2>/dev/null

Now reason explicitly about what makes a Persist change safe or unsafe:

SAFE   : appending a new OPTIONAL field to a device's State, gated so older snapshots
         (which lack it) deserialize with a sensible default, and newer Firecracker
         versions that don't know the field can still skip it.
UNSAFE : reordering existing fields           -> positional/format misread on old snapshots
         removing a field                      -> old snapshot has bytes the new code won't read
         changing a field's type/width         -> silent value corruption
         changing a default that boot relied on -> restored machine differs from saved one
ALWAYS : any State change is a SNAPSHOT_VERSION event. Bump the version, add a migration
         path if needed, and add a test that LOADS A SNAPSHOT WRITTEN BY THE OLD FORMAT.

# Find the snapshot-compatibility tests — the safety net for the above.
rg -ln "snapshot" tests/integration_tests/functional/ | rg -i "snapshot|persist|compat"
rg -n "snapshot_version|load.*snapshot|restore" tests/integration_tests/ | head

A maintainer reviewing a diff that touches any */persist.rs asks exactly three questions: (1) did the serialized State layout change? (2) if so, is there a version bump and a migration? (3) is there a test that loads a snapshot produced by the previous format? No to any of those, no merge.

Implementation Requirements / Deliverables

A Full snapshot (/tmp/snap.state + /tmp/snap.mem) created via the Paused→create→Resumed sequence, with file sizes reported and explained.
A successful restore on a fresh firecracker process, with GET / showing the restored machine running, and evidence it resumed at the saved instant.
snapshot-editor output mapping the state file's contents onto MicrovmState and the recorded snapshot version.
A Diff snapshot whose memory file is demonstrably smaller than the Full one, plus a one-paragraph explanation of track_dirty_pages and its runtime cost (note dev-preview status — verify on your branch).
A written explanation of the Persist trait and a concrete SAFE-vs-UNSAFE change list for one real device's State, citing the file you read.
A short note on what the UFFD backend changes about restore and why it matters at scale.

Troubleshooting

`PUT /snapshot/create` returns an error about state

You must PATCH /vm {"state":"Paused"} first. Create only works on a paused machine.

Diff snapshot create fails or is refused

You did not enable track_dirty_pages before boot. It cannot be turned on after the machine is running. Reboot with it set in /machine-config. Also confirm Diff is supported on your branch (rg -n "Diff" src/vmm/src/ and the CHANGELOG) — it is dev-preview.

Restore fails with a version or deserialization error

The snapshot was written by a different Firecracker version than the one restoring it, and the formats are incompatible. Confirm with snapshot-editor info. This is the exact failure mode the compatibility window exists to prevent — and exactly what a careless Persist change would inflict on users.

A state file of a few tens of KiB and a memory file approximately equal to mem_size_mib.
A restored microVM whose GET / reports it running, resuming from the paused instant.
snapshot-editor output showing a version stamp and structured device/VM state.
A Diff memory file substantially smaller than the corresponding Full one.

Stretch Goals

Snapshot under the jailer. Combine this lab with Lab 9.1: create and restore a snapshot from inside a jailer chroot, with the snapshot files inside the chroot. Note the path implications.
Round-trip a guest fact. Write a unique marker to a file in the guest (or set an env var in a running shell), snapshot, restore on a fresh process, and confirm the marker survived — proof the snapshot captured live state, not just config.
Edit the state file. Use snapshot-editor to read a device field, modify it, write it back, and observe the effect (or failure) on restore. This is how you build intuition for why hand-editing persisted state is dangerous.
Trace build_microvm_from_snapshot. rg -n "fn build_microvm_from_snapshot" src/vmm/src/builder.rs and read how the restore-side builder reconstructs the Vmm from MicrovmState — contrast it with build_microvm_for_boot.
Find a real snapshot-compat PR. gh pr list -R firecracker-microvm/firecracker --search "snapshot version persist" --state merged and read how a maintainer handled a Persist change: the version bump, the migration, the test.

Validation / Self-check

Answer without notes. They gate completion.

Why must the microVM be Paused before PUT /snapshot/create? What would be wrong with the snapshot otherwise?
The snapshot is two files. Name each, what it contains, and which one dominates the total size — and why.
What does the Persist trait require a device to provide, and why is its serialized State a compatibility contract rather than an internal detail?
List two changes to a device's Persist::State that are SAFE and two that are UNSAFE, and explain why the unsafe ones break cross-version restore without a red unit test.
What does track_dirty_pages enable, what is its runtime cost, and why is it off by default? What is the relationship between it and Diff snapshots?
What does the UFFD memory backend let you do that the File backend cannot, and why does that matter when restoring thousands of microVMs from one base?
You are reviewing a PR that adds a field to virtio-net's persisted state with no version bump and no compat test. State exactly what you'd request and the failure you're preventing.

Next: Lab 9.3 — Analyze a Performance Regression, where the density budget that makes snapshots worth shipping becomes a number you must measure and defend.

Open-Source Engineer & Contributor