Snapshotting

A snapshot is a complete, serialized capture of a running microVM that can be saved to disk and later resurrected — on the same host or a different one — so the restored VM continues from exactly where it was paused. This is the feature that makes fast cloning, "pre-warmed" function instances, and rapid scale-out possible: instead of booting a kernel and a userspace from scratch (tens to hundreds of milliseconds), you restore from a snapshot and resume in single-digit milliseconds.

This chapter covers the two files a snapshot consists of, the Full vs Diff distinction, the Persist trait every device implements, the MicrovmState that ties it all together, the create flow (Pause → create → Resume) and the load flow (PUT /snapshot/load → resume), how memory is restored lazily with MAP_PRIVATE COW or UFFD, and the backward-compatibility constraints that make snapshotting one of the hardest areas to contribute to safely.

Note: A snapshot is a serialization format that is part of Firecracker's public compatibility surface. A microVM snapshotted by version X must restore on a compatible version, and every device's serialized state is versioned. This is why changes to device state structs are reviewed harder than almost anything else: break the format and you break every operator who relies on cross-version restore.

The two files

rg -n "snapshot_path|mem_file_path|SnapshotCreateParams|CreateSnapshotParams|MemBackendType|mem_backend" src/vmm/src/
sed -n '1,60p' docs/snapshotting/snapshot-support.md

A snapshot is two files, produced together and required together to restore:

The state file is small and structured (it serializes Rust structs); the memory file is large (the size of guest RAM, or less for a diff). Restore needs both: the state file to recreate the machine and devices, the memory file to repopulate guest RAM.

Full vs Diff snapshots

rg -n "track_dirty_pages|SnapshotType|Full|Diff|enable_diff_snapshots|dirty_bitmap|KVM_GET_DIRTY_LOG" src/vmm/src/

A Full snapshot writes the entire guest memory. A Diff snapshot writes only the pages the guest modified since the previous snapshot, using KVM's dirty-page tracking (KVM_GET_DIRTY_LOG), which requires the microVM to have been configured with track_dirty_pages: true (the field was renamed from enable_diff_snapshots; use track_dirty_pages). Diff snapshots are smaller and faster to take but must be layered on a base — the rebase-snap tool merges a diff memory file onto a base memory file (find src/rebase-snap -name "*.rs").

The Persist trait

rg -n "trait Persist|impl Persist|fn save|fn restore|type State|type ConstructorArgs|Persist<" src/vmm/src/

Every component that holds state implements a Persist trait (locate the definition with rg -n "trait Persist" src/vmm/src/). Its shape is roughly:

#![allow(unused)]
fn main() {
pub trait Persist<'a> {
    type State;            // a serializable snapshot of this component
    type ConstructorArgs;  // what you need (besides State) to rebuild it
    type Error;
    fn save(&self) -> Self::State;
    fn restore(args: Self::ConstructorArgs, state: &Self::State) -> Result<Self, Self::Error>;
}
}

save() produces a plain, serializable State struct; restore() rebuilds the live component from that State plus whatever runtime arguments (ConstructorArgs) can't be serialized (file descriptors, the guest memory handle, the event manager). Every device — block, net, vsock, balloon, rng — implements it, as do the vCPUs and the VM-level state. This is the mechanism by which a heterogeneous machine is reduced to a flat, versioned data structure and reconstructed.

flowchart LR
    subgraph save
      Dev1["block.save()"] --> S1["BlockState"]
      Dev2["net.save()"] --> S2["NetState"]
      Vcpu["vcpu.save()"] --> S3["VcpuState"]
    end
    S1 --> MVM["MicrovmState"]
    S2 --> MVM
    S3 --> MVM
    MVM --> File["microVM state file (serialized)"]

MicrovmState: the aggregate

rg -n "struct MicrovmState|vm_state|device_states|vcpu_states|memory_state|VmInfo" src/vmm/src/persist.rs

MicrovmState (find it with rg -n "struct MicrovmState" src/vmm/src/persist.rs) is the top-level serialized structure: it aggregates the VM-level KVM state, the per-vCPU states, all device states, and the memory-region description. The create path builds it by calling save() across the machine; the load path consumes it, calling restore() across the machine. src/vmm/src/persist.rs and src/vmm/src/snapshot/ are the files to read; the builder entry point for restore is build_microvm_from_snapshot (rg -n "build_microvm_from_snapshot" src/vmm/src/builder.rs).

The create flow: Pause → create → Resume

rg -n "create_snapshot|CreateSnapshot|fn pause|fn resume|VmState|Paused|Resumed" src/vmm/src/

You cannot snapshot a running microVM — its state would be inconsistent. The flow is:

API=/tmp/fc.sock
# 1. Pause the microVM (vCPUs stop; device state quiesces).
curl -X PATCH --unix-socket $API --data '{"state":"Paused"}' http://localhost/vm
# 2. Create the snapshot: writes the state file and the memory file.
curl -X PUT --unix-socket $API --data \
 '{"snapshot_type":"Full","snapshot_path":"./vm.state","mem_file_path":"./vm.mem"}' \
 http://localhost/snapshot/create
# 3. Resume (the original VM keeps running) — or just kill it if you only wanted the snapshot.
curl -X PATCH --unix-socket $API --data '{"state":"Resumed"}' http://localhost/vm

sequenceDiagram
    participant Op as Operator
    participant API as API thread
    participant VMM as VMM thread
    Op->>API: PATCH /vm {Paused}
    API->>VMM: VmmAction::Pause
    VMM->>VMM: pause all vCPUs, quiesce devices
    Op->>API: PUT /snapshot/create
    API->>VMM: VmmAction::CreateSnapshot
    VMM->>VMM: save() across machine → MicrovmState
    VMM->>VMM: write state file + memory file
    Op->>API: PATCH /vm {Resumed}
    API->>VMM: VmmAction::Resume

The load flow: PUT /snapshot/load

rg -n "load_snapshot|LoadSnapshot|mem_backend|MemBackendType|resume_vm|Uffd|userfaultfd" src/vmm/src/

Restore happens on a fresh Firecracker process (no boot-source, no drives configured — the snapshot carries all of that). A single call recreates the machine:

curl -X PUT --unix-socket $API --data '{
  "snapshot_path": "./vm.state",
  "mem_backend": { "backend_path": "./vm.mem", "backend_type": "File" },
  "resume_vm": true
}' http://localhost/snapshot/load

Tip: The standalone mem_file_path on load is deprecated — use the mem_backend object. With backend_type: Uffd, Firecracker connects to a separate page-fault handler over a Unix socket and sends it the guest memory layout; that handler decides how to serve pages (from a file, over the network, etc.). UFFD is what makes restore lazy: pages load on first touch, not all up front.

How memory comes back: MAP_PRIVATE COW and on-demand

rg -n "MAP_PRIVATE|mmap|GuestMemoryMmap|MADV|userfaultfd|register_memory|copy_on_write" src/vmm/src/

For the File backend, Firecracker mmaps the memory file MAP_PRIVATE. That gives copy-on-write: the restored VM reads pages straight from the file (no copy), and only when it writes a page does the kernel make a private copy. Combined with the kernel's demand paging, this means restore doesn't have to read all of guest RAM up front — pages fault in as the guest touches them. The Uffd backend takes this further: page faults are delivered to a userspace handler that can fetch pages from anywhere (a slower tier, a remote store), which is the foundation of snapshot-at-scale strategies (see ../engineering/snapshotting-at-scale.md and ../masterclass/snapshotting/lab-02-uffd-page-fault-handler.md).

restore (File backend):
  mmap(mem_file, MAP_PRIVATE)  →  guest reads = page from file (shared)
                                  guest writes = COW, private copy
  no upfront copy of RAM → fast resume, pages fault in on demand

restore (Uffd backend):
  guest touches an unmapped page → page fault → userfaultfd → handler process
  handler supplies the page bytes → guest continues

Backward compatibility and versioning

rg -n "version|Version|FC_VERSION|app_version|SNAPSHOT_VERSION|Versionize|serde|bincode" src/vmm/src/snapshot/
sed -n '1,80p' docs/snapshotting/snapshot-support.md

The state file is a versioned serialization. Firecracker documents which versions can restore which snapshots (the support matrix in docs/snapshotting/), and there are real rules: you generally restore on the same or a newer compatible Firecracker version, the CPU model/features must be compatible (CPU templates exist partly to normalize this across heterogeneous hosts — see cpu-templates-and-cpuid.md), and adding/removing/reordering fields in a device's State struct can break the format. This is why snapshot-touching PRs demand integration tests that restore across versions, and why "Status: snapshot-compat" issues (../issue-roadmap/stage-8-snapshot-compat.md) are a distinct contributor track. Treat the format as an API, not an implementation detail.

Reading exercise

# 1. The Persist trait and a couple of its implementations.
rg -n "trait Persist" src/vmm/src/
rg -n "impl Persist" src/vmm/src/devices/virtio/block/
rg -n "impl Persist" src/vmm/src/devices/virtio/net/

# 2. MicrovmState and the create/load entry points.
rg -n "struct MicrovmState|fn create_snapshot|fn restore_from_snapshot|load_snapshot" src/vmm/src/persist.rs
rg -n "build_microvm_from_snapshot" src/vmm/src/builder.rs

# 3. Full vs Diff and dirty-page tracking.
rg -n "track_dirty_pages|SnapshotType|Diff|Full|KVM_GET_DIRTY_LOG" src/vmm/src/

# 4. The memory backend choice (File vs Uffd).
rg -n "MemBackendType|backend_type|Uffd|userfaultfd|MAP_PRIVATE" src/vmm/src/

# 5. Do a real create+restore by hand following the API blocks above.

# 6. Read the support/compat docs.
ls docs/snapshotting/

Answer:

1. What two files make up a snapshot, what does each hold, and why are both required to restore?
2. Explain Full vs Diff: what's in each memory file, what must be enabled for Diff, and what tool merges a diff onto a base?
3. Describe the Persist trait: what do save() and restore() return/take, and why does restore need ConstructorArgs separate from State?
4. List the three API calls of the create flow in order and explain why the VM must be paused first.
5. Contrast the File and Uffd memory backends. What makes restore lazy in each case?
6. Why is the snapshot format treated as a public compatibility surface, and name two changes that could break cross-version restore.

Common bugs and symptoms

Validation: prove you understand this

1. Draw the two-file structure of a snapshot and label what restore reads from each.
2. Explain Full vs Diff snapshots, including the configuration prerequisite and the rebase tool.
3. Write the Persist trait signature from memory and explain why restore takes both a State and ConstructorArgs.
4. Sequence the create flow and the load flow as API calls, and say where resume_vm matters.
5. Explain how MAP_PRIVATE COW and UFFD each make memory restore lazy, and when you'd choose UFFD.
6. Argue why the snapshot format is a compatibility surface and describe the testing a snapshot PR must include.

Next: mmds-metadata-service.md — a VMM-internal service the guest reads over the network, and a good example of state that is deliberately not snapshotted.