Snapshotting at Scale

The most important number in serverless is the cold start. When a Lambda function has not run recently, something has to create a fresh execution environment before the first byte of your code runs — and a microVM that boots in 125 ms is still 125 ms of latency the user pays. Snapshotting is the mechanism that drives that number toward zero. Instead of booting a kernel, initializing devices, and starting userspace every time, you do it once, capture the entire machine state, and then restore it — bringing a fully-booted, application-ready microVM back to life in single-digit milliseconds.

That is the promise. The engineering is where it gets hard, because snapshotting sits at the intersection of three of Firecracker's hardest constraints at once: fast switching (restore must be milliseconds, not seconds), low overhead (you cannot eagerly copy gigabytes of guest RAM), and isolation (you are about to resume a machine from a state file that, in a multi-tenant world, you may not fully trust). This chapter is about how Firecracker resolves those tensions, and where the open problems still are.

Note: This chapter assumes you have read the snapshotting deep dive and the guest memory deep dive. It does not re-explain the Persist trait or the API workflow — it asks the design questions: what does each mechanism cost, and what would you change?

The two-file model

A Firecracker snapshot is two files, and understanding why it is two is the key to the whole subsystem.

┌────────────────────────────┐      ┌─────────────────────────────────┐
│  microVM state file        │      │  memory file                    │
│  (snapshot_path)           │      │  (mem_file_path)                │
│                            │      │                                 │
│  • KVM vCPU state (regs,   │      │  • the entire guest RAM,        │
│    sregs, CPUID, MSRs,     │      │    page for page                │
│    LAPIC, ...)             │      │  • the BIG one: MiB to GiB      │
│  • device state (virtio    │      │                                 │
│    queues, config, ...)    │      │  restored by mmap, NOT by       │
│  • memory layout / regions │      │  reading the whole file         │
│  serde + bitcode encoded   │      │                                 │
│  small: KiB                │      │                                 │
└────────────────────────────┘      └─────────────────────────────────┘

The split exists because the two files have completely different size and access profiles. The state file is small (kilobytes) and must be deserialized in full — it is the description of what the machine is. The memory file is enormous (however much RAM the guest had) and must not be read in full at restore time, or you have thrown away the entire latency advantage. Restore speed lives or dies on never touching most of the memory file.

# Every device serializes itself via the Persist trait. Find the contract.
rg -rn "trait Persist|fn save|fn restore|SaveState|RestoreState" src/vmm/src/snapshot/ src/vmm/src/persist.rs | head

# The top-level save/restore orchestration.
rg -rn "fn create_snapshot|fn restore_from_snapshot|pub fn save|snapshot_state_to_file" src/vmm/src/persist.rs | head

Warning: The block devices' backing files are not part of the snapshot. The state file references drives by path; restoring a snapshot reattaches the same (or equivalent) rootfs. If you restore the same snapshot into many clones sharing one read-only rootfs, that is by design — but a writable rootfs shared across clones is a correctness and security bug. See docs/snapshotting/network-for-clones.md and the random-for-clones note for the analogous entropy problem.

The serialization format and the versioning burden

Snapshots are serialized with serde + the bitcode format (verify on your branch — this replaced the older bespoke "versionize" approach). Read the canonical description before anything else:

sed -n '1,60p' docs/snapshotting/versioning.md
rg -n "bitcode|serde|SNAPSHOT_VERSION|format version" src/vmm/src/snapshot/ docs/snapshotting/versioning.md | head

The critical design fact: snapshot format versions are independent of Firecracker versions. Each Firecracker release declares which snapshot format version it supports; on load it checks compatibility and refuses an incompatible file rather than silently corrupting state.

This is where the burden lives, and it is the heaviest tax in the whole subsystem. Every device's persisted state is part of a public, versioned format. The moment a snapshot taken by version N must restore on version N+k, you have a compatibility contract:

This is why a PR that touches a device's Persist impl gets scrutinized harder than almost anything else: a careless change can make every snapshot in a fleet of millions unrestorable. The maintainers will ask "what does this do to snapshot compatibility?" and "where is the cross-version restore test?" before they ask anything else.

# How cross-version compatibility is actually tested.
rg -rn "snapshot" tests/integration_tests/functional/ | rg -i "compat|version|restore" | head

COW and on-demand paging: why restore is fast

Restore does not read the memory file. It maps it. The guest's RAM is mmap'd MAP_PRIVATE over the memory file, so:

• The kernel does copy-on-write: pages are shared with the file (and, across clones, shared between microVMs) until the guest writes one, at which point the kernel copies just that page.
• Pages are faulted in on demand: the guest touches a page, the CPU faults, the kernel pulls that one page from the file. A microVM that only touches 8 MiB of a 512 MiB snapshot only ever reads 8 MiB.

# The mmap of guest memory at restore. Note MAP_PRIVATE / on-demand behavior.
rg -n "MAP_PRIVATE|MAP_NORESERVE|mmap|from_file|MmapRegionBuilder" src/vmm/src/vstate/memory.rs | head

That on-demand behavior is what makes restore "instant": the latency you pay at /snapshot/load is proportional to the state file (tiny) plus the few pages the guest touches before it serves its first request, not to the total RAM.

UFFD: taking control of the page-fault path

MAP_PRIVATE over a file works, but it hands the page-fault policy to the host kernel and the local filesystem. For serverless at scale you want control: serve pages from a remote store, from a deduplicated page cache, with prefetching, with metrics — none of which a plain file mapping gives you. That is what userfaultfd (UFFD) is for.

When you load a snapshot with mem_backend.backend_type = Uffd, Firecracker does not map the memory file itself. It registers the guest memory region with a userfaultfd and hands the file descriptor to a separate page-fault handler process over a Unix socket. When the guest faults on a page, the kernel notifies your handler, which decides where the page content comes from and injects it.

sequenceDiagram
    participant Guest as guest vCPU
    participant KVM
    participant FC as Firecracker
    participant UFFD as userfaultfd
    participant Handler as page-fault handler (separate process)
    FC->>UFFD: register guest memory region
    FC->>Handler: send region layout + uffd fd over UDS
    Guest->>KVM: access unmapped guest page
    KVM->>UFFD: page fault
    UFFD->>Handler: fault event (address)
    Handler->>Handler: fetch page (local file / remote store / dedup cache)
    Handler->>UFFD: UFFDIO_COPY the page in
    UFFD->>Guest: resume; access completes

# Firecracker's UFFD wiring — the userfaultfd crate, the handshake.
rg -rn "userfaultfd|Uffd|GuestRegionUffd|guest_memory_from_uffd|backend_type" src/vmm/src/persist.rs src/vmm/src/vmm_config/snapshot.rs | head

The design payoff: the page-fault handler is your code, outside Firecracker's seccomp cage, and it can implement arbitrary policy. The huge-page restore path requires UFFD precisely because the handler must know the page size — see huge pages and the UFFD masterclass lab.

Tip: The handler is a security boundary you own. It receives a guest's memory layout and serves its pages; a buggy handler is a way to corrupt or leak guest state across tenants. Treat it with the same suspicion as the VMM.

Diff snapshots and dirty-page tracking

A full snapshot writes the entire guest RAM. A diff snapshot (developer preview — verify) writes only the pages that changed since the base. The machinery is KVM's dirty-page logging, gated by track_dirty_pages on /machine-config:

# Dirty-page tracking: the KVM dirty log + the atomic bitmap on each region.
rg -n "track_dirty|dirty_bitmap|reset_dirty|get_dirty_log|AtomicBitmap|dump_dirty" \
  src/vmm/src/vstate/vm.rs src/vmm/src/vstate/memory.rs | head

The flow: with tracking on, KVM and Firecracker's per-region AtomicBitmap record which guest pages were written. A diff snapshot dumps only the dirty pages plus the state file; later, rebase-snap merges a diff onto its base to reconstruct a full memory file. The trade-off is direct:

The field rename enable_diff_snapshots → track_dirty_pages (in the CHANGELOG) is itself a signal: the subsystem is being hardened, and the gap between "developer preview" and GA is exactly the kind of area to own.

Restoring untrusted state: the security tension

Here is the design tension that makes snapshotting genuinely hard and not just an optimization. Firecracker's whole isolation argument rests on a small, audited attack surface. But /snapshot/load accepts a state file and a memory file and reconstructs an entire machine from them — vCPU register state, device queue positions, MSRs, the lot. If those files are attacker-controlled, you are feeding untrusted input straight into the most privileged reconstruction path in the VMM.

The honest summary: snapshot restore is the one place where the minimal-attack- surface argument is under the most pressure, because it is a large, complex, input-driven reconstruction. This is why snapshot PRs are reviewed so carefully and why fuzzing the restore path (docs/fuzzing.md) is valued work. If you want a domain where security and performance are in maximum tension, this is it.

# Fuzz targets and Kani proofs touching the snapshot/restore path.
rg -rln "fuzz" tests/ docs/fuzzing.md | head
rg -rln "kani::proof" src/vmm/src/ | head

The design tensions, summarized

A contributor who can hold all five of these in their head at once, and argue a PR against them, is operating at maintainer level.

Where to contribute

gh issue list --repo firecracker-microvm/firecracker \
  --search "snapshot in:title,body state:open" --limit 40
gh issue list --repo firecracker-microvm/firecracker \
  --search "uffd OR diff-snapshot OR track_dirty in:title,body state:open"

Concrete on-ramps: cross-version restore tests (close compatibility gaps); UFFD handler robustness and the example handler; diff-snapshot edge cases on the road to GA; fuzzing the restore path; documenting clone-safety pitfalls.

Next: density is the other half of the serverless economics — Oversubscription & Density. Or go deep on the hands-on side in the snapshotting masterclass.