Stage 8 — Snapshot Compatibility

What class of issue this is

Stage 8 is about freezing a running microVM and bringing it back to life — possibly on a different Firecracker binary than the one that saved it. A snapshot is two files: the microVM state file (KVM + device state) and the memory file (guest RAM). Every device, the vCPU/KVM state, and the machine config implement the Persist trait to serialize themselves into the aggregate MicrovmState, which is versioned so a newer binary can restore an older snapshot (and, within policy, vice versa). The bugs here are the subtlest in the codebase because they are invisible until restore: a Persist impl that drops a field, a MicrovmState version bump that isn't handled on the restore path, a restore edge case (a device that was mid-I/O, a feature negotiated before save), a cross-version mismatch that silently restores wrong state.

Concretely, a Stage 8 PR is one of:

• A Persist impl bug: a field added to a device's runtime state but not to its persisted state (or vice versa), so restore loses or corrupts it.
• A MicrovmState/state-file versioning issue: a version bump without the corresponding restore-side handling, or a field that needs version-gated (de)serialization.
• A restore edge case: in-flight virtqueue state, a device feature/status that must be re-established on restore, a memory-backend (File vs Uffd) handling bug.
• A cross-version compatibility bug: a snapshot from version X fails to restore (or mis-restores) on version Y in a way the support policy says should work.

Why it's at this difficulty

Snapshot compatibility is a wire-format contract that AWS customers depend on across upgrades — a silent restore bug is data corruption in production, the worst class of defect. You must understand every device's state, the serialization format, the version-negotiation logic, and the support policy (which versions are expected to interoperate). It is the deepest application of the compatibility mindset. Maps to Level 9; the snapshotting deep dive and the compatibility chapter are required reading.

What you must already understand

• The Persist trait and MicrovmState. Find the trait, the aggregate state, and the save/restore entry points:

rg -n "trait Persist|fn save|fn restore|struct MicrovmState|impl Persist" \
  src/vmm/src/persist.rs src/vmm/src/devices/ | head
rg -n "struct .*State\b|#\[derive\(.*Serialize|Versionize|serde" src/vmm/src/persist.rs | head

• The snapshot format and versioning. Firecracker historically used a Versionize-based format; the versioning approach has evolved — check what your branch uses (serde/bincode + an explicit version field, or Versionize):

ls src/vmm/src/snapshot/
rg -n "version|SnapshotHdr|magic|SNAPSHOT_VERSION|app_version|Versionize|deserialize" \
  src/vmm/src/snapshot/ | head

• The workflow. PATCH /vm {state:Paused} → PUT /snapshot/create → PATCH /vm {state:Resumed}; load via PUT /snapshot/load with mem_backend{backend_path, backend_type: File|Uffd} and resume_vm. (Field renamed enable_diff_snapshots→track_dirty_pages; standalone mem_file_path on load deprecated.)

rg -n "snapshot/create|snapshot/load|create_snapshot|restore_from_snapshot|mem_backend|Uffd" \
  src/vmm/src/rpc_interface.rs src/vmm/src/persist.rs | head

• The tooling. snapshot-editor and rebase-snap inspect/edit/rebase snapshots — useful for reproducing and for understanding the format:

ls src/snapshot-editor/ src/rebase-snap/

Representative tasks

How to approach one — worked example: a Persist impl that drops a field

Illustrative of the pattern. The rg finds the real device state and impl; do not trust paths — the crate merge moved persistence code.

Symptom: a field was added to a device's runtime struct (say a block device's io_engine-related state, or a net device's RX-buffer accounting) but the device's persisted state (*State) and restore were not updated, so after restore the device starts with a default value and misbehaves (a stuck queue, wrong throttling, a dropped in-flight request).

Step 1 — read the save and restore halves together

A Persist impl is two mirrored functions; bugs are always an asymmetry between them.

rg -n "impl Persist for Block|struct BlockState|fn save|fn restore" \
  src/vmm/src/devices/virtio/block/persist.rs
git log --oneline -n 8 -- src/vmm/src/devices/virtio/block/persist.rs

#![allow(unused)]
fn main() {
// save: device -> state
fn save(&self) -> BlockState {
    BlockState {
        // ... fields ...
        // BUG: a field that exists on the device is not copied into BlockState
    }
}
// restore: state -> device  (must be the exact inverse of save)
fn restore(constructor_args: ..., state: &BlockState) -> Result<Self, ...> {
    // ... rebuilds the device from state ...
}
}

Step 2 — open the discussion FIRST, then fix both halves and the version

Snapshot changes always start with a maintainer conversation — the format is a contract. Adding a field to a persisted state struct is a format change: it needs a version bump and version-gated handling so an old snapshot (without the field) still restores. Post your plan, then:

--- a/src/vmm/src/devices/virtio/block/persist.rs
+++ b/src/vmm/src/devices/virtio/block/persist.rs
@@  pub struct BlockState {
     // existing fields ...
+    // New in snapshot version N: persist the field that was being dropped.
+    pub pending_request_count: u32,
 }
@@  fn save(&self) -> BlockState {
     BlockState {
         // ...
+        pending_request_count: self.pending_request_count,
     }
 }
@@  fn restore(..., state: &BlockState) -> Result<Self, ...> {
     // ...
+    dev.pending_request_count = state.pending_request_count;

The version-gating side (so an older snapshot that lacks the field deserializes with a default) depends on your branch's mechanism — a serde #[serde(default)], or an explicit version match. Find it and follow the existing pattern exactly:

rg -n "serde\(default\)|SNAPSHOT_VERSION|match .*version|since_version" src/vmm/src/ | head

Step 3 — test save→restore round-trip AND cross-version

The decisive test creates a snapshot, restores it, and asserts the field survived — and, for a format change, that an older-format snapshot still restores.

rg -n "def test_.*snapshot|create_snapshot|restore|build_microvm_from_snapshot" \
  tests/integration_tests/functional/ | head
tools/devtool test -- -k snapshot

# pytest shape (illustrative): snapshot, restore, assert the device state survived.
def test_block_pending_requests_survive_snapshot(microvm_factory, ...):
    vm = microvm_factory.build(...)
    # drive some block I/O so pending_request_count != default
    snapshot = vm.snapshot_full()
    restored = microvm_factory.build_from_snapshot(snapshot)
    # assert the restored device behaves as if the field was preserved

Firecracker also has cross-version snapshot tests that build several prior releases and check restore both directions within the support window. If your change is a format bump, that suite is where it must be exercised — find it:

rg -n "build_versions|get_firecracker_binaries|prev_version|cross.version" tests/ | head

Warning: A snapshot bug is silent until restore, and a restore bug is silent until the guest misbehaves later. There is no margin for "looks right." Round-trip and cross-version tests, plus a maintainer review of the format change, are mandatory — this is why the stage's PRs have long threads and small diffs.

What a good PR looks like

• save and restore are exact inverses, and you proved it with a round-trip test that drives the device into a non-default state before snapshotting.
• Format changes are versioned, with handling so an older snapshot still restores (a default, a version match) — never a silent break of the support window.
• A cross-version test exercises restore across the relevant prior releases when the format changed.
• The discussion happened first. No snapshot PR should surprise a maintainer with a format change.
• No new state is persisted that doesn't need to be, and nothing guest-controlled is trusted on restore any more than at runtime (Stage 7 discipline carries over).
• CHANGELOG entry noting any snapshot-version implication; both functional and cross-version tests.

Graduation criteria — ready for the advanced stages when

• You have one merged snapshot PR — a Persist fix, a versioning fix, or a restore edge case — with a round-trip test and, if the format changed, a cross-version test.
• You can explain the two snapshot files, the Persist trait, how MicrovmState is versioned on your branch, and the save→pause→create→resume / load→resume workflow.
• You can reason about when a change is a wire-format change (needs a version bump + cross-version test) vs an internal-only change, and you open the discussion before touching the format.
• You treat "silent until restore" as the defining risk and test accordingly.

You have now worked the full save/restore contract. The remaining stages are cross-cutting concerns that ride on everything you have learned: Stage 9 (test determinism), Stage 10 (performance), Stage 11 (security), and Stage 12 (release-blocking triage).

Next: Stage 9 — Flaky Test Fixes.