Lab 3: Snapshot Compatibility & the Format as an API

Background

In Lab 1 you produced a base.state file and treated it as opaque; in Lab 2 you served the memory file by hand. This lab is about the state file — the small, structured serialization of MicrovmState — and the single most important fact about it: it is a public compatibility surface. A microVM snapshotted by Firecracker version X must restore on the versions the project promises it will, and every byte of the serialized layout is part of a contract with every operator who relies on cross-version restore. Break that layout casually and you do not produce a bug report — you produce a fleet of microVMs that will not come back.

This is the area that separates a contributor from a maintainer. Touching a device's emulation logic is reviewed for correctness; touching a device's serialized State struct is reviewed for whether it silently breaks restore for the entire installed base. You will dissect MicrovmState field by field, understand the current serialization stack (serde + bitcode — and why versionize is the historical predecessor you'll still find referenced), enumerate exactly what changes break compatibility and which are safe, use snapshot-editor and rebase-snap as the operator and maintainer tools they are, and internalize the review discipline a snapshot-format PR demands.

This is a review-it / trace-it lab. There is little to "run" in the build sense; there is a great deal to read precisely and reason about correctly, because the cost of being wrong here is borne by people who are not in the room.

Why This Lab Matters for Contributors

Snapshot-compat is a dedicated contributor track (issue-roadmap Stage 8). The maintainers gate format changes harder than almost anything else; a PR that adds a field to a State struct without the right versioning and tests will be sent back.
It is where the minimal device model and compatibility disciplines meet: every new device feature that holds state must serialize it, version it, and prove it restores across the supported window. You cannot add a stateful feature responsibly without this skill.
The judgment — "this change is format-breaking; that one is safe; here is the test that proves it" — is exactly what earns review trust on a single-vendor project with a high bar and two-approval merges. A contributor who reasons crisply about compatibility is one the maintainers can rely on near the format.

Prerequisites

Requirement	Why	Verify
Lab 1 and Lab 2	You have created, inspected, and restored snapshots and know the two files	`ls base.state base.mem`
The snapshotting deep dive, the Persist, MicrovmState, and versioning sections	The trait and the aggregate are the subject	you can write the `Persist` signature from memory
Contributor mindset: compatibility	The general discipline this lab specializes to snapshots	you understand why compatibility is a maintainer concern
Built `snapshot-editor` and `rebase-snap`	The operator/maintainer tools	`ls $B/snapshot-editor $B/rebase-snap`

cd ~/firecracker
B=build/cargo_target/x86_64-unknown-linux-musl/release
test -x $B/snapshot-editor && test -x $B/rebase-snap && echo "tools ready"
rg -q "trait Persist" src/vmm/src/ && echo "Persist present"

Step-by-Step Tasks

Step 1: Dissect `MicrovmState` field by field

The state file is a serialized MicrovmState. Read its definition and the structs it aggregates — do not trust the deep dive's sketch, read the real thing on your branch:

# The aggregate and its members. Path drifts — locate it.
rg -n "struct MicrovmState" src/vmm/src/persist.rs
rg -n "struct MicrovmState" -A 30 src/vmm/src/persist.rs

# What it aggregates: VM-level state, per-vCPU state, device states, memory layout.
rg -n "vm_state|vcpu_states|device_states|memory_state|VmInfo|version" src/vmm/src/persist.rs

Build a map of what is serialized. The shape (names verify on your branch):

Component of `MicrovmState`	Holds	Produced by
`VmInfo` / app version	the Firecracker version that wrote it, machine config	the VMM at create time
VM-level KVM state	irqchip, clock, pit, etc. (arch-specific)	the VM's `save()`
Per-vCPU state	registers, MSRs, CPUID, LAPIC, the KVM vCPU snapshot	each `Vcpu::save()`
Device states	each device's serialized `State` (block, net, vsock, balloon, rng, …)	each device's `Persist::save()`
Memory layout description	the guest memory regions (base, size) — not the bytes	the memory subsystem

Pick one device and read its State struct end to end — this is the unit of compatibility you will reason about for the rest of the lab:

# The block device's serialized state — a concrete State struct to anchor on.
rg -n "struct .*State|impl Persist|fn save|fn restore|Serialize|Deserialize" \
  src/vmm/src/devices/virtio/block/virtio/persist.rs
# Compare with the net device's.
rg -n "struct .*State|impl Persist" src/vmm/src/devices/virtio/net/persist.rs

Notice what is in a device State: queue indices, the device's configured features, rate-limiter bucket state, the device-specific config. Every one of those fields is part of the serialized layout. Reorder them, change a type, or remove one, and a state file written before your change deserializes into garbage after it.

Step 2: Establish the current serialization stack (and the historical one)

This is the most version-sensitive part of the whole curriculum. Verify it on your branch — the project migrated its serialization backend, and reading or citing the wrong one is a real error.

# What serializes MicrovmState today? Look for serde + bitcode (current) and
# versionize (historical predecessor) — see which actually appears.
rg -n "serde|bitcode|bincode|versionize|Versionize|Serialize|Deserialize|Snapshot" src/vmm/src/snapshot/
ls src/vmm/src/snapshot/
# The dependency list is the ground truth — what's actually pulled in?
rg -n "bitcode|versionize|serde|bincode" Cargo.toml src/vmm/Cargo.toml Cargo.lock | head

Era	Mechanism	What it did
Historical	`versionize` / `VersionMap`	A bespoke crate (FC-originated) that serialized structs with an explicit version map: each struct could declare which fields existed at which version, and the serializer translated between versions. Powerful but heavy and FC-specific.
Current (verify)	serde + bitcode	Standard serde derives (`Serialize`/`Deserialize`) with the `bitcode` binary codec. Simpler and faster; compatibility is managed at a higher level (an explicit format/version header and the support window) rather than per-field version maps.

Warning: Do not write "Firecracker uses versionize" in a PR or review comment without checking. It did; the current stack is serde + bitcode (verify on your branch and in the CHANGELOG). Citing the wrong serialization path is the fastest way to look like you haven't read the code — exactly the impression you must avoid near this subsystem.

Read the format/version header handling — the thing that decides whether a state file can even be deserialized by this binary:

rg -n "version|magic|header|SNAPSHOT_VERSION|FC_VERSION|app_version|format" src/vmm/src/snapshot/
sed -n '1,80p' docs/snapshotting/snapshot-support.md   # the support matrix — the contract

Step 3: Enumerate what breaks compatibility — and what is safe

This is the core deliverable: a precise table of which State-struct changes are format-breaking and which are not. Reason about it from how a binary serializer deserializes a struct: it reads fields in order, by type. Anything that changes the order, the types, or the set of fields a given version expects can break.

Change to a serialized `State` struct	Breaks restore?	Why
Reorder existing fields	Yes	A positional/binary codec reads fields in declaration order; reordering makes old bytes deserialize into the wrong fields.
Remove a field	Yes	Old state files still contain its bytes; the new layout no longer expects them — desync.
Change a field's type (e.g. `u32`→`u64`)	Yes	Different width/encoding; old bytes no longer parse as the new type.
Rename a field (same type, same position)	Usually no for binary codecs (positional), but yes for name-keyed formats	Depends on whether the codec is positional or name-tagged — verify your codec.
Add a field at the end	Depends	Safe only if the format/version logic gives a default for old snapshots that lack it; otherwise old files are too short. This is the case that needs explicit versioning + a test.
Add a new optional/`Option` field with a default	Usually the safe path	A defaulted/optional field lets a missing value decode for old snapshots — if the codec supports it. This is the disciplined way to extend a `State`.
Change a default value (not the layout)	No (layout-safe) but semantically risky	The bytes parse, but the meaning may differ across versions — a subtler compat hazard.
Add a whole new device with its own `State`	Depends	New devices restoring on old binaries fail; old snapshots on new binaries must tolerate the device's absence.

Write your own version of this table from the code on your branch, naming the codec and citing where the version header is checked. The table is worthless memorized; it is valuable derived.

Note: The general principle: a serialized struct is an append-with-defaults structure, never a free-edit one. You add fields carefully, with versioning and a default for old snapshots, and you basically never reorder, remove, or retype. Treat a State struct the way you would treat an on-the-wire protocol message — because that is exactly what it is.

Step 4: Use `snapshot-editor` as the operator's window into the format

snapshot-editor is how an operator (and a reviewer) inspects and edits a state file without a running Firecracker. Discover its subcommands — they vary across releases:

$B/snapshot-editor --help
$B/snapshot-editor info-vmstate --help     # subcommand names vary — verify
$B/snapshot-editor edit-memory --help 2>/dev/null

# The format version of a state file — the field that governs cross-version restore.
$B/snapshot-editor info-vmstate version --vmstate-path ./base.state

# Dump the vmstate structure — see the serialized device/vcpu states.
$B/snapshot-editor info-vmstate vm-state --vmstate-path ./base.state | sed -n '1,80p'

Connect tool to format: the version subcommand reads the same header your restore path checks in Step 2; the vm-state dump renders the same MicrovmState you dissected in Step 1. When a restore fails with a version error, this tool is how you confirm which version wrote the file — the first diagnostic, not a guess.

Step 5: Use `rebase-snap` — the Diff-snapshot compatibility tool

Diff snapshots (Lab 1 stretch goal) are a memory-file compatibility mechanism: a diff mem file contains only pages dirtied since a base, and must be merged onto that base before it can be restored as a whole. rebase-snap does that merge:

find src/rebase-snap -name '*.rs'
rg -n "fn main|base|diff|rebase|copy|pwrite|pread" src/rebase-snap/src/

# Produce a Diff snapshot (needs track_dirty_pages at boot), then rebase it onto the base.
# 1) boot with {"track_dirty_pages": true} ; take a Full base.mem ; do work ;
# 2) take a Diff (snapshot_type: Diff) -> a small diff.mem ;
# 3) merge:
$B/rebase-snap --base-file ./base.mem --diff-file ./diff.mem   # flag names vary — verify with --help
# Now base.mem is the merged image; restore from base.state + the rebased base.mem.

The point for this lab: rebase-snap operates on the memory file's compatibility (layering diffs), while the state file's compatibility is the versioned MicrovmState serialization. Two files, two distinct compatibility stories — keep them separate.

Step 6: Walk the review discipline of a snapshot-format PR

Now put on the maintainer's hat. Suppose a PR adds a field to the block device's State to persist a new feature. Walk what review must check — this is the deliverable judgment:

# What the PR touches, and the tests it must add.
rg -rln "snapshot|restore|cross.?version|compat|track_dirty" tests/integration_tests/
rg -n "snapshot|restore|version" tests/integration_tests/functional/test_snapshot*.py 2>/dev/null | head
sed -n '1,60p' docs/snapshotting/snapshot-support.md   # the documented support window

The review checklist a snapshot-format PR must satisfy:

Reviewer asks	Why	What proves it
Is the field added, not reordered/removed/retyped?	Only append-with-default is safe	the diff itself
Does an old snapshot still restore on the new binary?	Backward compat	an integration test that restores a pre-change state file
Does a new snapshot restore where the support window requires it?	Forward/cross-version compat	a cross-version restore test
Is the field defaulted for snapshots that lack it?	Old files are "too short" without a default	the default in code + a test
Is the support matrix / CHANGELOG updated if behavior changed?	The format is a documented contract	the docs diff and a CHANGELOG entry
Did the format version bump if required?	The header is how restore refuses incompatible files	the version handling + a test

Tip: The maintainer's recurring question — and the one this lab trains you to answer — is: "Given this change to a State struct, what must the author do so existing snapshots still restore, and new snapshots restore where the window requires it, and what tests prove it?" If you can answer that with the table above and a pointed rg at the integration tests, you are reviewing this subsystem at the level the project needs.

Implementation Requirements / Deliverables

A field-by-field map of MicrovmState and at least one device State struct, read from your branch, with each field's role.
A one-paragraph statement of the current serialization stack (serde + bitcode — verified on your branch) and how it differs from the historical versionize, with the version-header check located in the source.
Your own compatibility table: at least five specific State-struct changes classified as breaking or safe, each with the reason derived from how the codec deserializes.
snapshot-editor output showing a state file's version and a vmstate dump, tied back to the format header and MicrovmState.
A rebase-snap run (or a precise reading of its source) explaining that it is a memory-file compatibility tool, distinct from state-file versioning.
The review checklist applied, in writing, to one hypothetical "add a field to a device State" PR — what tests and docs it must include.

Troubleshooting

`snapshot-editor` subcommand not found / different name

Subcommands change across releases. Run $B/snapshot-editor --help and the per-command --help to discover the current names; do not copy them from this lab.

You can't tell whether the codec is positional or name-tagged

Read the codec, not the struct. rg -n "bitcode|serde|Serialize|Deserialize" src/vmm/src/snapshot/ and check the codec's docs: a binary positional codec (like bitcode) is field-order-sensitive; a self-describing/name-tagged format is not. This determines whether renames and reorders are safe — get it right before you classify any change.

A restore "works" in your dev test but you're unsure it's really compatible

Same-version restore is not a compat test. To test compatibility you must restore a state file written by a different (older) version on the new binary, and vice versa where the window requires it. The integration suite does this with pinned artifacts — rg -rln "snapshot" tests/integration_tests/ and read how it pins versions.

`rebase-snap` flags don't match this lab

The CLI flags vary — $B/rebase-snap --help. The concept is fixed (merge a diff mem file onto a base mem file); the exact flag names are not.

Expected Output

$ $B/snapshot-editor info-vmstate version --vmstate-path ./base.state
v3.0.0                       # the format/version header your restore path checks

$ rg -n "bitcode|versionize" src/vmm/src/snapshot/ Cargo.lock | head
src/vmm/src/snapshot/...:  ... bitcode ...        # current codec (verify)
# (versionize, if present at all, is historical / a dev dependency)

# Your compatibility verdict (excerpt):
reorder fields in BlockState     -> BREAKS  (positional codec reads in order)
add Option<u32> at end + default -> SAFE    (old snapshots decode the missing field)
change queue_index u16 -> u32    -> BREAKS  (width change; old bytes mis-parse)

Stretch Goals

Break it on purpose, observe the failure. On a throwaway branch, reorder two fields in a device State, rebuild, and try to restore a base.state written by the unmodified binary. Capture the exact error. Then fix it the right way (append
- default) and show the restore succeeds. This is the most instructive single exercise in the lab — do it.
Trace the version header end to end. Follow the format version from where it is written at create time to where it is checked at load time, and explain what happens when they mismatch. rg -n "version" src/vmm/src/snapshot/ src/vmm/src/persist.rs.
Read a real format-change PR. gh pr list --repo firecracker-microvm/firecracker --search "snapshot version compat" --state merged — find one that changed a State struct and study how the author handled versioning, tests, and the CHANGELOG.
Map the support window. From docs/snapshotting/snapshot-support.md, write the exact promise: which versions restore which snapshots, and what an operator can and cannot rely on across an upgrade.
Diff vs Full at the format level. Explain precisely what differs between a Diff and a Full snapshot's state file (hint: very little — the difference is in the memory file and the track_dirty_pages prerequisite). Where does rebase-snap fit and where does it not?

Validation / Self-check

Answer without notes. These gate completion.

What does the state file serialize, and name four components of MicrovmState.
What is the current serialization stack, and what is the historical one you'll still see referenced? How do you check which is in use?
Classify these and justify each: reorder fields; remove a field; add an Option<u32> at the end with a default; change a u16 to a u64.
What does the format version header do, and what tool reads it without a running Firecracker?
What is rebase-snap for, and why is it a memory-file tool rather than a state-file one?
A PR adds a field to a device State to persist a new feature. List the review checks and the tests it must include.
Why is the snapshot format treated as a public API rather than an implementation detail, and who pays when it breaks?

When you can dissect MicrovmState, name the codec correctly, classify any State-struct change as breaking or safe with the reason, and walk the review discipline of a format-change PR, you've completed the snapshotting masterclass — and you are equipped to contribute to the subsystem that gates maintainership.

Next: take the trust-boundary thread further into the security masterclass — restore is a code-execution surface, and the jailer/seccomp/threat-model intensive is where you audit it. Or carry the performance thread into the performance & density masterclass. Both deepen Level 9 and the Stage 8 snapshot-compat issue track.

Open-Source Engineer & Contributor