Lab 1: Create, Restore, and Clone

Background

In Lab 9.2 you created and restored a snapshot once, to prove it works. This lab makes the workflow second nature and then pushes past it into the pattern that makes snapshotting matter: restoring many clones from one base snapshot. That is not a toy — it is the literal mechanism behind serverless cold-start mitigation. A fleet keeps one "pre-warmed" snapshot of a booted, initialized function runtime and stamps out fresh, independent microVMs from it in single-digit milliseconds instead of booting each from scratch.

You will: run the full Full-snapshot create flow (PATCH /vm {Paused} → PUT /snapshot/create → PATCH /vm {Resumed}); restore on a fresh firecracker process with mem_backend: File; open both snapshot files with snapshot-editor to see what's actually in them; measure restore-and-resume latency; fan N clones out of one snapshot; and then — the part most tutorials skip — reckon honestly with the security and correctness caveats of restoring shared state. Cloned VMs that share a snapshot share more than they should: the same RNG state, the same secrets in memory, the same MAC and IP. Knowing exactly what's shared is the difference between using snapshots safely and shipping a vulnerability.

This is a trace-it / measure-it lab.

Why This Lab Matters for Contributors

• The create/restore API is a public surface; contributors who touch it must know its exact semantics (which fields are pre-boot-only, what resume_vm does, why restore needs a fresh process). You can't review an API change to /snapshot/* you can't drive flawlessly.
• The clone-many pattern exposes the COW model (snapshotting deep dive) at its most concrete: N processes mmap the same memory file MAP_PRIVATE and diverge on write. This is the substrate Lab 2's UFFD handler replaces.
• The security caveats are not academic. "Restored clones share entropy/secrets" is a real, repeatedly-rediscovered class of issue. A contributor who internalizes it writes safer code and reviews PRs with the right suspicion.

Prerequisites

• Lab 9.2 and the snapshotting deep dive.
• A built firecracker and snapshot-editor, a CI vmlinux and rootfs, and a working manual boot (Lab 1.3).

cd ~/firecracker
B=build/cargo_target/x86_64-unknown-linux-musl/release
test -x $B/firecracker && test -x $B/snapshot-editor && echo "binaries ready"
ls vmlinux-* *.ext4 2>/dev/null   # you need a kernel + rootfs

Step-by-Step Tasks

Step 1: Boot a base microVM and put a marker in it

Snapshots are most instructive when you can see state survive. Boot a microVM, then write a marker inside the guest so you can prove the restored clone resumes from the same point.

API=/tmp/fc-base.sock
rm -f $API
sudo $B/firecracker --api-sock $API &
FCPID=$!

curl -sX PUT --unix-socket $API \
 --data '{"kernel_image_path":"./vmlinux-6.1.x","boot_args":"console=ttyS0 reboot=k panic=1 nomodule"}' \
 http://localhost/boot-source
curl -sX PUT --unix-socket $API \
 --data '{"drive_id":"rootfs","path_on_host":"./ubuntu-24.04.ext4","is_root_device":true,"is_read_only":false}' \
 http://localhost/drives/rootfs
curl -sX PUT --unix-socket $API \
 --data '{"vcpu_count":1,"mem_size_mib":256}' http://localhost/machine-config
curl -sX PUT --unix-socket $API \
 --data '{"action_type":"InstanceStart"}' http://localhost/actions

In the guest serial console, leave a marker — e.g. a process counting seconds, or just note the uptime. The point is that the restored clone will continue this guest's runtime, not a fresh boot.

# in the guest:
root@guest:~# date +%s > /run/marker ; uptime

Step 2: Pause, create the snapshot, resume

The create flow is three calls. The VM must be paused first — a running VM's state is inconsistent (vCPUs mid-instruction, device queues in flight).

# 1. Pause: vCPUs stop, device state quiesces.
curl -sX PATCH --unix-socket $API --data '{"state":"Paused"}' http://localhost/vm

# 2. Create: write the state file and the memory file.
curl -sX PUT --unix-socket $API --data \
 '{"snapshot_type":"Full","snapshot_path":"./base.state","mem_file_path":"./base.mem"}' \
 http://localhost/snapshot/create

# 3. Resume the original (or kill it — you only needed the snapshot).
curl -sX PATCH --unix-socket $API --data '{"state":"Resumed"}' http://localhost/vm

ls -lh base.state base.mem

You now have two files. The base.state file is small and structured (serialized machine + device + vCPU state); base.mem is the size of guest RAM (256 MiB here).

# Done with the base VM:
curl -sX PUT --unix-socket $API --data '{"action_type":"SendCtrlAltDel"}' http://localhost/actions 2>/dev/null
kill $FCPID 2>/dev/null

Step 3: Inspect both files with snapshot-editor

Do not treat the snapshot as opaque. snapshot-editor reads the state file's structure and version. Subcommands vary across releases — discover them:

$B/snapshot-editor --help
$B/snapshot-editor info-vmstate --help    # subcommand names vary — verify

# The snapshot format version (THE field that governs cross-version restore):
$B/snapshot-editor info-vmstate version --vmstate-path ./base.state

# Dump the vmstate structure (vcpu state, device states, memory layout):
$B/snapshot-editor info-vmstate vm-state --vmstate-path ./base.state | sed -n '1,60p'

# The memory file is a flat dump. Its size should equal guest RAM (Full snapshot):
stat -c '%s' base.mem      # 256 MiB = 268435456 bytes
# Diff snapshots would be smaller and need a base — covered in Lab 3 / rebase-snap.

Note what's in the state file: the per-vCPU register/MSR/CPUID state, each device's serialized State, the memory-region description. This is the MicrovmState you'll dissect in Lab 3. Note what's not in it: the guest RAM itself (that's the mem file) and anything Firecracker deliberately doesn't snapshot (e.g. MMDS contents — see the snapshotting deep dive).

Step 4: Restore on a fresh firecracker (File backend)

Restore happens in a brand-new Firecracker process with no boot-source and no drives configured — the snapshot carries all of that. A single call rebuilds the machine via build_microvm_from_snapshot:

API2=/tmp/fc-restore.sock
rm -f $API2
sudo $B/firecracker --api-sock $API2 &

curl -sX PUT --unix-socket $API2 --data '{
  "snapshot_path": "./base.state",
  "mem_backend": { "backend_path": "./base.mem", "backend_type": "File" },
  "resume_vm": true
}' http://localhost/snapshot/load

On the restored guest's serial console, confirm continuity — your marker from Step 1 is present, and uptime continues from where the snapshot was taken (it does not reset to zero). That is the proof: this is resume, not reboot.

# in the RESTORED guest:
root@guest:~# cat /run/marker     # the same value you wrote in Step 1
root@guest:~# uptime              # continues from the snapshot point

Note: With the File backend, Firecracker mmaps base.mem MAP_PRIVATE. The restored VM reads pages straight from the file (no upfront copy); only on write does the kernel make a private copy (COW). Restore is fast because RAM faults in on demand, not all at once. Find this in the source: rg -n "MAP_PRIVATE|mmap|backend_type|File" src/vmm/src/.

Step 5: Measure restore-and-resume latency

Time the load call. For a fair number, restore many times and look at the distribution; a single run includes process-start noise.

restore_once() {
  local api=$1
  rm -f $api; sudo $B/firecracker --api-sock $api & local pid=$!
  # wait for the socket
  for i in $(seq 1 100); do [ -S $api ] && break; done
  local t0=$(date +%s%N)
  curl -sX PUT --unix-socket $api --data \
    '{"snapshot_path":"./base.state","mem_backend":{"backend_path":"./base.mem","backend_type":"File"},"resume_vm":true}' \
    http://localhost/snapshot/load >/dev/null
  local t1=$(date +%s%N)
  echo "restore+resume: $(( (t1 - t0) / 1000000 )) ms"
  kill $pid 2>/dev/null
}
for n in $(seq 1 10); do restore_once /tmp/fc-bench-$n.sock; done

Interpret it: the time splits between deserializing the state file (small, structured — fast) and faulting in memory (lazy with the File backend, so the resume is fast but the guest pays page faults as it touches RAM afterward). Restore should be far below a cold boot. Contrast against a from-scratch boot time (engineering/boot-time-optimization).

Tip: Firecracker's own integration tests measure this; the documented target is single-digit-millisecond restore for typical configs. Your numbers depend on RAM size, backend, and host. Record yours and explain the dominant cost.

Step 6: Clone many microVMs from one snapshot

This is the cold-start pattern. The same base.state and base.mem produce N independent microVMs, each in its own Firecracker process, each mmap-ing the mem file MAP_PRIVATE so writes diverge per-clone.

CLONES=5
for n in $(seq 1 $CLONES); do
  api=/tmp/fc-clone-$n.sock
  rm -f $api
  sudo $B/firecracker --api-sock $api &
  for i in $(seq 1 100); do [ -S $api ] && break; done
  curl -sX PUT --unix-socket $api --data \
    '{"snapshot_path":"./base.state","mem_backend":{"backend_path":"./base.mem","backend_type":"File"},"resume_vm":true}' \
    http://localhost/snapshot/load >/dev/null
  echo "clone $n up on $api"
done

All five share one read-only base memory file via COW. Touch RAM in one and the kernel copies just those pages for that clone; the others are unaffected. This is how one snapshot backs thousands of microVMs with minimal incremental memory.

Warning — clones are NOT distinct VMs in every sense. They started from identical state. Proceed to Step 7 before you ever do this with real workloads.

Step 7: Reckon with the security and correctness caveats

Restoring shared state is powerful and dangerous. Write up, concretely, what N clones of one snapshot share and why each is a problem:

The general principle: a snapshot is a copy of a specific moment, and anything that was supposed to be unique-per-VM is now duplicated. And the deeper threat-model point: restore reconstructs memory and device state from files — if those files are attacker-controlled, restore is a code-execution surface in privileged host code. Never restore a snapshot you did not produce or fully trust. This connects directly to the security masterclass and the threat model.

# Demonstrate the entropy problem (in two clones):
# in clone 1 and clone 2 guests, BEFORE any re-seed:
head -c 16 /dev/urandom | xxd      # compare — if identical, you've reproduced the bug

Implementation Requirements / Deliverables

• The full create flow run (Pause→create→Resume), with base.state and base.mem produced and their sizes noted.
• snapshot-editor output: the snapshot version and a dump of the vmstate structure, with a sentence on what's in the state file vs the mem file.
• A successful restore on a fresh process with mem_backend: File, proven by a surviving in-guest marker and a continuing uptime.
• A restore-latency measurement (≥5 runs) with a sentence on where the time goes.
• N (≥3) clones running simultaneously from one snapshot.
• The filled-in shared-state caveats table, plus a demonstration of at least one shared-state problem (e.g. identical /dev/urandom output across clones) and its mitigation.

Troubleshooting

PUT /snapshot/create returns an error about state

The VM wasn't paused. The create flow requires PATCH /vm {Paused} first. Confirm the pause succeeded (200) before creating.

Restore fails: "cannot load snapshot" / deserialization error

The state file was written by an incompatible Firecracker version, or you're pointing at the wrong file. Check the version with $B/snapshot-editor info-vmstate version --vmstate-path ./base.state and ensure the restoring binary is compatible (same or a supported newer version — Lab 3).

Restore succeeds but the guest hangs touching memory

With the File backend this usually means the mem file is wrong/truncated, or the memory layout in the state file doesn't match the mem file. Confirm base.mem is exactly guest-RAM-sized and unmodified. (With the Uffd backend this would mean the handler isn't serving pages — that's Lab 2.)

Restore complains drives/boot-source are configured

You tried to restore in a process that already had a boot-source or drives set. Restore needs a fresh process — the snapshot carries the full configuration.

Clones collide on the network / can't all reach the host

Expected — they share a MAC/IP (Step 7). Assign distinct tap devices and re-configure network identity per clone, or test clones without networking first.

Expected Output

$ ls -lh base.state base.mem
-rw-r--r-- 1 root root  18K base.state
-rw-r--r-- 1 root root 256M base.mem

$ $B/snapshot-editor info-vmstate version --vmstate-path ./base.state
v3.0.0            # (example — your snapshot/format version)

# restored guest:
root@guest:~# cat /run/marker
1718700000        # the same value written before the snapshot
root@guest:~# uptime
 ... up 4 min ... # continues, does not reset

# entropy bug across two un-reseeded clones:
clone1$ head -c 16 /dev/urandom | xxd  -> 3f a1 ... (IDENTICAL to clone2)
clone2$ head -c 16 /dev/urandom | xxd  -> 3f a1 ...

The surviving marker proves resume; the identical /dev/urandom proves the shared-entropy caveat is real.

Stretch Goals

1. Diff snapshots. Boot with track_dirty_pages: true, take a Full base, do work, then take a Diff snapshot (snapshot_type: Diff). Note the diff mem file is far smaller. Use rebase-snap to merge the diff onto the base and restore from the merged result. You'll formalize this in Lab 3.
2. Patched restore (resume_vm: false). Restore with resume_vm: false, then PATCH /network-interfaces/... to fix the network identity before resuming with PATCH /vm {Resumed}. This is the real fix for the MAC/IP caveat — patch the per-VM identity in the paused window.
3. Measure the COW saving. Run N clones and compare summed RSS against N × the memory-file size. The gap is what COW + the shared MAP_PRIVATE file saves. Use ps/smem//proc/<pid>/smaps_rollup.
4. Restore latency vs RAM size. Repeat Step 5's measurement for 128 / 512 / 1024 MiB guests. Plot restore time against RAM. With the File backend, where does the curve go, and why (lazy memory means resume is nearly RAM-independent; the cost moves into post-resume page faults)?

Validation / Self-check

Answer without notes. These gate completion.

1. List the three API calls of the create flow in order. Why must the VM be paused before creating?
2. What is in the state file vs the memory file? Which tool shows you the state file's structure and version?
3. Why must restore happen in a fresh Firecracker process with no boot-source or drives?
4. What does MAP_PRIVATE COW give you when N clones share one memory file, and what diverges per-clone?
5. Name three things N clones of one snapshot share that should be unique-per-VM, and the mitigation for each.
6. Why is restoring an untrusted snapshot a security risk, and what's the rule?
7. With the File backend, where does restore latency actually go, and why is resume nearly independent of RAM size?

When you can run create/restore/clone fluently, inspect both files, measure restore, and explain every shared-state caveat with a demonstration, you've completed Lab 1. Continue to Lab 2 — Build a UFFD Page-Fault Handler, where you replace the File backend's kernel-driven demand paging with a userspace handler you write yourself.