Compatibility, Stability, Performance

A maintainer's hardest reviews are not about whether your code is correct — they are about what your correct code is allowed to break. Firecracker is the substrate under AWS Lambda and Fargate and a public, versioned product that orchestrators and operators depend on. Three contracts are sacred: the API backward-compatibility contract, the snapshot version compatibility guarantee, and the performance envelope (boot time, memory overhead) that is the product's entire reason to exist. A change that quietly breaks any of them is a change that breaks production for people who trusted a version number.

This chapter is the compatibility lens you must apply before you write a change, not after a reviewer asks. It pairs with the snapshotting deep dive and the Level 9 performance-regression lab. Internalize it and you will flag your own compatibility risks before a maintainer does — which, more than any other single habit, is what separates a contributor from someone on the maintainer track.

The Three Compatibility Surfaces

Before any change, ask which of these it touches. If the answer is "none," you have an ordinary PR. If it touches any, you have a compatibility-sensitive PR and the rest of this chapter applies.

Plus the implicit fourth, SemVer / release policy: which of the above you may change, and in which kind of release, is governed by Firecracker's versioning and support policy. Read it:

cd ~/fc-src
ls docs/                                    # find the API-change runbook and snapshot/version docs
rg -n -i "semver|backward|compat|support|deprecat|breaking" docs/ README.md CHANGELOG.md | head

The API Backward-Compatibility Contract

Firecracker's REST API is a stable, versioned contract. Orchestrators — firecracker-containerd, the Go SDK, Kata, in-house controllers — encode exact request shapes against a version. The governing rule, enforced by the maintainers via the API-change runbook in docs/, is: evolve the API additively; do not break existing clients.

# The API contract itself:
sed -n '1,40p' src/firecracker/swagger/firecracker.yaml
# The runbook that says how it may change:
rg -ril "api.*change|backward|compat" docs/

What "additive" means in practice:

The enable_diff_snapshots → track_dirty_pages rename and the deprecation of the standalone mem_file_path (in favor of mem_backend) are textbook examples of the right way to change the API: the old name is deprecated (kept working, documented as going away, noted in the CHANGELOG) rather than ripped out, giving clients a migration window. When you must change API behavior, the pattern is add the new, deprecate the old, remove only after a documented window — and every step of it lands in CHANGELOG.md.

Warning: Tightening validation feels safe — you're rejecting "bad" input — but it is a breaking change if any client was sending input you now reject. The runbook treats it as one. If you need to reject something, consider whether it can be a warning first, and flag the break loudly in the PR.

Snapshot Version Compatibility

Snapshots are the deepest compatibility trap in Firecracker, because a snapshot is a serialized dump of internal state — KVM register state plus every device's state via the Persist trait — and it is written by one version and read by another, on a different host, possibly months later. Any change to what gets serialized or how can make an old snapshot unrestorable or, worse, restore it silently wrong.

# Every device implements Persist; its state struct IS the on-disk format.
rg -n "impl Persist|trait Persist|struct .*State" src/vmm/src/devices/virtio/ | head
rg -n "version|Version|SnapshotVersion" src/vmm/src/persist.rs src/vmm/src/snapshot/ | head

The discipline:

• A snapshot has two files — the microVM state file (snapshot_path: KVM + device state) and the memory file (mem_file_path/mem_backend: guest RAM). Both are versioned artifacts.
• Changing a Persist state struct changes the format. Adding a field, removing one, reordering — any of these must preserve the ability to load older snapshots that the support policy promises to load. That usually means versioned (de)serialization: read old layouts, write the new one.
• There is a round-trip test, and your change must pass it. A snapshot created on a supported prior version must restore. The cross-version snapshot compatibility tests in tests/ exist exactly to catch a careless Persist change.

# The snapshot-compat integration tests — run them when you touch any device state:
rg -n "snapshot|restore|version" tests/integration_tests/functional/ | grep -i snap | head
tools/devtool test -- tests/integration_tests/functional/test_snapshot_*.py

Note the difference between Full snapshots (GA) and Diff snapshots (developer preview, needs track_dirty_pages and dirty-page tracking): a change touching dirty-page tracking or the diff/base memory layout (rebase-snap) has its own compatibility surface on top of the state format. When in doubt, assume any device-state change is snapshot-affecting and prove otherwise with the round-trip test. The mechanics live in the snapshotting deep dive; the contract is yours to protect.

SemVer and the Release Policy

What you are permitted to break, and where, is governed by Firecracker's versioning and support policy. The shape of it (verify the exact current policy on your branch — it is version-sensitive):

• Breaking changes to the API or snapshot compatibility are gated to major/minor boundaries per the policy, never slipped into a patch release.
• Patch releases carry bug fixes and security fixes and must not break compatibility — which is precisely why CVE fixes (like CVE-2026-5747, fixed in 1.14.4/1.15.1) ship as patches across supported lines.
• Supported versions are a defined window; the support policy states which versions get fixes and which snapshots restore where.

rg -n -i "support|maintenance|release|version" README.md docs/ | grep -i -E "polic|support|window" | head

The practical takeaway: when you propose a breaking change, you are not just changing code — you are asking the maintainers to spend a major/minor version's compatibility budget on it. That budget is scarce. Expect to justify it against the alternative of an additive, non-breaking design, and expect "can this be done additively?" as the first question.

Performance Is a Gate, Not a Nice-to-Have

Firecracker's reason to exist is fast boot and tiny overhead: boots to app code in under ~125 ms, under ~5 MiB memory overhead per microVM, 20x+ oversubscription, thousands per host (NSDI '20). Those are not marketing numbers — they are the product. A change that adds 20 ms to boot or a megabyte of per-microVM overhead can be a regression that gets the PR rejected even if it is otherwise perfect, because at Lambda/Fargate scale that cost is multiplied by millions of microVMs.

# The performance suite that gates regressions:
ls tests/integration_tests/performance/
rg -n "boot.?time|latency|memory|throughput" tests/integration_tests/performance/ | head

If you touch the boot path or the steady-state footprint, run the performance tests and report the numbers in your PR. Volunteering "boot time unchanged within noise — before/after attached" is the performance equivalent of the attack-surface note: it answers the maintainer's question before they ask it. The boot-time-optimization and hugepages-and-memory-performance engineering chapters go deeper on what moves these numbers.

Why the Minimal Device Model Resists Features

Compatibility and minimalism are the same instinct viewed from two angles. Every feature you don't add is a compatibility surface you never have to maintain, an attack surface that never exists, and a performance cost never paid. The minimal device model philosophy is why "Firecracker should support X like QEMU does" is structurally hard to land: X is not free once; it is a permanent tax on every future change.

So when a maintainer resists a feature, they are protecting future compatibility, security, and performance, not being conservative for its own sake. A feature, once shipped, becomes a contract: clients depend on it, snapshots encode it, it sits on the boot/memory budget forever, and removing it later is itself a breaking change. The minimal device model is the project refusing to take on contracts it doesn't have to. Understanding this is what lets you frame a genuinely-needed feature in terms the maintainers can accept: minimal surface, additive, off by default, with a snapshot-compat and performance story already worked out.

What a Breaking Change Actually Costs

When a break is truly unavoidable, count the full cost so you can argue it honestly:

That list is why the maintainers' reflex is "make it additive." Most things that feel like they require a break can be done additively with a new field/endpoint and a deprecation of the old — slower, but it keeps every existing client and snapshot working. Reach for the break only when the additive path is genuinely impossible, and bring the full cost accounting when you do.

Validation: Prove You Understand This

1. Name the three compatibility surfaces and, for a change you might make, state which it touches and why.
2. Classify each as compatible or breaking, with the reason: add an optional field; rename a field; change a default; tighten validation; add a new endpoint.
3. Explain why a change to a device's Persist state struct is a snapshot-compatibility event, and name the test that would catch a careless one.
4. Describe the correct way to "rename" an API field without breaking clients, using the enable_diff_snapshots → track_dirty_pages precedent.
5. Identify a boot-path or memory change and state which performance test you'd run and what you'd report in the PR.
6. Explain, in terms of compatibility/security/performance cost, why the minimal device model resists adding features — and list the full cost of a genuinely-unavoidable breaking change.

You have absorbed this chapter when, before writing any change, you can predict which compatibility surface it touches and what test proves it safe — and you state that in your PR before a reviewer asks. The next chapter — The Path to Maintainership — is how this judgment, sustained over time, turns into trust and ownership.