Project 5: A CPU Template for a New Host Generation

A microVM that boots on one host and is snapshotted, then resumed on a different host, can crash, mis-detect features, or silently corrupt a guest if the two CPUs disagree about what the guest sees. The guest kernel and the language runtimes inside it probe the CPU once — at boot, via CPUID and a handful of model-specific registers (MSRs) — and they assume the answer is stable forever. KVM, left alone, passes most of the host CPU's identity straight through. So a guest that booted on an Intel Skylake host, learned it had AVX-512, and is then resumed on an Intel Cascade Lake (or worse, a host where AVX-512 is fused off) finds the world has changed underneath it. The fix is a CPU template: a declarative description of exactly which CPUID leaves and MSRs the guest is allowed to see, applied identically on every host in a fleet, so the guest's view of the CPU is normalized and snapshots are portable.

This project asks you to build and validate a CPU template for a host generation that Firecracker does not yet ship a good template for — or to materially improve cpu-template-helper, the in-tree tool for creating, inspecting, and verifying templates — and to prove portability: a microVM snapshotted under your template on host A resumes correctly on host B, with the guest seeing an identical CPU on both. The deliverable is a validated template plus a portability proof, and (the upstreamable half) a fix or improvement to the helper tooling or the normalization logic.

Note: Read the CPU templates and CPUID deep dive in full and do the KVM & vCPUs masterclass, especially Lab 3: CPUID and MSRs and Lab 4.3: a CPUID template. This brief assumes you already know what KVM_GET_SUPPORTED_CPUID / KVM_SET_CPUID2 do, what a CPUID leaf/subleaf and an MSR are, the difference between a static built-in template and a custom template applied via PUT /cpu-config, and why feature masking is a one-way street (you can hide a feature, you cannot conjure one the hardware lacks). If those are fuzzy, do the masterclass first — this brief will not re-derive CPUID.

Problem & motivation

Firecracker's threat-and-fleet model makes CPU normalization a first-class concern, for two reasons that compound:

• Snapshot portability. AWS Lambda and Fargate snapshot a warmed microVM and restore it later, potentially on a different physical host with a different CPU stepping or even a different microarchitecture within the same vendor family. If the guest's CPUID/MSR view differs between create-host and restore-host, the guest can fault on an instruction it thinks is available, mis-size a feature buffer (XSAVE area), or take a wrong runtime code path. The template is the contract that makes "snapshot here, restore there" safe: every host presents the same normalized CPU.
• Heterogeneous fleets. A real fleet has multiple CPU generations in service at once. Without templates, the lowest-common-denominator guest is whatever host it happened to land on. With a template, an operator pins the guest to a deliberately chosen feature set — typically the intersection that every host in the migration pool supports — so a workload behaves identically regardless of which host KVM scheduled it onto.

Firecracker ships some static templates (e.g. for specific Intel/AMD generations — verify which on your branch) and supports custom templates via /cpu-config. But host generations keep arriving, the masking rules for new CPUID leaves and new MSRs are fiddly, and the tooling to verify that a template actually produces an identical guest view across two hosts is exactly the kind of thing a sharp contributor improves. The normalization is also security-relevant: hiding a speculative-execution-related feature or a leaky MSR from the guest is part of the threat model, not just a portability nicety.

The motivation is leverage at the seam between KVM and the fleet: a correct template is the difference between "snapshots are portable" and "snapshots are a roulette wheel," and the verification tooling is what lets an operator trust a template before they bet a fleet on it.

What you'll build

Pick one of two tracks (they share most of the prerequisites and the validation discipline):

Both tracks produce: the template (or tooling change), a portability proof (a guest sees an identical CPU on two different hosts under your template), unit tests for any normalization logic you touch, an integration test, and a write-up of the feature-masking decisions and what they cost.

Prerequisites

• Level 4 (KVM, vCPUs, the run loop), especially Lab 4.3: a CPUID template.
• Level 9 and Lab 9.2: snapshot/restore — portability is a snapshot property.
• The CPU templates and CPUID deep dive and the KVM fundamentals deep dive.
• The KVM & vCPUs masterclass, all three labs.
• The snapshotting masterclass, especially Lab 3: snapshot compatibility — templates are the mechanism behind cross-host snapshot compat.
• Hardware reality check: you need access to two physically different CPUs (ideally two host generations from the same vendor) to prove portability. Two distinct bare-metal instance types in a cloud, or two machines, will do. Plan this before you start — it is the gating resource for the whole project.

Phased plan

Phase 0 — Build, dump your host CPU, and map the template path (1–2 days)

Start by making the abstract concrete: dump what your host CPU actually exposes, then trace how Firecracker turns a template into the KVM_SET_CPUID2/MSR-set calls a vCPU applies at boot.

tools/devtool build --release
# The CPU template tooling and config code — read it by role, never by line number:
rg -n "cpu_template|CpuTemplate|CustomCpuTemplate|StaticCpuTemplate|cpu-config" src/vmm/src/
rg -n "CpuidLeaf|CpuidModifier|RegisterModifier|MsrModifier|normalize|apply" src/vmm/src/cpu_config/
# The helper crate — its subcommands (dump / strip / verify, names vary, verify):
ls src/cpu-template-helper/ && rg -n "fn main|Subcommand|dump|strip|verify|Command" src/cpu-template-helper/src/
# Where a vCPU actually applies CPUID/MSRs at boot:
rg -n "set_cpuid2|SET_CPUID2|set_msrs|SET_MSRS|GET_SUPPORTED_CPUID|configure" src/vmm/src/vstate/vcpu/

Then dump the host:

# Dump this host's CPU config in the helper's template format (subcommand name — verify):
./build/cargo_target/*/release/cpu-template-helper dump --output host-A.json
# Inspect a few leaves by hand to anchor your understanding:
rg -n "leaf|0x1\b|0x7\b|0xd\b|msr|0x" host-A.json | head

Anti-staleness: the cpu-template-helper subcommands, the JSON schema of a custom template, the set of shipped static templates, and the module layout under src/vmm/src/cpu_config/ all move between releases. Confirm the current subcommands (--help), the current /cpu-config body shape in src/firecracker/swagger/firecracker.yaml, and read docs/cpu_templates/ (path — verify) on your branch. Check git log --oneline -- src/vmm/src/cpu_config/ src/cpu-template-helper/ for recent churn, and gh issue list --repo firecracker-microvm/firecracker --search "cpu template OR cpuid" for live work you might be duplicating.

Produce capstone-work/template-path.md: the trace from PUT /cpu-config (or a static template selection in machine-config) → the template's CPUID/MSR modifiers → KVM_GET_SUPPORTED_CPUID (the host's ceiling) → modifier application → KVM_SET_CPUID2 + MSR set on each vCPU at boot. Cite the code.

Phase 1 — Define the target and the masking rationale

Decide the exact CPU view you want the guest to see, and write down why for every non-obvious leaf. This is the design half, and it is where templates go wrong.

# Dump host A and host B (the two CPUs you will prove portability across):
# (on host A) cpu-template-helper dump --output host-A.json
# (on host B) cpu-template-helper dump --output host-B.json
# The intersection of what both hosts support is your portable ceiling:
diff <(jq -S . host-A.json) <(jq -S . host-B.json) | less   # see where they disagree

The discipline:

Produce capstone-work/template-design.md: the target feature set, the host-A/host-B intersection that justifies it, and a per-leaf/per-MSR rationale for every modifier — the document you would attach to the PR or issue.

Phase 2 — Build the template (Track A) or the helper change (Track B)

Track A: author the template JSON (CPUID leaf/subleaf modifiers and MSR modifiers), apply it via PUT /cpu-config pre-boot, and boot a microVM under it.

# Apply a custom template pre-boot, then boot (shape — verify against swagger.yaml):
curl -X PUT --unix-socket "$API" --data @my-template.json http://localhost/cpu-config
# ... boot-source, drives, machine-config, then InstanceStart ...
# Inside the guest, read back what the CPU now claims:
#   cat /proc/cpuinfo ; cpuid -1 (the cpuid tool) ; lscpu

Milestone 1: a microVM boots under your template and the guest's cpuid//proc/cpuinfo view matches your target exactly — every feature you masked is gone, nothing you didn't intend to touch changed.

Track B: implement the helper improvement (minimum diff, behind the existing cpu-template-helper structure), with unit tests for the normalization/diff logic.

# Run the helper's own tests, then add yours:
cargo test -p cpu-template-helper
cargo clippy --all --all-targets --all-features -- -D warnings

Phase 3 — Prove portability across two hosts (the headline)

This is the proof that makes the project credible. The claim is: under my template, a guest sees an identical CPU on host A and host B, and a snapshot crosses between them.

                       host A                          host B
                  (e.g. Skylake)                  (e.g. Cascade Lake)
  boot under template ──► guest cpuid view ───────► must be IDENTICAL ◄─── boot under template
        │                                                                        │
        ▼ snapshot/create                                              ▲ snapshot/load + resume
  state.file + mem.file ───────────── copy ──────────────────────────► guest resumes correctly

The two tests:

1. Identical-view test. Boot a microVM under your template on host A and on host B. Capture the guest's CPUID/MSR view on each (a guest-side dump of cpuid raw leaves + the MSRs the guest can read). Assert byte-for-byte equality of the parts the template controls.
2. Cross-host snapshot test. Snapshot a microVM on host A (PATCH /vm {Paused} → PUT /snapshot/create), copy the state+memory files to host B, PUT /snapshot/load + resume on host B, and assert the guest resumes and keeps running correctly (a workload marker, a clean dmesg, no #UD/illegal-instruction faults).

Tip: Without a template, run the same cross-host snapshot to show the failure mode first — the guest mis-detecting the CPU, or an illegal-instruction fault, or a feature flag flipping. Demonstrating the bug your template fixes is exactly the "test that fails without the change" discipline a reviewer wants (code style & trust).

Phase 4 — Tests, validation, and the write-up

• Unit tests for any normalization/modifier/diff code you touched (cargo test).
• A pytest integration test that boots under your template and asserts the guest CPU view (model the assertion on the existing CPU-template tests — rg -n "cpu_template|cpuid|def test_" tests/integration_tests/functional/ | head).
• The validation report: tools/devtool checkstyle + checkbuild --all output, the test commands, and — crucially for this project — the exact CPUs (vendor, family, model, stepping, microcode) of host A and host B.

Key code areas

aarch64 note: there is no CPUID; the equivalent normalization is over system registers (MIDR, feature ID registers) and the GIC. If you take aarch64, the template is a set of register modifiers — confirm the helper supports it on your branch before committing to it.

Design considerations & trade-offs

• Masking is one-way. You can hide a feature the host has; you cannot present one it lacks. Every target must be a subset of the intersection of every restore host. Get this wrong and the guest faults on restore — the exact failure templates exist to prevent.
• Intersection vs. richness. A template that targets the lowest common denominator is maximally portable but throws away performance features on newer hosts. A richer template shrinks the migration pool. State which trade-off you chose and why; there is no free lunch here, and naming the cost is a maintainer signal.
• Stability is the whole point. A template's value is that the guest's CPU view never changes across hosts and across restore. A template that's "mostly stable" is worse than none, because it hides the failure until production. Your portability proof must be exact, not approximate.
• Security masking is deliberate. Some leaves/MSRs are hidden for isolation reasons, not portability. Don't pass a feature through just because the host has it; check the deep dive and the threat model for what FC deliberately suppresses.
• Static vs. custom. A static template is baked into the binary and selected by name; a custom template is data applied at runtime. A new static template is a code change with a higher bar (it ships to everyone); a custom template is configuration. Scope Track A as a custom template first; propose a static one only with maintainer buy-in.
• Snapshot version interaction. The vCPU state in a snapshot includes the CPUID/MSR values. A template change can interact with snapshot compatibility across FC versions — verify against snapshot compat.

How to test & validate

• Correctness (single host): boot under the template; the guest's cpuid//proc/cpuinfo matches the target exactly — nothing masked is present, nothing unintended changed.
• Portability (two hosts): the identical-view test (byte-for-byte guest CPU view on host A and host B) and the cross-host snapshot test (snapshot on A, resume on B, guest runs clean). This is the deliverable's spine.
• Unit: cargo test for modifier-application / diff / normalization logic you touched.
• Integration: a pytest test under tests/integration_tests/functional/ that boots under the template and asserts the guest view; model it on the existing CPU-template tests.
• Negative control: show the failure without the template (the guest mis-detects or faults across hosts), so the value is undeniable.

Stretch goals

• A cpu-template-helper verify enhancement that, given two host dumps and a template, reports whether the template is applicable on both (i.e. every targeted feature is a subset of each host) before you ever boot — failing fast on an impossible target.
• aarch64 register template for a host where FC's coverage is thin — a genuinely under-served area.
• An intersection tool: given N host dumps, emit the maximal portable template (the intersection), so an operator can derive a fleet-wide template mechanically.
• A CHANGELOG-worthy normalization fix: if Phase 0 surfaces a leaf or MSR that FC normalizes incorrectly (a real bug), that fix is a clean, mergeable Firecracker PR on its own.

What a strong deliverable looks like

A strong deliverable is a validated template (or a helper improvement) with a reproducible cross-host portability proof: a guest that sees a byte-for-byte identical CPU on two different physical CPUs under your template, and a snapshot that crosses between them and resumes clean — with the masking rationale documented per leaf/MSR and the exact CPUs stated.

The upstreaming path:

1. The helper and the normalization logic are the mergeable surface. A new static template ships to everyone and needs maintainer buy-in and an issue first; a cpu-template-helper improvement (a better verify, a correctly handled leaf/MSR, an aarch64 gap closed) or a fix to a mis-normalized leaf is mergeable on its own merits. Check the live state: gh issue list --repo firecracker-microvm/firecracker --search "cpu template OR cpuid OR cpu-template-helper" and read docs/cpu_templates/.
2. Comment before you code. CPUID/MSR normalization is snapshot-compatibility-sensitive; confirm the maintainers want the change and agree on the target before you write the modifiers.
3. Bring the proof. A portability claim with a two-host, byte-for-byte identical guest view and a cross-host snapshot resume is exactly what gets a template PR taken seriously — numbers and hosts stated, not "it worked for me."

Even if no template lands upstream, a validated template plus a portability proof and a findings write-up is a portfolio-grade artifact: it demonstrates you understand the seam between KVM, the guest's CPU model, and fleet-scale snapshot portability — one of the hardest correctness problems in the codebase. A finished version at 90+ on the rubric is maintainer-grade CPU/KVM work.

Next: Project 6 — a boot-time / density benchmark harness to turn from correctness to measurement, or back to the portfolio overview.