CPU Templates and CPUID

The guest's view of the CPU is not the host's CPU. Firecracker decides exactly which CPU features, instruction-set extensions, and model identifiers the guest sees, by reading the host-supported CPUID leaves from KVM, transforming them, and writing them into the vCPU before boot — plus configuring a set of MSRs. CPU templates are the named, reusable specifications of those transformations. They exist for a concrete operational reason: a fleet of heterogeneous hosts (different CPU steppings, microcode, even vendors) must present a consistent virtual CPU so that a guest — and especially a snapshot taken on one host and restored on another — behaves identically everywhere. This chapter covers the GET_SUPPORTED_CPUID → transform → SET_CPUID2 pipeline, static vs custom templates, the MSR side, and the cpu-template-helper tooling.

After this chapter you will be able to: explain why CPUID must be normalized; trace where Firecracker reads, transforms, and sets CPUID/MSRs; distinguish a static built-in template from a custom one supplied via /cpu-config; and use cpu-template-helper to build and verify a template.

Note: CPUID normalization is a correctness and portability feature, not a performance tweak. A guest that sees feature X on host A and not on host B may execute an X instruction after a snapshot migration and take an illegal-instruction fault. Templates make the virtual CPU a stable contract. This is why snapshot portability and CPU templates are inseparable topics.

The CPUID pipeline

# Where Firecracker reads, transforms, and sets CPUID.
rg -n "get_supported_cpuid|GET_SUPPORTED_CPUID|set_cpuid2|SET_CPUID2|CpuId|cpuid" src/vmm/src/cpu_config/ src/vmm/src/arch/x86_64/ src/vmm/src/vstate/vcpu/

The flow, per vCPU, during boot (step 6 of the-boot-sequence.md):

flowchart LR
    A["Kvm::get_supported_cpuid()"] --> B["host-supported CPUID leaves"]
    B --> C["apply template transform: mask/set bits, fix topology, vendor, leaves"]
    C --> D["normalize per-vCPU: APIC id, topology (leaf 0xB), brand string"]
    D --> E["vcpu.set_cpuid2(cpuid)"]
    E --> F["guest sees this virtual CPU"]

The "transform" stage is where the template lives. Without a template, Firecracker still does a baseline normalization (topology, APIC IDs, disabling features it doesn't support); a template layers an explicit, named policy on top.

rg -n "fn normalize|fn process_cpuid|brand_string|leaf|0xB|topology|APIC" src/vmm/src/cpu_config/ src/vmm/src/arch/x86_64/

MSRs

CPUID is half the story; the guest CPU's behaviour is also shaped by Model-Specific Registers. Firecracker configures a curated MSR set (and a template can adjust it) via KVM_SET_MSRS.

rg -n "set_msrs|SET_MSRS|MSR_|msr|create_msr|supported_msrs" src/vmm/src/cpu_config/ src/vmm/src/arch/x86_64/

MSRs matter for the same portability reason: speculative-execution mitigation controls, TSC behaviour, and feature-enable bits live in MSRs, and a snapshot restored on a host with different MSR support can misbehave if they aren't pinned. Templates therefore cover CPUID and MSRs together as one CPU specification.

Static vs custom templates

There are two kinds of template, and the distinction is exactly where they come from.

rg -n "StaticCpuTemplate|CustomCpuTemplate|enum.*Template|cpu_template|CpuConfiguration" src/vmm/src/cpu_config/ src/vmm/src/vmm_config/

A static template is the common case: you name a known CPU baseline and Firecracker masks the host's CPUID down to that baseline's feature set, so an Intel-X-templated guest looks the same on any host new enough to support it. A custom template (the /cpu-config endpoint) is a JSON document that explicitly lists CPUID leaf/register modifiers and MSR modifiers — full control, used when the built-in baselines don't fit. Both feed the same set_cpuid2/set_msrs calls; the template is just the source of the transform.

Why templates exist (the operational argument)

   Host A (newer CPU)        Host B (older CPU)
   supports AVX-512          no AVX-512
        │                          │
   without a template:        without a template:
   guest sees AVX-512         guest sees no AVX-512
        │                          │
        └──── snapshot on A ───────┘
                    │
        restore on B: guest tries an AVX-512 instruction it "knew" it had
                    │
                    ▼   #UD illegal-instruction fault — guest crashes

With a template pinning the feature set to a common baseline, the guest on both hosts sees the same CPU, the snapshot is portable, and the migration is safe. This is the core reason AWS uses templates at fleet scale: consistent guest CPU across heterogeneous hardware, and snapshot/restore that survives a move to different silicon. The flip side is that a template can only expose features present on every target host — you trade some peak capability for uniformity.

See snapshotting.md for the restore path and why CPU state is part of the snapshot, and the oversubscription and density chapter for the fleet-scale picture.

cpu-template-helper and snapshot-editor

Firecracker ships a dedicated tool for building, inspecting, and verifying templates:

# The crate.
rg -n "cpu-template-helper" Cargo.toml ; ls src/cpu-template-helper 2>/dev/null
rg -n "fn main|dump|verify|template" src/cpu-template-helper/src/

The workflow is: on a representative host, dump to get a baseline; edit it down to the features common across your fleet; verify on each host class; then ship it as a custom template (or rely on a matching static one). This tooling is how you avoid the snapshot-migration #UD trap empirically rather than by guesswork.

Reading exercise

# 1. The read→transform→set pipeline.
rg -n "get_supported_cpuid|set_cpuid2|normalize|process_cpuid" src/vmm/src/cpu_config/ src/vmm/src/arch/x86_64/

# 2. MSR configuration.
rg -n "set_msrs|MSR_|create_msr|supported_msrs" src/vmm/src/cpu_config/ src/vmm/src/arch/x86_64/

# 3. Static vs custom template types.
rg -n "StaticCpuTemplate|CustomCpuTemplate|CpuConfiguration|cpu_template" src/vmm/src/cpu_config/ src/vmm/src/vmm_config/

# 4. The /cpu-config endpoint.
rg -n "cpu-config|cpu_config|CpuConfig" src/firecracker/src/api_server/ src/firecracker/swagger/firecracker.yaml

# 5. The helper tool.
ls src/cpu-template-helper ; rg -n "dump|verify" src/cpu-template-helper/src/

# 6. Inspect a guest's CPUID after boot (from inside the guest): `cpuid` or `lscpu`.

Answer:

1. Walk the CPUID pipeline from get_supported_cpuid to set_cpuid2, naming what the transform stage changes.
2. Why are MSRs configured alongside CPUID, and what kinds of MSRs matter for portability?
3. Distinguish a static template (from /machine-config) from a custom one (from /cpu-config): source, audience, and how each reaches set_cpuid2.
4. Explain the snapshot-migration #UD failure and how a template prevents it.
5. What is the trade-off a template forces on you regarding CPU features?
6. Describe the cpu-template-helper workflow: dump, edit, verify, ship.

Common bugs and symptoms

Validation: prove you understand this

1. Draw the read→transform→set CPUID pipeline and the ioctls at each end.
2. Explain why CPUID normalization is a correctness/portability feature, not a performance one.
3. Contrast static and custom templates and name the endpoint each comes from.
4. Walk the snapshot-migration failure that templates prevent, with the host-A/host-B example.
5. Explain why MSRs are part of a CPU template and give one MSR category that matters.
6. Describe how you'd use cpu-template-helper to produce a fleet-safe template.

Next: Interrupts and the irqchip — how devices and the in-kernel interrupt controller deliver interrupts to the vCPUs you just configured.