Lab 4.3: Inspect and Apply a CPU Template

This is a build-it lab about the vCPU's identity. Every vCPU presents a CPU to the guest: a model, a vendor string, and — critically — a set of feature flags the guest reads via the CPUID instruction and a set of MSRs (model-specific registers). Firecracker configures that identity before the vCPU ever runs, with KVM_SET_CPUID2 and KVM_SET_MSRS. A CPU template is a declarative way to normalize that identity — to make a guest see the same CPU regardless of which physical host it lands on. You will dump your host's CPUID/MSR configuration with cpu-template-helper, read a static template's JSON, apply one to a microVM, and observe the masked CPUID from inside the guest.

This is where "the vCPU runs guest code" becomes "the vCPU runs guest code as a specific, controlled CPU." It is the difference between a VM that works on your laptop and a fleet of microVMs whose snapshots can be moved between an Intel Skylake host and an Intel Cascade Lake host without the guest noticing.

Background

When the guest executes CPUID, the CPU returns information about itself: vendor (GenuineIntel), family/model/stepping, and a large bitfield of feature flags — does this CPU have AVX2? AVX-512? RDRAND? The TSC deadline timer? Software (glibc, the JVM, OpenSSL) reads these flags at startup and picks code paths accordingly. A program that detects AVX-512 and then runs on a CPU without it gets an illegal-instruction fault.

KVM, by default, would let the guest see most of the host's real CPUID (filtered through KVM_GET_SUPPORTED_CPUID). That's a problem for a fleet:

• Heterogeneous hosts. Your fleet has Skylake, Cascade Lake, and Ice Lake machines. A guest that boots on Ice Lake (and detects AVX-512) cannot be live-migrated or snapshot-restored onto Skylake (no AVX-512) — it would fault. The guest must see a lowest-common-denominator CPU.
• Snapshot portability. A snapshot captures the vCPU's CPUID/MSR state. Restoring it on a different host requires the restored CPU to look identical to the captured one. If the host CPUID leaks through, the snapshot is host-specific.
• Determinism / security. Masking off RDRAND, certain TSC features, or speculative-execution feature bits can be part of a hardening or determinism strategy.

A CPU template solves this by specifying exactly which CPUID leaves/registers and which MSRs the guest sees, as a transformation applied after KVM_GET_SUPPORTED_CPUID and before KVM_SET_CPUID2. Two flavors:

The tooling is the cpu-template-helper crate: it can dump the host's full configuration as a template, verify that a template applies cleanly on a host, and help you author a custom one.

Companion reading: CPU templates & CPUID deep dive, KVM fundamentals (KVM_GET_SUPPORTED_CPUID/KVM_SET_CPUID2), and snapshotting (why portability needs templates).

Why This Lab Matters for Contributors

CPU-template bugs are a real, recurring issue category: a feature flag not masked (breaking portability), a static template missing a fixup for a new microarchitecture, a custom-template validation gap, an MSR not round-tripped on restore. These are sharply-scoped, high-value PRs — exactly the kind a Level 4 graduate opens (see Lab 4.4). You cannot reason about snapshot compatibility (Level 9) or the threat model without understanding what a template controls. And "why does my guest see a different CPU after restore?" is a question you'll be asked the moment you touch this code.

Prerequisites

• Firecracker builds, including the helper crates: tools/devtool build.
• You can boot a microVM by hand (Lab 1.3) and reach a guest shell over the serial console.
• You read Lab 4.1 (where CPUID is set, conceptually: before first KVM_RUN).
• An x86_64 host. CPUID/MSR templates are most meaningful on x86; aarch64 has an analog (CPU registers) but this lab is x86-centric.

mkdir -p ~/fc-notes ; : > ~/fc-notes/reading-log-4.3.md

Note: Static template names (T2, T2S, T2CL, T2A, C3, …) are version- and vendor-sensitive: T2* are Intel, T2A is AMD, C3 is older. Verify which exist on your branch before applying one — an unsupported name on the wrong vendor host is rejected. Find them:

rg -n "StaticCpuTemplate|enum .*Template|\"T2\"|\"T2S\"|\"T2CL\"|\"T2A\"|\"C3\"" src/vmm/src/cpu_config/

Step 1 (10 min) — Locate the CPU-config code

# The crate that dumps/verifies/authors templates.
find . -path "*cpu-template-helper*" -name "*.rs" | head
ls cpu-template-helper/src 2>/dev/null

# Where CPUID/MSRs are configured and where a template is applied, in the VMM.
rg -n "set_cpuid2|SET_CPUID2|get_supported_cpuid|set_msrs|CpuConfiguration|apply.*[Tt]emplate" \
  src/vmm/src/cpu_config/ src/vmm/src/arch/x86_64/ src/vmm/src/vstate/vcpu/

In your log, name (file + function): where Firecracker calls KVM_GET_SUPPORTED_CPUID, where a template mutates the CPUID/MSR set, and where the result is pushed with KVM_SET_CPUID2. The pipeline is: supported CPUID (from KVM) → template transform → SET_CPUID2 (into the vCPU), all before the first KVM_RUN.

Step 2 (15 min) — Dump your host's CPU configuration

cpu-template-helper can emit a template that exactly describes this host. This is your ground truth for "what would the guest see with no masking."

# Find the built helper binary (path mirrors the firecracker binary).
HELPER=$(find build/cargo_target -name cpu-template-helper -type f | grep -E 'release|debug' | head -1)
echo "$HELPER"
"$HELPER" --help

# Dump this host's CPUID + MSR configuration as a (custom-template-shaped) JSON.
# The exact subcommand/flags are version-sensitive — read --help. Common shape:
"$HELPER" template dump --output /tmp/host-config.json   # verify subcommand on your branch
jq '.cpuid_modifiers | length, .msr_modifiers | length' /tmp/host-config.json 2>/dev/null

Tip: If template dump isn't the subcommand on your branch, --help shows the right one (it may be dump, template dump, or similar, and may need a running/temporary microVM to read the live config from). The helper exists precisely so you never hand-write a CPUID leaf table.

Open /tmp/host-config.json. It's a list of CPUID modifiers (each: a leaf number, a subleaf, and the eax/ebx/ecx/edx register values or a per-bit mask) and MSR modifiers (each: an MSR index and a value/mask). In your log, find and note:

• The leaf 0x1 ECX/EDX (the classic feature flags — SSE, AVX live here).
• The leaf 0x7 subleaf 0 EBX/ECX/EDX (AVX2, AVX-512, etc.).
• The vendor leaf 0x0 (the GenuineIntel/AuthenticAMD string in EBX/EDX/ECX).

Step 3 (15 min) — Read a static template's transform

Static templates live in source, not JSON files you edit. Read one to see what it changes:

# Find the static template definitions (e.g. T2, T2S, T2CL, C3) and what they mask.
rg -n "mod t2|fn t2|T2|cpuid|msr|mask|0x1, |leaf 0x7|0x7," src/vmm/src/cpu_config/x86_64/
find src/vmm/src/cpu_config -name "*.rs" | xargs rg -l "cpuid|CpuidEntry|template" | head

Pick T2 (a common Intel baseline). In your log, answer:

1. Which feature bits does it clear (mask off)? (Typically newer-than-baseline features so the guest can run on older Intel hosts — e.g. it may clear bits that distinguish Cascade Lake/Ice Lake from the baseline.)
2. Does it normalize the vendor/brand string or family/model? Why would you want a stable model number across a fleet?
3. Which MSRs does it touch?

Note: The point of T2 is "a guest that boots under T2 sees the same CPU on any T2-compatible Intel host." The template is the contract. Read the comments in the static-template source — they usually state which microarchitectures the template targets.

Step 4 (15 min) — Boot WITHOUT a template and record the guest's CPU

Baseline first. Boot a microVM with no cpu_template, get a guest shell, and capture what the guest sees:

# Pre-boot config: no cpu_template field (or "None").
curl -X PUT --unix-socket /tmp/fc.sock \
  --data '{"vcpu_count":2,"mem_size_mib":1024}' http://localhost/machine-config
# ... boot-source, drives, InstanceStart as in Lab 1.3 ...

In the guest, over the serial console:

# In the guest — the feature flags the kernel parsed from CPUID:
grep -m1 ^flags /proc/cpuinfo | tr ' ' '\n' | sort > /tmp/guest-flags-none.txt
grep -m1 "model name" /proc/cpuinfo
# If a `cpuid` tool is in the rootfs, dump raw leaves too:
cpuid -1 2>/dev/null | head -40 || echo "no cpuid tool; /proc/cpuinfo flags are enough"
wc -l /tmp/guest-flags-none.txt

Copy /tmp/guest-flags-none.txt out (over the console or vsock). This is the unmasked flag set.

Step 5 (15 min) — Boot WITH a static template and diff the flags

Now boot a fresh microVM (restart firecracker, fresh socket) and apply a static template:

curl -X PUT --unix-socket /tmp/fc.sock \
  --data '{"vcpu_count":2,"mem_size_mib":1024,"cpu_template":"T2"}' \
  http://localhost/machine-config
# ... boot-source, drives, InstanceStart ...

In the guest:

grep -m1 ^flags /proc/cpuinfo | tr ' ' '\n' | sort > /tmp/guest-flags-t2.txt
grep -m1 "model name" /proc/cpuinfo

Diff the two flag sets on the host:

diff /tmp/guest-flags-none.txt /tmp/guest-flags-t2.txt
# '<' lines = flags present without a template but MASKED OFF by T2.

The flags that disappear under T2 are exactly the feature bits the template cleared. In your log, list a few (e.g. an AVX-512 variant, a newer instruction set extension) and explain why clearing them makes the guest portable to an older host.

Warning: Applying an Intel template (T2/T2S/T2CL) on an AMD host (or T2A on Intel) is rejected — a template can only mask, never add, and can't pretend a different vendor. If InstanceStart fails with a CPU-template error, you're on the wrong vendor. Check lscpu | grep Vendor on the host and pick the matching template (or C3).

Step 6 (15 min) — Apply a CUSTOM template via /cpu-config

Static templates are built-in; a custom one is JSON you PUT /cpu-config before boot. Take your host dump from Step 2, trim it to a single deliberate change (e.g. clear one feature bit), and apply it:

# Validate your custom template applies on this host before booting with it:
"$HELPER" template verify --template /tmp/host-config.json   # verify subcommand name on your branch

# Pre-boot: load the custom template.
curl -X PUT --unix-socket /tmp/fc.sock \
  --data @/tmp/host-config.json http://localhost/cpu-config
# ... then machine-config (WITHOUT a static cpu_template), boot-source, drives, InstanceStart ...

Note: /cpu-config (custom) and /machine-config's cpu_template (static) are two paths to the same outcome — a transformed CPUID/MSR set. You generally use one or the other. Find the validation the API does: rg -n "cpu-config|CpuConfig|custom.*template|validate" src/vmm/src/ src/firecracker/.

In the guest, confirm your single masked bit is gone (grep flags /proc/cpuinfo). You've now driven both the static and custom paths and proven the masking end to end.

Implementation Requirements / Deliverables

• A host CPU-config dump (cpu-template-helper) saved to JSON, with the vendor leaf, leaf 0x1, and leaf 0x7 located and noted.
• A reading note: where Firecracker calls GET_SUPPORTED_CPUID, where the template transforms the set, and where SET_CPUID2 pushes it — all before the first KVM_RUN.
• Two guest flag dumps (/proc/cpuinfo) — no-template vs T2 (or your vendor's static template) — and a diff showing which flags T2 masks off.
• A one-paragraph explanation of why those specific flags are cleared (portability to older hosts), grounded in the static-template source comments.
• A custom template applied via PUT /cpu-config, with one deliberately-masked bit confirmed gone inside the guest.

Troubleshooting

cpu-template-helper --help shows different subcommands than this lab

Expected — the CLI evolves. The capabilities (dump host config, verify a template on a host) are stable in intent; --help is authoritative for the exact verbs and flags on your branch.

InstanceStart fails with a CPU-template / CPUID error

Almost always vendor mismatch (Intel template on AMD or vice versa) or an unsupported template name on your branch. lscpu | grep -i vendor; pick the matching static template; re-check the name with the rg from the prerequisites.

The diff between no-template and T2 is empty

Your host may already be a T2-baseline microarchitecture (nothing newer to mask), or /proc/cpuinfo flags granularity isn't fine enough to show the masked bits. Use a raw cpuid dump of the specific leaf (cpuid -1 -l 0x7) for a finer comparison, or pick a template that targets an older baseline than your host.

No cpuid tool and minimal /proc/cpuinfo in the rootfs

The Firecracker CI rootfs is minimal. /proc/cpuinfo's flags line is enough for this lab. If you want raw leaves, build a tiny static cpuid-dumping program into a scratch rootfs, or read the leaves from the host dump and reason about which the guest should/shouldn't see.

Expected Output

# Host, no template -> T2 diff (flags MASKED OFF by T2):
$ diff /tmp/guest-flags-none.txt /tmp/guest-flags-t2.txt
< avx512f
< avx512dq
< avx512cd
< ...            # newer-than-baseline features T2 clears for portability

(Your exact masked set depends entirely on the host microarchitecture and the template chosen. The result — a template removes a coherent set of newer feature flags so the guest looks like an older, fleet-wide-common CPU — is the point.)

Stretch Goals

1. Snapshot portability proof. Snapshot a microVM booted under T2 (snapshotting deep dive), then restore it and re-check /proc/cpuinfo. The CPU must be identical across restore. Now reason: why would restoring a no-template snapshot onto a different host be unsafe?
2. Author a real custom template. Use cpu-template-helper to dump two different hosts (or read two captured dumps) and compute the intersection — the largest template both can satisfy. That intersection is exactly what a fleet operator builds. Document the leaves where the hosts differ.
3. Find the MSR side. CPUID isn't the whole story — MSRs matter for TSC, speculative-execution controls, etc. Find where Firecracker sets MSRs (rg -n "set_msrs|SET_MSRS|MsrModifier" src/vmm/) and identify one MSR a template normalizes. Why is an un-normalized MSR a snapshot-restore hazard?
4. Trace the failure path. Deliberately apply a wrong-vendor template and read how the error propagates from the CPUID-application code, through the vCPU/builder, to the API response — the exact propagation chain you'll fix in Lab 4.4.

Validation / Self-check

Answer without notes. These gate completion.

1. What does the guest read with CPUID, and why does software (glibc, the JVM) care about the result at startup?
2. State the CPU-config pipeline in order: from KVM_GET_SUPPORTED_CPUID to the running vCPU. At which point does a template transform the set, and why must it all happen before the first KVM_RUN?
3. Why do CPU templates exist? Give the two primary reasons (heterogeneous hosts; snapshot portability) and explain each in one sentence.
4. What is the difference between a static template (/machine-config cpu_template) and a custom template (PUT /cpu-config)? When would you use each?
5. Why can a template only mask features and never add them? What happens if you apply an Intel template on an AMD host?
6. You applied T2 and a set of flags vanished from the guest's /proc/cpuinfo. Name the mechanism and explain why removing those flags makes the guest portable.
7. Why is restoring a no-template snapshot onto a different host model unsafe, and how does a template make it safe?

When you can dump the host config, diff masked flags between no-template and T2, apply a custom template, and explain portability, you've completed Lab 4.3. Continue to Lab 4.4: Fix a vCPU/KVM State Edge Case.