Lab 3: CPUID and MSRs in Depth

Background

A guest discovers what CPU it's running on by executing CPUID and reading MSRs (model-specific registers). Both are virtualized, and the VMM — not the hardware — controls the answers. This is the lever behind two of the hardest problems in Firecracker: making a snapshot taken on one host model resume safely on another, and keeping the guest from seeing (and exploiting) CPU features the operator never meant to expose. This lab makes you own that lever.

You'll work the full pipeline hands-on: ask KVM for the supported CPUID with KVM_GET_SUPPORTED_CPUID, see how Firecracker masks and edits the leaves (topology fix-ups, brand string, feature clears), install them with KVM_SET_CPUID2, and do the analogous dance for MSRs. You'll use cpu-template-helper to dump and diff what two configurations expose, then boot a guest and run cpuid/rdmsr inside it to confirm the guest sees exactly what the template dictated — and nothing else.

This is a build-and-instrument lab that extends Level 4 Lab 4.3 and the CPU templates & CPUID deep dive. That material introduced templates; here you go down to the individual leaf and register, prove the guest-visible effect, and reason about the security and snapshot-portability consequences from the bytes.

Why This Lab Matters for Contributors

Snapshot compatibility issues — a guest that resumes and then hits an illegal instruction, or a workload that behaves differently after migration — are almost always CPUID/MSR normalization problems. You can't triage one without being able to dump and diff the exposed CPU surface.
Security: every CPUID feature bit and MSR is attack surface. Masking is part of the threat model. A PR that exposes a leaf carelessly is a vulnerability; reviewing such a PR requires the skill this lab builds.
cpu-template-helper is a real in-tree tool maintainers use to author and verify templates. Learning it is learning the project's own workflow for this feature, which feeds issue-roadmap Stage 6.

Prerequisites

Lab 1 and Lab 2 complete — you can build a VMM and you know the fd levels.
A Firecracker build and the ability to boot a microVM by hand.
The cpuid and msr-tools packages available inside your guest rootfs (or a rootfs you can add them to). On the host, the cpuid utility helps too.

cd ~/firecracker
# Confirm cpu-template-helper builds (it's a workspace member):
rg -n "cpu-template-helper" Cargo.toml
ls src/cpu-template-helper/
mkdir -p ~/fc-notes ; : > ~/fc-notes/cpuid-msr.md

Note: CPUID leaves and MSR indices are an x86 topic; aarch64 has an analogous but different identity surface (system registers / MIDR_EL1 etc.). This lab is x86_64. The principle — the VMM controls what the guest is told it is, and normalizes it for portability and security — transfers; the registers don't.

Step-by-Step Tasks

Step 1 (15 min) — The three-step CPUID pipeline in the source

Read the pipeline before you run it. The flow is: get the supported set (system fd), edit it, set it on each vCPU (vCPU fd).

# 1) GET_SUPPORTED_CPUID — the menu of what's possible on this host+KVM:
rg -n "get_supported_cpuid|GET_SUPPORTED_CPUID|KVM_MAX_CPUID_ENTRIES|CpuId" src/vmm/src/
# 2) The edit/normalize stage — where leaves get masked and patched:
rg -n "normalize|cpuid|Leaf|brand_string|topology|0xb|leaf_0x1" src/vmm/src/cpu_config/
# 3) SET_CPUID2 — install on the vCPU before the first KVM_RUN:
rg -n "set_cpuid2|SET_CPUID2" src/vmm/src/vstate/vcpu/

In your notes, name the function that does each step and the module it's in. Identify one concrete edit Firecracker makes for correctness (e.g. patching the topology leaf 0xB/0x1 so the guest sees the configured core/thread count) and one it makes (or could make) for security (clearing a feature bit so the guest can't use a host capability).

Step 2 (15 min) — Dump the supported CPUID with a tiny program

Write a standalone program that asks KVM directly. This is KVM_GET_SUPPORTED_CPUID with no Firecracker in the way — the raw menu.

// dump-cpuid/src/main.rs  (Cargo deps: kvm-ioctls, kvm-bindings)
use kvm_bindings::KVM_MAX_CPUID_ENTRIES;
use kvm_ioctls::Kvm;

fn main() {
    let kvm = Kvm::new().expect("open /dev/kvm");
    // The supported-CPUID set is a SYSTEM-fd query.
    let cpuid = kvm
        .get_supported_cpuid(KVM_MAX_CPUID_ENTRIES)
        .expect("KVM_GET_SUPPORTED_CPUID");
    println!("function  index  eax       ebx       ecx       edx");
    for e in cpuid.as_slice() {
        println!(
            "{:#010x} {:#06x} {:#010x} {:#010x} {:#010x} {:#010x}",
            e.function, e.index, e.eax, e.ebx, e.ecx, e.edx
        );
    }
}

cargo new --bin dump-cpuid && cd dump-cpuid
# add kvm-ioctls / kvm-bindings to Cargo.toml (versions matching Firecracker)
# paste the source above, then:
cargo run | tee ~/fc-notes/supported-cpuid.txt

Find leaf 0x1 (feature flags in ecx/edx) and leaf 0xB (extended topology). Note that this is the host's full menu — Firecracker will hand the guest a subset.

Note: KVM_GET_SUPPORTED_CPUID is a system-level ioctl (it describes the host+KVM capability), but KVM_SET_CPUID2 is per-vCPU (each vCPU gets its installed view). That asymmetry — query once at the top, install per vCPU — is exactly why CPU templates are a pre-boot, per-vCPU concern.

Step 3 (20 min) — Inspect a real configuration with `cpu-template-helper`

cpu-template-helper is the project's tool for dumping the actual CPUID/MSR configuration Firecracker would apply, given a machine config. Find its subcommands and dump a baseline.

cd ~/firecracker
tools/devtool build
HELPER=./build/cargo_target/x86_64-unknown-linux-musl/debug/cpu-template-helper
# Discover the subcommands (dump / fingerprint / verify — names vary, check --help):
$HELPER --help
rg -n "Subcommand|dump|fingerprint|verify|template" src/cpu-template-helper/src/

Dump the CPU configuration for a minimal machine config (the tool typically needs a kernel + a config to instantiate a microVM and read back what would be applied):

# Shape — confirm the exact flags from --help on your branch:
$HELPER template dump --config <machine-config.json> --output ~/fc-notes/cpu-config-baseline.json 2>&1 | tee -a ~/fc-notes/cpuid-msr.md
# Inspect what it captured:
python3 -m json.tool ~/fc-notes/cpu-config-baseline.json | head -60

In cpu-config-baseline.json you'll see CPUID leaves and MSRs as modifier/value entries. Diff the leaf 0x1 feature bits here against your raw supported-cpuid.txt from Step 2. The difference is Firecracker's mask — record at least three feature bits that the supported set has but the applied config drops or pins.

Step 4 (20 min) — Build and apply a custom template; diff the surface

Create a custom CPU template that explicitly modifies a leaf — for example, clearing a feature bit — and use the helper to show the before/after exposed surface. The template format is JSON with per-leaf/per-MSR modifiers.

# Find the template schema / an example to copy:
rg -n "struct CustomCpuTemplate|CpuidLeafModifier|RegisterModifier|MsrModifier" src/vmm/src/cpu_config/
find . -name '*.json' | xargs grep -l "cpuid_modifiers\|msr_modifiers" 2>/dev/null | head

A minimal custom template (shape — confirm field names against the schema you just grepped) that masks one CPUID feature bit:

{
  "cpuid_modifiers": [
    {
      "leaf": "0x1",
      "subleaf": "0x0",
      "flags": 0,
      "modifiers": [
        { "register": "ecx", "bitmap": "0b00000000000000000000000010000000" }
      ]
    }
  ],
  "msr_modifiers": []
}

Apply it via /cpu-config at boot, or pass it to the helper's verify/dump path, and diff the exposed leaf 0x1 against the baseline. Record the exact bit you cleared and confirm the helper shows it gone.

# At boot time, a custom template goes via PUT /cpu-config (pre-boot):
curl -X PUT --unix-socket /tmp/fc.sock --data @my-template.json http://localhost/cpu-config

Step 5 (20 min) — Prove it from inside the guest

The whole point is the guest-visible effect. Boot a microVM with your template and run cpuid/rdmsr inside it; the masked bit must be absent.

# Boot as usual with the template applied (PUT /cpu-config before InstanceStart).
# Then, inside the guest over the serial console:
cpuid -1 -l 1            # leaf 1; check the ECX feature bits — your cleared bit is gone
cat /proc/cpuinfo | grep -o '^flags.*' | head -1   # the kernel's view of the same bits
# MSRs (needs CONFIG_X86_MSR + msr-tools):
modprobe msr 2>/dev/null
rdmsr 0x10               # TSC, as an example readable MSR

Compare the guest's leaf 0x1 ecx to the host's (from Step 2) and to your template. In your notes, write the three views side by side: host supported, Firecracker applied (helper), guest sees (cpuid inside). They must form a consistent chain: guest ⊆ applied ⊆ supported, with your explicit modifier visible at the applied→guest step.

Step 6 (15 min) — The MSR path

MSRs follow the same get/edit/set pattern with different ioctls. Read the wiring and then observe.

# The MSR ioctls and the supported-MSR list:
rg -n "KVM_GET_MSRS|KVM_SET_MSRS|GET_MSR_INDEX_LIST|get_msrs|set_msrs|msr_index_list" src/vmm/src/
# Which MSRs Firecracker saves/restores (this set is load-bearing for snapshots):
rg -n "msrs_to_save|MSR_|allowed_msrs|supported_msrs" src/vmm/src/

The MSRs Firecracker save/restores are exactly the ones a snapshot must carry to make the guest see a consistent CPU after restore. In your notes, name three MSRs in that set and say why each matters (e.g. an MSR that exposes a feature, one that holds architectural state). Connect this to Snapshotting: the snapshot's vCPU state is this MSR set plus the registers and CPUID.

Step 7 (15 min) — Reason about portability and security

Put the pieces together in writing. Two scenarios, both real:

Cross-host snapshot. A snapshot is taken on a host whose CPUID leaf 0x1 advertises feature X. It's restored on a host that lacks X. Walk through what the guest sees with and without a CPU template that masks X to a common baseline. What instruction or behavior breaks without the template? (Hint: a userspace program that cached CPUID at startup, or a JIT that emits X instructions.) This is the snapshot-compat failure mode in CPU terms.
Security mask. Pick a feature bit you'd want cleared for a hostile guest (a side-channel-relevant capability, or one that widens the host attack surface). Explain why a default-deny posture — expose the minimum the guest needs — is the right one, and how cpu-template-helper's verify step lets a maintainer prove a template still masks it after a code change.

Implementation Requirements / Deliverables

The three-step pipeline named in source: the function for GET_SUPPORTED_CPUID, the normalize/edit stage, and SET_CPUID2, each with its module.
Your raw supported-cpuid.txt from the standalone dumper, and the cpu-config-baseline.json from cpu-template-helper, with at least three feature bits Firecracker masks (supported but not applied).
A custom template that clears one named CPUID feature bit, and the helper-diff showing it gone from the applied config.
The three-view chain (host supported ⊇ applied ⊇ guest-sees) proven with cpuid run inside the guest, your cleared bit visibly absent.
Three save/restored MSRs named with why each matters for snapshots.
The two written scenarios from Step 7 (cross-host snapshot break; security mask).

Troubleshooting

`cpu-template-helper --help` shows different subcommands than above

Expected — the CLI evolves. Use --help and the rg "Subcommand" output as ground truth; the names dump/fingerprint/verify are representative, not guaranteed.

`cpuid` not present in the guest

Your rootfs lacks it. Add the cpuid package to the rootfs (or use a fuller rootfs), or read the bits from /proc/cpuinfo's flags line, which the kernel derives from the same CPUID. For MSRs you also need CONFIG_X86_MSR and msr-tools.

My masked bit is still present inside the guest

Three common causes: the template wasn't applied (confirm PUT /cpu-config returned success before InstanceStart); you masked the wrong register (ecx vs edx) or bit position; or the kernel re-derives the capability from a different leaf and the bit you cleared isn't the one the flag comes from. Diff the helper's applied config to confirm the modifier landed, then re-check which leaf/register the guest reads.

`KVM_SET_CPUID2` fails with `E2BIG`

You handed more leaves than the CpuId/kvm_cpuid2 buffer was allocated for. Allocate with KVM_MAX_CPUID_ENTRIES. Check rg -n "KVM_MAX_CPUID_ENTRIES|with_capacity" src/vmm/src/.

`rdmsr` returns "operation not permitted" / device missing

modprobe msr first; the guest kernel needs CONFIG_X86_MSR. Check the guest config: grep CONFIG_X86_MSR resources/guest_configs/*.

Expected Output

# host supported (Step 2), leaf 0x1 ecx (excerpt):
0x00000001 0x0000 ...  ecx=0x7ffafbff ...

# Firecracker applied (helper, baseline) leaf 0x1 ecx — a MASKED subset:
"leaf": "0x1", "ecx": "0x7ed8220b"     # several host bits cleared

# guest sees (cpuid -1 -l 1 inside the VM), after your custom mask of bit 7:
   ... ecx bit 7 (your target): 0    # confirmed absent

The exact hex differs by host. The result you're confirming is the chain: the guest's exposed ecx is a strict subset of the applied config, which is a strict subset of the host-supported set, and your explicit modifier is visibly the delta at the last step.

Stretch Goals

Topology fix-up. Configure vcpu_count: 4 with SMT on vs off and dump leaf 0xB/0x1 each time. Confirm Firecracker patches the core/thread counts so the guest's nproc and lscpu match the config — a correctness edit, not a security one.
Fingerprint two hosts. Run cpu-template-helper fingerprint on two different physical hosts (or two EC2 instance types) and diff. The diff is exactly the set of bits a portable snapshot template must normalize — the input to a real snapshot-compat investigation.
Verify-after-change. Make a trivial change to the normalization code, then run the helper's verify against a saved template. Watch it catch the drift. This is the maintainer's safety net for CPUID changes — the thing a PR reviewer relies on.
MSR round-trip. Snapshot a paused guest, dump the saved MSRs from the snapshot file with snapshot-editor, and confirm they match the msrs_to_save set. Bridge to Snapshotting Lab 1.

Validation / Self-check

Answer without notes. These gate completion.

Walk the CPUID pipeline: which ioctl gets the supported set, where does Firecracker edit it, which ioctl installs it, and at which fd level is each?
Give one CPUID edit done for correctness and one done for security, naming the leaf each touches.
Explain the chain guest ⊆ applied ⊆ host-supported. Where does a custom template's modifier appear in that chain?
Why does a cross-host snapshot need CPUID/MSR normalization? Describe a concrete way the guest breaks without it.
Which MSRs does Firecracker save/restore, and why is that set exactly what a snapshot must carry?
How would you prove, from inside a guest, that a feature bit is masked — and what are two reasons a "masked" bit might still appear?
What does cpu-template-helper's verify step protect against, and why is it the maintainer's safety net for changes to CPUID handling?

When you can dump the supported CPUID, mask a bit with a template, prove its absence inside the guest, and explain the snapshot-portability and security stakes, you've completed Lab 3 — and the KVM & vCPUs intensive. Continue to The Boot Process intensive to see how a real vmlinux gets loaded onto the vCPUs you now understand, or to issue-roadmap Stage 6 (vCPU & KVM) to put this to work on a real issue.

Open-Source Engineer & Contributor