Lab 2: VM-Exit Taxonomy — A Data-Driven Study of a Real Boot

Background

In Lab 1 you made VM exits by hand: an out here, an MMIO write there, a hlt to finish. You know what each exit is. This lab inverts the exercise: you take a real Firecracker boot — an actual vmlinux coming up on real virtio devices — and you count and classify every single VM exit it produces. Then you tie each class, and within MMIO each register offset, back to a concrete device and register.

This is a trace-it / study lab, and it is the single most leverage-dense thing you can measure about a VMM. Every exit is a context switch from guest to host and a slice of host code emulating a device. Performance is "exits per operation." Correctness is "did every exit get handled right." Security is "fewer exit handlers, less attack surface." Maintainers reason about Firecracker in exit counts; by the end of this lab you will too, with numbers you produced yourself.

This lab extends Level 4 Lab 4.2 — that lab introduced PIO vs MMIO and the virtio-MMIO register map. Here you build a complete taxonomy: not just "I see MMIO exits" but a full histogram of every exit class across the entire boot, decoded register by register, and the same data again from the production METRICS system so you learn both the throwaway and the mergeable way to get it.

Why This Lab Matters for Contributors

• "The guest is slow," "CPU is 100% at idle," "this device hangs" are all exit problems. The skill of producing an exit histogram and reading it is the diagnostic backbone of every performance and many correctness investigations.
• The minimal device model philosophy is an argument about exit handlers — fewer devices, fewer exit paths, less host code a hostile guest can reach. You can only evaluate a proposal to add a device if you can reason about the exits it adds.
• The production answer is the METRICS system, not eprintln!. Learning to read exit counts from metrics is what turns "I instrumented the run loop locally" into "here's an observability PR."

Prerequisites

• Lab 1 complete — you can build a VMM and you know the VcpuExit variants cold.
• You can boot a microVM by hand from your own build (Lab 1.3, Lab 1.1).
• You read the intensive index's exit-taxonomy table and Lab 4.2.

cd ~/firecracker
ls build/cargo_target/*/debug/firecracker 2>/dev/null || tools/devtool build
mkdir -p ~/fc-notes ; : > ~/fc-notes/exit-taxonomy.md

Note: This lab instruments the hottest path in Firecracker with throwaway logging. Everything in Steps 2–4 is for measurement only and must never go into a PR. Step 6 shows the production-safe path. Build in debug so the trace is readable; the shape of the histogram is what matters, not absolute speed.

Step-by-Step Tasks

Step 1 (10 min) — Predict the taxonomy before you measure

Science: write down your hypothesis first. Based on the intensive index and Lab 4.2, fill this table in your notes before running anything. You'll grade yourself against the measured result.

And within MMIO, predict which virtio-MMIO register offsets dominate (hint: recall the two from Lab 4.2).

Step 2 (15 min) — Find the run loop and the exit match

The instrumentation point is the match over VcpuExit. It lives in the arch-specific vCPU code, and x86_64 and aarch64 diverge — read the one for your host.

# The run loop and its exit match:
rg -n "fn run\b|run_emulation|match.*run\(\)|VcpuExit::" src/vmm/src/vstate/vcpu/
# The x86_64 handler (or aarch64.rs on ARM):
rg -n "VcpuExit::IoIn|VcpuExit::IoOut|VcpuExit::MmioRead|VcpuExit::MmioWrite|VcpuExit::Hlt|VcpuExit::Shutdown" \
  src/vmm/src/vstate/vcpu/x86_64.rs

In your notes, record the exact function and the file. Note whether the match is in x86_64.rs/aarch64.rs directly or in a shared mod.rs that calls into the arch handler — Firecracker factors the common loop from the arch-specific exit dispatch.

Step 3 (20 min) — Build a counting harness

Add a per-variant tally that prints a histogram on shutdown. This is cleaner than a trace! per exit (which floods the log and slows the boot enough to skew ratios). The shape:

#![allow(unused)]
fn main() {
// TEMPORARY (Lab 2) — remove before any PR. A per-exit-class counter.
// Add near the Vcpu struct (or as thread-locals); increment in each match arm.
use std::sync::atomic::{AtomicU64, Ordering};

static IO_EXITS: AtomicU64 = AtomicU64::new(0);
static MMIO_EXITS: AtomicU64 = AtomicU64::new(0);
static HLT_EXITS: AtomicU64 = AtomicU64::new(0);
static OTHER_EXITS: AtomicU64 = AtomicU64::new(0);
// Decoded MMIO register offsets -> count (use a small fixed set of offsets).
static MMIO_NOTIFY: AtomicU64 = AtomicU64::new(0);   // 0x050 QueueNotify
static MMIO_INTSTATUS: AtomicU64 = AtomicU64::new(0); // 0x060 InterruptStatus
static MMIO_INTACK: AtomicU64 = AtomicU64::new(0);    // 0x064 InterruptACK
static MMIO_STATUS: AtomicU64 = AtomicU64::new(0);    // 0x070 Status
}

In each match arm, bump the right counter (and decode the MMIO offset with addr & 0xfff, exactly as in Lab 4.2):

#![allow(unused)]
fn main() {
VcpuExit::IoIn(..) | VcpuExit::IoOut(..) => { IO_EXITS.fetch_add(1, Ordering::Relaxed); /* ...existing... */ }
VcpuExit::MmioRead(addr, _) | VcpuExit::MmioWrite(addr, _) => {
    MMIO_EXITS.fetch_add(1, Ordering::Relaxed);
    match addr & 0xfff {
        0x050 => MMIO_NOTIFY.fetch_add(1, Ordering::Relaxed),
        0x060 => MMIO_INTSTATUS.fetch_add(1, Ordering::Relaxed),
        0x064 => MMIO_INTACK.fetch_add(1, Ordering::Relaxed),
        0x070 => MMIO_STATUS.fetch_add(1, Ordering::Relaxed),
        _ => 0,
    };
    /* ...existing dispatch... */
}
VcpuExit::Hlt => { HLT_EXITS.fetch_add(1, Ordering::Relaxed); /* ...existing... */ }
other => { OTHER_EXITS.fetch_add(1, Ordering::Relaxed); /* ...existing... */ }
}

Print the histogram where the vCPU loop exits (or on a SIGTERM handler). The key discipline: count in the arm, don't change the behavior — keep every existing line that actually handles the exit.

Warning: Atomics here are for measurement convenience; in the real hot path even relaxed atomics cost. This is throwaway. The production way to count exits is METRICS (Step 6), which is already wired to be cheap and per-vCPU. Confirm instrumentation is gone with git diff before building anything you'll commit.

Step 4 (15 min) — Boot and capture the boot-time histogram

tools/devtool build
FC=./build/cargo_target/x86_64-unknown-linux-musl/debug/firecracker
sudo $FC --api-sock /tmp/fc.sock &
# Configure + InstanceStart exactly as in Lab 1.3 (kernel + rootfs + machine-config):
API=/tmp/fc.sock
curl -X PUT --unix-socket $API --data \
 '{"kernel_image_path":"./vmlinux","boot_args":"console=ttyS0 reboot=k panic=1 pci=off"}' \
 http://localhost/boot-source
curl -X PUT --unix-socket $API --data \
 '{"drive_id":"rootfs","path_on_host":"./rootfs.ext4","is_root_device":true,"is_read_only":false}' \
 http://localhost/drives/rootfs
curl -X PUT --unix-socket $API --data '{"vcpu_count":2,"mem_size_mib":1024}' http://localhost/machine-config
curl -X PUT --unix-socket $API --data '{"action_type":"InstanceStart"}' http://localhost/actions
# let it reach a login prompt, then shut down (Ctrl-A then X in the console, or kill)

Read the histogram your harness printed. Record it in ~/fc-notes/exit-taxonomy.md next to your Step 1 prediction and grade yourself.

Step 5 (20 min) — Drive disk I/O and re-measure the delta

Now isolate a device's contribution. Reset the counters (re-boot), attach a scratch drive, and generate a known amount of I/O inside the guest. The delta in QueueNotify is the block device's signature.

truncate -s 64M /tmp/scratch.ext4 && mkfs.ext4 -q /tmp/scratch.ext4
# Add BEFORE InstanceStart (drives are pre-boot config):
curl -X PUT --unix-socket $API --data \
 '{"drive_id":"scratch","path_on_host":"/tmp/scratch.ext4","is_root_device":false,"is_read_only":false}' \
 http://localhost/drives/scratch

Inside the guest, over the serial console:

# In the guest:
mkfs.ext4 -q /dev/vdb && mount /dev/vdb /mnt
dd if=/dev/zero of=/mnt/f bs=1M count=16 conv=fsync ; sync ; umount /mnt

Compare the histogram before and after the dd. Answer in your notes:

1. How many QueueNotify (kicks) did 16 MiB of dd add? It should be far fewer than 16,384 (one-per-4-KiB) — virtio batches a descriptor chain per kick. That batching ratio is the heart of virtio performance.
2. Is QueueNotify (guest→device) or InterruptStatus/InterruptACK (device→guest) more frequent during the dd, and why?
3. Did the PIO count move? (It shouldn't much — PIO is the console; dd to a disk isn't console traffic.)

Step 6 (20 min) — Get the same data from METRICS (the production way)

eprintln! is for you; METRICS is for a PR. Firecracker already counts vCPU exits in its metrics system. Find them and read them via FlushMetrics.

# The metrics structs — look for per-vCPU exit counters:
rg -n "METRICS|VcpuMetrics|exit_io|exit_mmio|metric_|SharedIncMetric" src/vmm/src/
rg -n "struct VcpuMetrics|io_in|io_out|mmio|hlt" src/vmm/src/logger/ src/vmm/src/

Configure metrics to a file, boot, run the same dd, flush, and read the JSON:

# Point metrics at a file before InstanceStart:
curl -X PUT --unix-socket $API --data \
 '{"metrics_path":"/tmp/fc-metrics.json"}' http://localhost/metrics
# ... boot, run dd in the guest ...
# Force a flush:
curl -X PUT --unix-socket $API --data '{"action_type":"FlushMetrics"}' http://localhost/actions
# Read the exit counters from the JSON (the exact key names — grep the source above):
cat /tmp/fc-metrics.json | python3 -m json.tool | grep -iE "vcpu|exit|mmio|io" 

Compare the METRICS exit counts to your hand-rolled histogram. They should agree in shape. Note any difference: METRICS may aggregate IoIn+IoOut, or count at a different granularity than your & 0xfff decode. In your notes, write down which metric corresponds to which of your counters. This mapping is the deliverable that matters — it's how you'd justify an exit-related claim in a PR using data the project already collects.

Step 7 (15 min) — Tie each exit class to a device and register

Assemble the full taxonomy. For each exit class, name the Firecracker code that handles it and the device behind it.

Fill the "where" column with the file:line you actually find. The point: you can now point at the exact handler for any exit you counted.

Step 8 (10 min) — Remove instrumentation, confirm clean

git diff --stat            # should show your throwaway edits
git checkout -- src/vmm/src/vstate/vcpu/   # revert them
git diff                   # must be empty
tools/devtool build        # confirm a clean build with no leftover edits

Implementation Requirements / Deliverables

• Your Step 1 prediction table, graded against the measured boot histogram.
• The full boot-time exit histogram: counts by class (PIO/MMIO/HLT/other) and, within MMIO, by decoded register offset.
• The dd delta: how many QueueNotify 16 MiB of writes produced, with the batching ratio (kicks ÷ 4-KiB-pages) and a one-line explanation.
• The same exit data read from METRICS via FlushMetrics, with a mapping from each metric key to your hand-rolled counter.
• The completed exit-class → device → handler table from Step 7, each "where" cell a real file:line on your branch.
• git diff clean — all instrumentation removed.

Troubleshooting

My histogram is empty / nothing printed

The print site never ran — the vCPU loop didn't exit cleanly, or the static counters are in the wrong module scope. Print on a signal handler or at the top of the Hlt/Shutdown arm so you get output even on an abrupt stop.

Every MMIO offset decodes as the same value / "other"

Your & 0xfff mask assumes 4-KiB-aligned device windows. Confirm the window size on your branch: rg -n "MMIO_LEN|0x1000" src/vmm/src/. If devices are wider, widen the mask.

I see almost no QueueNotify MMIO writes

That's KVM_IOEVENTFD doing its job: the kick is short-circuited to an eventfd the VMM thread polls, so it never reaches your match as an MMIO exit. This is correct and important — note it. Find the registration: rg -n "register_ioevent|IoEventAddress|KVM_IOEVENTFD" src/vmm/src/. Your QueueNotify count being near-zero is the system working as designed; the kicks show up as eventfd wakeups on the VMM thread, not vCPU exits.

METRICS keys don't match my counters

The metric names drift and may aggregate differently than your decode. That's expected — rg the real struct (VcpuMetrics) and map your counter to whatever the project actually exposes. The mapping is the deliverable.

The boot crawls and counts look inflated

Debug build + per-exit atomics is slow. Tolerable for one observation run. If a count is implausibly huge, you may be sampling a polled register; confirm you only counted and didn't alter the read that returns the register value.

Expected Output

=== VM-exit histogram (boot, 2 vCPUs) ===
MMIO        9021
PIO         1188
HLT            5
OTHER          0

=== MMIO by decoded register ===
0x050 QueueNotify     0      (short-circuited by IOEVENTFD — see troubleshooting)
0x060 InterruptStatus 3110
0x064 InterruptACK     980
0x070 Status            47   (device-setup burst)

Numbers vary widely by kernel, rootfs, and device set. The shape is the result: MMIO-dominated, interrupt handshake on top, QueueNotify near-zero because IOEVENTFD removes it from the exit path, console PIO second, a handful of HLTs, no error exits on a healthy boot.

Stretch Goals

1. Idle exit floor. After boot, leave the guest idle 30 s and count exits in that window. A healthy idle guest generates almost none — proof the in-kernel irqchip and IOEVENTFD keep idle guests out of userspace. A steady idle exit stream is a polling bug worth a real issue.
2. Exits per kilobyte. Run dd at 4/16/64 MiB and plot QueueNotify (or the block metric) vs bytes. Linear? Sub-linear (better batching at scale)? This is the start of I/O-engine reasoning and feeds Performance & Density Lab 3.
3. The IOEVENTFD accounting. Find where queue-notify is registered with KVM_IOEVENTFD and instrument the eventfd side (the VMM thread's EventManager) instead of the vCPU side. Now you can count kicks even though they never hit the run loop — the complete picture.
4. aarch64 contrast. If you have an ARM host, repeat Step 4 on aarch64.rs. The exit set differs (no PIO; MMIO + system-register exits). Diff the two taxonomies and explain the architectural reason.

Validation / Self-check

Answer without notes. These gate completion.

1. List every VcpuExit variant Firecracker's run loop handles and the guest event that causes each.
2. In a healthy boot, which exit class dominates and why? Which two virtio-MMIO register offsets would you expect on top if IOEVENTFD weren't short-circuiting the kick?
3. Why is QueueNotify near-zero in your vCPU-side histogram, and where do those kicks actually go?
4. A 16 MiB dd produced far fewer kicks than 4-KiB-pages. Name the mechanism and state the batching ratio you measured.
5. Where in the source is the exit match, and how do x86_64 and aarch64 diverge?
6. Map the production METRICS exit counters to your hand-rolled histogram. Why is the metrics path the one you'd cite in a PR?
7. Name the exact handler (file:line on your branch) for a PIO 0x3f8 exit and for an MMIO Status write.

When you can produce the full histogram, explain the IOEVENTFD short-circuit from the data, and read the same numbers out of METRICS, you've completed Lab 2. Continue to Lab 3 — CPUID and MSRs, where you stop counting what the guest does and start controlling what the guest is told it is.