Lab 4.1: Read the vCPU Run Loop

This is a trace-it lab over src/vmm/src/vstate/vcpu/ — the heart of Firecracker. You will find the vCPU run loop, read how it calls VcpuFd::run, how it matches on VcpuExit and dispatches I/O and MMIO exits to the device buses, and how it interleaves that with the VcpuEvent control protocol (Pause/Resume/Exit) coming from the VMM thread. You will not change behavior — you will instrument the loop with tracing and watch it run during a real boot, then map what you see onto the bare KVM_RUN loop you wrote in Lab 1.4.

The goal is not to memorize the file. It is to be able to stand at a whiteboard and draw one iteration of the loop — run, exit, dispatch, respond, loop — and name the function at every hop on your own branch.

Background

When the API thread receives InstanceStart, the builder spins up one OS thread per vCPU. Each thread runs a loop that is, stripped to its essence, identical to your Lab 1.4 toy:

loop {
    match vcpu_fd.run() {            // KVM_RUN — blocks until a VM exit
        Ok(exit) => handle(exit),    // emulate the device, return read data, ...
        Err(e)   => bail(e),
    }
}

Firecracker wraps this in two layers. The inner layer is KvmVcpu (in the arch file): it owns the VcpuFd, calls .run(), and turns a single VcpuExit into a single device dispatch (a write to the PIO bus, a read from the MMIO bus, …). The outer layer is Vcpu (in mod.rs): it owns the thread and the VcpuEvent/VcpuResponse channels, decides whether to be running at all, and handles pause/resume and shutdown requests from the VMM thread between (and during) KVM_RUN calls.

That two-layer split is the whole lab. The inner layer is "what does the guest need right now?" The outer layer is "should I even be in KVM_RUN, and what do I tell the VMM thread when something goes wrong?" Read them in that order.

Companion reading: the vCPU run loop & VM exits deep dive, KVM fundamentals, and kvm-ioctls & kvm-bindings (what VcpuFd::run and VcpuExit actually are).

Why This Lab Matters for Contributors

Every Level 4 issue category lives in this loop: a poorly-surfaced KVM error, an unhandled VcpuExit, a pause/resume race, a wrong dispatch to the bus. Maintainers triaging "the guest hangs," "InstanceStart returns but nothing boots," or "pausing under load deadlocks" read exactly these two files. You cannot reproduce — let alone fix — those issues without having read the run loop end to end. This is also the foundation for Lab 4.2 (the exit taxonomy) and Lab 4.4 (a vCPU edge-case fix).

Prerequisites

• Firecracker builds: tools/devtool build succeeds (Level 1).
• You completed Lab 1.4 — you have a working mental model of a raw KVM_RUN loop.
• You can boot a microVM by hand (Lab 1.3).
• A reading log:

mkdir -p ~/fc-notes
: > ~/fc-notes/reading-log-4.1.md

Note: Names like Vcpu, KvmVcpu, VcpuExit, VcpuEvent are stable across branches, but bodies are long and line numbers drift. Every step gives you the rg to locate the code on your checkout. Never cite a line number back to a maintainer — cite the function.

Step 1 (10 min) — Find the two layers

# The outer control layer (the thread + event protocol).
rg -n "struct Vcpu\b|impl Vcpu\b|fn run\b|fn run_emulation|VcpuEvent|VcpuResponse" \
  src/vmm/src/vstate/vcpu/mod.rs

# The inner KVM layer (the VcpuFd + the exit match). Architecture-specific.
rg -n "struct KvmVcpu|impl KvmVcpu|\.run\(\)|VcpuExit::" src/vmm/src/vstate/vcpu/x86_64.rs

# Confirm the arch split (there is an aarch64 sibling).
find src/vmm/src/vstate/vcpu -type f

In your reading log, write the file + function name for each of:

1. The function the VMM thread calls to start the vCPU thread (look for a spawn/thread::Builder in or near Vcpu).
2. The top-level loop that processes VcpuEvents.
3. The function that actually calls VcpuFd::run and matches VcpuExit.

Tip: The thread body is usually a method like Vcpu::run (the OS-thread entry) that loops on a state machine, calling an emulation step (often run_emulation or similar) which is where KVM_RUN happens. Trace the call chain: thread entry → state-machine loop → emulation step → vcpu_fd.run().

Step 2 (20 min) — Read the inner loop: one KVM_RUN, one exit

Open the arch file and find the match on VcpuExit. This is the direct analog of the switch (run->exit_reason) in your Lab 1.4 C/Rust toy.

rg -n "VcpuExit::(IoIn|IoOut|MmioRead|MmioWrite|Hlt|Shutdown|FailEntry|InternalError|SystemEvent)" \
  src/vmm/src/vstate/vcpu/x86_64.rs

Map each arm to what it does. Fill this table in your log from the actual code:

For each arm, answer in your log:

• Where do IoIn/MmioRead (reads) put the data the guest will see? (The bus writes into the data slice; for PIO that slice is backed by the kvm_run shared page at io.data_offset; for MMIO it is mmio.data.)
• What does the function return to the outer loop after a normal exit? (Usually an enum like VcpuEmulation::Handled vs Stopped/Interrupted — find its name with the rg below.)

rg -n "enum VcpuEmulation|VcpuEmulation::|Handled|Stopped|Interrupted" src/vmm/src/vstate/vcpu/

Note: The PIO bus and the MMIO bus are two different Bus instances (PortIODeviceManager / MMIODeviceManager on x86). A KVM_EXIT_IO goes to the former, a KVM_EXIT_MMIO to the latter, keyed by port/address. You'll dissect this routing in Lab 4.2; for now just note which bus each arm uses.

Step 3 (20 min) — Read the outer loop: the VcpuEvent state machine

The vCPU thread does not just spin on KVM_RUN. Between runs (and woken via a signal/event during a run) it processes control messages from the VMM thread. Find the protocol:

rg -n "enum VcpuEvent|enum VcpuResponse" src/vmm/src/vstate/vcpu/
rg -n "VcpuEvent::(Pause|Resume|Finish|SaveState|RestoreState|DumpCpuConfig)" src/vmm/src/vstate/vcpu/mod.rs

The outer loop is typically a small state machine with at least these states (find the real enum — names vary by branch): Paused and Running. In Running, it calls the emulation step (which runs KVM_RUN); in Paused, it blocks waiting for a VcpuEvent. The events you should locate:

Answer in your log:

1. How does the VMM thread interrupt a vCPU that is blocked inside KVM_RUN to deliver a Pause? (Look for a signal — Firecracker uses a dedicated signal so KVM_RUN returns EINTR; find it.)

rg -n "VCPU_RTSIG_OFFSET|sigrtmin|signal|EINTR|Interrupted" src/vmm/src/vstate/vcpu/ src/vmm/src/signal_handler.rs

2. When the emulation step returns an error (a KVM ioctl failed, an unhandled exit), how does the outer loop turn that into something the VMM thread sees? (Trace the error up to a VcpuResponse or a state transition to a "faulted/exited" state.)

Step 4 (20 min) — Instrument the loop and watch a boot

Now make the loop talk. Add temporary tracing at the exit dispatch so you can see the run loop run during a real boot. Find the exit match and add a log line per arm (or one before the match dumping the variant). Use the crate's logging macros — find how the file already logs:

rg -n "use log|tracing::|warn!|debug!|trace!|METRICS" src/vmm/src/vstate/vcpu/x86_64.rs | head

Add, at the top of the exit handling, something like:

#![allow(unused)]
fn main() {
// TEMPORARY instrumentation for Lab 4.1 — remove before any PR.
match exit_reason {
    VcpuExit::IoIn(port, _)    => log::trace!("vcpu exit: IoIn  port={port:#x}"),
    VcpuExit::IoOut(port, _)   => log::trace!("vcpu exit: IoOut port={port:#x}"),
    VcpuExit::MmioRead(addr, _)  => log::trace!("vcpu exit: MmioRead  addr={addr:#x}"),
    VcpuExit::MmioWrite(addr, _) => log::trace!("vcpu exit: MmioWrite addr={addr:#x}"),
    VcpuExit::Hlt              => log::trace!("vcpu exit: Hlt"),
    other                      => log::trace!("vcpu exit: {other:?}"),
}
}

Warning: This is throwaway instrumentation. It is on the hottest path in Firecracker — trace! on every exit will flood and slow the boot. That is fine for one observation run; remove it before you build anything you'll commit. Never PR debug logging on the run loop.

Build and boot with logging at Trace:

tools/devtool build
# Configure /logger with level "Trace" (or pass --level Trace), then boot a microVM
# exactly as in Lab 1.3, and tail the log file. The simplest path:
sudo ./build/cargo_target/x86_64-unknown-linux-musl/debug/firecracker \
  --api-sock /tmp/fc.sock --level Trace --log-path /tmp/fc.log &
# ... PUT /logger if needed, /boot-source, /drives, /machine-config, then InstanceStart ...
grep -c "vcpu exit:" /tmp/fc.log
grep "vcpu exit:" /tmp/fc.log | sort | uniq -c | sort -rn | head -30

Expected: thousands of exits during boot. The histogram should be dominated by a handful of MMIO addresses (virtio device registers) and a couple of PIO ports (the serial console 0x3f8, maybe the i8042 0x60/0x64). Save the histogram into your reading log — it is the empirical proof you read the right loop.

Step 5 (15 min) — Map it onto your Lab 1.4 toy and diagram it

Put your toy and Firecracker side by side:

Draw the loop. Reproduce (don't copy — draw from your reading) a flow of one iteration:

flowchart TD
    START([vCPU thread, state = Running]) --> CHK{VcpuEvent pending?}
    CHK -->|Pause| PAUSED[state = Paused<br/>block on event channel]
    PAUSED -->|Resume| START
    CHK -->|Finish| END([thread exits])
    CHK -->|none| RUN["KvmVcpu: vcpu_fd.run()  (KVM_RUN, blocks)"]
    RUN -->|returns VcpuExit| MATCH{match exit_reason}
    MATCH -->|IoIn/IoOut| PIO["PortIODeviceManager bus<br/>read/write port"]
    MATCH -->|MmioRead/Write| MMIO["MMIODeviceManager bus<br/>read/write addr"]
    MATCH -->|Hlt / Shutdown| STOP[signal shutdown to VMM thread]
    MATCH -->|FailEntry / InternalError| ERR[VcpuError -> VcpuResponse to VMM thread]
    PIO --> START
    MMIO --> START
    STOP --> END
    ERR --> END

Annotate your version with the real function names from your branch at each box.

Implementation Requirements / Deliverables

• A reading log naming, with file + function (not line number): the thread entry, the VcpuEvent state-machine loop, the emulation step that calls KVM_RUN, and the VcpuExit match.
• The completed VcpuExit → bus table from the actual code on your branch.
• A boot-time exit histogram (uniq -c) captured with temporary tracing, dominated by a few MMIO addresses and PIO ports.
• Your own mermaid/ASCII diagram of one loop iteration, annotated with real function names.
• The instrumentation removed from your tree (git diff is clean of the trace! additions).

Troubleshooting

grep -c "vcpu exit:" is 0

Logging isn't at Trace, or the build didn't pick up your edit. Confirm the log level (/logger body or --level Trace), confirm you rebuilt (tools/devtool build), and confirm you're running the binary you just built (the path under build/cargo_target/...), not an old one on $PATH.

The boot hangs after my edit

A trace! per exit can be slow enough to make a TCP/serial timeout fire, but it should still boot. More likely you accidentally changed control flow (e.g. swallowed the exit before the real dispatch). Your instrumentation must be additive — log, then fall through to the original match.

I can't tell which file has the run loop (x86 vs aarch64)

find src/vmm/src/vstate/vcpu -type f shows the per-arch files. On an x86 host you're in x86_64.rs; the architecture-independent control loop is in mod.rs. The cfg(target_arch) gates decide which compiles.

VcpuExit has variants my match doesn't list

Good — that's the point of reading on your branch. cargo doc -p kvm-ioctls --open lists the full enum; new variants (e.g. for newer KVM features) appear over time. Note any your run loop maps to a catch-all.

Expected Output

$ grep "vcpu exit:" /tmp/fc.log | sort | uniq -c | sort -rn | head
  4123 vcpu exit: MmioWrite addr=0xd0000050   # a virtio QueueNotify (kick)
  3880 vcpu exit: MmioRead  addr=0xd0000060   # InterruptStatus
  1190 vcpu exit: IoOut port=0x3f8            # serial console TX
   ...
     1 vcpu exit: Hlt

(Your exact addresses/ports differ — virtio-MMIO base, the device layout, and the kernel version all move them. The shape — MMIO-dominated, a serial port, a terminal Hlt — is what you're confirming.)

Stretch Goals

1. Find the EINTR path. Trigger a pause during boot (PATCH /vm {"state":"Paused"} mid-boot) and confirm in the log that KVM_RUN returned interrupted and the loop transitioned to Paused rather than treating EINTR as an error. Read exactly how the loop distinguishes "interrupted by our signal" from "real error."
2. Count exits by phase. Split the histogram into "before first userspace login prompt" vs "idle after boot." Idle should be near-zero exits — proof the in-kernel irqchip and IOEVENTFD fast path keep an idle guest out of userspace (preview of Lab 4.2 and the interrupts deep dive).
3. Read the aarch64 sibling. Even on an x86 host, read vstate/vcpu/aarch64.rs (or equivalent) and note how its VcpuExit handling differs (no PIO; MMIO + system-register exits). Any run-loop change you ever make must keep both arches consistent.

Validation / Self-check

Answer without notes. These gate completion.

1. Name the two layers of the vCPU run loop (the types) and which file each lives in. What does each own?
2. Which VcpuExit variants map to KVM_EXIT_IO, and which to KVM_EXIT_MMIO? Which Firecracker bus handles each, and why are they different buses?
3. For a guest read (IoIn/MmioRead), where does the data the guest will see come from — who fills the data slice, and how does it reach the guest?
4. How does the VMM thread pause a vCPU that is currently blocked inside KVM_RUN? What does KVM_RUN return, and how does the loop avoid treating that as a crash?
5. Draw one iteration of the loop from memory, including the VcpuEvent check, KVM_RUN, the exit match, and the two terminal paths (clean shutdown vs error).
6. When the emulation step hits an unrecoverable KVM error, how does that reach the VMM thread and, ultimately, the user?
7. Why is putting a trace! on every exit a non-starter for a real PR? What does that tell you about how Firecracker instruments the run loop in production (hint: METRICS)?

When you can draw the loop, fill the exit table from your branch, and show your boot histogram, you've completed Lab 4.1. Continue to Lab 4.2: VM Exits — MMIO and PIO.