Lab 4.2: VM Exits — MMIO and PIO

This is a trace-it lab that turns the exit match you read in Lab 4.1 into a precise, measured understanding of why the guest exits, what kind of exit it is, and how the data crosses the boundary. You will instrument the run loop to count and categorize exits during a boot and during disk I/O, and you will tie individual MMIO exits to specific virtio-MMIO register offsets. By the end you will be able to look at an exit address and say "that's a virtio-block QueueNotify" or "that's the serial console" without guessing.

The exit taxonomy is the single most leverage-dense thing you can understand about a VMM. Performance, correctness, and security all live here: every exit is a context switch from guest to host and a piece of host code emulating a device. Fewer exits = faster. A mishandled exit = a bug or a vulnerability.

Background

There are two fundamentally different ways a guest "talks to a device," and they produce two different VM exits.

PIO (port-mapped I/O) — the x86 in/out instructions, addressing a separate 16-bit port space (0x0000–0xFFFF). The guest executes out 0x3f8, al to send a byte to the serial port. KVM intercepts it and exits with KVM_EXIT_IO. Firecracker uses PIO only for legacy devices: the 16550 serial UART at 0x3f8, and a partial i8042 keyboard controller at 0x60/0x64 (only enough to catch a guest reset). No virtio device uses PIO in Firecracker.

MMIO (memory-mapped I/O) — the guest reads/writes a physical memory address that isn't backed by RAM. The CPU's page tables / EPT have no mapping there, so the access faults out to KVM, which exits with KVM_EXIT_MMIO. Firecracker uses MMIO for virtio devices: each virtio-MMIO device occupies a fixed block of guest-physical address space (a register window), and the guest's reads/writes of those registers become MMIO exits. This is the default transport — virtio-mmio, no PCI.

  guest instruction            CPU/KVM detects                 VM exit          Firecracker bus
 ───────────────────────────────────────────────────────────────────────────────────────────────
  out 0x3f8, al           →    I/O to port space          →    KVM_EXIT_IO   →   PortIODeviceManager
  in  al, 0x3f8           →    I/O from port space        →    KVM_EXIT_IO   →   PortIODeviceManager
  mov [0xd0000050], eax   →    write to unmapped phys     →    KVM_EXIT_MMIO →   MMIODeviceManager
  mov eax, [0xd0000060]   →    read  from unmapped phys   →    KVM_EXIT_MMIO →   MMIODeviceManager

How the data crosses the boundary differs by exit type, and you must know this cold:

kvm-ioctls hides the data_offset arithmetic: it hands you a &[u8]/&mut [u8] already pointing at the right bytes. But you should know what it's hiding — see kvm-ioctls & kvm-bindings.

Companion reading: vCPU run loop & VM exits, the MMIO bus & device manager, virtio transport (MMIO), serial console & legacy devices.

Why This Lab Matters for Contributors

"The guest is slow," "this device hangs," "CPU usage is 100% while idle" — these are exit problems. A device that exits on every byte instead of batching, an idle guest spinning on a polled register, a virtio fast-path that fell back to userspace: all show up as exit counts. Maintainers reason about Firecracker performance in exits per operation. And the security argument for the minimal device model is literally "fewer exit handlers = less host attack surface." This lab gives you the measurement skill those conversations require, and it's the direct setup for the virtio labs in Level 7.

Prerequisites

• You completed Lab 4.1 — you can find the exit match and add temporary tracing.
• You can boot a microVM by hand and attach a second drive (Lab 1.3).
• A reading log:

mkdir -p ~/fc-notes ; : > ~/fc-notes/reading-log-4.2.md

Note: The exact MMIO base address and per-device layout are computed at boot and not a fixed constant you should memorize. Find the layout on your branch with the rgs below; the offsets within a device's register window are from the virtio-MMIO spec and are stable.

Step 1 (10 min) — Find the two buses

A PIO exit and an MMIO exit go to two different Bus objects, keyed by address.

# The bus abstraction and the two managers.
rg -n "struct Bus|impl Bus|fn read|fn write|BusDevice" src/vmm/src/devices/
rg -n "PortIODeviceManager|MMIODeviceManager" src/vmm/src/device_manager/
# Where the run loop hands an exit to a bus.
rg -n "\.read\(|\.write\(|io_bus|mmio_bus|pio_bus" src/vmm/src/vstate/vcpu/x86_64.rs

In your log, write the call from the VcpuExit::IoOut arm to the PIO bus, and from VcpuExit::MmioWrite to the MMIO bus. Note that a Bus is essentially an interval map: address → BusDevice. The dispatch is "find the device whose range contains this address, offset into it, call read/write."

Step 2 (10 min) — Map the PIO devices

The PIO space in Firecracker is tiny. Find what's registered:

rg -n "0x3f8|0x2f8|0x3e8|0x2e8|SERIAL|com1|0x60|0x64|i8042|I8042" src/vmm/src/ \
  --glob '!**/tests/**'

Note: That's essentially it. There is no PCI config space (0xcf8/0xcfc), no legacy timer zoo, no BIOS. The smallness is the design — see the minimal device model philosophy.

Step 3 (15 min) — Map the virtio-MMIO register window

Every virtio device is a block of MMIO registers. Find where Firecracker places them and the register offsets:

# Where MMIO devices get their guest-physical address allocated.
rg -n "MMIO_LEN|mmio.*base|allocate|next_mmio|GuestAddress" src/vmm/src/device_manager/
# The virtio-MMIO register offsets (the constants below come from the spec).
rg -n "MAGIC|MagicValue|0x74726976|QUEUE_NOTIFY|INTERRUPT_STATUS|0x050|0x060|0x070" \
  src/vmm/src/devices/virtio/

The virtio-MMIO register map (offsets relative to a device's base address) you'll match exits against:

The two offsets you'll see most at runtime are 0x050 (QueueNotify) — the guest kicking the device after queuing a request — and 0x060 (InterruptStatus) — the guest checking why it was interrupted. Memorize those two. Full treatment in the virtio-MMIO transport deep dive and virtqueues.

Step 4 (20 min) — Instrument: count and categorize exits

Add temporary instrumentation that categorizes each exit. Find the exit match and add a counter. The cleanest approach is a small per-variant tally you print on shutdown; the quick approach is a trace! you post-process with uniq -c (as in Lab 4.1). Here is the quick version, extended to decode the virtio-MMIO offset:

#![allow(unused)]
fn main() {
// TEMPORARY (Lab 4.2) — remove before any PR. Categorize and decode each exit.
match exit_reason {
    VcpuExit::IoIn(port, _) | VcpuExit::IoOut(port, _) => {
        log::trace!("EXIT PIO  port={port:#06x}");
    }
    VcpuExit::MmioRead(addr, _) | VcpuExit::MmioWrite(addr, _) => {
        // Decode the offset within a 4 KiB-aligned virtio-mmio window.
        let off = addr & 0xfff;
        let reg = match off {
            0x000 => "Magic", 0x008 => "DeviceID", 0x030 => "QueueSel",
            0x050 => "QueueNotify(kick)", 0x060 => "IntStatus",
            0x064 => "IntACK", 0x070 => "Status", _ => "other",
        };
        log::trace!("EXIT MMIO addr={addr:#x} off={off:#05x} {reg}");
    }
    other => log::trace!("EXIT {other:?}"),
}
}

Warning: Throwaway only. This is the hottest path in Firecracker. Remove it before you build anything you'll commit; never PR run-loop logging. The production way to count exits is the METRICS system — find it with rg -n "METRICS|VcpuMetrics|exit" src/vmm/src/.

Build, boot, and capture:

tools/devtool build
sudo ./build/cargo_target/x86_64-unknown-linux-musl/debug/firecracker \
  --api-sock /tmp/fc.sock --level Trace --log-path /tmp/fc.log &
# boot as in Lab 1.3 (kernel + rootfs + machine-config + InstanceStart)

# Totals by class:
grep -oE "EXIT (PIO|MMIO|[A-Za-z]+)" /tmp/fc.log | sort | uniq -c | sort -rn
# MMIO exits by decoded register:
grep "EXIT MMIO" /tmp/fc.log | grep -oE "off=0x[0-9a-f]+ [A-Za-z()]+" | sort | uniq -c | sort -rn
# PIO exits by port:
grep "EXIT PIO" /tmp/fc.log | grep -oE "port=0x[0-9a-f]+" | sort | uniq -c | sort -rn

Record the three histograms. You should see: PIO dominated by 0x3f8 (console output); MMIO dominated by QueueNotify and IntStatus/IntACK, with a burst of Status/QueueSel/QueueDesc during device setup.

Step 5 (20 min) — Watch disk I/O drive block-device exits

Now isolate one device's exits. Boot with a second drive, then generate I/O inside the guest and watch the block device's MMIO window light up.

# Add a scratch drive before InstanceStart:
truncate -s 64M /tmp/scratch.ext4 && mkfs.ext4 -q /tmp/scratch.ext4
curl -X PUT --unix-socket /tmp/fc.sock \
  --data '{"drive_id":"scratch","path_on_host":"/tmp/scratch.ext4","is_root_device":false,"is_read_only":false}' \
  http://localhost/drives/scratch
# ... InstanceStart ...

Inside the guest (over the serial console), find the second virtio-block device and force real I/O:

# In the guest:
mkfs.ext4 -q /dev/vdb && mount /dev/vdb /mnt
dd if=/dev/zero of=/mnt/f bs=1M count=16 conv=fsync
sync ; umount /mnt

Then, on the host, slice the log to the window around your dd and count the block device's exits:

grep "EXIT MMIO" /tmp/fc.log | grep "QueueNotify" | wc -l   # kicks ≈ batched requests
grep "EXIT MMIO" /tmp/fc.log | grep "IntStatus"   | wc -l   # interrupt-status reads

In your log, answer:

1. How many QueueNotify (kicks) did 16 MiB of dd produce? It should be far fewer than 16,384 (one per 4 KiB) — virtio batches a chain of descriptors per kick. This is the whole point of virtqueues: amortize the exit.
2. Which is more frequent, QueueNotify (guest→device) or the device→guest interrupt path (IntStatus/IntACK)? Why?

Tip — the IOEVENTFD fast path. In production, the QueueNotify write doesn't even cause a userspace exit: Firecracker registers it with KVM_IOEVENTFD so the kick wakes an eventfd the VMM thread's EventManager is polling, without the vCPU leaving KVM_RUN. If you see QueueNotify MMIO writes in your trace, that's because IOEVENTFD short-circuits the eventfd but the address may still register an exit on some paths — go read where register_ioevent is called and reason about what should and shouldn't reach your match:

rg -n "register_ioevent|IoEventAddress|KVM_IOEVENTFD|ioeventfd" src/vmm/src/

This is the seam between this lab and the interrupts & irqchip deep dive.

Step 6 (10 min) — Tie an MMIO address back to a specific device

You have addresses; now name the device. The device manager knows which guest-physical range belongs to which device. Find the mapping:

rg -n "register_mmio|add_device|MMIODeviceInfo|DeviceType|virtio_device_id" src/vmm/src/device_manager/

The guest also tells you, via the kernel command line, where each virtio-MMIO device lives. From inside the guest:

# In the guest — the kernel cmdline lists virtio_mmio.device=SIZE@ADDR:IRQ entries.
cat /proc/cmdline | tr ' ' '\n' | grep virtio_mmio
# And the bound devices:
ls /sys/devices/platform/ | grep -i virtio
dmesg | grep -i virtio

Match an ADDR from /proc/cmdline to the high bits of an MMIO exit address in your host log. That's the device. Record one full mapping (e.g. "0xd0002000 → virtio-block vdb → IRQ N").

Implementation Requirements / Deliverables

• The PIO device map (ports → devices) and the virtio-MMIO register-offset table, both confirmed against your branch.
• Three boot-time histograms: exits by class, MMIO exits by decoded register, PIO exits by port.
• A measured QueueNotify count for a 16 MiB dd, with a one-line explanation of why it's far below 16,384.
• One MMIO address traced end to end: exit address → /proc/cmdline virtio_mmio.device= entry → named guest device.
• A note on where KVM_IOEVENTFD is registered and what it removes from the exit path.
• Instrumentation removed (git diff clean).

Troubleshooting

Every MMIO exit decodes as "other"

Your & 0xfff assumption (4 KiB-aligned device windows) may not match the layout on your branch, or the window size differs. Check the device window length: rg -n "MMIO_LEN|0x1000" src/vmm/src/. Adjust the mask to the real window size and re-decode.

I see no QueueNotify MMIO writes at all, only reads

That's IOEVENTFD doing its job — the kick is short-circuited to an eventfd and never reaches your match as an MMIO write. Confirm by finding the register_ioevent call for the queue-notify address. This is expected and correct; note it rather than "fixing" it.

The guest has no /dev/vdb

The second drive wasn't attached before InstanceStart (drives are pre-boot config) or the guest kernel lacks virtio-block. Confirm the PUT /drives/scratch returned 204 before InstanceStart, and grep CONFIG_VIRTIO_BLK the guest kernel config in resources/guest_configs/.

Exit counts are astronomically high and the boot crawls

That's the trace!-per-exit tax, plus possibly a polled register. It's tolerable for one observation run. If it's truly stuck, your guest may be spinning on a register that's returning a wrong value — check that you didn't alter the read path, only logged it.

Expected Output

$ grep -oE "EXIT (PIO|MMIO|[A-Za-z]+)" /tmp/fc.log | sort | uniq -c | sort -rn
   9021 EXIT MMIO
   1188 EXIT PIO
      1 EXIT Hlt

$ grep "EXIT MMIO" /tmp/fc.log | grep -oE "off=0x[0-9a-f]+ [A-Za-z()]+" | sort | uniq -c | sort -rn
   4502 off=0x050 QueueNotify(kick)
   3110 off=0x060 IntStatus
    980 off=0x064 IntACK
    ...

Numbers vary by kernel, rootfs, and device set; the shape — MMIO-dominated, kicks and interrupt status on top, console PIO second, a single terminal Hlt — is the result you're confirming.

Stretch Goals

1. Exits per kilobyte. Run dd at 4 MiB, 16 MiB, 64 MiB and plot QueueNotify count vs bytes. Is it linear? Sub-linear (better batching at scale)? This is the beginning of I/O-engine reasoning (io-engines).
2. Idle exit floor. After boot, leave the guest idle for 30 s and count exits in that window. A well-behaved idle guest should generate almost none — proof the in-kernel timer (PIT/LAPIC) and IOEVENTFD keep idle guests out of userspace. Any steady stream of exits at idle is a polling bug worth a real issue.
3. PIO write coalescing. The serial console exits per byte (out 0x3f8). Count console exits for a dmesg dump and reason about why the serial console is not a virtqueue and what that costs. (This is exactly why heavy logging slows a microVM.)
4. Add a production counter. Instead of trace!, wire a real METRICS counter for one exit class and read it via FlushMetrics. Compare it to your uniq -c. This is the shape of a mergeable observability PR.

Validation / Self-check

Answer without notes. These gate completion.

1. What guest instruction causes a KVM_EXIT_IO, and what causes a KVM_EXIT_MMIO? Give a concrete example address/port for each in Firecracker.
2. For each exit type, name the kvm_run fields that carry the request, and say where the data lives (the shared-page io.data_offset vs inline mmio.data[8]). What does kvm-ioctls hand you instead?
3. Which Firecracker bus handles PIO and which handles MMIO? How does each find the right device for an address?
4. What are the two virtio-MMIO register offsets you'll see most at runtime, and what does each mean?
5. A 16 MiB dd produced far fewer than 16,384 kicks. Explain the mechanism that batches requests per kick.
6. What does KVM_IOEVENTFD change about the QueueNotify exit, and why does that matter for performance? Where is it registered?
7. Why is an idle guest that still generates a steady stream of MMIO exits a bug, and how does the in-kernel irqchip relate to that?

When you can produce the three histograms, trace one MMIO address to a named guest device, and explain the IOEVENTFD short-circuit, you've completed Lab 4.2. Continue to Lab 4.3: Inspect and Apply a CPU Template.