The virtio-MMIO Transport

Every virtio device in Firecracker — block, net, vsock, balloon, rng — speaks the same wire protocol to its guest driver: virtio. But virtio is split into two halves. The device-type half (how a block request is laid out, what a net header contains) is per-device. The transport half — how the guest finds the device, negotiates features, sets up queues, kicks the device, and receives interrupts — is shared. Firecracker's default transport is virtio-MMIO: each device is a small block of memory-mapped registers at a fixed guest physical address plus one interrupt line. There is no PCI bus, no enumeration, no configuration space probing — the guest is simply told where each device lives via the kernel command line. This chapter dissects that register block, the device/driver split, feature negotiation, the status state machine, how a guest's MMIO access becomes a VM exit that lands on the right device, and how virtio-PCI (behind --enable-pci) differs.

After this chapter you will be able to: read the virtio-MMIO register map and say what each access does; explain the ACKNOWLEDGE → DRIVER → FEATURES_OK → DRIVER_OK handshake; trace a guest register write from a KVM_EXIT_MMIO to a specific device's write method; and explain why Firecracker chose MMIO over PCI.

Note: virtio-MMIO is the transport, not the device. The same MmioTransport wraps a block device or a net device indifferently; it knows nothing about sectors or Ethernet frames. Keep the transport (this chapter) and the queue mechanics (virtqueues.md) and the device logic (virtio-block.md) as three separate layers in your head.

Where the transport lives

# The MMIO transport wrapper — the object that owns the register block for one device.
rg -n "struct MmioTransport|impl MmioTransport|fn bus_read|fn bus_write" src/vmm/src/devices/virtio/
# The register offset constants (Magic, Version, DeviceID, QueueSel, ...).
rg -n "MMIO_MAGIC_VALUE|0x7472|VENDOR_ID|fn read|fn write" src/vmm/src/devices/virtio/mmio.rs
# How devices get placed on the MMIO bus and assigned an address + IRQ.
rg -n "MMIODeviceManager|register_mmio|allocate|IrqTrigger" src/vmm/src/device_manager/

The transport is one type — call it MmioTransport (verify the name on your branch) — that holds a reference to the inner VirtioDevice plus the transport-level state: the selected queue, the device status byte, the negotiated feature bits, and the interrupt status. The MMIODeviceManager owns a collection of these, each at a distinct guest physical address, and routes bus accesses to the right one.

The register block

A virtio-MMIO device occupies a fixed-size window (4 KiB is typical) of guest physical address space. The guest driver reads and writes 32-bit registers at fixed offsets within that window. These offsets are defined by the virtio specification, not by Firecracker — they are identical across every compliant VMM, which is why an unmodified Linux virtio_mmio driver works against Firecracker.

        guest physical address space
   ┌────────────────────────────────────────┐
   │  ...                                    │
   │  virtio_mmio.device=0x1000@0xd0000000:5 │  ← cmdline tells the guest: a device,
   │      ┌──────────────────────────┐       │     4 KiB window @ 0xd0000000, IRQ 5
   │      │ 0x000 MagicValue 'virt'  │       │
   │      │ 0x008 DeviceID = 2 (blk) │       │
   │      │ 0x030 QueueSel           │       │
   │      │ 0x050 QueueNotify  ◄── kick (write triggers eventfd via KVM_IOEVENTFD)
   │      │ 0x060 InterruptStatus    │       │
   │      │ 0x070 Status (state mach)│       │
   │      │ 0x080 QueueDesc addr     │       │
   │      │ 0x100 config (capacity)  │       │
   │      └──────────────────────────┘       │
   └────────────────────────────────────────┘

Note: The guest is never handed this map. It learns it from the kernel command line on x86 (virtio_mmio.device=SIZE@ADDR:IRQ, repeated per device) or from an FDT node on aarch64. There is no enumeration step. Find where Firecracker emits those cmdline fragments:

rg -n "virtio_mmio.device|add_virtio_mmio_device|cmdline" src/vmm/src/device_manager/ src/vmm/src/arch/

The device/driver split and feature negotiation

virtio is a contract between a device (Firecracker's emulation, the host side) and a driver (the guest kernel module). The two must agree on a feature set before any I/O happens. The device advertises a 64-bit feature bitmask via DeviceFeatures; the driver reads it, masks it down to the bits it understands and wants, and writes the result back via DriverFeatures. The intersection is the negotiated feature set, frozen at FEATURES_OK.

# What features each device offers — every virtio device implements avail_features().
rg -n "avail_features|fn features|acked_features|VIRTIO_F_|AVAIL_FEATURES" src/vmm/src/devices/virtio/

Feature bits you will meet constantly (the transport-level ones):

Each device type adds its own bits on top — VIRTIO_NET_F_MAC, VIRTIO_BLK_F_RO, etc. — covered in the per-device chapters. The negotiation is mandatory and ordered: the driver may not touch a queue until FEATURES_OK is set and read back successfully.

The status state machine

Status (offset 0x070) is how the driver narrates its progress to the device. The bits are cumulative — the driver ORs in the next bit and writes the whole byte. The device watches for the transition and reacts.

rg -n "ACKNOWLEDGE|DRIVER\b|FEATURES_OK|DRIVER_OK|DEVICE_NEEDS_RESET|FAILED|fn set_device_status|fn check_device_status" src/vmm/src/devices/virtio/

flowchart TD
    R["reset (Status=0)"] --> A["ACKNOWLEDGE (1): guest sees the device"]
    A --> D["DRIVER (2): guest has a driver for it"]
    D --> Feat["read DeviceFeatures, write DriverFeatures"]
    Feat --> FO["FEATURES_OK (8): driver accepts the feature set"]
    FO --> Check{device re-reads Status: is FEATURES_OK still set?}
    Check -->|no| Fail["FAILED (128): negotiation rejected"]
    Check -->|yes| Q["driver sets up queues: QueueDesc/Driver/Device + QueueReady"]
    Q --> DO["DRIVER_OK (4): driver is live — I/O may begin"]
    DO -->|device.activate| Live["device starts processing virtqueues"]

The bit that matters most to Firecracker's code is DRIVER_OK. When the driver finally ORs it in and writes Status, the transport calls the inner device's activate method. That is the moment the device wires up its eventfds, registers itself with the EventManager, and becomes ready to process queues. Before DRIVER_OK, the device is configured but inert; after it, the device is hot. Writing 0 to Status is a reset, which tears all of that down.

# The activate hook — every device implements this; it is the DRIVER_OK -> live transition.
rg -n "fn activate|fn is_activated|ACTIVATE_EVENT|register_runtime" src/vmm/src/devices/virtio/

From a guest register access to a device method

How does a guest writing to 0x050 become a Rust function call? Through a VM exit. Most virtio-MMIO registers are serviced as MMIO exits (vcpu-run-loop-and-vm-exits.md): the guest executes a store to a guest physical address that has no backing RAM, the CPU traps into KVM, KVM returns from KVM_RUN with exit_reason = KVM_EXIT_MMIO carrying phys_addr, data, len, and is_write. The vCPU thread looks the address up on the MMIO bus, finds the MmioTransport whose window contains it, computes the offset, and calls read/write.

rg -n "KVM_EXIT_MMIO|VcpuExit::MmioRead|VcpuExit::MmioWrite|mmio_bus|fn read|fn write" src/vmm/src/vstate/vcpu/ src/vmm/src/device_manager/

sequenceDiagram
    participant G as guest driver
    participant K as KVM
    participant Vc as vCPU thread
    participant Bus as MMIO bus
    participant Dev as MmioTransport/device
    participant Ev as VMM thread (EventManager)
    G->>K: store to QueueNotify (0x050)
    Note over K: address registered via KVM_IOEVENTFD
    K-->>Ev: signal the device's eventfd (no exit!)
    Ev->>Dev: EventManager wakes; device.process_queue()
    G->>K: store to Status / QueueSel (0x070 / 0x030)
    K->>Vc: KVM_RUN returns KVM_EXIT_MMIO
    Vc->>Bus: lookup phys_addr
    Bus->>Dev: transport.write(offset, data)

There is one crucial optimization on the kick path. QueueNotify writes are the hottest register access — every batch of I/O ends with one. Routing each through a full MMIO exit and bus lookup would be slow. Instead Firecracker registers the QueueNotify address with KVM via KVM_IOEVENTFD: a guest write to that exact address is converted by KVM itself into a signal on an eventfd, with no VM exit and no vCPU involvement. That eventfd is registered in the VMM thread's EventManager epoll set. So a kick wakes the VMM thread directly, off the vCPU's path. Completion interrupts go the other way: the device updates the used ring, then injects an IRQ via KVM_IRQFD (an eventfd wired to a guest interrupt line). Find both:

rg -n "KVM_IOEVENTFD|register_ioevent|IoEventAddress|KVM_IRQFD|register_irqfd|IrqTrigger|trigger" \
  src/vmm/src/device_manager/ src/vmm/src/devices/virtio/

virtio-MMIO vs virtio-PCI

rg -n "enable_pci|virtio.*pci|PciDevice|VirtioPciDevice|MsixConfig" src/vmm/src/devices/ src/firecracker/src/

The minimal-device-model philosophy (minimal-device-model-philosophy.md) explains the default: a PCI host bridge plus config space plus MSI-X is a lot of host code a malicious guest can poke at, and the boot-time enumeration costs milliseconds Firecracker does not want to spend. MMIO is "the device is there, full stop." PCI exists for workloads that need many devices or per-queue interrupt scaling, and it is opt-in.

Warning: virtio-PCI is the newer, larger-surface transport — CVE-2026-5747 was a vulnerability in the PCI transport, fixed in 1.14.4 / 1.15.1 (verify on your branch and the CHANGELOG). The MMIO default is the conservative, hardened path. Treat any PCI-transport change as security-sensitive.

Reading exercise

# 1. The transport wrapper and its read/write dispatch.
rg -n "struct MmioTransport|fn bus_read|fn bus_write|fn read|fn write" src/vmm/src/devices/virtio/mmio.rs

# 2. The register offsets — confirm Magic, QueueNotify@0x050, InterruptStatus@0x060, Status@0x070.
rg -n "0x00|0x50|0x60|0x70|MAGIC|QUEUE_NOTIFY|INTERRUPT_STATUS|STATUS" src/vmm/src/devices/virtio/mmio.rs

# 3. The status state machine and the activate hook.
rg -n "ACKNOWLEDGE|DRIVER_OK|FEATURES_OK|fn activate|fn is_activated" src/vmm/src/devices/virtio/

# 4. Feature negotiation on a real device.
rg -n "avail_features|acked_features|VIRTIO_F_VERSION_1|EVENT_IDX" src/vmm/src/devices/virtio/

# 5. The kick fast path (ioeventfd) and interrupt injection (irqfd).
rg -n "KVM_IOEVENTFD|register_ioevent|KVM_IRQFD|register_irqfd|IrqTrigger" \
  src/vmm/src/device_manager/ src/vmm/src/devices/virtio/

# 6. How the device's cmdline address fragment is generated.
rg -n "virtio_mmio.device|add_virtio_mmio_device" src/vmm/src/device_manager/ src/vmm/src/arch/

# 7. On a booted microVM, see what the guest discovered:
#    cat /proc/cmdline | tr ' ' '\n' | grep virtio_mmio
#    ls /sys/bus/virtio/devices

Answer:

1. What are the first three registers a virtio driver reads, and what does each tell it? What value must MagicValue hold and why?
2. Walk the status state machine. At which transition does the device call activate, and what does activate actually do?
3. Describe feature negotiation: which side writes DriverFeatures, and what does FEATURES_OK guarantee?
4. A guest writes to QueueNotify (0x050). Trace what happens — and explain why it does not take a VM exit, unlike a write to Status.
5. How does a guest find its virtio devices under MMIO when there is no PCI enumeration?
6. Give two reasons Firecracker defaults to MMIO over PCI, and one reason a workload might want PCI.

Common bugs and symptoms

Validation: prove you understand this

1. Draw the virtio-MMIO register block from memory with the offsets for MagicValue, DeviceID, QueueSel, QueueNotify, InterruptStatus, Status, and the queue address registers.
2. Explain the device/driver split and feature negotiation in one paragraph, naming who writes DeviceFeatures vs DriverFeatures.
3. Walk the full status state machine and say precisely what activate does and when it fires.
4. Explain why a QueueNotify write is handled by KVM_IOEVENTFD and not as an MMIO exit, and what that buys you.
5. Contrast virtio-MMIO and virtio-PCI on discovery, interrupt model, and attack surface; justify the MMIO default.
6. A guest's virtio-block device appears in /sys/bus/virtio but no reads ever complete. List the transport-level checks you would make, in order, to find where the handshake broke.

Next: Virtqueues — the split-ring data structure (descriptor table, available ring, used ring) that every transport drives. The single most important chapter in the device model.