The MMIO Bus and the Device Manager

When a guest reads or writes a device register, the CPU exits to KVM with a physical address, and that address arrives at a vCPU thread as a VcpuExit::MmioRead/MmioWrite (or IoIn/IoOut for port I/O). Something has to turn that bare address into "the virtio-block device's QueueNotify register" and call the right handler. That something is the Bus — an address-range-to-device map — and the device managers that populate it. Firecracker's DeviceManager wraps an MMIODeviceManager (the virtio-mmio and serial devices on x86 and aarch64), a PortIODeviceManager (x86 legacy PIO devices), and an ACPIDeviceManager. This chapter shows how devices are placed at fixed MMIO windows, advertised to the guest so its drivers can find them, registered during build, and how an MMIO exit is routed back to the owning device.

After this chapter you will be able to: explain how a VM-exit address becomes a device method call; locate the bus and the device managers; describe how a device's MMIO window and IRQ are allocated and advertised; and trace device registration during build_microvm_for_boot.

Note: Firecracker's default transport is virtio-mmio: each device lives at a fixed MMIO register block with a single IRQ, and the guest is told where each device is (there is no PCI bus to enumerate). The Bus is what connects a guest's access to that block back to the device object. Hold the two halves together — placement (host side) and advertisement (guest side) must agree exactly.

The Bus: address → device

# The Bus abstraction and how exits are routed through it.
rg -n "struct Bus|BusDevice|fn insert|fn read\b|fn write\b|get_device|BusRange" src/vmm/src/devices/ src/vmm/src/device_manager/

A Bus is essentially a sorted map from [start, start+len) address ranges to device handles. On an MMIO exit, the vCPU thread asks the MMIO bus "who owns this address?", gets the device and the offset within its range, and calls read/write on it.

flowchart TD
    Exit["VcpuExit::MmioWrite(addr, data)"] --> Lookup["mmio_bus.get_device(addr)"]
    Lookup -->|found| Dev["BusDevice + offset = addr - base"]
    Lookup -->|not found| Err["no device -> ignored / error"]
    Dev --> Call["device.write(offset, data)"]
    Call --> Effect["e.g. QueueNotify -> kick the virtqueue"]

There are two buses: the MMIO bus (driven by MmioRead/MmioWrite) and the PIO bus (driven by IoIn/IoOut, x86 only). The vCPU run loop dispatches to the right one by exit type — see vcpu-run-loop-and-vm-exits.md.

The device managers

DeviceManager is the umbrella; under it sit the specific managers.

rg -n "struct DeviceManager|MMIODeviceManager|PortIODeviceManager|ACPIDeviceManager" src/vmm/src/device_manager/

rg -n "fn register_mmio_virtio|fn register_virtio_device|register_mmio_serial|allocate|MMIO_LEN|mmio_base" src/vmm/src/device_manager/

The MMIODeviceManager is where the real action is for virtio. It does three things per device:

1. Allocate an MMIO window (a fixed-size block from the MMIO gap region) and an IRQ line (see interrupts-and-irqchip.md).
2. Register the device on the MMIO bus at that window, and register its interrupt eventfd with KVM via KVM_IRQFD, and its queue-notify with KVM_IOEVENTFD.
3. Record the window/IRQ so they can be advertised to the guest.

Fixed MMIO windows and advertising them to the guest

Because there is no PCI enumeration, the guest cannot discover devices — it must be told. Each virtio-mmio device occupies a fixed-size window in the MMIO gap (the reserved address range below 4 GiB — see the-boot-sequence.md), and Firecracker tells the guest about each window and IRQ.

   guest physical address space
   ┌─────────────────────────────────────────────────────────────┐
   │ low RAM │ ... │  MMIO gap (below 4 GiB)         │ high RAM    │
   │         │     │ ┌────────┐ ┌────────┐ ┌────────┐│  (>4 GiB)   │
   │         │     │ │virtio0 │ │virtio1 │ │virtio2 ││             │
   │         │     │ │@ A0:I0 │ │@ A1:I1 │ │@ A2:I2 ││             │
   │         │     │ └────────┘ └────────┘ └────────┘│             │
   └─────────────────────────────────────────────────────────────┘

rg -n "virtio_mmio.device|add_virtio_mmio|cmdline.*virtio|fdt.*virtio|MMIO_CFG|add_mmio_node" src/vmm/src/ src/vmm/src/arch/

The two sides must match: the bus registration (host) and the advertisement (guest) must use the same base address and IRQ, or the guest's driver will probe an address the bus doesn't route to that device, or wait on an interrupt line the device never raises. This is the single most common device-bring-up bug, and it is why the MMIODeviceManager records the window/IRQ at registration and replays them into the cmdline/FDT.

Registration during build

Device registration happens in the-boot-sequence.md, step 8, after memory, the VM fd, the irqchip, and the vCPUs exist, and before the vCPUs are released.

rg -n "attach_block_devices|attach_net_devices|attach_vsock|attach_balloon|attach_entropy|attach_legacy|register" src/vmm/src/builder.rs src/vmm/src/device_manager/

flowchart LR
    B["build_microvm_for_boot"] --> L["create legacy devices (serial, i8042)"]
    L --> Blk["attach block devices"]
    Blk --> Net["attach net devices"]
    Net --> Etc["attach vsock / balloon / rng (as configured)"]
    Etc --> Reg["MMIODeviceManager: allocate window+IRQ, insert on bus, irqfd+ioeventfd"]
    Reg --> Adv["advertise: cmdline virtio_mmio.device / FDT node / ACPI"]

For each configured device, the builder calls an attach_* helper that constructs the device object, hands it to the MMIODeviceManager for window/IRQ allocation and bus insertion, wires the eventfds to KVM, and records the advertisement data. The serial console and i8042 (legacy) are attached through their own paths (serial-console-and-legacy-devices.md). After all devices are registered and advertised, the boot params/cmdline/FDT are finalized and the vCPUs start.

Reading exercise

# 1. The Bus.
rg -n "struct Bus|BusDevice|fn insert|get_device|BusRange|fn read\b|fn write\b" src/vmm/src/devices/ src/vmm/src/device_manager/

# 2. The device managers.
rg -n "struct DeviceManager|MMIODeviceManager|PortIODeviceManager|ACPIDeviceManager" src/vmm/src/device_manager/

# 3. Window + IRQ allocation and KVM wiring.
rg -n "allocate|mmio_base|MMIO_LEN|register_irqfd|register_ioevent|register_mmio" src/vmm/src/device_manager/

# 4. Advertisement to the guest.
rg -n "virtio_mmio.device|add_virtio_mmio|fdt.*virtio|cmdline" src/vmm/src/ src/vmm/src/arch/

# 5. The attach_* calls in the builder.
rg -n "attach_block_devices|attach_net_devices|attach_vsock|attach_balloon|attach_entropy|attach_legacy" src/vmm/src/builder.rs

# 6. In a booted guest, see the advertised devices:
#    cat /proc/cmdline      (the virtio_mmio.device= tokens)
#    ls /sys/devices/platform/*.virtio_mmio

Answer:

1. How does a VcpuExit::MmioWrite(addr, data) become a call on a specific device? Name each step.
2. What three things does the MMIODeviceManager do when it registers a virtio device?
3. Why must the guest be told where devices are, and what are the two pieces of information per device?
4. What goes wrong if the advertised base/IRQ disagree with the bus registration?
5. Name the three managers under DeviceManager and what each owns.
6. Where in the boot sequence does device registration sit, and why there?

Common bugs and symptoms

Validation: prove you understand this

1. Draw the path from an MMIO exit address to a device write, including the bus lookup and offset.
2. Explain the Bus data structure and how get_device finds the owner of an address.
3. List the three managers under DeviceManager and the bus each drives.
4. Explain placement vs advertisement and why they must agree, with the x86 cmdline token format.
5. Describe what MMIODeviceManager does for one virtio device, including the two KVM eventfd registrations.
6. Place device registration in the boot sequence and justify its position relative to memory, irqchip, and vCPU start.

Next: virtio Transport (MMIO) — what actually lives inside each device's MMIO register block, and the virtio status/feature state machine.