Lab 7.2: Virtqueues and the MMIO Transport

Background

Lab 7.1 traced an I/O through the device model. This lab slows down and dissects the two structures that make the model work: the virtio-MMIO transport (how a driver finds and configures a device) and the split virtqueue (the shared-memory rings the I/O actually travels on). This is a mechanics lab — you will read the spec-defined register offsets straight out of Firecracker's code, walk the exact byte layout of virtq_desc/virtq_avail/virtq_used in guest memory, follow how the driver configures a queue from a cold start, and understand how ioeventfd/irqfd make the kick and interrupt cheap. Optionally, you will dump a live virtqueue out of a running guest's memory and read the rings by hand.

If Lab 7.1 was "watch the machine run," this lab is "take the cover off and name every gear." A contributor who can recite the MMIO register map and the descriptor-ring layout can read any device handler in the tree and can reason about an entire category of bugs — index wraparound, a missing WRITE-flag check, a register write handled at the wrong offset, a queue that never becomes ready.

Why This Lab Matters for Contributors

• The MMIO register map and the virtqueue layout are spec-fixed: they do not change between branches, so this is durable knowledge you will use for years.
• Most virtio correctness bugs are at this layer: a register handled at the wrong offset, an off-by-one in the ring index, a descriptor chain not validated against WRITE. Seeing the raw layout lets you spot them.
• It de-mystifies the virtio-transport-mmio and virtqueues deep dives — read both alongside this lab.

Prerequisites

• Lab 7.1 done — you have traced a block I/O and found the Queue and the block device's handler.
• A built firecracker and a bootable kernel + rootfs.
• Verify the transport and queue code exist:

rg -n "MmioTransport"            src/vmm/src/devices/virtio/
rg -n "struct Queue\b"           src/vmm/src/devices/virtio/
rg -n "0x74726976\|MAGIC"        src/vmm/src/devices/virtio/   # the MagicValue 'virt'

The virtio-MMIO Register Block

Each virtio-MMIO device is a fixed window of registers in guest physical memory — by default 4 KiB (0x1000) wide. The guest's virtio-mmio driver finds the window because Firecracker told it the address: on x86 via the kernel command line (virtio_mmio.device=0x1000@0xd0000000:5), on aarch64 via an FDT node. The register offsets are defined by the virtio spec and are identical everywhere.

Find these offsets in the code rather than trusting the table — the values are spec-fixed but the constant names drift:

# The MagicValue and the offset constants:
rg -n "0x74726976" src/vmm/src/devices/virtio/
rg -n "MMIO_MAGIC_VALUE\|MMIO_VERSION\|MMIO_DEVICE_ID\|QUEUE_NOTIFY\|QUEUE_SEL\|INTERRUPT_STATUS\|STATUS" \
   src/vmm/src/devices/virtio/

# Where the MmioTransport dispatches a guest read/write by offset:
rg -n "fn read\b\|fn write\b\|match offset\|0x00 =>\|0x70 =>\|0x50 =>" \
   src/vmm/src/devices/virtio/

Note: Firecracker's MmioTransport is the adapter between the bus (which hands it a (offset, data, is_write) from a KVM_EXIT_MMIO) and the VirtioDevice. A read of MagicValue returns a constant; a write to QueueSel selects a queue; a write to DriverFeatures records an accepted feature window; a write to Status drives the state machine and, on DRIVER_OK, calls the device's activate(). Reading the read/write dispatch is the single most instructive thing in this lab.

The Status State Machine and Feature Negotiation

The Status register (0x070) is a bitfield the driver writes, one bit at a time, to walk a strict handshake. Firecracker's transport enforces the order and acts on the transitions.

flowchart LR
    R[reset / Status=0] -->|driver sets ACKNOWLEDGE| A[ACKNOWLEDGE=1]
    A -->|driver sets DRIVER| D[DRIVER=2]
    D -->|driver reads DeviceFeatures,<br/>writes DriverFeatures| FN[feature negotiation]
    FN -->|driver sets FEATURES_OK| FO[FEATURES_OK=8]
    FO -->|device confirms FEATURES_OK still set| OK1[features locked]
    OK1 -->|driver writes queue addrs,<br/>QueueReady=1| QR[queues live]
    QR -->|driver sets DRIVER_OK| DOK["DRIVER_OK=4 -> activate()"]
    A -.->|on any failure| F[FAILED=128]
    D -.-> F
    FO -.-> F

Feature negotiation rides through DeviceFeatures/DriverFeatures: the device advertises a 64-bit feature mask (read in two 32-bit windows via DeviceFeaturesSel), the driver writes back the subset it accepts. Key feature bits you will see:

Find them:

rg -n "VIRTIO_F_VERSION_1\|VIRTIO_F_EVENT_IDX\|VIRTIO_F_INDIRECT_DESC\|VIRTIO_F_RING_PACKED\|avail_features\|fn features\|fn ack_features" \
   src/vmm/src/devices/virtio/

Warning: When the guest sets FEATURES_OK, the device must re-read Status and confirm the bit is still set; if the device cannot support the accepted feature set it leaves the bit clear, and the driver aborts. A device that blindly accepts any feature subset, or that reads a queue address before QueueReady, is a real bug class. The order in the state machine is not advisory.

The Split Virtqueue Layout (byte for byte)

Once the driver has written the three queue addresses and QueueReady=1, three structures sit in guest memory. All multi-byte fields are little-endian.

DESCRIPTOR TABLE  @ QueueDesc address,  QueueNum entries, 16 bytes each:
  offset  size  field
   +0      8     addr   (guest physical address of the buffer)
   +8      4     len    (buffer length in bytes)
   +12     2     flags  (bit0 NEXT=1, bit1 WRITE=2, bit2 INDIRECT=4)
   +14     2     next   (index of the next descriptor, valid iff NEXT set)

AVAILABLE RING    @ QueueDriver address  (driver -> device):
  +0   2   flags
  +2   2   idx           (driver bumps this after publishing a chain head)
  +4   2*N ring[QueueNum] (ring[idx % QueueNum] = head descriptor index)
  ...      used_event    (2 bytes, present iff EVENT_IDX negotiated)

USED RING         @ QueueDevice address  (device -> driver):
  +0   2   flags
  +2   2   idx           (device bumps this after completing a chain)
  +4   8*N ring[QueueNum] each: { u32 id; u32 len; }  (id = head descriptor; len = bytes written)
  ...      avail_event   (2 bytes, present iff EVENT_IDX negotiated)

A worked example — a block read with three descriptors:

desc[0] = { addr=H, len=16, flags=NEXT,        next=1 }   # virtio_blk_req header (device READS)
desc[1] = { addr=D, len=4096, flags=NEXT|WRITE, next=2 }  # data buffer (device WRITES)
desc[2] = { addr=S, len=1,  flags=WRITE,        next=0 }  # status byte (device WRITES)

driver: avail.ring[avail.idx % N] = 0   (head = desc index 0)
driver: avail.idx += 1
driver: write QueueNotify = 0            (kick queue 0)

device: sees avail.idx moved; pop chain starting at desc[0]
device: read header at H -> {type=IN, sector=2048}
device: pread(fd, buf, 4096, 2048*512) ; copy buf into guest address D
device: write VIRTIO_BLK_S_OK to guest address S
device: used.ring[used.idx % N] = { id=0, len=4096 }
device: used.idx += 1
device: InterruptStatus |= 1 ; raise IRQ (irqfd)

driver: interrupt; reaps used.ring; sees id=0 done, len=4096; read() returns

Confirm Firecracker reads these structures through vm-memory (it never trusts a guest pointer blindly — every access is a checked read_obj/write_obj against GuestMemoryMmap):

rg -n "read_obj\|write_obj\|GuestAddress\|checked_offset\|DescriptorChain\|avail_idx\|used_idx" \
   src/vmm/src/devices/virtio/

See vm-memory for why every guest access is bounds-checked — a guest descriptor addr/len is attacker-controlled, and an unchecked read is a host memory-safety bug.

How ioeventfd and irqfd Make It Fast

The doorbell (QueueNotify) and the completion interrupt are the two crossings between guest and host on the data path. KVM gives Firecracker two primitives to make each one cheap:

flowchart LR
    subgraph Kick path
      GW[Guest writes QueueNotify] --> IOE{ioeventfd registered<br/>for this addr?}
      IOE -->|yes| EFD[eventfd signaled in kernel<br/>NO userspace exit] --> EM[EventManager epoll wakes]
      IOE -->|no| EXIT[KVM_EXIT_MMIO -> bus -> transport] --> EM
    end
    subgraph Interrupt path
      DEV[Device done] --> IRQFD[write irqfd eventfd] --> INJ[KVM injects guest IRQ]
    end

• ioeventfd (KVM_IOEVENTFD): Firecracker registers the queue's notification address with KVM so that a guest write of a specific value to QueueNotify signals an eventfd inside the kernel — the vCPU thread does not exit to userspace. The EventManager epoll loop is already waiting on that eventfd, so the kick wakes the device directly. Without it, every kick is a KVM_EXIT_MMIO, a userspace dispatch through the bus, and a transport call.
• irqfd (KVM_IRQFD): the reverse direction. The device writes an eventfd, and KVM injects the corresponding guest interrupt — no KVM_RUN round trip to deliver it.

Find the registrations (they live in the device manager / transport, not the device):

rg -n "register_ioevent\|IoEventAddress\|KVM_IOEVENTFD\|register_irqfd\|KVM_IRQFD\|irq_evt\|interrupt_evt" \
   src/vmm/src/

Tip: This is why Firecracker can sustain real I/O without burning the host CPU on VM exits. The only required exit per I/O batch is zero on the kick side (ioeventfd) and zero on the interrupt side (irqfd) — the cost is one eventfd signal each way. See interrupts-and-irqchip and the-event-manager.

Step-by-Step Tasks

Step 1: Read the transport's read/write dispatch

Open the MmioTransport and read the function that handles a guest register access. Map each offset in the table above to the branch that handles it.

rg -n "MmioTransport" src/vmm/src/devices/virtio/
# Open the file the above points at and read its read()/write() methods.
rg -n "fn read\b\|fn write\b" $(rg -l "MmioTransport" src/vmm/src/devices/virtio/ | head -1)

In your reading log, answer: where is MagicValue returned? Where does a write to QueueSel change which queue subsequent writes target? Where does a write to Status reaching DRIVER_OK call activate()? Where is QueueNotify forwarded to the device's queue?

Step 2: Inspect the cmdline / FDT that advertises the device

The guest only knows where the register block is because Firecracker told it. Find where the virtio_mmio.device=... string (x86) or the FDT node (aarch64) is built.

# x86: the kernel cmdline parameter the guest's virtio-mmio driver parses.
rg -n "virtio_mmio.device\|virtio_mmio" src/vmm/src/

# aarch64: the FDT node for each MMIO device.
rg -n "virtio_mmio\|fdt\|FdtWriter\|create_virtio_node\|compatible" src/vmm/src/arch/aarch64/

Boot a microVM (as in Lab 7.1) and, inside the guest, read the actual cmdline and the discovered devices:

# Inside the guest:
cat /proc/cmdline                       # find the virtio_mmio.device=SIZE@ADDR:IRQ entries
ls /sys/bus/platform/devices/ | grep virtio   # the discovered MMIO devices
dmesg | grep -i virtio                  # the driver binding to each

Match the ADDR in /proc/cmdline to where the MMIODeviceManager placed the block device's register window. See the-mmio-bus-and-device-manager.

Step 3: Read the device's feature advertisement

Pick the block device and read what features it advertises and how it acks the driver's choice.

rg -n "fn features\|avail_features\|fn ack_features\|acked_features" src/vmm/src/devices/virtio/block/

Confirm VIRTIO_F_VERSION_1 is advertised (it must be, for the v2 transport) and note any device-specific features (e.g. VIRTIO_BLK_F_FLUSH, VIRTIO_BLK_F_RO). Cross-reference the virtio-block deep dive.

Step 4: Read the Queue's index bookkeeping

The most bug-prone part of any virtqueue implementation is the index arithmetic. Read how Firecracker's Queue tracks next_avail, computes ring[idx % size], and updates the used ring.

rg -n "next_avail\|next_used\|wrapping_add\|% self.size\|& (size - 1)\|fn pop\|fn add_used\|fn is_empty" \
   src/vmm/src/devices/virtio/

Answer: how does the queue detect new available entries (compare avail.idx to its own cursor)? How does it mask the index into the ring (modulo vs. power-of-two AND)? Where could an off-by-one or a wraparound bug live? This is exactly the shape of real virtio correctness issues.

Step 5 (optional, advanced): Dump a live virtqueue from guest memory

This is the payoff: read the rings out of a running guest's RAM by hand. It is fiddly and host-specific, so it is optional — but doing it once makes the layout permanent knowledge.

Approach (one of several):

1. Boot a microVM with one drive and a tiny memory size so the address space is easy to reason about. Note the block device's register address from /proc/cmdline.
2. From your trace in Lab 7.1 (or by adding a one-shot log line in the block device's activate), print the guest physical addresses of the descriptor table, avail ring, and used ring that the driver wrote (the QueueDesc/QueueDriver/QueueDevice values the transport received).
3. Firecracker's guest memory is a host mmap; the device already reads these structures via vm-memory. The simplest reliable "dump" is to add a temporary debug print in the device that walks the descriptor table and the rings using the same GuestMemoryMmap accessors and prints addr/len/flags/next for each descriptor and the current avail.idx/used.idx. This avoids any host-side address translation and uses the exact path the device uses.

#![allow(unused)]
fn main() {
// Temporary, inside the block device once a chain is popped (remove before any PR):
log::warn!("[vq] avail.idx={} used.idx={}", queue.avail_idx(mem), queue.used_idx(mem));
for d in chain.clone() {
    log::warn!("[vq] desc addr={:#x} len={} flags={:#x} next={}",
               d.addr.0, d.len, d.flags, d.next);
}
}

Find the real accessor names first (avail_idx/used_idx/the DescriptorChain fields differ by branch):

rg -n "fn avail_idx\|fn used_idx\|pub addr\|pub len\|pub flags\|pub next" src/vmm/src/devices/virtio/

Run a guest dd ... iflag=direct and read the dumped descriptors: you should see the header/data/status three-descriptor chain, with flags showing NEXT on the first two and WRITE on the data and status descriptors. That is the byte layout from this lab, alive in memory.

Warning: Anything that reads or prints guest memory is debug-only and never goes near a PR. It also runs on the VMM thread — keep it to a few lines and remove it after.

Implementation Requirements / Deliverables

• An annotated copy of the MMIO register table, each offset linked to the line in the transport's read/write dispatch that handles it (with the rg you used).
• A written walk of the status state machine and feature negotiation for the block device: the bit sequence the driver writes and what the device does at FEATURES_OK and DRIVER_OK.
• The virtio_mmio.device=... line from a booted guest's /proc/cmdline, matched to where the MMIODeviceManager placed the register window.
• A description of the descriptor-ring layout in your own words, with the three-descriptor block-read example.
• An explanation of how ioeventfd and irqfd remove VM exits on the kick and interrupt paths, with the rg that finds their registration.
• (Optional) A dump of a live virtqueue's descriptors and ring indices.

Troubleshooting

rg for the offset constants returns nothing

The constant names drift (MMIO_MAGIC_VALUE vs MAGIC_VALUE vs an inline 0x74726976). Search for the value 0x74726976 first, then read the surrounding module to find how the rest are named.

The guest's /proc/cmdline has no virtio_mmio.device

You may be on aarch64 (devices are advertised via the FDT, not the cmdline) or with --enable-pci (virtio-PCI enumeration instead of MMIO). On aarch64, inspect the FDT path from Step 2; under PCI, the discovery is different — this lab assumes the default MMIO transport.

The live-queue dump prints garbage addresses

You are likely printing host addresses or reading the wrong structure. Use the device's own GuestMemoryMmap accessors (the same ones the handler uses) so addresses are guest-physical and bounds-checked. Do not try to translate guest→host addresses yourself.

A register write seems to be ignored

Re-check the offset against the table and against the transport's dispatch — and check the state machine. Several registers are only meaningful in certain states (a queue address write before DRIVER is set, or after DRIVER_OK, may be rejected). The order is enforced.

Expected Output

# /proc/cmdline inside the guest (x86):
... virtio_mmio.device=0x1000@0xd0000000:5 virtio_mmio.device=0x1000@0xd0001000:6 ...

# A live virtqueue dump during a guest read (block, queue 0):
[vq] avail.idx=37 used.idx=36
[vq] desc addr=0x3f1a2000 len=16   flags=0x1   next=1     # NEXT  -> header
[vq] desc addr=0x3f1a3000 len=4096 flags=0x3   next=2     # NEXT|WRITE -> data
[vq] desc addr=0x3f1a4000 len=1    flags=0x2   next=0     # WRITE -> status

Your addresses and indices will differ; the shape — three descriptors, flags NEXT then NEXT|WRITE then WRITE — is what must hold.

Stretch Goals

1. Read the whole negotiation from the guest's view. Boot with Firecracker logging at a verbose level and correlate the device's status transitions (ACKNOWLEDGE → … → DRIVER_OK) with the guest dmesg virtio lines. Where exactly does activate() fire relative to the guest's DRIVER_OK write?
2. Find the EVENT_IDX optimization. If VIRTIO_F_EVENT_IDX is negotiated, the device can suppress interrupts and the driver can suppress kicks using used_event/avail_event. Find where Firecracker reads/writes those and reason about when an interrupt is skipped. See virtio-net-and-tap, where it matters most.
3. Trace an INDIRECT descriptor. If VIRTIO_F_INDIRECT_DESC is negotiated, a single descriptor with the INDIRECT flag points at an in-memory table of further descriptors. Find the code that follows it and reason about the extra validation it needs.
4. Compare to rust-vmm. Read the rust-vmm virtio-queue crate's ring logic and diff its descriptor-chain iteration against Firecracker's in-tree Queue. They implement the same spec; note where Firecracker diverged and why.

Validation / Self-check

Answer without notes; these gate completion.

1. What value does a guest read from MagicValue, at what offset, and why does the driver check it?
2. List the status bits in the order the driver sets them. Which bit triggers activate(), and what must the device verify when the driver sets FEATURES_OK?
3. Which three guest physical addresses does the driver write through the MMIO registers to set up a queue, and what structure lives at each?
4. In a block-read descriptor chain, which descriptors carry NEXT and which carry WRITE, and why?
5. How does the Queue decide there is new work, and how does it index into the available ring?
6. What does ioeventfd change about the kick path, and what does irqfd change about the interrupt path? How many VM exits does each save per I/O?
7. Why must every read of a guest descriptor's addr/len go through a bounds-checked vm-memory accessor rather than a raw pointer?

Cross-references: virtio-transport-mmio, virtqueues, the-mmio-bus-and-device-manager, vm-memory, rust-vmm virtio-queue, interrupts-and-irqchip.

Next: Lab 7.3 — Build It: A Custom Virtio Device.