Level 7: The Virtio Device Model

This is the level where Firecracker stops being "a thing that runs a kernel" and becomes "a thing your guest can do I/O against." Up to now you have driven KVM, walked the vCPU run loop, watched VM exits, and traced a kernel from an ELF file into guest memory. But a microVM with no disk and no network is useless. The way a Firecracker guest reaches a real disk, a real network, a real source of entropy, is virtio — a standardized, paravirtualized I/O model in which the guest and the host cooperate over shared memory rings instead of pretending to be real hardware. Firecracker implements virtio devices itself, in-tree, in src/vmm/src/devices/virtio/, and this level makes you read that code until a guest disk read is no longer magic.

You will learn the virtio spec essentials — the device/driver split, the virtio-MMIO transport and its register map, feature negotiation and the status state machine, the split virtqueue (descriptor table, available ring, used ring), and the kick/interrupt mechanism — and then you will watch Firecracker's own code implement every one of those pieces. You will trace a real guest disk read end to end, from the guest driver writing QueueNotify to the host calling pread and back. You will inspect the MMIO register block and the in-memory layout of a virtqueue. And then you will build a small virtio device of your own, following the shape of the simplest existing devices.

This curriculum will not hold your hand. It points you at the right parts of the codebase, gives you the right questions, and makes you run everything you read. Every type named below comes with the rg that finds it on your checkout, because the device code moves between branches and a contributor who quotes line numbers is already wrong. The facts here track roughly Firecracker v1.16; where something is version-sensitive it says "(verify on your branch)," and you are expected to actually verify.

Learning Objectives

By the end of Level 7 you must be able to:

1. State the device/driver split at the heart of virtio, and explain why a paravirtualized model is faster and safer than emulating real hardware register-for-register.
2. Read the virtio-MMIO register map (MagicValue, Version, DeviceID, QueueSel, QueueNum, QueueReady, QueueNotify, InterruptStatus, Status, the queue address registers) and explain what a guest write to each one does.
3. Walk the feature-negotiation handshake and the status state machine (ACKNOWLEDGE → DRIVER → FEATURES_OK → DRIVER_OK), and say what it means when the guest sets the FAILED bit.
4. Diagram a split virtqueue — the descriptor table (addr/len/flags/next, with NEXT/WRITE/INDIRECT flags), the available ring (driver → device), and the used ring (device → driver) — and describe a descriptor chain for a block request.
5. Explain the kick/interrupt mechanism: how the guest "kicks" by writing QueueNotify, how ioeventfd turns that write into a host eventfd signal, and how the device raises a guest interrupt via irqfd after writing the used ring.
6. Locate Firecracker's VirtioDevice trait, the Queue type, the MmioTransport, and the MMIODeviceManager, and explain the activate → process flow driven by the EventManager epoll loop.
7. Trace a virtio-block I/O from the guest's read() syscall down to a host pread on the backing file and back to the guest's completion interrupt — naming every hop in Firecracker's code.
8. Implement a minimal VirtioDevice of your own, register it with the MMIODeviceManager, advertise it to the guest, and exercise it.

Virtio in One Picture

Virtio is a contract between a driver (in the guest kernel) and a device (in the VMM). The contract has three layers, and Firecracker implements the bottom two:

1. The transport — how the driver finds the device and configures it. Firecracker's default is virtio-MMIO: a fixed block of memory-mapped registers per device, plus one interrupt line. No PCI enumeration; the guest is told where each device lives via the kernel command line on x86 (virtio_mmio.device=SIZE@ADDR:IRQ) or an FDT node on aarch64. (A virtio-PCI transport now exists behind --enable-pci — verify on your branch; note CVE-2026-5747 lived in that transport, fixed in 1.14.4/1.15.1.)
2. The virtqueue — the shared-memory ring buffers over which the driver and device pass I/O requests. The driver puts descriptors in guest RAM; the device reads them, does the work, and writes results back. This is the data plane, and it is where the speed comes from.
3. The device semantics — what the bytes in those descriptors mean. A block request, a network packet, a chunk of randomness. This is per-device-type.

           GUEST                                       HOST (Firecracker VMM thread)
  ┌──────────────────────────┐                ┌───────────────────────────────────────┐
  │  virtio-blk driver       │                │  Block device (VirtioDevice impl)      │
  │  (Linux kernel)          │                │  src/vmm/src/devices/virtio/block/     │
  └──────────┬───────────────┘                └───────────────▲───────────────────────┘
             │ 1. build descriptor chain in guest RAM         │ 4. pop chain, pread/pwrite
             │    (header | data | status)                    │    on the backing file
             ▼                                                │
   ┌───────────────────────────── guest physical memory ──────┴───────────────────┐
   │  descriptor table   │   available ring (driver→dev)   │   used ring (dev→driver) │
   └──────────────────────────────────────────────────────────────────────────────┘
             │ 2. write QueueNotify (MMIO)                    ▲ 5. write used ring
             ▼                                                │
   ┌──────────────────────┐   KVM_EXIT_MMIO / ioeventfd   ┌───┴──────────────────┐
   │  virtio-MMIO register│ ────────────────────────────► │ EventManager epoll   │
   │  block @ fixed addr  │   3. eventfd wakes the VMM     │ wakes the device     │
   └──────────────────────┘                               └──────────────────────┘
             ▲ 6. irqfd injects the completion interrupt — guest driver reaps the used ring
             └──────────────────────────────────────────────────────────────────────────────

The genius of the design: once the queue is set up, the fast path is just shared memory plus two eventfds. The guest writes a doorbell register (one VM exit, or zero if ioeventfd is used); the host does the work and signals an interrupt (irqfd). No per-byte trap-and-emulate. You will trace exactly this loop in Lab 7.1.

Note: "virtio" is always spelled lowercase, the transport is virtio-MMIO, the rings are virtqueues, and the guest side is the driver while the host side is the device. Getting the device/driver direction backwards is the single most common source of confusion when reading this code: the available ring is written by the driver, the used ring is written by the device. Burn that in.

The Split Virtqueue (the thing you must truly understand)

Everything in this level rests on the split virtqueue. There are three contiguous structures, all living in guest memory (Firecracker reads them through vm-memory):

Descriptor table — an array of virtq_desc, indexed by descriptor id:
  struct virtq_desc {
      le64 addr;    // guest physical address of a buffer
      le32 len;     // length of that buffer
      le16 flags;   // NEXT=1 (chains to `next`), WRITE=2 (device writes it), INDIRECT=4
      le16 next;    // id of the next descriptor in the chain (if NEXT set)
  }

Available ring — the DRIVER hands descriptor chains to the DEVICE:
  struct virtq_avail {
      le16 flags;
      le16 idx;        // driver increments this; device watches it
      le16 ring[QSIZE];// ring[idx % QSIZE] = head descriptor id of a new chain
  }

Used ring — the DEVICE returns completed chains to the DRIVER:
  struct virtq_used {
      le16 flags;
      le16 idx;        // device increments this; driver watches it
      struct virtq_used_elem { le32 id; le32 len; } ring[QSIZE];
  }

A single I/O is a descriptor chain: one or more descriptors linked by next/NEXT. For a block read the chain is three descriptors — a device-readable header (struct virtio_blk_req with type + sector), a device-writable data buffer (WRITE flag set — this is where the device deposits the disk contents), and a one-byte device-writable status. The driver writes the head descriptor's id into the available ring and bumps avail.idx; the device pops it, walks the chain, fills the data buffer, writes the status byte, then appends a used_elem and bumps used.idx. The WRITE flag is the access-control boundary: a descriptor the device writes must have WRITE set, and Firecracker must refuse to write a buffer that does not — getting this wrong is a class of real virtio bugs.

You will study this structure in depth in Lab 7.2 and the virtqueues deep dive. Firecracker's Queue type encapsulates the ring pointers and the index bookkeeping; the rust-vmm virtio-queue crate is the reference for the same logic.

How Firecracker Implements It

Every Firecracker virtio device implements one trait and is plugged into the machine by one manager.

flowchart TD
    G[Guest virtio driver] -->|MMIO write to register block| MT[MmioTransport]
    MT -->|reads/writes registers,<br/>tracks negotiated state| DEV[VirtioDevice impl<br/>block / net / vsock / balloon / rng]
    MT -.->|registered on the bus| BUS[MMIODeviceManager bus]
    KVM[KVM_EXIT_MMIO] -->|dispatched by phys_addr| BUS
    BUS --> MT
    EM[EventManager epoll loop] -->|queue/ioeventfd ready| DEV
    DEV -->|pop chain, do host I/O| Q[Queue]
    Q -->|read avail / write used| MEM[(GuestMemoryMmap)]
    DEV -->|raise IRQ via irqfd| KVM

The pieces, each with the command that finds it (paths drift — always rg):

• VirtioDevice — the trait every device implements: device_type(), queues(), interrupt_status(), read_config()/write_config(), features()/ack_features(), and activate(). rg -n "trait VirtioDevice" src/vmm/src/devices/virtio/
• MmioTransport — the per-device MMIO register adapter; it reads/writes the register block, drives feature negotiation and the status state machine, and forwards QueueNotify to the device. rg -n "struct MmioTransport|MmioTransport" src/vmm/src/devices/virtio/
• Queue — the split-virtqueue abstraction (avail/used indices, pop, add_used, descriptor chain iteration). rg -n "struct Queue\b" src/vmm/src/devices/virtio/
• MMIODeviceManager — places each device's register block at a guest physical address, owns the bus that dispatches a KVM_EXIT_MMIO to the right device, and allocates the IRQ. rg -n "struct MMIODeviceManager" src/vmm/src/device_manager/
• ActivateError / activate() — activate() is called once, when the guest finishes negotiation (sets DRIVER_OK); it registers the device's queue eventfds with the EventManager and spins up the data path. rg -n "fn activate" src/vmm/src/devices/virtio/

The activate → process flow is the spine of the device model and the thing most people get wrong on first read:

1. At boot, the device is constructed and registered on the bus but not active — its queues are not yet wired to the event loop.
2. The guest driver negotiates features and writes the queue addresses through the MMIO registers.
3. When the guest sets the DRIVER_OK status bit, the MmioTransport calls the device's activate(), which registers the queue notification eventfd(s) with the EventManager.
4. From then on, every guest "kick" (a QueueNotify write, delivered as an ioeventfd signal) wakes the EventManager, which calls the device's process_queue-style handler; the handler pops descriptor chains, does the host-side I/O, writes the used ring, and raises the interrupt.

The event loop is the rust-vmm event-manager crate; the bus and placement are the MMIO bus and device manager deep dive.

Required Reading

Read these in your own checkout and from the spec, in order, before the labs. In a mature project the best documentation is the code and the spec — treat both as primary source.

Confirm the device tree exists on your branch (names occasionally move):

# From the repo root of your firecracker checkout:
ls src/vmm/src/devices/virtio/
# Expect subdirs like: block/  net/  vsock/  balloon/  rng/  (+ queue, transport, persist, gen)
# If the layout moved, find a device by type rather than trusting the name:
rg -l "impl VirtioDevice" src/vmm/src/devices/virtio/

Source Code Areas to Inspect

Key Types Quick Reference

Memorize the role of each; predict the path before you run the command.

# Build the muscle memory — predict each directory before the result appears.
rg -n "trait VirtioDevice"        src/vmm/src/devices/virtio/
rg -n "struct Queue\b"            src/vmm/src/devices/virtio/
rg -n "MmioTransport"             src/vmm/src/devices/virtio/
rg -n "pub struct Block"          src/vmm/src/devices/virtio/block/
rg -n "0x74726976"                src/vmm/src/devices/virtio/   # the MagicValue 'virt'

Note: Several of these names are version-sensitive — whether the transport type is exactly MmioTransport, whether the entropy device is Entropy or Rng, and the precise module split (queue.rs vs a queue/ directory) all verify on your branch. If a command returns nothing, broaden it (drop the path, search the whole devices/virtio/) rather than concluding the type is gone.

GitHub Issue Categories for Level 7 Contributors

You now have the depth to take device-model issues — but the device path is privileged host code on the data plane, so scope carefully and lean on tests:

• Device config / validation — improving an error message or validation in the block/net/vsock config path; surfacing a misconfiguration the guest currently hits silently.
• Virtio correctness — descriptor-chain validation, the WRITE-flag access check, queue index-wraparound handling, feature-bit edge cases.
• io_engine / rate-limiter interactions in the block or net device.
• Device-level tests — integration tests that exercise a device path that is currently under-tested (a stretch from the testing level).

gh issue list --repo firecracker-microvm/firecracker \
  --label "Good first issue" --state open --limit 40
# Then narrow by area in the body/title:
gh issue list --repo firecracker-microvm/firecracker --state open --limit 60 \
  --search "virtio OR block OR virtqueue OR net OR vsock in:title,body"

Warning: "Add device X because QEMU has it" is not an accepted argument in this project — the minimal device model is a security posture, not an oversight. A new device is a large, hard sell. Correctness, validation, and test coverage on existing devices are where a Level 7 graduate contributes. See minimal-device-model-philosophy.

Deliverables

Demonstrate all of the following before advancing to Level 8:

• A completed reading log tracing a guest disk read from QueueNotify → KVM_EXIT_MMIO/ioeventfd → the block device's queue handler → pread → used ring → IRQ, with the rg you used at each hop (Lab 7.1).
• An annotated dump of the virtio-MMIO register block offsets and the negotiation/status sequence a guest performs, plus (optionally) a dump of a live virtqueue from guest memory (Lab 7.2).
• A working minimal VirtioDevice of your own (an echo/counter device), registered with the MMIODeviceManager, advertised to the guest, and exercised — with at least one unit test (Lab 7.3).
• A two-paragraph written explanation of the activate → process flow and of why the WRITE descriptor flag is an access-control boundary.

Common Mistakes

How to Verify Success

# 1. You can find the device model and name each device's type id from the code.
rg -n "fn device_type" src/vmm/src/devices/virtio/*/        # one per device
rg -n "TYPE_BLOCK\|TYPE_NET\|TYPE_RNG\|TYPE_BALLOON\|TYPE_VSOCK" src/vmm/src/devices/virtio/

# 2. You traced a real block I/O (Lab 7.1): your tracing fired on a guest `dd`/`fio` read.

# 3. You can read the MMIO register offsets straight from the code (Lab 7.2).
rg -n "0x74726976\|MagicValue\|QueueNotify\|QueueSel\|InterruptStatus" src/vmm/src/devices/virtio/

# 4. Your custom device (Lab 7.3) is reachable from the guest and its unit test passes.
#    (You will build with tools/devtool and exercise from inside a booted microVM.)

PR Profile: Level 7 Graduate

A Level 7 graduate can credibly open these kinds of PRs. The device data plane is privileged host code; every change here needs a test, and ideally an integration test that boots a guest.

You are not yet expected to land a whole new device upstream (that is a large, multi-PR effort with a high bar — be honest about that in Lab 7.3), or to change the snapshot/Persist format of a device (that is Level 9 and the snapshotting deep dive).

Next: Lab 7.1 — Trace a Virtio-Block I/O End to End.