virtio-queue

A virtio device is, at its core, a set of virtqueues: ring buffers in guest memory through which the driver (in the guest) hands buffers to the device (in the VMM) and gets them back. A split virtqueue is three arrays — a descriptor table, an available ring, and a used ring — all in guest RAM, all attacker-controlled. Parsing them correctly and safely is the most security- sensitive parsing a VMM does: every offset, every chained descriptor, every length is a value the guest chose, and a sloppy parser is a guest-to-host escape. rust-vmm factors this logic into virtio-queue. This chapter teaches the model the crate implements, which is the same model Firecracker implements in-tree.

After this chapter you can: explain the three rings and the descriptor-chain mechanics; name the Queue type's key operations (pop_descriptor_chain, add_used, needs_notification) and the QueueT/QueueOwnedT traits; describe DescriptorChain, Reader/Writer, and QueueState save/restore; distinguish split from packed; and explain why Firecracker keeps its own virtio code while this crate exists.

Note — the one nuance to internalize first: Firecracker maintains its own virtqueue and device code in src/vmm/src/devices/virtio/. It does not consume virtio-queue wholesale (verify on your branch with the rg below). We cover virtio-queue because it is the cleanest, best-documented expression of the exact same model — the descriptor chain, the used ring, the notification logic are identical concepts. Read the crate to understand the in-tree code.

# Does your branch depend on virtio-queue, or only mirror its model in-tree?
rg -n "virtio-queue|virtio-device|virtio-bindings" Cargo.toml src/vmm/Cargo.toml Cargo.lock
# Firecracker's in-tree virtqueue implementation:
rg -n "struct Queue|DescriptorChain|pop|add_used|used_ring|avail_ring|desc_table" src/vmm/src/devices/virtio/queue.rs
cargo doc -p virtio-queue --no-deps --open 2>/dev/null || echo "virtio-queue not a direct dep — read in-tree queue.rs"

docs.rs: docs.rs/virtio-queue.

The split virtqueue model

 GUEST RAM (all three rings are guest-controlled):

   descriptor table              available ring (driver → device)
   ┌───────────────────┐         ┌──────────────────────────────┐
   │ desc[0] addr,len, │         │ idx │ ring[0] ring[1] ... ... │
   │         flags,next│◄────────┤ (driver publishes head indices)│
   │ desc[1] ...       │         └──────────────────────────────┘
   │ desc[2] ...       │
   └───────────────────┘         used ring (device → driver)
        ▲   chained via            ┌──────────────────────────────┐
        │   NEXT flag → next        │ idx │ {id,len} {id,len} ...  │
        └───────────────────────────┤ (device publishes completed)  │
                                    └──────────────────────────────┘

The flow: the driver builds a descriptor chain (one or more descriptors linked by the NEXT flag) describing a buffer set, publishes the chain's head index in the available ring, bumps avail.idx, and kicks (writes QueueNotify in the MMIO register block — a KVM_EXIT_MMIO or, on the fast path, an ioeventfd). The device pops the chain, does the work (reads the request, does the host I/O), writes results into the WRITE-flagged descriptors, publishes {head_id, bytes_written} in the used ring, bumps used.idx, and raises an interrupt if the driver needs one.

The Queue type

virtio-queue's central type is Queue. It owns the three ring addresses (set during device configuration from the MMIO QueueDesc/QueueDriver/QueueDevice registers) and the head/tail bookkeeping, and exposes the operations a device backend calls.

The traits QueueT and QueueOwnedT abstract over queue access: QueueT is the shared interface (so a device can be generic over the queue type), and QueueOwnedT adds the owned-iteration methods like pop_descriptor_chain. This is the same trait-over-concrete pattern as vm-memory's GuestMemory — device code written against QueueT doesn't care about the concrete Queue.

Every operation that touches a ring takes a &M: GuestMemory (vm-memory.md) — because the rings are guest memory, and every access is a bounds-checked read_obj/write_obj. That is the safety story: ring parsing is built on vm-memory's checked accessors, so a wild descriptor address yields an Err, not a host out-of-bounds access.

DescriptorChain, Reader, and Writer

pop_descriptor_chain returns a DescriptorChain — an iterator over the descriptors in one chain, following the NEXT links (and dereferencing INDIRECT descriptors). Each descriptor tells you a guest buffer (addr, len) and whether it is device-readable (input to the device) or device-writable (WRITE flag, output back to the driver).

A device almost never wants to walk raw descriptors. virtio-queue provides Reader and Writer (in the desc / chain helpers) that present the readable descriptors of a chain as one logical input stream and the writable descriptors as one logical output stream — so a block device can "read the request header, then read the data" or "write the data, then write the status byte" across a multi-descriptor chain without manually tracking which descriptor it is in.

# In Firecracker's in-tree code, the equivalent walk:
rg -n "DescriptorChain|next_descriptor|VIRTQ_DESC_F_NEXT|VIRTQ_DESC_F_WRITE|is_write_only" src/vmm/src/devices/virtio/

#![allow(unused)]
fn main() {
// Shape of a backend draining one queue (virtio-queue API; FC's in-tree is analogous).
use virtio_queue::{Queue, QueueOwnedT};
use vm_memory::GuestMemoryMmap;

fn process_queue(queue: &mut Queue, mem: &GuestMemoryMmap) {
    while let Some(mut chain) = queue.pop_descriptor_chain(mem) {
        let head_index = chain.head_index();
        let mut written = 0u32;

        // Walk the chain: readable descriptors are device input, writable are output.
        for desc in &mut chain {
            if desc.is_write_only() {
                // device → guest: write results here (bounds-checked via mem)
                // written += do_output(mem, desc.addr(), desc.len());
                written += desc.len();
            } else {
                // guest → device: read request data here
                // let req = mem.read_obj::<ReqHeader>(desc.addr())?;
            }
        }

        // Publish completion and decide whether to interrupt.
        queue.add_used(mem, head_index, written).unwrap();
        if queue.needs_notification(mem).unwrap() {
            // trigger the device's interrupt eventfd (irqfd) — see event-manager / vmm-sys-util
        }
    }
}
}

needs_notification is not a formality. With the VIRTIO_F_EVENT_IDX feature negotiated, the driver tells the device when it actually wants to be interrupted (to batch completions and cut interrupt rate). Honoring it is a real performance lever and a real correctness requirement — interrupt too little and the guest stalls; too much and you waste cycles.

QueueState: save and restore

For snapshotting, the queue's position (its ring addresses, the next-available index, the used index, the notification flags) must be saved and restored exactly, so a restored microVM resumes mid-stream without losing or double-processing a buffer. virtio-queue exposes a QueueState (a plain, serializable snapshot of the queue's fields) and Queue↔QueueState conversions. Firecracker's in-tree queue has the equivalent — its virtio devices implement the Persist trait, and the queue state is part of what each device serializes.

rg -n "QueueState|Persist|save\(|restore\(|next_avail|next_used" src/vmm/src/devices/virtio/

Split vs packed

virtio-queue supports both split and packed (packed via additional state in the Queue/QueueState). Firecracker historically implements split virtqueues in its in-tree devices; whether packed is supported on your branch is a "verify it" question — rg for PACKED:

rg -n "PACKED|RING_PACKED|packed" src/vmm/src/devices/virtio/

Why Firecracker keeps its own virtio code

This is a deliberate engineering choice, and a good lens on the in-tree-vs-upstream trade-off from what-is-rust-vmm.md:

• Attack surface control. The virtqueue parser is the highest-value target in the whole VMM. Firecracker wants its exact implementation — its bounds checks, its error handling, its limits — under its own review and its own fuzzing, not abstracted behind a general-purpose crate that must serve every consumer.
• Tight integration. FC's devices, rate limiters, and MMIO transport are co-designed; the queue handling is woven into them.
• History. Firecracker predates the mature virtio-queue crate; its in-tree code came first and works.

The concepts are identical, which is why reading virtio-queue is the fastest way to understand src/vmm/src/devices/virtio/queue.rs. When you trace a virtio-block I/O in Lab 7.1 or read the virtqueues deep dive, you are reading Firecracker's version of exactly this model. And if you ever do contribute to the shared crate, Lab R3 is where you drive a real virtio-queue Queue directly.

Reading exercise

# 1. Is virtio-queue a direct dep, or mirrored in-tree?
rg -n "virtio-queue" Cargo.toml Cargo.lock
rg -n "struct Queue|DescriptorChain" src/vmm/src/devices/virtio/queue.rs

# 2. The crate's Queue/DescriptorChain API (pinned version).
cargo doc -p virtio-queue --no-deps --open 2>/dev/null || \
  find ~/.cargo/registry/src -maxdepth 2 -type d -name 'virtio-queue-*'

# 3. FC's pop / add_used / notification equivalents.
rg -n "fn pop|add_used|needs_notification|enable_notification|next_avail|next_used" src/vmm/src/devices/virtio/queue.rs

# 4. The descriptor-flag handling.
rg -n "VIRTQ_DESC_F_NEXT|VIRTQ_DESC_F_WRITE|is_write_only|INDIRECT" src/vmm/src/devices/virtio/

# 5. Queue save/restore for snapshots.
rg -n "QueueState|Persist|save|restore" src/vmm/src/devices/virtio/

# 6. Split vs packed support.
rg -n "PACKED|EVENT_IDX|RING_PACKED" src/vmm/src/devices/virtio/

Answer:

1. Draw the three rings and label the direction of each. Which two are written by the driver and which by the device?
2. Walk one buffer end to end: descriptor chain → available ring → kick → device pop → used ring → interrupt. Name the Queue method at each device-side step.
3. What is a DescriptorChain, and what do Reader/Writer add on top of raw descriptor iteration?
4. What does needs_notification honor, and what goes wrong if you always raise the interrupt? If you never do?
5. What does QueueState capture, and why must it be exact for snapshot restore?
6. Give two concrete reasons Firecracker keeps its own virtio code instead of consuming virtio-queue. What stays identical regardless?

Common bugs and symptoms

Next: vm-superio — the legacy device models (serial console, i8042, RTC) that Firecracker extracted upstream. Unlike virtio, these FC does consume from rust-vmm.