Lab R3: Drive a Virtqueue with virtio-queue + vm-memory

Background

A virtqueue is the data structure every virtio device uses to move buffers between the guest ("driver") and the host ("device"). It is the beating heart of virtio-block, virtio-net, virtio-vsock — all of it. And it is the part of the device model that stays abstract the longest, because the virtqueues deep dive describes three rings (descriptor table, available ring, used ring) in guest memory, manipulated from both sides, with index wrapping and flag bits, and reading about it is not the same as laying the bytes down yourself.

This lab makes it concrete. You will build a GuestMemoryMmap, lay out a split virtqueue by hand in that memory — descriptor table, available ring, used ring, at addresses you choose and align yourself — then play both roles. As the driver, you write a descriptor chain and publish it in the available ring (exactly what a guest kernel does on a "kick"). As the device, you construct a rust-vmm virtio-queue Queue over those same addresses, pop_descriptor_chain() to get the chain the driver published, walk its descriptors, read and write the buffers in guest memory, and add_used to signal completion (exactly what Firecracker's block/net devices do). Every step prints what it touched, so the ring mechanics become visible.

This is the hardest lab in the rust-vmm section and the most valuable. The payoff: after it, the descriptor-chain logic in src/vmm/src/devices/virtio/ reads as a variation on a program you wrote.

Note: virtio-queue provides the device-side Queue abstraction (pop chains, add used). It does not lay out the rings for you — that is the driver's job, done in guest memory. By doing the layout yourself you learn the wire format the spec defines and KVM/vm-memory transports. This is deliberately lower-level than calling the crate's test-only MockSplitQueue helper.

Why This Lab Matters for Contributors

• The virtqueues deep dive, virtio-block, and virtio-net all assume you can read a descriptor chain. This lab is where that becomes a skill, not a diagram.
• Firecracker maintains its own Queue in src/vmm/src/devices/virtio/queue.rs, closely modeled on rust-vmm's. Driving the upstream virtio-queue Queue here teaches the shape so the in-tree one is legible — and shows you what an upstream-vs-in-tree fork looks like (relevant for Lab R4).
• Bugs in virtio devices are almost always descriptor-handling bugs: a chain walked wrong, a used index off by one, a writable buffer treated as readable. You can only debug what you have built.
• The MMIO transport deep dive explains how the queue addresses get programmed (the guest writes them to MMIO registers). This lab is what sits behind those registers.

Prerequisites

• Lab R1 and Lab R2 complete — you are comfortable with GuestMemoryMmap and reading/writing guest memory.
• A Rust toolchain. No KVM is required for this lab — it is pure guest-memory + queue logic, so it runs on any platform (macOS/Windows included). That is a feature: you can study virtqueues without a Linux box.
• A Firecracker checkout to rg against.

rustc --version
git clone https://github.com/firecracker-microvm/firecracker.git ~/firecracker 2>/dev/null || true
# See Firecracker's own Queue and how its devices pop chains and add used buffers:
rg -n "pop_descriptor_chain\|pop_or_enable_notification\|add_used\|DescriptorChain" ~/firecracker/src/vmm/src/devices/virtio/

Step-by-Step Tasks

Step 1: Hold the split-virtqueue wire format in your head

A split virtqueue of size N is three contiguous regions in guest memory. The spec defines their byte layout exactly; you will write these bytes yourself.

                    SPLIT VIRTQUEUE  (queue size N, here N = 8)
 ┌──────────────────────────────────────────────────────────────────────┐
 │ Descriptor Table   @ DESC_TABLE_ADDR   (16 bytes × N)                  │
 │   each entry:  u64 addr | u32 len | u16 flags | u16 next               │
 │     flags: NEXT=1 (chain continues via `next`),  WRITE=2 (device-write)│
 ├──────────────────────────────────────────────────────────────────────┤
 │ Available Ring     @ AVAIL_ADDR        (driver → device)               │
 │   u16 flags | u16 idx | u16 ring[N] | (u16 used_event)                 │
 │     idx = total #chains ever made available (free-running, wraps)      │
 │     ring[idx % N] = head descriptor index of a published chain         │
 ├──────────────────────────────────────────────────────────────────────┤
 │ Used Ring          @ USED_ADDR         (device → driver)               │
 │   u16 flags | u16 idx | {u32 id; u32 len}[N] | (u16 avail_event)       │
 │     idx = total #chains ever completed (free-running, wraps)           │
 │     ring[idx % N] = {id = head desc index, len = bytes written}        │
 └──────────────────────────────────────────────────────────────────────┘

Two rules carry all the subtlety:

• Indices are free-running counters, not slots. avail.idx and used.idx only ever increase (mod 2¹⁶). You compute the slot with idx % N. The device remembers the last avail.idx it processed; the gap is the new work.
• The descriptor table is shared; the rings are one-directional. The driver fills descriptors and the available ring; the device fills the used ring. Each side only ever reads the other's ring.

Tip: Alignment matters on real hardware (desc table 16-byte, avail 2-byte, used 4-byte aligned). We place the three regions on separate 4 KiB pages, which satisfies every alignment and keeps the addresses easy to read in output.

Step 2: Create the project and pin the crates

cargo new --bin virtqueue-lab && cd virtqueue-lab

Cargo.toml:

[package]
name = "virtqueue-lab"
version = "0.1.0"
edition = "2021"

[dependencies]
# Versions current to mid-2026 — verify on crates.io; the virtio-queue API moves between minors.
virtio-queue = "0.17"
vm-memory = { version = "0.17", features = ["backend-mmap"] }

Warning — virtio-queue is the most API-volatile crate in this section. Between 0.x minors the descriptor types were reorganized (a desc module with split/packed submodules; Descriptor became a deprecated alias for a wire-format RawDescriptor). The code below targets 0.17; if your resolved version differs, cargo doc --open -p virtio-queue and adjust the import of Descriptor and the QueueOwnedT/QueueT method names. (verify on docs.rs for your version.) cargo tree -p vm-memory — virtio-queue and vm-memory must agree on the vm-memory version.

Step 3: Write the program

Replace src/main.rs. The program is in three acts: lay out the rings, be the driver, be the device.

use std::mem::size_of;
use virtio_queue::desc::split::Descriptor; // 0.17: split-ring descriptor type
use virtio_queue::{QueueOwnedT, QueueT, Queue};
use vm_memory::{Bytes, GuestAddress, GuestMemory, GuestMemoryMmap};

const QUEUE_SIZE: u16 = 8;
const MEM_SIZE: usize = 0x10_0000; // 1 MiB of guest memory

// Place each ring on its own page so addresses are obvious and alignment is satisfied.
const DESC_TABLE_ADDR: u64 = 0x1000;
const AVAIL_ADDR: u64 = 0x2000;
const USED_ADDR: u64 = 0x3000;
// Buffers the descriptors point at (data the driver provides / device fills).
const BUF_IN_ADDR: u64 = 0x4000; // driver-written, device reads ("request")
const BUF_OUT_ADDR: u64 = 0x5000; // device-written, driver reads ("response")

// Split-ring descriptor flags (virtio spec).
const VIRTQ_DESC_F_NEXT: u16 = 1; // buffer continues in `next`
const VIRTQ_DESC_F_WRITE: u16 = 2; // device-write-only (else device-read-only)

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let mem = GuestMemoryMmap::<()>::from_ranges(&[(GuestAddress(0), MEM_SIZE)])?;

    // ── ACT 1: lay out the descriptor table (the driver builds a 2-descriptor chain) ──
    // Descriptor 0: a READ buffer (device reads the "request" the driver wrote), chains to 1.
    // Descriptor 1: a WRITE buffer (device writes a "response" into it), end of chain.
    let request = b"PING from the driver";
    mem.write_slice(request, GuestAddress(BUF_IN_ADDR))?;
    println!("[driver] wrote request {:?} at {:#x}", std::str::from_utf8(request)?, BUF_IN_ADDR);

    let desc0 = Descriptor::new(BUF_IN_ADDR, request.len() as u32, VIRTQ_DESC_F_NEXT, 1);
    let desc1 = Descriptor::new(BUF_OUT_ADDR, 64, VIRTQ_DESC_F_WRITE, 0);
    // Write the two descriptors into the descriptor table (16 bytes each).
    mem.write_obj(desc0, GuestAddress(DESC_TABLE_ADDR))?;
    mem.write_obj(desc1, GuestAddress(DESC_TABLE_ADDR + 16))?;
    println!("[driver] desc[0] addr={:#x} len={} flags=NEXT next=1", BUF_IN_ADDR, request.len());
    println!("[driver] desc[1] addr={:#x} len=64 flags=WRITE next=0  (end of chain)", BUF_OUT_ADDR);

    // ── ACT 2: publish the chain in the available ring (the "kick") ──
    // Available ring layout: u16 flags @ +0, u16 idx @ +2, u16 ring[i] @ +4 + 2*i.
    // Publish head descriptor index 0 into ring slot 0, then bump idx to 1.
    let avail = GuestAddress(AVAIL_ADDR);
    mem.write_obj(0u16, avail)?; // flags = 0
    mem.write_obj(0u16, GuestAddress(AVAIL_ADDR + 4))?; // ring[0] = head index 0
    mem.write_obj(1u16, GuestAddress(AVAIL_ADDR + 2))?; // idx = 1  (one chain available)
    println!("[driver] published chain head=0 in avail.ring[0]; avail.idx now = 1  (KICK)");

    // ── ACT 3: be the DEVICE — build a Queue over the rings and process the chain ──
    let mut queue = Queue::new(QUEUE_SIZE)?;
    // Program the ring addresses (what the guest would write to the MMIO QueueDesc/Driver/Device
    // registers — see the MMIO transport deep dive). low/high split of each 64-bit address:
    queue.set_desc_table_address(Some(lo(DESC_TABLE_ADDR)), Some(hi(DESC_TABLE_ADDR)));
    queue.set_avail_ring_address(Some(lo(AVAIL_ADDR)), Some(hi(AVAIL_ADDR)));
    queue.set_used_ring_address(Some(lo(USED_ADDR)), Some(hi(USED_ADDR)));
    queue.set_size(QUEUE_SIZE);
    queue.set_ready(true);

    // Pop the chain the driver published. This reads avail.idx, sees 1 > last-seen 0,
    // and returns the chain whose head is avail.ring[0] = descriptor 0.
    let mut chain = queue
        .pop_descriptor_chain(&mem)
        .expect("[device] expected one available chain but found none");
    println!("[device] popped chain, head index = {}", chain.head_index());

    // Walk the chain: descriptor 0 (read) then descriptor 1 (write).
    let mut response_head: Option<(GuestAddress, u32)> = None;
    let mut total_written = 0u32;
    for (i, desc) in chain.by_ref().enumerate() {
        let writable = desc.flags() & VIRTQ_DESC_F_WRITE != 0;
        println!(
            "[device]   desc {i}: addr={:#x} len={} {}",
            desc.addr().0,
            desc.len(),
            if writable { "WRITE (device fills)" } else { "READ (device consumes)" }
        );
        if !writable {
            // Read the request the driver placed here.
            let mut buf = vec![0u8; desc.len() as usize];
            mem.read_slice(&mut buf, desc.addr())?;
            println!("[device]     read request: {:?}", std::str::from_utf8(&buf)?);
        } else {
            // Remember where to write the response.
            response_head = Some((desc.addr(), desc.len()));
        }
    }

    // Write a response into the device-writable buffer.
    if let Some((addr, cap)) = response_head {
        let response = b"PONG from the device";
        assert!(response.len() as u32 <= cap, "response exceeds buffer");
        mem.write_slice(response, addr)?;
        total_written = response.len() as u32;
        println!("[device]   wrote response {:?} into {:#x}", std::str::from_utf8(response)?, addr.0);
    }

    // ── ACT 3b: signal completion via the used ring ──
    // add_used writes {id = head index, len = bytes written} into used.ring[used.idx % N]
    // and bumps used.idx. The driver polls used.idx to discover completed chains.
    let head = chain.head_index();
    queue.add_used(&mem, head, total_written)?;
    println!("[device] add_used(head={head}, len={total_written}); used.idx bumped");

    // ── verify: read the used ring back as the DRIVER would ──
    // Used ring: u16 flags @ +0, u16 idx @ +2, then {u32 id; u32 len}[i] @ +4 + 8*i.
    let used_idx: u16 = mem.read_obj(GuestAddress(USED_ADDR + 2))?;
    let used_id: u32 = mem.read_obj(GuestAddress(USED_ADDR + 4))?;
    let used_len: u32 = mem.read_obj(GuestAddress(USED_ADDR + 8))?;
    println!("[driver] sees used.idx={used_idx}, used.ring[0] = {{id={used_id}, len={used_len}}}");

    // And read the response the device wrote, completing the round trip.
    let mut got = vec![0u8; used_len as usize];
    mem.read_slice(&mut got, GuestAddress(BUF_OUT_ADDR))?;
    println!("[driver] response buffer: {:?}", std::str::from_utf8(&got)?);
    Ok(())
}

// Split a 64-bit guest address into the low/high 32-bit halves the queue wants.
fn lo(addr: u64) -> u32 {
    addr as u32
}
fn hi(addr: u64) -> u32 {
    (addr >> 32) as u32
}

// (size_of imported for readers who want to assert the 16-byte descriptor size.)
const _: () = assert!(size_of::<Descriptor>() == 16);

Step 4: Run it

cargo run

Expected output:

[driver] wrote request "PING from the driver" at 0x4000
[driver] desc[0] addr=0x4000 len=20 flags=NEXT next=1
[driver] desc[1] addr=0x5000 len=64 flags=WRITE next=0  (end of chain)
[driver] published chain head=0 in avail.ring[0]; avail.idx now = 1  (KICK)
[device] popped chain, head index = 0
[device]   desc 0: addr=0x4000 len=20 READ (device consumes)
[device]     read request: "PING from the driver"
[device]   desc 1: addr=0x5000 len=64 WRITE (device fills)
[device]   wrote response "PONG from the device" into 0x5000
[device] add_used(head=0, len=20); used.idx bumped
[driver] sees used.idx=1, used.ring[0] = {id=0, len=20}
[driver] response buffer: "PONG from the device"

You just drove a virtqueue end to end: the driver published a 2-descriptor chain, the device popped it, read the read-only buffer, filled the write-only buffer, and posted the result to the used ring — and the driver observed the completion. That round trip is every virtio device's inner loop.

Note: add_used's len is "bytes the device wrote that the driver should consider valid." For a block read it is the bytes read into the guest buffer; for net RX it is the packet length. We pass total_written (the request length here, deliberately, to show the field is device-chosen — for a real response you would pass the response length). Read the spec's "used ring" section on what len means per device class.

Step 5: Map it to Firecracker's virtio devices

Firecracker keeps its own Queue (a fork/cousin of virtio-queue) and its devices use it exactly like you used the upstream one. Confirm:

Read devices/virtio/block/ or devices/virtio/net/ with this table open. The device's process function is pop a chain → parse the descriptors → do host I/O → add_used → signal an interrupt — the loop you just wrote, with real I/O in the middle and an IRQ at the end.

Implementation Requirements / Deliverables

• virtqueue-lab compiles and cargo run prints the full driver→device→driver round trip.
• You can draw the three rings from memory, including which side writes each, and where idx lives in the available and used rings.
• You can explain why avail.idx/used.idx are free-running counters and how the slot index is computed (idx % N).
• You can explain the NEXT and WRITE descriptor flags and why the device must check WRITE before deciding to read or fill a buffer.
• The Step 5 mapping table, each rg run against Firecracker, with the file noted.
• A short note (3–5 sentences) on what differs between rust-vmm's virtio-queue Queue and Firecracker's in-tree Queue, and why Firecracker forks it.

Troubleshooting

pop_descriptor_chain returns None

The device sees no new work. Either avail.idx was never bumped (you must write idx after the ring entry, and it must be > the device's last-seen value, which starts at 0), or the queue addresses were programmed wrong. Print mem.read_obj::<u16>(GuestAddress(AVAIL_ADDR + 2)) — it must be 1.

error[E0432]: unresolved import for Descriptor or the queue traits

virtio-queue API drift. In 0.17 the split descriptor is virtio_queue::desc::split::Descriptor and the traits are QueueT/QueueOwnedT. Older versions exported Descriptor at the crate root and used different trait names. cargo doc --open -p virtio-queue and fix the imports for your version. (verify on docs.rs.)

The chain has the wrong descriptors / walks off the end

Your next field or NEXT flag is wrong. desc0 must set VIRTQ_DESC_F_NEXT and next = 1; desc1 must clear NEXT and set next = 0. A NEXT-without-a-valid-next walks into an unwritten descriptor and yields garbage or an error.

add_used succeeds but the driver reads used.idx = 0

You read the wrong offset. used.idx is at USED_ADDR + 2 (after the 2-byte flags); the first used element {id, len} is at USED_ADDR + 4. Recheck the used-ring layout in Step 1.

Compile error: vm-memory version mismatch

virtio-queue's GuestMemory trait must be the same vm-memory your GuestMemoryMmap comes from. cargo tree -p vm-memory should show a single version. Pin vm-memory to match virtio-queue's requirement. (verify on crates.io.)

Expected Output

$ cargo run
    Finished `dev` profile [unoptimized + debuginfo] target(s)
     Running `target/debug/virtqueue-lab`
[driver] wrote request "PING from the driver" at 0x4000
[driver] desc[0] addr=0x4000 len=20 flags=NEXT next=1
[driver] desc[1] addr=0x5000 len=64 flags=WRITE next=0  (end of chain)
[driver] published chain head=0 in avail.ring[0]; avail.idx now = 1  (KICK)
[device] popped chain, head index = 0
[device]   desc 0: addr=0x4000 len=20 READ (device consumes)
[device]     read request: "PING from the driver"
[device]   desc 1: addr=0x5000 len=64 WRITE (device fills)
[device]   wrote response "PONG from the device" into 0x5000
[device] add_used(head=0, len=20); used.idx bumped
[driver] sees used.idx=1, used.ring[0] = {id=0, len=20}
[driver] response buffer: "PONG from the device"

Stretch Goals

1. Publish two chains and process both. Write a second chain (head = descriptor 2) into avail.ring[1], bump avail.idx to 2, and loop pop_descriptor_chain until it returns None. You now have the exact "drain the available ring" loop a real device runs on each kick.

2. Make a three-descriptor chain. Chain desc0 → desc1 → desc2 (two read buffers feeding one write buffer, say) and confirm the device walks all three. Real block requests are exactly this: a readable header descriptor, readable/writable data descriptors, and a writable status byte.

3. Implement needs_notification. After add_used, call queue.needs_notification(&mem) and print whether the device should raise an interrupt. This is the EVENT_IDX optimization that lets busy queues skip interrupts — rg -n "needs_notification\|EVENT_IDX\|used_event" ~/firecracker.

4. Trigger a malformed chain on purpose. Set desc0.next = 5 while only descriptors 0–1 are valid, and observe how pop_descriptor_chain/iteration handles the out-of-range index. This is the class of guest-supplied-garbage that virtio device code must defend against — the untrusted-guest threat model in miniature. Note how the crate (and Firecracker) bounds-check next.

5. Compare to Firecracker's Queue. Open src/vmm/src/devices/virtio/queue.rs and find the in-tree pop/add_used/index-wrapping code. List three differences from the upstream virtio-queue you just used, and form a hypothesis about why Firecracker forks rather than consumes it (history, control over panics, performance) — fuel for Lab R4.

Validation / Self-check

You are done when you can answer these without notes:

1. Name the three rings of a split virtqueue, which side (driver/device) writes each, and which side reads each.
2. What is in a single descriptor (four fields), and what do the NEXT and WRITE flags mean?
3. Why are avail.idx and used.idx free-running counters, and how do you turn one into a ring slot?
4. Walk the full lifecycle of one request: driver builds a chain → … → driver sees the completion. Name every ring touched and in what order.
5. What does pop_descriptor_chain do under the hood (which ring/index does it read), and what does add_used write and where?
6. Why must the device check the WRITE flag before reading or writing a descriptor's buffer, and what is the security consequence of getting it wrong with an untrusted guest?
7. How does Firecracker's in-tree Queue relate to rust-vmm's virtio-queue, and where did you find each in the source?

When you can lay out a virtqueue by hand and drive it from both sides, the entire virtio device model is open to you. Now step back from building blocks to the ecosystem that maintains them: learn how to contribute changes back upstream in Lab R4: Contribute to rust-vmm.