Lab 4: Build a Virtio Device

Background

This is a build-it lab, and the hardest in the intensive. You have traced block, net, and vsock end to end. Now you implement a complete custom virtio-MMIO device — not the stub you built in Level 7 Lab 7.3, but a working device with real config space, real feature negotiation, a real queue handler that does device-writable output, registration with the device manager, guest advertisement, and a guest program that exercises it. The device is deliberately simple in purpose — a "counter/echo" device — but complete in structure: every piece a real device has, except the upstreaming.

Your device, call it mintsg (a tiny made-up name), does two things:

1. Config space: exposes a 4-byte little-endian request counter the guest can read.
2. One virtqueue: the guest sends a buffer; the device reads it, optionally echoes it back into a device-writable descriptor, bumps the counter, completes the chain, and raises the interrupt.

That is enough to exercise the full VirtioDevice contract: device_type, queues, feature negotiation, read_config/write_config, activate, the queue handler, the EventManager subscription, MMIODeviceManager registration, and guest advertisement. Level 7 had you stop at "the guest discovers a type-42 device with no driver"; here you go further and drive the queue and config space from a guest program over /dev/mem, plus a proper unit test.

Be honest about scope. A learning device on your branch is achievable in this lab. Upstreaming a real device is a large, multi-PR effort with a very high bar — a spec rationale (the minimal-device-model philosophy means "QEMU has it" is not enough), a Persist impl with snapshot tests, integration tests, rate limiting if it does I/O, and two maintainer approvals. This lab teaches you the mechanics so you could take that on — it does not pretend you've shipped a device.

Why This Lab Matters for Contributors

• Implementing VirtioDevice end to end forces you to internalize the activate→process flow, the Queue API, feature/config plumbing, and EventManager wiring — knowledge you cannot fake when reviewing or fixing device code.
• The same moving parts are exactly what every real device-model PR touches; you will be able to read such a PR and know what's missing.
• It makes the minimal-device-model philosophy visceral: you will feel how much surface area even a trivial device adds, and why a new device is a hard sell.

Prerequisites

• Labs 1–3 complete.
• A built firecracker you can iterate on with tools/devtool build.
• Identify your two template devices — the simplest in the tree:

# rng (entropy) is the smallest data path; balloon is config-space-heavy.
rg -l "impl VirtioDevice for" src/vmm/src/devices/virtio/rng/
rg -l "impl VirtioDevice for" src/vmm/src/devices/virtio/balloon/
ls src/vmm/src/devices/virtio/rng/ src/vmm/src/devices/virtio/balloon/
# Read the exact trait surface on YOUR branch — it drifts.
rg -n "trait VirtioDevice" -A 50 src/vmm/src/devices/virtio/

You will copy the shape of rng (queue + activate) and balloon (config space). Read both before writing a line.

The Pieces You Must Build

flowchart TD
    T["1. VirtioDevice impl<br/>device_type · queues · features<br/>read_config/write_config · activate"] --> H["2. Queue handler<br/>pop chain · echo · bump counter · add_used · IRQ"]
    H --> S["3. EventManager subscription<br/>activate registers the queue eventfd"]
    T --> F["4. Feature negotiation<br/>advertise VERSION_1; ack what the driver sets"]
    T --> C["5. Config space<br/>expose the counter at offset 0"]
    H --> R["6. Register with MMIODeviceManager<br/>window + IRQ + cmdline entry"]
    R --> A["7. Advertise to the guest<br/>virtio_mmio.device=SIZE@ADDR:IRQ"]
    A --> X["8. Exercise it<br/>unit test + guest /dev/mem program"]

Pick a non-standard device_type so no in-kernel driver binds (net=1, block=2, rng=4, balloon=5, vsock=19 are taken). Use 42.

Step-by-Step Tasks

Step 1: Read rng and balloon, then create the module

# Answer, in your reading log, from rng:
rg -n "fn device_type|fn queues|fn avail_features|fn acked_features|fn set_acked_features|fn read_config|fn write_config|fn activate" \
   src/vmm/src/devices/virtio/rng/
rg -n "impl.*Subscriber.*for|fn process\b|fn init\b|EventSet|Events::with_data" \
   src/vmm/src/devices/virtio/rng/
# And how balloon does config-space state:
rg -n "fn read_config|fn write_config|config_space|ConfigSpace" \
   src/vmm/src/devices/virtio/balloon/

Create the module and register it where the others are declared:

mkdir -p src/vmm/src/devices/virtio/mintsg
rg -n "pub mod block|pub mod rng|pub mod balloon" src/vmm/src/devices/virtio/mod.rs
# add: pub mod mintsg;

Warning: Every signature, import path, and helper name below drifts between branches. Treat the skeleton as a map. The intended workflow is: paste it, then open rng/device.rs side by side and let the compiler error list be your checklist — copy rng's exact impl VirtioDevice block and replace the bodies. Do not fight the compiler from this text.

Step 2: The device struct and constructor

src/vmm/src/devices/virtio/mintsg/device.rs:

#![allow(unused)]
fn main() {
// SPDX-License-Identifier: Apache-2.0   (match an existing file's header exactly)

use std::sync::atomic::{AtomicU32, Ordering};
use std::sync::Arc;

use vmm_sys_util::eventfd::EventFd;
use vm_memory::GuestMemoryMmap;

// These paths drift — rg each one on your branch and fix:
//   rg -n "struct IrqTrigger|enum IrqType|trait VirtioDevice|struct Queue\b" src/vmm/src/devices/virtio/
use crate::devices::virtio::device::{IrqTrigger, IrqType, VirtioDevice};
use crate::devices::virtio::queue::Queue;
use crate::devices::virtio::ActivateError; // path varies

pub const MINTSG_DEV_TYPE: u32 = 42;     // non-standard: no in-kernel driver binds
pub const MINTSG_QUEUE_SIZE: u16 = 16;
pub const MINTSG_NUM_QUEUES: usize = 1;
pub const CONFIG_LEN: usize = 4;          // a single u32 counter, LE

pub struct Mintsg {
    // virtio plumbing
    avail_features: u64,
    acked_features: u64,
    queues: Vec<Queue>,
    queue_evts: Vec<EventFd>,
    irq_trigger: IrqTrigger,
    activated: bool,

    // device-specific state
    request_count: AtomicU32,             // exposed via config space
    mem: Option<GuestMemoryMmap>,
}

impl Mintsg {
    pub fn new() -> Result<Self, ActivateError> {
        // VIRTIO_F_VERSION_1 is bit 32 — mandatory for the v2 MMIO transport.
        // rg the real constant: rg -n "VIRTIO_F_VERSION_1|VERSION_1" src/vmm/src/devices/virtio/
        let avail_features: u64 = 1u64 << 32;
        let queues = vec![Queue::new(MINTSG_QUEUE_SIZE)];
        let queue_evts = vec![EventFd::new(libc::EFD_NONBLOCK)?];
        Ok(Mintsg {
            avail_features,
            acked_features: 0,
            queues,
            queue_evts,
            irq_trigger: IrqTrigger::new()?,
            activated: false,
            request_count: AtomicU32::new(0),
            mem: None,
        })
    }
}
}

Step 3: Implement the VirtioDevice trait (config + features)

This is the contract. Every method maps onto something you traced in Labs 1–3.

#![allow(unused)]
fn main() {
impl VirtioDevice for Mintsg {
    fn device_type(&self) -> u32 {
        MINTSG_DEV_TYPE
    }

    fn queues(&self) -> &[Queue] {
        &self.queues
    }
    fn queues_mut(&mut self) -> &mut [Queue] {
        &mut self.queues
    }
    fn queue_events(&self) -> &[EventFd] {
        &self.queue_evts
    }

    // ---- Feature negotiation ----
    // We advertise our features; the driver reads them, picks a subset, and writes
    // them back via set_acked_features during the status state machine (Lab: virtio-transport).
    fn avail_features(&self) -> u64 {
        self.avail_features
    }
    fn acked_features(&self) -> u64 {
        self.acked_features
    }
    fn set_acked_features(&mut self, acked: u64) {
        // A real device validates that the driver only acked bits we offered.
        self.acked_features = acked & self.avail_features;
    }

    // ---- Interrupts ----
    fn interrupt_status(&self) -> Arc<AtomicU32> {
        self.irq_trigger.irq_status.clone() // accessor name varies — rg it
    }
    fn interrupt_evt(&self) -> &EventFd {
        &self.irq_trigger.irq_evt           // accessor name varies — rg it
    }

    // ---- Config space: a 4-byte LE counter at offset 0 ----
    fn read_config(&self, offset: u64, data: &mut [u8]) {
        let bytes = self.request_count.load(Ordering::SeqCst).to_le_bytes();
        let off = offset as usize;
        for (i, b) in data.iter_mut().enumerate() {
            *b = bytes.get(off + i).copied().unwrap_or(0);
        }
    }
    fn write_config(&mut self, _offset: u64, _data: &[u8]) {
        // Counter is read-only from the guest; a real device would validate offset+len
        // and apply writable fields here.
    }

    fn is_activated(&self) -> bool {
        self.activated
    }

    // ---- Activation: called once when the guest sets DRIVER_OK ----
    fn activate(&mut self, mem: GuestMemoryMmap) -> Result<(), ActivateError> {
        self.mem = Some(mem);
        // Register the queue eventfd(s) with the EventManager so kicks reach process_queue().
        // Copy rng/device.rs::activate verbatim — the registration call differs by branch.
        self.activated = true;
        Ok(())
    }
}
}

Note on features: VIRTIO_F_VERSION_1 (bit 32) is mandatory for the modern (v2) MMIO transport — without it the driver refuses the device. You can also advertise device-specific feature bits in the low 24 bits; for the learning device, VERSION_1 alone is enough. The status state machine (ACKNOWLEDGE → DRIVER → FEATURES_OK → DRIVER_OK) that drives set_acked_features and then activate is the MMIO transport — re-read it if the ordering is fuzzy.

Step 4: The queue handler (the data path)

src/vmm/src/devices/virtio/mintsg/device.rs (or a sibling), modeled on rng's process fn:

#![allow(unused)]
fn main() {
impl Mintsg {
    pub fn process_queue(&mut self) -> Result<(), ()> {
        let mem = self.mem.as_ref().ok_or(())?.clone();
        let mut raised = false;

        // Drain all available chains.
        while let Some(desc_chain) = self.queues[0].pop(&mem) {
            self.request_count.fetch_add(1, Ordering::SeqCst);

            // Walk the chain. Two descriptors expected:
            //   - a device-READABLE input descriptor (no WRITE flag), and
            //   - a device-WRITABLE output descriptor (WRITE flag set) to echo into.
            // Copy the readable bytes into the writable one. ALWAYS check the WRITE flag
            // before writing a descriptor — an unchecked write is a host memory-safety bug.
            let mut written: u32 = 0;
            // (Pseudocode — adapt to your branch's DescriptorChain iteration API:)
            //
            //   let head = desc_chain.head_index();
            //   let mut in_buf = Vec::new();
            //   for desc in desc_chain {
            //       if desc.is_write_only() {
            //           let n = mem.write_slice(&in_buf, desc.addr).ok();
            //           written = ...;
            //       } else {
            //           mem.read_slice(&mut in_buf, desc.addr)...;
            //       }
            //   }

            // Return the chain to the guest's used ring.
            self.queues[0].add_used(&mem, desc_chain.head_index(), written)
                .map_err(|_| ())?;
            raised = true;
        }

        if raised {
            // Set InterruptStatus and fire the irqfd. Name varies — diff against rng/block.
            self.irq_trigger.trigger(IrqType::Vring).map_err(|_| ())?;
        }
        Ok(())
    }
}
}

The handler is invoked from your Subscriber impl when the queue eventfd fires. Copy the Subscriber/process wiring from rng:

rg -n "impl.*Subscriber.*for|fn process\b|fn init\b|EventSet|Events::with_data|register" \
   src/vmm/src/devices/virtio/rng/

Put the subscriber in src/vmm/src/devices/virtio/mintsg/event_handler.rs, mirroring rng's split.

Warning: This handler runs on the VMM thread — the data plane for every device. No blocking, no slow allocation in the loop. And validate the chain: only write WRITE descriptors, clamp len, treat every guest addr/len/next as hostile (see the virtqueues deep dive). The bounds-checking is the whole reason the minimal device model exists — get it wrong and you have a guest→host escape.

Step 5: Register the device with the MMIODeviceManager

Find where rng/block are constructed and attached at boot, and add yours behind a simple gate (a config flag, an env var, or unconditionally on your learning branch).

rg -n "MMIODeviceManager|register_mmio|attach.*device|build.*rng|build.*block" \
   src/vmm/src/builder.rs src/vmm/src/device_manager/
# The registration call that places the window, allocates the IRQ, and bus-registers:
rg -n "fn register_mmio_virtio|fn register_virtio_device|register_mmio" \
   src/vmm/src/device_manager/

Add a call that constructs Mintsg::new() and registers it exactly as rng is registered. The manager assigns the register window's guest physical address and the IRQ — capture them; you need them in Step 7.

Step 6: Feature negotiation and config space are now wired

You do not write the negotiation loop — the transport does, calling your avail_features, set_acked_features, read_config, write_config as the guest walks the status state machine. Your job was to implement those methods correctly (Step 3). Confirm the transport calls them:

rg -n "read_config|write_config|avail_features|set_acked_features|ack_features|DRIVER_OK|FEATURES_OK" \
   src/vmm/src/devices/virtio/mmio.rs 2>/dev/null \
   src/vmm/src/devices/virtio/transport/ 2>/dev/null

When the guest reads config offset 0, the transport routes it to your read_config and returns the counter. When the guest negotiates features, the transport calls set_acked_features with the bits the driver chose. You implemented both — this step is verification, not new code.

Step 7: Advertise to the guest and boot

The register window exists, but the guest only learns of it via the cmdline (x86) or FDT (aarch64). Registration usually appends virtio_mmio.device=SIZE@ADDR:IRQ automatically — confirm:

rg -n "virtio_mmio.device|add_virtio_device_to_cmdline|append.*virtio_mmio" src/vmm/src/

tools/devtool build
ARCH=$(uname -m); BIN=build/cargo_target/${ARCH}-unknown-linux-musl/debug/firecracker
# Boot as in Lab 1, then in the guest:
cat /proc/cmdline | tr ' ' '\n' | grep virtio_mmio.device
dmesg | grep -i "virtio.*mmio"

Type 42 has no in-kernel driver, so the kernel discovers the MMIO device but binds nothing — that is expected and proves your transport, window, and advertisement are correct. You will talk to it from userspace (Step 8), not via an in-kernel driver.

Step 8: Exercise it — unit test, then a guest /dev/mem program

A. Unit test the queue path (most reliable, do this first). No guest needed.

rg -n "#\[cfg\(test\)\]|GuestMemoryMmap::from_ranges|fn create_virtio_mem|add_avail|VirtqDesc|MockSplitQueue" \
   src/vmm/src/devices/virtio/rng/ src/vmm/src/devices/virtio/test_utils* 2>/dev/null

#![allow(unused)]
fn main() {
#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn mintsg_counts_and_echoes() {
        // 1. Build a small GuestMemoryMmap (copy rng/block's test helper).
        // 2. Lay out a 2-descriptor chain: [0]=readable input "hi", [1]=WRITE output.
        // 3. Set avail.ring[0]=0 and avail.idx=1.
        // 4. Construct Mintsg, activate(mem), call process_queue().
        // 5. Assert: request_count == 1, used.idx advanced, output desc holds "hi".
    }

    #[test]
    fn mintsg_config_reads_counter() {
        // Construct Mintsg, bump request_count, read_config(0, &mut buf[..4]),
        // assert the LE bytes match the counter.
    }
}
}

cargo test -p vmm mintsg            # in the dev container
# or: tools/devtool test -- integration_tests/build/test_unittests.py

B. Guest /dev/mem program (the real-hardware proof). Since no kernel driver binds, drive the device's MMIO register block and config space directly from a guest userspace program. You need the device's register base address (the ADDR from /proc/cmdline). Read the MMIO register map: MagicValue 0x74726976 @0x000, DeviceID @0x008, config space at @0x100.

/* mintsg_poke.c — compile in the guest: gcc -O2 -o mintsg_poke mintsg_poke.c
   Run as root in the guest. ADDR is the virtio_mmio.device base from /proc/cmdline. */
#include <stdio.h>
#include <stdlib.h>
#include <stdint.h>
#include <fcntl.h>
#include <unistd.h>
#include <sys/mman.h>

int main(int argc, char **argv) {
    if (argc < 2) { fprintf(stderr, "usage: %s 0xADDR\n", argv[0]); return 1; }
    uint64_t base = strtoull(argv[1], NULL, 0);
    int fd = open("/dev/mem", O_RDWR | O_SYNC);
    if (fd < 0) { perror("open /dev/mem"); return 1; }
    /* Map one page covering the MMIO register block. */
    volatile uint8_t *p = mmap(NULL, 0x1000, PROT_READ | PROT_WRITE,
                               MAP_SHARED, fd, base & ~0xFFFUL);
    if (p == MAP_FAILED) { perror("mmap"); return 1; }
    volatile uint32_t *reg = (volatile uint32_t *)(p + (base & 0xFFF));

    /* MagicValue @0x000 must read 0x74726976 ("virt"); DeviceID @0x008 must be 42. */
    printf("Magic=0x%08x (want 0x74726976)\n", reg[0x000/4]);
    printf("DeviceID=%u (want 42)\n",          reg[0x008/4]);

    /* Config space starts at 0x100; our counter is the first 4 bytes. */
    printf("counter=%u\n", reg[0x100/4]);
    return 0;
}

# Guest:
ADDR=$(cat /proc/cmdline | tr ' ' '\n' | grep -m1 virtio_mmio.device | sed 's/.*@//; s/:.*//')
./mintsg_poke "$ADDR"
#   Magic=0x74726976 (want 0x74726976)   <- transport register block is live
#   DeviceID=42 (want 42)                <- it's YOUR device
#   counter=0                            <- config space read worked via read_config()

Reading Magic and DeviceID proves the transport register block is wired; reading the counter at config offset 0 proves your read_config is reachable from the guest. Driving the queue from /dev/mem (writing the queue-address registers and QueueNotify) is fiddly and host-dependent — the unit test is the authoritative queue proof; the /dev/mem poke is the transport/config proof.

Tip: /dev/mem access requires the guest kernel to allow it (CONFIG_DEVMEM, and not CONFIG_STRICT_DEVMEM blocking the MMIO range) and the guest running as root. If /dev/mem is locked down, fall back to the unit test for full coverage and treat the poke as a stretch.

Implementation Requirements / Deliverables

• A new mintsg device under src/vmm/src/devices/virtio/ that compiles under tools/devtool build with clippy clean (-D warnings).
• A full impl VirtioDevice for Mintsg: device_type, queues, feature negotiation (avail_features/acked_features/set_acked_features validating the acked subset), config space (read_config exposing the counter), and activate.
• A queue handler that pops a chain, bumps the counter, echoes into a WRITE descriptor with WRITE-flag validation, calls add_used, and raises the interrupt.
• The device registered with the MMIODeviceManager and discovered by the guest kernel (visible in /proc/cmdline and dmesg).
• At least two unit tests: one driving the queue handler against mock guest memory (asserting counter + used ring + echoed bytes), one asserting read_config returns the counter.
• The guest /dev/mem program reading MagicValue, DeviceID == 42, and the config-space counter.
• A short write-up of what a real upstream device additionally needs: spec rationale, a Persist impl + snapshot compat tests, integration tests under tests/, rate limiting if it does I/O, swagger schema + CHANGELOG, and two maintainer approvals.

Troubleshooting

The trait impl won't compile — missing/extra methods

The VirtioDevice surface differs by branch. Don't patch against guesses — open rng/device.rs, copy its entire impl VirtioDevice block, and replace the bodies. The compiler error list is the current trait checklist.

activate() is never called

Either the guest never reached DRIVER_OK (no driver bound — expected for type 42; test the queue via the unit test, not the guest), or your registration didn't place the device on the transport. Confirm the window appears in /proc/cmdline; if not, the MMIODeviceManager registration is wrong.

Guest /dev/mem poke reads Magic=0 or wrong DeviceID

You're reading the wrong base address or the page mapping is off. Recompute ADDR from /proc/cmdline, align the mmap to a page boundary (the code masks ~0xFFF), and index the register within the page (base & 0xFFF). If Magic is right but DeviceID isn't 42, your registration placed a different device at that window — check ordering.

set_acked_features lets the driver ack bits you never offered

That's a real bug — mask with avail_features (as the skeleton does). Add a unit test that acks a superset and asserts only your offered bits survive.

The unit test's pop returns None

Your available ring is wrong: set avail.ring[0] to the head descriptor index and bump avail.idx to 1 before pop. Copy an existing device test's chain-construction helper rather than hand-rolling it.

Clippy fails the build (-D warnings)

Firecracker treats clippy warnings as errors — the same gate a real PR faces. Run tools/devtool fmt and cargo clippy --all --all-targets --all-features -- -D warnings; fix every lint.

Expected Output

# Build + clippy clean with the new module:
> tools/devtool build
   Compiling vmm ...
    Finished dev [unoptimized] profile

# Guest discovers the device (type 42, no driver bound — expected):
$ cat /proc/cmdline | tr ' ' '\n' | grep virtio_mmio.device
virtio_mmio.device=0x1000@0xd0003000:8

# Guest /dev/mem poke:
$ ./mintsg_poke 0xd0003000
Magic=0x74726976 (want 0x74726976)
DeviceID=42 (want 42)
counter=0

# Unit tests pass:
> cargo test -p vmm mintsg
test devices::virtio::mintsg::tests::mintsg_counts_and_echoes ... ok
test devices::virtio::mintsg::tests::mintsg_config_reads_counter ... ok

Stretch Goals

1. Add a device-specific feature bit. Advertise a low-bits feature (e.g. "echo supported") in avail_features, change behavior based on whether the driver acked it, and unit-test both paths.
2. Make config space writable. Add a writable config field (e.g. a "reset counter" trigger), apply it in write_config with proper offset/len validation, and exercise it from the /dev/mem poke.
3. Implement Persist. Give Mintsg a Persist impl (save returns the counter + acked_features; restore rebuilds), mirroring an existing device, and snapshot/restore a microVM that has it. This is required for any real device — see the snapshotting masterclass and Lab 3 there.
4. Write the guest kernel driver. For full realism, write a tiny out-of-tree guest kernel module that binds virtio type 42, posts a buffer on the queue, and reads the echo back — the other half of the boundary this lab deliberately skipped.
5. Add an integration test. Write a pytest under tests/ that boots a microVM with mintsg and asserts the guest discovers it. That's the shape a real device PR must include.
6. Read a real device PR. Find the GitHub PR that added an existing device (e.g. rng/entropy) and list everything it touched beyond these eight pieces — config plumbing, swagger, CHANGELOG, docs, tests. That list is the honest cost of a mergeable device.

Validation / Self-check

Answer without notes; these gate completion.

1. What does device_type() return, and why a non-standard id like 42?
2. Walk the feature-negotiation flow: who calls avail_features, who calls set_acked_features, and why must set_acked_features mask against the offered bits?
3. When is activate() called (in terms of the status state machine), and what must it register so guest kicks reach your handler?
4. In your queue handler, which ring do you read for work and which do you write on completion, and which call raises the guest interrupt?
5. Why must you check the WRITE flag before writing any descriptor, and what bug does skipping it create?
6. How does a guest userspace program reach your config space without a kernel driver, and what does reading MagicValue/DeviceID prove?
7. Name three things a real, upstreamable device needs that mintsg does not — and why the minimal-device-model philosophy makes a new device a hard sell.

Cross-references: virtqueues, virtio-transport-mmio, virtio-rng-entropy, the-mmio-bus-and-device-manager, the-event-manager, minimal-device-model-philosophy, Level 7 Lab 7.3.

This completes the Virtio Devices intensive. You have traced block, net, and vsock end to end and built a complete device of your own. Next, take the device-model issues you are now equipped for from the issue-roadmap virtio stage, or move to the Snapshotting intensive — where the Persist impl you stubbed here becomes a first-class concern.