Lab 7.3: Build It — A Custom Virtio Device

Background

This is a build-it lab, and it is the hardest in the level. You have traced a real device (Lab 7.1) and dissected the transport and the rings (Lab 7.2). Now you will implement a new virtio device of your own — a deliberately tiny echo/counter device — by following the exact shape of the simplest existing devices in the tree. You will implement the VirtioDevice trait (device_type, queues, features, read_config/write_config, activate), wire the device's queue eventfd into the EventManager, register the device with the MMIODeviceManager so it gets a register window and an IRQ, advertise it to the guest, and exercise it from inside a booted microVM.

Be honest with yourself about scope. A learning device that lives on your branch — never sent upstream — is achievable in this lab. Upstreaming a real new device is a large, multi-PR effort with a very high bar: it needs a spec rationale (the minimal-device-model philosophy means "QEMU has it" is not enough), snapshot/Persist support, integration tests, rate-limiting if it does I/O, and two maintainer approvals. This lab teaches you the mechanics of the device model from the inside so that you could take that on later — it does not ask you to ship one.

Pick the simplest possible semantics. Your device has one virtqueue. The guest sends a buffer; the device reads it, computes something trivial (echo the bytes back into a writable descriptor, or just count requests and bump a config field), and completes the chain. No host I/O, no rate limiter, no second queue. Every line of complexity you add is a line you must debug on the hardest path in Firecracker.

Why This Lab Matters for Contributors

  • Implementing VirtioDevice from scratch forces you to internalize the activate → process flow, the Queue API, and the feature/config plumbing — knowledge you cannot fake when reviewing or fixing device code.
  • Even though you will not upstream this device, the same five moving parts (trait impl, queue handler, event subscription, manager registration, guest advertisement) are exactly what every real device-model PR touches.
  • It makes the minimal-device-model-philosophy visceral: you will feel how much surface area even a trivial device adds.

Prerequisites

  • Lab 7.1 and Lab 7.2 complete — you can trace a device, read the transport, and read the Queue.
  • A built firecracker you can iterate on with tools/devtool build.
  • Identify your two template devices (the simplest in the tree):
# The entropy (rng) device is the smallest data path; balloon is config-heavy but small.
rg -l "impl VirtioDevice" src/vmm/src/devices/virtio/rng/
rg -l "impl VirtioDevice" src/vmm/src/devices/virtio/balloon/
ls src/vmm/src/devices/virtio/rng/ src/vmm/src/devices/virtio/balloon/

You will copy the shape of these. Read both before writing a line of your own.

The Five Pieces You Must Build

flowchart TD
    T[1. VirtioDevice impl<br/>device_type/queues/features/config/activate] --> H[2. queue handler<br/>pop chain, do trivial work, add_used, raise IRQ]
    H --> S[3. EventManager subscription<br/>activate registers the queue eventfd]
    T --> R[4. Register with MMIODeviceManager<br/>get a register window + IRQ]
    R --> A[5. Advertise to the guest<br/>cmdline virtio_mmio.device / FDT node]
#PieceTemplate to copyWhat it does
1VirtioDevice implrng/ deviceThe trait: type id, queues, features, config read/write, activate.
2Queue handlerrng/ device's process fnPop a chain, do the trivial work, write used ring, signal interrupt.
3EventManager subscriptionany device's activate + Subscriber implMake queue kicks reach your handler.
4MMIODeviceManager registrationhow block/rng are wired in builder.rs/device_manager/Place the register window, allocate the IRQ.
5Guest advertisementthe virtio_mmio.device cmdline builderTell the guest where the device lives.

Pick a device type id that does not collide. Firecracker uses net=1, block=2, rng=4, balloon=5, vsock=19. For a learning device, use a clearly non-standard id (e.g. 42) so nothing in the guest tries to bind a real driver to it — you will talk to it from userspace via /dev/mem or a tiny test, not via an in-kernel virtio driver.

Note: A real device needs a guest-side driver. The Linux kernel has drivers for the standard device types; for a brand-new type you would need a guest kernel module too. To keep this lab to one side of the boundary, your "guest exercise" reads/writes the device's MMIO config space directly (which exercises read_config/write_config), and/or you validate the queue path with a unit test that drives a Queue against a mock GuestMemoryMmap. That is enough to prove the device works without writing a kernel driver.

Step-by-Step Tasks

Step 1: Read the two template devices end to end (45 min)

Do not skip this. Open rng/ and read every file. Answer, in your reading log:

rg -n "fn device_type"  src/vmm/src/devices/virtio/rng/
rg -n "fn queues\b\|queues:\|fn queues_mut" src/vmm/src/devices/virtio/rng/
rg -n "fn features\|avail_features\|fn ack_features\|acked_features" src/vmm/src/devices/virtio/rng/
rg -n "fn read_config\|fn write_config" src/vmm/src/devices/virtio/rng/
rg -n "fn activate" src/vmm/src/devices/virtio/rng/
rg -n "impl.*Subscriber.*for\|fn process\b\|fn handle_.*event\|fn init\b" src/vmm/src/devices/virtio/rng/
  • What does device_type() return, and where is the constant?
  • How many queues does it declare, and what sizes?
  • How does it advertise features, and which does it ack?
  • What does activate() register with the EventManager?
  • Which function is the queue handler, and how does it pop / process / add_used / signal?

Then skim balloon/ for how a device with config-space state implements read_config / write_config (you will reuse that for your counter).

Step 2: Create the module skeleton (20 min)

Create a new device subdirectory. Mirror the file split your template uses (often a mod.rs / device.rs for the struct + trait impl, an event_handler.rs for the Subscriber, and a persist.rs you can stub).

mkdir -p src/vmm/src/devices/virtio/echo
# Register the module wherever the parent declares the others:
rg -n "pub mod block\|pub mod rng\|pub mod balloon\|mod block" src/vmm/src/devices/virtio/mod.rs
# add: pub mod echo;

Skeleton (src/vmm/src/devices/virtio/echo/device.rs). Names, imports, and trait signatures drift between branches — paste this, then let the compiler and a side-by-side diff against rng/device.rs correct every signature. Treat it as a map, not as copy-paste-ready code.

#![allow(unused)]
fn main() {
// SPDX-License-Identifier: Apache-2.0  (match the header style of an existing file)

use std::sync::atomic::{AtomicU32, Ordering};
use std::sync::Arc;

use vmm_sys_util::eventfd::EventFd;
use vm_memory::GuestMemoryMmap;

use crate::devices::virtio::device::{IrqTrigger, VirtioDevice}; // path varies — rg it
use crate::devices::virtio::queue::Queue;                       // path varies — rg it

// A non-standard learning type id so no real guest driver binds to it.
pub const ECHO_DEV_TYPE: u32 = 42;
pub const ECHO_QUEUE_SIZE: u16 = 16;
pub const ECHO_NUM_QUEUES: usize = 1;

pub struct Echo {
    // virtio plumbing
    avail_features: u64,
    acked_features: u64,
    queues: Vec<Queue>,
    queue_evts: Vec<EventFd>,
    irq_trigger: IrqTrigger,           // wraps the irqfd + InterruptStatus (rg the real type)
    activated: bool,

    // device-specific state: a request counter exposed via config space
    request_count: AtomicU32,
    mem: Option<GuestMemoryMmap>,
}

impl Echo {
    pub fn new() -> Result<Self, ActivateError> {
        // VIRTIO_F_VERSION_1 is bit 32 — mandatory for the v2 MMIO transport.
        let avail_features = 1u64 << 32; // (1 << VIRTIO_F_VERSION_1); rg the constant
        let queues = vec![Queue::new(ECHO_QUEUE_SIZE)];
        let queue_evts = vec![EventFd::new(libc::EFD_NONBLOCK)?];
        Ok(Echo {
            avail_features,
            acked_features: 0,
            queues,
            queue_evts,
            irq_trigger: IrqTrigger::new()?,
            activated: false,
            request_count: AtomicU32::new(0),
            mem: None,
        })
    }
}
}

Step 3: Implement the VirtioDevice trait (40 min)

This is the core. Every method maps onto something you read in Lab 7.2. Copy the exact method signatures from your branch's trait — they differ:

rg -n "trait VirtioDevice" -A 40 src/vmm/src/devices/virtio/

#![allow(unused)]
fn main() {
impl VirtioDevice for Echo {
    fn device_type(&self) -> u32 {
        ECHO_DEV_TYPE
    }

    fn queues(&self) -> &[Queue] {
        &self.queues
    }
    fn queues_mut(&mut self) -> &mut [Queue] {
        &mut self.queues
    }
    fn queue_events(&self) -> &[EventFd] {
        &self.queue_evts
    }

    fn avail_features(&self) -> u64 {
        self.avail_features
    }
    fn acked_features(&self) -> u64 {
        self.acked_features
    }
    fn set_acked_features(&mut self, acked: u64) {
        self.acked_features = acked;
    }

    fn interrupt_status(&self) -> Arc<AtomicU32> {
        self.irq_trigger.irq_status.clone() // rg the real accessor name
    }
    fn interrupt_evt(&self) -> &EventFd {
        &self.irq_trigger.irq_evt          // rg the real accessor name
    }

    // Config space: expose the request counter at offset 0 (4 bytes, little-endian).
    fn read_config(&self, offset: u64, data: &mut [u8]) {
        let count = self.request_count.load(Ordering::SeqCst).to_le_bytes();
        for (i, b) in data.iter_mut().enumerate() {
            let idx = offset as usize + i;
            *b = count.get(idx).copied().unwrap_or(0);
        }
    }
    fn write_config(&mut self, _offset: u64, _data: &[u8]) {
        // read-only config for the learning device; reject or ignore writes
    }

    fn is_activated(&self) -> bool {
        self.activated
    }

    // Called once, when the guest sets DRIVER_OK. Register the queue eventfd with
    // the EventManager so kicks reach your handler. Diff against rng/device.rs::activate.
    fn activate(&mut self, mem: GuestMemoryMmap) -> Result<(), ActivateError> {
        self.mem = Some(mem);
        // register self (as a Subscriber) and self.queue_evts[0] with the EventManager.
        // The exact registration is in your branch's activate() — copy it verbatim.
        self.activated = true;
        Ok(())
    }
}
}

Warning: The trait surface (queues vs queues_mut, whether set_acked_features exists, whether interrupt_status returns an Arc<AtomicU32> or something wrapped) changes between branches. Do not fight the compiler from this skeleton — open rng/device.rs, copy its impl VirtioDevice block, and edit the bodies. That is the intended workflow.

Step 4: Implement the queue handler (the data path) (40 min)

When the guest kicks queue 0, your handler pops the chain, does the trivial work (echo / count), and completes it. Model it on the rng device's process function.

#![allow(unused)]
fn main() {
impl Echo {
    fn process_queue(&mut self) -> Result<(), ()> {
        let mem = self.mem.as_ref().ok_or(())?;
        let mut used_any = false;

        while let Some(head) = self.queues[0].pop(mem) {
            // Walk the descriptor chain. For an echo device:
            //   - a device-readable descriptor (input bytes), and
            //   - a device-writable descriptor (WRITE flag) to echo into.
            // For a counter device, you can ignore the payload and just count.
            self.request_count.fetch_add(1, Ordering::SeqCst);

            let mut written: u32 = 0;
            // (Optional echo:) copy from the readable desc into the WRITE desc using
            // mem.read_slice / mem.write_slice via the descriptor addresses. ALWAYS check
            // the WRITE flag before writing a descriptor — see Lab 7.2.

            // Return the chain to the guest.
            self.queues[0].add_used(mem, head.index, written);
            used_any = true;
        }

        if used_any {
            // Mark InterruptStatus and raise the IRQ (irqfd). rg the real call.
            self.irq_trigger.trigger(IrqType::Vring)?; // name varies — diff against rng/block
        }
        Ok(())
    }
}
}

The handler is invoked from your Subscriber impl when the queue eventfd fires. Copy the Subscriber/process wiring from your template:

rg -n "impl.*Subscriber.*for\|fn process\b\|fn init\b\|EventSet\|Events::with_data" \
   src/vmm/src/devices/virtio/rng/

Warning: This handler runs on the VMM thread. Do nothing blocking, nothing slow, no allocation in a hot loop you can avoid. And validate the chain: only write descriptors with the WRITE flag, clamp lengths, and treat every guest-supplied addr/len as hostile. An unchecked write here is a host memory-safety bug — exactly the class of issue the minimal device model exists to minimize.

Step 5: Register the device with the MMIODeviceManager (30 min)

Find where existing devices are constructed and attached during boot, and add yours behind a simple gate (a config flag, an env var, or unconditionally for your learning branch).

# Where block/rng/etc. are built and registered:
rg -n "MMIODeviceManager\|register_mmio\|attach.*device\|build.*block\|build.*rng" \
   src/vmm/src/builder.rs src/vmm/src/device_manager/

# The registration call you must mirror (it places the window + allocates an IRQ + bus-registers):
rg -n "fn register_mmio_virtio\|fn register_virtio_device\|register_mmio" src/vmm/src/device_manager/

Add a call that constructs Echo::new() and registers it the same way the rng device is registered. The manager will assign the register window's guest physical address and the IRQ; capture them — you need them for Step 6.

Step 6: Advertise the device to the guest (20 min)

The register window exists, but the guest does not know about it unless Firecracker adds it to the cmdline (x86) or FDT (aarch64). The MMIODeviceManager registration usually appends the virtio_mmio.device=SIZE@ADDR:IRQ cmdline entry automatically — confirm yours did:

rg -n "virtio_mmio.device\|add_virtio_device_to_cmdline\|append.*virtio_mmio" src/vmm/src/

Boot and verify the guest sees it:

tools/devtool build
ARCH=$(uname -m); BIN=build/cargo_target/${ARCH}-unknown-linux-musl/debug/firecracker
# boot as in Lab 7.1, then inside the guest:
cat /proc/cmdline | tr ' ' '\n' | grep virtio_mmio.device
dmesg | grep -i "virtio.*42\|unknown virtio device"   # the kernel will note an unbound type-42 device

Because type 42 has no in-kernel driver, the guest kernel will discover the MMIO device but not bind a driver — that is fine and expected for a learning device. The discovery itself proves your transport, register window, and advertisement are correct.

Step 7: Exercise it (30 min)

You have two ways to prove the device works, in increasing realism:

A. Unit test the queue path (most reliable). This needs no guest driver. Build a Queue over a mock GuestMemoryMmap, hand-craft a descriptor chain, call your handler, and assert the used ring and the counter advanced. Copy the test harness from an existing device's tests:

rg -n "#\[cfg(test)\]\|fn create_virtio_mem\|GuestMemoryMmap::from_ranges\|fn add_avail\|mock" \
   src/vmm/src/devices/virtio/rng/ src/vmm/src/devices/virtio/test_utils* 2>/dev/null

#![allow(unused)]
fn main() {
#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn echo_counts_one_request() {
        // 1. Build a small GuestMemoryMmap (copy the helper rng/block tests use).
        // 2. Lay out a descriptor chain + avail ring entry in that memory.
        // 3. Construct Echo, activate(mem), then call process_queue().
        // 4. Assert request_count == 1 and the used ring idx advanced.
    }
}
}

Run it:

tools/devtool test -- integration_tests/build/test_unittests.py   # or cargo test in the container
# or, scoped:
cargo test -p vmm echo

B. Poke the config space from the guest (optional). Read the device's config space (offset 0x100, the request counter) over /dev/mem from inside the guest at the device's register address. This is fiddly and host-dependent, but reading the counter via read_config proves the transport end-to-end. Treat it as a stretch.

Tip: Lead with the unit test. It is deterministic, runs in CI shape, and directly exercises the exact code real device PRs are judged on. The guest-side poke is a nice-to-have.

Implementation Requirements / Deliverables

  • A new echo (or counter) device under src/vmm/src/devices/virtio/ that compiles as part of tools/devtool build.
  • A full impl VirtioDevice for Echo with device_type, queues, features, read_config, activate, modeled on the rng device.
  • A queue handler that pops a chain, advances a counter (and optionally echoes), calls add_used, and raises the interrupt — with WRITE-flag validation on any descriptor it writes.
  • The device registered with the MMIODeviceManager and discovered by the guest kernel (visible in /proc/cmdline and dmesg).
  • At least one unit test that drives the queue handler against a mock guest memory and asserts the used ring and counter advanced.
  • A short write-up of what a real upstream device would additionally need (spec rationale, Persist/snapshot support, integration tests, rate limiting, two approvals) — proving you understand the gap between a learning device and a mergeable one.

Troubleshooting

The trait impl will not compile — missing/extra methods

The VirtioDevice trait surface differs by branch. Do not patch the skeleton method by method against guesses — open rng/device.rs, copy its entire impl VirtioDevice block, and replace the bodies. The compiler error list is your checklist of the current trait.

activate() is never called

Either the device never reached DRIVER_OK (no driver bound, which is expected for type 42 — in that case test via the unit test, not via the guest), or your registration did not place the device on the bus/transport. Confirm the register window appears in the guest /proc/cmdline; if it does not, your MMIODeviceManager registration is wrong.

The guest kernel panics or hangs at boot after adding the device

A malformed cmdline entry (virtio_mmio.device=...) or an overlapping register window will derail discovery. Re-check that the manager assigned a non-overlapping address and a free IRQ, and that the SIZE@ADDR:IRQ string is well-formed. Boot without your device to confirm the rest still works, then re-add it.

The unit test's pop returns None

You did not lay out the available ring correctly: set avail.ring[0] to the head descriptor index and bump avail.idx to 1 before calling pop. Re-read the layout in Lab 7.2 and copy an existing device test's chain-construction helper.

Clippy fails the build (-D warnings)

Firecracker treats clippy warnings as errors. Run tools/devtool fmt and cargo clippy --all --all-targets --all-features -- -D warnings; fix every lint before you consider the lab done — this is the same gate a real PR faces.

Expected Output

# Build succeeds with the new module:
> tools/devtool build
   Compiling vmm ...
    Finished dev [unoptimized] profile

# Guest sees the device discovered (type 42, no driver bound — expected):
$ cat /proc/cmdline | tr ' ' '\n' | grep virtio_mmio.device
virtio_mmio.device=0x1000@0xd0003000:8
$ dmesg | grep -i virtio
virtio-mmio virtio-mmio.3: Failed to identify device   # or: unknown device type 42 — fine for a learning device

# The unit test passes:
> cargo test -p vmm echo
test devices::virtio::echo::tests::echo_counts_one_request ... ok

Stretch Goals

  1. Make echo real. Implement the device-writable path: copy the bytes from the readable descriptor into the WRITE descriptor, set len correctly in add_used, and write a guest-side test (a tiny kernel module or a /dev/mem poke) that sends "hello" and reads it back.
  2. Add config-space writes. Let the guest write a value to config space that the device echoes back, and add a unit test that exercises read_config/write_config round-trips.
  3. Implement Persist. Give your device a Persist impl so it can be snapshotted, mirroring an existing device. This is exactly the Level 9 / snapshotting concern, and it is required for any real device.
  4. Add an integration test. Write a pytest under tests/ that boots a microVM with your device and asserts the guest discovers it. This is the shape of the test a real device PR must include.
  5. Read a real device PR. Find the PR that added an existing device (e.g. entropy/rng) on GitHub and list everything it touched beyond the five pieces here — config plumbing, the swagger schema, the changelog, docs. That list is the honest cost of a real device.

Validation / Self-check

Answer without notes; these gate completion.

  1. What does device_type() return for your device, and why did you choose a non-standard id?
  2. When is activate() called, and what must it register so that guest kicks reach your handler?
  3. In your queue handler, which ring do you read to find work and which do you write on completion, and which call raises the guest interrupt?
  4. Why must you check the WRITE flag before writing any descriptor, and what bug does skipping it create?
  5. How did the guest kernel come to know your device's register window exists?
  6. Why is a unit test against mock guest memory a more reliable proof than a guest-side poke for a non-standard device type?
  7. Name three things a real, upstreamable device needs that your learning device does not — and why the minimal-device-model philosophy makes a new device a hard sell.

Cross-references: virtqueues, virtio-transport-mmio, virtio-block, virtio-rng-entropy, the-mmio-bus-and-device-manager, rust-vmm virtio-queue, minimal-device-model-philosophy.

This completes Level 7. Next: Level 8 — Real Issue Contribution, where you leave curated labs behind and take a real GitHub issue end to end. The device-model issues you are now equipped for are catalogued in the issue-roadmap stage on virtio devices.