The Virtio Device Model — Intensive

Everything a Firecracker guest touches that isn't raw CPU or raw RAM goes through virtio. Disk I/O is virtio-block. The network is virtio-net over a host TAP. The host↔guest control channel is virtio-vsock. Entropy, the memory balloon, pmem — all virtio. If you cannot trace a virtio device from the guest driver's kick down to the host syscall and back through the used ring, you cannot debug, review, or extend the half of Firecracker that operators actually file bugs against. This masterclass makes you do exactly that, four times, for the three devices that matter most — and then makes you build one.

You have already met the pieces. Level 7 taught you the virtqueue (the three rings, the descriptor chain, the kick and the interrupt), the MMIO transport (the register block, the status state machine, feature negotiation), and the MMIO bus and device manager (how a device gets a register window and an IRQ). This intensive does not re-teach those — it uses them, relentlessly, on real hardware paths you instrument and measure. Where Level 7 had you trace block once and stub a counter device, here you trace block and net and vsock end to end with strace, tcpdump, fio, and your own log points, and you build a complete custom virtio-MMIO device with real queue processing, config space, feature negotiation, and snapshot honesty.

Note: "Virtio device" in Firecracker means a Rust struct that implements the VirtioDevice trait, lives under src/vmm/src/devices/virtio/, runs its queue handler on the VMM thread's EventManager epoll loop, and talks to the guest over the virtio-MMIO transport (PCI exists behind --enable-pci — verify on your branch). Hold that definition in your head; every lab here is a different instance of it.

What you will be able to do

After this intensive you can:

1. Trace a guest block I/O from the descriptor chain through the MMIO kick, the ioeventfd, the device queue handler, a host pread/pwrite (or io_uring_enter), the used ring, and the injected IRQ — and measure each engine.
2. Stand up a host TAP, route it, attach a virtio-net interface, SSH into the guest, and trace an RX and a TX frame through the TAP fd and the two virtqueues while watching tcpdump.
3. Configure virtio-vsock, run host↔guest socket traffic in both directions, and explain the CID/port multiplexing and the host-Unix-socket bridge that firecracker-containerd and Kata depend on.
4. Implement a complete custom virtio-MMIO device — VirtioDevice trait, activate, queue processing via the EventManager, config space, feature negotiation — register it, advertise it to the guest, and exercise it.
5. Apply a token-bucket rate limiter to a device and watch it throttle in real syscall traces.
6. Speak about the device model the way a maintainer does: surface area, security boundaries, the Persist requirement, and why "QEMU has it" is not an argument.

The shape every virtio device shares

Before the labs, fix the skeleton. Every device in src/vmm/src/devices/virtio/ is the same five moving parts; the labs differ only in what fills them.

┌──────────────────────────────────────────────────────────────────────────┐
│ GUEST                                                                      │
│   virtio-blk / virtio-net / virtio-vsock driver in the guest kernel        │
│     builds descriptor chains in GUEST memory, kicks QueueNotify (MMIO)     │
└───────────────────────────────┬────────────────────────────────────────────┘
                                │ MMIO write → KVM_IOEVENTFD (no VM exit to userspace)
                                ▼
┌──────────────────────────────────────────────────────────────────────────┐
│ VMM THREAD  (one EventManager epoll loop)                                  │
│   queue eventfd fires → device's queue handler runs:                       │
│     1. Queue::pop  → walk chain, bounds-check every addr/len/next          │
│     2. do the work → host pread/pwrite (block), TAP read/write (net),      │
│        socket bridge (vsock), trivial compute (your device)                │
│     3. add_used(head, len); used.idx++                                      │
│     4. raise IRQ via irqfd; InterruptStatus |= USED_RING                    │
└───────────────────────────────┬────────────────────────────────────────────┘
                                │ KVM_IRQFD → guest interrupt
                                ▼
                         guest driver reaps used ring

The trait is the contract. Find it on your branch — do not trust a remembered signature, it drifts:

# The VirtioDevice trait surface every device implements.
rg -n "trait VirtioDevice" -A 40 src/vmm/src/devices/virtio/
# The device subdirectories you will trace and copy from.
ls src/vmm/src/devices/virtio/
# Confirm each device implements the trait.
rg -l "impl VirtioDevice for" src/vmm/src/devices/virtio/

The four labs

Do them in order. Block is the simplest interesting device (one queue, a fixed three-descriptor request shape) and teaches the path; net adds a second queue and an external fd (the TAP); vsock adds a per-connection state machine and a host socket bridge; building one forces you to internalize all of it from the inside. Lab 4 is deliberately deeper than Level 7 Lab 7.3 — there you stubbed a counter; here you build a working device with real config space and feature negotiation and run a guest program against it.

Prerequisites

This intensive assumes Level 7 complete. Concretely:

• You finished Level 7: you traced a virtio-block I/O (Lab 7.1), dissected the rings and the MMIO transport (Lab 7.2), and stubbed a custom device (Lab 7.3).
• You have internalized the virtqueues deep dive and the MMIO transport deep dive. Re-read them if the words "available ring", "used ring", "kick", "ioeventfd", and "DRIVER_OK" do not produce instant mental pictures.
• You can build with tools/devtool build, boot a microVM by hand (Lab 1.3), and read the vCPU run loop and VM exits (Level 4).
• You have root on a Linux host with a working /dev/kvm (these labs create TAP devices, run strace -p, and tcpdump — all of which need privilege). Run firecracker without the jailer for these labs so strace -p, TAP paths, and backing files are straightforward; the jailer is Level 9 and the security masterclass.

Verify your checkout has the three devices and the queue code before you start:

# All four must return hits. If a path is empty, the layout moved — find by role.
rg -l "impl VirtioDevice for" src/vmm/src/devices/virtio/block/
rg -l "impl VirtioDevice for" src/vmm/src/devices/virtio/net/
rg -l "impl VirtioDevice for" src/vmm/src/devices/virtio/vsock/
rg -n "struct Queue\b"        src/vmm/src/devices/virtio/queue.rs

# And that you can build and boot. (Default libc is musl; verify your arch.)
tools/devtool build
ARCH=$(uname -m)
ls -l build/cargo_target/${ARCH}-unknown-linux-musl/debug/firecracker

How these labs connect to real contribution

The device model is where a large share of Firecracker's open issues live. The issue-roadmap stage on virtio devices catalogues them: rate-limiter accounting bugs, io_uring vs Sync discrepancies, TAP error handling, vsock connection-state edge cases, descriptor-validation hardening. Every one of those is debugged by walking the exact paths you trace here. A contributor who can produce a clean strace correlating a guest sector to a host file offset, or a tcpdump showing a frame crossing the TAP, writes bug reports and fixes that maintainers take seriously — because the evidence is in the trace, not in a guess.

The deep dives are your reference throughout. Keep them open:

Begin with Lab 1 — Virtio-Block, End to End: set up a backing file, attach it, run fio, and trace one I/O all the way to a host pread and back.