Project 1: Design and Build a New virtio-MMIO Device

This is the hardest and most complete build in the curriculum. You will take a virtio device from nothing — a device type ID, a feature set, a register block on the MMIO bus, a virtqueue, host-side I/O, an interrupt back to the guest — all the way to a guest that loads a driver and uses it. You will touch the device manager, the MMIO transport, the virtqueue machinery, the event loop, guest memory, the builder, snapshot persistence, the API, and the seccomp filter. By the end you will understand the Firecracker device model the way only someone who has added a device to it does.

It is also the project most likely to not land upstream, and you must internalize that before you start. Firecracker's minimal-device-model philosophy treats every emulated device as host attack surface a malicious guest can reach. "QEMU has it" is not an argument. A genuinely new device needs an RFC, a clear serverless use case, and maintainer buy-in. So the realistic upstream framing of this project is one of: (a) a substantial extension to an existing device that the maintainers already want, (b) a new device that mirrors an existing virtio-spec device with a concrete Firecracker use case, or (c) the device built to maintainer quality locally as the definitive proof that you can build one, with an RFC-grade design note that opens the conversation. All three are real work.

Note: Prerequisites are non-negotiable for this one. You must have done Level 7 end to end, the virtio masterclass — especially Lab 4: Build a virtio device — and read the virtio-transport-mmio, virtqueues, the-mmio-bus-and-device-manager, interrupts-and-irqchip, and the-event-manager deep dives. This brief assumes you already know what a split virtqueue, a kick, an irqfd, and the ACKNOWLEDGE→DRIVER→FEATURES_OK→DRIVER_OK status handshake are. It will not re-derive them.

Problem & motivation

A microVM is only as useful as the devices it exposes. Firecracker ships a tight set — net, block, vsock, rng, balloon, a serial console, a partial i8042 — all over virtio-MMIO (with a newer virtio-PCI transport behind --enable-pci; verify on your branch, and note CVE-2026-5747 was in the PCI transport, fixed in 1.14.4/1.15.1). Each one is a self-contained implementation living in src/vmm/src/devices/virtio/, wired onto the MMIO bus, driven by the EventManager epoll loop, and made snapshot-restorable through the Persist trait.

Adding a device — or substantially extending one — is the canonical "deep, multi-file, full-stack" Firecracker change. It is where every subsystem you have studied separately meets:

• The guest driver negotiates features and sets up a virtqueue in guest RAM.
• The MMIO transport decodes the guest's register reads/writes (MagicValue, Version, DeviceID, QueueSel, QueueNotify, Status, the queue address registers).
• The virtqueue logic walks the descriptor chain, reads the available ring, does the host-side work, writes the used ring, and raises an interrupt.
• The device manager placed the register block and the IRQ, and told the guest via the kernel cmdline (virtio_mmio.device=SIZE@ADDR:IRQ on x86) or the FDT node (aarch64).
• The builder constructed the device; persistence saves and restores its state; the API configures it; the seccomp filter must permit any new syscalls it makes.

The motivation is mastery and leverage. There is no deeper single demonstration of "I understand Firecracker" than a working device. And the extension framing — making an existing device do something the maintainers want — is genuinely upstreamable.

What you'll build

Choose one of three scopes. Scope your Phase 1 to the smallest of these that still teaches the full loop.

Whichever you pick, the artifact is the same: a device that negotiates features, processes a virtqueue, does real host I/O, interrupts the guest, survives snapshot/restore, and is exercised by a guest driver — with unit tests and a pytest integration test.

Prerequisites

• Level 7: The Virtio Device Model — all labs, especially the custom-device project.
• Virtio masterclass and its build-a-device lab.
• Deep dives: virtqueues, virtio-transport-mmio, the-mmio-bus-and-device-manager, interrupts-and-irqchip, the-event-manager, guest-memory-management, snapshotting.
• rust-vmm: virtio-queue (the ring logic), vm-memory (descriptor reads/writes), kvm-ioctls (KVM_IRQFD/KVM_IOEVENTFD).
• The minimal-device-model philosophy essay — read it before you decide to upstream anything.

Phased plan

The discipline here is that Phase 0 and Phase 1 are pure reading and the smallest possible device. Do not write feature code before you can trace an existing device end to end.

Phase 0 — Pick a model device and trace it end to end (2–3 days)

Pick the existing device closest to what you'll build (rng is the simplest complete example; block and net are the richest). Build it, then trace every layer.

tools/devtool build --release
# The smallest full device — read it cover to cover first:
rg -n "VIRTIO_ID|device_type|fn process|fn activate|Queue" src/vmm/src/devices/virtio/rng/
# How devices register on the bus and get an MMIO range + IRQ:
rg -n "MMIODeviceManager|register_mmio|allocate|virtio_mmio.device" src/vmm/src/device_manager/
# The MMIO register decode (MagicValue/Version/DeviceID/QueueNotify/Status):
rg -n "0x74726976|MagicValue|QueueNotify|DeviceStatus|MmioTransport" src/vmm/src/devices/virtio/
# How a device is built and wired into the running Vmm:
rg -n "fn attach.*device|build_microvm_for_boot|fn activate" src/vmm/src/builder.rs src/vmm/src/device_manager/

Produce capstone-work/device-path.md: a single diagram tracing a guest kick → KVM_IOEVENTFD → an epoll event in the VMM thread → the device's process_queue → descriptor-chain walk → host I/O → used-ring write → irqfd → guest IRQ. Cite the files by role (found via rg), never by line number. This is your execution-path-mastery artifact; you cannot skip it.

Phase 1 — The smallest device that processes one queue (the core build)

Stand up a minimal device with: a device type ID, one virtqueue, the VirtioDevice trait implemented (device type, features, config space, activate), and a process_queue that pops descriptors, does a trivial host operation (echo/zero/count), writes the used ring, and signals the IRQ.

# The trait every device implements — find its exact name and methods on your branch:
rg -n "trait VirtioDevice|fn device_type|fn queues|fn activate|fn read_config|fn write_config" src/vmm/src/devices/virtio/
# A model `process` + interrupt-signal to copy the shape from:
rg -n "fn process_queue|add_used|signal_used_queue|IrqTrigger|interrupt" src/vmm/src/devices/virtio/rng/ src/vmm/src/devices/virtio/block/

Wire it onto the MMIO bus and register its activation eventfd with the EventManager so a guest kick wakes it.

#![allow(unused)]
fn main() {
// Sketch — names/shape vary by branch; rg the real trait first.
impl VirtioDevice for Echo {
    fn device_type(&self) -> u32 { ECHO_DEVICE_ID }      // your chosen type ID
    fn queues(&self) -> &[Queue] { &self.queues }
    fn interrupt_trigger(&self) -> &IrqTrigger { &self.irq }
    fn activate(&mut self, mem: GuestMemoryMmap) -> Result<(), ActivateError> {
        // register the queue-event fds with epoll; mark DRIVER_OK reached
        Ok(())
    }
    // read_config/write_config for the device-specific config space
}

fn process_queue(&mut self) {
    while let Some(head) = self.queue.pop(&self.mem) {
        // walk the descriptor chain: read input descriptors, write output ones
        let len = self.handle(head, &self.mem);
        self.queue.add_used(&self.mem, head.index, len);
    }
    self.signal_used_queue();   // raise the IRQ via irqfd
}
}

Milestone 1: a unit test drives a fake virtqueue in guest memory through one request and asserts the used ring and the host side-effect.

Phase 2 — Snapshot/restore and the API

A Firecracker device that does not survive snapshot/restore is incomplete. Implement the Persist trait: a serializable state struct, save, and restore that reconstructs the device and re-activates its queues.

rg -n "trait Persist|fn save|fn restore|impl Persist" src/vmm/src/ src/vmm/src/devices/virtio/
rg -n "VirtioDeviceState|persist|snapshot" src/vmm/src/devices/virtio/block/persist.rs

Then expose configuration through the API (a pre-boot PUT endpoint and a VmResources field), and add the parsed-request → VmmAction plumbing.

# The API surface and the action enum the API thread sends to the VMM thread:
rg -n "VmmAction|enum VmmAction" src/vmm/src/rpc_interface.rs
rg -n "PutDevice|parse_put|fn into_parsed_request" src/firecracker/src/api_server/
rg -n "struct VmResources|fn set_.*device|build_device" src/vmm/src/resources.rs

Milestone 2: a pytest integration test configures the device over the socket, boots, exercises it from the guest, snapshots, restores, and exercises it again.

Phase 3 — The guest driver story

A device with no driver is untestable from inside the guest. You have three options, in increasing effort:

Document the driver path in capstone-work/driver.md: the feature bits negotiated, the queue layout, and exactly how the guest exercises the device.

Phase 4 — Seccomp, hardening, and the design note

Any new syscall your device makes (a new ioctl, a new fd type) must be added to the seccomp filter for the vmm thread category, or Firecracker will be killed at runtime.

rg -n "ioctl|openat|read\b|write\b" resources/seccomp/x86_64.json | head
# Trace which thread category your device runs in (vmm) and what it needs:
rg -n "vmm|vcpu|api|default_action|SyscallRule" resources/seccomp/

Write the RFC-grade capstone-work/design.md: the use case, the device contract, the feature bits, the security argument (why this surface is acceptable), the snapshot-compat story, and the alternatives you rejected. This is the document that would open the upstream conversation.

Key code areas

Anti-staleness: the device-model code is refactored regularly (it was all merged into vmm in a large reorg). Never trust a path or a method name from this brief without an rg on your own branch, and check the CHANGELOG.md and git log --oneline -- src/vmm/src/devices/virtio/ for recent churn before you scope. Confirm the PCI transport state with rg -n "enable_pci|VirtioPci" src/.

Design considerations & trade-offs

• Which transport? virtio-MMIO is the default and the simpler target; the virtio-PCI transport (behind --enable-pci) is newer and was the site of a CVE. Build on MMIO first. Decide explicitly and write it down.
• One queue or many? Multi-queue is where real net/block performance lives, but it multiplies the snapshot state and the epoll wiring. Phase 1 should be single-queue. A multi-queue extension of an existing device is itself a strong Scope-A project.
• Synchronous vs. async host I/O. Block has Sync and io_uring engines for a reason (see io-engines). If your device does real I/O, decide whether it blocks the VMM thread (simple, but a slow op stalls all devices) or goes async (correct, but harder).
• Config space and feature negotiation are a compatibility contract. Once a guest negotiates a feature, you cannot silently change its meaning. Get the bits right; document them.
• Snapshot compatibility is forever. A device's Persist state is part of the snapshot format. Adding a field later means a versioned, backward-compatible change. Design the state struct as if it will outlive you.
• Every byte is attack surface. A malicious guest controls the descriptor chain: lengths, addresses, ring indices. Validate every guest-supplied offset and length against guest memory bounds before you dereference. This is the single most important correctness-and-security property of the whole project.

How to test & validate

• Unit tests (cargo test): build a fake virtqueue in a test GuestMemoryMmap, push descriptors, call process_queue, assert the used ring and the host effect. Test malformed chains (bad indices, lengths past memory end, write-only descriptors used as input) and assert you reject them — this is the security test.

• Integration test (pytest, required for new functionality): add a test under tests/ that configures the device over the API, boots a microVM, exercises the device from the guest, then snapshots and restores and re-exercises it.

  rg -n "def test_|microvm|api_socket|snapshot" tests/integration_tests/ | head
  tools/devtool test -- -k your_device_test
  

• The gates:

  tools/devtool fmt
  tools/devtool checkstyle
  tools/devtool checkbuild --all     # clippy is warnings-as-errors
  

• Restart/seccomp check: run the microVM with the production seccomp filter (not --no-seccomp) and confirm it is not killed — that proves you covered every syscall.

Stretch goals

• Make the device multi-queue and show the throughput change with a benchmark (ties into Project 6).
• Add diff-snapshot support (track_dirty_pages) and confirm the device state round-trips through a diff snapshot, not just a full one.
• Wire a metric for the device into the metrics subsystem (logging-and-metrics deep dive).
• Port the device to the virtio-PCI transport behind --enable-pci and compare the wiring — a deep lesson in why MMIO is the default.
• Add a Kani harness for the descriptor-chain bounds checks (the Kani label marks formal-verification work the maintainers value).

What a strong deliverable looks like

A strong deliverable is a device that negotiates features, processes a virtqueue correctly against an adversarial guest, does real host I/O, interrupts the guest, survives snapshot/restore, and is exercised by a real guest driver in a pytest test — plus the RFC-grade design note that makes the security and use-case argument.

The upstreaming path depends on your scope:

1. Scope A (extend an existing device) is the one that lands. Find the live issue first — gh issue list --repo firecracker-microvm/firecracker --search "virtio multi-queue OR offload OR <your feature>" — comment your intent, and open a focused PR. This is mergeable on its own merits.
2. Scope B (a new spec device) needs an RFC. Open the design note as a GitHub issue, link the minimal-device-model philosophy, and negotiate scope before writing the bulk of the code.
3. Scope C (pedagogical) is local. The deliverable is the working device, the tests, the write-up, and the design note — proof you can build one. It is a portfolio piece, not a PR.

Whatever the scope: one logical change per commit, every commit git commit -s, a CHANGELOG.md entry, the pytest test in the same PR, and ≥2 maintainer approvals if it lands. A finished Scope-A extension at 90+ on the rubric is a real, merged Firecracker contribution in the hardest subsystem in the codebase.

Next: Project 2 — a UFFD page-fault handler if you want to follow the device through to snapshot/restore, or back to the portfolio overview to pick by area.