event-manager

The VMM thread does not poll. It sits in an epoll loop: it blocks in epoll_wait until one of the file descriptors it cares about becomes ready — a virtio device's ioeventfd (the guest kicked a queue), the API wake-up eventfd (a control action arrived), a rate-limiter timer, host stdin (a console keystroke) — and then dispatches to whoever owns that fd. Managing a set of fds, their interest masks, and the "when this fd is ready, call this handler" mapping, cleanly and safely, is the job of rust-vmm's event-manager. It is the VMM threading model's engine and the subject of the event manager deep dive — and, like vm-superio and seccompiler, it is a crate Firecracker donated upstream: it was derived from Firecracker's own EventManager.

After this chapter you can: name the core types (EventManager, EventSubscriber/MutEventSubscriber, EventOps, Events, EventSet, SubscriberId); explain the subscriber model and how registration works; implement a subscriber; and find where Firecracker's VMM thread and devices use it.

Note: This is the third Firecracker donation. The subscriber pattern you implement here is the exact pattern every Firecracker device follows on the epoll loop. Learn it once and the whole device-dispatch story falls into place.

# Confirm the dependency and the pinned version (verify on your branch):
rg -n "event-manager" Cargo.toml src/vmm/Cargo.toml Cargo.lock
cargo doc -p event-manager --no-deps --open

docs.rs: docs.rs/event-manager.

The problem: an epoll loop with many owners

Raw epoll is awkward to share. One thread owns the epoll fd; many independent components (each device, the API plumbing, the timers) each have their own fds and their own logic for what to do when ready. Without an abstraction you get a giant match on raw fd numbers and manual bookkeeping for which fd belongs to which device, what interest mask each wants, and how to add/remove fds at runtime. event-manager replaces that with a registry of subscribers: each component registers the fds it cares about and an object that knows how to handle them; the manager runs the loop and calls the right object when its fd fires.

flowchart TD
    EM["EventManager (owns the epoll fd)"]
    EM -->|epoll_wait fires fd| D1["block device subscriber"]
    EM -->|epoll_wait fires fd| D2["net device subscriber"]
    EM -->|epoll_wait fires fd| API["API wake-up eventfd subscriber"]
    EM -->|epoll_wait fires fd| CON["serial/stdin subscriber"]
    D1 -. "registered (eventfd, EventSet::IN, SubscriberId)" .-> EM
    D2 -. register .-> EM

The core types

The trait split — EventSubscriber (shared &self) vs MutEventSubscriber (&mut self) — exists because some subscribers need mutable state in process (most devices do: they mutate their queues), and some don't. Firecracker's devices are MutEventSubscriber.

find ~/.cargo/registry/src -maxdepth 2 -type d -name 'event-manager-*' \
  -exec rg -n "trait EventSubscriber|trait MutEventSubscriber|struct EventOps|struct Events|struct EventSet|fn process|fn init|fn run\b|add_subscriber" {} +

Implementing a subscriber

A subscriber implements two methods: init, where it tells the manager which fds it wants (via EventOps), and process, where it does the work when one of those fds is ready. Here is a minimal subscriber that watches one EventFd and reacts when it fires — the exact pattern a device follows.

#![allow(unused)]
fn main() {
use event_manager::{EventOps, Events, EventSet, MutEventSubscriber};
use vmm_sys_util::eventfd::EventFd;

struct MyDevice {
    queue_evt: EventFd,   // the ioeventfd the guest "kicks" (vmm-sys-util)
}

impl MutEventSubscriber for MyDevice {
    // Called once when registered: declare interest in our fd.
    fn init(&mut self, ops: &mut EventOps) {
        ops.add(Events::with_data(
            &self.queue_evt,
            0,                 // a token you choose to identify this fd
            EventSet::IN,      // readable
        )).expect("register queue_evt");
    }

    // Called when one of our fds is ready.
    fn process(&mut self, events: Events, _ops: &mut EventOps) {
        if events.event_set().contains(EventSet::IN) {
            // Drain the eventfd (the guest kicked a queue)...
            let _ = self.queue_evt.read();
            // ...then do the real work: pop the virtqueue, do host I/O,
            // add_used, raise the completion interrupt. (See virtio-queue.md.)
        }
    }
}
}

Registration returns a SubscriberId; the manager calls init once, then calls process every time epoll reports one of that subscriber's fds ready. To add or remove fds later (a device that gains/loses a queue at runtime), the subscriber uses the EventOps handed to it. The loop driver is the manager:

#![allow(unused)]
fn main() {
use event_manager::{EventManager, SubscriberOps};

let mut mgr: EventManager<MyDevice> = EventManager::new().unwrap();
let _id = mgr.add_subscriber(MyDevice { queue_evt });
loop {
    // Block until something is ready, then dispatch to the matching subscriber's
    // process(). This is the VMM thread's entire life.
    mgr.run().unwrap();
}
}

Warning: process runs on the VMM thread, synchronously, holding up the whole loop until it returns. A device handler that blocks (a slow host syscall, a lock it can't get) stalls every device and the API wake-up too. This is the mechanical reason behind the threading-model rule that the VMM loop must never block — see the common bugs in the threading model.

How Firecracker uses it

# The VMM thread's EventManager and its run loop:
rg -n "EventManager|event_manager|add_subscriber|\.run\(\)|SubscriberOps" src/vmm/src/

# Devices implementing the subscriber trait:
rg -n "impl MutEventSubscriber|impl EventSubscriber|fn process\b|fn init\b|EventOps|EventSet::IN" src/vmm/src/devices/

# The API wake-up eventfd is also a subscriber on this loop:
rg -n "api_event_fd|EventFd|process.*api|EventSet" src/vmm/src/

The unifying picture: the VMM thread is one EventManager::run() loop, and every device, timer, and the API plumbing is a subscriber on it. That is why a Firecracker device is fundamentally "a struct that implements MutEventSubscriber and owns some eventfds." When you write a virtio device in Lab 7.3, the subscriber boilerplate is event-manager.

The connection to KVM is the ioeventfd: Firecracker registers a device's queue eventfd with KVM via KVM_IOEVENTFD so that a guest write to the device's QueueNotify MMIO register signals the eventfd directly in the kernel — no VM exit round-trip to userspace for the notification. The eventfd becomes ready, epoll returns, the subscriber's process runs. That is the virtio fast path, and event-manager is the userspace half of it.

sequenceDiagram
    participant G as guest driver
    participant K as KVM
    participant E as ioeventfd (vmm-sys-util EventFd)
    participant M as EventManager (VMM thread)
    participant D as device subscriber
    G->>K: write QueueNotify MMIO (kick)
    K->>E: KVM_IOEVENTFD signals the eventfd (no VM exit to userspace)
    E->>M: epoll_wait returns
    M->>D: process(Events{IN})
    D->>D: pop chain, host I/O, add_used, raise IRQ

Reading exercise

# 1. The dependency and pinned version.
rg -n "event-manager" Cargo.toml Cargo.lock

# 2. The traits and types (pinned version).
cargo doc -p event-manager --no-deps --open

# 3. Firecracker's EventManager and run loop.
rg -n "EventManager|add_subscriber|\.run\(\)" src/vmm/src/

# 4. A real device subscriber: its init + process.
rg -n "impl MutEventSubscriber|fn init\b|fn process\b" src/vmm/src/devices/virtio/

# 5. The API wake-up eventfd as a subscriber.
rg -n "api_event_fd|process.*Vmm|drain|mpsc" src/vmm/src/ src/firecracker/src/

# 6. The ioeventfd registration (the fast-path link to KVM).
rg -n "register_ioevent|KVM_IOEVENTFD|ioeventfd|queue_evt" src/vmm/src/

Answer:

1. Why does the VMM thread use event-manager instead of a hand-rolled match on raw fd numbers? What does the manager own?
2. Name the six core types and what each is for. What is the difference between EventSubscriber and MutEventSubscriber, and which do Firecracker devices use?
3. Implement (in pseudocode from memory) a subscriber that watches one eventfd: what goes in init, what goes in process?
4. Why must process never block, and what is the system-wide consequence if it does?
5. List four kinds of subscriber on Firecracker's VMM loop and what each one's process does.
6. Explain the ioeventfd fast path: how a guest kick reaches a subscriber's process without a VM exit to userspace, and which crate provides each piece.

Common bugs and symptoms

Next: seccompiler — the crate (originally Firecracker's) that compiles JSON seccomp filters to BPF and applies them per-thread, locking down the exact threads whose loops you just learned.