The Event Manager

The VMM thread does not poll devices in a busy loop, and it does not spawn a thread per device. Instead it runs a single epoll-based event loop — the EventManager — that sleeps until a file descriptor becomes ready and then dispatches to whichever component registered interest in that fd. A virtio device's queue notification (an ioeventfd), the API thread's wake-up eventfd, a vCPU's signal-driven event, a TAP fd becoming readable — all of these are file descriptors, and all of them are multiplexed through one epoll instance on one thread. This is the heart of Firecracker's data-plane concurrency model.

This chapter covers the rust-vmm event-manager crate (EventManager, MutEventSubscriber, EventOps, EventSet), how the VMM thread registers subscribers (devices, the API eventfd, vCPU events), the run-loop dispatch, how a device's queue ioeventfd drives processing, and why this single-threaded, fd-driven design is the right fit for a minimal VMM.

Note: event-manager is an external rust-vmm crate (Firecracker donated it upstream), so the trait definitions live in the dependency, not in the vmm crate — but every subscriber that implements those traits lives in vmm. When you rg for MutEventSubscriber, the impls are in-tree; the trait is in ~/.cargo. See ../rust-vmm/event-manager.md.

The core types

# Where the EventManager is constructed and run in Firecracker.
rg -n "EventManager|MutEventSubscriber|SubscriberOps|EventOps|EventSet|add_subscriber|register" src/vmm/src/
# The trait you implement is in the external crate; find its impls in-tree:
rg -n "impl MutEventSubscriber" src/vmm/src/

Type	From	Role
`EventManager`	`event-manager` crate	Owns the `epoll` fd and the set of subscribers; the run loop.
`MutEventSubscriber`	`event-manager` crate	The trait a component implements to receive events. Two methods: `init` and `process`.
`EventOps`	`event-manager` crate	The handle a subscriber uses to add/modify/remove the fds it watches.
`EventSet`	`event-manager` (re-exports `vmm-sys-util`)	The interest flags: `IN` (readable), `OUT`, `ERROR`, `HANG_UP`.
`EventFd`	`vmm-sys-util`	The eventfd wrapper used for `ioeventfd`, the API wake-up, etc.

A subscriber implements MutEventSubscriber:

#![allow(unused)]
fn main() {
impl MutEventSubscriber for MyDevice {
    fn init(&mut self, ops: &mut EventOps) {
        // register the fds I care about, e.g. my queue's ioeventfd
        ops.add(Events::with_data(&self.queue_evt, MY_TOKEN, EventSet::IN)).unwrap();
    }
    fn process(&mut self, events: Events, ops: &mut EventOps) {
        // an fd I registered is ready — figure out which and do the work
    }
}
}

init is called once when the subscriber is added (it declares its fds); process is called every time one of those fds is ready.

Registration: who subscribes

rg -n "add_subscriber|register|Arc::new(Mutex::new|EventManager::new|fn run_event_loop|run\(\)" src/vmm/src/

During build_microvm_for_boot (and the snapshot-restore equivalent), the builder registers every event source as a subscriber with the EventManager:

Subscriber	fd it watches	What `process` does
each virtio device	its queue `ioeventfd`(s), plus backend fds (TAP, block file, vsock socket)	drain the virtqueue, do the I/O, update the used ring, raise the IRQ
the API connection	the API eventfd that the API thread signals after sending a `VmmAction`	read the action off the mpsc channel and execute it
the `Vmm` itself	exit/control events	orchestrate shutdown, etc.
rate limiters	a timer fd	re-enable a throttled queue when tokens refill

flowchart TD
    EM["EventManager (epoll fd)"]
    Block["virtio-block: queue ioeventfd"] --> EM
    Net["virtio-net: queue ioeventfd + TAP fd"] --> EM
    Vsock["virtio-vsock: queue ioeventfd + socket"] --> EM
    Api["API eventfd (signaled by API thread)"] --> EM
    RL["rate-limiter timerfd"] --> EM
    EM -->|"fd ready"| Dispatch["dispatch to that subscriber's process()"]

The Vmm is the natural owner; subscribers are typically held as Arc<Mutex<…>> so the EventManager and the rest of the VMM can both reach them.

The run loop

rg -n "fn run|event_manager.run|\.run\(\)|EVENT_LOOP|loop \{" src/vmm/src/
rg -n "run_with_api|run_without_api" src/firecracker/src/

The VMM thread's main body is, in essence, loop { event_manager.run() }. Each run():

calls epoll_wait and blocks until at least one registered fd is ready (or a timeout);
for each ready fd, looks up the owning subscriber and calls its process(events, ops);
returns; the outer loop calls run() again.

VMM thread:
  ┌───────────────────────────────────────────────┐
  │ loop {                                         │
  │   n = epoll_wait(...)        ← sleeps here     │
  │   for each ready fd:                           │
  │       subscriber.process(events, ops)          │
  │ }                                              │
  └───────────────────────────────────────────────┘

Because everything runs on one thread, a subscriber's process must not block — it does bounded work (drain a queue, do one batch of I/O) and returns, so other ready fds get serviced. This cooperative discipline is why Firecracker's device code is written to be non-blocking and to bound its per-call work.

The thread wiring (which thread runs the EventManager, how the API eventfd is connected) is in api_server_adapter::run_with_api() / run_without_api() in the firecracker binary, not in main.rs — see the-vmm-threading-model.md and api-server-and-action-channel.md.

How an ioeventfd drives a device

rg -n "ioeventfd|KVM_IOEVENTFD|register_ioevent|queue_evt|queue_notify|QueueNotify" src/vmm/src/

This is the connection between KVM and the event loop, and it is the whole reason the VMM thread can sleep instead of spin. When the guest "kicks" a virtqueue by writing the device's QueueNotify MMIO register, you do not want that to become a VM exit that the VMM has to handle synchronously. Instead Firecracker registers an ioeventfd with KVM (KVM_IOEVENTFD): KVM is told "when the guest writes to this MMIO address, just signal this eventfd and let the guest keep running." That same eventfd is registered with the EventManager. So:

sequenceDiagram
    participant Guest as Guest driver
    participant KVM
    participant EM as EventManager (VMM thread)
    participant Dev as Device.process()
    Guest->>KVM: write QueueNotify MMIO (kick)
    KVM->>KVM: matches registered ioeventfd
    KVM-->>EM: signal eventfd (no VM exit to VMM)
    Note over Guest,KVM: guest vCPU keeps running
    EM->>Dev: fd ready → process()
    Dev->>Dev: drain virtqueue, do I/O
    Dev->>KVM: KVM_IRQFD → inject completion IRQ

The guest's kick and the VMM's processing are decoupled: the vCPU thread doesn't stall waiting for the VMM, and the VMM thread sleeps until there's actual work. Completion is delivered back to the guest by raising an interrupt via KVM_IRQFD (another eventfd, in the other direction). This pair — ioeventfd in, irqfd out — is the virtio fast path, and the EventManager is what makes the inbound half efficient. See virtio-transport-mmio.md and interrupts-and-irqchip.md.

Why this design

A minimal VMM wants: low idle overhead (sleep, don't spin), a small attack surface (no thread-per- device sprawl), and predictable latency. One epoll loop on one thread delivers all three. Threads are expensive at the densities Firecracker targets (thousands of microVMs per host); an idle microVM with an epoll-sleeping VMM thread costs almost nothing. And single-threaded device emulation sidesteps a large class of data races inside the VMM. The cost — that one slow process can delay others — is acceptable precisely because the device model is tiny and every handler is written to be non-blocking.

Reading exercise

# 1. Find where the EventManager is created and run.
rg -n "EventManager::new|event_manager|\.run\(\)" src/vmm/src/
rg -n "run_with_api|run_without_api" src/firecracker/src/

# 2. Find the subscriber impls (the trait is external; impls are in-tree).
rg -n "impl MutEventSubscriber" src/vmm/src/

# 3. See how a device registers its queue eventfd in init().
rg -n "fn init|EventOps|EventSet::IN|add\(Events" src/vmm/src/devices/virtio/

# 4. Find the ioeventfd registration with KVM.
rg -n "ioeventfd|register_ioevent|KVM_IOEVENTFD" src/vmm/src/

# 5. Find the API eventfd that wakes the loop when an action arrives.
rg -n "eventfd|EventFd|api_event|to_vmm|from_api" src/vmm/src/ src/firecracker/src/

# 6. Trace one device end to end: pick block or net and read its process().
rg -n "fn process" src/vmm/src/devices/virtio/block/

Answer:

Name the four core event-manager types and which crate each comes from.
What two methods does MutEventSubscriber require, and when is each called?
List the subscribers the builder registers and the fd each watches.
Describe one iteration of the run loop. Why must a process implementation never block?
Explain the role of ioeventfd: what does it let KVM do instead of forcing a VM exit to the VMM?
Give three reasons a single-threaded epoll loop is the right model for a minimal, high-density VMM.

Common bugs and symptoms

Symptom	Root cause	Where to look
Device never processes guest requests	forgot to register its eventfd in `init`, or wrong `EventSet`	the device's `init`/`EventOps.add`
Guest kick has no effect; high latency	`ioeventfd` not registered with KVM; falling back to VM exits	`register_ioevent`/`KVM_IOEVENTFD`
Whole VMM stalls / one device starves others	a `process` blocks or loops unboundedly	the offending `process`; bound its work
API requests hang	API eventfd not registered, or the action channel not drained in `process`	API subscriber; mpsc receive
Throttled device never recovers	rate-limiter timerfd not re-registered after refill	rate limiter subscriber; rate-limiting-token-bucket.md
`process` called but does nothing	event token/fd mismatch — handler can't tell which fd fired	the token/`data` you passed to `Events::with_data`

Validation: prove you understand this

Explain the EventManager model in one paragraph: epoll, subscribers, init vs process.
List every event source registered on the VMM thread and what each one's process does.
Walk a single run-loop iteration and state the non-blocking contract on subscribers.
Draw the ioeventfd path from a guest kick to the device's process() and explain why the vCPU does not stall.
Explain how completion is delivered back to the guest (the other half: irqfd).
Defend the single-thread design against "why not a thread per device," citing density, attack surface, and races.

Next: logging-and-metrics.md — the observability surface the VMM thread maintains alongside the event loop, and how operators read it.

Open-Source Engineer & Contributor