The VMM Threading Model

A running Firecracker process is exactly one microVM. There is no "VM manager" multiplexing several guests inside one process — if you want ten microVMs, you run ten firecracker processes, each in its own jail. Inside that single process the work is split across three classes of thread: the API thread (the HTTP control plane on the Unix socket), the VMM thread (which owns the Vmm struct and runs the epoll event loop driving all device emulation), and one thread per vCPU (each spinning in its own KVM_RUN loop). This chapter shows you where each thread is spawned, what it owns, how the API thread hands work to the VMM thread over an mpsc channel plus a wake-up eventfd, and — most importantly — why the control plane is deliberately kept off the data plane's fast path.

After this chapter you will be able to: name the three thread classes and what each owns; explain why a slow API request can never stall a vCPU; locate every thread-spawn site on your checkout; and trace a control message from the socket to the Vmm struct and back.

Note: "one process = one microVM" is the single most clarifying fact about Firecracker's design. It is why the threading model is small, why there is no global lock contention between guests, and why the security story (jailer + seccomp) can be per-process. Hold it in your head through every other chapter.

The three thread classes

# Find where threads are spawned. Do not trust a count — run this and read the call sites.
rg -n "thread::Builder|\.spawn\(|spawn_named|name\(" src/vmm/src/ src/firecracker/src/
# The API thread + channel wiring lives in the binary, not in vmm:
rg -n "fn run_with_api|fn run_without_api|ApiServer|api_thread" src/firecracker/src/

Thread	Spawned where	Owns	In the fast path?
API thread	`firecracker` binary, in the `api_server_adapter` (`run_with_api`)	The `ApiServer`, the HTTP parsing, the Unix-socket listener	No — never
VMM thread	The process's main thread becomes the VMM thread after setup	The `Vmm` struct, the `EventManager` epoll loop, all device emulation, MMDS, rate limiters	Yes — services device I/O
vCPU thread × N	One per configured vCPU, spawned during build/boot	A `Vcpu`/`KvmVcpu`, its `VcpuFd`, the `KVM_RUN` loop	Yes — runs guest code

Under --no-api (config-file boot), the API thread does not exist at all — run_without_api parses the config file, builds the microVM, and runs the same VMM loop. That is your proof that the API thread is pure control plane: the guest runs fine without it.

                          one firecracker process (one microVM)
   ┌──────────────────────────────────────────────────────────────────────┐
   │  API thread            VMM thread                vCPU thread × N       │
   │  ┌──────────┐          ┌──────────────┐          ┌──────────────┐      │
   │  │ HTTP on  │  mpsc    │  Vmm struct  │  Vcpu    │ KVM_RUN loop │      │
   │  │ UDS      │ ───────► │  EventManager│  Event   │ (one vCPU)   │      │
   │  │ micro-   │ ◄─────── │  epoll loop  │ ◄──────► │ handles PIO/ │      │
   │  │ http     │  mpsc    │  devices     │  Vcpu    │ MMIO exits   │      │
   │  └──────────┘ +eventfd │  MMDS, rate  │  Response└──────────────┘      │
   │                        │  limiters    │          ┌──────────────┐      │
   │                        └──────────────┘   ...    │ vCPU thread 2│      │
   │                                                  └──────────────┘      │
   └──────────────────────────────────────────────────────────────────────┘

What each thread owns

The ownership boundaries are not advisory — they are how the code stays free of locks on the hot path.

The VMM thread owns the Vmm struct. Find it:

rg -n "pub struct Vmm|impl Vmm" src/vmm/src/lib.rs
rg -n "struct EventManager|EventManager::new|\.run\(" src/vmm/src/

The Vmm struct holds the VM file descriptor, the device managers, guest memory handles, and the vCPU control handles. The VMM thread sits in the EventManager epoll loop (the rust-vmm event-manager crate — see the-event-manager.md): it blocks on epoll_wait until a device's eventfd, a timer, or the API wake-up eventfd becomes ready, then dispatches to the matching Subscriber. Device emulation — pulling a request off a virtqueue, doing the host pread/write, updating the used ring, injecting the completion interrupt — all happens on this one thread.

Each vCPU thread owns its Vcpu. Find the entry point:

rg -n "fn run\b|fn start_threads|fn run_emulation|KVM_RUN|VcpuExit" src/vmm/src/vstate/vcpu/

A vCPU thread's life is the KVM_RUN loop (vcpu-run-loop-and-vm-exits.md): ioctl(vcpufd, KVM_RUN) blocks while the guest executes on the physical CPU; when the guest does something the VMM must handle (a PIO/MMIO access, a halt), KVM returns and the thread reads the exit reason and dispatches it — synchronously, on the vCPU thread — to the PIO or MMIO bus. The bus may end in a device whose backing work the VMM thread will later complete. This is the data plane.

The API thread owns the HTTP server. It parses bytes off the socket into a ParsedRequest, turns that into a VmmAction, and hands it off — it never touches guest memory, never touches a VcpuFd, never blocks the VMM loop for longer than it takes to enqueue a message.

The action channel: mpsc + a wake-up eventfd

The API thread and the VMM thread are connected by a pair of std::sync::mpsc channels and one eventfd. The pattern is the heart of the control plane.

# Locate the action types and the channel/eventfd wiring.
rg -n "type ApiRequest|type ApiResponse|VmmAction|VmmData|VmmActionError" src/vmm/src/rpc_interface.rs
rg -n "mpsc::channel|EventFd|api_event_fd|from_api|to_api" src/firecracker/src/ src/vmm/src/

Direction	Carries	Type (verify on your branch)
API → VMM	A control command	`ApiRequest = Box<VmmAction>`, sent over an mpsc `Sender`
API → VMM	"wake up, there's a message"	an `EventFd` registered in the VMM's epoll set
VMM → API	The result	`ApiResponse = Box<Result<VmmData, VmmActionError>>` over an mpsc `Sender`

The mpsc channel is asynchronous, so the VMM thread is not constantly polling it. To get the VMM thread to notice a new action without busy-waiting, the API thread also writes to a dedicated eventfd that is registered in the VMM's epoll set. The next epoll_wait returns, the VMM thread drains the mpsc channel, dispatches each VmmAction, and sends the ApiResponse back. The API thread blocks on the response channel and then writes the HTTP reply.

sequenceDiagram
    participant C as curl (orchestrator)
    participant A as API thread
    participant E as eventfd
    participant V as VMM thread (epoll)
    C->>A: PUT /drives/rootfs  (HTTP over UDS)
    A->>A: ParsedRequest -> VmmAction::InsertBlockDevice
    A->>V: mpsc send Box<VmmAction>
    A->>E: write(1)  (wake the loop)
    V->>V: epoll_wait returns; drain mpsc
    V->>V: dispatch via Preboot/Runtime controller
    V->>A: mpsc send Box<Result<VmmData, VmmActionError>>
    A->>C: HTTP 204 / 400 with body

Dispatch is split by lifecycle phase. Before the microVM starts, actions go through the PrebootApiController (it can mutate VmResources: add drives, set machine config, then handle StartMicroVm). After boot, actions go through the RuntimeApiController (pause/resume, snapshot, flush metrics, ctrl-alt-del — the runtime-legal subset). See api-server-and-action-channel.md for the full request→action→response path and the swagger contract.

rg -n "PrebootApiController|RuntimeApiController|build_microvm|StartMicroVm" src/vmm/src/rpc_interface.rs

The vCPU threads have their own separate control channels — VcpuEvent/VcpuResponse — that the VMM thread uses to pause and resume individual vCPUs (for snapshotting, for clean shutdown). Locate them:

rg -n "VcpuEvent|VcpuResponse|Pause|Resume|enum VcpuEvent" src/vmm/src/vstate/vcpu/

Why the control plane is off the fast path

This is the design decision the whole model exists to enforce. A vCPU thread must be able to run guest code at near-native speed; the only reason it should ever stop is a VM exit it must service. If a slow or malicious API request — a huge JSON body, a client that stalls mid-request — could block a vCPU, the guest's performance would be hostage to the control plane.

So the split is strict: the API thread does control, the data threads do data, and they communicate only by message-passing. Consequences you can reason about:

A blocked or slow HTTP client cannot stall a vCPU or the device loop. The worst case is that the next control action is delayed.
The VMM thread services the API eventfd as just one more epoll source among the device eventfds, so control work is interleaved fairly with device work and never starves it.
Most runtime API actions are cheap (pause, flush metrics). The expensive one — snapshot create — deliberately pauses the vCPUs first, so there is no contention to fight.
Under --no-api there is no control plane thread at all, and nothing about the guest changes. That is the cleanest proof that the control plane is genuinely off the fast path.

Warning: Do not "optimize" by having the API thread reach directly into the Vmm struct or a VcpuFd. The whole safety and performance story depends on the API thread owning nothing shared with the data plane except the channels. Cross-thread access to KVM fds or guest memory from the API thread is a bug class, not a shortcut.

Reading exercise

# 1. Every thread spawn site in the codebase.
rg -n "thread::Builder|\.spawn\(|spawn_named" src/vmm/src/ src/firecracker/src/

# 2. The with-API vs no-API entry points and the channel wiring.
rg -n "fn run_with_api|fn run_without_api|mpsc::channel|api_event_fd" src/firecracker/src/

# 3. The action channel types.
rg -n "type ApiRequest|type ApiResponse|enum VmmAction|VmmData|VmmActionError" src/vmm/src/rpc_interface.rs

# 4. The two API controllers and the boot boundary.
rg -n "PrebootApiController|RuntimeApiController|StartMicroVm|build_microvm" src/vmm/src/rpc_interface.rs

# 5. The vCPU thread entry and its control channel.
rg -n "fn run\b|VcpuEvent|VcpuResponse|KVM_RUN" src/vmm/src/vstate/vcpu/

# 6. Watch the threads on a running microVM (boot one first, then):
ps -L -p "$(pgrep -n firecracker)" -o tid,comm

Answer:

Name the three thread classes and exactly what each one owns. Which two are on the data fast path?
Walk a PUT /machine-config from the socket to the Vmm and back: which thread does each step, and where does the eventfd come in?
Why is there an eventfd in addition to the mpsc channel? What would break if you removed it?
What is the difference between the PrebootApiController and the RuntimeApiController, and what event flips you from one to the other?
Under --no-api, which thread is missing, and what does that prove about the control plane?
The vCPU control channel (VcpuEvent/VcpuResponse) is separate from the API action channel. Why not reuse one channel for both?

Common bugs and symptoms

Symptom	Root cause	Where to look
API call hangs forever	VMM thread blocked in a device handler and never returns to `epoll_wait`; response never sent	device `Subscriber::process`; the VMM `EventManager` loop
`InstanceStart` returns OK but guest never runs	vCPU threads spawned but never sent the "start" signal, or KVM setup failed silently	vCPU spawn + `VcpuEvent` start; `build_microvm_for_boot`
Snapshot create hangs	vCPUs not actually paused before state is read; pause `VcpuResponse` never received	pause path; `VcpuEvent::Pause`/`VcpuResponse`
Control action applied at the wrong time (e.g. add drive after boot)	Dispatched through the wrong controller; preboot-only action reached `RuntimeApiController`	`Preboot`/`Runtime` dispatch in `rpc_interface.rs`
API thread reads stale/garbage VM state	API thread reached into shared state instead of going through the channel	any direct `Vmm`/`VcpuFd` access from the API thread
High guest latency correlated with API traffic	Control work accidentally placed on a vCPU or blocking the VMM loop	confirm the work is on the API thread / off the epoll loop

Validation: prove you understand this

Draw the three-thread diagram from memory and label what each thread owns and which are in the fast path.
Explain the full lifecycle of one control action: the two channels, the eventfd, the two controllers, and which thread runs each stage.
Explain in one paragraph why a slow HTTP client cannot stall a vCPU. What is the worst thing it can do?
Why does --no-api prove the control plane is off the fast path? What changes about the guest when the API thread is absent?
Why do the vCPU threads have a separate control channel from the API action channel? Give one concrete operation that uses it.
A maintainer rejects a PR that has the API thread call directly into the Vmm struct "to avoid the channel overhead." Defend that rejection on both safety and performance grounds.

Next: The API Server and the Action Channel — the full request→VmmAction→response path and the micro-http server that feeds it.

Open-Source Engineer & Contributor