Level 3: Architecture and the Threading Model

Levels 1 and 2 got you building Firecracker, running its tests, booting a microVM by hand, and opening a trivial pull request. You can drive the API socket and you can find your way around the src/ workspace. Level 3 is where you stop treating Firecracker as a black box that answers curl and start seeing the process behind the socket: a handful of threads, a single control-plane channel, an epoll loop, and one hard rule that explains almost every design decision you will meet for the rest of this curriculum.

That rule is: one Firecracker process is exactly one microVM. There is no pool, no scheduler, no multiplexing of guests inside a process. A serverless host runs thousands of microVMs by running thousands of firecracker processes, each jailed, each owning one guest. Inside one such process, the work is split across three classes of thread — the API thread, the VMM thread, and one thread per vCPU — and the boundary between them is the single most important architectural fact in the codebase. The control plane (HTTP configuration over the Unix socket) is deliberately kept off the data plane (the vCPUs running guest code through KVM_RUN). Get this boundary wrong in a patch and you will either block a vCPU on a lock or race the VMM state; get it right and you are writing code the way the maintainers do.

This level is mostly reading and tracing. You will follow a configuration request from the socket to the point where it mutates VmResources; you will enumerate the live threads of a running microVM and map each one to its spawn site in the source; and you will build a small read-only API action end-to-end. By the end you should be able to open the codebase cold and, for any API request, name the thread that parses it, the channel it travels on, the controller that dispatches it, and the struct it mutates — without a search engine and without a memorized line number.

Learning Objectives

By the end of Level 3 you must be able to:

1. Explain the three-thread-class model (API thread, VMM thread, one thread per vCPU) and why "one process = one microVM" is a load-bearing invariant, not an implementation detail.
2. Trace a configuration request — e.g. PUT /machine-config or PUT /drives/{id} — from the HTTP bytes on the UDS, through the API thread's parser, to the VmmAction that the VMM thread dispatches, and back to the HTTP response.
3. Describe the control-plane channel: how a ParsedRequest becomes a VmmAction, is boxed as an ApiRequest, is sent over an std::sync::mpsc channel, and how an eventfd wakes the VMM's epoll loop so it processes the request.
4. Distinguish PrebootApiController (before the microVM boots) from RuntimeApiController (after InstanceStart) and say which actions each one accepts.
5. Explain the EventManager epoll loop: what a MutEventSubscriber is, what subscribers register (the device fds, the API wake-up eventfd, the vCPU exit signals), and why the VMM thread is event-driven rather than a busy loop.
6. Locate and read VmResources, the aggregate pre-boot configuration, and the builder (build_microvm_for_boot / build_and_boot_microvm) that turns it into a running Vmm.
7. Name where the threads and channels are actually wired — run_with_api / run_without_api in the API-server adapter, not main.rs.
8. State, from memory, why the control plane is off the fast path and what would go wrong if it weren't.

The Three Thread Classes (and the One-Process Rule)

A running firecracker process — after you've sent InstanceStart for a 2-vCPU microVM — has threads in three roles. There is no fourth role.

The API thread is optional: under --no-api (the config-file boot path) there is no HTTP server at all, because the machine is configured from a JSON file at startup and then started immediately. The VMM thread always exists. The vCPU threads exist only after the guest is started — before InstanceStart there are no vCPUs running, only configuration accumulating in VmResources.

                         ONE firecracker PROCESS = ONE microVM
┌───────────────────────────────────────────────────────────────────────────────┐
│                                                                                 │
│   API thread                 VMM thread                       vCPU threads      │
│   (control plane)            (the Vmm + EventManager)          (the data plane)  │
│  ┌──────────────┐           ┌────────────────────────┐       ┌──────────────┐   │
│  │ micro-http   │  ApiReq   │  epoll_wait():         │       │ loop {        │   │
│  │  accept loop │ ────────► │   • API wake-up eventfd│       │  ioctl(KVM_RUN)│  │
│  │  on the UDS  │  (mpsc +  │   • device queue evfds │  ◄──► │  match exit { │   │
│  │              │  eventfd  │   • vcpu exit signals  │ Vcpu  │   IO  => ...  │   │
│  │ parse HTTP   │  wake)    │  dispatch VmmAction:   │ Event │   MMIO=> ...  │   │
│  │  → VmmAction │ ◄──────── │   Preboot|RuntimeApi   │ /Resp │  } }          │   │
│  └──────────────┘  ApiResp  │   Controller           │       └──────────────┘   │
│                             └────────────────────────┘         × vcpu_count     │
│                                                                                 │
│   seccomp-BPF filter category applied per thread: api | vmm | vcpu              │
└───────────────────────────────────────────────────────────────────────────────┘
                          │  ioctl() on /dev/kvm
                          ▼
                  KVM (Linux kernel module)

Note: The per-thread seccomp categories (api, vmm, vcpu) in resources/seccomp/<arch>.json are not a coincidence — they exist because the threads have sharply different jobs and therefore sharply different allowed syscall sets. The threading model and the security model are the same model viewed from two angles. You will see this again in Level 9.

The reason this split exists is performance and safety under multi-tenant load. The vCPU threads must run guest code with as little host interference as possible; the VMM thread must service device I/O the instant a queue is kicked; and neither must ever be stalled by a human (or an orchestrator) sending a configuration HTTP request. So configuration is handled by a separate thread that never touches the Vmm directly — it sends a message and waits. That message-passing boundary is the control-plane channel.

The Control-Plane Channel: API → VMM

The API thread and the VMM thread share no mutable state. They communicate over a pair of std::sync::mpsc channels plus an eventfd:

The flow, in one breath: the API thread parses HTTP into a ParsedRequest, converts it to a VmmAction (a giant enum — one variant per control operation), boxes it as an ApiRequest, sends it down the mpsc channel, then writes to the eventfd to wake the VMM's epoll loop, and blocks waiting for the ApiResponse. The VMM thread's epoll returns from epoll_wait because the eventfd is readable, drains the channel, dispatches the VmmAction through the active controller, produces a Result<VmmData, VmmActionError>, and sends it back. The API thread wakes, serializes the VmmData (or the error) into an HTTP response, and writes it to the socket.

sequenceDiagram
    participant C as curl (orchestrator)
    participant API as API thread<br/>(micro-http)
    participant CH as mpsc channel<br/>+ eventfd
    participant EM as VMM thread<br/>(EventManager epoll)
    participant CTL as Preboot/Runtime<br/>ApiController
    participant R as VmResources / Vmm

    C->>API: PUT /machine-config {vcpu_count, mem_size_mib}
    API->>API: parse HTTP → ParsedRequest → VmmAction
    API->>CH: send(Box<VmmAction>)  [ApiRequest]
    API->>CH: eventfd.write(1)   (wake the VMM)
    API-->>API: block on ApiResponse recv
    EM->>CH: epoll_wait returns: API eventfd readable
    EM->>CH: recv()  → Box<VmmAction>
    EM->>CTL: handle_request(VmmAction)
    CTL->>R: mutate VmResources (preboot)<br/>or query/act on Vmm (runtime)
    R-->>CTL: Ok(VmmData) | Err(VmmActionError)
    CTL-->>CH: send(Box<Result<VmmData, VmmActionError>>)  [ApiResponse]
    CH-->>API: recv() → ApiResponse
    API->>C: HTTP 204 / 200 + JSON, or 400 + error

Two controllers sit at the dispatch point, and which one is active depends on whether the guest has booted:

• PrebootApiController is active before StartMicroVm. It accepts the configuration actions (PutBootSource, InsertBlockDevice, SetVmConfiguration, InsertNetworkDevice, …), each of which mutates VmResources. It is also the controller that handles StartMicroVm itself — the action that runs the builder and transitions the process into the running state.
• RuntimeApiController is active after the microVM is running. It accepts the runtime actions (pause/resume, snapshot create, MMDS updates, metrics flush, balloon updates, drive/net rate-limiter patches, GET queries of live state). It can no longer accept most pre-boot config — the machine is already built.

Tip: When you read an error like The requested operation is not supported after starting the microVM, you are looking at the boundary between these two controllers. The VmmAction arrived, but the active controller refused it. Tracing which controller is live is half of debugging an API-state bug.

The deep mechanics of this channel — the exact VmmAction variant set, the boxing, the eventfd wake, the controller dispatch, and the back-compat rules for changing the enum — are the subject of the API server and action channel deep dive. Read it alongside Lab 3.1.

The VMM Thread is an Event Loop

The VMM thread does not poll. It blocks in epoll_wait inside the rust-vmm event-manager crate and wakes only when one of its registered file descriptors becomes readable. Each source of work registers itself as a MutEventSubscriber with an EventManager:

When epoll_wait returns, the EventManager calls process(...) on each ready subscriber, which does the actual work — copy a block request off a virtqueue and issue the host pread, drain the API channel and dispatch the action, etc. This is why the control plane is cheap: an idle microVM's VMM thread is asleep in epoll_wait, and an API request is just one more readable fd among the device fds. The configuration path and the I/O fast path share the same loop but are entirely separate events.

The architecture of this loop — subscriber registration, the init/process lifecycle, the ownership rules, and the gotchas (a subscriber that never re-arms its fd, a process that blocks the whole loop) — is covered in the Event Manager deep dive. The full threading rationale, including why the vCPU threads talk to the VMM thread over a separate VcpuEvent/VcpuResponse channel pair, is in the VMM threading model deep dive. Both are required reading for this level, not optional.

From Configuration to a Running microVM: VmResources and the Builder

Two structures anchor the pre-boot world:

• VmResources — the aggregate of everything you configured over the API before boot: the boot source, the block devices, the network interfaces, the MachineConfig (vCPUs, memory, SMT, CPU template, dirty-page tracking), vsock, balloon, MMDS config, entropy, logger/metrics. Every PrebootApiController action ends up mutating a field of VmResources. It is the single source of truth that the builder consumes.
• The builder (builder.rs) — the functions that turn VmResources into a live Vmm: build_microvm_for_boot (build from a fresh config), build_and_boot_microvm (build then start the vCPUs), and build_microvm_from_snapshot (restore). The builder is where guest memory is allocated and registered with KVM, the kernel is loaded, the device manager places the virtio-MMIO devices, the vCPUs are created and configured, and the EventManager subscribers are wired up.

   PrebootApiController                    builder.rs
   handles each PUT:                       build_and_boot_microvm(VmResources):
   ┌───────────────────┐    StartMicroVm   ┌──────────────────────────────────┐
   │ PutBootSource     │ ─────────────────►│ 1. create Vm + guest memory      │
   │ InsertBlockDevice │                   │ 2. KVM_SET_USER_MEMORY_REGION    │
   │ SetVmConfiguration│  mutate           │ 3. load kernel into guest mem    │
   │ InsertNetworkDev. │  fields of        │ 4. DeviceManager: place virtio   │
   │ ...               │  VmResources      │    MMIO devices + register fds   │
   └───────────────────┘                   │ 5. create + configure vCPUs      │
            │                              │ 6. spawn vCPU threads (KVM_RUN)  │
            ▼                              │ 7. hand Vmm to the EventManager  │
        VmResources  ─────────────────────►└──────────────────────────────────┘
                                                     │
                                                     ▼  now RuntimeApiController is active

Where are the threads and channels actually created? Not in main.rs. The wiring lives in the API-server adapter, in run_with_api() (the normal path) and run_without_api() (the --no-api config-file path). run_with_api spawns the API thread, creates the mpsc channels and the wake-up eventfd, builds the EventManager, and runs the VMM event loop on the main thread; run_without_api skips the API thread and feeds a parsed config straight into the builder. Find them with the rg commands below — the file name has drifted across refactors, so do not trust a path from memory.

Required Reading

Read these before and after the labs. Confirm each exists on your checkout first; if a path differs, the rg next to it relocates the content.

# One command to confirm the spine of this level is present on your branch:
rg -n "enum VmmAction|PrebootApiController|RuntimeApiController" src/vmm/src/rpc_interface.rs
rg -n "fn build_and_boot_microvm|fn build_microvm_for_boot" src/vmm/src/builder.rs
rg -rn "fn run_with_api|fn run_without_api" src/firecracker/src/

Source Code Areas to Inspect

You are reading, not modifying (until Lab 3.3). Skim these crates and modules; do not read top to bottom — rg for the specific type or function.

The control plane (the firecracker binary)

The action channel and resources (the vmm crate)

The VMM core and the event loop (the vmm crate)

# Confirm event-manager is an external dependency, not vendored:
rg -n "event-manager|event_manager" Cargo.toml src/vmm/Cargo.toml
# Find every MutEventSubscriber implementation — these are your subscribers:
rg -rn "impl MutEventSubscriber for" src/vmm/src/ | head -40

Key Types Quick Reference

Each type below comes with the rg that finds it on your branch. Run them; do not trust a remembered path.

GitHub Issue Categories for Level 3

A graduate of this level can credibly pick up issues in these areas. Find them with the label filters in Lab 3.1 and the issue roadmap:

• API validation and error messages — a PUT accepts a value it should reject, or returns a confusing VmmActionError. The fix lives in the parser or the preboot controller. (See issue-roadmap stage 4.)
• Preboot vs runtime state errors — an action that should be allowed (or rejected) at a given lifecycle stage isn't. The fix lives in the controller dispatch.
• Documentation of the API surface — the swagger (firecracker.yaml) drifting from the actual parser, or docs/design.md lagging the code.
• Small read-only additions — surfacing a piece of live state through a GET, exactly the shape of Lab 3.3.

Deliverables

Demonstrate all of the following before advancing to Level 4:

• A reading-log trace of PUT /machine-config (or PUT /drives/{id}) from the UDS bytes to the VmResources field it mutates, with a file path for every hop (Lab 3.1).
• Instrumentation you added (an eprintln!/tracing line) that proves the request crossed the mpsc channel and woke the VMM thread, with the captured output (Lab 3.1).
• A live enumeration of a running microVM's threads (ps -T / /proc/<pid>/task) showing the fc_api, fc_vmm, and fc_vcpu N thread names, each mapped to its spawn site in source (Lab 3.2).
• A one-paragraph explanation of why the control plane is off the fast path, naming the channel, the eventfd, and the EventManager (Lab 3.2).
• A working, curl-able read-only API action you added end-to-end — swagger entry, parser, a VmmAction, controller handling, and a VmmData response (Lab 3.3).
• From memory: name the two controllers and one action each accepts but the other rejects.

Common Mistakes

How to Verify Success

# 1. You can locate the action channel and both controllers by grep, not by memory:
rg -n "enum VmmAction|PrebootApiController|RuntimeApiController" src/vmm/src/rpc_interface.rs

# 2. You can find where the threads + channels are wired (NOT main.rs):
rg -rn "fn run_with_api|fn run_without_api" src/firecracker/src/

# 3. You can enumerate a live microVM's threads (start one first, then):
#    ps -T -p "$(pgrep -n firecracker)" -o spid,comm
#    cat /proc/"$(pgrep -n firecracker)"/status | grep Threads

# 4. You can name the builder entry point:
rg -n "fn build_and_boot_microvm|fn build_microvm_for_boot" src/vmm/src/builder.rs

When you can open the codebase cold and, for any API request, point at the thread that parses it, the channel it rides, the controller that dispatches it, and the field it mutates, you are ready for Level 3's labs — and for the run loop in Level 4.

PR Profile: Level 3 Graduate

A contributor who has genuinely completed this level can credibly open these classes of PR:

Avoid, for now: anything that adds a new device, touches the vCPU run loop, or changes the VmmAction/snapshot wire format — those need Levels 4, 7, and 9. Adding surface area to the API is a high-bar change even when you understand it; a read-only addition is the safe first step.

Next: Lab 3.1 — Trace an API Request from the Socket to the VMM.