Lab 3.2: The Threading Model and the EventManager

In Lab 3.1 you traced one request across the API/VMM thread boundary. This lab makes the threading model physical: you will start a real microVM, enumerate its operating-system threads with ps -T and /proc/<pid>/task, read the thread names the OS shows you (fc_api, fc_vmm, fc_vcpu N), and map each one back to the line of Rust that spawned it. Then you will read the heart of the VMM thread — the EventManager epoll loop from the rust-vmm event-manager crate — and enumerate what subscribes to it: the device file descriptors, the API wake-up eventfd, and the vCPU event channels. Finally you will articulate, precisely, why the control plane is kept off the data plane.

This is a trace-it lab with a strong "observe the live process" component. There is no production code to write, but you will run commands against a running microVM and correlate the kernel's view of the process with the source.

Background

A Firecracker process implements exactly three thread classes (the Level 3 overview introduces them, and the VMM threading model deep dive covers them in depth):

Three things make this model what it is:

1. One process is one microVM. Threads are not shared across guests; there is no in-process multiplexing. A host runs thousands of guests by running thousands of processes.
2. The VMM thread is event-driven, not busy. It sleeps in epoll_wait until an fd it registered becomes readable. Every source of work — a guest kicking a virtqueue, the API thread sending an action, a periodic metrics flush — is a readable fd handled by a MutEventSubscriber.
3. The control plane is off the fast path. Configuration requests ride a separate channel + an eventfd; they are one more readable fd in the same epoll loop, never a lock on a vCPU and never a blocking call in device emulation.

Note: The thread comm names (fc_api, fc_vmm, fc_vcpu N) are set explicitly in the source with a set_name/prctl(PR_SET_NAME)-style call. Linux truncates comm to 15 characters, so long names may appear clipped in ps/top. Verify the exact strings on your branch with the rg in Step 4 — they have been tidied across releases.

Why This Lab Matters for Contributors

A maintainer reasons about Firecracker bugs in terms of which thread is misbehaving. "The vCPU is spinning at 100%" points at the KVM_RUN loop; "the VMM stalls under load" points at a blocking process() in the EventManager; "configuration hangs" points at the API channel and the eventfd. To debug a hang or a deadlock you must be able to attach to a specific thread, know what loop it is supposed to be in, and recognize when it is stuck somewhere it shouldn't be. The EventManager loop in particular is a place where a single mistake — a subscriber that blocks, or one that forgets to re-arm its fd — degrades every device on the microVM. You cannot review a device PR or diagnose a stall without this model in your hands.

Prerequisites

• Firecracker builds: tools/devtool build (Level 1).
• You can boot a microVM by hand with a kernel + rootfs (see Level 1, Lab 1.3). You need a real running guest for the thread-enumeration half.
• You completed Lab 3.1 (you know the API→VMM channel).
• You have read the Event Manager deep dive intro.
• Linux tools: ps, top, and /proc (you are on the Linux host running Firecracker, not inside the guest).

mkdir -p ~/firecracker-notes
: > ~/firecracker-notes/threading-3.2.md

Note: Run these commands on the host, against the firecracker process — not inside the guest. If you jailer-launch in production the process lives in its own PID namespace; for this lab launch firecracker directly so the host ps//proc sees it plainly.

Part A — Enumerate the Threads of a Live microVM (budget: 40 min)

Step 1 (8 min) — Boot a 2-vCPU microVM

Use two vCPUs so you can see two vCPU threads. Adjust the kernel/rootfs paths to your Lab 1.3 artifacts.

ARCH=$(uname -m)
BIN=build/cargo_target/${ARCH}-unknown-linux-musl/debug/firecracker
API=/tmp/fc-3.2.socket
rm -f "$API"

sudo "$BIN" --api-sock "$API" &
sleep 0.3

curl -sS -X PUT --unix-socket "$API" \
  --data '{"kernel_image_path":"./vmlinux-6.1.x","boot_args":"console=ttyS0 reboot=k panic=1 pci=off"}' \
  http://localhost/boot-source
curl -sS -X PUT --unix-socket "$API" \
  --data '{"drive_id":"rootfs","path_on_host":"./ubuntu-24.04.ext4","is_root_device":true,"is_read_only":false}' \
  http://localhost/drives/rootfs
# TWO vCPUs — this is the point:
curl -sS -X PUT --unix-socket "$API" \
  --data '{"vcpu_count":2,"mem_size_mib":512}' http://localhost/machine-config

# Before starting: how many threads now? (API + VMM, no vCPUs yet.)
FC_PID=$(pgrep -n firecracker)
echo "PID=$FC_PID"
cat /proc/$FC_PID/status | grep '^Threads:'

Note the thread count before InstanceStart: you should see roughly 2 (API + VMM). The vCPU threads do not exist yet — they are spawned by the builder at start. Now start the guest:

curl -sS -X PUT --unix-socket "$API" \
  --data '{"action_type":"InstanceStart"}' http://localhost/actions
sleep 0.5
cat /proc/$FC_PID/status | grep '^Threads:'

The count should jump by vcpu_count (here +2, to ~4). You have just watched the vCPU threads come into existence at boot.

Step 2 (8 min) — Name every thread

Three views of the same fact. Run all three and compare.

FC_PID=$(pgrep -n firecracker)

# (a) ps -T: one row per thread, with the comm name:
ps -T -p "$FC_PID" -o spid,tid,comm

# (b) top in thread mode (-H), one-shot batch:
top -H -b -n1 -p "$FC_PID" | sed -n '7,30p'

# (c) the raw kernel view: one directory per thread, comm in each:
for t in /proc/$FC_PID/task/*; do
  printf '%s\t%s\n' "$(basename "$t")" "$(cat "$t/comm")"
done

Expected (names verify on your branch; ids differ):

  SPID    TID COMMAND
 40121  40121 firecracker        ← could be the main/VMM thread
 40122  40122 fc_api             ← the API thread
 40123  40123 fc_vmm             ← the VMM/EventManager thread
 40124  40124 fc_vcpu 0          ← vCPU 0 (KVM_RUN loop)
 40125  40125 fc_vcpu 1          ← vCPU 1

Tip: comm is capped at 15 bytes by the kernel. If you see a truncated name, that's why. The source string may be longer; ps/top show the clipped version. Cross-check against the rg in Step 4 rather than the truncated string.

Log it:

cat >> ~/firecracker-notes/threading-3.2.md <<'EOF'
## Live threads (vcpu_count=2)
Before InstanceStart: ~2 threads (API + VMM, no vCPUs)
After  InstanceStart: ~4 threads (+2 vCPUs)
TID    comm        role
<..>   fc_api      API thread, micro-http accept loop
<..>   fc_vmm      VMM thread, EventManager epoll loop
<..>   fc_vcpu 0   vCPU 0, KVM_RUN loop
<..>   fc_vcpu 1   vCPU 1, KVM_RUN loop
EOF

Step 3 (8 min) — Observe what each thread is doing

A thread's state tells you which loop it's in. Sample the stack/state of each.

FC_PID=$(pgrep -n firecracker)

# Per-thread scheduler state (S = sleeping/blocked, R = running):
ps -T -p "$FC_PID" -o tid,comm,stat

# What syscall is each thread parked in right now? (root may be required)
for t in /proc/$FC_PID/task/*; do
  tid=$(basename "$t")
  printf '%s %-12s %s\n' "$tid" "$(cat "$t/comm")" "$(cat "$t/wchan" 2>/dev/null)"
done

Read the result. On an idle guest:

• fc_api is blocked in an accept/epoll (waiting for the next HTTP request).
• fc_vmm is blocked in epoll_wait (the EventManager loop, waiting for a device/API event).
• each fc_vcpu N is blocked inside the KVM_RUN ioctl — the guest is idle (it hlt-ed), so the vCPU is parked in the kernel until an interrupt.

This is the threading model at rest: three loops, all asleep, each waiting on its own kind of event. Generate some I/O in the guest (e.g. dd on its console) and re-sample — you'll see fc_vmm and a vCPU flicker to R as virtqueue kicks wake the EventManager.

Step 4 (8 min) — Map each thread to its spawn site in source

Now correlate the kernel's view with the code. Find where each thread is named and spawned.

# Where the thread comm names are set:
rg -rn "fc_vcpu|fc_vmm|fc_api|set_name|PR_SET_NAME|prctl" src/vmm/src/ src/firecracker/src/ | head

# The vCPU thread spawn site:
rg -rn "thread::Builder|spawn\(|fn start_threads|fn start_vcpus|Builder::new\(\).name" src/vmm/src/ | rg -i "vcpu|builder" | head

# The API thread spawn site (in the adapter, not main.rs):
rg -rn "thread::Builder|spawn\(|run_with_api|ApiServer" src/firecracker/src/ | head

# The vCPU run loop itself (the KVM_RUN body):
rg -rn "fn run|KVM_RUN|VcpuExit|fn run_emulation" src/vmm/src/vstate/vcpu/ | head

What you should find and confirm:

1. The vCPU threads are spawned in the builder (builder.rs) when the microVM starts — one thread::Builder::new().name("fc_vcpu N")...spawn(...) per vCPU, each entering the vCPU's run loop. This is why the thread count jumps at InstanceStart.
2. The API thread is spawned in run_with_api (the adapter), running the micro-http server.
3. The VMM thread is the main thread that runs the EventManager loop after the builder hands it the Vmm (or a dedicated thread — verify which on your branch).

Log it:

cat >> ~/firecracker-notes/threading-3.2.md <<'EOF'
## Spawn sites
fc_vcpu N : src/vmm/src/builder.rs  (start_vcpus / start_threads) -> Vcpu::run() -> KVM_RUN loop in src/vmm/src/vstate/vcpu/
fc_api    : src/firecracker/src/ run_with_api -> micro-http server
fc_vmm    : the EventManager loop (main/VMM thread) after the builder returns the Vmm
EOF

Part B — The EventManager Epoll Loop (budget: 35 min)

Step 5 (8 min) — Find the loop and confirm it is external

The EventManager is the rust-vmm event-manager crate — an external dependency, not vendored code.

# Confirm it is an external dependency:
rg -n "event-manager" Cargo.toml src/vmm/Cargo.toml
# Find where Firecracker drives the loop (the run() call on the VMM thread):
rg -rn "EventManager|event_manager|\.run\(|epoll" src/vmm/src/ src/firecracker/src/ | rg -i "event_manager|EventManager|\.run\(" | head
# (optional) read the crate source the build pulled in:
find ~/.cargo -path '*event-manager*/src/lib.rs' 2>/dev/null | head

The shape is: the VMM thread calls event_manager.run() (or run_with_timeout(...)) in a loop. Each call does one epoll_wait, then dispatches process(...) to every subscriber whose fd is ready. Read the crate's run/dispatch if you found it — it is small and worth 5 minutes.

Step 6 (10 min) — Enumerate the subscribers

A MutEventSubscriber is anything that registers fds with the EventManager and gets process-ed when they fire. Enumerate every implementor in Firecracker.

# Every subscriber type:
rg -rn "impl MutEventSubscriber for" src/vmm/src/ src/firecracker/src/

# What each subscriber registers (look for the init/register/add calls):
rg -rn "fn init\b|register\(|add\(|Events::new|EventSet::" src/vmm/src/devices/ | head -30

You should find subscribers in these roles (the table — confirm the exact types on your branch):

The crucial observation: the API eventfd is just one subscriber among the device fds. The same epoll loop that services a virtio-net packet also services a PUT /drives/{id}. That is the mechanical realization of "the control plane shares the loop but not the fast path" — they are different events on one epoll_wait.

Log it:

cat >> ~/firecracker-notes/threading-3.2.md <<'EOF'
## EventManager subscribers (MutEventSubscriber impls)
- API eventfd      -> a VmmAction is waiting on the channel
- virtio-block     -> queue eventfd / io completion
- virtio-net       -> TAP fd + queue eventfds
- virtio-vsock     -> host unix socket + queue eventfds
- serial console   -> input fd
- metrics          -> timer fd
The API eventfd is ONE subscriber among the device fds: same loop, separate events.
EOF

Step 7 (8 min) — The vCPU↔VMM channel is separate

The vCPU threads do not go through the EventManager for control. They talk to the VMM thread over a separate pair of channels.

rg -rn "VcpuEvent|VcpuResponse|VcpuHandle|enum VcpuEmulation" src/vmm/src/vstate/vcpu/ | head
rg -rn "Pause|Resume|SaveState|RestoreState|Finish" src/vmm/src/vstate/vcpu/ | rg -i "VcpuEvent" | head

Confirm: there is a VcpuEvent / VcpuResponse channel pair (Pause, Resume, SaveState, …) distinct from the ApiRequest/ApiResponse channel. When you PATCH /vm {"state":"Paused"}, the API action reaches the VMM thread over the API channel, and the VMM thread then sends a VcpuEvent::Pause to each vCPU over the vCPU channel. Two channels, two purposes: the API channel is human/orchestrator control; the vCPU channel is the VMM thread commanding its own vCPUs.

   API thread ──ApiRequest/Response──► VMM thread ──VcpuEvent/Response──► vCPU threads
   (control plane)                     (EventManager)                    (KVM_RUN loops)

Step 8 (9 min) — Articulate why the control plane is off the fast path

This is the conceptual deliverable. Write it in your own words; here is the argument to converge on.

The vCPU threads run guest code with minimal host interference — a vCPU spends its life in the KVM_RUN ioctl and should return to it as fast as possible. The VMM thread services device I/O the instant a queue is kicked. If configuration were handled on these threads — say, if a vCPU took a lock on the Vmm to apply a config change — then a slow or malicious control request could stall a vCPU mid-guest-execution, or block device emulation for every device, harming the data plane of a tenant who isn't even the one sending the request. By routing all control through a separate API thread that owns nothing but a channel, and by making the VMM thread process that channel as just another epoll event (cheap, non-blocking, microsecond-scale — you measured this in Lab 3.1's stretch goal), Firecracker guarantees that configuration latency and the data-plane fast path do not interact. The threading model is, at bottom, a latency-isolation argument — and it is the same argument as the per-thread seccomp categories: different jobs, different trust, hard boundaries.

Log it: write a one-paragraph version naming the API channel, the eventfd, the EventManager loop, and the vCPU KVM_RUN loop.

Step 9 — Clean up

sudo pkill -f "fc-3.2.socket" 2>/dev/null
rm -f /tmp/fc-3.2.socket

The Model at a Glance

flowchart TD
    subgraph proc[ONE firecracker process = ONE microVM]
      API["fc_api thread<br/>micro-http accept loop"]
      VMM["fc_vmm thread<br/>EventManager.run(): epoll_wait"]
      V0["fc_vcpu 0<br/>KVM_RUN loop"]
      V1["fc_vcpu 1<br/>KVM_RUN loop"]
    end
    API -->|ApiRequest + eventfd wake| VMM
    VMM -->|ApiResponse| API
    VMM -->|VcpuEvent: Pause/Resume| V0
    VMM -->|VcpuEvent: Pause/Resume| V1
    V0 -->|VcpuResponse / exit signal fd| VMM
    V1 -->|VcpuResponse / exit signal fd| VMM
    DEV[device fds: block/net/vsock/serial/timer] -->|readable| VMM
    KVM[(/dev/kvm)]
    V0 -->|ioctl KVM_RUN| KVM
    V1 -->|ioctl KVM_RUN| KVM

Implementation Requirements / Deliverables

• A ~/firecracker-notes/threading-3.2.md listing every live thread of a 2-vCPU microVM by TID and comm, the thread count before vs after InstanceStart, and each thread's spawn site in source.
• An enumeration of the EventManager subscribers (MutEventSubscriber impls) and what fd each registers.
• A note confirming the vCPU↔VMM channel (VcpuEvent/VcpuResponse) is separate from the API channel.
• A one-paragraph, in-your-own-words explanation of why the control plane is off the fast path, naming the channel, the eventfd, the EventManager loop, and the KVM_RUN loop.

Troubleshooting

ps -T shows only one thread

You sampled before InstanceStart, or the process forked/exec'd and you grabbed the wrong PID. Use pgrep -n firecracker to get the newest PID, and sample after you've sent InstanceStart.

Thread names are all firecracker (not fc_vcpu/fc_api)

Either your branch predates the explicit naming, or comm truncation is hiding them. Confirm with the rg for set_name/PR_SET_NAME in Step 4 — if the source sets names but ps doesn't show them, read /proc/<pid>/task/<tid>/comm directly, which is authoritative.

/proc/<pid>/task/*/wchan is empty or 0

Kernel hardening (kernel.yama / restricted /proc) can blank wchan for non-root. Run the loop under sudo, or fall back to ps -T -o tid,comm,stat (the stat column still shows S/R/D states).

The vCPU threads show as running (R) at 100% on an idle guest

That can be a real bug (a busy KVM_RUN loop), or your guest is genuinely busy. Confirm by checking inside the guest (top on the serial console). A correctly idle guest hlts and its vCPU parks in the KVM_RUN ioctl in state S/D. A spinning vCPU on a truly idle guest is exactly the kind of edge case Level 4, Lab 4.4 hunts.

I can't find impl MutEventSubscriber

The trait may be re-exported under a different path after the crate merge. Broaden the search: rg -rn "MutEventSubscriber|EventSubscriber|Subscriber" src/vmm/src/ | head. The device subscribers are under src/vmm/src/devices/.

Expected Output

$ ps -T -p $(pgrep -n firecracker) -o tid,comm,stat
   TID COMMAND         STAT
 40121 firecracker     Sl
 40122 fc_api          Sl
 40123 fc_vmm          Sl
 40124 fc_vcpu 0       Sl
 40125 fc_vcpu 1       Sl

(All sleeping on an idle guest: API in accept, VMM in epoll_wait, vCPUs in KVM_RUN.)

Stretch Goals

1. Watch a vCPU exit live. strace -f -e trace=ioctl -p $(pgrep -n firecracker) 2>&1 | grep KVM_RUN | head while you generate guest I/O. You'll see the KVM_RUN ioctls returning and being re-entered — the run loop you'll read in Level 4.
2. Count subscribers as devices are added. Boot with one drive, then with a drive + a net iface + a vsock, and compare how many fds the VMM thread's epoll watches (ls /proc/<pid>/task/<vmm-tid>/fd | wc -l is a crude proxy). More devices = more subscribers = more readable fds in the same loop.
3. Pause and re-enumerate. PATCH /vm {"state":"Paused"}, then re-sample the vCPU threads. They should leave the KVM_RUN loop and park waiting on a VcpuEvent::Resume. Find the parked state in /proc and correlate it with the pause handling in vstate/vcpu/.
4. Find the blocking-in-process() foot-gun. Read one device's process implementation and reason about what would happen if it did a synchronous, slow host call there. Which other devices would be starved? (This is a real review concern for device PRs — see Level 7.)

Validation / Self-check

Answer without looking back:

1. A microVM is configured with vcpu_count: 4 but not yet started. How many threads does the process have, and how many after InstanceStart? Why the difference?
2. Name the loop each thread class runs and the syscall each is typically blocked in on an idle guest.
3. The API wake-up eventfd is "just one subscriber among the device fds." Explain, in those terms, what it means to say the control plane shares the loop but is off the fast path.
4. When you PATCH /vm {"state":"Paused"}, two different channels are involved before the guest is actually paused. Name both and the order they're used.
5. What is a MutEventSubscriber, and what two things must one do (besides existing) to participate in the EventManager loop?
6. Why would a slow, blocking call inside one device's process() harm unrelated devices on the same microVM? Which thread is at risk?
7. Where in source is a vCPU thread spawned, and what loop does it enter? Where is the API thread spawned, and why is it not in main.rs?

When you can answer all seven and your threading log maps every live thread to its source, you've completed Lab 3.2. Continue to Lab 3.3: Build It — A Custom API Action.