Lab 3.1: Trace an API Request from the Socket to the VMM

This is a trace-it lab. You will not write production code, but you will add temporary instrumentation, rebuild Firecracker, and watch a real request cross the thread boundary. You pick one concrete configuration request — the canonical choice is PUT /machine-config, with PUT /drives/{id} as the alternative — and follow it through every hop: the HTTP bytes arriving on the Unix domain socket, the API thread parsing them into a ParsedRequest, the conversion to a VmmAction, the boxed ApiRequest going down the mpsc channel, the eventfd waking the VMM thread's EventManager epoll loop, the PrebootApiController dispatching the action, the mutation of VmResources, and the ApiResponse coming back to become an HTTP response.

The deliverable is a reading-log artifact with a file path for every hop, plus captured output from instrumentation that proves the message crossed the channel. You will reuse this skill in every issue lab and in the capstone. The discipline is the one this whole curriculum drills: navigate and instrument; never memorize line numbers.

Background

A Firecracker process splits work across three thread classes (see the Level 3 overview). The control plane — HTTP configuration over the UDS — runs on the API thread and is deliberately isolated from the VMM thread (which owns the Vmm and runs the EventManager epoll loop) and the vCPU threads. The two control-plane threads share no mutable state; they pass messages:

HTTP on the UDS → micro-http accept loop (API thread)
   → ParsedRequest → VmmAction → Box<VmmAction> (ApiRequest)
   → mpsc send + eventfd.write(1)  ───────────────►  VMM thread epoll_wait wakes
   → recv() → PrebootApiController.handle_request(VmmAction)
   → mutate VmResources → Ok(VmmData) | Err(VmmActionError)
   → Box<Result<...>> (ApiResponse) → mpsc send back
   → API thread serializes → HTTP 204/200/400 on the UDS

PUT /machine-config is the ideal subject: it is pre-boot (so it hits PrebootApiController), it has clear, typed fields (vcpu_count, mem_size_mib, smt, cpu_template, track_dirty_pages), and it mutates a single, easy-to-find field of VmResources (the MachineConfig). It exercises the whole channel without dragging in device construction.

Deep-dive companions for this lab: api-server-and-action-channel.md (the mechanics of the channel and the controllers), the-event-manager.md (the epoll loop that the eventfd wakes), and the-vmm-threading-model.md (why the two threads are split at all).

Why This Lab Matters for Contributors

When an issue says "PUT /machine-config accepts vcpu_count: 0 and then panics at boot" or "a PATCH after InstanceStart returns a confusing error," the first thing a maintainer does is locate the exact code on the path: which parser built the action, which controller dispatched it, which VmResources field it touched. Almost every API-validation and lifecycle-state bug in Level 8, the issue roadmap, and the capstone starts with exactly this trace. If you cannot get from a curl to the VmResources setter in a few minutes, you cannot triage. This lab makes that automatic.

Prerequisites

mkdir -p ~/firecracker-notes
: > ~/firecracker-notes/reading-log-3.1.md

Verify your build is current:

ARCH=$(uname -m)
BIN=build/cargo_target/${ARCH}-unknown-linux-musl/debug/firecracker
ls -l "$BIN" || echo "build first: tools/devtool build"

Note: Struct and function names in this lab are stable across recent branches; paths and line numbers are not. Every step gives you an rg/find to locate the hop on your checkout. If a name has drifted (the crate merge into vmm moved several files), the rg still lands you in the right neighborhood. Build the debug binary for this lab — it's faster to rebuild and your eprintln!/tracing lines show up plainly.

Part A — Read the Path (budget: 45 min)

Step 1 (5 min) — Confirm the request live, without booting

You don't need a guest to exercise PUT /machine-config; it's pre-boot config. Start the VMM with an API socket and send the request.

ARCH=$(uname -m)
BIN=build/cargo_target/${ARCH}-unknown-linux-musl/debug/firecracker
API=/tmp/fc-3.1.socket
rm -f "$API"

sudo "$BIN" --api-sock "$API" &
FC_PID=$!
sleep 0.3

# The request we will trace:
curl -sS -X PUT --unix-socket "$API" \
  --data '{"vcpu_count":2,"mem_size_mib":1024,"smt":false}' \
  -w '\nHTTP %{http_code}\n' \
  http://localhost/machine-config

# Read it back to prove it took effect:
curl -sS --unix-socket "$API" http://localhost/machine-config ; echo

Expected: the PUT returns HTTP 204 (no content) and the subsequent GET returns a JSON object echoing vcpu_count: 2, mem_size_mib: 1024. Leave the process running; you'll come back to it. Now find the code that produced that 204.

Step 2 (8 min) — Where the socket bytes are parsed (the API thread)

The HTTP server lives in the firecracker binary, not in vmm. Find the API server and the parsing entry point.

# The API server module:
find src/firecracker -type d -name api_server
ls src/firecracker/src/api_server/

# The parsed-request type and the dispatch on (method, path):
rg -rn "struct ParsedRequest|enum RequestAction|try_from_request|fn parse" src/firecracker/src/api_server/ | head -30

# The per-endpoint parser for machine-config:
rg -rn "machine-config|machine_config|MachineConfig" src/firecracker/src/api_server/ | head

You are looking for two things:

  1. The router: the code that matches (PUT, "/machine-config") and routes to a per-endpoint parser. It is keyed on the HTTP method and the first path segment.
  2. The per-endpoint parser for machine-config, which deserializes the JSON body into a MachineConfig (or an MachineConfigUpdate for PATCH) and wraps it as a VmmAction.

Read the parser. Note that it does not validate everything itself and it does not touch any Vmm state — it produces a ParsedRequest whose action is a VmmAction. Find which variant:

rg -n "VmmAction::" src/firecracker/src/api_server/ | rg -i "machineconfig|vmconfig" | head

You should see something like VmmAction::SetVmConfiguration(...) (verify the exact variant name on your branch — the enum is in rpc_interface.rs).

Log it:

cat >> ~/firecracker-notes/reading-log-3.1.md <<'EOF'
## PUT /machine-config
1. src/firecracker/src/api_server/  — router matches (PUT,/machine-config)
   -> per-endpoint parser deserializes body -> MachineConfig
   -> ParsedRequest { action = VmmAction::SetVmConfiguration(cfg) }   (verify variant on branch)
   -> NO Vmm state touched here; this is the API thread only
EOF

Step 3 (8 min) — The VmmAction enum and the channel types

Now cross into the vmm crate, where the action enum and the channel message types live.

# The control-plane command enum:
rg -n "enum VmmAction" src/vmm/src/rpc_interface.rs
# The exact machine-config variant:
rg -n "SetVmConfiguration|UpdateVmConfiguration|MachineConfig" src/vmm/src/rpc_interface.rs | head
# The boxed channel message types:
rg -n "type ApiRequest|type ApiResponse|enum VmmData|enum VmmActionError" src/vmm/src/rpc_interface.rs

Read the declarations:

  • VmmAction is a large enum — one variant per control operation. Your request is one variant.
  • type ApiRequest = Box<VmmAction> — the action is boxed before it goes on the channel (the enum is big; boxing keeps the channel message small).
  • type ApiResponse = Box<Result<VmmData, VmmActionError>> — the reply is a boxed Result. VmmData is the success payload (it has a variant per query/result); VmmActionError is the failure.

Log it:

cat >> ~/firecracker-notes/reading-log-3.1.md <<'EOF'
2. src/vmm/src/rpc_interface.rs
   - VmmAction::SetVmConfiguration(MachineConfigUpdate)   (one variant of a big enum)
   - type ApiRequest  = Box<VmmAction>                    (boxed onto the mpsc channel)
   - type ApiResponse = Box<Result<VmmData, VmmActionError>>
EOF

Step 4 (10 min) — Where the channel + eventfd are created and the send happens

The action is sent and the VMM is woken by the API-server adapter, not main.rs. Find the wiring.

# The thread + channel + eventfd wiring (NOT main.rs):
rg -rn "fn run_with_api|fn run_without_api" src/firecracker/src/
# The mpsc channel(s) and the API wake-up eventfd:
rg -rn "channel\(\)|sync_channel|EventFd|api_event_fd|to_vmm|from_api" src/firecracker/src/ | head -30
# The actual send + wake on the API side:
rg -rn "\.send\(|api_event_fd.*write|\.write\(1" src/firecracker/src/ | head

Read run_with_api. Confirm for yourself:

  1. It creates a pair of mpsc channels (api_request / api_response) and an eventfd.
  2. It spawns the API thread (the micro-http server) holding the sender end and the eventfd.
  3. It builds the EventManager, registers the eventfd as a subscriber, and runs the VMM loop.

On the API side, sending a request is two operations: sender.send(Box::new(action)) and event_fd.write(1). The send queues the message; the write is what makes epoll_wait return on the VMM thread. Miss the second and the VMM never wakes — a classic deadlock shape. Confirm both exist together.

Log it:

cat >> ~/firecracker-notes/reading-log-3.1.md <<'EOF'
3. src/firecracker/src/  (run_with_api adapter — NOT main.rs)
   - creates mpsc api_request/api_response channels + an eventfd
   - spawns API thread (micro-http); registers the eventfd with EventManager
   - API side: sender.send(Box<VmmAction>)  THEN  event_fd.write(1)  (wake the VMM)
   - then blocks on api_response.recv()
EOF

Step 5 (8 min) — The VMM wakes and dispatches through the controller

On the VMM thread, the eventfd is a MutEventSubscriber. When it becomes readable, EventManager calls its process, which drains the channel and dispatches the action.

# The subscriber that handles the API eventfd on the VMM side:
rg -rn "impl MutEventSubscriber|fn process|api_event_fd|handle_preboot_request|handle_request" src/firecracker/src/ src/vmm/src/ | rg -i "api|preboot|runtime" | head

# The two controllers and their dispatch:
rg -n "PrebootApiController|RuntimeApiController|fn handle_request|fn handle_preboot_request" src/vmm/src/rpc_interface.rs | head

Read the dispatch in rpc_interface.rs. Find the big match self (or match request) over VmmAction inside PrebootApiController. Locate the arm for your variant — for SetVmConfiguration it calls into VmResources to update the machine config and returns Ok(VmmData::Empty) (or the relevant data) on success, or an Err(VmmActionError::...) on a bad value. This is the moment the control command becomes a state mutation.

Log it:

cat >> ~/firecracker-notes/reading-log-3.1.md <<'EOF'
4. VMM thread: API eventfd subscriber.process() drains the channel
   -> PrebootApiController.handle_request(VmmAction::SetVmConfiguration(cfg))
   -> match arm calls into VmResources to set machine config
   -> Ok(VmmData::Empty) | Err(VmmActionError::MachineConfig(...))
EOF

Step 6 (6 min) — The mutation of VmResources, and the response home

Find the VmResources setter the controller arm calls.

rg -n "struct VmResources|fn set_vm_config|fn update_vm_config|machine_config|fn vm_config" src/vmm/src/resources.rs | head
rg -n "struct MachineConfig|fn update|fn check" src/vmm/src/vmm_config/machine_config.rs | head

Read the setter. This is where the validation that should happen pre-boot lives (e.g. rejecting vcpu_count == 0, or mem_size_mib below a floor). The mutated field becomes the source of truth the builder consumes at InstanceStart.

The Ok(VmmData)/Err(VmmActionError) returned by the controller is boxed as an ApiResponse, sent back over the response channel, received by the blocked API thread, and serialized into the HTTP response — a 204 for an empty success, a JSON body for a GET, or a 400 with the error for a rejection.

Log it:

cat >> ~/firecracker-notes/reading-log-3.1.md <<'EOF'
5. src/vmm/src/resources.rs / vmm_config/machine_config.rs
   - VmResources setter validates + stores the new MachineConfig (source of truth for the builder)
   - controller returns ApiResponse = Box<Result<VmmData, VmmActionError>>
   - VMM thread sends it back -> API thread recv() -> HTTP 204 / 200+JSON / 400+error
EOF

Part B — Prove It with Instrumentation (budget: 30 min)

Reading is half the lab. Now you prove the message crossed the thread boundary by adding three temporary log lines — one on the API side before the send, one inside the controller dispatch, one after the VmResources mutation — rebuilding, and watching them fire in order with different thread contexts.

Step 7 (10 min) — Add temporary instrumentation

Firecracker uses the log/tracing ecosystem, but for a throwaway trace the bluntest tool is best: eprintln! to stderr, which is visible the moment the binary runs. Add these three lines (adjust to the exact functions you found above — the rg from Steps 4–6 named them).

Warning: This is throwaway instrumentation. Never commit eprintln! — clippy and tools/devtool checkstyle will reject it, and it bypasses the logger and seccomp expectations. You will git restore it at the end of the lab. If you want a permanent version, use the tracing macros the codebase already imports (rg -n "use tracing|info!\|debug!" src/vmm/src/rpc_interface.rs).

On the API side, just before the sender.send(...) you found in Step 4:

#![allow(unused)]
fn main() {
// TEMP-3.1: prove the API thread sent the action and is about to wake the VMM.
eprintln!("[3.1][api  ][{:?}] sending VmmAction over channel", std::thread::current().id());
}

Inside PrebootApiController's dispatch (Step 5), at the top of handle_request (or in the machine-config arm):

#![allow(unused)]
fn main() {
// TEMP-3.1: prove the VMM thread received and is dispatching the action.
eprintln!("[3.1][vmm  ][{:?}] dispatching VmmAction = {:?}", std::thread::current().id(), request);
}

After the VmResources mutation (Step 6), in the machine-config setter:

#![allow(unused)]
fn main() {
// TEMP-3.1: prove the mutation happened on the VMM thread.
eprintln!("[3.1][vmm  ][{:?}] VmResources machine_config updated", std::thread::current().id());
}

Tip: If request isn't Debug-printable where you put the second line (it may be moved into the match), print a literal string instead: "dispatching SetVmConfiguration". The point is the thread id, not the payload.

Step 8 (10 min) — Rebuild and re-run the request

# Rebuild the debug binary (fast incremental rebuild):
tools/devtool build

ARCH=$(uname -m)
BIN=build/cargo_target/${ARCH}-unknown-linux-musl/debug/firecracker
API=/tmp/fc-3.1.socket
rm -f "$API"

# Run in the FOREGROUND so stderr prints to your terminal:
sudo "$BIN" --api-sock "$API" 2>/tmp/fc-3.1.stderr &
sleep 0.3

curl -sS -X PUT --unix-socket "$API" \
  --data '{"vcpu_count":2,"mem_size_mib":1024,"smt":false}' \
  -w 'HTTP %{http_code}\n' http://localhost/machine-config

# Look at what your instrumentation printed:
grep '\[3.1\]' /tmp/fc-3.1.stderr

Expected output (thread ids will differ on your machine):

[3.1][api  ][ThreadId(2)] sending VmmAction over channel
[3.1][vmm  ][ThreadId(1)] dispatching VmmAction = SetVmConfiguration(...)
[3.1][vmm  ][ThreadId(1)] VmResources machine_config updated

The two facts to see, not just believe:

  1. The first line has a different ThreadId from the second and third. That difference is the API/VMM thread boundary you read about. The action was created on one thread and dispatched on another.
  2. The order is api → vmm → vmm. The send-then-wake handed off to the epoll loop, which dispatched and mutated.

Step 9 (5 min) — A negative case: a rejection never mutates

Send a deliberately invalid request and watch the dispatch line fire but the mutation line not:

curl -sS -X PUT --unix-socket "$API" \
  --data '{"vcpu_count":0,"mem_size_mib":1024}' \
  -w 'HTTP %{http_code}\n' http://localhost/machine-config
grep '\[3.1\]' /tmp/fc-3.1.stderr | tail -4

If your branch rejects vcpu_count: 0 (it should — verify the floor in machine_config.rs), you'll see the dispatching line but no machine_config updated line, and HTTP 400. That is the controller returning Err(VmmActionError) before touching VmResources. You have just located the exact validation that an "accepts a bad value" bug would live in.

Step 10 (5 min) — Clean up

sudo kill "$FC_PID" 2>/dev/null
sudo pkill -f "fc-3.1.socket" 2>/dev/null
# Remove every temporary line you added — leave the tree clean:
git restore src/firecracker/src/ src/vmm/src/
git status --short    # must be empty
rm -f /tmp/fc-3.1.socket /tmp/fc-3.1.stderr

Warning: Confirm git status --short is empty before you consider the lab done. Leaving eprintln! in your tree is the fastest way to fail checkstyle on your next, real PR.

The Full Path at a Glance

sequenceDiagram
    participant C as curl
    participant API as API thread<br/>(micro-http, ThreadId A)
    participant CH as mpsc + eventfd
    participant EM as VMM thread<br/>(EventManager, ThreadId B)
    participant CTL as PrebootApiController
    participant VR as VmResources

    C->>API: PUT /machine-config {vcpu_count,mem_size_mib}
    Note over API: api_server/ parser:<br/>JSON → MachineConfig → ParsedRequest
    API->>API: VmmAction::SetVmConfiguration(cfg)
    API->>CH: sender.send(Box<VmmAction>)
    API->>CH: event_fd.write(1)
    API-->>API: block on api_response.recv()
    EM->>CH: epoll_wait wakes (API eventfd readable)
    EM->>CH: recv() → Box<VmmAction>
    EM->>CTL: handle_request(SetVmConfiguration)
    CTL->>VR: set machine config (validate + store)
    alt valid
        VR-->>CTL: Ok(())
        CTL-->>CH: Box(Ok(VmmData::Empty))
        CH-->>API: api_response.recv()
        API->>C: HTTP 204
    else invalid (e.g. vcpu_count=0)
        VR-->>CTL: Err(...)
        CTL-->>CH: Box(Err(VmmActionError::MachineConfig(...)))
        CH-->>API: api_response.recv()
        API->>C: HTTP 400 + error JSON
    end

Implementation Requirements / Deliverables

  • A ~/firecracker-notes/reading-log-3.1.md with a file path for every hop: parser → VmmAction → channel/eventfd wiring → controller → VmResources setter.
  • The three instrumentation lines added, the rebuild done, and the captured stderr showing the api → vmm → vmm order with two distinct thread ids.
  • The negative-case capture: an invalid PUT that dispatches but does not mutate, returning 400.
  • A clean tree afterward (git status --short empty).
  • A one-sentence answer to: what does the eventfd do that the mpsc send does not?

Troubleshooting

The curl hangs forever

Your eventfd write is missing, or you instrumented the wrong send. The mpsc send alone does not wake the VMM — the API thread will queue the message and block on the response, while the VMM sleeps in epoll_wait. Re-check Step 4: the API side must do send(...) and event_fd.write(1). If your own edit accidentally returned early before the wake, restore and retry.

My eprintln lines don't appear

You either rebuilt the wrong profile (you ran the release binary but edited for debug, or vice versa), or your edit is in a function that isn't on this path. Confirm the binary you ran is the one you just rebuilt (ls -l the timestamp), and that the rg from Steps 4–6 actually pointed at the function you edited. Re-run the rg for SetVmConfiguration to be sure you're in the right arm.

Both lines show the same thread id

You likely instrumented two points that are both on the API thread, or both on the VMM thread. Re-read: the send is on the API thread; the dispatch and mutation are on the VMM thread. If the send line shows ThreadId(1) (the main/VMM thread), you instrumented the wrong send — find the one inside the API thread's request handler, not a config-file path.

checkstyle fails on my next PR

You left an eprintln! behind. Run git restore over src/firecracker/src/ and src/vmm/src/ and confirm git status --short is empty. Clippy treats stray debug printing as an error in this repo.

HTTP 400 on a request you think is valid

Read the error body — it's a VmmActionError serialized to JSON, and it names the rejection. That is the validation in the VmResources/MachineConfig setter doing its job. Compare your JSON against the swagger (rg -n "machine-config" src/firecracker/swagger/firecracker.yaml).

Expected Output

A complete reading log plus a captured trace:

$ grep '\[3.1\]' /tmp/fc-3.1.stderr
[3.1][api  ][ThreadId(2)] sending VmmAction over channel
[3.1][vmm  ][ThreadId(1)] dispatching VmmAction = SetVmConfiguration(...)
[3.1][vmm  ][ThreadId(1)] VmResources machine_config updated

$ git status --short
            # (empty — clean tree)

Stretch Goals

  1. Trace PUT /drives/{id} instead. It's still pre-boot, but the controller arm calls VmResources::set_block_device / insert_block_device, which constructs more state than a plain machine-config update. Find the variant (rg -n "InsertBlockDevice|InsertNetworkDevice" src/vmm/src/rpc_interface.rs) and follow it to the device config in vmm_config/drive.rs.
  2. Trace a runtime action. Boot a microVM, then PATCH /vm {"state":"Paused"}. This time the active controller is RuntimeApiController, and the action acts on the live Vmm, not VmResources. Add an instrumentation line in the runtime dispatch and confirm the same channel carries it but a different controller handles it.
  3. Measure the round trip. Wrap the API-side send and the response recv with Instant::now() and print the elapsed time. It will be microseconds — visible proof that the control plane is cheap and does not touch the fast path.
  4. Find the seccomp angle. Open resources/seccomp/$(uname -m).json and find the api and vmm filter categories. The send/recv on the channel and the eventfd write are syscalls each category must permit. Which syscalls (write, read, futex, eventfd) appear in both? (Preview of Level 9.)

Validation / Self-check

Answer without looking back at this lab:

  1. Which thread parses the HTTP request into a ParsedRequest, and which thread executes the match arm that mutates VmResources? How did your instrumentation prove they are different threads?
  2. The API thread does two operations to deliver a request to the VMM. Name both, and say exactly what goes wrong if the second is omitted.
  3. What are the concrete types of ApiRequest and ApiResponse, and why is the VmmAction boxed?
  4. Which controller handled your PUT /machine-config — PrebootApiController or RuntimeApiController — and what determines which one is active?
  5. In the negative case (vcpu_count: 0), the dispatching line fired but the updated line did not. In which file and function does that rejection happen, and what does the controller return?
  6. Where are the mpsc channels and the wake-up eventfd actually created — name the function, and say why it is not main.rs.
  7. A bug report says "PUT /machine-config accepts a memory size that crashes the build at boot." Name the two files you would open first and why.

When you can answer all seven and your reading log has a file path for every hop, you've completed Lab 3.1. Continue to Lab 3.2: The Threading Model and the EventManager.