Step 3: Execution-Path Analysis

You have a deterministic repro. Now you trace the code that produces it — from the entry point where the trigger arrives down to the component that emits the symptom. The deliverable is a map: every hop from "request hits the socket" (or "vCPU takes this exit," or "the device pops this descriptor") to the line where behavior goes wrong, each hop located by a command you ran, not a line number you trusted.

This is where Levels 3–7 pay off. You already know the architecture — the three thread classes, the action channel, the run loop, the MMIO bus, the virtqueue. Step 3 is applying that map to this bug instead of reading it in the abstract.

Goal

Produce capstone-work/execution-path.md: an ordered list of code locations the trigger flows through, each with the rg/find that located it (anti-staleness — never a bare line number), a diagram of the path with the suspect node marked, and a single confirmed observation point where you have seen the value turn wrong (via a log line or a print), not merely reasoned that it must.

Anchor at the Entry Point

Every Firecracker bug enters through one of a small number of doors. Identify yours from the repro, then find the door's code with rg:

Note (anti-staleness): Run these on your checkout. The big crate merge put most subsystems under src/vmm/src/, but exact paths drift between branches and the next refactor will move something. Name structs and functions by role, then locate them — never cite a line number from this guide or the issue thread.

Trace, Don't Guess

Follow the trigger value hop by hop. At each hop, record the function (located by rg), what it does to the value, and where it hands off. The classic Firecracker control-plane path looks like this — adapt it to your bug:

curl PUT /machine-config
        │  (API thread)
        ▼
ApiServer parses request ──► ParsedRequest ──► VmmAction::UpdateMachineConfig(cfg)
        │  rg -n "UpdateMachineConfig" src/vmm/src/rpc_interface.rs
        ▼   Box<VmmAction> over std::sync::mpsc; eventfd wakes the VMM thread
PrebootApiController / RuntimeApiController dispatches the action
        │  rg -n "PrebootApiController|RuntimeApiController" src/vmm/src/rpc_interface.rs
        ▼
VmResources::update_machine_config(cfg)   ◄── validation lives here or nearby
        │  rg -n "fn update_machine_config|fn set_vcpu" src/vmm/src/
        ▼
... builder consumes VmResources at InstanceStart ...
build_microvm_for_boot ──► create vCPUs / map memory / build devices
        │  rg -n "fn build_microvm_for_boot" src/vmm/src/builder.rs
        ▼
[symptom surfaces here]

For a data-plane (virtio) bug the path is different — kick → queue → descriptor chain → host I/O → used ring → IRQ — and you trace it with the virtqueues deep dive and virtio-block deep dive open. For a vCPU bug it is the run loop and VM exits. Reopen the deep dive for your subsystem and walk its "reading exercise" against your repro — that is exactly the muscle this step uses.

Cross-reference the threading model as you go: which thread is this code on? A bug that looks like a race may be a control-plane action racing the VMM epoll loop — see the VMM threading model and the event manager.

Instrument to Confirm the Observation Point

Reading is a hypothesis; a log line is evidence. You must see the value turn wrong, not deduce it. Firecracker gives you three instruments, in increasing order of intrusiveness.

1. Existing logs and metrics

Turn the logger up and read what the code already emits before adding anything:

# Boot with a log file at TRACE/DEBUG and watch the relevant subsystem.
curl -X PUT --unix-socket $API \
  --data '{"log_path":"/tmp/fc.log","level":"Debug","show_level":true,"show_log_origin":true}' \
  http://localhost/logger
rg -n "<subsystem keyword>" /tmp/fc.log

See logging and metrics for what's already instrumented (per-device metrics often pinpoint a queue stall or a rejected request without any code change).

2. log-instrument tracing

The repo ships a log-instrument macro/crate for function-entry/exit tracing (dev/CI tooling). Locate how it's used and enable it for the path you're tracing:

rg -n "log_instrument|instrument" src/vmm/src/ | rg -i "<subsystem>"

3. A temporary println! / tracing probe

The bluntest instrument: drop a probe at the line where you suspect the value flips, print the value plus its expected, rebuild, run the repro, observe.

#![allow(unused)]
fn main() {
// TEMPORARY — remove before committing. Confirms the observation point.
eprintln!("DEBUG vcpu_count={} (expected reject when 0)", cfg.vcpu_count);
}

tools/devtool build && ./repro.sh 2>&1 | rg DEBUG

The line where the probe shows a wrong (or wrongly-accepted) value, when the probe one hop upstream shows a right value, is your observation point. Record it.

Warning: Every temporary probe comes out before you commit. tools/devtool checkstyle and clippy-as-errors will catch stray println!/dbg!, but the habit to build is: instrument, observe, remove, then re-instrument elsewhere. The probe's job is to find the line; the line's job is to be cited in your doc.

Localize: Find the Component, Not Yet the Cause

The output of tracing is localization — naming the component and the specific function where correct becomes incorrect. You are not yet diagnosing why (that is Step 4); you are proving where.

A clean localization statement looks like:

The trigger (vcpu_count: 0) is accepted at the API layer (VmResources::update_machine_config, located by rg -n "fn update_machine_config" src/vmm/src/) without a lower-bound check. The wrong value survives until build_microvm_for_boot (in src/vmm/src/builder.rs) tries to create zero vCPUs, where the symptom surfaces. Observation point: a probe in update_machine_config prints vcpu_count=0 and the function returns Ok, confirming validation is the gap, not vCPU creation.

Note what that statement does: it names two candidate sites (where the value is accepted and where the symptom surfaces) and uses the observation point to distinguish them. That distinction is the whole game in Step 4.

Diagram the Path

Add a diagram to execution-path.md — a mermaid flowchart or an ASCII box chain — with the suspect node clearly marked. A reviewer should be able to open each cited file at each hop and follow your trace without asking you a question.

flowchart TD
    A["curl PUT /machine-config {vcpu_count:0}"] --> B["ApiServer → ParsedRequest"]
    B --> C["VmmAction::UpdateMachineConfig"]
    C --> D["VmResources::update_machine_config<br/>(NO lower-bound check)"]
    D -->|Ok, value=0 survives| E["build_microvm_for_boot"]
    E --> F["create 0 vCPUs → symptom"]
    style D fill:#fdd,stroke:#c00

Deliverable for Step 3

• capstone-work/execution-path.md with an ordered hop list, each hop located by an rg/find command (no bare line numbers).
• A path diagram (mermaid or ASCII) with the suspect node marked.
• A confirmed observation point: the file + function (located, with a commit SHA recorded) where you saw the value turn wrong, and the instrument you used to see it.
• A one-paragraph localization statement naming the candidate fix site vs. the symptom site.
• All temporary probes removed from the tree.

Rubric Hooks

This is the Execution-path analysis dimension (18 pts) almost verbatim: located citations at every layer, an accurate diagram, and a confirmed observation point separate the maintainer-ready trace from "I think it's in the device manager." A doc with role-named, rg-located references pinned to a commit SHA scores high; vague prose ("the API handles it") scores low. See the evaluation rubric.

Validation / Self-check

Before advancing to Step 4:

1. You can name every hop from trigger to symptom, and locate each with a command.
2. You have observed (not merely reasoned) the value turning wrong, at one specific line, recorded with the commit SHA it was at.
3. Your diagram marks the suspect node and a reviewer could follow it cold.
4. You can state which thread each hop runs on (API / VMM / vCPU).
5. You have two candidate sites named: where the bad value is accepted and where the symptom surfaces — and you know which is which.
6. Every temporary probe is removed; git status is clean except your notes.

Then go to Step 4: Root-Cause Identification.