Debugging and Profiling — Intensive

Every other masterclass teaches you how a subsystem is supposed to work. This one teaches you what to do when it doesn't — when a microVM hangs before it ever prints a login prompt, when a guest's KVM_RUN loop wedges in an MMIO exit, when the VMM vanishes with no log line, or when a change that "obviously" couldn't matter adds 8 ms to boot. The skill that separates a serious Firecracker contributor from a tourist is not knowing the code; it is knowing, under pressure, which tool answers which question, and being able to drive that tool to a root cause instead of guessing.

Firecracker is harder to debug than an ordinary process for reasons that are intentional. It is seccomp-jailed, so the syscalls a debugger relies on may be denied. It runs as three classes of thread — the API thread, the VMM thread, and one thread per vCPU — and a hang in one looks nothing like a hang in another. And half of "the program" is a guest kernel running on real silicon behind the KVM boundary, where your host debugger cannot follow it. So you need two debuggers, not one: a GDB stub inside Firecracker that speaks the remote serial protocol to debug the guest, and an ordinary host GDB/LLDB attached to the firecracker process to debug the VMM. Knowing which side a symptom lives on, and reaching for the right debugger, is the first thing this intensive trains.

Note: This intensive assumes you have completed Levels 4–5 — you can read the vCPU run loop, you understand the threading model, and you can drive the pytest integration harness. Debugging is not a Level-1 skill bolted on at the end; it is what you do once you already know what the code is meant to do, so you can see where reality diverges.

The four questions a contributor actually asks

Before any specific tool, internalize the order of operations. The mistake beginners make is reaching for a debugger (slow to set up, easy to get lost in) when a metrics flush or a Trace-level log line (one curl, instant) would have answered the question. Start at the top and descend only when the cheaper instrument is inconclusive.

flowchart TD
    Start{"What is the symptom?"} --> Guest["guest misbehaves<br/>(hangs in kernel, wrong behaviour)"]
    Start --> VMM["the VMM misbehaves<br/>(crash, hang, wrong emulation)"]
    Start --> Slow["it's slow / boots slowly"]
    Start --> Regress["it worked last week"]

    Guest --> GdbStub["Firecracker gdb feature<br/>target remote /tmp/gdb.socket<br/>hbreak in the guest kernel"]
    VMM --> Logs["metrics flush + Trace logs first<br/>(seccomp.num_faults? signals.sigsegv?)"]
    Logs --> HostGdb["then host gdb/lldb on the<br/>firecracker PID — break in the run loop"]
    Slow --> BootTimer["BootTimer device (magic byte 123)<br/>+ test_boottime.py + log_instrument"]
    Regress --> Bisect["deterministic repro<br/>+ git bisect run"]

Memorize the first box under each symptom: the GDB stub for guest misbehaviour, a metrics/log pass then host GDB for VMM misbehaviour, the BootTimer for slowness, and git bisect for a regression. Those cheap first moves route you to the right heavyweight tool.

The crucial distinction this intensive forces you to make is guest versus VMM. They are different programs, on different sides of the KVM boundary, debugged with different tools:

The two debuggers are not interchangeable, and confusing them wastes hours: setting a host breakpoint on start_kernel will never fire (that symbol runs in the guest), and a guest target remote will never show you Vcpu::run (that runs in the VMM). Lab 1 drills exactly this boundary.

What this intensive builds on

This intensive is the hands-on counterpart to two deep dives you should have read:

• Logging and metrics — the logger (PUT /logger, levels), the metrics counter tree (PUT /metrics, FlushMetrics), the counters that must stay zero (seccomp.num_faults, signals.sigsegv), and the log-instrument tracing facility. Lab 2 turns all of it into a workflow.
• Signals, shutdown, and reset — what a SIGSYS/SIGBUS/SIGSEGV VMM exit means, which is the difference between "the sandbox worked" and "there is a memory bug." When the VMM vanishes, this is the chapter that tells you where to look.

It also leans on the boot-time optimization essay (the BootTimer mechanism and the regression-gate philosophy) and the testing framework (because a git bisect is only as good as the scripted test it runs).

The three labs

• Lab 1: GDB a microVM — build Firecracker with the gdb feature, enable gdb_socket_path on machine-config, attach a GDB to the guest and break in start_kernel; then attach host gdb/lldb to the firecracker process and break in the VMM run loop. Live both sides of the KVM boundary in one session and never confuse them again.
• Lab 2: Tracing and metrics — wire up the logger (levels) and the metrics system, read the emitted counter JSON, turn on log_instrument instrumentation, use the BootTimer for boot timing, and assemble all of it into a repeatable observability workflow for diagnosing a slow or failing boot — without ever attaching a debugger.
• Lab 3: Reproduce and bisect — the core debugging discipline. Take a regression (behaviour or performance), write a deterministic reproduction, drive git bisect run with a scripted pass/fail test to find the offending commit in O(log n) steps, and read the diff to root-cause. This is the skill every other lab feeds into.

Deliverables

• Build a gdb-enabled Firecracker and attach a debugger to a running guest kernel, setting and hitting a breakpoint in start_kernel.
• Attach host gdb/lldb to the firecracker process and break inside the VMM run loop, and explain in one sentence why each breakpoint lives where it does.
• Produce a metrics-and-logs triage report for a failing boot: which counter moved, which log line confirmed it, and what it ruled in or out.
• Measure a real boot-to-userspace number with the BootTimer and break it down by phase.
• Write a deterministic reproducer and a git bisect run script that converges on a known introducing commit, then read that commit's diff and explain the root cause.

Common mistakes

How to verify success

cd ~/src/firecracker
# 1. You can build the gdb-enabled binary and find the stub module.
rg -n "feature = \"gdb\"|gdb_socket_path" src/ Cargo.toml
ls src/vmm/src/gdb/

# 2. You can find the BootTimer and the metrics flush.
rg -n "BootTimer|MAGIC_VALUE_SIGNAL_GUEST_BOOT_COMPLETE" src/vmm/src/devices/pseudo/boot_timer.rs
rg -n "FlushMetrics|flush_metrics" src/vmm/src/

# 3. You can drive the boot-time test and a scripted bisect target.
./tools/devtool test -- integration_tests/performance/test_boottime.py 2>&1 | tail -15

If those commands land on real files on your branch (they will move — that is why you run them rather than trust a line number), and you can articulate guest vs VMM without hesitating, you are ready.

PR Profile: a graduate of this intensive

A contributor who can reproduce, bisect, and root-cause is doing the part of the work the maintainers value most highly, because it is the part that does not scale and cannot be faked.

Begin with Lab 1: GDB a microVM. When you finish all three, the sibling Performance & Density intensive takes the boot-time and bisect skills you build here and turns them on Firecracker's headline numbers.