Lab 1: GDB a microVM

Background

There are two programs running when a microVM is live, and they are on opposite sides of the KVM boundary. One is firecracker — an ordinary (jailed) Linux process running on the host, with its API thread, its VMM thread, and its vCPU threads. The other is the guest: a Linux kernel and your rootfs, executing on the physical CPU under hardware virtualization, where the host's view stops at ioctl(vcpufd, KVM_RUN). A host debugger attached to firecracker can see Vcpu::run and the device emulation; it cannot see start_kernel or a guest page fault, because that code never runs as host instructions — KVM runs it directly on the silicon and only hands control back to the VMM on a VM exit.

So debugging a microVM means choosing a debugger by which side the bug is on:

  • To debug the guest (the kernel, a driver, guest userspace), you use Firecracker's gdb feature: a GDB stub compiled into the VMM that speaks the GDB Remote Serial Protocol over a Unix socket. Your gdb, loaded with the guest's vmlinux symbols, connects to that socket and drives the guest through KVM. This is the same idea as QEMU's -s -S gdbserver, or KGDB, but built into Firecracker and using KVM's hardware breakpoints.
  • To debug the VMM (device emulation, the run loop, the API path, snapshot/restore, a seccomp denial), you attach an ordinary host gdb or lldb to the firecracker PID, exactly as you would any Rust binary.

This lab makes you do both, in one session, against one microVM, so the boundary becomes muscle memory.

Warning: GDB support is a debug-build feature, gated behind a Cargo feature and intended for development, not production. It widens what a connected debugger can do to the guest and is not part of the hardened, seccomp-locked production path. Never enable gdb on a production binary. (Verify the feature name and the gating on your branch — see Step 1.)

Why This Lab Matters for Contributors

  • The single most common wasted afternoon in VMM work is debugging the wrong program — setting a host breakpoint on a guest symbol, or vice versa. Internalizing guest vs VMM here saves that afternoon forever.
  • When you contribute to the boot sequence or guest memory, being able to stop the guest at start_kernel and inspect its early state is the difference between reasoning about the zero page/e820 from the code and seeing the kernel consume it.
  • When you contribute to the vCPU run loop or a virtio device, breaking in the VMM at the exact VcpuExit or MMIO dispatch lets you watch an emulation decision happen instead of inferring it.
  • The GDB stub itself is real Firecracker code (src/vmm/src/gdb/) you could one day contribute to — register coverage, address translation, multi-core support are all live edges.

Prerequisites

# Verify your toolchain before you start.
cd ~/src/firecracker
which gdb || echo "install gdb"
ls -l /dev/kvm                       # must be read+write to you
git rev-parse --short HEAD           # record the commit you are debugging on

Step 1: Build Firecracker with the gdb feature

The GDB stub is compiled in only when you ask for it. Find the feature and the field it enables — do not trust the names from memory, they are version-sensitive.

cd ~/src/firecracker
# The Cargo feature that pulls in the GDB stub, and the gdbstub crate it uses.
rg -n "^gdb|feature = \"gdb\"|gdbstub" Cargo.toml src/vmm/Cargo.toml src/firecracker/Cargo.toml

# The stub module itself: target.rs (the gdbstub Target impl), event_loop.rs, arch/.
ls src/vmm/src/gdb/
rg -n "gdb_socket_path|GdbServer|target_remote|run_loop|hbreak|HwBreakpoint" src/vmm/src/gdb/

You should find a gdb/ module containing (names verify on your branch) mod.rs, target.rs, event_loop.rs, and an arch/ subdirectory — Firecracker implements the gdbstub crate's Target trait over KVM's guest-debug ioctls. Now build with the feature on. A debug build is what you want for debugging (symbols, no inlining surprises):

# Build the firecracker binary with the gdb feature enabled (debug profile).
./tools/devtool build --debug -- --features "gdb"

# Confirm the binary exists (arch/profile may differ — find it, don't assume).
find build/cargo_target -type f -name firecracker | rg "debug"
FC=$(find build/cargo_target -type f -name firecracker | rg "debug" | head -1); echo "$FC"

Note: --features "gdb" is passed through devtool to cargo. If the flag plumbing differs on your branch, rg -n "features|gdb" tools/devtool shows how devtool forwards cargo args. The ground truth is always cargo build --features "gdb" somewhere underneath.

Step 2: Get a guest kernel with debug info

You cannot set a symbolic breakpoint (hbreak start_kernel) without symbols, and the guest debug config needs a couple of kernel options. The CI vmlinux images may be stripped; for this lab build or fetch a vmlinux compiled with at least these options (find the guest configs in-tree):

ls resources/guest_configs/
# The options the GDB workflow needs (verify against docs/gdb-debugging.md on your branch):
rg -n "CONFIG_DEBUG_INFO|CONFIG_FRAME_POINTER|CONFIG_SCHED_MC" resources/guest_configs/

The documented requirements are CONFIG_DEBUG_INFO=y and CONFIG_FRAME_POINTER=y; the docs also recommend disabling multi-core scheduler heuristics (CONFIG_SCHED_MC=n, CONFIG_SCHED_MC_PRIO=n) because the stub's multi-core handling is limited. Read the canonical instructions:

sed -n '1,120p' docs/gdb-debugging.md

For the rest of this lab, assume you have ./vmlinux (with symbols) and a rootfs ./rootfs.ext4, as in Lab 1.3.

Step 3: Boot a microVM with the GDB socket enabled

Enabling the stub is a machine-config field: gdb_socket_path. Set it before InstanceStart, with a single vCPU (multi-vCPU guest debugging is limited). Two ways:

Config-file boot (simplest — the whole machine in one JSON):

cat > vm.json <<'EOF'
{
  "boot-source": {
    "kernel_image_path": "./vmlinux",
    "boot_args": "console=ttyS0 reboot=k panic=1 pci=off nomodule"
  },
  "drives": [
    {"drive_id": "rootfs", "path_on_host": "./rootfs.ext4",
     "is_root_device": true, "is_read_only": false}
  ],
  "machine-config": {
    "vcpu_count": 1,
    "mem_size_mib": 256,
    "gdb_socket_path": "/tmp/gdb.socket"
  }
}
EOF

# Start the gdb-enabled binary. It will set up the VM, then BLOCK waiting for gdb
# to connect on /tmp/gdb.socket before the guest runs a single instruction.
sudo "$FC" --no-api --config-file vm.json

API boot (set gdb_socket_path via PATCH /machine-config pre-boot, then InstanceStart):

API=/tmp/fc.sock
sudo "$FC" --api-sock "$API" &
curl -X PUT --unix-socket "$API" --data \
  '{"kernel_image_path":"./vmlinux","boot_args":"console=ttyS0 reboot=k panic=1 pci=off nomodule"}' \
  http://localhost/boot-source
curl -X PUT --unix-socket "$API" --data \
  '{"drive_id":"rootfs","path_on_host":"./rootfs.ext4","is_root_device":true,"is_read_only":false}' \
  http://localhost/drives/rootfs
# The key line: enable the gdb socket on machine-config, pre-boot.
curl -X PATCH --unix-socket "$API" --data \
  '{"gdb_socket_path":"/tmp/gdb.socket"}' http://localhost/machine-config
curl -X PUT --unix-socket "$API" --data \
  '{"action_type":"InstanceStart"}' http://localhost/actions
# InstanceStart returns, but the guest is HALTED at entry, waiting for gdb.

Note: With the GDB socket set, Firecracker stops the guest at the kernel entry point and waits for a debugger — like QEMU's -S. Nothing in the guest runs until you continue from gdb. That is exactly what you want to break at start_kernel.

Step 4: Attach GDB to the GUEST and break in the kernel

In a second terminal, start gdb on the guest vmlinux and connect to the socket. The connection is a Unix socket (the GDB remote protocol over a UDS):

gdb ./vmlinux

(gdb) target remote /tmp/gdb.socket
Remote debugging using /tmp/gdb.socket
0x000000000... in ?? ()

# Hardware breakpoint at the kernel's C entry. Use hbreak, not break --
# software breakpoints rewrite guest memory, which is fragile this early;
# KVM hardware breakpoints are what the stub supports cleanly.
(gdb) hbreak start_kernel
Hardware assisted breakpoint 1 at 0xffffffff...: file init/main.c, line ...

(gdb) continue
Continuing.

Breakpoint 1, start_kernel () at init/main.c:...
...
(gdb) backtrace
(gdb) info registers rip rsp rsi          # rsi held the zero-page address at boot
(gdb) print boot_command_line             # the kernel cmdline you passed

You are now stopped inside the guest kernel, on real virtualized hardware, with full symbols. Things to do here to prove the boundary to yourself:

# Step a few instructions of guest code.
(gdb) stepi
(gdb) stepi

# Break later, in device setup, to watch a virtio-mmio probe.
(gdb) hbreak setup_arch
(gdb) continue

# Pause a running guest at any time with Ctrl+C (it interrupts vCPU 1),
# then 'continue' to resume. 'kill' / the documented exit ends the session.

Tip: Only a limited subset of CPU registers is exposed by the stub, and on aarch64 guest virtual-address translation is limited (4 KiB pages, not all physical address sizes). If info registers or reading a guest pointer behaves oddly, that is a known stub limitation, not your mistake — rg -n "registers|translate|read_addr" src/vmm/src/gdb/arch/ shows what is implemented.

Step 5: Now debug the VMM — host gdb/lldb on the firecracker process

Detach from the guest (or leave it paused) and switch sides. The VMM is an ordinary host process. Find its PID and the threads.

# Find the firecracker process and its threads (API, VMM, one per vCPU).
FCPID=$(pgrep -n firecracker); echo "$FCPID"
ps -L -p "$FCPID" -o tid,comm     # the thread names: fc_api, fc_vmm, fc_vcpu N

Attach host gdb (or lldb -p "$FCPID"). Because Firecracker is seccomp-filtered, attaching may be blocked by the production filter; for VMM debugging start the binary with --no-seccomp (development only — it changes the security posture, so never draw production conclusions from a --no-seccomp run):

# (Re)start for VMM debugging with seccomp off so ptrace/gdb can attach cleanly.
# sudo "$FC" --no-api --config-file vm.json --no-seccomp   (dev only)
sudo gdb -p "$FCPID"

Set a breakpoint in the run loop — the host code, not the guest. Locate it first; the symbol path is version-sensitive:

# Find the vCPU run loop and the VM-exit dispatch in the VMM.
rg -n "fn run\b|KVM_RUN|VcpuExit::|fn run_emulation|handle_mmio|fn process" \
  src/vmm/src/vstate/vcpu/

# Break where the vCPU loop dispatches a VM exit. Tab-complete the mangled Rust path;
# the role is "the function that matches on VcpuExit". Example shape:
(gdb) break vmm::vstate::vcpu::*::run_emulation
(gdb) continue
# ... trigger guest I/O (e.g. log in, run `ls`) ...
Thread 4 "fc_vcpu 0" hit Breakpoint 1, ... run_emulation (...)
(gdb) bt
(gdb) info threads          # see fc_vmm and the fc_vcpu threads
(gdb) thread apply all bt   # a "where is every thread" snapshot — the VMM analogue of a thread dump

To break in the VMM thread (device emulation, the EventManager loop) instead of a vCPU thread:

rg -n "fn process|fn handle_event|run_event_loop|EventManager|fn write\b|fn read\b" \
  src/vmm/src/devices/virtio/block/ src/vmm/src/devices/virtio/mmio.rs

(gdb) break vmm::devices::virtio::block::*::process_queue
(gdb) continue
# ... do a guest disk read ...   then inspect the request, the descriptor chain, guest memory.

This is the payoff: the same microVM, debugged from both sides. The guest gdb stopped you at start_kernel; host gdb stops you at the VMM code that emulates the hardware the guest is talking to.

Step 6: Tell the two debuggers apart, deliberately

Run the experiment that cements the boundary. With the guest gdb attached:

(gdb) break run_emulation        # a VMM symbol
Function "run_emulation" not defined.    # the guest has no idea what that is

With host gdb attached to firecracker:

(gdb) break start_kernel         # a guest symbol
Function "start_kernel" not defined.     # the VMM binary has no guest kernel symbols

Each debugger only knows its own program. That single fact — that start_kernel and run_emulation live in different symbol tables on different sides of KVM_RUN — is the whole lab.

Implementation Requirements / Deliverables

  • A gdb-enabled debug build of firecracker, located with find (not a memorized path).
  • A microVM booted with gdb_socket_path set, halted at guest entry.
  • A guest gdb session that hits hbreak start_kernel and prints a backtrace + boot_command_line.
  • A host gdb/lldb session that breaks in the vCPU run loop (a VcpuExit dispatch) and in a VMM-thread device path, with thread apply all bt captured once.
  • A one-paragraph write-up: for three concrete symptoms (a guest kernel panic, a block-read that returns wrong bytes, a VMM crash on InstanceStart), state which debugger you would reach for and why.

Troubleshooting

target remote /tmp/gdb.socket hangs or "Connection refused"

The socket only exists once Firecracker has set up the VM and is waiting. Confirm the binary was built with --features "gdb" (strings "$FC" | rg gdb or just re-run Step 1's rg), that gdb_socket_path was set before InstanceStart, and that the path matches exactly. ls -l /tmp/gdb.socket should show a socket.

hbreak start_kernel says "Function not defined"

Your vmlinux has no symbols, or you loaded the wrong file. gdb ./vmlinux then info functions start_kernel. If empty, rebuild the guest kernel with CONFIG_DEBUG_INFO=y (Step 2). A stripped CI kernel will not work for symbolic breakpoints.

The breakpoint is hit but registers/memory look wrong

Expected within limits: the stub exposes a subset of registers and limited address translation (especially aarch64). Check what src/vmm/src/gdb/arch/ actually implements; do not assume full register coverage. For multi-vCPU oddities, rebuild the guest with CONFIG_SCHED_MC=n and use a single vCPU.

Host gdb can't attach to firecracker ("Operation not permitted")

Either seccomp is blocking the ptrace path (start the dev run with --no-seccomp), or ptrace_scope is locked down on your host (cat /proc/sys/kernel/yama/ptrace_scope; sudo the gdb, or run firecracker and gdb as the same user). Remember --no-seccomp changes the posture — re-verify any fix under the real filter.

Rust symbol names won't tab-complete

They are mangled and the module path moves. Find the function with rg first (Steps 5), then in gdb use rbreak with a regex on the role, e.g. rbreak run_emulation, and pick from the list.

Expected Output

A guest session that looks like:

(gdb) target remote /tmp/gdb.socket
(gdb) hbreak start_kernel
(gdb) c
Breakpoint 1, start_kernel () at init/main.c:...
(gdb) bt
#0  start_kernel () at init/main.c:...

and a host session where info threads shows fc_vmm plus one fc_vcpu N per configured vCPU, and a breakpoint in the run loop fires when the guest does I/O.

Stretch Goals

  1. Watch a virtio-block request cross the boundary. Set a guest breakpoint in the block driver's submit path and a host breakpoint in process_queue. Trigger one read; step the request from the guest driver, through the kick, to the VMM's descriptor-chain handling. You have now watched one request traverse the KVM boundary in both debuggers.
  2. Read the GDB stub's Target impl. Open src/vmm/src/gdb/target.rs and map the gdbstub Target methods (read_registers, write_addr, breakpoint ops) onto the KVM guest-debug ioctls they call. Note exactly which registers are exposed and which are stubbed — that is a real contribution surface.
  3. Break on a guest page fault. Set a watchpoint or a breakpoint in the guest fault path and a host breakpoint where guest memory is mapped (rg -n "GuestMemoryMmap|fn get_slice|MAP_" src/vmm/src/vstate/memory.rs) to see the same physical page from both sides.

Validation / Self-check

Answer without notes; these gate completion:

  1. Why can a host gdb attached to firecracker never set a working breakpoint on start_kernel, and why can the guest gdb never break on run_emulation?
  2. What does gdb_socket_path go under in the configuration, and when (relative to InstanceStart) must it be set?
  3. Why does the workflow use hbreak rather than break for the guest kernel, and what KVM feature makes hbreak possible?
  4. Name two documented limitations of the GDB stub and where in src/vmm/src/gdb/ you would confirm them.
  5. Why is --no-seccomp sometimes needed for the VMM side, and why must you never report a finding from a --no-seccomp run as a production conclusion?
  6. Given "the guest hangs with no console output after InstanceStart," which debugger do you reach for first, and what is the very first breakpoint you set?
  7. In host gdb, what is the VMM analogue of a thread dump, and what does it tell you that a single backtrace does not?

Next: Lab 2: Tracing and metrics — when you should not reach for a debugger at all, and how to diagnose a failing boot from counters and logs without attaching to anything.