Level 5: Testing and Debugging

Up to here you have read the engine and made small, surgical changes. From this level on, the rule hardens: no change ships without a test, and a Firecracker maintainer's first question on almost every pull request is "where's the test, and is it deterministic?" CONTRIBUTING is explicit — new functionality requires integration tests, and you may not lower unit-test coverage. This level makes Firecracker's test stack your home so that you can answer both questions before anyone asks.

Firecracker's test stack has two distinct halves that you must stop conflating. The first is Rust unit tests — #[cfg(test)] modules compiled into each crate and run with cargo test, testing pure logic (rate-limiter token math, config validation, parsing, serialization round-trips) with no microVM in sight. The second is a Python pytest integration suite under tests/, driven by tools/devtool test, that builds the real firecracker binary, spawns it as a real process, configures it over the real REST API, boots a real guest, and SSHes in to assert end-to-end behavior. Both run in CI on x86_64 and aarch64. On top of those sit the style/build gates (tools/devtool fmt, checkstyle, checkbuild --all) and Kani formal-verification proofs for a handful of unsafe-adjacent invariants.

This is the level where you learn how a mature, security-critical project defends itself. By the end you can run a single test of either kind, read its failure, drive a real microVM under a debugger, and reach for the cheapest test that still proves your change. This curriculum will not hold your hand here either: it points you at tests/framework/, gives you the questions, and makes you run every command.

Note: The integration suite is the load-bearing half. tools/devtool test runs pytest, not cargo test. The Rust unit tests are cargo test, but the suite that gates your PR — the one that boots real microVMs — is Python. Internalize this split now; mixing them up wastes hours.

Learning Objectives

By the end of Level 5 you must be able to:

1. Explain Firecracker's two-layer test stack — Rust #[cfg(test)] unit tests vs the pytest integration suite — and choose the right layer for a given change.
2. Run a single Rust unit test (cargo test, scoped to one crate/module/function) and a single pytest integration test (tools/devtool test -- <path>::<name>), and read each failure correctly.
3. Describe the pytest framework: the Microvm helper, the microvm_factory fixture, the guest_kernel/rootfs artifact fixtures, and the uvm/uvm_plain convenience fixtures.
4. Trace exactly what an integration test does to a running firecracker: spawn the process, configure over the Unix socket, InstanceStart, SSH into the guest, assert, tear down.
5. Write a focused, table-driven Rust unit test for an under-covered function and run it without spinning up the whole suite.
6. Write a new pytest that boots a microVM and asserts an end-to-end behavior (a second drive, a NIC, a metrics field), using the Microvm helper and SSH-into-guest utilities.
7. Debug a misbehaving microVM with the tools that actually work here: the Firecracker log and metrics, the serial console, the API socket, strace, and gdb against a vCPU thread.
8. Reproduce a flaky test deterministically, identify the race/timing/ordering assumption, and fix it without time.sleep-as-synchronization.

The Two-Layer Test Stack

Firecracker tests at two altitudes. Picking the lower one whenever it can prove your change is the single most valuable testing skill — unit tests run in milliseconds; an integration test builds a binary, boots a kernel, and SSHes into a guest.

                         What does your change touch?
                                     │
        ┌────────────────────────────┴────────────────────────────┐
        │                                                          │
  pure logic in one crate                          observable microVM behavior
  (rate-limiter math, a config                     (a drive appears as /dev/vdb,
   validator, a parser, a                           a NIC gets configured, a
   serialization round-trip)                         metric updates, an API
        │                                            error is returned, a boot
        ▼                                            succeeds/fails)
  ┌──────────────────┐                                        │
  │ Rust unit test   │                                        ▼
  │ #[cfg(test)] mod  │                          ┌──────────────────────────────┐
  │ run: cargo test   │                          │ pytest integration test       │
  └──────────────────┘                          │ tests/integration_tests/...   │
        (ms; no VM)                              │ run: tools/devtool test       │
                                                 │ (builds binary, boots a guest)│
                                                 └──────────────────────────────┘

flowchart TD
    Q{What are you proving?} --> U[Pure logic in one crate]
    Q --> I[Observable microVM behavior]
    Q --> S[Wire/serialization round-trip]
    Q --> P[Formal invariant on unsafe-adjacent code]
    U --> UT["Rust #[cfg(test)] unit test<br/>cargo test -p vmm module::test"]
    S --> UT
    I --> IT["pytest in tests/integration_tests/<br/>tools/devtool test -- path::name"]
    P --> K["Kani proof harness<br/>cargo kani / labeled 'Kani'"]

Tip: Clippy is warnings-as-errors in CI: cargo clippy --all --all-targets --all-features -- -D warnings. A green cargo test that fails clippy still fails the PR. Run tools/devtool checkstyle before you push.

The pytest Integration Framework, From Orbit

The integration suite is a normal pytest project with a heavyweight fixture layer that manufactures real microVMs. You will dissect it in Lab 5.1; here is the map so the labs have something concrete underneath.

# Confirm the framework exists and see its shape — run this, don't trust the prose.
find tests -maxdepth 2 -type d | sort
ls tests/framework/
rg -n "class Microvm" tests/framework/microvm.py
rg -n "def microvm_factory|def uvm|def guest_kernel|def rootfs" tests/conftest.py

tests/
├── conftest.py                  ← session/test fixtures: microvm_factory, guest_kernel, rootfs, uvm…
├── pytest.ini                   ← markers, options
├── pyproject.toml               ← deps / config
├── framework/                   ← the framework package (the helper layer)
│   ├── microvm.py               ← the Microvm class (THE central helper)
│   ├── microvm_helpers.py       ← guest-side helpers (verify on your branch)
│   ├── artifacts.py             ← kernels / rootfs / CI artifacts
│   ├── jailer.py                ← JailerContext (run firecracker under the jailer)
│   ├── http_api.py / *_api      ← the Api wrapper around the REST socket (verify name)
│   ├── utils.py, utils_*.py     ← ssh, networking, cpu templates, …
│   └── ab_test.py               ← A/B (before/after) performance testing
├── host_tools/                  ← C/Rust guest-side test binaries, network helpers
└── integration_tests/
    ├── functional/              ← test_drive_virtio.py, test_net.py, test_api.py, test_metrics.py…
    ├── performance/             ← test_boottime.py, test_snapshot.py, test_huge_pages.py
    ├── security/                ← test_vulnerabilities.py, test_sec_audit.py
    └── build/                   ← test_sanitizers.py, build-matrix checks

The central abstraction is the Microvm class in tests/framework/microvm.py. A test does not talk to curl and a socket directly — it asks the microvm_factory fixture for a Microvm, then drives it through Python methods that wrap the REST API:

Warning: Never hard-code paths, ports, or socket names in a test. The framework hands you a unique session root, a TAP per interface, and a fresh socket. Reusing a name across tests is the classic source of the flake you will fix in Lab 5.4.

Artifacts: Where Kernels and Rootfs Come From

An integration test needs a guest kernel and a root filesystem. The framework supplies them as fixtures so a test never downloads or builds them inline:

# How tests get a kernel + rootfs — read the fixtures, then the artifact loader.
rg -n "def guest_kernel|def rootfs|ALL_GUEST_KERNELS|artifact_dir" tests/conftest.py tests/framework/artifacts.py

Because guest_kernel and several config fixtures are parametrized, one def test_x(...) silently becomes many test cases — one per kernel, per io-engine, per snapshot type. That is deliberate breadth, and it is also a source of flakiness when a test makes an assumption that holds on one kernel but not another.

Required Reading

Read these before Lab 5.1. Confirm each exists on your checkout with the command shown; do not trust a path you have not ls'd.

# One command to confirm the whole reading list resolves on your branch:
ls tests/README.md tests/framework/microvm.py tests/conftest.py tests/framework/artifacts.py \
   tests/integration_tests/functional/test_metrics.py \
   tests/integration_tests/functional/test_drive_virtio.py CONTRIBUTING.md

Source Code Areas to Inspect

# Where the unit-testable logic clusters — run these to find #[cfg(test)] density.
rg -l "#\[cfg\(test\)\]" src/vmm/src | head -40
rg -c "#\[test\]" src/vmm/src/rate_limiter/mod.rs src/vmm/src/vmm_config/*.rs 2>/dev/null

Key Types and Fixtures Quick Reference

Always locate these with the command shown — paths drift between branches.

GitHub Issue Categories for Level 5

These are issue classes a Level 5 graduate can credibly take — see Issue Roadmap Stage 1 and Stage 9.

Deliverables

You must demonstrate all of the following before advancing to Level 6:

• You can run a single Rust unit test and a single pytest integration test by name, and read each failure (Lab 5.1).
• You traced exactly what one existing integration test does to a running firecracker — every API call, the boot, the SSH assertion, the teardown (Lab 5.1).
• A focused, table-driven Rust #[test] you wrote for a previously under-covered function, run with cargo test, that fails when you break the function (Lab 5.2).
• A new pytest that boots a microVM and asserts an end-to-end behavior (a second drive as /dev/vdb, or a NIC, or a metrics field), passing under tools/devtool test (Lab 5.3).
• A reproduced, diagnosed, and correctly fixed flaky test — made deterministic, with the race explained (Lab 5.4).
• From memory: the two-layer stack, how to run one test of each kind, and why time.sleep is not synchronization.

Common Mistakes

How to Verify Success

# 1. Run a single Rust unit test, scoped tight.
tools/devtool test -- --help   # see how the runner forwards args (then:)
# Inside the dev container you can also: cargo test -p vmm rate_limiter::tests::test_token_bucket_create

# 2. Run a single integration test by node id.
tools/devtool test -- integration_tests/functional/test_metrics.py -k metrics -v

# 3. Run only the functional suite, stop on first failure.
tools/devtool test -- integration_tests/functional/ -x

# 4. Reproduce a specific parametrized case and keep its artifacts.
tools/devtool test -- "integration_tests/functional/test_drive_virtio.py::test_partuuid_boot" -v

# 5. The style gate the maintainers run on every PR.
tools/devtool checkstyle

When you can scope a test of either kind, read its failure from the captured log/metrics, write a new test at the right altitude, and make a flaky one deterministic, you are ready for Level 6.

PR Profile: Level 5 Graduate

A Level 5 graduate can credibly open these pull requests:

A Level 5 graduate writes the test first, runs it red, makes it green, and runs the style gate before pushing — exactly the workflow the maintainers expect.

Next: Lab 5.1 — The pytest Integration Framework.