Lab 1.2: Run the Unit and Integration Tests

Background

Firecracker has two test surfaces, and you must understand both because they are different machines. The unit tests are ordinary Rust #[test] functions compiled and run by cargo test — they test individual functions and structs in isolation, in-process, with no real microVM. The integration suite is pytest, living in tests/, which builds the binary, starts actual Firecracker processes, boots actual microVMs, drives them over the API socket, and asserts on end-to-end behavior. The integration suite is the project's primary correctness signal, and new functionality requires integration tests — CONTRIBUTING.md is explicit about it.

Both are wrapped by tools/devtool test, which runs them inside the same Docker dev container as the build so the environment matches CI. On top of the tests sit the style and build gates — cargo fmt, clippy as warnings-as-errors, cargo sort, and the Python/Markdown formatters — which CI enforces and which a maintainer will bounce your PR for failing. Knowing how to run all of this locally, and how to scope a run to a single test so you are not waiting twenty minutes for one assertion, is a core contributor skill.

Why This Lab Matters for Contributors

  • A PR that fails CI wastes everyone's time. You must reproduce the full CI gate locally: tools/devtool test, checkstyle, checkbuild --all, clippy -D warnings.
  • You will spend real time reading test output — a pytest failure trace, a panicked unit test, a clippy lint. Learning to read it now pays off in every later level, especially the testing level where you write tests and fix flaky ones.
  • Knowing whether a behavior is covered by a unit test or an integration test tells you where to add coverage for a fix — a distinction the test framework deep dive builds on.

Prerequisites

  • Lab 1.1 complete — you have a green tools/devtool build and a working /dev/kvm. The integration tests boot real microVMs, so KVM access is mandatory here.
# Confirm the build and KVM before running anything:
ls -l /dev/kvm
tools/devtool build >/dev/null && echo "build OK"

Step-by-Step Tasks

Step 1: Understand the two test machines

            tools/devtool test
                    │
        ┌───────────┴────────────────────────────────┐
        ▼                                             ▼
  cargo test (unit)                          pytest  (tests/)
  ───────────────────                        ─────────────────────────────
  • #[test] fns inside src/                  • integration_tests/*.py
  • in-process, no microVM                   • starts real firecracker procs,
  • fast, deterministic                        boots real microVMs over the API
  • tests a fn/struct in isolation           • needs /dev/kvm, network, root-ish
  • lives next to the code it tests          • the PRIMARY correctness signal

Before running anything, see what's there:

ls tests/                              # the pytest tree
ls tests/integration_tests/           # grouped by area: api, functional, performance, security, ...
rg -l "#\[cfg\(test\)\]|#\[test\]" src/vmm/src | head   # where unit tests live (next to source)

Step 2: Run the unit tests

The unit tests are cargo test, but run them inside the container so the toolchain matches. Many unit tests still touch KVM, so keep /dev/kvm accessible.

# Run the whole workspace's unit tests via devtool's test entry point.
# On many branches the unit tests are exposed through a pytest wrapper; the most direct route is:
tools/devtool test -- integration_tests/build/test_unittests.py

If you want raw cargo test (fast inner loop, inside the container or with the pinned toolchain on your host):

# Inside the dev container / with the pinned toolchain installed:
cargo test --workspace --target $(uname -m)-unknown-linux-musl
# Scope to one crate to go faster:
cargo test -p vmm

Note: The exact filename of the pytest unit-test wrapper (test_unittests.py and its path) is version-sensitive — verify on your branch with find tests -name '*unittest*'. The concept is stable: unit tests = cargo test; the pytest suite shells out to it as one of its checks.

Step 3: Run a single unit test

You almost never want the whole suite while iterating. cargo test takes a substring filter:

# Run only tests whose name contains the substring (e.g. anything mentioning "rate_limiter"):
cargo test -p vmm rate_limiter
# Run one exact test and show its stdout even on success:
cargo test -p vmm <exact_test_name> -- --exact --nocapture
# List candidate test names without running them:
cargo test -p vmm -- --list | head

Pick any real test name from the list and run it once. Watch the running N tests / test result: ok lines — that format is what you will scan in every future run.

Step 4: Run the integration (pytest) suite

The integration tests are the heart of the suite. The full run is long and boots many microVMs; scope it. pytest's -k selects tests by substring expression, and you pass pytest args after the --:

# Run a single integration-test file:
tools/devtool test -- integration_tests/functional/test_api.py

# Run only tests whose name matches a -k expression (e.g. machine-config tests):
tools/devtool test -- -k "machine_config"

# Combine: one file, one matching test, verbose:
tools/devtool test -- integration_tests/functional/test_api.py -k "machine_config" -v

Find good targets to scope against:

ls tests/integration_tests/functional/        # test_api.py, test_drives.py, test_net.py, ...
rg -n "^def test_" tests/integration_tests/functional/test_api.py | head

Step 5: Read the test output

A passing pytest run ends with a green summary line; a failure gives you a traceback plus captured Firecracker logs. Learn the anatomy:

tests/integration_tests/functional/test_api.py::test_machine_config[...]  PASSED   [ 42%]
                                                                          ▲ outcome  ▲ progress

FAILED test_api.py::test_drives[vda] - AssertionError: expected 204, got 400
    ────────────── Captured stdout / fc log ──────────────
    ... the actual JSON the API returned, the firecracker log lines ...

When an integration test fails, the first thing you read is the captured Firecracker log, not the Python traceback — the traceback tells you which assertion tripped, but the FC log tells you what the VMM actually did. This is the same instinct you will use debugging real issues in Level 8.

For unit tests, a failure is a panic with a file:line and a left/right mismatch:

thread 'vstate::vcpu::tests::some_test' panicked at 'assertion failed: `(left == right)`
  left: `2`, right: `1`', src/vmm/src/vstate/vcpu/...

Step 6: Run the style and build gates

CI runs more than tests. Reproduce the gates a maintainer will check:

# Auto-format everything (cargo fmt + clippy --fix + cargo sort + black/isort/mdformat):
tools/devtool fmt

# The aggregate style check (what CI verifies — fmt, sort, license headers, etc.):
tools/devtool checkstyle

# Build every target/feature combination CI builds:
tools/devtool checkbuild --all

# Clippy as warnings-as-errors — this is exactly the CI invocation:
cargo clippy --all --all-targets --all-features -- -D warnings

Tip: CONTRIBUTING.md recommends wiring tools/devtool fmt and the checks as a git pre-commit/pre-push hook so you never push a formatting or clippy failure. Set that up now; it saves a CI round-trip on every PR. Find the suggested hook in CONTRIBUTING.md: rg -n -i "hook|pre-commit|pre-push" CONTRIBUTING.md.

Step 7: Know what CI actually runs

Your local gates should mirror CI. Read the CI definition so there are no surprises:

ls .github/workflows/ 2>/dev/null
rg -l "devtool|cargo test|clippy|checkstyle|pytest" .buildkite/ .github/ tools/ 2>/dev/null

Firecracker's CI runs the build, the unit tests, the pytest integration suite (across architectures), the style/clippy gates, and — for some changes — Kani formal-verification proofs (the Kani label). You do not need to run Kani in Level 1, but know it exists:

rg -l "kani" tests/ src/ 2>/dev/null | head

Implementation Requirements / Deliverables

  • The unit tests pass (cargo test / the pytest unit wrapper) on your checkout.
  • You ran a single unit test by name with --nocapture and read its output.
  • You ran a single integration test file and a -k-filtered subset, and both passed.
  • You read at least one failure (induce one on a throwaway branch if needed — change an assertion) and identified the file:line and the captured FC log.
  • tools/devtool checkstyle and cargo clippy ... -D warnings pass clean on an unmodified checkout.

Troubleshooting

Integration tests fail with "no /dev/kvm" or permission errors

The integration suite boots real microVMs and needs KVM.

ls -l /dev/kvm
groups | tr ' ' '\n' | grep -E 'kvm|docker'
sudo setfacl -m u:${USER}:rw /dev/kvm   # if group membership is awkward

devtool runs tests in a container with /dev/kvm passed through; if the device is missing on the host, the container cannot see it either.

A test is flaky (passes on rerun)

Some integration tests are timing- or network-sensitive. Before calling it flaky, rerun it in isolation to confirm it is non-deterministic and not failing because of something you changed:

tools/devtool test -- integration_tests/functional/test_net.py -k "the_test_name" --count=5  # if pytest-repeat is available

Flaky tests are a real category with their own GitHub label; in Level 1 you observe and report them, you fix them in Level 5, Lab 4. Do not paper over a flake by re-running until green and committing.

Tests need root / network setup

Networking tests create TAP devices and need elevated privileges or specific host setup. If a whole class of net/vsock tests fails identically, it is environment, not code — read the test's setup fixture (conftest.py) to see what it expects:

rg -n "tap|TAP|netns|CAP_NET" tests/ | head
find tests -name conftest.py

Clippy fails on code you didn't touch

You may be on a Rust channel newer than the pin (new lints). Build/lint inside devtool so the pinned toolchain is used, not your system Rust:

cat rust-toolchain.toml   # confirm the pinned channel
tools/devtool checkbuild --all

The pytest run is enormous and slow

You ran the whole suite. Always scope with a file path and/or -k. The full suite is for CI and pre-PR confidence, not the inner loop:

tools/devtool test -- integration_tests/functional/test_api.py -k "config" -v

Expected Output

$ cargo test -p vmm rate_limiter
   Compiling vmm ...
running 7 tests
test rate_limiter::tests::test_token_bucket_create ... ok
test rate_limiter::tests::test_rate_limiter_default ... ok
...
test result: ok. 7 passed; 0 failed; 0 ignored

$ tools/devtool test -- integration_tests/functional/test_api.py -k "machine_config" -v
tests/.../test_api.py::test_machine_config[...] PASSED                    [100%]
======================= 1 passed, N deselected in Ns =======================

$ tools/devtool checkstyle
[Firecracker devtool] checkstyle ... OK

Stretch Goals

  1. Map a unit test to its source. Pick one unit test you ran in Step 3, find the function it exercises, and read both:

    rg -n "fn the_function_under_test" src/vmm/src/

    State in one sentence what invariant the test guards.

  2. Map an integration test to an API endpoint. Open test_api.py, pick a test_ function, and identify which REST endpoint (/machine-config, /drives/{id}, …) it drives and what response it asserts. Cross-reference the API endpoint map.

  3. Reproduce a failure deterministically. On a throwaway branch, break one assertion in a unit test, run it, and practice reading the panic. Revert. The goal is fluency with the failure format before you ever face a real one.

  4. Time the gates. Run time tools/devtool checkbuild --all and time tools/devtool test -- <one file>. Knowing how long the gates take shapes how you structure a PR's local validation.

Validation / Self-check

You are done when you can answer these without notes:

  1. What is the difference between Firecracker's unit tests and its integration tests, and which one boots a real microVM?
  2. Why do the integration tests require /dev/kvm but the build does not?
  3. How do you run a single integration test by name, and a single unit test by name?
  4. When an integration test fails, what do you read first — the Python traceback or the captured FC log — and why?
  5. Name the four local gates that mirror CI, and which one is "warnings-as-errors."
  6. What does CONTRIBUTING.md require for new functionality, test-wise, and where would such a test live?

When you can answer all six, proceed to Lab 1.3 — Boot Your First microVM.