Lab 5.1: The pytest Integration Framework

Background

Firecracker's integration suite is the half of the test stack that actually boots microVMs. It is a pytest project under tests/, but the part that makes it special is the framework package (tests/framework/) and the fixtures (tests/conftest.py) that manufacture a real, jailed firecracker process, hand it a kernel and rootfs, configure it over the REST API, boot it, and let you SSH into the guest to assert. When CONTRIBUTING says "new functionality requires integration tests," it means a test in this suite.

This is a trace-it lab. You will not write production code. You will read the framework, run an existing integration test, and trace — concretely, with commands — exactly what that test does to a running firecracker: which process it spawns, which API endpoints it hits in which order, when the guest boots, how it reaches into the guest, and how everything is torn down. By the end you can open any test in integration_tests/functional/ and know what it touches without guessing.

Note: You need a working dev container and the ability to boot a microVM (Level 1). The integration suite runs inside tools/devtool's container, which already has the test kernels, rootfs artifacts, Python deps, and KVM access wired up.

Why This Lab Matters for Contributors

• Every feature PR you ever open will add or change a test in this suite. You must read it fluently.
• The Microvm helper is the API you will use in Lab 5.3 to write your own integration test — you cannot write one until you have read how one works.
• Knowing the spawn → configure → boot → assert → teardown lifecycle is exactly what you need to debug a flaky test in Lab 5.4.
• This is the practical counterpart to the threading-model deep dive and the API server deep dive: you watch the control plane drive a real process.

Prerequisites

• Completed Level 1 (build + boot a microVM) and Level 3 (the API → VMM action path).
• A checkout of firecracker-microvm/firecracker and a working dev container.

# Verify you can run the suite at all (this prints the collected tests, runs nothing heavy).
tools/devtool test -- integration_tests/functional/test_api.py --collect-only -q | head -20

If that lists test node ids, you are ready.

Step 1: Map the framework package

Do not read top-to-bottom. Find the load-bearing pieces by role.

# The whole framework package, by file.
ls -1 tests/framework/

# THE central class — the helper every test drives.
rg -n "class Microvm" tests/framework/microvm.py

# Its lifecycle methods (locate, don't memorize line numbers).
rg -n "def spawn|def basic_config|def add_drive|def add_net_iface|def start\b|def kill|def make_snapshot" \
   tests/framework/microvm.py

# The api wrapper and ssh/serial access points.
rg -n "self.api|def ssh\b|def ssh_iface|def serial_input|def flush_metrics" tests/framework/microvm.py

Expected: class Microvm resolves in tests/framework/microvm.py, and the method names above all appear. Note what spawn() does (launch the process + open the socket) versus start() (boot the guest) versus basic_config() (one shot to set boot-source + root drive + machine-config).

Now the supporting modules:

rg -n "class MicroVMFactory|class JailerContext|class Artifact|def ssh" tests/framework/*.py

Tip: Microvm.spawn() and Microvm.start() are different verbs. spawn() starts the firecracker process (an empty VMM listening on its socket). start() issues PUT /actions {InstanceStart} and boots the guest. A test that calls spawn() but never start() exercises only pre-boot API behavior.

Step 2: Read the fixtures that feed a test

A test function's parameters are fixtures. The ones that matter most:

# The lifecycle + artifact fixtures, with one-line context.
rg -n "@pytest.fixture" -A2 tests/conftest.py | rg -n "def microvm_factory|def guest_kernel|def rootfs|def uvm|def io_engine|def vcpu_count|def mem_size_mib|def pci_enabled" -A1

The key realization: because guest_kernel (and io_engine, pci_enabled, …) are parametrized, a single def test_x(...) becomes many cases. Prove it:

# How many cases does one test expand to? Count the node ids.
tools/devtool test -- integration_tests/functional/test_metrics.py --collect-only -q | rg -c "::"

A number greater than the count of def test_ functions means parametrization is multiplying your test. That breadth is the point — and the source of "passes on kernel A, fails on kernel B" flakes.

Step 3: Read one real test end-to-end

Open a small functional test and read the lifecycle literally.

# A representative test: spawn → configure → start → ssh → assert.
sed -n '1,90p' tests/integration_tests/functional/test_metrics.py

You will see the canonical shape (your branch's exact code may differ — verify):

def test_net_metrics(uvm):
    test_microvm = uvm
    test_microvm.spawn()          # launch the firecracker process; open the API socket
    test_microvm.basic_config()   # PUT /boot-source + PUT /drives/rootfs + PUT /machine-config
    test_microvm.add_net_iface()  # create a TAP, PUT /network-interfaces/eth0
    test_microvm.start()          # PUT /actions {InstanceStart} → the guest boots

    # reach into the booted guest over SSH and do work that moves a counter
    test_microvm.ssh.run("sync")

    metrics = test_microvm.flush_metrics()   # PUT /actions {FlushMetrics}; parse the JSON
    # assert on a field, e.g. that net RX/TX counters advanced

Map each line to the fact sheet:

Warning: basic_config() hides three API calls. When a test fails "during config," run with -v and read the framework log to see which PUT returned non-2xx. Do not assume it was the one you changed.

Step 4: Watch the test drive a real process

Trace the actual side effects. Run one test verbosely and keep its artifacts, then inspect what the framework created.

# Run a single test; -v shows each step, -s lets prints through.
tools/devtool test -- integration_tests/functional/test_metrics.py -k net_metrics -v -s

# The framework gives each test a unique session/results dir; find recent firecracker artifacts.
# (Inside the container; paths under /srv or a tmp root — verify on your branch.)
rg -n "results_dir|test_fc_session_root_path|chroot|jail" tests/conftest.py tests/framework/microvm.py | head

While a longer test runs, in a second shell you can see the real process and its socket:

# The actual firecracker process the test spawned.
pgrep -af firecracker

# Its API Unix socket (the test talks to THIS).
find /srv /tmp -name "*.socket" 2>/dev/null | head

# Poll the instance state directly — the same thing the test's `api` object does.
SOCK=$(find /srv /tmp -name "*.socket" 2>/dev/null | head -1)
curl -s --unix-socket "$SOCK" http://localhost/   # GET / → {"state":"Running",...}

This is the whole point of the lab: a "test" is a Python program that spawns the same firecracker binary you ran by hand in Level 1, sends it the same JSON over the same kind of socket, and asserts on the result. There is no magic.

Step 5: Trace teardown and isolation

The reason tests don't clobber each other is isolation in the fixtures. Find it:

# How a microVM is killed and its resources reclaimed.
rg -n "def kill|reap|cleanup|shutdown|__del__|atexit" tests/framework/microvm.py
rg -n "def reap_orphans|netns_factory|def microvm_factory" tests/conftest.py

Note three isolation mechanisms:

1. Unique paths — each Microvm gets its own socket, chroot/jail dir, and results dir. No two tests share a socket name.
2. Per-test network namespaces — netns_factory hands out (and reuses) network namespaces so a TAP/IP in one test cannot collide with another.
3. Reaping — reap_orphans / the factory's teardown kills the firecracker process even when an assertion throws, so a failed test does not leak a running VMM.

sequenceDiagram
    participant T as Test (pytest)
    participant F as microvm_factory
    participant M as Microvm helper
    participant FC as firecracker process
    participant G as Guest
    T->>F: request a Microvm (with guest_kernel, rootfs)
    F->>M: build (unique sock/jail/netns)
    T->>M: spawn()
    M->>FC: exec firecracker --api-sock …
    T->>M: basic_config(), add_net_iface()
    M->>FC: PUT /boot-source, /drives, /machine-config, /network-interfaces
    T->>M: start()
    M->>FC: PUT /actions {InstanceStart}
    FC->>G: boot kernel → login over serial/network
    T->>M: ssh.run("...") / flush_metrics()
    M->>G: SSH command / FC: PUT /actions {FlushMetrics}
    G-->>T: result → assert
    Note over F,FC: teardown (even on failure): kill() reaps the process, frees netns

Step 6: Run, scope, and read a failure

Practice the three things you'll do constantly: scope to one test, make it fail on purpose, read the failure.

# Scope to one node id.
tools/devtool test -- "integration_tests/functional/test_api.py::test_api_machine_config" -v

# A glob across a file.
tools/devtool test -- integration_tests/functional/test_api.py -k "machine_config" -v

# Run the whole functional dir but stop on first failure (fast feedback).
tools/devtool test -- integration_tests/functional/ -x

To see what a real failure looks like, temporarily make an assertion impossible in a copy (never commit this):

# Read where assertions live, then break one locally to study the failure output.
rg -n "assert " tests/integration_tests/functional/test_api.py | head

When it fails, read in this order: (1) the assertion line and values, (2) the captured Firecracker log and metrics the framework attaches, (3) the serial output if the guest never came up. The framework saves these to the test's results dir — find it:

rg -n "results_dir|log_file|metrics_file|console|serial" tests/framework/microvm.py | head

Deliverables

• You located class Microvm and named what spawn(), basic_config(), start(), ssh, and flush_metrics() each do.
• You explained why one def test_* expands into multiple node ids (parametrized guest_kernel and config fixtures).
• You traced one real functional test line-by-line to the API endpoints and KVM mechanisms it drives.
• While a test ran, you found the live firecracker process and its socket, and queried GET / yourself.
• You named the three isolation mechanisms (unique paths, per-test netns, reaping on failure).
• You scoped a single test, ran it, and read its failure from the captured log/metrics/serial.

Troubleshooting

tools/devtool test can't find the container or fails to build

Run tools/devtool build first; the test runner needs a built binary and the dev image. Check docker ps and that /dev/kvm is accessible from inside the container.

A test errors in spawn() with "address already in use" / socket exists

A previous run leaked a process or socket. pgrep -af firecracker and kill stragglers; remove stale sockets under the session root. This is exactly the isolation failure mode Lab 5.4 addresses.

ssh.run(...) times out

The guest didn't get a working NIC, or it isn't booted. Confirm add_net_iface() ran before start(), then read the serial output to see whether the kernel reached a login prompt.

Collection works but every case is skipped

Check markers and your arch — some tests are gated by @pytest.mark.skipif on x86_64/aarch64 or on a feature flag. rg -n "@pytest.mark.skip" tests/integration_tests/functional/<file>.

Expected Output

$ tools/devtool test -- integration_tests/functional/test_metrics.py -k net_metrics -v
...
integration_tests/functional/test_metrics.py::test_net_metrics[vmlinux-6.1.x] PASSED
integration_tests/functional/test_metrics.py::test_net_metrics[vmlinux-5.10.x] PASSED
...
1 passed (×N kernels), 0 failed

(Exact kernel tags and counts depend on ALL_GUEST_KERNELS on your branch.)

Stretch Goals

1. Open test_drive_virtio.py and list every API endpoint the file touches (rg -n "api\.|add_drive|basic_config" tests/integration_tests/functional/test_drive_virtio.py). This is your reconnaissance for Lab 5.3.
2. Find a test that uses make_snapshot() and trace the pause → snapshot → resume calls back to the snapshotting deep dive.
3. Find how the framework asserts on a Firecracker log line (not just an API result) and write down the helper name.
4. Run one test with -Dpci / --pci-style parametrization (rg -n "pci_enabled" tests/) and confirm it runs both transports.

Validation / Self-check

Answer without notes; these gate completion:

1. What is the difference between Microvm.spawn() and Microvm.start(), and which one boots the guest?
2. Which three REST endpoints does basic_config() hide behind one call?
3. Why does one def test_* produce several pytest node ids, and what fixture is usually responsible?
4. Name the three isolation mechanisms that stop two integration tests from clobbering each other.
5. When a test fails after start(), what three captured artifacts do you read, and in what order?
6. How would you run just one parametrized case of one test, verbosely?
7. Where does the Microvm helper actually send the JSON you configure — what is on the other end of self.api?

Next: Lab 5.2 — Add a Missing Unit Test, where you drop down to the Rust layer and prove a function's contract in milliseconds.