Step 6: Testing

Your Step 2 reproducer proved the bug exists; your Step 5 fix made it stop. That is necessary and nowhere near sufficient. The tests you ship in the PR have a different job from your throwaway repro: they are permanent regression protection that has to encode every trigger condition, survive other people's machines on both architectures, and convince two maintainers that the fix is correct and that nothing around it broke.

The rule that governs this step, the same one Firecracker reviewers apply: a test that would have passed before your fix is not a test of your fix. Every test you add must be red on main without your change and green with it. If you cannot make it red on main, you have not written a test of your bug — you have written a test of something else.

Goal

A test (integration in tests/, unit in a #[cfg(test)] module, or both) that is red before your fix, green after, encodes every trigger condition from Steps 2 and 4, and runs deterministically under tools/devtool test on the CI matrix. Plus all three gates green on the branch: tools/devtool checkstyle, tools/devtool checkbuild --all, and coverage that does not drop.

CONTRIBUTING is explicit and non-negotiable: new functionality needs integration tests, and unit-test coverage must not decrease — it is expected to increase. Treat both as hard requirements, not aspirations.

Firecracker's Two Test Layers

Firecracker has exactly two layers you write into, and you must pick the right one for what the bug actually needs. Cheaper (unit) is better; reach for the pytest integration harness when a unit test genuinely cannot exercise the path.

Note: Firecracker's primary integration harness is pytest in tests/ — not raw cargo test. The Python suite spins up actual firecracker processes, drives the Unix socket, and asserts on real microVM behavior. The framework lives in tests/framework/ (confirm: ls tests/framework/; you will see microvm.py, http_api.py, artifacts.py, defs.py). cargo test covers the in-process Rust units beneath it.

Find which functional test file your bug belongs near — there is almost always an existing one for your subsystem:

# Don't guess the path; list the real test files and pick the matching subsystem.
ls tests/integration_tests/functional/        # test_api.py, test_net.py, test_metrics.py, test_serial_io.py, ...
ls tests/integration_tests/security/          # test_seccomp.py, test_jail.py, test_vulnerabilities.py, ...
rg -n "def test_" tests/integration_tests/functional/test_api.py | head

Add your test to the file that already owns the subsystem (an API-validation bug → test_api.py; a metrics bug → test_metrics.py; a seccomp regression → security/test_seccomp.py). A new file is justified only for a genuinely new area.

The Unit Test (when the logic is unit-reachable)

If the fix site from Step 4 is a pure function or a method on a struct you can construct in a test — an API validator, an arithmetic computation, a parser — the primary regression guard is a Rust unit test right next to the code. It is fast, it pins the exact code path, and it runs on every cargo test.

Find the existing test module in the file you changed and extend it rather than inventing a parallel structure:

# Locate the in-file test module you will add to.
rg -n "#\[cfg\(test\)\]|mod tests|#\[test\]" src/vmm/src/resources.rs

Cover the trigger condition explicitly, and add a negative control — the nearest valid case that must still behave as before. The negative control is what proves your fix is scoped, not a blanket behavior change:

#![allow(unused)]
fn main() {
#[cfg(test)]
mod tests {
    use super::*;

    // THE FIX: vcpu_count == 0 must be rejected at the config boundary.
    #[test]
    fn test_machine_config_rejects_zero_vcpus() {
        let mut resources = VmResources::default();
        let cfg = MachineConfigUpdate { vcpu_count: Some(0), ..Default::default() };
        let err = resources.update_machine_config(&cfg).unwrap_err(); // FAILS on main (was Ok)
        assert!(matches!(err, MachineConfigError::InvalidVcpuCount));
    }

    // NEGATIVE CONTROL: a valid count must still succeed — the fix is scoped.
    #[test]
    fn test_machine_config_accepts_one_vcpu() {
        let mut resources = VmResources::default();
        let cfg = MachineConfigUpdate { vcpu_count: Some(1), ..Default::default() };
        resources.update_machine_config(&cfg).unwrap(); // unchanged by the fix
    }
}
}

Tip: Prove the test is a test of your fix by stashing the production change and re-running. git stash -- src/vmm/src/resources.rs && cargo test test_machine_config_rejects_zero_vcpus must fail; git stash pop and it must pass. If it passes both ways, it is not testing your bug.

For a guest-reachable (virtio) bug, the unit test must treat the descriptor fields as hostile and assert that a malformed input is rejected, not just that the one reported value works. The reported value is one input; the attacker will try others. Construct an adversarial descriptor (oversized length, out-of-bounds address, bad index) and assert the handler returns an error instead of panicking or reading out of bounds — that is the real regression guard for the untrusted data plane.

Run only your new units while iterating:

tools/devtool test -- ... # (the pytest path; below)
# Rust units, fast loop:
cargo test -p vmm test_machine_config_rejects_zero_vcpus test_machine_config_accepts_one_vcpu

The Integration Test (the required pytest)

CONTRIBUTING requires integration tests for new functionality, and many bugs only manifest through a real booting microVM. The integration test drives the actual binary over the actual API socket and asserts on real behavior. This is the test a maintainer trusts most, because it exercises the same surface an operator does.

The framework gives you a microvm fixture (or builder) that handles the jailer, the socket, the artifacts, and teardown. Read two nearby tests in the file you are extending and imitate their shape — do not invent your own setup:

# tests/integration_tests/functional/test_api.py  (extend the existing file)

def test_negative_machine_config_zero_vcpus(uvm_plain):
    """vcpu_count == 0 must be rejected at config time with a clear 4xx,
    not fail later at InstanceStart. Regression test for #NNNN."""
    test_microvm = uvm_plain
    test_microvm.spawn()
    test_microvm.basic_config()  # sane defaults

    # THE FIX: the API rejects the bad value up front.
    response = test_microvm.api.machine_config.put(vcpu_count=0, mem_size_mib=1024)
    # red on main (request was accepted, failure surfaced later); green with the fix:
    assert response.status_code == 400
    assert "vcpu" in response.json()["fault_message"].lower()


def test_machine_config_one_vcpu_still_ok(uvm_plain):
    """NEGATIVE CONTROL: a valid vcpu_count is unaffected by the fix."""
    test_microvm = uvm_plain
    test_microvm.spawn()
    test_microvm.basic_config(vcpu_count=1)
    test_microvm.start()                      # boots to InstanceStart as before
    assert test_microvm.state == "Running"

Warning: The exact fixture names, the API client surface (test_microvm.api.<resource>.put/patch/get), and the assertion helpers drift between branches. Do not memorize them — read the file you are editing. Run rg -n "def test_|\.api\.|basic_config|\.spawn\(\)" tests/integration_tests/functional/test_api.py and copy the current idiom of the two tests nearest yours.

For bugs whose trigger is a device, snapshot, or boot behavior rather than an API status code, assert on the observable that Step 2 used to detect the bug:

Run just your integration test through devtool (it builds the binary, sets up the container, and invokes pytest):

# -- forwards args to pytest; -k selects by name, -s shows output while iterating.
tools/devtool test -- integration_tests/functional/test_api.py -k zero_vcpus -s

Encode Every Trigger Condition

Go back to your Step 2 repro and Step 4 root cause and list the conditions under which the bug fires. Each one becomes an assertion or a parameterization — not a comment, an assertion. If the bug only triggers with mem_size_mib above a threshold, or only with a diff snapshot, or only on a particular io_engine, encode that exact condition. pytest's @pytest.mark.parametrize lets you sweep the boundary cheaply:

@pytest.mark.parametrize("vcpu_count", [0])           # the failing input(s)
def test_negative_machine_config_invalid_vcpus(uvm_plain, vcpu_count):
    ...

A test that hits the bug only on the one value you happened to report is weaker than one that asserts the class of bad input is rejected (the boundary and just past it). For a guest data-plane fix, parametrize across several malformed descriptor shapes — that is the difference between "fixed the reported case" and "hardened the path."

Snapshot, Seccomp, and Performance: Special Cases

Three Firecracker test concerns recur and have dedicated homes:

• Snapshot compatibility. If your fix touched any device's Persist state or the serialized layout (Step 4/5 flagged this), you owe a snapshot round-trip test: create on this build, restore, resume, assert the guest is intact. A fix that changes what a snapshot serializes without a version story and a compat test will be blocked. See snapshotting and Stage 8.
• Seccomp / attack surface. If you touched a syscall surface or a seccomp filter, extend tests/integration_tests/security/test_seccomp.py / test_jail.py so the allowed/denied set is asserted, not assumed. A widened filter that no test pins is a review red flag.
• Performance. If the bug or fix is on a hot path (the run loop, the virtio fast path), the performance suite lives in tests/integration_tests/performance/, and CI runs A/B comparisons. You usually do not add a perf test for a correctness fix, but you must be able to show the fast path is unchanged in the common case — Step 7 covers the measurement.

Determinism: the Non-Negotiable

A test that passes on your machine and flakes on a CI runner is worse than no test — it becomes a flaky-test issue with your name on it. Enforce determinism:

• No sleep() to wait for state. Poll the API / a log line / a metric with a bounded retry; the framework provides helpers (rg -n "retry|wait_for|poll" tests/framework/). Wait on a condition, never a wall-clock duration.
• No order-dependent assertions over a dict/set whose iteration order is unspecified.
• Clean teardown. The microvm fixture handles process/socket/jailer cleanup; don't leak a firecracker process that poisons the next test.
• Both architectures. CI runs x86_64 and aarch64. If your fix is arch- specific (an arch/x86_64/ vs arch/aarch64/ path), mark/skip appropriately and verify the other arch still passes — don't assume.

Prove determinism by hammering the test before you trust it:

# Re-run many times; any single failure means it's not done.
tools/devtool test -- integration_tests/functional/test_api.py -k zero_vcpus \
  --count=20 || echo "FLAKE — not done"   # pytest-repeat if available; else loop in a shell

Run the Full Gate Suite

Before you call Step 6 done, run the three gates CONTRIBUTING and CI enforce, in this order, on the branch:

tools/devtool checkstyle      # cargo fmt + clippy + sorting + python/markdown style
tools/devtool checkbuild --all  # builds all targets/features; clippy is -D warnings here
tools/devtool test            # the full integration suite (or a scoped -- subset while iterating)

checkbuild --all and CI run clippy as warnings-as-errors (cargo clippy --all --all-targets --all-features -- -D warnings). A clippy warning on your new test or fix is a free review comment — fix it now. For coverage, the project tracks it (CI has a coverage pipeline); your unit test should raise line coverage on the path you fixed, never lower it. If you only added an integration test for logic that is also unit-reachable, add the unit test too — integration tests don't always count toward the unit-coverage gate.

Deliverable for Step 6

• A test at the lowest viable layer (unit if the logic is in-process, integration if it needs a real microVM) that is red on main, green with the fix — verified by stashing the fix.
• A negative control proving the fix is scoped (the nearest valid case still behaves as before).
• Every trigger condition from Steps 2/4 encoded as an assertion or a parametrization; for guest-reachable bugs, multiple adversarial inputs.
• The integration test required by CONTRIBUTING, added to the existing subsystem test file, using that file's current fixture idiom.
• Snapshot/seccomp/perf coverage iff the fix touches those surfaces (or an explicit, written "not touched" note for Step 7).
• Determinism proven: no sleep-waits, clean teardown, re-run without a flake, both architectures considered.
• tools/devtool checkstyle, checkbuild --all, and test green on the branch; coverage not reduced.

Rubric Hooks

This is the Tests dimension (16 pts): integration red-before/green-after, a unit test where the logic is unit-reachable, a negative control, every trigger encoded, and determinism. A red-before/green-after integration test plus a scoped unit test with a negative control scores high; a test that would have passed on main, a sleep-based wait, or no unit test for unit-reachable logic scores low. Coverage discipline and the required integration test also feed PR craft. See the evaluation rubric.

Validation / Self-check

Before advancing to Step 7:

1. Every new test fails on main and passes with your fix — you verified by stashing the production change and re-running, not by reasoning about it.
2. You included a negative control that would catch an over-broad fix.
3. You chose the lowest viable layer; you did not write a pytest integration test for logic a unit test reaches, nor a unit test for behavior that only a real microVM exhibits.
4. Every Step 2/4 trigger condition is encoded as an assertion; a guest-reachable fix is tested against multiple adversarial inputs.
5. No sleep-based waits, no order-dependent assertions, clean teardown; you ran it repeatedly without a flake and considered both architectures.
6. checkstyle, checkbuild --all, and test are green on the branch, clippy is warning-free, and coverage did not drop.

Then go to Step 7: Validation.