Stage 9 — Flaky Test Fixes

What class of issue this is

Stage 9 is de-flaking: taking a test that fails nondeterministically — green most runs, red occasionally on the same code — and making it deterministic, without weakening what it checks. In Firecracker the flakiness almost always lives in the pytest integration suite under tests/, because those tests boot real microVMs, drive a real API socket, time real boots and real I/O, and talk to real host resources (TAP devices, sockets, the dev kernel). A race between the test and the microVM's lifecycle, a hard-coded timeout that's too tight on a loaded CI host, or a resource leaked from a previous test all produce flakes. Unit-level (cargo test) flakes happen too, usually from a timing assumption or a non-deterministic ordering.

Concretely, a Stage 9 PR is one of:

• Replacing a fixed sleep/timeout with a poll-until-condition (wait for the actual state, not a guessed duration).
• Fixing a race between the test and the microVM (asserting before the guest reached the expected state, reading metrics before a flush, closing a socket the VMM still uses).
• Fixing resource isolation/leakage between tests (a leftover TAP interface, an un-cleaned API socket, a port collision) that makes a test fail depending on what ran before it.
• Fixing a timing-sensitive assertion (a boot-time or throughput bound that is occasionally exceeded on a busy host) — tightening how it's measured, not loosening the bound to hide the flake.

Why it's at this difficulty

De-flaking demands the opposite skill from feature work: you must reproduce something that happens 1-in-50 runs, find the timing assumption behind it, and prove your fix by running the test hundreds of times without a failure — all while keeping the assertion meaningful. It needs the test-harness fluency of Level 5 (how microvm_factory, fixtures, and the pytest helpers work) but not deep subsystem knowledge, which is why many contributors interleave Stage 9 work with every other stage. Pairs with the event manager deep dive when the race is in the VMM's epoll loop rather than the test.

What you must already understand

• The pytest harness. How a test builds and boots a microVM, and what helpers wait on state:

ls tests/ tests/integration_tests/
rg -n "def test_|microvm_factory|@pytest.fixture|wait_for|poll|assert_eq" tests/ | head -30
rg -n "def serial|def ssh|def get_metrics|def flush_metrics|def wait" tests/framework/ 2>/dev/null | head

• How to run one test repeatedly and surface a flake. pytest with the right flags / a loop:

# Run a single test many times to reproduce a flake:
tools/devtool test -- -k test_name --count 200          # if pytest-repeat is available; verify
# Or loop it and stop on first failure:
tools/devtool test -- -k test_name -x

• Determinism levers. Randomized unit tests print a seed; reproduce with it. Find how randomness is seeded on your branch:

rg -n "seed|rand::|StdRng|from_seed|proptest|RUST_TEST" src/ tests/ | head

• The known-flaky markers. Firecracker may skip/quarantine flakes with a pytest marker or an issue link — find the convention so you can un-mark them when fixed:

rg -n "skip|xfail|flaky|@pytest.mark|TODO.*flak" tests/ | head

Representative tasks

How to approach one — worked example: a fixed-sleep race

Illustrative of the pattern. Run the grep to find a real candidate on your branch.

Symptom: an integration test boots a microVM, time.sleep(2), then asserts the guest reached a state (a file exists, a service is up, a metric incremented). On a busy CI host the boot takes longer than two seconds occasionally, so the assertion fires too early and the test is red.

Step 1 — reproduce the flake

rg -n "time.sleep|sleep\(" tests/integration_tests/ | head
tools/devtool test -- -k test_boot_thing -x      # run until it fails
# If it won't fail locally, add artificial load or run with --count high.

Step 2 — find what the sleep was waiting for and poll that instead

The fix is never "make the sleep longer" — it is to wait on the actual observable condition with a bounded timeout, so the test is fast when the host is fast and patient when it's slow:

--- a/tests/integration_tests/functional/test_boot_thing.py
+++ b/tests/integration_tests/functional/test_boot_thing.py
@@
-    vm.start()
-    time.sleep(2)                       # racy: boot may not be done
-    assert vm.ssh.run("test -f /ready").returncode == 0
+    vm.start()
+    # Poll the real readiness signal with a generous bound instead of guessing a duration.
+    def _ready():
+        return vm.ssh.run("test -f /ready").returncode == 0
+    wait_for(_ready, timeout_s=30, poll_interval_s=0.2)

Use the harness's existing wait helper (rg -n "def wait_for|retry|@retry" tests/framework/) rather than rolling your own loop. If the condition is a metric, flush first:

vm.flush_metrics()                 # PUT /actions {"action_type":"FlushMetrics"}
metrics = vm.get_all_metrics()
assert metrics["block"]["read_count"] > 0

Step 3 — prove the fix with volume, and un-quarantine

A de-flake is only credible if you ran it enough to be confident. State the run count in the PR:

tools/devtool test -- -k test_boot_thing --count 300   # 0 failures

If the test was skipped/quarantined behind a marker or a tracking issue, remove the marker in the same PR and reference the issue you are closing:

rg -n "test_boot_thing" tests/   # find the skip/xfail marker to delete

Tip: A flake that you "fixed" but can only run 20 times is not fixed. The whole point of the stage is determinism — reviewers want a number. If you genuinely cannot reproduce it, say so in the PR and add the diagnostic instrumentation (a log/metric/dump on failure) that will catch it next time, rather than blindly widening a timeout.

A second pattern — a seeded unit-test flake

Illustrative.

Symptom: a randomized cargo test occasionally fails. The fix is to reproduce with the printed seed, find the assumption the random input breaks (an ordering, an off-by-one at a boundary the seed happened to hit), and fix the code or the assertion — never delete the randomness, which is what found the bug.

rg -n "seed|StdRng::seed_from_u64|proptest!" src/ | head
# reproduce with the failing seed the test framework printed, then fix the real assumption.

A seeded flake is often a genuine bug in disguise (a real boundary case), so treat it like a Stage 3+ fix once you find the cause — it may merit its own behaviour PR.

What a good PR looks like

• The assertion is unchanged in strength. You made the test deterministic, not lenient. A PR that loosens a bound to hide a flake will be bounced.
• No fixed sleeps where a poll-until-condition belongs; the test is fast on a fast host.
• A run count in the description. "Ran 300×, 0 failures" is the evidence reviewers want.
• Quarantine markers removed in the same PR, closing the tracking issue.
• A seeded flake is reproduced and root-caused, not papered over — and if it's a real bug, split out a behaviour fix.
• CHANGELOG usually not needed (test-only); gates green.

Graduation criteria — ready to move on when

• You have de-flaked at least three tests (un-quarantined and fixed), each with a stated high-volume run proving determinism.
• You can reproduce a randomized failure from its seed and a timing flake under artificial load.
• You reach for a poll-until-condition by reflex and treat a fixed sleep in a test as a smell.
• You can tell a "real bug surfaced as a flake" from a genuine test-harness race, and route each to the right kind of PR.

Determinism is the precondition for the next stage: you cannot measure performance on a flaky baseline. Stage 10 uses the stable harness you now trust to make measured, CI-guarded performance improvements.

Next: Stage 10 — Performance Improvements.