Lab 5.4: Fix It — A Flaky Test

Background

A flaky test passes sometimes and fails sometimes with no code change. It is worse than a missing test: it trains everyone on the team to ignore red CI ("just re-run it"), which is exactly how a real regression slips through. Firecracker's integration suite is especially exposed to flakiness because every test boots a real microVM — real processes, real kernels, real networking, real timing — and runs across a parametrized matrix of guest kernels and config knobs on shared CI.

This is a fix-it lab. You will learn the discipline: reproduce the flake intermittently, find the assumption that is sometimes false (a race, a timing assumption, an ordering dependency, a resource clash), and make the test deterministic — by waiting on the real signal instead of sleeping, by isolating shared state, and by removing the assumption rather than papering over it. The unforgivable "fix" is adding a longer time.sleep or muting the test; you will do neither.

Note: The cardinal rule of flaky tests, valid in every test framework: time.sleep is not synchronization. A sleep that's "long enough" on your laptop is too short on a loaded CI box and wastefully slow everywhere. Replace sleeps with a poll on the actual condition.

Why This Lab Matters for Contributors

• "Fix flaky test_X" is a real, recurring issue class (see Issue Roadmap Stage 9) and a high-trust contribution: it directly improves everyone's CI signal.
• Diagnosing a flake forces you to understand the microVM lifecycle timing you studied in Lab 5.1 — when is the guest actually booted, when has an async API action actually completed.
• The skills (poll-don't-sleep, isolate shared resources, reproduce-under-stress) transfer directly to diagnosing production races, which is what the debugging masterclass builds on.

Prerequisites

• Completed Lab 5.1 and Lab 5.3.
• You can run a single integration test repeatedly under tools/devtool test.

# You can re-run the same test many times (the core flake-hunting move).
tools/devtool test -- integration_tests/functional/test_api.py -k machine_config --count=10 2>/dev/null \
  || tools/devtool test -- integration_tests/functional/test_api.py -k machine_config -v

(If pytest-repeat's --count isn't available on your branch, loop the command in a shell — see Step 2.)

Step 1: Recognize the four flake archetypes

Almost every flaky integration test is one of these. Learn to name them on sight.

flowchart TD
    F[Test is flaky] --> R{Reproduce 1-in-N?}
    R -- only alone vs in-suite --> O[Ordering dependency]
    R -- only on a variant --> P[Parametrization assumption]
    R -- under concurrency/load --> C{Error text?}
    C -- in use / collision --> RC[Resource clash]
    C -- expected X, none yet --> T[Timing / premature assertion]
    O --> FixO[Isolate setup/teardown]
    P --> FixP[Fix per-variant or reasoned skip]
    RC --> FixR[Unique names + netns + reap]
    T --> FixT[Poll the real signal, no sleep]

Step 2: Reproduce it intermittently

You cannot fix what you cannot reproduce. The whole game is to make a 1-in-50 failure happen on demand. Three levers: repeat, parallelism/load, and the failing variant.

# Lever 1: repeat the same test many times.
for i in $(seq 1 30); do
  tools/devtool test -- "integration_tests/functional/test_api.py::test_api_machine_config" -q \
    || { echo "FAILED on iteration $i"; break; }
done

# Lever 2: add CPU load so timing-sensitive tests fail faster (simulate a busy CI box).
#   In another shell:  stress-ng --cpu $(nproc) --timeout 120s   # if available
#   then re-run the loop above.

# Lever 3: pin the exact failing parametrization (read the node id from a red CI run).
tools/devtool test -- "integration_tests/functional/test_foo.py::test_bar[vmlinux-5.10.x-Async]" -v

Tip: When you finally catch a failure, save everything — the captured Firecracker log, serial output, and metrics from the test's results dir. A flake's evidence is gone on the next (passing) run. rg -n "results_dir|log_file|console|metrics" tests/framework/microvm.py to find where they land.

Step 3: Walk a representative flaky scenario

Here is a concrete, instructive example you'll recognize in the wild: asserting on a metric immediately after triggering guest I/O.

# FLAKY — do not imitate.
def test_block_metrics_flaky(uvm_plain):
    vm = uvm_plain
    vm.spawn(); vm.basic_config(); vm.add_net_iface(); vm.start()

    vm.ssh.run("dd if=/dev/vda of=/dev/null bs=1M count=16")  # generate block reads
    import time
    time.sleep(1)                                             # <-- the lie
    metrics = vm.flush_metrics()
    assert metrics["block"]["read_count"] > 0                 # sometimes 0

Why it's flaky: the guest's dd and the VMM's metric accounting are asynchronous with respect to the test thread. time.sleep(1) usually covers the gap, but on a loaded host the read completion, the metric increment, and the FlushMetrics flush can race. One second is sometimes not enough, and it's always wasted when it is.

The fix is to poll the real condition until it's true or a timeout fires — the integration-suite analog of an assertBusy. Firecracker's framework provides a wait/retry helper; find it:

# A poll-until-true helper (name varies: wait_for, retry, eventually, assert_eventually).
rg -n "def wait_for|def retry|def eventually|Timeout|def check_" tests/framework/utils*.py

# DETERMINISTIC — poll the real signal, no sleep.
from framework.utils import wait_for  # verify the import path on your branch

def test_block_metrics(uvm_plain):
    vm = uvm_plain
    vm.spawn(); vm.basic_config(); vm.add_net_iface(); vm.start()

    vm.ssh.run("dd if=/dev/vda of=/dev/null bs=1M count=16")

    def reads_recorded():
        m = vm.flush_metrics()
        return m["block"]["read_count"] > 0

    # Retry flush+check until true or timeout; raises with the last value on failure.
    wait_for(reads_recorded, timeout_s=10)

If your branch has no generic helper, write the loop explicitly — the point is poll a condition, not sleep a guess:

import time
deadline = time.monotonic() + 10
last = None
while time.monotonic() < deadline:
    last = vm.flush_metrics()["block"]["read_count"]
    if last > 0:
        break
    time.sleep(0.1)            # a *retry interval*, not a synchronization sleep
assert last and last > 0, f"block read_count never advanced (last={last})"

The distinction is everything: a retry interval bounds how often you re-check a condition; a synchronization sleep assumes the condition is true by then. The first is deterministic; the second is a bet.

Step 4: Fix the other archetypes when you meet them

Resource clash. If two tests (or a leaked process) fight over a socket/TAP/IP:

# Find hard-coded names that should be unique.
rg -n "\.socket\"|tap0|169\.254|/tmp/firecracker|127\.0\.0\.1:" tests/integration_tests/

Replace literals with the framework's unique allocations (the Microvm's own socket path, a TAP from netns_factory). Confirm teardown reaps the process even on failure (rg -n "def kill|reap|finally" tests/framework/microvm.py).

Ordering dependency. If a test passes alone but fails in the suite:

# Run alone, then in the file, then with a different order seed.
tools/devtool test -- "integration_tests/functional/test_foo.py::test_bar" -v
tools/devtool test -- integration_tests/functional/test_foo.py -v
tools/devtool test -- integration_tests/functional/test_foo.py -p no:randomly -v   # if order plugin present

The fix is always the same shape: the test must create the state it needs and remove the state it created. Don't rely on what ran before.

Parametrization assumption. If only one kernel/io-engine/arch fails, the assertion is variant- specific. Either make it correct for all variants (preferred) or add a reasoned, documented skip — never a blanket one:

@pytest.mark.skipif(
    platform.machine() != "x86_64",
    reason="MPTable-specific check; aarch64 uses FDT (see arch deep dive)",
)

Warning: A skip/xfail without a tracking reason is the same disease as @Ignore: the test rots and the gap is invisible. If you must mute temporarily, link an issue in the reason and open the issue. Muting is a stopgap, never the deliverable.

Step 5: Prove the fix — pass under stress

A flake fix is only credible if it survives the conditions that exposed the flake. Re-run many times, under load, and show it green every time.

# 50 iterations; any failure aborts and prints the iteration.
for i in $(seq 1 50); do
  tools/devtool test -- "integration_tests/functional/test_block_metrics.py::test_block_metrics" -q \
    || { echo "STILL FLAKY at iteration $i"; exit 1; }
done
echo "50/50 green"

# Bonus: run it with CPU pressure in another shell (stress-ng) during the loop.

You must also confirm you didn't slow the suite down: the deterministic poll should finish faster than the old sleep on the common path (it returns as soon as the condition holds, instead of always waiting the full sleep).

flowchart LR
    A[Reproduce 1-in-N] --> B[Classify archetype]
    B --> C[Remove the assumption:<br/>poll / isolate / unique names]
    C --> D[Re-run 50x under load]
    D --> E{All green & not slower?}
    E -- no --> B
    E -- yes --> F[fmt + checkstyle + commit -s]

Step 6: How Firecracker labels and handles flaky tests

Know the project's actual workflow so your PR fits it:

# Existing skips/xfails and their reasons — read the precedent.
rg -n "pytest.mark.skip|xfail|flaky|@retry|AwaitsFix|TODO.*flak" tests/integration_tests/ | head
# Any retry/flaky markers in the framework?
rg -n "flaky|rerun|retry" tests/pytest.ini tests/framework/*.py

• Flaky failures show up as red CI on PRs that didn't touch the code — the signal that prompts an issue (Type: Bug, often tagged as a flaky/CI issue).
• A temporary mute must carry a reason and a tracking issue link; the real fix removes the mute.
• The PR that fixes a flake explains the race in the description ("the metric increment is asynchronous to the flush; we now poll until it advances or 10s elapses") so reviewers can verify the reasoning, not just the diff.

Then commit:

tools/devtool fmt && tools/devtool checkstyle
git checkout -b fix/flaky-block-metrics
git add tests/integration_tests/functional/test_block_metrics.py
git commit -s -m "test: make block-metrics test deterministic (poll instead of sleep)"

Deliverables

• You reproduced a flaky test intermittently (repeat + load, or a pinned failing variant) and classified its archetype.
• You identified the false assumption (race / timing / ordering / resource clash).
• You replaced the bad mechanism with a deterministic one — poll-with-timeout, unique resources, proper setup/teardown — with no time.sleep-as-synchronization.
• You re-ran the test 50× (ideally under load) and it passed every time, without slowing the suite.
• Your commit message / PR explains the race, not just the diff; any mute carries a reason + issue.

Troubleshooting

I can't reproduce the flake locally

CI machines are more loaded and more parallel than your laptop. Add CPU pressure (stress-ng), raise the iteration count, and pin the exact failing parametrization from the red CI node id. If it only fails on aarch64, you need that arch.

My poll loop "fixes" it but the suite got slower

You're polling too coarsely (large interval) or your timeout is huge. Use a small retry interval (~50–100 ms) and a generous-but-bounded timeout; the loop should return immediately once the condition holds.

The flake is actually a real bug in Firecracker

Sometimes the test is right and the code races. Then the fix moves into src/ (a missing synchronization, an off-by-one in metric accounting). That's a Level 8 contribution — see Lab 8.1. The integration test that catches it stays.

Removing a sleep makes it fail every time now

Good — that means the sleep was masking a genuinely premature assertion. Now you can see the real ordering and write the correct wait condition.

Expected Output

$ for i in $(seq 1 50); do tools/devtool test -- "...test_block_metrics" -q || break; done
.................................................. (50 dots)
50/50 green

Stretch Goals

1. Find a real skipped/xfail'd integration test on your branch, read its tracking issue, and write down whether the mute is still justified (rg -n "skip\|xfail" tests/integration_tests/ -l).
2. Write a tiny helper in tests/framework/ that wraps poll-with-timeout (if one doesn't exist) and refactor one sleep-based test to use it.
3. Reproduce a resource-clash flake on purpose: hard-code a socket name in a copy of a test, run two in parallel, watch the collision, then fix it with unique names.
4. Take the test_block_metrics deterministic version and confirm it passes across all parametrized kernels and both io-engines.

Validation / Self-check

Answer without notes; these gate completion:

1. State the four flake archetypes and the one-line fix for each.
2. Why is time.sleep(N) not synchronization, and what is it sometimes legitimately used for?
3. What is the difference between a retry interval and a synchronization sleep?
4. A test passes alone but fails in the suite. Which archetype is it, and what's the fix?
5. Why must you reproduce the flake before changing anything, and why under load?
6. When is muting a test acceptable, and what must accompany it?
7. How do you prove a flake fix is credible, and how do you confirm you didn't slow the suite?

You have completed Level 5. You can now run, write, and debug tests at both layers and make a flaky one deterministic. Next: Level 6 — The Boot Process and Guest Memory, where the tests you write start asserting on the earliest moments of a microVM's life.