Lab 5.4: Fix It — Un-Ignore a Flaky Test

Lab type: Fix-It — flaky-test archaeology, reproduction, and repair Estimated time: 120 min Tez modules: tez-tests, tez-dag Key classes: real @Ignored tests in the checkout

Background

Every long-lived Java project accumulates @Ignored tests. Some were disabled because they are flaky (pass sometimes, fail sometimes with no code change); others because they need a resource the CI box doesn't have, or because they encode an assumption that turned out to be invalid. A flaky test is almost always a symptom of a real problem — a race, a timing assumption, or a DrainDispatcher that was not drained — and muting it silently drops coverage.

In this lab you will triage the real @Ignore inventory in the Tez checkout, reconstruct why each was disabled using git log and git blame, pick the genuinely flaky one, run it in a loop to characterize the failure, and reason about what a credible "un-ignore" patch requires. You will not be able to hand-wave: the JIRA history is real and you will read it.

Warning: "Un-ignore it and hope it passes once" is not a fix. A flaky test passes usually, so a single green run proves nothing. The bar for un-ignoring is: reproduce the flake, find the root cause, fix it, and show N consecutive clean runs — the same bar the Tez maintainers hold.

Why This Lab Matters for Contributors

• Flaky CI taxes every contributor: red builds get ignored, real regressions hide in the noise. Fixing a flaky test is a visible, gratefully-received contribution.
• Un-ignoring a test is a concrete, mergeable PR with a clean before/after: one fewer @Ignore, one more reliably-running test.
• The triage skill — classify each @Ignore as flaky / resource-gated / invalid — is exactly what a committer does when someone proposes re-enabling a test.

Prerequisites

• Labs 5.1–5.3 complete; export TEZ_SRC=/path/to/tez; mvn install -DskipTests -q succeeded.
• You understand DrainDispatcher.await() from Lab 5.2 (the #1 unit-test flakiness cause in Tez).
• Read the Failure Handling and Testing Framework deep dives.

Step-by-Step Tasks

Step 1: Inventory every @Ignore in the checkout

cd "$TEZ_SRC"
rg -n "@Ignore" --type java | sed 's|.*/tez-|tez-|'

At the time of writing, the real inventory (yours may drift — re-run the command) is:

Step 2: Triage — not every @Ignore is flaky

This is the core skill. Classify each entry before you touch anything. Use git blame + git log to read the disabling commit:

# Find the commit that added a specific @Ignore line, then read its message.
git blame -L '/testRandomFailingInputs/,+1' -- tez-tests/src/test/java/org/apache/tez/test/TestFaultTolerance.java
git blame -L '/testDuplicateTaskCompletion/,+1' -- tez-dag/src/test/java/org/apache/tez/dag/app/dag/impl/TestVertexImpl.java

The real history:

# TestFaultTolerance#testRandomFailingInputs  -> commit 4389ce8cf3
TEZ-3232. Disable randomFailingInputs in testFaulttolerance to unblock other tests. (hitesh)

# TestVertexImpl#testDuplicateTaskCompletion  -> commit 38b410e311 (TEZ-109 "Create tests for VertexImpl")
@Ignore // FIXME fix verteximpl for this test to work

# TestDAGImpl#testVertexSuccessfulCompletionUpdates -> TEZ-431 addendum
@Ignore // Duplicate completions from a vertex would be a bug. Invalid test.

Now classify:

Note: A huge fraction of @Ignored tests in mature projects are not flaky — they are benchmarks or bug-markers. Proposing to re-enable a resource-gated perf test on the normal CI box would be rejected. Triage first; pick the flaky one; leave the rest with a clear-eyed explanation.

Step 3: Read the flaky test and its JIRA

git show 4389ce8cf3 --stat          # what TEZ-3232 changed
sed -n '/testRandomFailingInputs/,/^  }/p' tez-tests/src/test/java/org/apache/tez/test/TestFaultTolerance.java

The test itself (TestFaultTolerance):

@Ignore
@Test (timeout=240000)
public void testRandomFailingInputs() throws Exception {
  Configuration testConf = new Configuration(false);
  testConf.setBoolean(TestInput.TEZ_FAILING_INPUT_DO_RANDOM_FAIL, true);
  testConf.setFloat(TestInput.TEZ_FAILING_INPUT_RANDOM_FAIL_PROBABILITY, 0.5f);
  DAG dag = new FailingDagBuilder(FailingDagBuilder.Levels.SIX)
      .withName("testRandomFailingInputs").withConf(testConf).build();
  runDAGAndVerify(dag, DAGStatus.State.SUCCEEDED);
}

Record:

1. The injected randomness: inputs fail with probability 0.5, over a six-level DAG. That is the flakiness source — the fault-tolerance machinery must recover from a random pattern of input failures, and sometimes it exhausts retries or hits a race.
2. The JIRA verdict (TEZ-3232): it was disabled to unblock other tests, i.e. its intermittent failures were reddening CI for everyone. This is the classic "mute to stop the bleeding" move — the right temporary action, the wrong permanent one.

Step 4: The flaky-test debugging playbook

Before you reproduce, internalize the playbook. Tez flakiness comes in a small number of archetypes:

How Tez devs actually handle this in CI: search the history and you will find a long line of real flaky-fix JIRAs — the maintainers fix them one by one rather than muting wholesale.

git log --oneline -i --grep="flaky" | head -20

Real examples you will see (all merged fixes, not mutes): TEZ-4206 (TestSpeculation flaky), TEZ-3664 (flaky due to writing to /tmp), TEZ-3325 (TestDAGImpl.testCounterLimits), TEZ-2846 (TestCommit.testVertexCommit_OnDAGSuccess), TEZ-4123 (TestMRRJobsDAGApi timeout — notable because it was reverted once and relanded, showing how hard flaky fixes are to get right).

Step 5: Reproduce the flakiness in a loop

Un-ignore the test locally only (do not commit yet): delete its @Ignore line. A flaky test tells you nothing in one run, so loop it. It is an integration test (boots the mini-cluster), so keep the loop small and expect minutes per iteration:

cd "$TEZ_SRC"
# Remove the @Ignore on testRandomFailingInputs first (edit the file), then:
for i in $(seq 1 10); do
  echo "=== run $i ==="
  mvn test -pl tez-tests -Dtest='TestFaultTolerance#testRandomFailingInputs' -q 2>&1 \
    | grep -E "Tests run|BUILD" | tail -2
done

Record the pass/fail pattern. Interpret it:

• Randomly failing → confirmed flaky (the expected outcome here). Note how often and capture a failing run's Surefire -output.txt for the stack trace.
• Always failing → the behavior regressed since 2016, or the test is now invalid; a deterministic bug.
• Always passing → the underlying fault-tolerance bug may have been fixed since TEZ-3232; then the credible patch is "un-ignore + evidence of N clean runs + note that the original flake no longer reproduces."

Tip: For a unit flake (e.g. anything in TestVertexImpl) you can loop far faster and, crucially, control determinism by ensuring every event fire is followed by dispatcher.await(). For an integration flake like this one, you control the input via the injected TEZ_FAILING_INPUT_RANDOM_FAIL_PROBABILITY — lowering or pinning it changes the failure pattern and helps you bisect whether the flake is in fault injection or in recovery.

Step 6: Characterize the failure

From a captured failing run, answer:

grep -n "FAILED\|Exception\|retries\|attempt\|SUCCEEDED\|Vertex failed\|Task failed" \
  tez-tests/target/surefire-reports/org.apache.tez.test.TestFaultTolerance-output.txt | head -40

1. Which archetype (Step 4) does it match?
2. Does the DAG fail because recovery exhausted retries (too many random input failures at 0.5), or because of a race in the recovery path (an event processed out of order)?
3. Is the flake in the test's expectation (asserting SUCCEEDED when the injected failure rate can legitimately exhaust retries) or in production recovery logic?

This distinction determines the fix. If a 0.5 failure probability over six levels can legitimately exceed the retry budget, the test's expectation is wrong (it should tolerate DAG failure or use a lower probability) — a test fix. If recovery should always succeed but sometimes races, it is a production fix in the fault-tolerance path. Both are credible; you must say which and back it with evidence.

Step 7: What a credible un-ignore patch looks like

You are not expected to land a production fault-tolerance fix in this lab. You are expected to produce the artifact a committer would need to even consider un-ignoring:

1. Archaeology: the disabling commit (4389ce8cf3, TEZ-3232) and its stated reason.
2. Reproduction: the loop from Step 5 and the observed pass/fail ratio.
3. Characterization: the archetype and the root-cause hypothesis from Step 6, with the log evidence.
4. The change: either the removed @Ignore plus the fix, or a reasoned argument that the flake no longer reproduces (with the N clean runs to prove it).
5. Evidence of stability: N consecutive clean runs (Tez reviewers expect this — a single green run is not evidence).

# Prove stability after your change (raise N if the flake was rare):
PASS=0; N=20
for i in $(seq 1 $N); do
  if mvn test -pl tez-tests -Dtest='TestFaultTolerance#testRandomFailingInputs' -q >/dev/null 2>&1; then
    PASS=$((PASS+1)); fi
done
echo "clean runs: $PASS / $N"

Draft the JIRA:

Summary: Re-enable TestFaultTolerance#testRandomFailingInputs (disabled by TEZ-3232)
Component: test
Description:
  TEZ-3232 disabled testRandomFailingInputs to unblock CI. Investigation:
    - Repro: ran the test N times; observed <X> failures out of N (or: 0 failures — no longer reproduces).
    - Archetype: <injected-randomness recovery / timing trap>.
    - Root cause: <recovery exhausted retries at p=0.5 / race in ... / no longer reproduces after TEZ-####>.
    - Change: <lowered fail probability to p / fixed ... / removed @Ignore only>.
  Evidence: <N> consecutive clean runs after the change.

Warning: If you cannot reproduce the flake and cannot explain why it stopped, do not propose un-ignoring. "It passed 20 times on my laptop" is weaker than a maintainer's memory of it failing nightly. A fix you cannot validate is not a fix.

Deliverables

• The full @Ignore inventory (Step 1) and the triage table classifying each as flaky / bug-marker / invalid / resource-gated (Step 2).
• Git archaeology for the flaky target: the disabling commit hash and JIRA (TEZ-3232) and its reason.
• A reproduction loop (Step 5) with the observed pass/fail pattern.
• A characterization: the archetype, the root-cause hypothesis, and the log evidence (Step 6).
• A written statement of what a credible un-ignore patch would contain, plus the N-clean-runs command output (Step 7). A production fix is a stretch goal, not required.

Troubleshooting

Stretch Goals

1. Fix a unit flake instead. Unit flakes loop in seconds. Find any TestVertexImpl/TestDAGImpl test that fires an event and asserts without an intervening dispatcher.await(), add the await(), and show it is now deterministic. This is the most common real Tez flaky-fix shape.
2. Bisect the fault-injection. For testRandomFailingInputs, sweep TEZ_FAILING_INPUT_RANDOM_FAIL_PROBABILITY from 0.1 to 0.5 and plot pass rate vs probability. Determine whether the flake is "recovery can't keep up past probability p" (test-expectation bug) or present even at low p (production race).
3. Read a real merged flaky fix. git show the commit for TEZ-3664 (flaky due to /tmp writes) or TEZ-4206 (TestSpeculation) and write two sentences on the actual root cause and fix — real examples of the archetypes in Step 4.
4. Propose the honest outcome. If your loop is always green, write the "re-enable + N clean runs + note the original flake no longer reproduces" patch and argue why a committer should trust it.

Validation / Self-check

1. Why is un-ignoring a test and running it once not evidence that it is fixed?
2. Triage the real inventory: which @Ignored tests are flaky, which are bug-markers, which are invalid, which are resource-gated? Give one example of each and how you can tell them apart.
3. What commit and JIRA disabled testRandomFailingInputs, and what reason did it give?
4. What is the injected-randomness source in that test, and why does it cause flakiness?
5. Name three flaky-test archetypes in Tez and the fix direction for each. Which one is the dominant unit-test cause, and what is its one-line fix?
6. How do you distinguish a flaky test whose expectation is wrong from one whose production code races? Why does the distinction change the fix?
7. What five things must a credible un-ignore patch contain?

When you can triage the @Ignore inventory, reconstruct why a test was disabled, reproduce and classify its flakiness, and describe the evidence a real un-ignore patch needs, you have completed Level 5. Continue to Level 6: Hive/Tez Integration, or revisit the Testing Framework deep dive when you take a real flaky-test JIRA.