Step 2: Reproduction

You do not have a bug until you have a failing test. Stack traces in JIRA comments are circumstantial evidence; a deterministic, automated reproducer is proof. Until you have one, every hypothesis in Step 4 is unverifiable and every "fix" in Step 5 is theater.

The rule that governs this step — the repro-first rule — is absolute: no production code changes until a test fails on a clean checkout of apache/tez:master with an assertion error. Not a compilation error, not a timeout, not a setup NullPointerException — an assertion error that says the code did the wrong thing. That red bar is your Step 2 success criterion and your regression guard for the rest of the Capstone.

Goal: a JUnit test that fails on master without your patch, in under two minutes, on five out of five runs.

Reproduction strategy by bug class

The right harness is the cheapest one that reliably reproduces the bug. Reaching for MiniTezCluster when a DrainDispatcher unit test would do costs you 90 seconds per iteration for the rest of the Capstone. Match the bug class to the harness before you write a line.

Start at the top of that table and only move down when the cheaper harness genuinely cannot reproduce. A state-machine bug reproduced as a MiniTezCluster job is a reproduction you will regret every time you re-run it.

Where reproducers live

Find and read the harness before you write against it.

cd ~/tez-src   # your clone of apache/tez

# The in-JVM YARN + DAGAppMaster harness.
find tez-tests -name "MiniTezCluster.java"
# tez-tests/src/test/java/org/apache/tez/test/MiniTezCluster.java

# Its lightest consumer — read setUp/tearDown top to bottom.
grep -n "MiniTezCluster\|BeforeClass\|AfterClass" \
  tez-tests/src/test/java/org/apache/tez/test/TestOrderedWordCount.java

# The canonical "wire up a cluster, submit a small DAG, assert on output" example.
grep -n "MiniTezCluster\|submitDAG\|waitForCompletion" \
  tez-tests/src/test/java/org/apache/tez/test/TestTezJobs.java

# Local mode — no YARN at all, everything in one JVM. Fastest end-to-end.
grep -n "TEZ_LOCAL_MODE\|LocalClient" \
  tez-tests/src/test/java/org/apache/tez/test/TestLocalMode.java

For pure state-machine reproducers (no YARN, no shuffle), the pattern lives in the tez-dag impl tests:

grep -ln "DrainDispatcher" tez-dag/src/test/java/org/apache/tez/dag/app/dag/impl/*.java
# TestVertexImpl.java  TestTaskImpl.java  TestTaskAttempt.java  TestDAGImpl.java  ...

DrainDispatcher is Hadoop's synchronous testing dispatcher: you dispatch() events, then await() drains the queue on the calling thread so every handler runs before await() returns. That gives you deterministic event ordering and no real threading — the two properties that make a state-machine race reproduce on every machine. The full treatment is in Step 6 and the testing-framework deep dive.

Four reproducer templates

Pick the one that matches your bug class. Every template's job is one failing assertion.

Template A — State-machine reproducer (drive handle() directly)

When the bug is "an event arrives in an unexpected state and the machine NPEs, wedges, or drops a task," you drive the state machine by handing events to the impl directly and draining between phases. No MiniTezCluster. Read TestVertexImpl's setUp() and its createVertex(...) helper first — do not build the collaborator zoo (AppContext, event handler, EdgeManager) from scratch; extend the existing test class with a new @Test.

package org.apache.tez.dag.app.dag.impl;

// imports: DrainDispatcher, VertexEventType, VertexEventTaskCompleted,
// VertexEventSourceTaskAttemptCompleted, TezTaskID, static assertEquals ...

public class TestVertexImplTezNNNNRepro {

  private DrainDispatcher dispatcher;
  private VertexImpl vertex;

  @Before
  public void setUp() throws Exception {
    // Reuse the SAME construction path as TestVertexImpl.setUp().
    dispatcher = new DrainDispatcher();
    // ... register handlers, build AppContext, createVertex(...) ...
    vertex.handle(new VertexEvent(vertex.getVertexId(), VertexEventType.V_INIT));
    dispatcher.await();
  }

  @Test(timeout = 10_000)
  public void reproTaskCompletionBeforeRouteEvent() throws Exception {
    // 1. Drive the vertex to RUNNING.
    vertex.handle(new VertexEvent(vertex.getVertexId(), VertexEventType.V_START));
    dispatcher.await();
    assertEquals(VertexState.RUNNING, vertex.getState());

    // 2. Inject the events in the exact order the JIRA describes — the race window.
    TezTaskID t0 = vertex.getTask(0).getTaskId();
    vertex.handle(new VertexEventTaskCompleted(t0, TaskState.SUCCEEDED));
    // Do NOT await yet: interleave the second event to recreate the reorder.
    vertex.handle(new VertexEventSourceTaskAttemptCompleted(/* ... */));
    dispatcher.await();

    // 3. The assertion that fails on master, passes with the fix.
    assertEquals(VertexState.SUCCEEDED, vertex.getState());
    //            ^ on master this is FAILED because of the bug
  }
}

Principles: drive with handle(), not through a scheduler; dispatcher.await() between phases you want ordered, and withhold it where the bug needs two events queued together; assert on getState() or a counter, never on log output. The event type names (V_INIT, V_START, V_TASK_COMPLETED, V_SOURCE_TASK_ATTEMPT_COMPLETED, V_ROUTE_EVENT) are real — confirm the set for your class with grep -n "V_\|TA_\|T_" tez-dag/src/main/java/org/apache/tez/dag/app/dag/event/*EventType.java.

Template B — Scheduler / container reproducer (mock RM, drainable callback)

Scheduler bugs reproduce against a mocked TezAMRMClientAsync with a drainable app callback — no cluster. This is the exact shape TEZ-4580 used to reproduce slow preemption. Read it first:

git show 9efa6f14d -- tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestTaskScheduler.java

The skeleton:

@Test(timeout = 5000)
public void reproPreemptionTezNNNN() throws Exception {
  TezAMRMClientAsync<CookieContainerRequest> mockRMClient =
      spy(new AMRMClientAsyncForTest(new AMRMClientForTest(), 100));

  Configuration conf = new Configuration();
  conf.setInt(TezConfiguration.TEZ_AM_PREEMPTION_PERCENTAGE, 50);
  TaskSchedulerContext mockApp = setupMockTaskSchedulerContext(/* ... */, conf);
  TaskSchedulerContextDrainable drainableAppCallback = createDrainableContext(mockApp);
  TaskSchedulerWithDrainableContext scheduler =
      new TaskSchedulerWithDrainableContext(drainableAppCallback, mockRMClient);
  scheduler.initialize();
  scheduler.start();

  // Drive the exact allocation/priority sequence from the JIRA, then drain:
  scheduler.allocateTask(mock(TaskAttempt.class), taskAsk, null, null, priority, cookie, null);
  scheduler.getProgress();          // updates highest priority / triggers preemption
  drainableAppCallback.drain();

  // Assert on RM interactions — this is where the bug shows.
  verify(mockRMClient, times(3)).releaseAssignedContainer(any());
  //                    ^ on master this is the wrong count
}

drainableAppCallback.drain() is the scheduler-side equivalent of dispatcher.await() — it flushes the async callback queue so your verify(...) counts are deterministic. See the scheduler deep dive.

Template C — Shuffle / correctness reproducer (local mode first)

When the bug is "output is wrong" — missing rows, duplicated rows, a partial sort — reproduce at the lowest layer that still exercises real shuffle. Try local mode first (one JVM, real IPO stack, no YARN): set TEZ_LOCAL_MODE=true on the TezConfiguration, submit a small DAG with deterministic input, assert on the output. TestLocalMode and TestPipelinedShuffle are your models. Only escalate to MiniTezCluster if the bug needs a real fetch across container boundaries.

@Test(timeout = 120_000)
public void reproPartitionedOutputMissingRows() throws Exception {
  TezConfiguration tezConf = new TezConfiguration();
  tezConf.setBoolean(TezConfiguration.TEZ_LOCAL_MODE, true);
  // deterministic input: fixed seed if random, known row count.
  Path input = writeKnownInput(/* rows = */ 10_000, /* seed = */ 42L);

  DAG dag = buildTwoVertexScatterGatherDAG(input, output);
  TezClient client = TezClient.create("repro", tezConf);
  client.start();
  try {
    DAGClient dagClient = client.submitDAG(dag);
    DAGStatus status = dagClient.waitForCompletionWithStatusUpdates(
        EnumSet.of(StatusGetOpts.GET_COUNTERS));
    assertEquals(DAGStatus.State.SUCCEEDED, status.getState());

    long outputRows = countRows(output);
    // On master this is 9_973 (27 rows lost in shuffle). With the fix: 10_000.
    assertEquals(10_000L, outputRows);
  } finally {
    client.stop();
  }
}

waitForCompletionWithStatusUpdates(Set<StatusGetOpts>) is the real DAGClient API — it blocks until the DAG finishes and optionally pulls counters. Use it, not a Thread.sleep poll loop.

Template D — MiniTezCluster reproducer (real cluster, when you must)

Recovery, container-reuse-across-attempts, and true end-to-end bugs need the in-JVM YARN cluster. TEZ-4569 (SCATTER_GATHER + BROADCAST hangs on DAG recovery) reproduced exactly this way — MiniDFSCluster + MiniTezCluster + a real MR-input DAG + an AM-kill. Read it as your model:

git show 44c4f1ec9 -- tez-tests/src/test/java/org/apache/tez/test/TestAMRecoveryAggregationBroadcast.java

private static MiniTezCluster tezCluster;

@BeforeClass
public static void setup() throws Exception {
  Configuration conf = new Configuration();
  tezCluster = new MiniTezCluster(TestYourReproTezNNNN.class.getName(),
      /*numNodeManagers=*/ 1, /*numLocalDirs=*/ 1, /*numLogDirs=*/ 1);
  tezCluster.init(conf);
  tezCluster.start();
}

@AfterClass
public static void tearDown() {
  if (tezCluster != null) { tezCluster.stop(); tezCluster = null; }
}

MiniTezCluster is the slowest harness; use it only when a cheaper one cannot see the bug. See the YARN-integration deep dive.

The reproduction artifact standard

A reproducer is not a scratch file you delete. It is a committed artifact on a scratch branch that another engineer can run. Meet this standard before you call Step 2 done:

1. A committed test on a scratch branch. Name it after the JIRA: Test<Component>Tez<NNNN>Repro.java (the Repro suffix is for your workflow — you rename it to a real test name in Step 6). Commit it so git stash/git checkout experiments in Step 4 never lose it.
2. The exact one-line run command, recorded. e.g. mvn test -pl tez-dag -Dtest=TestVertexImplTezNNNNRepro.
3. Exact environment recording. In capstone-work/repro.md, write down: the commit you reproduced on (git rev-parse HEAD), java -version, mvn -v, OS, and the module. Line numbers move between branches; the SHA is what makes your citations reproducible for the reviewer.
4. Symptom and trigger conditions, separately. The symptom is what the user observes ("DAG wedges in RUNNING"). The trigger conditions are the precise circumstances ("only when the last task completes before the route event is processed"). Bisect the triggers experimentally — remove one condition at a time and watch the symptom disappear. Each condition that makes it vanish is one your test must encode and your fix must address.

The capstone-work/repro.md template

# Reproduction: TEZ-NNNN

## Symptom
<one sentence — the observable wrong behavior>

## Trigger conditions
- <condition 1 — verified: symptom disappears when removed>
- <condition 2>

## Environment
- Reproduced on: master @ <sha from `git rev-parse HEAD`>
- java -version: <...>   mvn -v: <...>   OS: <...>
- Regression? Passes on: <tag/sha> | n/a (never worked)

## Automated repro
- File: tez-dag/src/test/.../TestVertexImplTezNNNNRepro.java
- Method: reproTaskCompletionBeforeRouteEvent
- Command: mvn test -pl tez-dag -Dtest=TestVertexImplTezNNNNRepro
- Result on master: FAIL (expected SUCCEEDED, got FAILED)

## State-machine trace at failure
<paste the transitioned-from log excerpt — see below>

Logging: see what the state machine actually did

A reproducer without logs is half a reproducer — you will stare at these logs all through Step 4. Tez test modules ship a log4j 1.x log4j.properties under src/test/resources/ (confirm which module you are in; find . -name log4j.properties | grep -v target). The stock file logs at info to stdout. Add the packages your bug lives in:

# tez-dag/src/test/resources/log4j.properties — append to the stock file.
log4j.logger.org.apache.tez.dag.app.dag.impl.DAGImpl=DEBUG
log4j.logger.org.apache.tez.dag.app.dag.impl.VertexImpl=DEBUG
log4j.logger.org.apache.tez.dag.app.dag.impl.TaskImpl=DEBUG
log4j.logger.org.apache.tez.dag.app.dag.impl.TaskAttemptImpl=DEBUG
log4j.logger.org.apache.tez.common.AsyncDispatcher=DEBUG

# Shuffle bugs — add these instead (tez-runtime-library module):
log4j.logger.org.apache.tez.runtime.library.common.shuffle.impl.ShuffleManager=DEBUG
log4j.logger.org.apache.tez.runtime.library.common.shuffle.Fetcher=DEBUG
log4j.logger.org.apache.tez.runtime.library.common.shuffle.orderedgrouped.FetcherOrderedGrouped=DEBUG

# Scheduler bugs — tez-dag:
log4j.logger.org.apache.tez.dag.app.rm.TaskSchedulerManager=DEBUG
log4j.logger.org.apache.tez.dag.app.rm.YarnTaskSchedulerService=DEBUG

The most useful single line the state machines emit is the transition record. Every impl logs " transitioned from " on a state change — grep the surefire output for it:

grep -h "transitioned from" tez-dag/target/surefire-reports/*.txt | head -40

That is your state-transition trace, and it is exactly what you diagram in Step 3. For MiniTezCluster runs, the container logs (your task's stderr and the AM's syslog) land under the module's target/:

find tez-tests/target -name "syslog" -path "*container*" -mmin -30

Minimizing a Hive-reported bug to pure Tez

Many Tez bugs arrive through Hive: the reporter attached a Hive query and a plan, not a Tez repro. You cannot land a fix, or write a Tez regression test, against a Hive query — you must minimize it to a pure-Tez DAG that reproduces the same failure. This is a skill in its own right; the full drill is in Hive-on-Tez Lab H5. The shape:

1. Get the Tez DAG plan Hive generated (EXPLAIN / the dag.dot from the query).
2. Identify the minimal vertex/edge structure that carries the failure — usually two or three vertices with one edge type (SCATTER_GATHER or BROADCAST).
3. Rebuild that structure with DAG.create(...) / Vertex.create(...) and the test processors (TestProcessor, TestInput, TestOutput, or SimpleTestDAG) instead of Hive operators.
4. Confirm the pure-Tez DAG reproduces the same symptom, then throw the Hive query away. Your reproducer, your root cause, and your fix are all pure Tez from here.

If you cannot minimize it, the bug may genuinely be in Hive's Tez usage, not in Tez — in which case it is the wrong Capstone issue. Say so in the JIRA and pick another.

Verify determinism: five runs, five fails

A reproducer that fails once is a coin flip you happened to catch. Run it five times; all five must fail on the same assertion.

cd ~/tez-src
for i in 1 2 3 4 5; do
  echo "=== Run $i ==="
  mvn test -pl tez-dag -Dtest=TestVertexImplTezNNNNRepro -q 2>&1 | tail -20
done

If you see 4 FAIL / 1 PASS, the race window is not pinned. Do not add Thread.sleep — that is the wrong answer, and Step 6 will make you remove it. Instead: drain the dispatcher between every event and inject the conflicting events in a controlled order; or gate a producer thread with a CountDownLatch until the consumer is at a known state. If the bug genuinely depends on external timing (GC, network), fall back to a @RepeatedTest-style stress loop and assert a failure rate above 50% — less ideal, acceptable for some shuffle races, and you must say so in the JIRA.

Timeboxing and when you cannot reproduce

Timebox reproduction to one week (Week 1 of the budget). If you cannot get to a deterministic failing test in that time, you are in one of two situations, and both have a specific move.

You cannot reproduce because the report is under-specified. Bisect the report against your own environment before you give up:

• Version deltas. Does the JIRA name a version? Check out that tag and try there; then check out master. If it reproduces on the old tag and not on master, it is already fixed — say so in the JIRA and close it. If the reverse, you have a regression and a known-good/known-bad pair to git bisect in Step 4.
• Config deltas. The reporter almost certainly ran with non-default config. Diff their settings against defaults and add them one at a time — often a single tez.* key is the missing trigger condition.

You have bisected and still cannot reproduce. Ask the reporter on the JIRA. Be specific and make it cheap for them to answer. A model comment:

I'm trying to reproduce this on master (@ <sha>, JDK 21) and cannot yet. Could you share: (1) the Tez version and Hadoop version you saw this on; (2) the full tez-site.xml (or the non-default tez.* keys); (3) whether this is a fresh DAG or only reproduces after an AM restart; and (4) the DAG shape (number of vertices and the edge type between them)? I have a minimal two-vertex SCATTER_GATHER DAG that does not trigger it, so I suspect a config or edge-type condition I'm missing. Thanks!

If the reporter goes silent for a week and you still cannot reproduce, this is not your Capstone issue. Move on — an unreproducible bug is not a bug you can credibly fix, and grinding on it burns the budget you need for the issue you can land.

The repro-first rule and its exceptions

Repro-first is the default and you should treat it as inviolable. The narrow, honest exceptions:

• Trivially-visible defects (a wrong constant, an obviously-inverted condition you can prove by reading three lines). Even here, you still write the failing test — you just write it knowing the answer instead of discovering it. The test is the deliverable, not the discovery process.
• Hardware/environment-only failures you cannot reproduce on any harness. These are rarely good Capstone issues; if you take one, your "reproducer" is a stress loop with a failure-rate assertion, and you must be explicit about that in the JIRA and PR.

Neither exception lets you skip the test. There is no version of a landed Tez fix without a test that would have caught the bug.

Validation / Self-check

By the end of Step 2 you must have:

1. A committed test under <module>/src/test/java/... named Test<Component>Tez<NNNN>Repro.java, on a scratch branch.
2. That test fails on a clean master with an assertion error — not a setup error, not a timeout.
3. Five consecutive runs produce the same failure on the same assertion.
4. The failure happens in under 120 seconds per run.
5. A log4j.properties snippet enabling DEBUG on the relevant Tez packages, and a captured state-transition trace (grep "transitioned from") at the failure point.
6. capstone-work/repro.md complete — symptom, trigger conditions (each verified by removal), exact environment (SHA, JDK, OS), and the one-line run command.
7. You chose the cheapest harness that reliably reproduces (state-machine unit > scheduler mock > local-mode DAG > MiniTezCluster).

Then go to Step 3: Execution Path Analysis.