Lab 8.1 — Find and Reproduce a Real JIRA Issue

Lab type: Research & Reproduce Estimated time: 2–4 hours (varies wildly by issue)

Background

A bug you cannot reproduce on demand is a bug you cannot fix with confidence — and a Tez committer knows it. The reproducer is the single most valuable artifact in a contribution: it makes the root cause provable, makes the fix verifiable, and is the first thing a reviewer looks for. Before you write one line of a fix, you need something that fails the same way every time on current master.

This lab takes a real open issue from issues.apache.org/jira/projects/TEZ and turns its prose bug report into a deterministic reproducer — ideally a failing JUnit test you will ship in the PR, at minimum a recorded local-mode DAG run or MiniTezCluster run. The governing rule: it must fail on master without your change, every run, and pass after the fix. If it does not fail on master, you reproduced the wrong thing.

Why This Lab Matters for Contributors

• Committers triage by reproducibility. "Cannot reproduce" closes more Tez issues than any fix.
• A repro separates the symptom (what the reporter saw) from the trigger conditions (what is actually required to provoke it). That separation is the start of root-cause work.
• A repro promoted to a JUnit test becomes your regression guard for free — and it is exactly what Lab 8.2 turns into a merged PR.
• Posting a clean repro on a stale JIRA is itself a respected contribution, even before any code.

Prerequisites

• A real Tez checkout built green on master. Record the commit:

  cd ~/src/oss-repos/tez
  git checkout master && git pull
  git rev-parse --short HEAD          # note this hash — it goes in your JIRA comment
  mvn clean install -DskipTests -Dmaven.javadoc.skip=true
  

• JDK 21+ and Maven 3.9.14+ (README.md states the requirement).
• An Apache JIRA account (self-service at issues.apache.org) so you can comment.
• You have read the Level 8 index and know the JIRA↔GitHub-PR workflow.

Step-by-Step Tasks

Step 1 — Mine the JIRA with real JQL

Go to issues.apache.org/jira/projects/TEZ, click Advanced search, and paste JQL. These queries are genuinely useful — each targets a different tractability signal:

Open, unassigned, small-priority bugs in the AM/runtime:

project = TEZ AND status = Open AND resolution = Unresolved
AND priority in (Minor, Trivial)
AND component in ("DAG", "Shuffle", "Runtime")
ORDER BY updated DESC

Bugs nobody has claimed (no assignee) — your safest starting pool:

project = TEZ AND status = Open AND assignee is EMPTY
AND type = Bug ORDER BY created DESC

Issues with an attached patch or a reproduction already in the thread (someone did the hard part):

project = TEZ AND status in (Open, "In Progress")
AND (attachments is not EMPTY OR text ~ "reproduce")
ORDER BY updated DESC

Error-message / diagnostics work (feeds directly into Lab 8.3):

project = TEZ AND status = Open AND resolution = Unresolved
AND (summary ~ "error message" OR summary ~ "diagnostic"
     OR summary ~ "NPE" OR text ~ "confusing")
ORDER BY updated DESC

Recently-active issues (a responsive reporter can confirm your repro):

project = TEZ AND status = Open AND updated >= -180d
ORDER BY updated DESC

Tip: Sort by updated DESC, not created. A five-year-old issue that no one has touched is often already fixed elsewhere, obsolete, or abandoned by its reporter.

Step 2 — Triage by reproducibility signals

For each candidate, score it before you invest an hour. The signals that predict "reproducible in an afternoon":

Write a triage note (you will paste a version of this on the JIRA):

Issue: TEZ-XXXX — <title>
Symptom (reporter's words): <quote>
Stack trace? <yes/no; which class at top of Tez frames>
Suspected component: <tez-dag state machine / shuffle / scheduler>
Assignee / open PR? <none — confirmed>
Repro harness I'll try: <unit TestVertexImpl / local-mode DAG / MiniTezCluster>
Master commit I'm on: <hash from prerequisites>

Step 3 — Choose the reproduction harness for the bug class

Tez has three reproduction levels. Pick the lowest-cost one that reliably fails — a committer will thank you for a unit test and groan at an unnecessary cluster test.

The unit-test harness. Open TestVertexImpl and read its setup/setupPostDagCreation — it is the template. The pattern is: build a DrainDispatcher, mock(AppContext.class), wire the mock's getHadoopShim(), getContainerLauncherName(...), etc., construct a VertexImpl, feed it events, then dispatcher.await() and assert state. That is how a state-machine bug becomes a 30-line deterministic test. Note the conventions the file already uses (you must match them in Lab 8.2): nearly every method is annotated @Test(timeout = ...), and synchronization is DrainDispatcher, never Thread.sleep.

# Read the harness before you write anything
sed -n '/void setup()/,/^  }/p' \
  tez-dag/src/test/java/org/apache/tez/dag/app/dag/impl/TestVertexImpl.java | head -60
grep -c "@Test(timeout" tez-dag/src/test/java/org/apache/tez/dag/app/dag/impl/TestVertexImpl.java

The local-mode harness. Every DAG in tez-examples runs in-process with -local (see TezExampleBase, which flips TezConfiguration.TEZ_LOCAL_MODE to true when it sees the flag, and also enables TezRuntimeConfiguration.TEZ_RUNTIME_OPTIMIZE_LOCAL_FETCH). This is the fastest way to reproduce a runtime/shuffle bug end-to-end without YARN — build a minimal input, run WordCount or OrderedWordCount with -local, and watch the failure.

The MiniTezCluster harness. For recovery/scheduling bugs, TestAMRecoveryAggregationBroadcast is a live example (added by TEZ-4569, below): it stands up MiniDFSCluster + MiniTezCluster, submits a DAG, kills the AM to force recovery, and asserts the result. Read it before writing your own — it shows the recovery-config knobs (DAG_RECOVERY_MAX_UNFLUSHED_EVENTS, RecoveryService.TEZ_TEST_RECOVERY_DRAIN_EVENTS_WHEN_STOPPED) that make AM-restart deterministic.

Step 4 — Reproduce, then minimize

Get any reproduction first, then strip it to the minimum that still fails:

1. Reproduce the symptom (run the DAG, drive the events, whatever provokes it).

2. Vary one factor at a time and record which ones matter:

   Factor	Still reproduces?	Conclusion
   Edge type (SCATTER_GATHER vs BROADCAST)	only with BROADCAST?	edge type is the trigger
   With vs without AM restart	only after restart?	it is a recovery bug
   1 task vs N tasks	both?	not a parallelism/skew bug
   Vertex-manager plugin type	only RootInputVertexManager?	plugin path is implicated

3. The set of factors that must hold is your trigger condition. Everything else is noise to delete from the test.

Record, precisely:

• The exact exception and stack trace (top Tez frame is your fix-site anchor).
• Whether it is deterministic or intermittent (a race — pin it as best you can and say so).
• The master commit hash it fails on.

A minimal state-machine reproduction skeleton — the shape you fill in from the harness you just read (this is a template, not copy-paste code; wire the mocks the way TestVertexImpl.setup() does):

@Test(timeout = 5000)
public void testMyReproduction() {
  // 1. Build the DAG plan with ONLY the trigger conditions from Step 4
  //    (e.g. one vertex with the specific edge/plugin combination).
  setupPostDagCreation();                 // harness helper: DrainDispatcher + mocked AppContext
  initVertex(v);                          // drive the state machine to the point of failure
  // 2. Feed the event that provokes the bug
  dispatcher.getEventHandler().handle(new VertexEvent(v.getVertexId(), VertexEventType.V_INIT));
  dispatcher.await();                     // NEVER Thread.sleep — drain the dispatcher instead
  // 3. Assert the WRONG behavior you observed (this is what fails on master)
  assertEquals("vertex should not be FAILED", VertexState.RUNNING, v.getState());
}

Run it and confirm it is red on master. If it is green, you asserted current behavior — go back to Step 4 and re-derive the trigger.

Step 5 — Map symptom → root cause

Trace from the symptom to the wrong line. The real fix site is often ~10 lines above the throw:

1. Start at the exception message; grep the literal string to find the throw site.

   grep -rn 'the exact message text' tez-dag/src/main/java tez-runtime-library/src/main/java
   

2. Read the method around it. Walk the call stack backwards from the trace.
3. Identify the single line whose logic is wrong (a null not guarded, a condition inverted, a field read before it is set).

Worked Examples: Real Closed JIRAs as Repro Models

Study how three merged fixes were reproduced. Read each commit in your checkout — do not take these summaries on faith.

TEZ-4308 — the trivial-but-real error-message fix (harness: none needed)

git show 5eeccf0e3 --stat
git show 5eeccf0e3

A one-file change (+6/−5) to ShuffleScheduler.java that fixed a genuinely user-facing defect: the "too many fetch failures" error concatenated "...insufficient progress!" + "failureCounts=" + ... with no separator, producing progress!failureCounts=. The fix added a space and bracketed the fields: "...insufficient progress: [failureCounts=" + ... + "]". Repro lesson: for a message-quality bug the "reproducer" is just reading the string-building code and the log it emits — no test harness required, and it still merged. This is the archetype Lab 8.3 builds on.

TEZ-4569 — SCATTER_GATHER + BROADCAST hangs on DAG recovery (harness: MiniTezCluster)

git show 44c4f1ec9 --stat      # VertexImpl.java (+45/-18) + a 509-line recovery test
git show 44c4f1ec9 -- tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/VertexImpl.java

The symptom — a DAG hangs on AM recovery when a vertex mixes SCATTER_GATHER and BROADCAST inputs — is not reproducible in a plain unit test; it needs a real AM restart. The author added TestAMRecoveryAggregationBroadcast in tez-tests, which uses MiniDFSCluster + MiniTezCluster and forces recovery. The fix itself extracted the tangled skip-initialization condition in VertexImpl into a readable canSkipInitialization() and added a usesRootInputVertexManager() predicate so a vertex driven by RootInputVertexManager can start without waiting on parents. Repro lesson: match the harness to the bug — a hang on recovery mandates the cluster harness, and the fix's payoff was as much readability (a named predicate) as behavior.

TEZ-4699 — validation hardening shipped with its test (harness: plain unit test)

git show bc265069f --stat      # CSVResult.java (+53/-28) + new TestCSVResult.java (+136)
git show bc265069f

A path-traversal hardening of CSVResult.java (in tez-tools) that added a validateOutputFile() guard rejecting a filename that escapes the working directory (it normalizes the target path and checks targetPath.startsWith(baseDir)), shipped together with a brand-new TestCSVResult.java. Repro lesson: the fix and its test land in the same PR; the test is the executable reproduction of "a malicious filename escapes the output dir." This is the exact fix-with-test discipline Lab 8.2 demands.

Note: Every hash above is from this repository's history. Re-run git show <hash> yourself — if a hash has drifted, find the commit by git log --grep="TEZ-4308" (the JIRA id is stable even when the hash is not).

Step 6 — Document the Repro on JIRA (properly)

Post a comment that lets a committer reproduce in one command. Professional, concise, complete:

Reproduced on master @ <short hash> (mvn clean install -DskipTests green).

Minimal trigger: <the stripped-down conditions from Step 4>.

Repro (fails on master):
  mvn test -pl <module> -Dtest=<YourReproTest>#<method>

Observed: <exact exception / wrong state / hang>, stack top at
  <Class>.<method> (org.apache.tez...impl.<Class>).
Expected: <the correct behavior>.

I'm working on a fix and will open a PR referencing this issue.

This does three things: proves the bug is real, claims the issue (prevents duplicate work), and gives the reviewer the command they will run on your eventual PR. If you do only this — post a clean, runnable repro on a stale issue — you have already made a real contribution.

Deliverables

• A triage note (issue id, symptom, reproducibility signals, assignee check, plan).
• A stated minimal trigger — what must be true, and what is irrelevant.
• The master commit hash the bug fails on (and a seed/thread note if it is a race).
• A reproducer in the correct harness that fails on master: a tez-dag unit test, a local-mode DAG run, or a MiniTezCluster test.
• A one-command way to run it.
• A JIRA comment documenting the repro (or a draft ready to post).

Troubleshooting

Stretch Goals

1. Reduce the reproducer to its absolute minimum — fewest vertices, fewest tasks, smallest harness — so a reviewer reads it in 20 seconds.
2. Write the repro at two levels (unit + MiniTezCluster) and decide which you would ship. Justify the lowest-cost choice in a JIRA comment.
3. git log -S bisect: find roughly when the behavior was introduced.

   git log -S "the symbol or string you grepped" -- \
     tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/ | head
   

4. Find a stale open TEZ issue with no repro and post one — the highest-leverage thing you can do before writing any fix.

Validation

1. Does your reproducer fail on clean master? Show the red output and the commit hash.
2. State the minimal trigger conditions — what must be true, what is irrelevant?
3. Which harness did you choose, and why was it the lowest-cost one that reliably fails?
4. Was the root cause where the stack trace pointed, or did you have to trace upward?
5. Is there a comment near the bug site describing the intended behavior — and was it wrong?
6. Could a committer reproduce from your JIRA comment alone, in one command?
7. How will this exact artifact prove your fix works in Lab 8.2?

Cross-references: Lab 8.2: Implement the Fix, Capstone step 2: Reproduction, patch quality, JIRA review.