Lab H4: Bug Attribution

Background

Lab H3 surfaced the root-cause stack trace. This lab decides who owns it. A failing Hive-on-Tez query may be a Hive bug, a Tez runtime-library bug, a Tez AM bug, a Tez MR-compat bug, a YARN bug, an HDFS bug, a JVM issue, a user error, or plain infrastructure. Filing on the wrong project wastes the reporter's day and the maintainer's, and — worse — it trains the community to distrust your triage.

Attribution is the keystone contributor skill at this boundary, the Tez analog of the OpenSearch curriculum's "core vs plugin vs Dashboards vs Lucene" attribution. This lab gives you a mechanical decision tree, a package→project→module table, and — the part that makes it real — three actual fixed Tez JIRAs that originated as Hive-reported issues, dissected to show how the bug crossed layers. Only commits that were read in the Tez checkout appear here; you will re-read them yourself.

The ownership boundary is the subject of the Hive integration deep-dive; the attribution muscle you build here is exercised again at contribution time in Level 6.

Why This Lab Matters for Contributors

The community's trust in you is largely a function of your attribution accuracy. A contributor who consistently files on the right project with a stack trace that proves the attribution gets their issues picked up quickly. One who guesses gets ignored. The attribution is not an opinion — it is a claim you back with the top actionable frame of a trace and, ideally, a repro that runs without the other layer (the pure-Tez repro you build in H5).

The hardest and most valuable cases are the ones that cross layers: a symptom a user saw in Hive whose fix landed in Tez. The three real JIRAs below are exactly those. Learn to see the seam.

Prerequisites

• H3 complete; you can produce a root-cause trace.
• ~/tez-src present with full git history (git -C ~/tez-src log --oneline | head).
• ~/hive-src present for the module layout.
• Comfort reading git show output.

The Ownership Map

Before the mechanics, the concepts. Draw the line by responsibility, not by where the exception happened to be thrown:

The subtlety: an exception thrown in Tez code can be Hive's fault (Hive handed Tez a malformed plan) and vice versa. The map is about who must change code to fix it, which is why the stack-trace rule below looks at the top frame you can change, then adjusts using the Caused by: chain.

The Decision Tree

flowchart TD
  S[Have the root-cause trace from H3]
  S --> T1[Find top frame whose package you can change]
  T1 --> P{Package prefix?}
  P -->|org.apache.hadoop.hive.*| H[Hive]
  P -->|org.apache.tez.runtime.library.*| TR[Tez tez-runtime-library]
  P -->|org.apache.tez.runtime.* not .library| TRI[Tez tez-runtime-internals]
  P -->|org.apache.tez.dag.app.*| TA[Tez tez-dag / AM]
  P -->|org.apache.tez.dag.api.* / client.*| TC[Tez tez-api]
  P -->|org.apache.tez.mapreduce.* / hadoop.mapred.*| TM[Tez tez-mapreduce]
  P -->|org.apache.hadoop.yarn.*| Y[YARN]
  P -->|org.apache.hadoop.hdfs.*| HD[HDFS]
  P -->|user package| U[User code]
  P -->|java.* sun.* jdk.*| J[Walk down one frame]
  J --> T1
  H --> CD[Read Caused by chain]
  TR --> CD
  TRI --> CD
  TA --> CD
  TM --> CD
  Y --> CD
  CD --> R[Root cause may shift the owner]
  R --> END[File on the project that owns the actionable code]

The rule in one sentence: find the top frame in actionable code, name its package prefix, read off the project, then let the Caused by: chain adjust.

Package → Project → Module Table

Verify the module list against your tree so the mapping is grounded:

find ~/tez-src -maxdepth 2 -name pom.xml | sort

Real Boundary Case 1 — TEZ-738 (Hive symptom, Tez AM state-machine fix)

Find and read the commit yourself:

git -C ~/tez-src log --grep=Hive -i --oneline | grep -i "738\|preempt"
git -C ~/tez-src show 4ba6f07b9

The Hive-reported symptom. The JIRA title is "Hive query fails with Invalid event: TA_CONTAINER_PREEMPTED at SUCCEEDED." A user ran a Hive query; a container that had already succeeded got preempted by YARN; Tez's TaskAttemptImpl state machine had no transition for a TA_CONTAINER_PREEMPTED event arriving in the SUCCEEDED state, so it threw "Invalid event" and failed the DAG. To the user this looked like a Hive failure — their Hive query died.

Where the fix landed. Purely in Tez, in the AM state machine. The one-line change adds TA_CONTAINER_PREEMPTED to the set of events tolerated at TaskAttemptStateInternal.SUCCEEDED:

git -C ~/tez-src show 4ba6f07b9 -- \
  tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/TaskAttemptImpl.java

You will see the .addTransition(SUCCEEDED, SUCCEEDED, EnumSet.of(TA_TIMED_OUT, ...)) gain TA_CONTAINER_PREEMPTED. Attribution: Tez tez-dag (AM), component AM. The trace, had you seen it, would have had a top actionable frame in org.apache.tez.dag.app.dag.impl.TaskAttemptImpl — a textbook "org.apache.tez.dag.app.*" attribution, even though the user experienced it as a Hive query dying. This is the seam: Hive symptom, Tez cause, YARN trigger.

Real Boundary Case 2 — TEZ-140 (large-DAG Hive failure, Tez state machines)

git -C ~/tez-src show 9138f7906

The Hive-reported symptom. "Tez/hive task failure on large DAG with Invalid event: TA_SCHEDULE at KILLED." Again a Hive user, again an "Invalid event" — this time a TA_SCHEDULE arriving after the attempt was already KILLED, on large DAGs where the race window is wide. Same family as Case 1: a Hive-visible failure whose cause is a missing/incorrect transition in Tez's TaskAttemptImpl, TaskImpl, and VertexImpl state machines.

git -C ~/tez-src show 9138f7906 --stat

The diff touches three Tez state-machine classes in tez-dag. Attribution: Tez tez-dag. The lesson repeated: "Invalid event: X at STATE" in a Hive-on-Tez failure is a Tez state-machine attribution nearly every time — the package prefix will be org.apache.tez.dag.app.dag.impl.*. When you see that phrase, you can attribute before you even open the full trace, then confirm with the top frame.

Pattern: Both TEZ-738 and TEZ-140 are "Invalid event" bugs. Memorise the shape — Invalid event: <EVENT> at <STATE> — and its owner: Tez AM state machine. It is one of the highest-signal attribution shortcuts at this boundary.

Real Boundary Case 3 — TEZ-2741 (Hive schema change, Tez MR-compat, then reverted)

The most instructive case, because it shows the boundary is genuinely hard — the fix was committed and then reverted the next day.

git -C ~/tez-src show 5149cc486        # the fix
git -C ~/tez-src show 8126d2eb3        # the revert
git -C ~/tez-src log --oneline --grep="TEZ-2741"

The Hive-reported symptom. "Hive on Tez does not work well with Sequence Files Schema changes." When a Hive table backed by SequenceFiles had a schema change across files, reads misbehaved. The suspected cause was in Tez's MR-compat split reader, org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat — the code Tez uses to group input splits, which Hive's HiveSplitGenerator builds on.

The fix (5149cc486). Four lines in TezGroupedSplitsInputFormat resetting key and value on the schema boundary:

git -C ~/tez-src show 5149cc486 -- \
  tez-mapreduce/src/main/java/org/apache/hadoop/mapred/split/TezGroupedSplitsInputFormat.java

The revert (8126d2eb3). Committed one day later, reverting the change (with a CHANGES.txt conflict noted). The fix did not hold — evidence that this bug straddled the Hive/Tez boundary so tightly that the Tez-side change was not the right home, or caused a regression.

What this teaches. The package prefix said org.apache.hadoop.mapred.split.* → Tez tez-mapreduce. That attribution was plausible and a committer acted on it — yet the fix was reverted, meaning the real resolution lived elsewhere (Hive's use of the reader, or a different Tez approach). Attribution can be right about the layer and still wrong about the fix. When a bug sits exactly on the seam, expect to file on both projects and cross-reference, and expect iteration. This is why H5's "reproduce without the other layer" is decisive: a pure-Tez repro of TEZ-2741 (a TezGroupedSplitsInputFormat test with no Hive) would have proven whether Tez alone could reproduce it.

Stack-Trace Forensics: Package Prefix → Layer

Practice the mechanical read on the four canonical shapes.

When the Caused by: chain shifts the owner

The top frame names the proximate code; the Caused by: chain can move the attribution. Read the whole chain before you commit. Three concrete adjustments:

• Top frame Hive, root cause user. Top is ...hive.ql.exec.tez.MapRecordSource, root is ClassNotFoundException: com.example.udf.X. The proximate code is Hive, but the missing class is the user's UDF jar — attribution is user, no JIRA. Had the root been ClassNotFoundException on a Hive class (org.apache.hadoop.hive.ql.exec.*), it would be a Hive packaging bug.
• Top frame Tez, root cause infra. Top is ...tez.runtime.library...ShuffleScheduler, root is java.net.ConnectException. The shuffle code surfaced a network failure — if it happens once, infrastructure; if the fetcher gives up below its configured retry limit, Tez. The tez.runtime.shuffle.fetch.failures.limit value decides.
• Top frame Tez AM, no Caused by:. An OutOfMemoryError with the first Tez frame in org.apache.tez.dag.app.dag.impl.VertexImpl and no deeper cause — a Tez AM sizing or allocation issue (tez.am.resource.memory.mb), filed with profile evidence.

The rule: name the owner from the top actionable frame, then let the deepest actionable Caused by: frame override it if it points at a different package.

Cross-project shapes worth memorising:

Reproduce in Isolation: Take Hive Out

The strongest attribution is a repro that runs the DAG without Hive. Tez ships synthetic DAGs and example jobs whose edge/processor semantics mirror what Hive emits. Confirm they exist in your tree, then use them as templates:

# Broadcast (map-join analog) and sort-merge (shuffle-join analog) examples:
ls ~/tez-src/tez-examples/src/main/java/org/apache/tez/examples/HashJoinExample.java
ls ~/tez-src/tez-examples/src/main/java/org/apache/tez/examples/SortMergeJoinExample.java
# A synthetic multi-stage (Map→Reduce→Reduce) DAG, the MRR analog of group-by+order-by:
ls ~/tez-src/tez-tests/src/main/java/org/apache/tez/mapreduce/examples/MRRSleepJob.java
# Fault-injection test DAGs for shuffle/failure reproduction:
ls ~/tez-src/tez-tests/src/test/java/org/apache/tez/test/SimpleTestDAG.java
ls ~/tez-src/tez-tests/src/test/java/org/apache/tez/test/TestFaultTolerance.java

If the pure-Tez analog reproduces the symptom, Hive is exonerated and you file on Tez with a Hive-free repro — the gold standard, and the whole subject of H5.

Deliverables

• Each of TEZ-738, TEZ-140, TEZ-2741 read via git show and summarised in your own words: the Hive symptom, the Tez module the fix touched, and (for TEZ-2741) why it was reverted.
• The decision tree and the package→module table saved.
• Four traces (from H3 or synthesized) attributed in writing, each naming the top actionable frame and the owning module.
• For one attribution, the pure-Tez analog (hashjoin/sortmergejoin/mrrsleep) you would use to prove it without Hive.

Troubleshooting

Stretch Goals

1. Mine more boundary JIRAs. Run git -C ~/tez-src log --grep=Hive -i --oneline | head -30, read two commits you have not seen, and dissect how each crossed the layer.
2. Reproduce TEZ-738's shape. Using TestFaultTolerance/SimpleTestDAG, drive a preemption-like event and confirm the state machine now tolerates it. Which transition would you break to re-open the bug?
3. Prove exoneration. Take a Hive shuffle-join symptom and reproduce it with SortMergeJoinExample alone. Write the one-paragraph attribution the repro justifies.
4. Attribute a version-skew case. Induce a NoSuchMethodError by mixing Tez jars, and show why it is not a code bug in either project.

Validation / Self-check

1. State the one-sentence attribution rule and the role of the Caused by: chain.
2. TEZ-738 was reported as a Hive failure but fixed in Tez. Which module, and what is the generalizable pattern for "Invalid event: X at STATE" failures?
3. Why is TEZ-2741 the most instructive of the three, given that its fix was reverted? What does that say about attributing seam bugs?
4. Map each of the four canonical top frames to its owning Tez/Hive module.
5. You suspect a shuffle-join bug is Tez, not Hive planning. Which pure-Tez example reproduces it without Hive, and what does a successful repro prove?
6. Given org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat at the top of a trace, which project and module, and which real JIRA is the precedent?
7. When do you file on both Hive and Tez, and what artifact makes that defensible?

You can now attribute a Hive-on-Tez failure to the right project and prove it. Next, Lab H5: Reproducing Bugs turns an attributed bug into a minimal, Hive-free reproducer a maintainer will actually run.