Lab H6: Writing a Diagnostic Patch

Background

The previous five labs assumed you could always reach the truth: read the DAG (H2), walk the logs (H3), attribute the layer (H4), reproduce it (H5). Sometimes you can't. The bug only happens on the reporter's locked-down cluster, the diagnostic that would tell you why isn't emitted, and you cannot reproduce locally. The contributor's move then is not to guess — it is to improve the diagnostics at the boundary so the next occurrence explains itself.

This lab is about writing diagnostics as a first-class contribution. You will learn where Tez surfaces diagnostics to callers (the real DAGStatus.getDiagnostics accumulation chain, quoted from the checkout), what Hive shows versus swallows, and you will dissect three real Tez commits that improved diagnostics — two that added detail, one that removed misleading detail. Then you write your own improvement at the Hive/Tez boundary, with a test, and decide — applying the H4 attribution skill to fixes — whether the improvement belongs in Hive or in Tez.

This closes the section. The counter/diagnostic machinery is the counters & diagnostics deep-dive; contribution mechanics are in Level 6.

Why This Lab Matters for Contributors

Diagnostic patches are the most reviewable and most welcomed contributions a newcomer can make. They are low-risk (they don't change behavior), high-value (every future debugger benefits), and they demonstrate exactly the judgment maintainers look for: you understand the boundary well enough to know what information is missing at the seam. Many committers' first merged Tez patches were diagnostic improvements — the three you dissect below are real, small, and merged.

There is also a deeper skill here. A diagnostic improvement is subject to the same attribution question as a bug fix: should the better message live in Hive (which swallowed it) or in Tez (which emitted the raw event)? Answering that well is H4 applied to fixes, and it is what turns "I added a log line" into "I put the right information at the right layer."

Prerequisites

• H4 and H5 complete.
• ~/tez-src with full git history.
• Ability to build Tez (mvn install -DskipTests) and run a module's unit tests.
• You can name @Private/@Unstable annotations and why diagnostic config is marked so (see compatibility).

Step 1: Where Tez Surfaces Diagnostics — the Real Chain

Before you improve a diagnostic, know exactly how one reaches a caller. Trace it in the checkout; do not take this on faith.

Accumulation. The AM builds up a List<String> of diagnostics on the DAG as events arrive. Read the accumulator and the reader:

grep -n "addDiagnostic\|List<String> getDiagnostics\|private.*diagnostics" \
  ~/tez-src/tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/DAGImpl.java | head

You will see DAGImpl.getDiagnostics() return the diagnostics list under a read lock, and many addDiagnostic(...) call sites — including the vertexFailed/vertexKilled composers you verified in H3:

"Vertex failed" + ", vertexName=" + vertex.getName()
               + ", vertexId="   + vertex.getVertexId()
               + ", diagnostics=" + vertex.getDiagnostics()

Handoff to the client status. DAGImpl.getDAGStatus(...) copies that list into a DAGStatusBuilder:

grep -n "getDAGStatus\|setDiagnostics" \
  ~/tez-src/tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/DAGImpl.java | head

The key line is status.setDiagnostics(diagnostics); inside getDAGStatus. That builder serialises into the protobuf the client reads.

Client read. On the caller's side (Hive is a caller), DAGStatus.getDiagnostics() returns the protobuf-backed list:

grep -n "getDiagnostics\|getDiagnosticsList" \
  ~/tez-src/tez-api/src/main/java/org/apache/tez/dag/api/client/DAGStatus.java
# public List<String> getDiagnostics() { return proxy.getDiagnosticsList(); }

So the full chain is:

flowchart LR
  E[Event arrives in AM] --> A["DAGImpl.addDiagnostic(msg)<br/>appends to List&lt;String&gt;"]
  A --> G["DAGImpl.getDAGStatus()<br/>status.setDiagnostics(diagnostics)"]
  G --> PB[DAGStatus protobuf]
  PB --> C["DAGStatus.getDiagnostics()<br/>proxy.getDiagnosticsList()"]
  C --> HIVE["Hive TezTask reads it,<br/>shows or swallows"]

There is a parallel chain at the vertex level (VertexStatus.getDiagnostics() → proxy.getDiagnosticsList()). Improving a diagnostic means putting a better string into addDiagnostic at the right site, or making Hive show more of what the chain already carries.

Step 2: What Hive Shows vs Swallows

Hive's TezTask monitors the DAGClient, and on failure pulls DAGStatus.getDiagnostics() and prints it to the console — but truncated and wrapped (the "Last 4096 bytes" you saw in H3). Two failure modes for the reporter:

Deciding which lever to pull is the attribution question from Step 6. Read how Hive consumes the status:

grep -n "getDiagnostics\|DAGStatus\|console\|LOG.error" \
  $(grep -rln "class TezTask" ~/hive-src/ql/src/java/) | head

Step 3: Dissect Real Diagnostic Improvements

Read each commit in your checkout. These are real, merged, and small.

3a. TEZ-2511 — add the exit code (Tez tez-dag)

git -C ~/tez-src show 333edd0f0

The whole fix changes one message in TaskSchedulerEventHandler from "Container failed. " to "Container failed, exitCode=" + exitStatus + ". ", and adds a unit test (testContainerExceededPMem) asserting the new string "Container failed, exitCode=-104. Exceeded Physical Memory". That single number — -104 is KILLED_EXCEEDED_PMEM — turns "a container failed, no idea why" into "the NM killed it for exceeding physical memory," which is a YARN memory attribution the reader can act on immediately. Lesson: the smallest useful diagnostic is often one number, and it ships with a test that pins the exact string.

3b. TEZ-3246 — add the actor identity (Tez tez-api)

git -C ~/tez-src show b63d7faf5

"Improve diagnostics when DAG killed by user." Before, a killed DAG's diagnostic was the constant "Kill Dag request received from client" — useless when you're trying to find who killed it. The fix adds getClientInfo() to DAGClientHandler, composing the caller's UGI and remote address (UserGroupInformation.getCurrentUser() + Server.getRemoteAddress()) into the message, so it becomes "Sending client kill from <user> at <address> to dag <id>". The test is updated to assert the message contains "Sending client kill from". Lesson: attribute the action, not just the fact — "who and from where," and assert on contains(...) so the test tolerates the variable parts.

3c. TEZ-3858 — remove misleading detail (Tez tez-dag)

git -C ~/tez-src show 87d7c145f

"Misleading dag level diagnostics in case of invalid vertex event." Here the fix makes the message shorter. The old DAG-level message interpolated event.getType(): "Invalid event " + event.getType() + " on Vertex " + .... At the DAG level that event type was misleading, so the fix drops it: "Invalid event on Vertex " + vertex.getLogIdentifier(). Lesson: a diagnostic improvement is not always more text. Misleading detail is worse than no detail — removing a field that points debuggers the wrong way is a real, mergeable improvement.

Step 4: The Three Diagnostic Patterns

With the real commits as models, here are the three tools. Reach for the lightest one that answers the question.

Pattern 1 — Boundary INFO logging

A boundary is where control crosses subsystems: Hive→Tez submit (TezTask → TezClient.submitDAG), Tez AM→container, container→processor, task→shuffle. INFO at a boundary is cheap and gives the next debugger a trail. Rules (from the real commits): tag every line with the JIRA id so the reporter can grep it; INFO not DEBUG (no log-level change required); structured key={} placeholders; log only what's needed.

// Illustrative — a boundary INFO in Hive's TezTask.submit, tagged for grep.
long t0 = System.nanoTime();
LOG.info("HIVE-XXXX diag: submitting DAG name={} vertices={}",
    dag.getName(), dag.getVertices().size());
DAGClient client = session.getSession().submitDAG(dag);
LOG.info("HIVE-XXXX diag: submitDAG returned in {} ms",
    TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - t0));

Pattern 2 — a new TezCounter

Counters aggregate across tasks and persist in the UI and summary. Add one when there is a count to track. Read the real enums (verified in H2):

grep -n "REDUCE_INPUT_GROUPS\|SHUFFLE_BYTES\|OUTPUT_RECORDS" \
  ~/tez-src/tez-api/src/main/java/org/apache/tez/common/counters/TaskCounter.java

// New enum member, JIRA-tagged in a comment:
/** TEZ-XXXX diag: shuffle fetch retries on this task. */
SHUFFLE_FETCH_RETRIES,
// Increment at the hot spot:
context.getCounters().findCounter(TaskCounter.SHUFFLE_FETCH_RETRIES).increment(1);

Pattern 3 — a debug config switch

For diagnostics too noisy to run by default, gate them behind a @Private @Unstable config key so the cost is opt-in and it's clear the key is temporary:

@Private @Unstable
public static final String TEZ_AM_DIAGNOSTICS_VERBOSE = "tez.am.diagnostics.verbose";
public static final boolean TEZ_AM_DIAGNOSTICS_VERBOSE_DEFAULT = false;

The reporter sets tez.am.diagnostics.verbose=true, reproduces, and captures. The switch is removed when the real fix lands — it is not a supported API.

Step 5: Guided Contribution — Improve a Boundary Diagnostic

Now do it for real, following the shape of TEZ-2511/3246/3858.

1. Find a poor diagnostic at the Hive/Tez boundary. Candidates: a submit-time failure whose message doesn't name the missing resource; a vertex-failure diagnostic that omits the vertex's parallelism; a shuffle failure that doesn't say which upstream host. Reproduce it (H5) or find it by reading addDiagnostic call sites.
2. Write the improved message. Add the one missing fact (a number, an identity, a host) — or remove a misleading one (TEZ-3858). Keep it one line, structured.
3. Add the test. Assert the new string with contains(...) for variable parts, as TEZ-3246 does. Find the sibling test:

   grep -rln "class TestTaskSchedulerEventHandler\|class TestDAGClientHandler" \
     ~/tez-src/tez-dag/src/test/java/
   

4. Prove no behavior change. mvn test -pl tez-dag (or the relevant module) stays green; a diagnostic patch must not alter behavior.
5. Write the JIRA/PR per design-via-JIRA and patch quality: what fact was missing, the before/after message, the test.

Step 6: Both-Sides Consideration — Hive or Tez?

The attribution skill from H4, now applied to the fix. When a boundary diagnostic is poor, the improvement can live on either side, and the right side is not always where you noticed the problem.

flowchart TD
  S[Poor diagnostic at the boundary]
  S --> Q1{Does Tez already carry<br/>the information?}
  Q1 -->|No, Tez never emits it| TEZ[Fix in Tez: add it at the<br/>addDiagnostic call site]
  Q1 -->|Yes, Tez has it| Q2{Does Hive show it<br/>to the user?}
  Q2 -->|No, Hive truncates/swallows| HIVE[Fix in Hive: surface more<br/>of DAGStatus.getDiagnostics]
  Q2 -->|Yes, but it's misleading| Q3{Is the misleading part<br/>Tez's message or Hive's framing?}
  Q3 -->|Tez's message| TEZ2[Fix in Tez: TEZ-3858 shape]
  Q3 -->|Hive's framing| HIVE2[Fix in Hive: reword the console output]

Rule of thumb: information that only Tez knows (container exit code, upstream host, scheduler state) belongs in a Tez diagnostic; information only Hive knows (which operator, which row, which vectorization decision) belongs in a Hive diagnostic; a truncation problem belongs in Hive. Getting this right is what makes your diagnostic patch land in the right project the first time — the same discipline as attributing a bug.

Deliverables

• The verified grep output for the DAGImpl.addDiagnostic → getDAGStatus → DAGStatus.getDiagnostics chain, annotated.
• TEZ-2511, TEZ-3246, TEZ-3858 each read via git show and summarised: what fact was added or removed, and the principle.
• One diagnostic improvement (patch or diff) at the Hive/Tez boundary, with a test asserting the new message and green module tests.
• A written both-sides decision: does your improvement belong in Hive or Tez, and why, using the Step 6 flowchart.

Troubleshooting

Stretch Goals

1. Mine more diagnostic commits. git -C ~/tez-src log --grep=diagnostic -i --oneline | head; read two more and classify each as add-detail, remove-misleading, or surface-more.
2. Improve a truncation. Find where Hive truncates DAGStatus.getDiagnostics on the console and propose surfacing the full trace to the HS2 log even when the console is truncated. Decide: Hive or Tez? (Hint: truncation is Hive's framing.)
3. Counter + log together. For a shuffle-retry diagnostic, add both a SHUFFLE_FETCH_RETRIES counter (how often) and one INFO line (first occurrence detail), and justify using both.
4. Write the JIRA a maintainer merges. Draft the full TEZ JIRA for your Step 5 improvement, modelled on TEZ-2511's shape (message + test + one-line rationale).

Validation / Self-check

1. Trace the diagnostic chain from an AM event to Hive reading it, naming the three methods (addDiagnostic, getDAGStatus/setDiagnostics, DAGStatus.getDiagnostics).
2. TEZ-2511 added a single number to a message. What number, what did it let a reader attribute, and what shipped alongside the message change?
3. TEZ-3858 made a diagnostic shorter. Why is removing a field an improvement, and which field did it remove?
4. Give the three diagnostic patterns and the one question each best answers.
5. A boundary diagnostic is poor and Tez already carries the information. Where does the fix belong, and why?
6. Why must a diagnostic patch be provably behavior-neutral, and how do you demonstrate it?
7. State the rule of thumb for Hive-vs-Tez diagnostic ownership in one sentence, and give a real precedent for each side.

This closes the Hive-on-Tez Labs. You can now trace a SQL query into a DAG (H1), capture and inspect that DAG (H2), walk a failure to its root cause (H3), attribute it to the right project (H4), reproduce it minimally and Hive-free (H5), and improve the diagnostics at the boundary so the next failure explains itself (H6). That six-step toolkit — attribute, locate, capture, reproduce, diagnose, improve — is the practising Tez-committer skill at the Hive/Tez seam.