Lab 6.2 — Debug a Failed Hive-on-Tez Query

Lab type: Fix-It (diagnostics + root-cause analysis) Estimated time: 150 min Tez modules: tez-dag, tez-runtime-library Hive classes (verify in your Hive checkout): TezTask, DagUtils

Background

A Hive-on-Tez query can fail in four very different places, and the whole skill of debugging one is telling them apart fast:

                 ┌─────────────────────────────────────────────┐
   HiveQL ─────▶ │ 1. COMPILE-TIME  (SemanticAnalyzer/TezCompiler) │  Hive only
                 └─────────────────────────────────────────────┘
                 ┌─────────────────────────────────────────────┐
   TezWork ────▶ │ 2. DAG-SUBMIT    (DagUtils / TezClient)      │  Hive→Tez seam
                 └─────────────────────────────────────────────┘
                 ┌─────────────────────────────────────────────┐
   DAG runs ───▶ │ 3. RUNTIME TASK  (processor/UDF/SerDe, OOM)  │  Tez executes Hive code
                 └─────────────────────────────────────────────┘
                 ┌─────────────────────────────────────────────┐
                 │ 4. SHUFFLE       (fetch failure, merge OOM)  │  Tez runtime-library
                 └─────────────────────────────────────────────┘

Each layer emits diagnostics in a different place: compile-time errors land on the Hive console immediately; submit errors come from TezClient; runtime and shuffle failures surface as Tez AM diagnostics (Vertex failed, Task failed) that Hive relays, with the real detail only in the task logs you pull with yarn logs.

In this lab you take a worked failure — a ClassNotFoundException from a missing UDF jar — and walk the entire diagnostic chain: Hive console → Tez AM Vertex failed → task attempt log → root cause → the Hive operator behind the failed vertex. Then you build a reusable symptom→cause→where-to-look table for the whole failure taxonomy.

Why This Lab Matters for Contributors

When a user files "my query fails on Tez," the maintainer's first job is layer attribution: is this Hive's compiler, Hive's runtime operator code, or Tez's shuffle? Get it wrong and you send the fix to the wrong project and burn a review cycle. The Vertex failed string the user pastes is a Tez diagnostic, but the cause is usually inside the vertex, in code Tez merely runs. Learning to read that string, drill to the task log, and land on the right layer is the difference between a triage comment that closes an issue and one that thrashes for a week.

Prerequisites

• Lab 6.1 complete — you can map vertices to operators.
• A working local Hive-on-Tez (same setup as Lab 6.1).
• yarn CLI available and log aggregation enabled.
• Tez checkout handy for grepping diagnostic strings.

Note: Every Tez diagnostic string in this lab is quoted from the Tez checkout by class, and a grep is given so you confirm it. The exit codes and Hive console text will vary by version — trust your run, use these as the map.

Step-by-Step Tasks

Step 1 — Reproduce a runtime task failure (missing UDF jar)

The cleanest reproducible Hive-on-Tez failure is a UDF whose class is not on the task classpath. Register a UDF pointing at a jar you will not add, then use it in a query that forces it to run inside a Tez vertex:

-- Point at a class that isn't on the cluster (no ADD JAR).
CREATE TEMPORARY FUNCTION myudf AS 'com.example.MissingUdf';

-- Force it to execute inside a Map vertex.
SELECT myudf(dept) FROM employees;

Depending on version this may fail at compile time (class resolved on the client) or at runtime (class resolved in the task). To reliably get a runtime failure, use a UDF that exists on the client but whose jar you did not distribute to the cluster, or induce a different runtime fault — a bad-data cast also works:

-- Runtime cast failure: a non-numeric value forced through a numeric cast.
CREATE TABLE dirty (s STRING) STORED AS ORC;
INSERT INTO dirty VALUES ('123'),('not_a_number'),('456');
SELECT CAST(s AS INT) + 1 FROM dirty;   -- one row will fail in the task

Run it and capture the exact Hive console output. You are looking for the shape:

FAILED: Execution Error, return code 2 from
  org.apache.hadoop.hive.ql.exec.tez.TezTask
Vertex failed, vertexName=Map 1, vertexId=vertex_..._1_00,
  diagnostics=[Task failed, taskId=task_..._1_00_000000,
  diagnostics=[TaskAttempt 0 failed, info=[ ... ]]]

That nested string is your whole map: TezTask (Hive) → Vertex failed (Tez DAGImpl) → Task failed (Tez VertexImpl) → TaskAttempt N failed (Tez TaskAttemptImpl).

Step 2 — Read the diagnostic string as a layer stack

The console message is a nesting of diagnostics from four classes across two projects. Confirm where each fragment is emitted:

# "Vertex failed" — emitted by the DAG state machine:
grep -rn "\"Vertex failed\"" \
  tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/DAGImpl.java

# "Task failed" — emitted by the vertex state machine:
grep -rn "\"Task failed\"" \
  tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/VertexImpl.java

In the Tez checkout, DAGImpl builds the Vertex failed, vertexName=..., vertexId=..., diagnostics=... string and VertexImpl builds the nested Task failed, taskId=..., diagnostics=.... The return code 2 from ... TezTask wrapper is Hive — it is TezTask surfacing the Tez DAGStatus to the user.

The innermost info=[...] is where the real cause lives — and it is almost always truncated in the console. That is why you go to the logs next.

Step 3 — Pull the task attempt log

The console shows the vertex; the cause is in the failed attempt's container log. Get the application id from the Hive console (application_..._NNNN) and:

# All AM + container logs for the app:
yarn logs -applicationId application_..._0001 > /tmp/app.log

# The AM's own view of the failure (vertex/task state transitions):
yarn logs -applicationId application_..._0001 -log_files syslog \
  | grep -E "Map 1|Task failed|TaskAttempt|diagnostics" | head -40

# The failing container's stdout/stderr/syslog (the real stack trace):
yarn logs -applicationId application_..._0001 \
  -containerId container_..._000002 | sed -n '1,200p'

In the container log you will find the actual Java exception — a ClassNotFoundException, a HiveException wrapping a NumberFormatException, or an OOM. This is the root cause; everything above was routing.

Note: The AM syslog and the container log are different files. The AM syslog tells you which task/attempt failed and how the state machine reacted; the container log tells you why. Read both, in that order.

Step 4 — Map the failed vertex back to the Hive operator tree

You have vertexName=Map 1 and a stack trace. Now close the loop: which Hive operator inside Map 1 threw? Re-run the EXPLAIN from Lab 6.1's technique:

EXPLAIN SELECT CAST(s AS INT) + 1 FROM dirty;

Map 1 contains TableScan → Select. The cast lives in the Select operator's expression — so a NumberFormatException from Map 1 maps to the Select operator evaluating CAST(s AS INT). For the UDF case, the failing operator is the one containing the GenericUDF invocation. Write the mapping down:

Vertex "Map 1"  ─▶  MapTezProcessor
                     └─ TableScan(dirty)
                        └─ Select: CAST(s AS INT) + 1   ← threw NumberFormatException

That single arrow — from a Tez vertex name to a Hive operator — is the deliverable of this lab. It is exactly what a maintainer writes in a triage comment.

Step 5 — Attribute the layer and route the fix

For your reproduced failure, fill the row that decides the whole ticket:

The subtle, contributor-grade insight: a runtime data error is not a Tez bug, but "the diagnostic didn't tell me which row / which operator" often is a legitimate Tez or Hive diagnostics-improvement PR. Knowing the difference is the value you add.

Step 6 — Build the failure taxonomy table

Generalise. Produce this table (fill the "where to look" column with the exact command you would run), which becomes your personal runbook:

Verify the two runtime Tez strings you will lean on most:

grep -rn "\"Vertex failed\"" tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/DAGImpl.java
grep -rn "\"Task failed\""   tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/VertexImpl.java

Step 7 — The memory-failure variant (worked)

The most common real Hive-on-Tez production failure is a reducer OOM. Its console signature is exitCode: -104 (YARN killed the container for exceeding its memory request). Walk the same chain, but note the fix is a config, not code:

Confirm the Tez task-memory key is real:

grep -rn "task.resource.memory.mb" \
  tez-api/src/main/java/org/apache/tez/dag/api/TezConfiguration.java

The diagnostic chain is identical — TezTask → Vertex failed → Task failed → TaskAttempt failed, exitCode -104 — but layer attribution says: not a code bug, a resourcing bug, fixed by raising hive.tez.container.size and re-running. Add this as a second runbook entry.

Deliverables

• The captured Hive console diagnostic for your reproduced failure.
• The AM syslog excerpt (Vertex failed / Task failed) and the container log excerpt (the real stack trace), both saved.
• The vertex → operator mapping diagram (Step 4).
• The completed layer-attribution row (Step 5).
• The failure taxonomy table (Step 6) with real where-to-look commands.
• A one-paragraph runbook entry for the -104 reducer OOM (Step 7).

Troubleshooting

Stretch Goals

1. Improve a diagnostic. Read VertexImpl's Task failed string. Could it include the operator or input name? Sketch (do not necessarily submit) a patch that enriches it, and identify the test in tez-dag/src/test/.../dag/impl/ you would extend. This is a real class of accepted Tez PR.
2. Force a DAG-submit failure. Kill the Tez session mid-query, or submit with a bad tez.lib.uris, and capture the submit-layer diagnostic. Contrast its shape with the runtime Vertex failed — a different class emits it.
3. Trace a shuffle failure now. Induce a fetch failure (you will do this properly in Lab 7.1) and note that its diagnostic mentions InputReadErrorEvent — a shuffle-layer signature absent from the pure runtime-task failures above.

Validation

1. Name the four failure layers and one console signature that distinguishes each.
2. In Vertex failed, vertexName=Map 1, diagnostics=[Task failed, ...], which Tez class emits the outer Vertex failed and which emits the nested Task failed? Which project emits the return code 2 from ... TezTask wrapper?
3. Why must you read the container log and not just the AM syslog to find the root cause?
4. Given vertexName=Map 1 and a NumberFormatException, which Hive operator most likely threw, and how did you determine that?
5. Is a ClassNotFoundException from a missing UDF jar a Tez bug? Justify the layer attribution.
6. What does exitCode: -104 mean, and is the fix code or configuration?
7. For a fetch-failure diagnostic, which level of this apprenticeship do you escalate to, and what string in the diagnostic told you it was shuffle?

When you can attribute a Vertex failed string to its layer and walk it to the Hive operator, you are ready for Level 7 — Runtime and Shuffle, where you finally drop below the seam and modify the shuffle code itself. For the full-depth version of this lab, see Lab H3: Debug a Query.