Hive-on-Tez Labs

Hive on Tez is the production context that has carried Tez through the last decade. Every large Hive deployment that isn't on Spark is on Tez. Understanding the Tez/Hive boundary is therefore not a niche skill — it is the production debugging skill for both projects, and it is the skill that most reliably turns a curious reader into a committer whose JIRAs get merged.

These six labs (H1–H6) work from a SQL query down through Hive compilation, into a Tez DAG, into running tasks, and back out through failure attribution and remediation. They are deliberately hands-on: every step has commands to run against your Tez checkout (~/tez-src) and your Hive checkout (~/hive-src). They build directly on the Hive Integration deep dive — read that first for the conceptual map; these labs make it muscle memory. The gentler on-ramp is Level 6, which introduces the SQL-to-DAG pipeline before you attempt the full attribution and reproduction workflow here.

Why integration bugs are the hardest — and the most valuable

A single-repo contributor can read all of server/ or all of tez-dag/ and still be useless on the issues users actually file, because those issues live at the boundary between Hive and Tez, not inside either project:

• A query returns the wrong row count. Is it a Hive operator-semantics bug, a Tez shuffle/partitioner bug, or an auto-parallelism interaction that changed the partitioning?
• A vertex hangs in INITIALIZING. Is Hive's DynamicPartitionPruner waiting for an event, or did Tez's RootInputInitializerManager mis-route it?
• A container OOMs during a join. Is Hive sizing the hash table wrong, or is Tez failing to honor a memory request?

None of these can be answered from one repository. Integration bugs are hard for three compounding reasons:

1. The stack trace lies about ownership. A MapJoinOperator OOM is rooted in a TezChild JVM frame; a wrong result surfaces in Hive output but may originate in a Tez edge manager. The top frame names the scene of the crime, not the culprit.
2. State crosses the boundary invisibly. HiveServer2's session pool keeps Tez AMs and their containers alive across queries and even across users. Bugs that only appear under pooling never reproduce in a standalone hive CLI run.
3. Two release trains. Hive builds against published Tez artifacts, so a Tez change can break Hive with zero failing tests in the Tez repo, and vice versa. The two projects version-skew constantly in the field.

That difficulty is exactly why the skill is valuable. The contributor who can say "this is a Tez cartesianproduct edge-manager bug, here is a 30-line reproducer DAG with no Hive on the classpath, and here is why it is not a Hive bug" is worth ten who can only say "the join is broken." The Tez/Hive boundary is one of the most-asked-about areas on both project mailing lists, and a crisp attribution is the single highest-leverage thing you can attach to a report.

Prerequisites

A working Hive-on-Tez setup is the shared prerequisite for all six labs. Lab H1 walks you through standing one up end to end; the later labs assume it exists. At minimum you need:

If you only have one of Hive 3 vs Hive 4, the labs work either way — they call out the delta where it matters. The integration boundary is a small, stable set of Hive classes; verify they exist in your tree before you start:

find ~/hive-src -path "*ql/exec/tez/TezTask.java"
find ~/hive-src -path "*ql/exec/tez/DagUtils.java"
find ~/hive-src -path "*ql/exec/tez/TezSessionPoolManager.java"
find ~/hive-src -path "*ql/exec/tez/TezSessionState.java"
find ~/hive-src -path "*ql/exec/tez/MapRecordSource.java"
find ~/hive-src -path "*ql/exec/tez/ReduceRecordSource.java"

org.apache.hadoop.hive.ql.exec.tez.TezTask                  — Hive's "execute on Tez" task
org.apache.hadoop.hive.ql.exec.tez.DagUtils                 — Builds Tez DAG from Hive plan
org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager    — Pools Tez sessions
org.apache.hadoop.hive.ql.exec.tez.TezSessionState          — One Hive session = one Tez AM
org.apache.hadoop.hive.ql.exec.tez.MapRecordSource          — Map-side record source
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource       — Reduce-side record source

If any are missing, your Hive tree may be too old. Hive 3.1.x and 4.0.x both have all six. On the Tez side, the labs lean on classes you can confirm now:

grep -n "public synchronized DAG addVertex" ~/tez-src/tez-api/src/main/java/org/apache/tez/dag/api/DAG.java
grep -n "class ShuffleVertexManager" \
  ~/tez-src/tez-runtime-library/src/main/java/org/apache/tez/dag/library/vertexmanager/ShuffleVertexManager.java
ls ~/tez-src/tez-runtime-library/src/main/java/org/apache/tez/runtime/library/cartesianproduct/

Two failure classes you will learn to separate

Almost every Hive-on-Tez bug falls into one of two classes, and the labs teach you to tell them apart before you write a single line of a report:

• Crashes (an exception, a killed vertex, an OOM). These surface as a "Vertex failed" message with a stack trace. The trap is that the top frame names the container's current activity, not the code that is wrong. A MapJoinOperator OOM crashes in a TezChild JVM but is a Hive hash-table-sizing bug; a fetch failure crashes in a Tez Fetcher frame but is often a YARN aux-service deployment problem. H3 and H4 exist to push you past the top frame to the owning project.
• Wrong results (no crash at all). The query succeeds and returns bad data. These are the more dangerous class, because nothing alerts on them. They almost always live in Hive operator semantics — unless you can reproduce the same DAG shape with synthetic data and no Hive classes on the classpath, at which point it becomes a Tez shuffle, partitioner, or edge-manager bug. H5 is entirely about building that Hive-free reproducer, because it is the only proof that separates the two.

Holding both classes in mind is what stops you filing a Tez JIRA for a Hive bug or a core Hive JIRA for a Tez shuffle regression. Every lab below is built to sharpen exactly that judgment.

The Tez/Hive boundary, at a glance

The boundary is one Hive class — TezTask — and a handful of supporting utilities. Above the boundary, Hive owns: SQL parsing, semantic analysis, logical plan, physical plan (MapWork/ReduceWork). Below the boundary, Tez owns: DAG execution, task scheduling, shuffle, recovery.

flowchart TD
  subgraph Hive
    A[SQL Query] --> B[Parser]
    B --> C[Semantic Analyzer]
    C --> D[Logical Plan]
    D --> E[Physical Plan<br/>MapWork / ReduceWork]
    E --> F[TezTask.execute]
    F --> G[DagUtils.createVertex<br/>DagUtils.createEdge]
    G --> H[DAG object]
  end
  subgraph Tez
    H --> I[TezSession.submitDAG]
    I --> J[DAGAppMaster<br/>tez-dag]
    J --> K[Vertex tasks<br/>tez-runtime-internals]
    K --> L[Shuffle I/O<br/>tez-runtime-library]
  end

That TezTask → DagUtils → DAG → submitDAG sequence is the entire integration surface. The six labs below walk it from the top (Lab H1) to the runtime (Lab H6).

Lab index

Skills matrix

Each lab is chosen to teach one diagnostic skill you cannot get from reading alone:

Reading order

H1 and H2 are foundational — do them in order. H3 and H4 are the debugging-and-attribution pair that build on each other: H3 teaches you to read a cross-stack failure, H4 teaches you to attribute it. H5 and H6 are the contributor-facing skills you need to file a useful Hive-on-Tez JIRA from a production observation — turning an attributed bug into a minimal reproducer (H5) and into a diagnostics patch that helps the next person (H6).

If you are arriving here from the Capstone, H4 and H5 are the most directly relevant. If you have not yet read the Hive Integration deep dive, read it first — these labs assume you know how TezTask hands a DAG to TezClient.submitDAG and how the session pool keeps AMs warm.

Where the real work happens

The labs are written so that, when you encounter a production issue, you can execute one crisp routine:

1. Read the stack trace and attribute it (H4).
2. Locate the SQL that produced the DAG (H1).
3. Capture the DAG and find the relevant vertex (H2).
4. Identify the failing task and its log (H3).
5. Reproduce it minimally on MiniTezCluster (H5).
6. Attach a diagnostic patch to a JIRA to get more data from the reporter (H6).

That six-step routine, executed crisply, is what gets Hive-on-Tez JIRAs resolved. Notice that it deliberately front-loads attribution (step 1) before location and reproduction: you decide which project owns the bug first, so every subsequent step is spent in the right codebase. A misattribution at step 1 wastes every step after it — which is why H4, the attribution lab, is the hinge the whole section turns on.

Validation for the section

You have absorbed the Hive-on-Tez section when, given a freshly-failing query in a production Hive-on-Tez deployment, you can:

1. Within 10 minutes, identify which project owns the failure (Hive / Tez / YARN).
2. Within 30 minutes, locate the relevant code on both sides of the boundary.
3. Within 1 hour, capture the DAG and the failing task's log.
4. Within a day, produce a minimum reproducer on MiniTezCluster.
5. Within a week, file a JIRA on the right project with all the data needed.

That is the standard a Hive-on-Tez committer holds themselves to. The labs build the muscle.