Level 6: Hive/Tez Integration

Hive-on-Tez is the single largest consumer of the Tez API in existence. The overwhelming majority of Tez bug reports that reach the mailing list — a slow query, a shuffle fetch failure, a reducer OOM, a counter that reads wrong — originate as a Hive query. If you want to fix the bugs real operators file, you must be able to stand at the boundary between the two projects and say, with confidence, "this half is Hive's job, that half is Tez's job, and the symptom lives here."

This level is the on-ramp to that skill. It teaches you the layering end-to-end — HiveQL down to a submitted Tez DAG — and it teaches you to read a Hive-on-Tez failure like a stack of layers, peeling from the console message down to the AM diagnostics down to the task logs. The deeper, PR-grade treatment lives in the hive-on-tez-labs section; this level positions you to work through it.

Learning Objectives

By the end of Level 6 you must be able to:

1. Draw the full translation pipeline — HiveQL → SemanticAnalyzer → TezCompiler → TezWork → DagUtils → Tez DAG API → TezClient.submitDAG — and name, for each hop, the class that owns it and which project it belongs to.
2. State precisely what is Hive's job and what is Tez's job: Hive owns compilation, the operator tree, physical Work units, and DAG construction; Tez owns DAG execution — scheduling, vertices, edges, shuffle, and the ShuffleVertexManager that Hive attaches.
3. Map an EXPLAIN plan's vertices (Map 1, Reducer 2) onto the Tez DAGImpl runtime objects of the same name, and back again.
4. Take a Hive-on-Tez failure and locate its layer — compile-time, DAG-submit, runtime task, or shuffle — from the shape of the diagnostic string alone.
5. Read a Vertex failed, vertexName=... diagnostic emitted by Tez and walk it back to the Hive operator tree that produced that vertex.
6. Decide, for a given fix, whether the patch belongs in Tez or in Hive — and articulate why.

What Hive Does With Tez, and Where the Seam Is

Every Hive query that runs on the Tez engine (SET hive.execution.engine=tez) travels this path:

flowchart TD
    SQL["HiveQL text"] --> AST["ParseDriver → AST (ANTLR)"]
    AST --> SA["SemanticAnalyzer<br/>AST → QB → Operator tree (logical plan)"]
    SA --> TC["TezCompiler<br/>logical → physical: MapWork / ReduceWork"]
    TC --> TW["TezWork<br/>graph of BaseWork nodes + edges"]
    TW --> DU["DagUtils.createVertex / createEdge<br/>TezWork → Tez DAG objects"]
    DU --> API["Tez DAG API<br/>DAG / Vertex / Edge / VertexManagerPluginDescriptor"]
    API --> SUB["TezTask → TezClient.submitDAG(DAG)"]
    SUB --> EXEC["Tez AM: DAGImpl / VertexImpl / TaskImpl<br/>schedule, shuffle, run"]

    subgraph HIVE["Hive (hive-exec / ql)"]
      AST
      SA
      TC
      TW
      DU
      SUB
    end
    subgraph TEZ["Tez (tez-api / tez-dag / tez-runtime-library)"]
      API
      EXEC
    end

Read the seam carefully, because it is the whole point of this level:

• Hive owns everything above the API call. Parsing, semantic analysis, the operator tree, the physical plan (MapWork/ReduceWork), and the construction of the DAG object — assembling vertices, wiring edges, choosing the ShuffleVertexManager, and setting the payload that controls auto-parallelism — are all Hive code. DagUtils is the factory; it produces Tez API objects but it is a Hive class.
• Tez owns everything below the API call. Once TezClient.submitDAG(dag) returns a DAGClient, Hive is a spectator. The Tez AM (DAGImpl, VertexImpl, TaskImpl, TaskAttemptImpl) schedules and runs the DAG; tez-runtime-library executes the shuffle; the vertices are named exactly what Hive named them (Map 1, Reducer 2).

The consequence is a rule you will use constantly when triaging bugs: if the wrong thing happened during compilation or DAG construction, it is a Hive bug; if the wrong thing happened during execution of a correctly-built DAG, it is a Tez bug. A GROUP BY that produced the wrong number of reducers may be a Hive planning bug or a Tez ShuffleVertexManager reconfiguration bug — and telling them apart is exactly the discrimination this level trains.

How This Level Relates to hive-on-tez-labs

This level is deliberately smaller than the full Hive-on-Tez labs. Think of the relationship this way:

Do the two labs here first. They give you the vocabulary — TezWork, DagUtils, vertex names, the diagnostic stack — that the deeper section assumes. When you finish, go to Lab H1: SQL → DAG and work forward.

Required Reading

Before the labs, read (or skim and bookmark):

• Deep dive: Hive integration — the architecture of the seam.
• Hive-on-Tez Labs — overview — where this level hands off.
• Lab H1: SQL → DAG and Lab H3: Debug a Query — the full versions of this level's two labs.
• This level's labs:
  • Lab 6.1 — Trace a Hive SQL Query to the Generated Tez DAG
  • Lab 6.2 — Debug a Failed Hive-on-Tez Query

Source Areas

Hive side — verify in your own Hive checkout

Hive source is not part of your Tez checkout. Clone Hive alongside it and use grep/find to confirm every class before you cite it — Hive moves classes between releases, so trust your checkout, not this table.

# In your Hive checkout, confirm the classes exist before trusting any claim:
for c in SemanticAnalyzer TezCompiler TezWork DagUtils TezTask; do
  find . -name "$c.java" -path "*org/apache/hadoop/hive*"
done

Tez side — in your Tez checkout

These are the Tez classes Hive drives. Cite them by module + class, never by line number.

# In your Tez checkout, confirm the auto-parallelism config key is real:
grep -rn "enable.auto-parallel" \
  tez-runtime-library/src/main/java/org/apache/tez/dag/library/vertexmanager/ShuffleVertexManager.java

Key Classes — Quick Reference

Note: The vertex names in the Tez runtime — Map 1, Reducer 2 — are strings chosen by Hive in DagUtils. That is why a Tez-side diagnostic like vertexName=Reducer 2 can be read straight off a Hive EXPLAIN plan. This naming coincidence is your single most useful debugging bridge.

Deliverables

Before advancing to Level 7, you must produce:

• A labelled diagram of one real query (a GROUP BY) traced from HiveQL to a submitted Tez DAG, with every vertex, edge type, and the ShuffleVertexManager on the reduce vertex — attributed to the Hive class that created each piece (Lab 6.1).
• The EXPLAIN output for that query with each vertex line mapped to a Tez runtime vertex of the same name, confirmed against the DAG summary (hive.tez.exec.print.summary=true).
• A failure-triage writeup: given a Vertex failed diagnostic, the layer it belongs to, the exact log you would pull next (console → AM → yarn logs), and the Hive operator subtree behind the named vertex (Lab 6.2).
• A one-paragraph, from-memory statement of the Hive/Tez seam: what each project owns and where submitDAG divides them.

Common Mistakes

How to Verify Success

You are ready to move on when you can, without notes:

# 1) Point at the seam in the Tez checkout — the method Hive calls.
grep -rn "public.*submitDAG" tez-api/src/main/java/org/apache/tez/client/TezClient.java

# 2) Show the Tez-side diagnostic strings you will read in Lab 6.2.
grep -rn "\"Vertex failed\"" tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/DAGImpl.java
grep -rn "\"Task failed\""   tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/VertexImpl.java

# 3) Show the reduce-vertex plugin Hive attaches, and its auto-parallelism flag.
grep -rn "enable.auto-parallel" \
  tez-runtime-library/src/main/java/org/apache/tez/dag/library/vertexmanager/ShuffleVertexManager.java

If each command lands where you expect and you can narrate why — that submitDAG is the boundary, that DAGImpl/VertexImpl emit the diagnostics Hive surfaces, and that the reduce plugin is Hive-attached but Tez-executed — you understand the seam.

PR Profile: Level 6 Graduate

A contributor who has completed this level can credibly open or review these kinds of PRs:

• Triage a "Hive query is slow/broken" report and correctly route it: "this is a Hive planning issue, closing as not-a-Tez-bug" or "this is a Tez ShuffleVertexManager reconfiguration bug, here is the failing vertex."
• Add or improve a diagnostic in DAGImpl/VertexImpl so the string Hive surfaces to the user names the offending vertex and cause more precisely — the kind of small, high-value change reviewers love.
• Reproduce a Hive-reported bug as a pure Tez DAG, stripping Hive out of the picture so the fix and its test live entirely in Tez (the skill the hive-on-tez-labs section makes production-grade).
• Review a Hive-side DagUtils change for its Tez consequences: does it set a legal EdgeProperty? does its ShuffleVertexManager payload parse?

You cannot yet write the runtime-library fix that a shuffle bug demands — that is Level 7, where you drop below the seam and modify the shuffle and processor code itself. But you can now find which side of the seam any Hive-on-Tez bug lives on, which is where every one of those PRs begins.

Next: Level 7 — Runtime and Shuffle →