Hive on Tez

Hive is the largest single consumer of Tez. Roughly 70% of bug reports filed against Tez originate in a Hive query; many "Tez bugs" turn out to be Hive bugs, and vice versa. A Tez committer who cannot read across the Hive/Tez boundary will misattribute failures, close valid reports, and miss the real regressions. This chapter walks the full layering from SQL to a running DAG, the session machinery HiveServer2 uses to keep AMs warm, and — critically — the Tez features Hive leans on hardest, each verified against the Tez source you have checked out.

A note on method: this book assumes a Tez checkout at ~/tez-src (all Tez commands below run from its root and are verified against current master). Hive source is not assumed present in a fixed state — Hive-side claims are explicitly marked [Hive-side] and always paired with a grep to run in your own Hive checkout (~/hive-src). If a grep comes up empty, your Hive version differs; trust the grep, not this chapter.

After this chapter you can:

• Name every layer between a SQL string and TezClient.submitDAG, and say which project owns each.
• Explain how HiveServer2's session pool maps onto Tez sessions and AMs.
• Point at the exact Tez classes Hive uses for auto-parallelism, cross joins, dynamic partition pruning, and unions — in the Tez tree.
• Attribute a Hive-on-Tez failure to Hive, Tez, or YARN from a stack trace.

The layering: SQL → DAG

[Hive-side] Hive's compile pipeline for Tez execution:

SQL string
  → ParseDriver (AST)
  → SemanticAnalyzer (operator tree)
  → TezCompiler (physical optimization for Tez)
  → TezWork: a graph of BaseWork nodes (MapWork, ReduceWork, MergeJoinWork, UnionWork)
  → TezTask.execute: DagUtils turns TezWork into a Tez DAG
  → TezSessionState → TezClient.submitDAG

Everything above TezTask is pure Hive: parsing, semantic analysis, logical and physical optimization. TezCompiler is the Tez-specific physical compiler — it decides shuffle boundaries, join algorithms, and which Tez edge type each BaseWork connection becomes. Verify the pipeline in your Hive tree:

grep -rn "class TezCompiler" ~/hive-src/ql/src/java --include=*.java
grep -rn "class TezWork\|class BaseWork" ~/hive-src/ql/src/java/org/apache/hadoop/hive/ql/plan/ | head
ls ~/hive-src/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/

Key files at the boundary:

flowchart TD
  subgraph Hive["Hive (ql module)"]
    A[SQL] --> B[SemanticAnalyzer<br/>operator tree]
    B --> C[TezCompiler<br/>physical plan]
    C --> D[TezWork<br/>MapWork / ReduceWork / MergeJoinWork]
    D --> E[TezTask.execute]
    E --> F[DagUtils.createDag<br/>createVertex / createEdge]
  end
  subgraph TezAPI["Tez public API (tez-api)"]
    F --> G[DAG.create + Vertex.create + Edge.create]
    G --> H[TezClient.submitDAG]
  end
  subgraph TezDAG["Tez AM (tez-dag)"]
    H --> I[DAGAppMaster]
    I --> J[Vertices, tasks, containers]
  end

The DAG-construction surface Hive calls into is exactly the public API you studied in The DAG Model and TezClient: DAG.addVertex, DAG.addEdge, Vertex.setVertexManagerPlugin, Vertex.setConf, DAG.setCallerContext — all in tez-api/src/main/java/org/apache/tez/dag/api/. Confirm the surface from the Tez side:

grep -n "public synchronized DAG addVertex\|public synchronized DAG setCallerContext" \
  tez-api/src/main/java/org/apache/tez/dag/api/DAG.java
grep -n "public Vertex setVertexManagerPlugin\|public Vertex setConf" \
  tez-api/src/main/java/org/apache/tez/dag/api/Vertex.java

[Hive-side] Find the single call site where Hive hands over control:

grep -rn "submitDAG" ~/hive-src/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ | head

Vertex names: "Map 1", "Reducer 2"

[Hive-side] The vertex names you see in Hive's console output (Map 1: 42/42 Reducer 2: 1/1), in EXPLAIN output, and in the Tez UI are assigned by Hive when TezWork is built — each BaseWork gets a name like Map 1 or Reducer 2, and DagUtils passes that string straight into Vertex.create. Tez treats the name as an opaque identifier: it keys vertex lookups, appears in AM logs (Vertex vertex_... [Map 1]), and names the vertex in ATS events. There is no semantic mapping to recover — the name in a Tez AM log is the name in the Hive plan.

# Hive side: where the names are minted
grep -rn "\"Map \"\|\"Reducer \"" ~/hive-src/ql/src/java/org/apache/hadoop/hive/ql/plan/TezWork.java

# Tez side: the name is just the Vertex identity
grep -n "vertexName" tez-api/src/main/java/org/apache/tez/dag/api/Vertex.java | head -5

Practical consequence: when a user reports "Reducer 2 failed", EXPLAIN the query and read the operator tree under Reducer 2 — that tells you which Hive operators were executing inside the failing Tez tasks.

TezTask.execute — high-level flow

[Hive-side]

grep -n "public int execute\|build(\|submit(" \
  ~/hive-src/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezTask.java | head -20

1. Acquire a TezSessionState from the pool (or open a new one).
2. build(...) — DagUtils turns each BaseWork into a Tez Vertex and each TezEdgeProperty into a Tez Edge; localizes added jars/files.
3. submit(...) → tezClient.submitDAG(dag).
4. Poll dagClient.getDAGStatus(...) until terminal, printing progress.
5. Surface Tez counters and diagnostics back into Hive's result state.

Two DAG-level things Hive sets that matter for debugging, both plain tez-api calls you can verify from the Tez side:

• Caller context. DAG.setCallerContext(CallerContext) — Hive stamps the query ID and query text so YARN and ATS can trace a DAG back to its SQL. CallerContext lives in tez-api (org.apache.tez.client.CallerContext).
• ACLs. DAGAccessControls (org.apache.tez.common.security.DAGAccessControls in tez-api) — who may view/modify the DAG via DAGClient.

grep -n "class CallerContext" tez-api/src/main/java/org/apache/tez/client/CallerContext.java
grep -n "class DAGAccessControls" tez-api/src/main/java/org/apache/tez/common/security/DAGAccessControls.java
# [Hive-side] confirm Hive sets both:
grep -rn "setCallerContext\|DAGAccessControls" ~/hive-src/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ | head

What Hive configures on a DAG

[Hive-side] Hive copies a large set of tez.* keys from hive-site.xml/session config into the DAG's configuration. The ones that matter most in production, each a real key in tez-api/.../TezConfiguration.java (grep the constant to see docs and defaults):

grep -n "TEZ_QUEUE_NAME\|TEZ_AM_STAGING_DIR\b\|TEZ_LIB_URIS\b\|TEZ_AM_CONTAINER_REUSE_ENABLED\|TEZ_SESSION_AM_DAG_SUBMIT_TIMEOUT_SECS\|TEZ_AM_SESSION_MODE\b" \
  tez-api/src/main/java/org/apache/tez/dag/api/TezConfiguration.java

Per-vertex settings go through Vertex.setConf(String, String) — a real method on tez-api's Vertex — which is how Hive applies, e.g., different sort buffer sizes to map and reduce vertices.

Tez features Hive relies on — verified from the Tez side

This is the heart of the chapter. Each subsection names the Tez classes in your checkout, then the Hive grep that shows the usage.

ShuffleVertexManager and auto reducer parallelism

tez-runtime-library, package org.apache.tez.dag.library.vertexmanager:

ls tez-runtime-library/src/main/java/org/apache/tez/dag/library/vertexmanager/
grep -n "class ShuffleVertexManager" \
  tez-runtime-library/src/main/java/org/apache/tez/dag/library/vertexmanager/ShuffleVertexManager.java

ShuffleVertexManager extends ShuffleVertexManagerBase. Its contract: a reduce-side vertex starts with a pessimistic (large) parallelism; as source tasks finish, they emit VertexManagerEvents carrying output-size stats; the manager sums them and, once enough sources have reported, shrinks the vertex via VertexManagerPluginContext.reconfigureVertex. The knobs, from ShuffleVertexManager (module tez-runtime-library, class org.apache.tez.dag.library.vertexmanager.ShuffleVertexManager):

public static final String TEZ_SHUFFLE_VERTEX_MANAGER_ENABLE_AUTO_PARALLEL =
    "tez.shuffle-vertex-manager.enable.auto-parallel";

plus tez.shuffle-vertex-manager.desired-task-input-size, tez.shuffle-vertex-manager.min-src-fraction, tez.shuffle-vertex-manager.max-src-fraction, and tez.shuffle-vertex-manager.min-task-parallelism. The decision logic is determineParallelismAndApply(...) in ShuffleVertexManagerBase — read it; it is the single most production-relevant algorithm in the runtime library.

[Hive-side] hive.tez.auto.reducer.parallelism=true makes DagUtils install ShuffleVertexManager with auto-parallel enabled on reduce vertices:

grep -rn "ShuffleVertexManager" ~/hive-src/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java
grep -rn "hive.tez.auto.reducer.parallelism" ~/hive-src/common/src ~/hive-src/ql/src | head

Symptom signature: reducer counts in the Tez UI that differ from the EXPLAIN plan's reducer counts are normal with auto-parallelism on — not a bug. Wrong results after a parallelism change, however, mean a partitioner/edge-manager interaction bug: attribute to Tez.

Cross joins: the CartesianProduct edge

tez-runtime-library, package org.apache.tez.runtime.library.cartesianproduct (added under TEZ-3230, extended by TEZ-3654 and TEZ-3708 — verify with git log --oneline --grep=TEZ-3230):

ls tez-runtime-library/src/main/java/org/apache/tez/runtime/library/cartesianproduct/

The two classes that matter most:

public class CartesianProductVertexManager extends VertexManagerPlugin {

public class CartesianProductEdgeManager extends EdgeManagerPluginOnDemand {

(both attributed: tez-runtime-library, org.apache.tez.runtime.library.cartesianproduct), plus CartesianProductConfig, which serializes the partitioned/fair choice and source-vertex list into the plugin payloads. This is Tez's only in-tree production user of EdgeProperty.DataMovementType.CUSTOM — the edge manager computes, for every (task-in-A, task-in-B) combination, which destination task consumes it, so a cross join needs no shuffle-by-key at all. FairCartesianProductVertexManager handles the unpartitioned case with output-size-aware grouping.

[Hive-side] Hive uses this for cross joins when hive.tez.cartesian-product.enabled=true:

grep -rn "CartesianProduct" ~/hive-src/ql/src/java/org/apache/hadoop/hive/ql | head

Dynamic partition pruning: InputInitializer events

The Tez-side API surface, all in tez-api:

• org.apache.tez.runtime.api.InputInitializer — runs in the AM to generate splits/events for a root input. The contract (module tez-api, class org.apache.tez.runtime.api.InputInitializer):

public abstract List<Event> initialize()
    throws Exception;

public abstract void handleInputInitializerEvent(List<InputInitializerEvent> events)
    throws Exception;

• org.apache.tez.runtime.api.events.InputInitializerEvent — created via InputInitializerEvent.create(targetVertexName, targetInputName, payload); sent by a running task in one vertex, routed by the AM to the InputInitializer of another vertex's root input.
• InputInitializerContext.registerForVertexStateUpdates(...) — lets the initializer defer until upstream vertices reach a chosen state.

Routing on the AM side is RootInputInitializerManager.handleInitializerEvents in tez-dag (org.apache.tez.dag.app.dag.RootInputInitializerManager):

grep -n "handleInitializerEvents\|handleInputInitializerEvents" \
  tez-dag/src/main/java/org/apache/tez/dag/app/dag/RootInputInitializerManager.java

[Hive-side] Hive's implementation: HiveSplitGenerator (an InputInitializer subclass) delays split generation for a fact table; map tasks scanning the dimension table evaluate the join-key filter and emit InputInitializerEvents carrying the surviving partition values; DynamicPartitionPruner (running inside HiveSplitGenerator in the AM) collects them and prunes fact-table partitions before splits are computed. That is why DPP can eliminate whole partitions with zero I/O.

grep -rn "class HiveSplitGenerator\|class DynamicPartitionPruner" \
  ~/hive-src/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/
grep -rn "InputInitializerEvent" ~/hive-src/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ | head

sequenceDiagram
  participant DimMap as Dim-table map task
  participant AM as Tez AM (RootInputInitializerManager)
  participant Init as HiveSplitGenerator (InputInitializer)
  participant Fact as Fact-table vertex
  DimMap->>AM: InputInitializerEvent(target="Map 1", input="fact", payload=keys)
  AM->>Init: handleInputInitializerEvent(events)
  Init->>Init: DynamicPartitionPruner drops partitions
  Init->>AM: initialize() returns split events
  AM->>Fact: parallelism + splits set, vertex starts

Failure signature: a query hangs with a vertex in INITIALIZING forever — the initializer is waiting for events that will never arrive (upstream vertex failed or the event was mis-targeted). Check AM logs for RootInputInitializerManager and the pruner's expected-event counters.

UNION: vertex groups

Tez-side: DAG.createVertexGroup, GroupInputEdge (tez-api, org.apache.tez.dag.api.GroupInputEdge) and ConcatenatedMergedKeyValuesInput (tez-runtime-library, org.apache.tez.runtime.library.input) — a union of N source vertices feeds one consumer as a single logical input.

ls tez-api/src/main/java/org/apache/tez/dag/api/ | grep -i group
ls tez-runtime-library/src/main/java/org/apache/tez/runtime/library/input/ | grep -i concat
# [Hive-side]
grep -rn "VertexGroup\|GroupInputEdge" ~/hive-src/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java | head

Container pre-warming

Tez-side: TezClient.preWarm(PreWarmVertex) — a real method; PreWarmVertex lives in tez-api (org.apache.tez.dag.api.PreWarmVertex). It runs a trivial DAG whose only purpose is to make the AM allocate and hold containers so the first real query skips allocation latency.

grep -n "public synchronized void preWarm" tez-api/src/main/java/org/apache/tez/client/TezClient.java
# [Hive-side]
grep -rn "hive.prewarm.enabled\|preWarm" ~/hive-src/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ | head

Operator → IPO mapping

[Hive-side] Hive operators run inside a Tez task — they are not Tez constructs. The mapping happens at the vertex I/O boundary, and every Tez class in this table exists in tez-runtime-library or tez-mapreduce:

ls tez-runtime-library/src/main/java/org/apache/tez/runtime/library/input/ \
   tez-runtime-library/src/main/java/org/apache/tez/runtime/library/output/ \
   tez-mapreduce/src/main/java/org/apache/tez/mapreduce/input/ \
   tez-mapreduce/src/main/java/org/apache/tez/mapreduce/output/
# [Hive-side] which of these Hive actually names:
grep -rn "OrderedPartitionedKVOutput\|OrderedGroupedKVInput\|UnorderedKVOutput\|UnorderedKVInput" \
  ~/hive-src/ql/src/java/org/apache/hadoop/hive/ql/exec/tez | head

The edge types Hive maps its TezEdgeProperty onto are the four members of EdgeProperty.DataMovementType in tez-api: ONE_TO_ONE, BROADCAST, SCATTER_GATHER, CUSTOM (cartesian product uses CUSTOM).

TezProcessor itself is intentionally thin — an adapter that drains Tez readers into Hive's Operator.process(Object row, int tag) convention and closes outputs. All the interesting logic is the Hive operator chain; all the interesting data movement is the Tez I/O classes above. When a stack trace crosses from org.apache.hadoop.hive.ql.exec frames into org.apache.tez.runtime.library frames, you are looking at exactly this boundary.

TezSessionState and the session pool

[Hive-side] A Tez session (session-mode TezClient, see TezClient) keeps one AM alive across DAG submissions. HiveServer2 multiplies this: TezSessionPoolManager maintains a pool of warm sessions so a query never pays AM startup latency.

Pool flow:

1. HS2 starts; optionally launches N session AMs per queue.
2. A query arrives; HS2 borrows an idle TezSessionState (or opens one).
3. TezTask submits its DAG to the session's AM; the AM holds containers across DAGs (Tez-side: Container Reuse).
4. Session returned to the pool; the AM sits idle awaiting the next DAG.
5. If no DAG arrives within tez.session.am.dag.submit.timeout.secs (Tez-side constant TEZ_SESSION_AM_DAG_SUBMIT_TIMEOUT_SECS), the AM shuts itself down — the pool must then reopen the session.

grep -rn "getSession\|returnSession" \
  ~/hive-src/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java | head

Debugging consequence: session pooling means AM state (cached objects, localized resources, container-held tokens) survives across queries from different users. A class of "works standalone, fails under HS2" bugs comes from exactly this reuse. Lab H3 makes you chase one.

A note on LLAP: Hive's LLAP mode replaces YARN containers with long-lived daemons; the Tez AM still coordinates but uses a different task-scheduler plugin (Hive-side LlapTaskSchedulerService). Do not extrapolate container-based debugging to LLAP; this book treats it as out of scope.

Bug attribution: where does it really live?

flowchart TD
  S[Failure observed] --> Q1{Top frames in<br/>org.apache.hadoop.hive.ql.exec?}
  Q1 -- yes --> H1[Hive bug: file HIVE]
  Q1 -- no --> Q2{Frames in org.apache.tez.runtime<br/>IFile / Fetcher / TezChild?}
  Q2 -- yes --> T1[Tez runtime bug: file TEZ]
  Q2 -- no --> Q3{Container launch / allocation /<br/>NM-shuffle connect failure?}
  Q3 -- yes --> Y1[YARN or deployment: check aux-service, queues]
  Q3 -- no --> Q4{Wrong results, no crash?}
  Q4 -- yes --> Q5{Same DAG shape reproducible<br/>with synthetic data, no Hive?}
  Q5 -- no --> H1
  Q5 -- yes --> T1

Wrong-result bugs almost always live in Hive (operator semantics) unless you can reproduce the same DAG shape with synthetic data and no Hive classes on the classpath — the skill drilled in Lab H5. The full attribution method is Lab H4; the gentler introduction is Level 6.

Reading exercise

Tez side (run from ~/tez-src):

1. Read determineParallelismAndApply in tez-runtime-library/.../vertexmanager/ShuffleVertexManagerBase.java. Write down the exact condition under which parallelism is not changed.
2. ls tez-runtime-library/src/main/java/org/apache/tez/runtime/library/cartesianproduct/ — read CartesianProductConfig and explain what "partitioned" vs "fair" means for the edge manager choice.
3. Read InputInitializer end to end in tez-api (it is short). What does initialize() return, and who consumes it?
4. grep -n "CUSTOM" tez-api/src/main/java/org/apache/tez/dag/api/EdgeProperty.java — which constructor is reserved for custom edges?

Hive side (run from ~/hive-src; expect version drift):

5. grep -rn "submitDAG" ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ — find the one call site and read 30 lines around it.
6. grep -rn "ShuffleVertexManager\|setVertexManagerPlugin" ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java — under exactly which conditions does Hive install which manager?
7. grep -rn "class DynamicPartitionPruner" ql/src/java — how many events does it expect, and what happens on timeout?

Common bugs and symptoms

Validation: prove you understand this

1. Name every layer between a SQL string and DAGAppMaster receiving a DAG, and mark each as Hive-owned or Tez-owned.
2. On a SCATTER_GATHER edge, name the Hive operator on each side and the Tez Output/Input class pair that moves the bytes.
3. Explain the ShuffleVertexManager auto-parallelism loop: what events carry the statistics, which method decides, and which context call applies the change. All three answers are class/method names in your Tez checkout.
4. A user's star-join query hangs with the fact-table vertex INITIALIZING. Walk the DPP event path and name the two places (one Hive class, one Tez class) where the hang can originate.
5. Why is a MapJoinOperator OOM a Hive bug even though the stack trace is rooted in a Tez container JVM? What single config would you change to confirm?
6. Given "works in CLI, fails in HS2", name two properties of session pooling that could explain it and the grep you would run in each tree.