Lab H1: SQL → DAG

Background

A user writes one line of SQL:

SELECT a, COUNT(*) FROM t GROUP BY a ORDER BY a;

Hive compiles it into a Tez DAG with three vertices and two edges, ships that DAG to a Tez Application Master, and streams the result back. Every production Hive-on-Tez incident you will ever debug lives somewhere on that compilation-and-execution path. If you cannot walk the path deliberately — parser → semantic analyzer → logical operator tree → physical plan (MapWork/ReduceWork inside a TezWork) → TezTask → DagUtils.createVertex/createEdge → submitted DAG → running vertices — you cannot attribute a failure to the right layer, and every later lab in this section (H2 through H6) assumes you can.

This lab builds that muscle end to end. You stand up one reproducible Hive-on-Tez environment, run the three flavours of EXPLAIN on a join-and-group-by query, map each Hive operator to the TezWork unit it lands in, read the Hive source that turns that plan into a Tez DAG, then inspect the resulting DAG as a Graphviz .dot file. You finish with a labelled DAG diagram and a table that maps every operator to its runtime home — a mapper vertex, a reducer vertex, or an edge.

The boundary you are learning is documented in the Hive integration deep-dive; read its opening before you start so you know the class names by role.

Why This Lab Matters for Contributors

When a Hive-on-Tez JIRA lands, the reporter almost always gives you a SQL query and a stack trace, nothing more. The committer's first move is to rebuild the DAG in their head from the SQL — how many vertices, which edges, where the shuffle is, which vertex runs which operator. A contributor who can do that in thirty seconds triages ten issues in the time it takes a novice to triage one. A contributor who cannot will file a Tez bug for a Hive planner problem, or a Hive bug for a Tez shuffle problem, and burn a maintainer's goodwill.

The specific skill is plan literacy: reading EXPLAIN output the way a systems programmer reads assembly. It is the foundation for the attribution methodology in H4 and the reproduction discipline in H5. You are not learning to use Hive; you are learning to see the DAG through the SQL.

Prerequisites

• A working shell with git, docker (or a local Hadoop+Hive), dot (Graphviz), and a JDK 8 or 11 on PATH.
• A local Tez checkout at ~/tez-src (this book uses the real tree at that path). Confirm with git -C ~/tez-src log --oneline -1.
• A local Hive checkout at ~/hive-src for reading the compiler source. You do not build it in this lab; you grep it. Any Hive 3.1.x or 4.0.x tree works.
• You have read the Hive integration deep-dive intro and can name TezTask and DagUtils by role.
• Roughly 4 GB free RAM for the container.

Step 1: Stand Up One Canonical Environment

Pick one recipe and stick to it for all six labs. This book standardises on the official Apache Hive 4 container, because it is the single most reproducible Hive-on-Tez setup and Hive 4 ships Tez as its only execution engine (MapReduce was removed).

# Canonical recipe: Apache Hive 4.0.1, embedded metastore, Tez engine.
docker run -d --name hive4 -p 10000:10000 -p 10002:10002 \
  --env SERVICE_NAME=hiveserver2 \
  apache/hive:4.0.1

# Wait for HiveServer2, then open beeline inside the container:
docker exec -it hive4 beeline -u 'jdbc:hive2://localhost:10000/'

Note: Pin the version. apache/hive:4.0.1 is explicit and reproducible; apache/hive:latest drifts and will silently change your operator names and edge types between runs. If you are on Hive 3.x from a release tarball instead, the labs still work — the deltas are called out where they matter, and Hive 3 keeps MapReduce as an option, so you must SET hive.execution.engine=tez; explicitly.

Adaptation for a real pseudo-distributed cluster. The container runs Tez tasks in Hive's default embedded fashion; to inspect a DAG as it runs inside YARN (Tez UI, yarn logs, AM-side .dot files) you need Tez pointed at a YARN ResourceManager. That is a heavier setup and you will do it in H2 and H3; for H1, whatever mode the container uses is fine, because the DAG structure is identical either way. Confirm which mode you are in — it changes nothing about this lab but you should always know:

SET tez.local.mode;

Verify the real config key exists in the Tez tree so you trust the name:

grep -n 'TEZ_LOCAL_MODE ' \
  ~/tez-src/tez-api/src/main/java/org/apache/tez/dag/api/TezConfiguration.java
# public static final String TEZ_LOCAL_MODE = TEZ_PREFIX + "local.mode";

Step 2: Create the Study Tables

Use a join-and-group-by query so the DAG has more than one edge type. Two tiny tables:

CREATE TABLE t (a INT, b STRING) STORED AS ORC;
CREATE TABLE d (a INT, label STRING) STORED AS ORC;   -- small dimension

INSERT INTO t VALUES (1,'x'),(1,'y'),(2,'z'),(3,'p'),(3,'q'),(3,'r');
INSERT INTO d VALUES (1,'one'),(2,'two'),(3,'three');

The query under study, exercising a join and a group-by and an order-by:

SELECT t.a, d.label, COUNT(*) AS c
  FROM t JOIN d ON t.a = d.a
  GROUP BY t.a, d.label
  ORDER BY t.a;

Because d is tiny, Hive's cost-based optimiser will normally choose a map join (broadcast the small side) for the join, a shuffle for the group-by, and a single-reducer stage for the total-order ORDER BY. That gives you all three edge shapes in one DAG — exactly what you want to learn to recognise.

Step 3: The Three EXPLAINs

Hive gives you three progressively more expensive views of the same query. Run all three and save each.

3a. EXPLAIN — the planned physical DAG

EXPLAIN
SELECT t.a, d.label, COUNT(*) AS c
  FROM t JOIN d ON t.a = d.a
  GROUP BY t.a, d.label
  ORDER BY t.a;

Look for the Tez stage and its Edges: and Vertices: blocks. You will see vertex names like Map 1, Map 2, Reducer 3, Reducer 4 and edge annotations like BROADCAST_EDGE (map join) and SIMPLE_EDGE (shuffle). Write down the vertex/edge topology; this is the plan.

3b. EXPLAIN FORMATTED — the machine-readable plan

EXPLAIN FORMATTED
SELECT t.a, d.label, COUNT(*) AS c
  FROM t JOIN d ON t.a = d.a GROUP BY t.a, d.label ORDER BY t.a;

This emits JSON with the same content plus explicit edge type fields and per-vertex operator trees with row-schema annotations. It is what you parse programmatically and what you attach to a JIRA. You dissect this in depth in H2.

3c. EXPLAIN ANALYZE — the plan annotated with actual runtime counts

EXPLAIN ANALYZE
SELECT t.a, d.label, COUNT(*) AS c
  FROM t JOIN d ON t.a = d.a GROUP BY t.a, d.label ORDER BY t.a;

EXPLAIN ANALYZE actually runs the query and annotates each operator with the rows it really emitted (shown against the optimiser's estimate). This is your single best tool for spotting bad cardinality estimates — the root cause of a large class of "Hive-on-Tez is slow" reports. When the estimate says 5 rows and the runtime says 5 million, the join order the optimiser chose was wrong, and that is a Hive problem, not a Tez one.

Tip: Save all three. The habit of capturing EXPLAIN, EXPLAIN FORMATTED, and EXPLAIN ANALYZE for any query you are debugging is the cheapest diagnostic you have, and the three together let you separate planning faults (bad plan) from execution faults (bad DAG runtime).

Also capture the two structural views for completeness:

EXPLAIN AST  SELECT t.a, d.label, COUNT(*) FROM t JOIN d ON t.a=d.a GROUP BY t.a,d.label ORDER BY t.a;
EXPLAIN CBO  SELECT t.a, d.label, COUNT(*) FROM t JOIN d ON t.a=d.a GROUP BY t.a,d.label ORDER BY t.a;

EXPLAIN AST shows the parse tree (a Lisp-style tree of TOK_* nodes); EXPLAIN CBO shows the Calcite relational algebra after cost-based optimisation, before it is lowered into Hive operators.

Step 4: Operator Tree → TezWork

Read the operator symbols in the EXPLAIN output. The common ones:

The physical planner packs these operators into Work units — MapWork, ReduceWork, MergeJoinWork — and the whole graph of Work units is a TezWork. TezWork is the Hive-side representation of the DAG, one step before it becomes a Tez DAG object. Read its shape in the Hive source (name it by role, let the grep find the current file):

grep -rln "class TezWork"     ~/hive-src/ql/src/java/
grep -rln "class MapWork"     ~/hive-src/ql/src/java/
grep -rln "class ReduceWork"  ~/hive-src/ql/src/java/
grep -rln "class TezCompiler" ~/hive-src/ql/src/java/

TezCompiler is the physical compiler that produces the TezWork; skim it to see the operator-tree-to-work assignment happen. For our query, the Work units are:

Note: The exact vertex numbering and whether the small side gets its own Map vertex depends on the Hive version and the CBO decision. Record your topology from your EXPLAIN; do not trust this table over your own output. If your build chose a shuffle join instead of a map join (e.g. stats were missing), you will see a MERGEJOIN and a SIMPLE_EDGE where this table shows MAPJOIN and a broadcast.

Step 5: TezTask — The Boundary

TezTask is the one Hive class that executes a TezWork on Tez. Find it and read the entry point by role:

grep -rln "class TezTask" ~/hive-src/ql/src/java/
grep -n  "public int execute" $(grep -rln "class TezTask" ~/hive-src/ql/src/java/)

TezTask.execute(...) does, in order:

1. Acquire a TezSessionState (a pooled Tez session = one Tez AM) via TezSessionPoolManager.
2. Build a Tez DAG from the TezWork using DagUtils.
3. Submit the DAG through the session (TezSessionState → TezClient.submitDAG).
4. Monitor the returned DAGClient to completion.
5. Surface counters and diagnostics back to the Hive console.

Find the DAG-building call (name varies by version — createDag, build, etc.):

grep -n "DagUtils\|dagUtils" $(grep -rln "class TezTask" ~/hive-src/ql/src/java/)

Above this call, Hive owns everything (parse, analyse, plan). Below it, TezClient.submitDAG in tez-api takes over and Tez owns execution. That single call site is the entire integration surface — memorise it. Verify the Tez side of the boundary in your Tez tree:

grep -n "public synchronized DAGClient submitDAG" \
  ~/tez-src/tez-api/src/main/java/org/apache/tez/client/TezClient.java

Step 6: DagUtils.createVertex / createEdge

DagUtils is where a Hive Work becomes a Tez Vertex and a Hive edge becomes a Tez Edge. Read the two methods by role:

grep -rln "class DagUtils" ~/hive-src/ql/src/java/
grep -n "createVertex\|public Vertex " $(grep -rln "class DagUtils" ~/hive-src/ql/src/java/)
grep -n "createEdge\|EdgeProperty"     $(grep -rln "class DagUtils" ~/hive-src/ql/src/java/)

For our query, DagUtils produces:

And the edges, whose EdgeProperty uses a Tez DataMovementType. Verify the four movement types exist in your Tez tree — these are the vocabulary of every DAG you will ever read:

grep -n "ONE_TO_ONE\|BROADCAST\|SCATTER_GATHER\|CUSTOM" \
  ~/tez-src/tez-api/src/main/java/org/apache/tez/dag/api/EdgeProperty.java

The "single reducer for ORDER BY" is Hive forcing Reducer 4 to parallelism 1 (with no LIMIT), so all rows land on one task and sort globally.

Step 7: Inspect the Resulting DAG

Reading source tells you what should be built. Now capture what was built. Tez can dump every DAG it runs as a Graphviz .dot file, gated by one config flag. Verify the real key and default in your Tez tree:

grep -n "TEZ_GENERATE_DEBUG_ARTIFACTS" \
  ~/tez-src/tez-api/src/main/java/org/apache/tez/dag/api/TezConfiguration.java
# tez.generate.debug.artifacts  (default false)

Read where the AM acts on it — this is the code that writes the .dot and the tez-dag.pb.txt plan text file:

grep -n "writeDebugArtifacts\|generateDAGVizFile\|writePBTextFile" \
  ~/tez-src/tez-dag/src/main/java/org/apache/tez/dag/app/DAGAppMaster.java
grep -n "TEZ_PB_PLAN_TEXT_NAME" \
  ~/tez-src/tez-api/src/main/java/org/apache/tez/dag/api/TezConstants.java
# TEZ_PB_PLAN_TEXT_NAME = "tez-dag.pb.txt"

Turn it on and run the query:

SET tez.generate.debug.artifacts=true;
SELECT t.a, d.label, COUNT(*) AS c
  FROM t JOIN d ON t.a = d.a GROUP BY t.a, d.label ORDER BY t.a;

The AM writes <dagId>_priority.dot (and a <dagId>-tez-dag.pb.txt plan dump) into one of its log dirs. On the container / AM node:

find / -name '*_priority.dot' 2>/dev/null
dot -Tpng <dagId>_priority.dot -o ~/tez-notes/hive-h1-dag.png

Warning: The file is written from the AM, into the AM's log directory — on a real cluster that is the AM's node, not your client. You will chase this again in H2; for now, if you are in local/embedded mode the file is on your machine.

Also turn on the runtime summary, which prints the executed topology and per-vertex counters after the query:

SET hive.exec.print.summary=true;
SELECT t.a, d.label, COUNT(*) AS c
  FROM t JOIN d ON t.a = d.a GROUP BY t.a, d.label ORDER BY t.a;

Confirm the vertex names and count in the summary match the .dot and match your EXPLAIN. Any mismatch is a runtime decision (auto-parallelism, CBO re-planning) and is exactly the kind of thing H2 teaches you to chase.

Step 8: The Operator → Runtime-Home Table

The payoff. Fill in, for your captured DAG, where each operator actually runs. This is the mental model you carry into every later lab.

Save the mermaid diagram of your actual topology:

flowchart TD
  M2["Map 2<br/>MapTezProcessor<br/>TS(d) → RS"]
  M1["Map 1<br/>MapTezProcessor<br/>TS(t) → MAPJOIN → GBY(partial) → RS"]
  R3["Reducer 3<br/>ReduceTezProcessor<br/>GBY(final) → RS"]
  R4["Reducer 4<br/>ReduceTezProcessor<br/>SEL → FS  (parallelism 1)"]
  M2 -->|"BROADCAST"| M1
  M1 -->|"SCATTER_GATHER (partition on a,label)"| R3
  R3 -->|"SCATTER_GATHER (sort on a)"| R4

Deliverables

• ~/tez-notes/hive-h1-explain.txt — EXPLAIN, EXPLAIN FORMATTED, EXPLAIN ANALYZE, EXPLAIN AST, EXPLAIN CBO all saved.
• ~/tez-notes/hive-h1-dag.png rendered from the real _priority.dot.
• The operator → runtime-home table above, filled with your topology.
• The mermaid DAG diagram of your actual run.
• The grep results for createVertex/createEdge in DagUtils, saved.
• One sentence per edge stating why Hive chose that DataMovementType.

Troubleshooting

Stretch Goals

1. Break the map join. Run SET hive.auto.convert.join=false; and re-EXPLAIN. Watch the BROADCAST edge become a SCATTER_GATHER and a second reducer appear. Explain the new topology in one paragraph.
2. Add a LIMIT. Append LIMIT 2 to the query and re-EXPLAIN. Does the total-order reducer stay at parallelism 1? What did Hive change about the sort?
3. Cross-check the plan text. Open the <dagId>-tez-dag.pb.txt the AM wrote and match each VertexPlan's processorDescriptor class name to the table in Step 6.
4. Trace one operator into Tez. For the RS at the end of Map 1, find in Hive's DagUtils which Tez output class it becomes (an ordered-partitioned KV output). Grep the class name in ~/tez-src/tez-runtime-library to confirm it exists.

Validation / Self-check

1. Name the exact Hive class and method that is the last Hive code to run before Tez owns execution, and the Tez method it calls.
2. For our query, how many vertices and edges does the DAG have, and what DataMovementType is each edge? Why did Hive pick each one?
3. Which Hive operator becomes a Tez Edge rather than living inside a Processor, and why is that the natural boundary?
4. What does EXPLAIN ANALYZE tell you that EXPLAIN FORMATTED cannot, and what class of Hive bug does that difference expose?
5. Where does the .dot file get written, which config flag enables it, and why might it not be on the machine you ran the query from?
6. If the same SQL produced a MERGEJOIN instead of a MAPJOIN, what one fact about the tables changed, and how would the DAG's edges change?
7. Point (by grep) to the Hive source that assigns operators to MapWork vs ReduceWork. Name the compiler class.

You can now trace any Hive query from SQL to a running Tez topology. Next, Lab H2: Inspecting the Hive-Emitted DAG turns this into a production capture-and-inspect discipline.