Lab 6.1 — Trace a Hive SQL Query to the Generated Tez DAG

Lab type: Reproduce & Trace Estimated time: 150 min Tez modules: tez-dag, tez-runtime-library, tez-api Hive classes (verify in your Hive checkout): SemanticAnalyzer, TezCompiler, TezWork, DagUtils, TezTask

Background

When you run

SELECT dept, COUNT(*) FROM employees GROUP BY dept;

on Hive with hive.execution.engine=tez, Hive compiles the SQL into a physical plan of MapWork/ReduceWork units, wraps them in a TezWork graph, and hands that to DagUtils, which builds a real Tez DAG — vertices named Map 1, Reducer 2, edges of type SCATTER_GATHER, and a ShuffleVertexManager on the reduce vertex. TezTask submits it via TezClient.submitDAG and the Tez AM (DAGImpl) runs it.

In this lab you will reproduce that translation on your own machine, read the EXPLAIN plan, map every plan vertex onto a runtime Tez vertex of the same name, and confirm the topology two independent ways (the Hive summary and the Tez .dot graph). You come out able to trace any Hive query to its DAG.

Why This Lab Matters for Contributors

Almost every Tez performance or correctness bug arrives described in Hive terms: "my GROUP BY spawns 1009 reducers," "the join vertex hangs." To fix it you must translate that Hive vocabulary into Tez runtime objects — vertices, edges, vertex managers — because that is where the Tez code you would patch actually lives. This lab builds the translation reflex. Without it, you cannot even read the bug report; with it, you can point at the exact VertexImpl or ShuffleVertexManager behind the symptom.

Prerequisites

• Level 6 index read; you can state the Hive/Tez seam.
• Your Tez checkout builds (mvn -q -DskipTests install succeeds).
• Docker installed or a Hadoop + Hive tarball you can run locally.
• Comfort with EXPLAIN output from Lab H1.
• You have read Lab H2: Inspect the DAG or are ready to.

Note: Hive is not in your Tez checkout. Everywhere this lab names a Hive class, a grep/find command follows so you confirm it against your Hive version. Never cite a Hive symbol you have not seen in your own tree.

Step-by-Step Tasks

Step 1 — Stand up a local Hive-on-Tez

You need the smallest thing that compiles SQL to a Tez DAG and runs it. Two recipes; pick one and pin the versions you actually use — adapt every version string below to what you install.

Recipe A — Docker (fastest). The Apache Hive project publishes a single-container image that boots HiveServer2 with an embedded Derby metastore. Check current tags on Docker Hub / the Hive site and substitute the tag you find; the shape is:

# Adapt the tag to a real published Hive 4.x tag you verify on Docker Hub.
docker run -d --name hive4 \
  -p 10000:10000 -p 10002:10002 \
  --env SERVICE_NAME=hiveserver2 \
  apache/hive:4.0.1

# Wait for HiveServer2, then connect with the bundled beeline:
docker exec -it hive4 beeline -u 'jdbc:hive2://localhost:10000/'

Recipe B — Tarballs (full control). If you want to see the Tez jars and configs on disk, install a matched trio and set them up by hand:

# Versions are EXAMPLES — verify current releases and adapt all three.
HADOOP=3.3.6 ; HIVE=4.0.1 ; TEZ=0.10.4
# 1) unpack hadoop, hive, tez tarballs
# 2) put tez's jars in HDFS (or file:///) and point tez.lib.uris at them
# 3) in hive-site.xml set:
#      hive.execution.engine = tez
#      hive.tez.container.size = 1024   (adapt to your RAM)
# 4) start a local metastore (Derby is fine for a lab) and HiveServer2

Whichever you choose, record the exact Hive and Tez versions — Hive×Tez compatibility is version-sensitive, and your notes are worthless without them:

-- inside beeline
SET hive.execution.engine;   -- must print: tez

Expected: hive.execution.engine=tez. If it says mr or spark, set it: SET hive.execution.engine=tez;. Write the Hive and Tez versions into your lab notes now.

Step 2 — Create a tiny, deterministic dataset

Small and deterministic so the counters are predictable:

CREATE TABLE employees (id INT, dept STRING) STORED AS ORC;

INSERT INTO employees VALUES
  (1,'eng'),(2,'eng'),(3,'eng'),
  (4,'sales'),(5,'sales'),
  (6,'ops');

Six rows, three distinct dept values (eng=3, sales=2, ops=1). Keep this in your notes — you will predict counters against it in Step 6.

Step 3 — Read the EXPLAIN plan and extract the vertices

EXPLAIN SELECT dept, COUNT(*) FROM employees GROUP BY dept;

Find the Stage: Stage-1 / Tez block. For a single-GROUP BY query you expect two vertices and one edge:

and the edge line Reducer 2 <- Map 1 (SIMPLE_EDGE). In Hive's EXPLAIN, SIMPLE_EDGE is the shuffle edge — it becomes Tez SCATTER_GATHER. Copy the whole Stage-1 block into your notes; it is the left-hand column of your mapping.

Note: Hive names map vertices Map N and reduce vertices Reducer N. Those exact strings become the Tez vertex names. Hold onto them.

Step 4 — Turn on the Hive-side DAG summary

Hive can print a per-vertex runtime summary after the query. Enable it and run:

SET hive.tez.exec.print.summary=true;
SELECT dept, COUNT(*) FROM employees GROUP BY dept;

After the result rows, Hive prints a summary table listing each vertex (Map 1, Reducer 2), its task count, and timing. This is your first independent confirmation of the topology: the vertex names here must match the EXPLAIN plan. Save the summary block.

If your build labels the setting differently, grep the Hive conf for it: grep -rn "print.summary" $(find . -name HiveConf.java) in your Hive checkout.

Step 5 — Find the same vertices on the Tez side

Now cross the seam. The vertices Hive named are, at runtime, VertexImpl objects inside the Tez AM (DAGImpl). You can see the names two ways.

(a) The Tez .dot graph. Tez writes a Graphviz file of the submitted DAG when debug artifacts are enabled. Confirm the switch and the writer in your checkout:

# The config flag (default false):
grep -rn "generate.debug.artifacts" \
  tez-api/src/main/java/org/apache/tez/dag/api/TezConfiguration.java

# The code that writes "<dagId>.dot":
grep -rn "\.dot\|Generating DAG graphviz file" \
  tez-dag/src/main/java/org/apache/tez/Utils.java

Enable it for your query and rerun:

SET tez.generate.debug.artifacts=true;
SELECT dept, COUNT(*) FROM employees GROUP BY dept;

A <dagId>.dot file appears in the AM log directory. Its node names are Map 1 and Reducer 2; each edge is labelled with dataMovement=SCATTER_GATHER and schedulingType=.... Render it:

dot -Tpng <dagId>.dot -o dag.png   # or paste the .dot into any Graphviz viewer

The edge label text is built in tez-dag's org.apache.tez.Utils — it emits dataMovement= and schedulingType= for each edge. That string is your proof that the Hive SIMPLE_EDGE became a Tez SCATTER_GATHER edge.

(b) The Tez UI / AM logs. If you have the Tez UI or ATS wired up, the DAG view shows the same two named vertices. Otherwise grep the AM syslog for the vertex names (you will do this properly in Lab 6.2):

yarn logs -applicationId <appId> -log_files syslog | grep -E "Map 1|Reducer 2"

Step 6 — Build the correlation table

Now write the artifact this whole lab exists to produce: a table that maps every EXPLAIN plan vertex to its Tez runtime object and predicts its counters.

The counter names are real Tez TaskCounter enum members — verify:

grep -nE "INPUT_RECORDS_PROCESSED|OUTPUT_RECORDS|REDUCE_INPUT_GROUPS" \
  tez-api/src/main/java/org/apache/tez/common/counters/TaskCounter.java

Now run with the summary on and fill in the actual numbers. The partial aggregate in Map 1 (map-side GROUP BY) is why Map 1 OUTPUT_RECORDS is 3, not 6 — you should be able to explain that from the operator tree.

Step 7 — Attribute each DAG element to the Hive class that built it

The final discrimination: for each piece of the DAG, name the Hive class that constructed it. This is what makes you able to route a bug to the right project.

Run each grep in your Hive checkout (top four) and Tez checkout (last). The point lands when you can say: "Reducer 2's ShuffleVertexManager was attached by Hive's DagUtils but is executed by Tez's runtime-library — so a reducer-count bug could be in either project, and I know how to tell which."

Deliverables

• Recorded Hive and Tez versions and the working SET hive.execution.engine=tez.
• The EXPLAIN Stage-1 / Tez block for the GROUP BY query, saved.
• The hive.tez.exec.print.summary=true output, saved.
• The rendered Tez .dot graph (dag.png) or its raw text, showing dataMovement=SCATTER_GATHER.
• The completed correlation table (Step 6) with predicted and actual counters, and your one-line explanation of the partial-aggregate count.
• The attribution table (Step 7) with every grep run against your own checkouts.

Troubleshooting

Stretch Goals

1. Add ORDER BY and watch a vertex appear. Run EXPLAIN on SELECT dept, COUNT(*) FROM employees GROUP BY dept ORDER BY dept;. You now get a third vertex (Reducer 3) with parallelism 1 — Hive forces a single reducer for total order. Confirm parallelism 1 in the .dot graph and explain, from Lab H1, why.
2. Provoke auto-parallelism. With a larger table and hive.tez.auto.reducer.parallelism=true, the reduce vertex's task count is set at runtime by ShuffleVertexManager, not at compile time. Compare the EXPLAIN reducer count against the summary's actual task count and reconcile them via the plugin.
3. Diff two Hive versions. If you can run two Hive versions against the same Tez, EXPLAIN the same query on each and diff the plans. Any difference in vertex/edge shape is a compatibility surface — exactly the kind of thing that blocks a Tez release.

Validation

Answer these from your artifacts, not from memory:

1. In the GROUP BY query, which Hive class turns SIMPLE_EDGE into a Tez SCATTER_GATHER edge, and which Tez class executes that edge's shuffle?
2. Why is Map 1 OUTPUT_RECORDS = 3 and not 6 for the six-row input? Name the operator responsible.
3. Where did the string Reducer 2 originate — Hive or Tez — and where do you see it on each side?
4. What config flag makes Tez write the .dot file, what is its default, and which Tez class writes the file?
5. If the query produced 1009 reducers unexpectedly, name the two projects the bug could live in and the one class in each you would open first.
6. Which TaskCounter member tells you the number of distinct dept groups, and on which vertex do you read it?
7. Adding ORDER BY created a vertex with parallelism 1 — what forces that, and is it a Hive decision or a Tez decision?

When you can answer all seven and hand over both tables, proceed to Lab 6.2 — Debug a Failed Hive-on-Tez Query, or go deep with Lab H2: Inspect the DAG.