Lab H2: Inspecting the Hive-Emitted DAG

Background

Lab H1 derived the DAG by reading Hive's compiler. That is the right way to understand a DAG, but in production you rarely get to re-derive — you inherit a running or finished artifact and must capture what Hive actually submitted, including every runtime decision the plan didn't show. Auto-parallelism can collapse forty reducers to four; a VertexManager can add edges after submission; vectorization can silently fall back. None of that is visible in EXPLAIN.

This lab is deep inspection of the running artifact. You will:

1. Read the DAG plan protobuf the AM writes to disk (tez.generate.debug.artifacts).
2. Read the vertex and edge properties Hive set — payloads, processor descriptors, and the ShuffleVertexManager auto-parallelism knobs — against the real config keys in the Tez source.
3. Pull per-vertex counters after a run and correlate EXPLAIN vertex names to the DAG vertex names.
4. Learn the edge type each query shape produces: map join → BROADCAST, group-by → SCATTER_GATHER, global ORDER BY → a single-reducer SCATTER_GATHER.

The relevant deep-dives are Hive integration for the boundary and counters & diagnostics for the counter mechanism. Read the latter's intro before Step 4.

Why This Lab Matters for Contributors

The gap between the planned DAG and the executed DAG is where half of all Hive-on-Tez performance JIRAs live. A reporter says "my query is slow"; the plan looks fine; the truth is that auto-parallelism gave one vertex a single task for a 40 GB input, or a broadcast edge is shipping a hash table that turned out to be 3 GB. You cannot see any of that from EXPLAIN. You have to read the artifact.

Committers are trusted with runtime decisions precisely because they can read the submitted DAGPlan and the counters and say "the plan was fine; the ShuffleVertexManager under-provisioned this vertex — here is the desired-task-input-size that caused it." That sentence is the difference between a triaged issue and a punted one.

Prerequisites

• You completed H1 and have the two study tables (t, d).
• ~/tez-src and ~/hive-src present; dot installed.
• You can run a query and find its DAG's log directory (container or AM node).
• You read the counters & diagnostics deep-dive intro.

Step 1: Capture the Plan Protobuf and the .dot

Turn on debug artifacts (verified in H1) and run a query. The AM writes two files per DAG: a Graphviz .dot and a human-readable protobuf text dump of the whole DAGPlan.

SET tez.generate.debug.artifacts=true;
SET hive.exec.print.summary=true;
SELECT t.a, d.label, COUNT(*) AS c
  FROM t JOIN d ON t.a = d.a GROUP BY t.a, d.label ORDER BY t.a;

Confirm the file names against the real constants in the Tez tree:

grep -n "TEZ_PB_PLAN_TEXT_NAME\|TEZ_PB_PLAN_BINARY_NAME" \
  ~/tez-src/tez-api/src/main/java/org/apache/tez/dag/api/TezConstants.java
# TEZ_PB_PLAN_BINARY_NAME = "tez-dag.pb"
# TEZ_PB_PLAN_TEXT_NAME   = "tez-dag.pb.txt"

Read the code that produces them, so you trust exactly what is dumped and when:

grep -n "writeDebugArtifacts\|writePBTextFile\|generateDAGVizFile" \
  ~/tez-src/tez-dag/src/main/java/org/apache/tez/dag/app/DAGAppMaster.java

You will see that writeDebugArtifacts runs only when the flag is true, and that it calls Utils.generateDAGVizFile(...) for the .dot and writePBTextFile(...) for the <dagId>-tez-dag.pb.txt. Locate and open the text plan:

find / -name '*tez-dag.pb.txt' 2>/dev/null

The .pb.txt is a protobuf text rendering of the DAGPlan: a name, a repeated vertex list (each with a processorDescriptor and input/output descriptors), and a repeated edge list. This is the ground-truth artifact — everything below is a way of reading pieces of it.

Step 2: EXPLAIN FORMATTED as the Machine-Readable Mirror

Capture the plan Hive intended, as JSON, and diff it in your head against the .pb.txt the AM actually built.

EXPLAIN FORMATTED
SELECT t.a, d.label, COUNT(*) AS c
  FROM t JOIN d ON t.a = d.a GROUP BY t.a, d.label ORDER BY t.a;

Structure (varies by Hive version — trust your output):

{
  "STAGE PLANS": {
    "Stage-1": {
      "Tez": {
        "Edges:": {
          "Reducer 3": [{"parent": "Map 1", "type": "SIMPLE_EDGE"}],
          "Map 1":     [{"parent": "Map 2", "type": "BROADCAST_EDGE"}],
          "Reducer 4": [{"parent": "Reducer 3", "type": "SIMPLE_EDGE"}]
        },
        "Vertices:": {
          "Map 1": {"Map Operator Tree:": ["..."], "Execution mode:": "vectorized"},
          "Map 2": {"Map Operator Tree:": ["..."]},
          "Reducer 3": {"Reduce Operator Tree:": ["..."]},
          "Reducer 4": {"Reduce Operator Tree:": ["..."]}
        }
      }
    }
  }
}

Two Hive edge names map to two Tez DataMovementTypes:

Verify the Tez names are real:

grep -n "ONE_TO_ONE\|BROADCAST\|SCATTER_GATHER\|CUSTOM" \
  ~/tez-src/tez-api/src/main/java/org/apache/tez/dag/api/EdgeProperty.java

Step 3: Vertex and Edge Properties Hive Set

Now read the two levers that matter most at runtime: the processor payload and the auto-parallelism configuration on the shuffle vertices.

3a. Processor descriptors

In the .pb.txt, each vertex has a processorDescriptor.className. For a Hive DAG these are MapTezProcessor and ReduceTezProcessor. Confirm they exist Hive-side:

grep -rln "class MapTezProcessor\|class ReduceTezProcessor" ~/hive-src/ql/src/java/

The MapRecordProcessor/ReduceRecordProcessor are the Hive classes those Tez processors delegate to for the actual operator pipeline — the same classes you will meet in stack traces in H3:

grep -rln "class MapRecordProcessor\|class ReduceRecordProcessor" ~/hive-src/ql/src/java/

3b. ShuffleVertexManager auto-parallelism

A SCATTER_GATHER vertex is usually managed by ShuffleVertexManager, which can decrease a vertex's parallelism at runtime based on the actual data size its upstream tasks produced. This is the single most common source of "planned N reducers, ran M" surprises. Read the real config keys and defaults directly from the source — do not trust any key name you did not verify here:

grep -n "public static final String TEZ_SHUFFLE_VERTEX_MANAGER" \
  ~/tez-src/tez-runtime-library/src/main/java/org/apache/tez/dag/library/vertexmanager/ShuffleVertexManager.java

The keys you will care about (verified against the current Tez tree):

Note: Hive typically enables auto-parallelism for its reducers and sets a desired task input size derived from hive.exec.reducers.bytes.per.reducer. When a reducer vertex ran with far fewer tasks than EXPLAIN implied, this is almost always why. The min-src-fraction/max-src-fraction pair controls the slow-start ramp — how eagerly downstream tasks begin before upstream finishes.

Inspect the actual values Hive sent by dumping the session config:

SET -v;   -- dumps every effective config, including tez.shuffle-vertex-manager.*

Save that dump; you will reuse it as the environment capture in H5.

Step 4: Per-Vertex Counters After the Run

Counters are how the runtime tells you what each vertex actually did. With hive.exec.print.summary=true, Hive prints a per-vertex table after the query. The per-task and per-vertex counters come from Tez's TaskCounter; the DAG-level ones from DAGCounter. Read the real enum members so you name them correctly:

grep -n "INPUT_RECORDS_PROCESSED\|OUTPUT_RECORDS\|REDUCE_INPUT_GROUPS\|SHUFFLE_BYTES\|SPILLED_RECORDS" \
  ~/tez-src/tez-api/src/main/java/org/apache/tez/common/counters/TaskCounter.java
grep -n "NUM_SUCCEEDED_TASKS\|TOTAL_LAUNCHED_TASKS\|WALL_CLOCK_MILLIS" \
  ~/tez-src/tez-api/src/main/java/org/apache/tez/common/counters/DAGCounter.java

For our 6-row/3-row study query, map the counters to expected values:

Verify against your actual summary. A counter that is zero where you expected nonzero (e.g. OUTPUT_RECORDS = 0 on the final reducer) is a real bug signal, and the shape of the reproducer for it is exactly what H5 teaches.

Tip: INPUT_GROUPS exists in the enum but is marked "not used at the moment" in the source — the live counter is REDUCE_INPUT_GROUPS. Always confirm a counter is actually emitted before you build an assertion on it; the enum lists more than the runtime fills.

Step 5: Correlate EXPLAIN Names to DAG Vertex Names

Hive names its vertices Map 1, Reducer 3, etc., in EXPLAIN. Tez names them the same in the DAGPlan — Hive passes the name straight through in DagUtils. That is not guaranteed for every framework on Tez, so verify it for Hive:

grep -n "setName\|getName\|vertexName" $(grep -rln "class DagUtils" ~/hive-src/ql/src/java/) | head

Build the correlation table for your run:

When the names don't line up (rare, but happens with LLAP or with dynamically-added vertices), you have found a runtime re-plan — note it; it is a legitimate thing to file about if it makes diagnostics confusing.

Step 6: Edge Types by Query Shape

The whole point of reading DAGs is pattern recognition. Memorise which query shape emits which edge, so you can predict a DAG from SQL and spot when the DAG doesn't match.

Note on global sort: Hive does not use a Tez range-partition edge for a plain ORDER BY; it forces the final reducer to parallelism 1 and lets a single task perform the total sort. If you add LIMIT, Hive can use a top-N short-circuit, and the shape changes — verify with EXPLAIN rather than assuming. Confirm the single-reducer behavior in your .pb.txt (Reducer 4 has one task) and in the summary.

Step 7: The Four-Way Cross-Check

When all four views agree, you have ground truth. When they disagree, the disagreement is the finding.

Each row of the second table is a diagnosis. "Planned 40, ran 4, and the input was 40 GB" is a desired-task-input-size misconfiguration; "planned N, ran N+1" is a CBO re-plan; each points at a different owner.

Deliverables

• The <dagId>-tez-dag.pb.txt saved and annotated: for each vertex, its processor class; for each edge, its data-movement type.
• ~/tez-notes/hive-h2-explain-formatted.json and a rendered .dot PNG.
• The SET -v dump, with the tez.shuffle-vertex-manager.* lines highlighted and their effect explained in one line each.
• The per-vertex counter table filled with your numbers, cross-checked against the 6/3-row expectations.
• The EXPLAIN-name → DAG-vertex-name correlation table.
• A one-paragraph note on any disagreement among the four views, with its cause.

Troubleshooting

Stretch Goals

1. Force an auto-parallelism cut. Load t with enough rows that the group-by reducer would plan several tasks, then lower tez.shuffle-vertex-manager.desired-task-input-size and watch the summary's task count change while the .pb.txt plan stays the same.
2. Read a bucketed map join. Create bucketed tables, run a bucketed map join, and find the CUSTOM_EDGE in the .pb.txt. Name the custom edge-manager class Hive set.
3. Diff two Hive versions. Run the same query on Hive 3 and Hive 4 containers and diff the two .pb.txt files. Explain any change in vertex count or edge type.
4. Byte-volume forensics. From the Tez UI (or counters), find the bytes crossing the broadcast edge. If the "small" side were actually 2 GB, what counter would warn you, and which knob (hive.auto.convert.join.noconditionaltask.size) governs it?

Validation / Self-check

1. Which two files does the AM write when tez.generate.debug.artifacts=true, what are their exact names, and which method writes each?
2. Give the five tez.shuffle-vertex-manager.* keys and their defaults from memory, then verify each with a grep. Which one most directly causes "planned N, ran M"?
3. A reducer vertex planned for 40 tasks ran with 4. Name the mechanism and the two config values you would inspect to explain it.
4. Map each of the four Hive EXPLAIN edge names to its Tez DataMovementType.
5. For a plain global ORDER BY with no LIMIT, what does Hive do to the final reducer, and how do you confirm it in the artifact?
6. Which counter tells you distinct group count on a group-by reducer, and why must you verify it is actually emitted before asserting on it?
7. You have four views of a DAG and they disagree on vertex count. Which view is "intended," which is "submitted," which is "ran," and how does the disagreement localise the cause?

You can now capture and read a Hive-submitted DAG four ways and reconcile them. Next, Lab H3: Debugging a Failed Query takes a failing DAG and walks the diagnostics chain to its root cause.