Lab H5: Reproducing Bugs

Background

Lab H4 attributed a bug to a layer. This lab makes that attribution executable. A JIRA without a reproducer drifts; a JIRA with a clean, minimal, deterministic reproducer gets picked up. "Clean" here has a precise meaning: the smallest schema × smallest data × smallest query × smallest config-delta that still triggers the bug, runnable in under a minute — and, when the attribution is to Tez, a pure-Tez reproducer that uses no Hive at all.

That last move is the one that separates a triaged Tez JIRA from a punted one. If you attributed a shuffle bug to tez-runtime-library in H4, the proof is a Tez DAG built from tez-examples/tez-tests classes that reproduces the symptom with the same edge and processor semantics Hive used — and no Hive on the classpath. This lab teaches the minimization discipline, the Hive→pure-Tez conversion, deterministic re-runs, environment capture, and a full worked example end to end.

The reproduction harnesses live in the testing framework world; the pure-Tez templates are the real classes you verified in H4.

Why This Lab Matters for Contributors

Reproduction is where you convert a claim into evidence. Maintainers triage by reproducibility: a bug they can run in a minute on MiniTezCluster (or as a JUnit test) gets fixed; a bug that needs a 200-node cluster and a customer's data gets a "please provide a repro" and dies. The contributor who attaches the repro is the person who gets the fix credited.

The minimization discipline also protects you: a shrunk repro often reveals the true trigger (it was the 1024-row boundary, not the data content) and sometimes reveals that your attribution was wrong. And converting a Hive repro to a pure-Tez repro forces you to state, in code, exactly which edge/processor semantics you believe are at fault — the sharpest possible attribution.

Prerequisites

• H4 complete; you have an attributed bug (real or the worked one below).
• ~/tez-src and ~/hive-src present.
• Ability to build and run a Tez example (tez-examples) and a MiniTezCluster/ MiniHS2 test.
• The SET -v config dump from H2.

The Four Reduction Axes

Minimise along four independent axes. Reduce one, re-test, and stop the moment reduction kills the repro.

The fourth axis is the one novices forget. A repro that carries thirty SET lines from a production hive-site.xml is not minimized — most of those settings are irrelevant, and the two that matter are the actual trigger. Binary-search the config deltas the same way you binary-search the data.

Step 1: Local Harness — MiniHS2 + MiniTezCluster

MiniHS2 is a single-JVM HiveServer2 running against a MiniTezCluster (single-JVM YARN). Together they reproduce a Hive-on-Tez bug in seconds. Find the reference classes:

find ~/hive-src/itests -name "MiniHS2.java" | head
find ~/tez-src/tez-tests -name "MiniTezCluster.java"

A reproducer skeleton (Hive 3/4 style — adapt the builder to your version):

public class TestMyBugRepro {
  private MiniHS2 miniHS2;

  @Before public void setUp() throws Exception {
    HiveConf conf = new HiveConf();
    conf.set("hive.execution.engine", "tez");
    miniHS2 = new MiniHS2.Builder().withConf(conf).withMiniMR().build();
    miniHS2.start(new HashMap<>());
  }

  @After public void tearDown() throws Exception { miniHS2.stop(); }

  @Test public void reproBug() throws Exception {
    try (Connection c = DriverManager.getConnection(miniHS2.getJdbcURL());
         Statement s = c.createStatement()) {
      s.execute("CREATE TABLE t (a INT) STORED AS ORC");
      s.execute("INSERT INTO t VALUES (1),(1),(2),(3),(3),(3)");
      ResultSet rs = s.executeQuery("SELECT a, COUNT(*) FROM t GROUP BY a ORDER BY a");
      // assert the correct behavior, or expect the exception
    }
  }
}

Run with mvn test -pl itests -Dtest=TestMyBugRepro. This is your Hive-side repro. If the attribution is to Tez, you will also build the Hive-free version in Step 5.

Step 2: Reduce the Schema and Data

Schema. From a 200-column production table, keep only columns the failing query references, then simplify their types (DECIMAL(38,10) → DECIMAL(10,2) if precision is not load-bearing; STRUCT<...> → STRING; drop partitioning unless the partition is the trigger). Re-test after each cut.

Data. Binary-search the row count and, at the smallest triggering count, vary the distinct-key cardinality and the skew independently:

-- Controlled synthetic data: N rows, K distinct keys.
INSERT INTO t
SELECT CAST(pos % 8 AS INT)           -- 8 distinct keys
FROM (SELECT posexplode(split(space(1023),' ')) AS (pos, x)) g;  -- 1024 rows

Record the exact boundary: "reproduces at ≥ 1024 rows with ≥ 8 distinct keys; does not at 1023 rows." A sharp boundary is itself a clue (a 1024 boundary smells like a vectorized batch size of 1024).

Tip: Hive's tez-examples ships JoinDataGen (joindatagen in ExampleDriver) for generating join inputs at controlled cardinalities, and JoinValidate (joinvalidate) for asserting join correctness. Verify and reuse them rather than hand-rolling a generator: ls ~/tez-src/tez-examples/src/main/java/org/apache/tez/examples/JoinDataGen.java

Step 3: Reduce the Query and Config

Remove clauses one at a time, re-testing:

1. Drop ORDER BY — does it still fail? (If the bug is in the total-order reducer, it won't.)
2. Drop the aggregate.
3. Drop WHERE predicates one by one.
4. Collapse a multi-table join to two tables, then tiny-on-tiny.
5. Flip hive.auto.convert.join=false to turn a map join into a shuffle join — does the surface change? (This also tells you which edge the bug rides on, feeding Step 5.)

Then strip config: start from your SET -v dump, remove every non-default setting, and add back only the ones whose removal hides the bug. The minimal config-delta is usually one or two lines. A three-line query with a two-line config beats a thirty-line query with a whole hive-site.xml.

Step 4: Deterministic Re-Runs

A repro that fails 1-in-5 is nearly useless to a maintainer. Pin the sources of nondeterminism:

For Tez, tez.local.mode=true (verified in H1) removes YARN and runs everything in one JVM — the most deterministic mode, and it makes the repro a plain JUnit test. Verify:

grep -n "TEZ_LOCAL_MODE " \
  ~/tez-src/tez-api/src/main/java/org/apache/tez/dag/api/TezConfiguration.java

If the bug only reproduces with real multi-container shuffle, say so explicitly and use MiniTezCluster with multiple containers; some bugs genuinely need it.

Step 5: Convert the Hive Repro to a Pure-Tez Repro

This is the lab's centerpiece. When H4 attributed the bug to Tez, prove it by building a Tez DAG with the same edge and processor semantics Hive used — and no Hive. Use the real tez-examples/tez-tests classes as templates. Verify each name before you cite it:

ls ~/tez-src/tez-examples/src/main/java/org/apache/tez/examples/HashJoinExample.java
ls ~/tez-src/tez-examples/src/main/java/org/apache/tez/examples/SortMergeJoinExample.java
ls ~/tez-src/tez-examples/src/main/java/org/apache/tez/examples/OrderedWordCount.java
ls ~/tez-src/tez-tests/src/main/java/org/apache/tez/mapreduce/examples/MRRSleepJob.java
ls ~/tez-src/tez-tests/src/test/java/org/apache/tez/test/SimpleTestDAG.java

Map your Hive DAG to its pure-Tez analog:

OrderedWordCount and HashJoinExample extend TezExampleBase and build their DAG with Edge/EdgeProperty directly — read one to see how to set a BROADCAST vs SCATTER_GATHER edge in a handful of lines:

grep -n "EdgeProperty\|DataMovementType\|OrderedPartitionedKVEdgeConfig\|UnorderedKVEdgeConfig" \
  ~/tez-src/tez-examples/src/main/java/org/apache/tez/examples/HashJoinExample.java

For a deterministic shuffle-failure repro, Tez has built-in fetch-failure injection. Verify the config and the format:

grep -n "shuffle.fetch.testing.errors.enable\|TEZ_RUNTIME_SHUFFLE_FETCH_ENABLE_TESTING_ERRORS" \
  ~/tez-src/tez-runtime-library/src/main/java/org/apache/tez/runtime/library/api/TezRuntimeConfiguration.java

The injection string is maphost#mapvertex#probability#features (e.g. host1#*#100 means fail all fetches from host1). Read the doc comment in that file for the exact grammar. With it, a shuffle bug you attributed in H4 becomes a deterministic pure-Tez JUnit test — the strongest artifact you can attach to a TEZ JIRA.

Step 6: Capture the Environment

A repro without the environment triplet is not reproducible. Capture:

• The version triplet: Tez / Hive / Hadoop (SELECT version();, hive --version, hadoop version).
• The full Hive config: SET -v; (from H2).
• The Tez config actually in effect (the tez.* subset of SET -v).
• The JDK version.
• The EXPLAIN FORMATTED and, if it ran, EXPLAIN ANALYZE.

Bundle it:

mkdir -p ~/tez-notes/hive-h5-repro
# ddl.sql, gen.sql, query.sql, minimal-config.sql, explain.txt,
# amlog-fragment.txt, container-log-fragment.txt, PureTezRepro.java
cat > ~/tez-notes/hive-h5-repro/README.md <<'EOF'
# Repro for HIVE-XXXXX / TEZ-XXXX
Tez: 0.10.x  Hive: 4.0.x  Hadoop: 3.3.x  JDK: 11
Setup:   hive -f ddl.sql && hive -f gen.sql
Repro:   hive -f minimal-config.sql -f query.sql
Pure-Tez: mvn test -Dtest=PureTezRepro    (no Hive on classpath)
Expected: <oracle>.  Actual: <symptom>.
EOF
tar czf ~/tez-notes/hive-h5-repro.tar.gz -C ~/tez-notes hive-h5-repro

Step 7: Worked Example End to End

The report. "A COUNT(*) group-by returns a wrong count for one key on a large ORC table; only with vectorization on." (Illustrative — the shape is what matters, and the numbers below are synthesized, not a real run.)

Attribute (H4). Symptom is a wrong result, not an exception. Wrong results in aggregation are almost always Hive (operator/vectorization) or the cross-stage combine. Hypothesis: Hive vectorized group-by. Provisional owner: Hive.

Minimize schema. Only the group key and COUNT(*) matter → CREATE TABLE t (a INT) STORED AS ORC;.

Minimize data. Binary-search finds the boundary at exactly one vectorized batch:

reproduces at row count = 1024, distinct keys = 1
does not reproduce at 1023, or with keys ≥ 2

Minimize config. Strip everything; two lines survive removal:

SET hive.vectorized.execution.enabled=true;
SET hive.vectorized.execution.reduce.enabled=true;

Deterministic. Single split, single reducer, inline data — fails every run.

Convert to pure-Tez? Here the boundary matters. If the wrong count reproduces with a plain Tez MRR job (OrderedWordCount/MRRSleepJob) that has no Hive vectorization, the bug is in Tez's shuffle/combine — re-attribute to Tez. If it only reproduces through Hive's vectorized reducer, Hive owns it (confirmed) and the pure-Tez job runs clean. Either way, the pure-Tez run is the deciding experiment.

Bundle per Step 6, note "Expected 1024, Actual 1023," attach, file on the project the deciding experiment pointed to.

Production-to-Test Translation

When a real production bug arrives with no reproducer, you run the pipeline in reverse:

1. Get the query — from the reporter, from hive.log, or from the HS2 operation log.
2. Get the schema — SHOW CREATE TABLE on every table involved.
3. Get a data sample — a few thousand rows, PII anonymised (replace strings with fixed synthetic tokens, keep the distribution since skew is often the trigger).
4. Get the version triplet — Tez / Hive / Hadoop.
5. Reproduce — stand up MiniHS2, load schema + sample, run the query.
6. If it reproduces, reduce — the four axes.
7. If it does not reproduce, expand — this is the direction novices skip. Add data until you cross the trigger, add nodes/concurrency, enable the production-only settings (speculation, LLAP). A bug that needs 40 GB or three nodes is still reproducible; you just have to find the threshold and document it.

A one-day cycle for a complex production bug is fast; a one-week cycle is normal for something subtle. Either way the deliverable is the same bundle from Step 6.

Warning: Anonymise before you attach. A JIRA is public. Strip real table names, column names that leak business meaning, and any data values. Replace them with t, a, b and synthetic values — which also shrinks the repro, so anonymisation and minimisation are the same motion.

When MiniTezCluster Won't Reproduce

If none reduce, document that the repro requires N nodes and attach the best evidence (logs, counters, .pb.txt) you have.

Deliverables

• A hive-h5-repro.tar.gz bundle with DDL, data-gen, minimal query, minimal config, EXPLAIN, log fragments, and the version triplet.
• A TestMyBugRepro.java (Hive-side) skeleton adapted to your bug.
• A PureTezRepro.java built from a verified tez-examples/tez-tests template that reproduces (or provably does not reproduce) the symptom without Hive.
• A written record of the four-axis minimization: the exact schema/data/query/config boundary at which the repro flips.

Troubleshooting

Stretch Goals

1. Turn a shuffle bug into a deterministic test. Use SimpleTestDAG + fetch-failure injection to reproduce the H3 shuffle failure with 100% reliability, no Hive.
2. Prove a re-attribution. Take a "wrong count" bug you called Hive and try to reproduce it with MRRSleepJob/OrderedWordCount. If it reproduces, re-attribute to Tez and rewrite the JIRA.
3. Fit the repro in a JIRA comment. Compress the worked example to a DDL + 3-line query
   • 2-line config that pastes into a single comment. Test that it stands alone.
4. Cardinality vs skew. For a join bug, use JoinDataGen to separate "needs high cardinality" from "needs skew," and state which the bug requires.

Validation / Self-check

1. Name the four reduction axes and give the stop condition for each. Which one do novices forget?
2. Why is a pure-Tez reproducer the strongest attribution artifact, and which real example classes reproduce a map join and a shuffle join without Hive?
3. How do you make a shuffle-failure repro deterministic instead of hoping a node dies? Name the config flag and the injection-string grammar.
4. What is the "deciding experiment" in the worked example, and how does its outcome move the attribution between Hive and Tez?
5. Give the environment triplet you must always capture, and one command for each.
6. A repro fails 1-in-5 runs. List three sources of nondeterminism and how you pin each.
7. Your repro shrank until it stopped reproducing. What does that tell you about the last thing you removed?

You can now minimize a Hive-on-Tez bug and prove its attribution with a Hive-free repro. Next, Lab H6: Writing a Diagnostic Patch covers the case where you cannot reproduce locally and must improve the diagnostics at the boundary itself.