Level 5: Testing and Debugging

Up to now you have read the engine and made small, reviewed changes. From here on, no patch lands without a test. The first question a Tez committer asks on almost every JIRA is "where's the test that fails without your change and passes with it?" The second is "does it pass reliably?" This level makes the Tez test suite your home: its three tiers, the DrainDispatcher-driven state-machine harness, the MiniTezCluster + MiniDFSCluster integration rig, and the debugging discipline you use when a test — or production — misbehaves.

Apache Tez has one of the most complete test suites in the Hadoop ecosystem: on the order of a thousand unit tests plus a full in-JVM YARN/HDFS integration harness. The single most valuable skill you build here is picking the cheapest tier that still proves your change, then making the assertion deterministic.

Note: Tez's state-machine tests are built on Hadoop's StateMachineFactory and a test-only DrainDispatcher (a subclass of YARN's AsyncDispatcher whose await() busy-loops until the event queue is empty). This lets a test fire an event and deterministically observe the resulting state — no sleeps, no races. Integration tests, by contrast, boot real YARN and HDFS in the test JVM and are where timing bugs and flakiness live. Both are covered below.

Learning Objectives

By the end of Level 5 you must be able to:

1. Name the three test tiers in Tez (unit / mini-cluster / full-cluster), choose the right one for a given change, and give the Maven command that runs each.
2. Explain how MiniTezCluster (extends Hadoop's MiniYARNCluster) and MiniDFSCluster are wired together in a @BeforeClass, and what each serviceInit / serviceStart / serviceStop step does.
3. Read and run the canonical mini-cluster integration test, TestTezJobs#testOrderedWordCount, and distinguish what it exercises from what a unit test exercises.
4. Dissect the TestVertexImpl harness: how it builds a VertexImpl with mocked services and drives VertexEvents through a DrainDispatcher, and how it asserts on VertexState.
5. Build a coverage matrix of which (VertexState, VertexEventType) transitions have tests, find a gap, and write a new transition test that follows the existing patterns.
6. Write a full mini-cluster integration test from scratch that submits a multi-vertex DAG, asserts on the final DAGStatus state, and reads back TezCounters.
7. Find a real @Ignored test, reconstruct why it was disabled via git log/git blame, and reason about what a credible "un-ignore" patch requires.

The Three Test Tiers

Everything lives in each module's src/test/java. Reach for a heavier tier only when a lighter one cannot exercise the behavior.

# Point this at your real checkout; the labs assume $TEZ_SRC is set.
export TEZ_SRC=/path/to/tez           # e.g. ~/src/tez
cd "$TEZ_SRC"
find . -name "MiniTezCluster.java"    # -> tez-tests/src/test/java/org/apache/tez/test/MiniTezCluster.java
find . -name "DrainDispatcher.java"   # -> tez-common/src/test/java/org/apache/tez/common/DrainDispatcher.java

flowchart TD
    Q{What does your change touch?} --> U[A single class / a state transition / a parser]
    Q --> M[A whole DAG: submission, shuffle, counters, recovery]
    Q --> S[Scale / real-cluster behavior]
    U --> UC["Unit test<br/>(TestVertexImpl, TestDAGImpl, TestTaskImpl)"]
    M --> MC["Mini-cluster test<br/>(TestTezJobs, TestFaultTolerance, TestRecovery)"]
    S --> SC["Full cluster<br/>(CI only)"]

Warning: Naming and location are load-bearing. TestOrderedWordCount (tez-tests/src/main/java/org/apache/tez/mapreduce/examples/TestOrderedWordCount.java) looks like a JUnit test but is a runnable Tool example in src/main — it has no @Test methods, so -Dtest=TestOrderedWordCount runs nothing. The real mini-cluster JUnit test for ordered word count is TestTezJobs#testOrderedWordCount in src/test. Match the class you run to where it lives.

How the Mini-Cluster Is Wired

MiniTezCluster extends MiniYARNCluster. In a single JVM it starts an in-process YARN ResourceManager and NodeManager(s); a test pairs it with a MiniDFSCluster (NameNode + DataNodes) when it needs real HDFS. The Tez ApplicationMaster runs as a normal YARN application inside that mini-YARN.

   JUnit test JVM
   ├── MiniDFSCluster        (NameNode + N DataNodes)   -> remoteFs = dfsCluster.getFileSystem()
   └── MiniTezCluster        (extends MiniYARNCluster)
        ├── ResourceManager
        ├── NodeManager(s)   (ShuffleHandler aux-service, DefaultContainerExecutor)
        └── Tez AM (DAGAppMaster)  <- launched per DAG / per session as a YARN app

Inspect the real lifecycle (never trust line numbers — grep):

grep -n "extends MiniYARNCluster\|serviceInit\|serviceStart\|serviceStop\|APPJAR\|TEZ_LIB_URIS" \
  "$TEZ_SRC"/tez-tests/src/test/java/org/apache/tez/test/MiniTezCluster.java

Key facts you should be able to state from that file:

Required Reading

Read these deep dives before the labs; they are the conceptual backbone this level makes concrete.

• Testing Framework — the three tiers, the arrange/send/drain/assert recipe, and the do/don't patterns. Read first.
• State Machines — StateMachineFactory, addTransition, and how an undefined (state, event) pair throws InvalidStateTransitonException.
• Vertex Lifecycle — the VertexState graph you will be testing.
• Counters & Diagnostics — TezCounters, TaskCounter, and how to read a failed DAG's diagnostics.
• YARN Integration and Local Mode — what the mini-cluster is standing in for.

Source Areas to Inspect

Debugging a Test

Three tools cover almost every test-debug situation in Tez.

1. Attach a debugger to a single test method. Surefire forks a JVM per test run; pause it and attach your IDE's remote debugger on port 5005:

# Pauses the forked test JVM until a debugger attaches on 5005, then runs the one method.
cd "$TEZ_SRC"
mvn test -pl tez-dag -Dtest='TestVertexImpl#testInvalidEvent' -Dmaven.surefire.debug
# In IntelliJ: Run > Attach to Process / "Remote JVM Debug" -> localhost:5005.
# Breakpoint in VertexImpl.handle(...) and step through the transition.

2. TRACE logging on exactly the package you care about. For a mini-cluster test, raise the log level for one package so the DAG lifecycle is visible without drowning in YARN/HDFS noise. The captured output lands in the Surefire -output.txt:

mvn test -pl tez-tests -Dtest='TestTezJobs#testOrderedWordCount' \
  -Dtez.root.logger=DEBUG,CLA 2>&1 | grep -i "vertex\|dag state"
grep -n "SUCCEEDED\|RUNNING\|Vertex.*transition" \
  tez-tests/target/surefire-reports/org.apache.tez.test.TestTezJobs-output.txt | head

3. DrainDispatcher.await() — the deterministic wait. In unit tests you never Thread.sleep to wait for an event to be processed. DrainDispatcher (a test-only AsyncDispatcher subclass) exposes await(), which busy-loops until the event queue is empty. Fire an event, call await(), then assert — the state is guaranteed settled. This is the single most important habit in Tez unit testing and the subject of Lab 5.2.

Tip: The final states of the two machines you assert on: a DAG ends in DAGStatus.State SUCCEEDED / KILLED / FAILED / ERROR; a vertex ends in VertexState SUCCEEDED / KILLED / FAILED / ERROR. ERROR specifically means "the state machine received an event it had no transition for" — a bug, not a normal outcome.

Key Classes Quick Reference

The Labs

Deliverables

Demonstrate all of the following before advancing to Level 6:

• You ran TestTezJobs#testOrderedWordCount to green and can explain, step by step, how MiniDFSCluster and MiniTezCluster are wired in @BeforeClass (Lab 5.1).
• A new (VertexState, VertexEventType) transition test added to TestVertexImpl, following the DrainDispatcher harness, that passes alongside the existing suite (Lab 5.2).
• A from-scratch mini-cluster integration test that submits a multi-vertex DAG, asserts DAGStatus.State.SUCCEEDED, and reads back a TezCounter value you derived by hand (Lab 5.3).
• A real @Ignored test picked from the checkout, its disable-history reconstructed from git log/git blame, and a written analysis of what un-ignoring it would take (Lab 5.4).
• From memory: the three tiers and their Maven commands; why DrainDispatcher.await() beats Thread.sleep; and why an undefined (state, event) pair drives a vertex to ERROR.

Common Mistakes

How to Verify Success

cd "$TEZ_SRC"
# 1. The whole module builds and its unit tests pass.
mvn install -DskipTests -q
mvn test -pl tez-dag -Dtest=TestVertexImpl -q 2>&1 | tail -5

# 2. You can run one integration test and read its Surefire report.
mvn test -pl tez-tests -Dtest='TestTezJobs#testOrderedWordCount' -q 2>&1 | tail -10
ls tez-tests/target/surefire-reports/

# 3. You can enumerate the state machine and its test coverage.
grep -c "addTransition" tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/VertexImpl.java
grep -c "public void test" tez-dag/src/test/java/org/apache/tez/dag/app/dag/impl/TestVertexImpl.java

When you can pick the right tier, boot the mini-cluster, drive a state machine deterministically through a DrainDispatcher, and reconstruct why a test was disabled, you have the testing fluency every Tez committer expects.

PR Profile: Level 5 Graduate

A contributor who has completed this level ships PRs that a committer can merge with confidence:

• Every change carries a test in the right tier, named and located so Surefire actually runs it.
• State-machine changes come with a TestVertexImpl/TestDAGImpl transition test that arranges the state, fires the event, await()s, and asserts on the resulting VertexState and any emitted event (e.g. DAGEventType.INTERNAL_ERROR).
• DAG-behavior changes come with a MiniTezCluster integration test that asserts the final DAGStatus state and a specific TezCounter, not just "it ran."
• Flaky-test work is backed by archaeology (the original JIRA), a reproduction, a root-cause classification, and evidence of N consecutive clean runs — never a silent @Ignore.

Continue to Level 6: Hive/Tez Integration.