Lab 5.1: MiniTezCluster and TestOrderedWordCount

Lab type: Read & Run Estimated time: 90 min Tez modules: tez-tests, tez-examples Key classes: org.apache.tez.test.MiniTezCluster, org.apache.tez.test.TestTezJobs

Background

An integration test boots real subsystems and runs a real DAG; a unit test mocks everything and drives one class. This lab is your first hands-on encounter with the integration tier. You will read the MiniTezCluster harness, trace exactly how the canonical TestTezJobs#testOrderedWordCount wires a MiniDFSCluster and a MiniTezCluster together in @BeforeClass, run it for real, and read its logs.

MiniTezCluster extends MiniYARNCluster (Hadoop). Inside one JVM it starts an in-process YARN ResourceManager and NodeManager(s); the test additionally starts a MiniDFSCluster (NameNode + DataNodes) so the DAG reads and writes real HDFS. The Tez ApplicationMaster (DAGAppMaster) then runs as an ordinary YARN application inside that mini-YARN. No external Hadoop install is required.

Note: The name is a trap. TestOrderedWordCount (tez-tests/src/main/java/org/apache/tez/mapreduce/examples/TestOrderedWordCount.java) is a runnable Tool example, not a JUnit test — it lives in src/main and has no @Test methods. The JUnit test that actually boots the mini-cluster and runs ordered word count is TestTezJobs#testOrderedWordCount in src/test, and the DAG it submits is org.apache.tez.examples.OrderedWordCount from tez-examples. TestSecureShuffle does invoke the TestOrderedWordCount example directly (via wordCount.run(args)) — that's the one place the example is exercised from a test.

Why This Lab Matters for Contributors

• Almost every DAG-level bug fix must ship an integration test in exactly this shape. If you cannot read TestTezJobs, you cannot add a test a committer will accept.
• Understanding what the mini-cluster boots — and what it costs — is the difference between a 3-second unit test and a 40-second integration test. Committers push back hard on using the heavy tier for logic that a unit test could prove.
• The @BeforeClass DFS+Tez wiring here is copy-paste boilerplate you will reuse verbatim in Lab 5.3.

Prerequisites

• Level 4 complete (you understand VertexImpl, DAGImpl, and the DAG model).
• export TEZ_SRC=/path/to/tez pointing at your real checkout.
• cd "$TEZ_SRC" && mvn install -DskipTests -q has succeeded (the AM jar must exist on disk, or the mini-cluster throws TezUncheckedException).
• Read the Testing Framework and YARN Integration deep dives.

cd "$TEZ_SRC"
find . -name "MiniTezCluster.java"                        # tez-tests/src/test/java/.../test/MiniTezCluster.java
find . -name "TestTezJobs.java"                           # tez-tests/src/test/java/.../test/TestTezJobs.java
find . -name "OrderedWordCount.java" -path "*examples*"   # tez-examples/src/main/java/.../examples/OrderedWordCount.java

Step-by-Step Tasks

Step 1: Read the MiniTezCluster lifecycle

grep -n "extends MiniYARNCluster\|public MiniTezCluster\|serviceInit\|serviceStart\|serviceStop\|APPJAR\|TEZ_LIB_URIS\|waitForAppsToFinish" \
  "$TEZ_SRC"/tez-tests/src/test/java/org/apache/tez/test/MiniTezCluster.java

Open the file and answer, citing the method you found it in (not a line number):

The constructor you will use most:

// From MiniTezCluster.java — (testName, numNodeManagers, numLocalDirs, numLogDirs)
public MiniTezCluster(String testName, int noOfNMs, int numLocalDirs, int numLogDirs) {
  super(testName, noOfNMs, numLocalDirs, numLogDirs);
}

Tip: serviceStop calling waitForAppsToFinish() is why a mini-cluster test can appear to "hang" for up to 30 seconds at teardown when a DAG did not finish cleanly. It is polling YARN for still-running apps and then killing them. This is a real source of slow test suites — remember it for Lab 5.4.

Step 2: Trace the @BeforeClass in TestTezJobs

grep -n "MiniDFSCluster\|MiniTezCluster\|@BeforeClass\|@AfterClass\|remoteFs\|localFs\|fs.defaultFS\|TEZ_AM_SLEEP_TIME_BEFORE_EXIT" \
  "$TEZ_SRC"/tez-tests/src/test/java/org/apache/tez/test/TestTezJobs.java | head -30

The setup is the reusable boilerplate. It looks like this in the source (TestTezJobs, setup()):

@BeforeClass
public static void setup() throws IOException {
  localFs = FileSystem.getLocal(conf);
  try {
    conf.set(MiniDFSCluster.HDFS_MINIDFS_BASEDIR, TEST_ROOT_DIR);
    dfsCluster = new MiniDFSCluster.Builder(conf).numDataNodes(2).format(true).racks(null).build();
    remoteFs = dfsCluster.getFileSystem();
  } catch (IOException io) {
    throw new RuntimeException("problem starting mini dfs cluster", io);
  }
  if (mrrTezCluster == null) {
    mrrTezCluster = new MiniTezCluster(TestTezJobs.class.getName(), 1, 1, 1);
    conf.set("fs.defaultFS", remoteFs.getUri().toString()); // use HDFS
    conf.setLong(TezConfiguration.TEZ_AM_SLEEP_TIME_BEFORE_EXIT_MILLIS, 500);
    mrrTezCluster.init(conf);
    mrrTezCluster.start();
  }
}

Answer:

Step 3: Trace the test body

grep -n "public void testOrderedWordCount\|generateOrderedWordCountInput\|verifyOutput\|OrderedWordCount job\|job.run" \
  "$TEZ_SRC"/tez-tests/src/test/java/org/apache/tez/test/TestTezJobs.java

The test (TestTezJobs, testOrderedWordCount()) does exactly this:

@Test(timeout = 60000)
public void testOrderedWordCount() throws Exception {
  String inputDirStr = "/tmp/owc-input/";
  Path inputDir = new Path(inputDirStr);
  Path stagingDirPath = new Path("/tmp/owc-staging-dir");
  remoteFs.mkdirs(inputDir);
  remoteFs.mkdirs(stagingDirPath);
  generateOrderedWordCountInput(inputDir, remoteFs);          // writes input to HDFS

  String outputDirStr = "/tmp/owc-output/";
  Path outputDir = new Path(outputDirStr);

  TezConfiguration tezConf = new TezConfiguration(mrrTezCluster.getConfig());
  tezConf.set(TezConfiguration.TEZ_AM_STAGING_DIR, stagingDirPath.toString());

  OrderedWordCount job = new OrderedWordCount();
  Assert.assertTrue("OrderedWordCount failed",
      job.run(tezConf, new String[]{"-counter", inputDirStr, outputDirStr, "2"}, null) == 0);
  verifyOutput(outputDir, remoteFs);                          // asserts the ordered output
}

Note what the assertions actually check. The primary gate is job.run(...) == 0 (the DAG succeeded). Then verifyOutput → verifyOrderedWordCountOutput reads the output file and asserts the counts are correct and in order. Read generateOrderedWordCountInput and confirm this arithmetic yourself:

• For each i in 1..10, the word a_i is written (11 - i) times to each of two files.
• So a_10 appears once per file, a_1 ten times per file. Combined count of a_i across both files is (11 - i) * 2.
• verifyOrderedWordCountOutput asserts exactly Assert.assertEquals((long)(11 - currentCounter) * 2, ...) walking currentCounter from 10 down to 1.

Step 3b: What this integration test exercises that a unit test cannot

This is the conceptual payoff of the lab. Contrast the two tiers on the same VertexImpl:

An integration test is the only place the whole path is exercised end to end: split calculation, container allocation, the umbilical, shuffle, counters, and the DAG state machine all interacting. That coverage is expensive, which is exactly why you reserve it for behavior a unit test cannot prove. The MiniDFSCluster here is not decoration — it gives the DAG a real filesystem so MRInput computes real splits and MROutput writes real files that verifyOutput reads back.

Step 4: Run it for real

Build first (the mini-cluster needs the AM jar on disk), then run the single method:

cd "$TEZ_SRC"
mvn install -DskipTests -q
mvn test -pl tez-tests -Dtest='TestTezJobs#testOrderedWordCount' 2>&1 | tail -25

Expected shape of the result (the exact numbers/timings vary, but the last lines look like):

Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
BUILD SUCCESS

If you instead see Unable to find a test class or No tests were executed, you almost certainly ran -Dtest=TestOrderedWordCount (the example) instead of -Dtest='TestTezJobs#testOrderedWordCount'.

Step 5: Read the logs and the report

Surefire captures per-test stdout/stderr and the mini-cluster's logs:

ls tez-tests/target/surefire-reports/
# Plain-text dump of everything the test JVM printed, incl. AM + YARN + DFS logs:
sed -n '1,60p' tez-tests/target/surefire-reports/org.apache.tez.test.TestTezJobs.txt
# Grep the captured output for the DAG lifecycle:
grep -n "Submitting dag\|DAG completed\|SUCCEEDED\|Starting MiniTezCluster" \
  tez-tests/target/surefire-reports/org.apache.tez.test.TestTezJobs-output.txt | head

In the captured output you should be able to see, in order: Starting MiniTezCluster (from MiniTezCluster.serviceStart), the DAG being submitted, task attempts running, and the DAG reaching SUCCEEDED. This is the log trail you will grep in every future integration-test debugging session.

Step 6: Time it and locate the overhead

time mvn test -pl tez-tests -Dtest='TestTezJobs#testOrderedWordCount' -q 2>&1 | tail -3

Answer: where does the wall-clock time go — booting MiniDFSCluster + MiniTezCluster, running the DAG, or the waitForAppsToFinish() teardown (up to 30s)? Cross-reference the timestamps in the -output.txt log. This is why integration tests use @BeforeClass (boot once per class) rather than @Before (boot per method).

Step 7: Survey the neighbourhood

grep -rl "MiniTezCluster" "$TEZ_SRC"/tez-tests/src/test/java/ | sort

Pick one that is not TestTezJobs — good choices are TestFaultTolerance (kills tasks/inputs), TestRecovery (AM restart), or TestSecureShuffle (SSL shuffle). Read its @BeforeClass and one @Test. Answer:

1. What scenario does it cover that testOrderedWordCount does not?
2. Does it reuse the same MiniTezCluster instance across methods (@BeforeClass), or start a fresh one per method (@Before, as TestSecureShuffle does because it toggles SSL)?

Deliverables

• Written answers to the Step 1–3 question tables, each citing a real method name (not a line number).
• A green run of TestTezJobs#testOrderedWordCount, with the final Tests run: 1, Failures: 0 line pasted in.
• The DAG-lifecycle log lines you grepped out of the Surefire -output.txt.
• A one-paragraph note on where the wall-clock time went (Step 6) and why @BeforeClass is used.
• A one-paragraph comparison of your Step 7 neighbour test vs testOrderedWordCount.

Troubleshooting

Stretch Goals

1. Add a counter assertion. Copy testOrderedWordCount's setup into a throwaway method, but submit the DAG via TezClient.submitDAG(OrderedWordCount.createDAG(...)) and assert dagClient.getDAGStatus(EnumSet.of(StatusGetOpts.GET_COUNTERS)).getDAGCounters() is non-null. This is the bridge to Lab 5.3.
2. Turn on DEBUG for one package. Add a log4j/slf4j override so org.apache.tez.dag.app logs at DEBUG, re-run, and find where the DAG transitions to RUNNING then SUCCEEDED in the log.
3. Read TestSecureShuffle. It is the one test that drives the TestOrderedWordCount example directly (wordCount.run(args)). Trace how it constructs args and why it starts a fresh MiniTezCluster per @Before (it toggles SSL between runs).

Deeper Understanding

Validation / Self-check

1. What class does MiniTezCluster extend, and what does that superclass boot?
2. Why does -Dtest=TestOrderedWordCount run zero tests, and what is the correct command?
3. In TestTezJobs.setup(), what is the exact wiring order of MiniDFSCluster, remoteFs, fs.defaultFS, and MiniTezCluster? Why does that order matter?
4. Where is the input written for testOrderedWordCount vs testOrderedWordCountDisableSplitGrouping?
5. Derive the expected combined count of the word a_3 across both input files, and show it matches the verifyOrderedWordCountOutput assertion.
6. Why can serviceStop make a passing test appear to hang for up to 30 seconds?
7. Why is @BeforeClass (not @Before) the right lifecycle hook for the mini-cluster?

When you can boot the mini-cluster, run the canonical DAG test, and read its log trail, continue to Lab 5.2: Add a Missing TestVertexImpl Transition Test.