Lab 1.3: Run a Simple Tez DAG Locally

Background

Apache Tez supports a local mode that runs an entire DAG — Application Master, vertices, and tasks — inside a single JVM, with no YARN cluster and no HDFS. This is the primary environment for rapid development and for most of the integration tests you will read later. Being able to run a DAG locally, watch its lifecycle in the logs, and reason about what each line means is the bridge between "I built Tez" (Lab 1.1) and "I understand what Tez does."

You will run a real shipped example — OrderedWordCount from the tez-examples module — in local mode, then read its log output as the DAG, its vertices, and its tasks move through their state machines. OrderedWordCount is the canonical Tez example: it tokenizes text, sums word counts, and emits words ordered by frequency, using the full public DAG API (DAG, Vertex, Edge, MRInput/MROutput, and an ordered partitioned shuffle edge).

This lab depends on a clean build (Lab 1.1). The mechanics of local mode are dissected further in the local-mode deep-dive; the DAG structure you run here is the subject of the DAG-model deep-dive.

Why This Lab Matters for Contributors

• Local mode is how you verify a behavior change without standing up a cluster — your daily loop.
• Every integration test in tez-tests is built on the same local/mini-cluster infrastructure.
• Reading a real DAG's construction gives concrete meaning to the state-machine code you study in Level 4 — the log lines here are state transitions.
• Local-mode-only bugs and DAG-API usability issues are a real, accessible contribution surface.

Prerequisites

• Lab 1.1 complete, including Step 7 (the tez-dist tarballs).
• The JDK/protoc prerequisites from Lab 1.1 still on your PATH.
• You know your build's project version string (mvn help:evaluate -Dexpression=project.version -q -DforceStdout).
• ~1 GB free disk for input/output scratch under /tmp.

Understanding Tez Local Mode

Local mode is switched on by a single configuration key. Read the real definition and its default:

grep -n "TEZ_LOCAL_MODE\b\|local.mode.without.network\|OPTIMIZE_LOCAL_FETCH" \
  tez-api/src/main/java/org/apache/tez/dag/api/TezConfiguration.java

You will find the verified keys (do not hard-code the string values; read them):

Crucially, you rarely set these by hand for a shipped example. TezExampleBase — the base class of every example in tez-examples — accepts a -local command-line flag that does the wiring for you. Confirm it in the source:

grep -n "LOCAL_MODE\|TEZ_LOCAL_MODE\|OPTIMIZE_LOCAL_FETCH\|disableSplitGrouping" \
  tez-examples/src/main/java/org/apache/tez/examples/TezExampleBase.java

When you pass -local, TezExampleBase sets TEZ_LOCAL_MODE=true, turns on TEZ_RUNTIME_OPTIMIZE_LOCAL_FETCH, and disables split grouping — exactly what a single-JVM run needs. That is the supported, real recipe; use it rather than hand-rolling -Dtez.local.mode=true.

Anatomy of OrderedWordCount

Read the example before running it:

sed -n '1,120p' tez-examples/src/main/java/org/apache/tez/examples/OrderedWordCount.java
grep -n "Vertex.create\|Edge.create\|addVertex\|addEdge\|newBuilder" \
  tez-examples/src/main/java/org/apache/tez/examples/OrderedWordCount.java

The DAG has three vertices (the names are string constants — Tokenizer and Summation come from WordCount, Sorter is defined in OrderedWordCount):

[Tokenizer]  MRInput(TextInputFormat) → emits (word, 1)
     │  OrderedPartitionedKV edge  (key=Text, value=IntWritable, HashPartitioner)
     ▼
[Summation]  sums per word → writes (count, word)   ← count is the KEY
     │  OrderedPartitionedKV edge  (key=IntWritable, value=Text, HashPartitioner)
     ▼
[Sorter]     1 task; the ordered edge already sorted by count → MROutput(TextOutputFormat)

flowchart TD
    A["Tokenizer<br/>TokenProcessor<br/>MRInput → (word, 1)"]
    B["Summation<br/>SumProcessor<br/>sum → (count, word)"]
    C["Sorter<br/>NoOpSorter (1 task)<br/>→ MROutput"]
    A -->|"OrderedPartitioned KV<br/>(Text, IntWritable)"| B
    B -->|"OrderedPartitioned KV<br/>(IntWritable, Text)"| C

The key insight: the sort happens on the edge, not in a processor. SumProcessor writes the count as the key; the ordered partitioned edge into Sorter sorts and groups by that count key, so NoOpSorter (a SimpleMRProcessor) only forwards the already-ordered data to MROutput. This is the essence of the Tez model — behavior is expressed through edge properties and I/O configuration, not just processor code. You will build the same shape from scratch, with integers instead of words, in Lab 1.4.

Step-by-Step Tasks

Step 1: Prepare Sample Input

mkdir -p /tmp/tez-lab/input
cat > /tmp/tez-lab/input/words.txt << 'EOF'
the quick brown fox jumps over the lazy dog
the dog barked at the fox
quick brown dog
EOF
rm -rf /tmp/tez-lab/output   # MROutput refuses to overwrite an existing dir

Step 2: Build the Examples and Assemble a Runnable Classpath

The cleanest, self-contained way to run from a source build is the exploded full distribution you produced in Lab 1.1 Step 7 — it contains every Tez jar plus the bundled Hadoop and third-party jars, so no separate Hadoop install is needed.

# Ensure examples + the dist are built:
mvn package -DskipTests -Pnoui -pl tez-examples -am -q
mvn package -DskipTests -Pnoui -pl tez-dist -am -q

# The assembly emits a 'dir' format alongside the tarball:
VERSION=$(mvn help:evaluate -Dexpression=project.version -q -DforceStdout)
export TEZ_DIST="tez-dist/target/tez-${VERSION}"
ls "$TEZ_DIST"        # tez-*.jar at the root; third-party + hadoop jars under lib/

The distribution layout is defined by tez-dist/src/main/assembly/tez-dist.xml: the Tez module jars land at the root (/) and their runtime dependencies under /lib. So a classpath of "$TEZ_DIST/*:$TEZ_DIST/lib/*" is everything you need.

Note: If you use the minimal tarball instead, Hadoop jars are excluded by design (see tez-dist-minimal.xml), so you must append $(hadoop classpath) from a real Hadoop install. The full dist avoids that.

Step 3: Run OrderedWordCount in Local Mode

The -local flag is the supported switch (Step "Understanding Tez Local Mode"). Options come before the positional arguments <input> <output> [numPartitions]:

java -cp "$TEZ_DIST/*:$TEZ_DIST/lib/*" \
  org.apache.tez.examples.OrderedWordCount \
  -local -counter \
  /tmp/tez-lab/input \
  /tmp/tez-lab/output \
  1

-counter also prints DAG counters at the end (it maps to the COUNTER_LOG option in TezExampleBase). You can equally launch through the examples driver — the tez-examples jar's manifest main class is org.apache.tez.examples.ExampleDriver, which registers orderedwordcount among other examples:

grep -n "addClass" tez-examples/src/main/java/org/apache/tez/examples/ExampleDriver.java
java -cp "$TEZ_DIST/*:$TEZ_DIST/lib/*" \
  org.apache.tez.examples.ExampleDriver orderedwordcount \
  -local /tmp/tez-lab/input /tmp/tez-lab/output 1

Tip: If a run fails because the output dir exists, rm -rf /tmp/tez-lab/output and retry. This FileAlreadyExistsException is Hadoop's MROutput refusing to clobber — the same behavior you'd hit on a cluster.

Step 4: Verify the Output

cat /tmp/tez-lab/output/part-*

Words appear ordered by count (the exact tie-break order among equal counts depends on the sort):

1	jumps
1	over
1	lazy
1	barked
1	at
2	quick
2	brown
2	fox
3	dog
4	the

Step 5: Read the Execution Log — Map Lines to State Transitions

This is the real payoff. During the run, Tez logs the lifecycle of the DAG, each vertex, and each task. The log-line formats are stable and worth memorizing; confirm the exact strings in source so you can grep them in any future run:

grep -n "Running DAG:" tez-dag/src/main/java/org/apache/tez/dag/app/DAGAppMaster.java
grep -n "transitioned from" \
  tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/DAGImpl.java \
  tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/VertexImpl.java
grep -n "DAG completed, dagId=" tez-dag/src/main/java/org/apache/tez/dag/app/DAGAppMaster.java

The verified formats you will see in the console:

For each vertex transition line, name the state pair. For example a Tokenizer line moving NEW → INITIALIZING → INITED → RUNNING → SUCCEEDED is the exact path you will trace in the vertex-lifecycle deep-dive. Write down the sequence you observe for one vertex — you will reconcile it against the state machine in Level 4.

Step 6: Turn Up the Volume (Optional)

To see every transition clearly, raise the log level for the impl package and re-run:

java -cp "$TEZ_DIST/*:$TEZ_DIST/lib/*" \
  -Dtez.root.logger=INFO,console \
  org.apache.tez.examples.OrderedWordCount \
  -local /tmp/tez-lab/input /tmp/tez-lab/output2 1 2>&1 | grep "transitioned from"

Count how many distinct vertex/DAG transitions a three-vertex DAG produces. It is more than you expect — which is why the state machines need 100+ tests (Lab 1.2).

Deliverables

• OrderedWordCount -local runs to completion and writes a part-* file.
• The output is ordered by count, and you can explain why the edge, not a processor, sorts it.
• You located the four real log-line formats (Running DAG, DAG transition, vertex transition, DAG completed) in the tez-dag source with grep.
• You wrote down the full state sequence for one vertex from the log.
• You can state which config key enables local mode and which flag on TezExampleBase sets it.

Troubleshooting

Stretch Goals

1. Watch the transitions live. Re-run piping through grep "transitioned from" and reconstruct the DAG-level path (NEW → INITED → RUNNING → SUCCEEDED). Confirm it against dag-model.

2. Change parallelism. Run with numPartitions = 2 (the third positional arg). Observe two part-* files and two Summation task transitions in the log. Reason about which vertex's parallelism that argument actually controls (read createDAG).

3. Try another shipped example in local mode. WordCount, SimpleSessionExample, HashJoinExample, and JoinDataGen/JoinValidate are all registered in ExampleDriver. Run:

   java -cp "$TEZ_DIST/*:$TEZ_DIST/lib/*" org.apache.tez.examples.ExampleDriver
   

   with no args to list them, then run wordcount -local and compare its log to orderedwordcount.

4. Preview an added vertex. Before you build one from scratch in Lab 1.4, sketch (on paper) how you would insert a filter vertex between Summation and Sorter that drops words with count < 2: which two edges change, and what processor code the new vertex needs. Lab 1.4 makes you actually write it.

5. (Cluster, optional/awareness) On a real pseudo-distributed YARN, local mode is not used. Instead you upload the dist tarball to HDFS and set tez.lib.uris:

   grep -n "tez.lib.uris\|use.cluster.hadoop-libs\|mapreduce_shuffle" \
     docs/src/site/markdown/install.md
   

   Set tez.lib.uris to the HDFS path of tez-<version>.tar.gz, ensure YARN's NodeManager aux-services includes the shuffle handler (mapreduce_shuffle), and drop -local. This is the real deployment path you will care about only from Level 5 onward.

Validation / Self-check

You are done when you can answer these without notes:

1. Which single configuration key enables local mode, what is its default, and which TezExampleBase command-line flag turns it on for a shipped example?
2. In OrderedWordCount, what does the sorting — a processor or an edge — and how do you know from the code?
3. What is the exact log-line format that reports a vertex state transition, and what does the bracketed [...] part contain?
4. Why does the second run fail if you don't delete the output directory, and where does that behavior come from?
5. What is the difference between the full and minimal dist for the purpose of running an example locally, and what do you have to add if you use the minimal one?
6. Trace the DAG-level state path you observed from the logs (NEW → … → SUCCEEDED).
7. On a real cluster you would drop -local — what two things must be configured instead (tez.lib.uris and the NodeManager shuffle aux-service)?

Where to go next: you have now run someone else's DAG and watched it live. In Lab 1.4 — Number Pipeline DAG you build the same three-vertex shape from scratch, in code you own, and assert the result. Deepen the mechanics with the local-mode, vertex-lifecycle, and dag-model deep-dives, then carry the skill into Level 2.