Overview & Prerequisites

This section is the on-ramp. Before you read a single line of DAGImpl.java or trace a DAG through TaskSchedulerManager, you need a working build, the ability to run a DAG in local mode, an Apache JIRA identity wired for contribution, and a clear mental map of how the curriculum is structured. This page gets you there. Budget two to four hours for a cold setup; most of that is the first Maven build downloading the world and provisioning protoc.

This curriculum will not hold your hand. It assumes you are a strong backend / distributed-systems engineer who can read unfamiliar Java without a guide. What it will do is point you at the exact parts of Apache Tez that matter, give you the right questions, and make you prove competence at each gate. The setup below is the first gate: if you cannot build Tez and run OrderedWordCount in local mode, nothing else in the curriculum will work.

Who This Curriculum Is For

This is an engineering apprenticeship for people who want to contribute to Apache Tez at a serious level and eventually operate as a committer or PMC-aware engineer. It is built for engineers who:

• Have 3+ years of Java and are comfortable in large, unfamiliar codebases.
• Know the Hadoop ecosystem at least conceptually — YARN, HDFS, MapReduce, and that Hive runs on Tez.
• Understand distributed execution: task graphs, scheduling, data movement, shuffle, speculative execution, fault tolerance.
• Want to do real work — reproduce a JIRA, fix it, write a test, and defend the patch on a mailing list — not just fix typos.

If Tez's premise (a DAG execution engine on YARN that replaced chained MapReduce jobs under Hive) is new to you, that is fine — the Introduction and the Warm-Up build it. What you cannot skip is the Java and the willingness to read code.

Prerequisites self-check

You are ready to start if you can answer yes to most of these. Where you cannot, the Warm-Up and Level 1 close the gap — but be honest, because a shaky baseline makes every later failure confusing.

Hardware & OS Requirements

Tez builds and runs its unit tests comfortably on a developer laptop; the mini-cluster integration tests are the heavy part.

• OS: Linux or macOS. Windows works only via WSL2 — the mini-cluster tests assume a POSIX filesystem and shell.
• RAM: 16 GB is the practical floor. The build + IDE + a MiniTezCluster test can push past 8 GB.
• Disk: ~10 GB for the source, the Maven local repository (~/.m2), and build output.
• CPU: Any modern multi-core. Level 9's benchmark DAG and JMH runs want a quiet machine — close other load when you measure.

What You Are Setting Up

+--------------------------------------------------------------------------+
|  Your laptop                                                            |
|                                                                        |
|   JDK 21  --- runs --->  IntelliJ / your editor                        |
|      |                                                                 |
|      |  imports as a Maven project                                     |
|      v                                                                 |
|   ~/src/oss-repos/tez  <-- git clone -- github.com/apache/tez          |
|      |                                                                 |
|      |  mvn install  (protoc provisioned by protoc-jar-maven-plugin)   |
|      v                                                                 |
|   java ... OrderedWordCount -local  --->  a DAG runs in ONE JVM        |
|      ^                                                                 |
|      |  you read counters / attach a profiler here                     |
|   git format-patch / PR  --->  JIRA + dev@tez.apache.org review        |
+--------------------------------------------------------------------------+

Two facts shape everything:

1. Tez builds with Maven (multi-module). There is no build-tool wrapper checked in (.mvn/ holds only project config), so you install Maven yourself. Protobuf is handled by the protoc-jar-maven-plugin, which provisions a matching protoc during the build — you usually do not install protoc by hand.
2. Contribution happens on Apache infrastructure. Issues live in Apache JIRA; code review happens on GitHub pull requests against apache/tez and on dev@tez.apache.org. There is no DCO/CLA gate like some projects, but there is a culture of tests-with-patches and mailing-list discussion — see the Release & PMC section.

Step 1 — Install the toolchain

java -version      # 21 for master; 8/11 for 0.10.x
mvn -version       # 3.6.3+
git --version      # 2.x

Note on Java version: always check the branch. master is 1.0.0-SNAPSHOT and requires JDK 21; the releases people actually run (0.10.3–0.10.5) build on JDK 8. Confirm with grep -n "maven.compiler.release\|javaVersion" pom.xml on your checkout.

Step 2 — Clone the repositories

mkdir -p ~/src/oss-repos && cd ~/src/oss-repos

# The engine — your home for the whole curriculum.
git clone https://github.com/apache/tez.git
cd tez
git log --oneline -5          # confirm you're on master
git tag | grep release-0.10   # see the release line users run

# OPTIONAL — Hadoop, for YARN API context and integration reference (Levels 6–8).
cd ~/src/oss-repos
git clone https://github.com/apache/hadoop.git

Warning: the first mvn install downloads a large dependency set and provisions protoc. Do it on a good connection and expect 15–30 minutes the first time. Subsequent builds use the Maven cache and are far faster.

Read these in the repo root before you build — they are the project's own contract:

cd ~/src/oss-repos/tez
ls README.md INSTALL.md pom.xml

Step 3 — First build and first run

cd ~/src/oss-repos/tez

# Build everything, skip tests and javadoc for speed on the first pass.
mvn install -DskipTests -Dmaven.javadoc.skip=true

# Run a single test class — your fast feedback loop for the whole curriculum.
mvn test -pl tez-dag -Dtest=TestDagAwareYarnTaskScheduler -q 2>&1 | tail -10

Now run a real DAG. Local mode (-local) runs the AM and all tasks in one JVM — the simplest way to run, debug, and profile a DAG:

# Build just the examples and their deps.
mvn package -DskipTests -pl tez-examples -am -q

mkdir -p /tmp/tez-lab/input
printf 'the quick brown fox\nthe lazy dog\nquick brown dog\n' > /tmp/tez-lab/input/words.txt

TEZ_HOME=~/src/oss-repos/tez
CP=$(echo $TEZ_HOME/tez-*/target/tez-*.jar | tr ' ' ':'):$(hadoop classpath)
rm -rf /tmp/tez-lab/output
java -cp "$CP" org.apache.tez.examples.OrderedWordCount \
  -local -counter /tmp/tez-lab/input /tmp/tez-lab/output 1

You want the DAG to finish SUCCEEDED and (with -counter) print an aggregated TezCounters block. If mvn test passes a known-good class and OrderedWordCount completes in local mode, your environment is solid.

Note on protoc: if the build fails with a protobuf/protoc error, the protoc-jar-maven-plugin could not provision the compiler. Check network access, or set PROTOC_PATH to a locally installed protoc matching the protobuf.version in pom.xml (currently 3.25.5).

Step 4 — Import into IntelliJ (as a Maven project)

1. IntelliJ → File → Open → select the ~/src/oss-repos/tez directory (the folder, not a single file). Choose Open as Project and trust it.
2. Let IntelliJ detect the Maven build and finish the initial import (it indexes tez-api, tez-dag, tez-runtime-library, …). This takes a while the first time.
3. Set the Project SDK to JDK 21 (for master) in Project Structure → Project.
4. Verify navigation: Go to Class and open TezClient, DAGAppMaster, DAGImpl, DagAwareYarnTaskScheduler. If those resolve, your index is healthy.

Tip: the single highest-leverage skill in this curriculum is setting a breakpoint in a state-machine transition (e.g. in TaskAttemptImpl) and stepping through a local-mode DAG. Get run/debug-from-the-gutter working early.

Step 5 — Set up Apache JIRA and the mailing lists

Tez tracks work in Apache JIRA and discusses it on mailing lists. Get plugged in now so you are not a stranger when you open your first issue or patch.

Learn the JIRA fields you will use constantly: Component (tez-dag, tez-api, tez-runtime-library, …), Priority, Fix Version, and the beginner labels for a first issue. The Release & PMC section explains how the mailing lists and voting actually work.

How the Curriculum Is Organized

The curriculum is 9 sequential levels of core engineering, 5 supporting sections, and a capstone.

L1 Hadoop & Tez Foundation ─▶ L2 Contributor Onboarding ─▶ L3 Tez Architecture
   ─▶ L4 DAG State Machine Internals ─▶ L5 Testing & Debugging
   ─▶ L6 Hive/Tez Integration ─▶ L7 Runtime & Shuffle
   ─▶ L8 Real Issue Contribution ─▶ L9 Advanced Committer / PMC ─▶ Capstone

   Deep Dives ....... referenced by L3, L4, L7
   Hive-on-Tez Labs . around L6
   Issue Roadmap .... pick real issues as you progress
   Release & PMC .... L9 and the capstone

The deep dives are not optional reading — they are where the real depth lives. A level says "trace DAG submission"; the DAG App Master deep dive is where you learn how DAGAppMaster actually wires the run. Treat the levels as the spine and the deep dives as the muscle.

The mindset progression this curriculum drives you along:

reader ─▶ builds & runs Tez ─▶ reads the engine ─▶ fixes a real JIRA
      ─▶ writes tests that catch real bugs ─▶ proves perf & guards compatibility
      ─▶ reviews others' patches ─▶ committer candidate

How to Use the Labs

Every lab follows the same shape, so you always know where you are:

• Background and Why This Lab Matters for Contributors — the why before the how.
• Prerequisites — what must already work (usually a green build and a prior lab).
• Step-by-Step Tasks — numbered, with real mvn/java/git/rg commands and expected output.
• Deliverables — checkboxes you must satisfy.
• Troubleshooting, Stretch Goals, and a Validation / Self-check that gates completion.

Rules for the labs:

1. Run every command. This is a hands-on apprenticeship; reading is not doing.
2. When a lab gives an rg/find, run it rather than trusting a line number — Tez code moves between branches, so the curriculum points you at code with commands, not fabricated line numbers.
3. Do not advance past a lab's Validation section until you can answer it without notes.
4. Keep a scratch branch for experiments so git bisect and reverts stay clean.

You Are Ready When…

Confirm every box before opening Level 1:

• java -version reports the JDK your branch needs (21 for master); mvn -version is 3.6.3+.
• ~/src/oss-repos/tez is cloned and you have read README.md and INSTALL.md.
• mvn install -DskipTests -Dmaven.javadoc.skip=true completes cleanly.
• mvn test -pl tez-dag -Dtest=TestDagAwareYarnTaskScheduler passes.
• OrderedWordCount -local runs a DAG to SUCCEEDED and prints counters.
• IntelliJ imported the project as Maven and Go to Class finds DAGAppMaster.
• You have an Apache JIRA account and are subscribed to dev@tez.apache.org.

# A 5-minute "am I ready" smoke test, from ~/src/oss-repos/tez:
java -version
mvn -version | head -1
mvn test -pl tez-dag -Dtest=TestDagAwareYarnTaskScheduler -q 2>&1 | tail -5

If any box is unchecked, fix it now. A broken baseline means every later mvn test and every DAG run produces confusing failures that hide the real work.

Where to Go Next

• Tez Warm-Up: From Data Engineer to Source Contributor — the most important page in this section. Run Tez as a user first, then bridge to the org.apache.tez.* source. Read this before Level 1.
• 16-Week Plan — a calendar mapping Levels 1–9 + capstone onto 16 weeks, with weekly reading, hands-on tasks, JIRA practice, and exit checkpoints.
• Milestones: M1–M9 — the competence gates, each with skills, self-check questions, and a rubric.

Continue to the Warm-Up, or jump straight to Level 1: Hadoop and Tez Foundation.