Lab 2.1: Navigate the Repository Structure

Background

Apache Tez is a Maven multi-module project — the root pom.xml lists seventeen modules, and the build order between them is not alphabetical; it is a dependency graph. A contributor who cannot say, from memory, which module owns a class and what that module is allowed to depend on will edit the wrong layer, break a public API by accident, or spend twenty minutes hunting for a file a committer finds in five seconds.

This lab is a guided tour. You will read the real root pom.xml, walk every module, derive the inter-module dependency arrows from the module POMs, learn to locate any class three different ways, and read the .github, .asf.yaml, and dev-support files that govern how your future PRs are built and checked. You will not change any code — but the muscle memory you build here makes every later lab fast.

Why This Lab Matters for Contributors

• "Where does this live?" is the question you answer dozens of times per PR. Seconds instead of minutes compounds.
• Editing the right layer matters: a change in tez-api is a public API change with compatibility implications; the same logic in tez-runtime-library is internal. Reviewers hold them to different bars.
• Reading a module POM's <dependency> block tells you what you may call from where — and why some "obvious" imports are architecturally forbidden.
• The .github/.asf.yaml/dev-support files are the contract your PR is judged against. Knowing them up front saves a red CI run.

Prerequisites

• Lab 1.1 complete — you can run mvn against a Tez checkout.
• A clone of apache/tez (your fork is fine; fork setup is Lab 2.2).
• git, grep/rg, and find on your PATH.

Step-by-Step Tasks

Step 1: Read the Real Module List

Do not trust any hand-drawn tree — read the source of truth. The <modules> block in the root POM is the authoritative, ordered list:

cd /path/to/tez
grep -n "<module>" pom.xml

You should see seventeen modules. The order is the coarse build order. Here is what each one is, its key packages, and when you would touch it:

Note: tez-plugins is an aggregator whose active submodules depend on a profile. On the default hadoop28 profile they are tez-protobuf-history-plugin, tez-yarn-timeline-history, tez-yarn-timeline-history-with-acls, tez-yarn-timeline-cache-plugin, tez-yarn-timeline-history-with-fs, tez-history-parser, and tez-aux-services. Verify with sed -n '/hadoop28/,/\/profile/p' tez-plugins/pom.xml.

Warning: The class you are looking for may not be in the module its name suggests. The ID classes (TezTaskAttemptID and friends) are in package org.apache.tez.dag.records but live in tez-common, not tez-dag. And TezUtils is in package org.apache.tez.common but lives in tez-api, not tez-common. Package ≠ module. Always verify with find.

Step 2: Derive the Dependency Direction

Modules declare their inter-dependencies explicitly, and Maven uses those to order the build. Extract the tez-to-tez edges yourself instead of trusting a diagram:

for m in tez-api tez-common tez-runtime-internals tez-runtime-library \
         tez-mapreduce tez-dag tez-examples tez-tests; do
  echo "== $m depends on =="
  grep -A2 "<groupId>org.apache.tez" "$m/pom.xml" \
    | grep "artifactId" | grep -v "$m<" | sort -u
done

The arrows this reveals (read "→" as "depends on"):

hadoop-shim  ── (base; no tez deps)
tez-api      ── (near-base; only build-time tez-javadoc-tools)
tez-common          → tez-api, hadoop-shim
tez-runtime-internals → tez-api, tez-common, hadoop-shim
tez-runtime-library → tez-api, tez-common, tez-runtime-internals
tez-mapreduce       → tez-api, tez-common, tez-runtime-library, tez-runtime-internals, hadoop-shim
tez-dag             → tez-api, tez-common, tez-runtime-library, tez-runtime-internals, hadoop-shim
tez-examples        → tez-api, tez-common, tez-runtime-library, tez-mapreduce
tez-tests           → tez-api, tez-common, tez-runtime-library, tez-mapreduce, tez-examples, tez-dag

Two rules fall out of this, and they explain a lot of review feedback:

1. tez-api is at the bottom. Everything depends on it; it depends on almost nothing. That is why it is the public API and why changes there are scrutinised — you cannot break it without breaking everyone above.
2. The arrows point one way. tez-common may use tez-api, but tez-api may not use tez-common. If a reviewer says "this helper belongs in tez-common, not tez-api," it is because putting it in tez-api would either be an unwanted public API or create a cycle.

Tip: To see the fully resolved graph including external jars: mvn dependency:tree -pl tez-dag | grep "org.apache.tez".

Step 3: Walk tez-dag — the Application Master

This is where you will spend most of Levels 3–4. Get its shape:

ls tez-dag/src/main/java/org/apache/tez/dag/app
find tez-dag/src/main/java -name "*Impl.java" -path "*dag/impl*"

The state machines are the heart of it. Confirm DAGImpl is built on Hadoop's StateMachineFactory:

grep -n "StateMachineFactory" tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/DAGImpl.java | head
grep -c "addTransition" tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/VertexImpl.java

The addTransition count is the number of state-machine edges in VertexImpl — a rough measure of why that class is the one people fear. You will read these in Level 3.

Step 4: Walk tez-runtime-library — I/O and Shuffle

The data-plane module. Shuffle fetch and external sort live here:

find tez-runtime-library/src/main/java -name "*Input*.java"  -path "*/library/*" | grep -v test
find tez-runtime-library/src/main/java -name "Fetcher.java"
find tez-runtime-library/src/main/java -name "ShuffleManager.java"

Note the shuffle infrastructure is under .../common/shuffle. When a JIRA is filed against "shuffle retry" or "fetch failure," this is the module and package.

Step 5: Locate Any Class — Three Ways

The core skill, fastest first:

# 1. By file name — when you know the class:
find . -name "VertexImpl.java" -not -path "*/target/*"

# 2. By declaration — when you want the exact definition site:
grep -rn "class DAGAppMaster" --include=*.java . | grep -v target

# 3. By behavior/constant — when you only know what it does:
grep -rln "TEZ_LOCAL_MODE" --include=*.java . | grep -v target

Practice until reflex. Find each and note the module it resolves in:

for c in TezClient TezConfiguration DAGAppMaster VertexImpl \
         ShuffleManager Fetcher MRInput TezTaskAttemptID TezUtils; do
  echo "== $c =="
  find . -name "$c.java" -path "*/main/*" -not -path "*/target/*"
done

Confirm the two traps from Step 1: TezTaskAttemptID resolves under tez-common (.../dag/records/), and TezUtils under tez-api (.../common/) — package names lie about module location, find does not.

Step 6: Read .asf.yaml — the Contribution Model as Config

.asf.yaml is not documentation about the process; it is the process, enforced by ASF infra. Read it:

cat .asf.yaml

What to extract, and why it matters for your PRs:

Step 7: Read .github/ and dev-support/

ls -R .github
cat .github/workflows/build.yml
ls dev-support dev-support/spotless
head -60 dev-support/tez-personality.sh

Note three things:

1. .github contains only workflows/build.yml. There is no pull_request_template.md and no issue templates — because issues are on JIRA. Do not expect a checklist to be auto-filled into your PR; write the body yourself.
2. build.yml runs mvn clean install -DskipTests across a Java [21, 25] × [ubuntu-latest, macos-latest] matrix on every push and PR to master. That is a compile gate, not a test gate.
3. dev-support/tez-personality.sh is the Yetus personality — it decides which precommit checks fire for which files. Read personality_file_filter: a changed .java, .proto, or pom.xml triggers javac, spotbugs, checkstyle, and javadoc; a .sh or Jenkinsfile triggers shellcheck; a .md or .txt triggers codespell. dev-support/spotless/license.java is the exact ASF header spotless will enforce on any new file you add.

The tests themselves run under Jenkins via the Jenkinsfile (Yetus rel/0.15.1 in Docker). Skim it to see that it diffs origin/master...HEAD, runs test-patch.sh, and posts a GitHub comment with emoji votes — that comment is the precommit result you will respond to in Lab 2.2.

Deliverables

• A module table you filled in yourself from grep -n "<module>" pom.xml — all seventeen, each with role and key package.
• The dependency-arrow list from Step 2, produced by your own loop over the module POMs.
• The resolved file path (from memory, then verified) for: DAGAppMaster, VertexImpl, ShuffleManager, Fetcher, TezTaskAttemptID, TezUtils.
• Two sentences explaining why tez-api sits at the bottom of the dependency graph and what that implies for changes to it.
• A note of the four .asf.yaml settings from Step 6 and what each one forces you to do.
• Confirmation, in your own words, that .github has no PR template and what that means for how you write a PR body.

Troubleshooting

Stretch Goals

1. Draw the real dependency DAG. From the Step 2 output, render a graph (by hand or with mvn dependency:tree) and verify it is acyclic. Find the longest path from hadoop-shim to tez-tests.
2. Find every .proto. Run find . -name "*.proto" -not -path "*/target/*" | sort. For each, name its module and one message it defines. Protobuf changes are wire-format changes — note which module owns the DAG plan protos.
3. Map the checkstyle gate to a rule. Open tez-build-tools/src/main/resources/checkstyle/checkstyle.xml, find the LineLength module, and record the max. Then find one file that would violate it and confirm with mvn checkstyle:check -pl <its-module>.
4. Trace a config key to its owner. Pick TEZ_AM_RESOURCE_MEMORY_MB (or any key), find its declaration in TezConfiguration (tez-api), and then find a reader of it in tez-dag. You have just crossed the public-API → internal boundary the dependency graph predicts.

Validation / Self-check

You are done when you can answer these without notes:

1. How many modules does the root pom.xml list, and which single module is the public API surface?
2. In which direction may dependencies point between tez-api and tez-common, and how do you verify it from a POM?
3. Which module physically contains TezTaskAttemptID, and why is its module different from what its package name suggests?
4. Give the three ways to locate a class; which do you use when you know only the behavior?
5. What does .asf.yaml's enabled_merge_buttons tell you about how your commits will land, and what does features: issues: false tell you about where bugs are filed?
6. Which file decides which Yetus precommit checks run for a changed .java file, and what are those checks?

Next: Lab 2.2 — Prepare a Patch Using Apache Practices.