Reading a 200k+ LOC Apache Codebase

Apache Tez is roughly 200,000 lines of Java across 17 Maven modules. No single human holds it all in their head — not even the founders who wrote most of it, most of whom no longer commit. The skill is not memory; it is navigation. This chapter gives you the strategies that work on Tez specifically, and a full worked reading protocol on one real class.

Everything here assumes a clone at ~/tez-src (git clone https://github.com/apache/tez.git ~/tez-src).

Module Map First

Before reading any code, learn the module shape. Run this once and pin the output:

cd ~/tez-src
find . -maxdepth 2 -name pom.xml -not -path '*/target/*' | sort

The modules that matter for ~90% of work, in the order data flows through them:

The rest — hadoop-shim, hadoop-shim-impls, tez-dist, tez-examples, tez-ext-service-tests, tez-ui — you touch rarely. Tez follows the Hadoop convention: code in <module>/src/main/java, tests in <module>/src/test/java, protobufs in <module>/src/main/proto.

The module dependency mental model

The Maven reactor order is the dependency order — a module can only depend on modules built before it. Read it from the parent pom.xml:

grep -A30 "<modules>" ~/tez-src/pom.xml | grep module

The mental model that follows: tez-api and tez-common are the foundation and depend on almost nothing internal. tez-dag (the AM) depends on them but not the other way around — the API cannot import AM internals, which is why tez-api is safe to read first and why a change there ripples everywhere. tez-runtime-library is where user-facing Input/Output/ Processor implementations live; it depends on the runtime and API but the AM does not depend on it. When you find yourself confused about "can class A see class B?", the reactor order answers it before you open the file.

Strategy 1: Start From an Entry Point, Trace Inward

Tez has exactly two entry points worth memorizing, and every trace starts at one of them.

The client entry is TezClient.submitDAG(DAG):

grep -n "submitDAG" tez-api/src/main/java/org/apache/tez/client/TezClient.java | head

You will find public synchronized DAGClient submitDAG(DAG dag) which branches to submitDAGSession (session mode) or submitDAGApplication (one AM per DAG). That method builds a DAGPlan and ships it over RPC. That arc — API builds a plan, sends it to the AM — is the canonical client-side read.

The AM entry is DAGAppMaster.main:

grep -n "public static void main" tez-dag/src/main/java/org/apache/tez/dag/app/DAGAppMaster.java

Everything the AM does hangs off that main. The server side of submitDAG lands in DAGClientHandler (note the real path — it is under dag/api/client/, not dag/app/):

find tez-dag/src/main/java -name DAGClientHandler.java
# tez-dag/src/main/java/org/apache/tez/dag/api/client/DAGClientHandler.java

The reading order for anything AM-side:

tez-api (what users build)      TezClient.submitDAG → DAGPlan
   ↓
tez-dag (what the AM does)       DAGClientHandler → DAGAppMaster → DAGImpl state machine
   ↓
tez-runtime-internals            LogicalIOProcessorRuntimeTask (what tasks run)
   ↓
tez-runtime-library              the Inputs/Outputs the task uses

Strategy 2: State-Machine-First for the AM

The hard core of tez-dag is four Hadoop-style finite state machines, one per entity in the DAG hierarchy: DAGImpl, VertexImpl, TaskImpl, TaskAttemptImpl (tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/). If you try to read these top-to-bottom you will drown — they are thousands of lines. Read the state-machine table instead. Each Impl class registers its transitions in a static StateMachineFactory block near the top. Find it:

grep -n "addTransition\|StateMachineFactory" \
  tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/VertexImpl.java | head -30

That block is the executable spec for the entity: it maps (current state, event) → (new state, transition class). To understand "what happens when a vertex's source completes," you find the event type (VertexEventType.V_SOURCE_VERTEX_STARTED or similar) in that table and jump straight to the named transition class. You never read the whole file; you read the one transition you care about. This is how committers navigate the AM, and it is the single most important Tez-specific reading trick.

Strategy 3: Protobufs Are the Source of Truth for Anything Serialized

Anything that crosses a process boundary (client → AM, AM → task, AM → history) is defined in protobuf. The protos are the contract; the Java is the implementation. Enumerate them — do not guess at names:

find ~/tez-src -name "*.proto" -not -path '*/target/*' | sort

The ones you will actually meet:

When you see a class like DAGProtos.DAGPlan, the generated code lives in target/generated-sources/ after a build. Don't read the generated Java; read the .proto. Practical rule: if you are changing a field that appears in a proto, you are changing wire compatibility. Stop and read Compatibility before you write the change.

Strategy 4: git log -S, Blame, and Code-Age Awareness

Two git tools replace most speculative reading.

git log -S answers "when and why did this string appear or disappear?":

git log -S "reconfigureVertex" --oneline -- tez-dag/
git log -S "reconfigureVertex" --oneline -- tez-api/

Pick the oldest commit and read it — the TEZ-NNNN in its message is the design discussion:

git show <sha> | head -30   # look for "TEZ-NNNN" in the first line

git log --follow shows a file's whole churn history, across renames, and teaches you code age — whether you are reading stable bedrock or a hot spot:

git log --follow --oneline -- \
  tez-runtime-library/src/main/java/org/apache/tez/dag/library/vertexmanager/ShuffleVertexManager.java | wc -l

That command returns around 50 commits for ShuffleVertexManager, spanning from TEZ-338 ("Determine reduce task parallelism") in the earliest days to recent Spotless hygiene. A file with 50 commits over a decade is a load-bearing, heavily-iterated class — read its history before you touch it. A file with three commits, all from one feature, is young and probably still owned by whoever wrote it. Age tells you how much implicit design you are about to walk into.

Strategy 5: Tests Are Executable Spec

The Tez test suite is the cheapest way to learn what a class does. For any class Foo.java, look for TestFoo.java; the test method names alone form a behavior spec:

find ~/tez-src -name "TestShuffleVertexManager.java"
grep -o "public void test[A-Za-z0-9]*" \
  $(find ~/tez-src -name TestShuffleVertexManager.java)

That prints behaviors like testLargeDataSize, testAutoParallelismConfig, and testSchedulingWithPartitionStats — a one-line map of what the class guarantees. For runtime behavior, the integration tests in tez-tests/ run full DAGs on a MiniTezCluster and are the gold standard for "how does this actually behave":

ls ~/tez-src/tez-tests/src/test/java/org/apache/tez/test/

Read the test before guessing. The author already encoded the behavior you are about to reverse-engineer.

Worked Protocol: Reading ShuffleVertexManager End to End

ShuffleVertexManager is a perfect specimen: it is @Public @Evolving, it drives Tez's signature feature (auto-reduce parallelism — deciding at runtime how many reducer tasks a vertex should have), and it has a rich, verifiable history. Here is the full protocol, with the real commands and what each step reveals.

Step 1 — Locate it and read the class contract (10 min)

find ~/tez-src -path '*vertexmanager/ShuffleVertexManager.java' -not -path '*/target/*'
sed -n '1,120p' \
  tez-runtime-library/src/main/java/org/apache/tez/dag/library/vertexmanager/ShuffleVertexManager.java

The header tells you three things immediately: the class is annotated @Public @Evolving (users depend on it, but the API may still change — see Compatibility); it extends ShuffleVertexManagerBase; and it declares its configuration surface as public constants — TEZ_SHUFFLE_VERTEX_MANAGER_ENABLE_AUTO_PARALLEL (default false), TEZ_SHUFFLE_VERTEX_MANAGER_DESIRED_TASK_INPUT_SIZE (default 100 * MB), and the slow-start min/max source-completion fractions. Those constants are the feature's knobs.

Step 2 — Read the history to find the design churn (15 min)

git log --follow --oneline -- \
  tez-runtime-library/src/main/java/org/apache/tez/dag/library/vertexmanager/ShuffleVertexManager.java

Scanning the ~50 lines of output, the design story emerges without opening a browser:

• The origin is TEZ-338 / TEZ-398 / TEZ-481 ("Determine reduce task parallelism", "Fix Reduce auto-parallelism after TEZ-398") — auto-parallelism was there almost from the start.
• TEZ-2242 ("Refactor ShuffleVertexManager code") and later TEZ-3395 ("Refactor ShuffleVertexManager to make parts of it re-usable") explain why ShuffleVertexManagerBase exists: the class was split so a fair-routing variant could share logic. If you had read only the current code you would wonder why the base class is there; the history tells you.
• Bug-fix commits like TEZ-3452 ("Auto-reduce parallelism calculation can overflow with large…") and TEZ-3222 ("Reduce messaging overhead for auto-reduce parallelism") mark the sharp edges — the exact scenarios that broke in production. These are the lines to read most carefully, because they encode hard-won corrections.

Step 3 — Jump from a suspicious line to its JIRA (5 min)

Pick any line whose intent is unclear and blame it:

git log -S "partition stats" --oneline -- \
  tez-runtime-library/src/main/java/org/apache/tez/dag/library/vertexmanager/

This surfaces TEZ-3303 ("Have ShuffleVertexManager consume more precise partition stats"). Open https://issues.apache.org/jira/browse/TEZ-3303 and you have the design discussion for why partition statistics are threaded through the manager at all. The next chapter, Design via JIRA, is entirely about turning that TEZ-NNNN into the "why."

Step 4 — Confirm behavior against the tests (10 min)

grep -o "public void test[A-Za-z0-9]*" \
  ~/tez-src/tez-runtime-library/src/test/java/org/apache/tez/dag/library/vertexmanager/TestShuffleVertexManager.java

testAutoParallelismConfig tells you exactly which config combinations enable the feature; testSchedulingWithPartitionStats tells you how partition stats change scheduling decisions. Read those two test methods and you understand the class's contract better than any comment would give you.

Step 5 — Record it (5 min)

Append the trace to a reading log (see below), citing file and JIRA for each hop. If you can reproduce this protocol tomorrow on a different class — say DagAwareYarnTaskScheduler, whose history is the worked example in Design via JIRA — you have the navigation skill.

Keep a Reading Log

Committers have working memory of the codebase because they wrote it. You don't. Compensate with notes. Keep one file and append a dated entry every time you trace a path:

mkdir -p ~/tez-notes
cat >> ~/tez-notes/reading-log.md <<'EOF'

## 2026-07-05 — ShuffleVertexManager auto-parallelism
- @Public @Evolving, extends ShuffleVertexManagerBase (split in TEZ-3395)
- knobs: ...ENABLE_AUTO_PARALLEL (default false), ...DESIRED_TASK_INPUT_SIZE (100MB)
- origin TEZ-338/398/481; precise partition stats added in TEZ-3303
- overflow bug fixed in TEZ-3452; messaging overhead reduced in TEZ-3222
- behavior pinned by TestShuffleVertexManager.testAutoParallelismConfig
EOF

Re-reading three months later, the log is gold. Without it, you re-trace the same path from zero and waste an afternoon.

Validation Artifacts

After this chapter you should have produced and kept:

1. ~/tez-notes/module-map.md — one sentence per module, derived from the reactor order.
2. ~/tez-notes/reading-log.md — the ShuffleVertexManager trace above, with file + JIRA per hop.
3. The StateMachineFactory transition block of one Impl class, read and understood.
4. One git log -S command you ran and the TEZ-NNNN it surfaced.

When you can run the worked protocol on a class you have never seen, without re-reading this page, you have the navigation skill. The next chapter — Design via JIRA — tells you where the design decisions behind that code actually lived.