Appendices & Reference

The rest of this curriculum is meant to be read: first principles, then the data model, then the real org.apache.tez.* classes, then a lab where you prove you understood it. This appendix is the opposite kind of document. It is meant to be looked up. When you are mid-task — staring at a TaskAttemptImpl state transition in the AM log, drafting a patch, tuning tez.runtime.io.sort.mb, or trying to remember which class turns a DataMovementEvent into a fetch — you do not want to re-read a 600-line deep dive. You want a table, an exact name, and a one-click jump to the chapter that explains it.

That is what these six reference documents give you. They are deliberately tables-first, comprehensive, and heavily cross-linked back into the rest of the book. Nothing here is new material; everything points at a chapter that teaches the concept in full, or at a file in the Tez source tree you can open right now. Think of the appendix as the index card layer that sits on top of the curriculum.

Note: Everything in these files was extracted from a current Apache Tez master checkout — real fully-qualified class names, real configuration key strings, real enum values. Nothing is invented. But Tez moves: classes get refactored, defaults change across releases, enums gain values. Every table tells you the grep that confirms the current truth in your checkout. When the appendix and your grep disagree, your grep wins — and that is a documentation fix worth a patch.

The six reference documents

Note: Document 5 is the only one that is not a pure lookup table — it is a comparative essay, because "why is Tez shaped like this?" is a question best answered by contrast. Read it once early; the other four you will return to constantly.

What each document is not

Reference material is as useful for what it excludes as what it includes, so a few boundaries are worth stating up front:

• The glossary defines terms; it does not teach the subsystem. Every entry is one to four sentences and a link. If an entry leaves you wanting the full mechanism, that is by design — follow the link.
• The key-classes map is organized by module, not by request path. If you are tracing a flow end-to-end (client submit → AM → task), the deep dives follow the flow; the map answers the orthogonal question "given this class name, where am I?"
• The config reference is a curated ~70 keys, not all of them. Tez has hundreds of config fields; this is the subset a maintainer actually reaches for. The complete, generated list lives in tez-default-template.xml (see below).
• The state & event map lists the enums and the routing, not the full transition tables. The transition tables live in the *Impl classes themselves (DAGImpl, VertexImpl, TaskImpl, TaskAttemptImpl) and are too large and too version-specific to mirror faithfully; the map gives you the vocabulary to read them.

How the appendix relates to the rest of the book

The curriculum has several layers, and the appendix indexes all of them:

flowchart LR
    A["Appendix<br/>(look up)"] --> DD["deep-dives/<br/>(subsystem reference)"]
    A --> OV["overview/<br/>(the mental model)"]
    A --> LV["level-1..9/<br/>(graded curriculum)"]
    A --> HV["hive-on-tez-labs/<br/>(the marquee consumer)"]
    A --> CM["contributor-mindset/<br/>(how to ship a patch)"]

• The deep dives (../deep-dives/index.md) are the subsystem reference: chapters mirroring the DAG lifecycle and the runtime data path — dag-model.md, logical-physical.md, state-machines.md, scheduler.md, shuffle-sort.md, and more. The glossary and key-classes map point here most often.
• The overview (../overview/index.md) is the one-sitting mental model of what Tez is and why it exists.
• The levels (../level-1/index.md … ../level-9/index.md) are the graded, hands-on curriculum that consumes all of the above, each ending in labs/.
• The Hive-on-Tez labs (../hive-on-tez-labs/index.md) trace Tez through its single most important production consumer.
• The contributor mindset chapters (../contributor-mindset/index.md) cover the non-code half of contributing: reading the codebase, design-via-JIRA, patch quality, and responding to review.

How to use these while contributing

The appendix earns its keep in the middle of real work. A few concrete patterns:

You hit an unfamiliar term in a JIRA or PR. Start in the Glossary. Get the one-sentence version, then follow the chapter link only if you need depth. Most terms resolve in ten seconds.

You have a stack trace and need to orient. Take the most specific class name in the trace and look it up in Key Classes by Module. That tells you which module owns the failure and which chapter explains it — so you read the right 600 lines, not all of them. A frame in org.apache.tez.runtime.library.common.shuffle.orderedgrouped is shuffle; a frame in org.apache.tez.dag.app.dag.impl is the AM state machines.

You are tuning or reproducing a bug. Open Configuration Keys Reference. Find the setting, note its @ConfigurationScope (AM, DAG, VERTEX, or CLIENT — which controls whether you can override it per-vertex), and read the chapter the row points at before you change anything in a patch.

You are reading a state transition. Open State Machine & Event Map. The state enums are tiny and stable; the event-type enums and the routing table tell you which *Impl class is the producer and which dispatcher carries the event.

You are arguing about design. Open Tez vs Spark vs MapReduce for the honest trade-offs — no marketing, just execution-model, scheduling, shuffle, and memory specifics.

A worked example

Say a JIRA reports: "reducers hang on a large Hive query; the AM log shows attempts stuck in START_WAIT." Here is how the appendix carries you through it without reading a single full deep dive first:

1. START_WAIT is unfamiliar. The State Machine & Event Map tells you it is a TaskAttemptStateInternal value meaning "scheduled with the task scheduler, waiting for a container." So the attempt is created but has no container yet — this is a scheduling/allocation problem, not a task-execution one.
2. Which class owns that? The Key Classes by Module points you at TaskAttemptImpl (the attempt state machine) and, since containers are the issue, DagAwareYarnTaskScheduler and AMContainerImpl in tez-dag.
3. The routing table in the event map shows the chain TA_SCHEDULE → S_TA_LAUNCH_REQUEST → TaskSchedulerManager. If the attempt is stuck at START_WAIT, either the launch request never left or YARN never returned a container — so you look at the scheduler and RM heartbeat.
4. The Configuration Keys Reference surfaces the suspects: tez.am.container.reuse.locality.delay-allocation-millis, tez.am.am-rm.heartbeat.interval-ms.max, and whether the queue (tez.queue.name) is starved.
5. The Glossary reminds you what delay scheduling and container reuse mean if the locality-delay path looks relevant.

Only now — with the problem localized to one class and three config keys — do you open ../deep-dives/scheduler.md and read the 600 lines that actually matter. That is the appendix working as intended: it converts a vague symptom into a precise place to look.

Reading a package prefix at a glance

Because the appendix is cross-linked to real fully-qualified class names, it helps to internalize what a package prefix tells you about which layer you are in. This table is the fastest orientation you can carry in your head; the Key Classes by Module doc expands each row:

When a stack-trace frame or a JIRA snippet names a class, match its prefix here first: it tells you whether you are looking at the client, the AM control plane, or the task-side data plane before you read a single line of the class. Those three layers rarely share a bug, so identifying the layer usually halves the search.

The verification reflex

Two commands settle almost every "is this still true?" question. From a Tez source checkout:

# Confirm a class still exists and see its package / role:
grep -rn "class TaskAttemptImpl" tez-dag/src/main/java/

# Confirm a config key, its constant, and its default (all in the *Configuration class):
grep -rn "TEZ_AM_CONTAINER_REUSE_ENABLED" \
  tez-api/src/main/java/org/apache/tez/dag/api/TezConfiguration.java

And from the generated config template, which is the project's own source of truth for every public key and default:

# Every @ConfigurationProperty key with its default lands here at build time:
find . -name tez-default-template.xml

Keep this appendix open in one pane and your checkout in another. The reference is fast; the grep is authoritative. Use both.