Deep Dives: Reading Order

This directory is the reference core of the curriculum: 21 deep-dive chapters that together explain how Apache Tez turns a DAG object into a distributed computation running across a YARN cluster. Each chapter is self-contained and verified against a real Tez checkout, but they form a dependency graph — read in the order below the first time through, then use this page as a lookup index when you return to fix a bug.

Every chapter follows the same shape: a "what you can do after this" opening, role-based sections that each start with a runnable grep/find, real code excerpts attributed by module + class (never by line number, because code moves between branches), at least one diagram, a reading exercise, a "common bugs and symptoms" table, and a Validation: prove you understand this gate. Treat that gate as the definition of "read."

The system in one picture

A Tez job has four tiers: the client builds a plan and launches an AM; the YARN layer schedules containers; the AM (DAGAppMaster) drives the DAG's state machines and hands work to containers; the tasks run your input/processor/output code and shuffle data between stages.

 ┌───────────────────────────────────────────────────────────────────────┐
 │ CLIENT JVM (tez-api)                                                    │
 │   DAG.create() → verify() → DAGPlan(proto)                             │
 │   TezClient.start() ─────────────► YARN RM: submitApplication          │
 │   TezClient.submitDAG() ──(RPC or staged tez-dag.pb)──► AM             │
 │   DAGClient.getDAGStatus()/waitForCompletion()  ◄── status             │
 └───────────────────────────────────────────────────────────────────────┘
              │ launches                                 ▲ RPC status/kill
              ▼                                          │
 ┌───────────────────────────────────────────────────────────────────────┐
 │ DAGAppMaster JVM (tez-dag)                                              │
 │   DAGClientServer ── AsyncDispatcher (single thread, the only mutation)│
 │        │            ├─ DAGImpl ─ VertexImpl ─ TaskImpl ─ TaskAttemptImpl│
 │        │            ├─ TaskSchedulerManager ─► YARN RM (AMRM heartbeat) │
 │        │            ├─ ContainerLauncherManager ─► YARN NM (launch)     │
 │        │            ├─ TaskCommunicatorManager (umbilical RPC server)   │
 │        │            └─ HistoryEventHandler / RecoveryService (ATS, log) │
 └───────────────────────────────────────────────────────────────────────┘
              │ launches containers                      ▲ umbilical: getTask,
              ▼                                          │ heartbeat, done, events
 ┌───────────────────────────────────────────────────────────────────────┐
 │ TASK CONTAINER JVM (tez-runtime-internals + tez-runtime-library)       │
 │   TezChild → LogicalIOProcessorRuntimeTask                            │
 │   Inputs → Processor → Outputs;  Sorter/IFile → ShuffleManager/Fetcher │
 └───────────────────────────────────────────────────────────────────────┘

flowchart TB
    subgraph Client["Client JVM · tez-api"]
        DAG["DAG / Vertex / Edge"] --> TCL["TezClient"]
        DC["DAGClient (poll/kill)"]
    end
    subgraph YARN["YARN"]
        RM["ResourceManager"]
        NM["NodeManagers"]
    end
    subgraph AM["DAGAppMaster JVM · tez-dag"]
        DISP["AsyncDispatcher"]
        DI["DAGImpl→VertexImpl→TaskImpl→TaskAttemptImpl"]
        SCH["TaskSchedulerManager"]
        CL["ContainerLauncherManager"]
        TCM["TaskCommunicatorManager"]
    end
    subgraph Task["Task container JVM · runtime"]
        RT["LogicalIOProcessorRuntimeTask"]
        IPO["Inputs → Processor → Outputs"]
    end
    TCL -->|submitApplication| RM
    RM -->|launch AM| AM
    TCL -->|submitDAG RPC| DISP
    DISP --> DI
    SCH <-->|AMRM heartbeat| RM
    CL -->|launch container| NM
    NM --> Task
    Task <-->|umbilical| TCM
    DC -->|getDAGStatus / tryKillDAG| DISP

The through-line: events are the only way to mutate AM state, everything the client sees is a snapshot, and every plan is self-sufficient so the AM can run it (or recover it) without calling back to the client. Hold those three ideas and the 21 chapters stop feeling like trivia.

The module map — a maintainer's mental model

Before the chapters, internalize where code lives. These are the Maven modules you will cd into (verified against the checkout's root pom.xml):

Two more you will meet but this section does not cover in depth: tez-plugins (ATS/Timeline history backends, aux shuffle service) and tez-examples (OrderedWordCount and friends, the smallest real DAGs to trace).

Tip: The single most important boundary is tez-api vs tez-dag. The Edge you build (org.apache.tez.dag.api.Edge, tez-api) is a dumb plan record; the Edge that routes events (org.apache.tez.dag.app.dag.impl.Edge, tez-dag) owns the EdgeManager. Same name, different worlds. When a colleague says "look at VertexImpl / Edge / DAG," always ask which module.

Reading order and rationale

Group 1 — The DAG model and the client

These four define "what is a Tez job" and how it is submitted and observed, before any execution machinery. Start here; without the DAG model in your head every later chapter reads as noise.

Group 2 — AM lifecycle and dispatch

How the AM comes up and how it mutates state. These must precede every per-entity lifecycle chapter.

Group 3 — Per-entity lifecycle

Read 8 → 9 → 10 in order; each refers back to events (7) and state-machine primitives (6).

Group 4 — Input / Processor / Output

The code inside the task JVM (tez-runtime-internals + tez-runtime-library).

Group 5 — Shuffle, sort, and counters

Do not debug a shuffle problem without reading 13 cold first.

Group 6 — Scheduling and resources

Group 7 — Modes and integrations

Group 8 — Failure, recovery, and testing

Which chapter answers which question

What a maintainer should internalize from each chapter

• dag-model — the plan is immutable and self-sufficient; verify() is the contract; the two Edge classes.
• logical-physical — parallelism is often decided in the AM, not the client; VertexManagerPlugins reshape the graph legally.
• tez-client — three submission paths behind one method; session start() launches the AM, non-session defers it; the IPC-size fallback.
• dag-client — status is a snapshot with a source; the AM→cache→Timeline→RM waterfall; the server-side long-poll makes waitForCompletion cheap.
• dag-app-master — the single-threaded dispatcher is the only mutation path; recovery is per-DAG; the scheduler/launcher/communicator are plugins.
• state-machines / event-routing — every transition is an event; handlers must not block; the state machine is the source of truth.
• vertex / task / task-attempt lifecycle — the exact states and the legal transitions; where failure and kill diverge.
• ipo-abstractions / tez-runtime — the task's initialize→start→run→close contract; the umbilical is the task's only line home.
• shuffle-sort / counters — where bytes and time actually go; how to measure it.
• scheduler / container-reuse / yarn-integration — how containers are asked for, kept, and reclaimed; the YARN contracts underneath.
• local-mode / hive-integration — the two most common real-world entry points into Tez.
• failure-handling / testing-framework — how Tez tolerates faults, and how you prove your change does too.

Order vs index

The deep-dives are an index: they exist to be looked up. Your first pass should follow the groups above. When you return to fix a bug, jump straight to the most relevant chapter and follow its cross-links. Do not skip the Validation section at the end of any chapter — it is the gate before you may claim to have read it, and the difference between "I saw the code" and "I can change the code."