Apache Tez Open-Source Contributor Curriculum

Welcome to the Apache Tez Open-Source Contributor Curriculum — a complete, implementation-heavy roadmap for engineers who want to become serious Apache Tez contributors and eventually operate at the level of a core contributor, committer, or PMC-aware engineer.

What This Curriculum Is

This is not a tutorial. It is a structured engineering apprenticeship built around how Apache Tez is actually developed, tested, reviewed, and maintained by its committers and PMC members.

Every level is tied to real Apache Tez source code, real JIRA issue patterns, real test infrastructure, and real contribution workflows. The labs mirror the work an Apache Tez committer actually does — reading state machine code, tracing DAG execution paths, debugging shuffle failures, reproducing reported issues, and preparing patches for community review.

The curriculum will not hold your hand. It will point you at the right parts of the codebase, give you the right questions to ask, and push you to develop the muscle memory of someone who works at this level habitually.

What Apache Tez Is

Apache Tez is a general-purpose DAG execution engine that runs on Apache YARN. It is the layer between a query planner and the cluster: a higher-level system (overwhelmingly Apache Hive, historically also Pig and Cascading) compiles a job into a directed acyclic graph of computation, and Tez runs that graph — requesting containers from YARN, scheduling tasks onto them, moving data between stages, and driving the whole thing to completion inside a single ApplicationMaster.

Why Tez exists

MapReduce forces every computation into a rigid Map → Shuffle → Reduce shape. A complex analytical query — a multi-join SQL statement, say — becomes a chain of MapReduce jobs, and between every job the intermediate results are written to HDFS and read back. That materialization is pure overhead, and the fixed two-stage shape wastes work.

Tez removes both costs. It lets a planner express an arbitrary DAG where:

• Vertices are computation stages (a Processor with parallelism).
• Edges describe data movement between stages (one-to-one, broadcast, scatter-gather shuffle).
• Containers are reused across tasks, eliminating repeated JVM startup.
• Data is pipelined between stages instead of round-tripping through HDFS.
• A single AM orchestrates the entire job, so scheduling and reuse decisions see the whole graph.

That is why Tez is dramatically faster than chained MapReduce for multi-stage queries, and why it became the default execution engine under Hive.

Where Tez sits in the stack

        Hive / Pig  (query planner)          <- compiles SQL/scripts to a DAG
              |
              v
   +---------------------------+
   |        Apache Tez         |             <- your work lives here
   |  TezClient -> DAGAppMaster |
   |  scheduling, state machines,           (org.apache.tez.*)
   |  shuffle, container reuse |
   +---------------------------+
              |
              v
        Apache YARN  (containers, RM/NM)     <- cluster resource management
              |
              v
        HDFS / object store  (data)

You will spend your time in the middle box: the AM (DAGAppMaster) that turns a submitted DAG into running work, the state machines (DAGImpl, VertexImpl, TaskImpl, TaskAttemptImpl) that model its lifecycle, the scheduler (DagAwareYarnTaskScheduler) that negotiates with YARN, and the runtime (tez-runtime-library) that implements the shuffle and sort.

Who This Is For

This curriculum is designed for strong backend and distributed systems engineers who:

• Have 3+ years of Java development experience (Maven-based projects)
• Are familiar with Hadoop, YARN, or MapReduce at a conceptual level
• Understand distributed systems fundamentals: scheduling, fault tolerance, partitioning, shuffle
• Want to contribute to Apache open-source at a serious level — not just fix typos

You should be comfortable with:

• Reading large, unfamiliar Java codebases without a guide
• git workflows, reading diffs, working with patch-based reviews
• The Hadoop ecosystem at a high level: YARN, HDFS, MapReduce, Hive
• Distributed execution concepts: task graphs, data movement, speculative execution

Why Contribute to Tez

Contributing to Tez is a specific kind of career investment, and it is worth being clear-eyed about what you get:

• You learn a production distributed execution engine from the inside. Tez is a real, battle-tested system that runs enormous Hive workloads. Understanding how its AM schedules, reuses containers, and recovers from failure is transferable to any large-scale data or scheduling system you will ever build.
• The surface area is legible. Unlike a sprawling, fast-churning project, Tez is bounded and stable enough that a determined engineer can build a genuine mental model of the whole engine — client, AM, runtime — in a few months. That is rare and valuable.
• Your contributions matter to real users. Because Tez sits under Hive, a scheduling or shuffle fix you land is felt by every Hive-on-Tez cluster that upgrades into it.
• It is a credible path to Apache committership. The bar is high and the timeline is long, but the meritocracy is real: sustained, high-quality contribution is recognized.

Project Health — an Honest Assessment

You should choose where to invest your effort with clear eyes, so here is the honest picture. Apache Tez is a mature project in a maintenance phase. That is not a criticism; it is the defining fact about contributing here, and it shapes everything.

• Development cadence is steady, not explosive. The master branch is versioned 1.0.0-SNAPSHOT, but the line users actually run is 0.10.x — 0.10.3 shipped 2024‑01, 0.10.4 2024‑08, 0.10.5 2025‑05. Commit volume runs on the order of ~50 commits per year, concentrated among a small, stable set of committers.
• The recent work is exactly what a mature engine needs: build/CI modernization (Yetus, JDK 21 support, dependency and license hygiene), targeted performance fixes, scheduler correctness, and flaky-test stabilization — not sweeping new features.
• Verify all of this yourself. The house rule of this curriculum is don't trust a doc over the repository. Run these against a fresh checkout:

cd ~/src/oss-repos/tez
git log --oneline --since="2024-01-01" | wc -l   # ~50/yr — a maintenance cadence
git shortlog -sn --since="2024-01-01" | head      # a handful of active committers
git tag | grep "release-0.10"                      # the released 0.10.x line
git log -1 --format=%cd                            # how recent is HEAD?

Why maintenance phase is good for you as a learner. A stable codebase is a better teacher than a churning one: the abstractions have settled, the design decisions are recorded in JIRA, and the code you read today will still be the code tomorrow. And the bar being high — every merge into an engine under Hive is scrutinized — means the review you get here will make you a genuinely better engineer. This curriculum leans into that: it teaches you to write the tests and prove the performance and compatibility properties that a maintenance-phase project actually merges.

What You Will Be Able to Do

After completing this curriculum, you will be able to:

How to Use This Curriculum

Work through the 9 levels sequentially. Do not skip levels. Each level builds directly on the previous one, and the labs depend on the conceptual foundations laid earlier.

Beyond the 9 levels, the curriculum includes five additional sections:

The curriculum closes with a Capstone Project — a full contribution cycle from issue reproduction to merged patch and engineering write-up.

Start with the Overview & Prerequisites to get your environment built and verified, then work the levels in order.

Required Tools

Before starting Level 1, ensure you have the following installed and working:

Java 21 for master (Java 8/11 for the 0.10.x release line — match the branch target)
Apache Maven 3.6.3 or newer
Git 2.x
IntelliJ IDEA (strongly recommended) or Eclipse with M2E
Hadoop matching the branch's hadoop.version (currently 3.4.2) for running DAGs

You will also need:

• A clone of the Apache Tez repository (GitHub mirror of the Apache GitBox repo)
• A clone of the Apache Hadoop repository (for YARN API context and integration reference)
• An account on Apache JIRA (free to create)
• Subscription to the Apache Tez mailing lists:
  • dev@tez.apache.org — development discussion (required)
  • issues@tez.apache.org — JIRA notifications (optional but useful)

Note on Java version: the target moves between branches. master (1.0.0-SNAPSHOT) sets maven.compiler.release=21; the 0.10.x releases build on Java 8. Always check pom.xml at the root of the branch you are working on. The full setup is in Overview & Prerequisites.

Apache Tez at a Glance

Key Modules

You will spend the majority of your time in these modules:

Key Classes (High-Level Preview)

If half of these are unfamiliar, that is expected. By Level 4 you will read them without a guide.

Apache Tez Community

Apache Tez is a mature project with an active but selective community. The codebase reflects years of careful design decisions, many of which are documented in JIRA issues, design documents, and mailing list threads rather than in code comments.

What the community values:

• Patches that include tests
• Issues that include a clear reproduction case
• Comments that demonstrate you have read the existing code
• Contributors who engage respectfully and patiently
• Sustained contribution over time, not one-off patches

The path from contributor to committer is measured in years, not weeks. That is intentional. The Apache meritocracy rewards sustained, high-quality contribution — not volume of patches. In a maintenance-phase engine underneath Hive, judgment matters more than throughput: the ability to write a test that pins a scheduler behavior, to prove a performance change with numbers, and to spot a compatibility break in a one-line diff is exactly what earns trust here.

This curriculum will help you build the habits and depth of understanding that make that path realistic.

Begin with the Overview & Prerequisites, then Level 1: Hadoop and Tez Foundation.