Level 9: Advanced Committer / PMC-Level Contributor

This is the last level before the capstone, and it is the one that changes how you read every line of a diff. Up to now you have been a contributor: you found a JIRA, reproduced it, fixed it, wrote a test, and attached a patch. A committer carries a different burden. A committer is the person who says no — who blocks a green, well-reviewed, genuinely useful patch because it allocates a TezCounters object on a path that runs once per task per heartbeat, or because it changes the wire order of a protobuf field that a running Hive cluster deserializes during a rolling upgrade. The contributor optimizes for "does my change work." The committer optimizes for "does Tez still work — under Hive, at 50,000 tasks, on the branch users actually run in production — after my change merges and the next release ships into it."

This curriculum will not hold your hand here. By Level 9 you can build Tez, trace a DAG from TezClient.submitDAG through DAGAppMaster to a running TaskAttemptImpl, read the state machines, debug a shuffle failure, and argue a design on a JIRA. What you build now is judgment — the two reflexes that every Tez committer applies to every patch they review, including their own, plus the governance literacy that turns a trusted contributor into a committer and then a PMC member.

Learning Objectives

By the end of Level 9 you must be able to:

1. Write scheduler behavior tests the way the project does — driving a real DagAwareYarnTaskScheduler or YarnTaskSchedulerService against a mocked AMRMClient, using the drainable-callback harness in TestTaskSchedulerHelpers, and asserting on the exact YARN RM calls (addContainerRequest, releaseAssignedContainer, updateBlacklist) a behavior should produce.
2. Treat performance as a correctness property on hot paths: profile before you change, write a JMH microbenchmark against a real class, confirm with a benchmark DAG at scale, and bisect a regression with git bisect run — never claim a win without before/after numbers.
3. Reason about backward compatibility the way an embedded engine must: Tez is the runtime under Hive and Pig, so a change to a public tez-api type, a protobuf record, a serialized Writable, or a TezConfiguration default is a compatibility event, not a cleanup.
4. Review another contributor's patch the way a committer does — checking test coverage, allocation on hot paths, compatibility, and whether the fix is the minimum-impact change or a clever refactor that smuggles in risk.
5. Understand how Apache governance actually works: the committer path, how release votes are run and counted, and what a PMC is accountable for.

Where Tez Sits, and Why the Bar Is Higher Here

Tez is a mature, maintenance-phase Apache project. The master branch is versioned 1.0.0-SNAPSHOT, but the releases people run are the 0.10.x line (0.10.3 shipped 2024‑01, 0.10.4 2024‑08, 0.10.5 2025‑05). Activity is steady rather than explosive — on the order of ~50 commits a year, concentrated among a handful of committers (Laszlo Bodor, Raghav Aggarwal, Ayush Saxena, and a few others carry most of it). Verify this yourself; it is the single most important fact about contributing here:

cd ~/src/oss-repos/tez
git log --oneline --since="2024-01-01" | wc -l      # ~50/yr — a maintenance cadence
git shortlog -sn --since="2024-01-01" | head        # a small, stable committer set
git tag | grep "release-0.10"                        # the line users actually run

That maturity raises the bar, it does not lower it. In a fast-moving project a mediocre patch gets swept along by momentum. In a maintenance-phase engine that sits underneath Hive, every merge is scrutinized because there is no churn to hide behind and the downstream blast radius is enormous. The two things committers guard hardest are exactly the two skills this level drills: performance (Tez's whole reason to exist is being faster than MapReduce — a regression is an existential bug) and backward compatibility (breaking Hive-on-Tez is the worst thing you can ship).

The Two Constant Committer Concerns

Strip away everything else and a Tez committer's review reduces to defending two invariants on every change. Internalize these; they are the lens for the whole level.

Concern 1 — Performance (it stays fast while it changes)

Tez exists because MapReduce forced every computation through HDFS between stages. The container-reuse, pipelined-shuffle, single-AM design is a performance argument. A path that runs once per record, once per task, once per heartbeat, or inside the AM dispatcher is hot: a small allocation or an accidental O(n²) there compounds across every task in every DAG on every cluster. Committers do not accept "it should be faster" — they accept "here is the JMH microbenchmark, here is the benchmark-DAG wall-clock, here is the git bisect that fingered the commit." The discipline is drilled in Lab 9.2 and framed in Stage 10: Performance Improvements.

A real example you will dissect: TEZ‑4250 (Optimise TaskImpl::getCounters) changed TaskImpl.getCounters() to stop allocating and populating a fresh TezCounters on every call, doing the aggregation only when speculation is actually enabled. Read the commit before you believe the description:

git show 9aeb17b4b --stat        # TEZ-4250, tez-dag TaskImpl/TaskAttempt

Concern 2 — Backward compatibility (it stays working while it changes)

Tez is not an application; it is a library other projects link and a protocol other processes speak. A change is a compatibility event if it touches: a public @InterfaceAudience.Public type in tez-api; a protobuf message (the AM↔task and client↔AM RPC records); a Writable's readFields/write order; recovery/history data on disk; or a TezConfiguration default that silently changes behavior. Get it wrong and you do not get a red unit test — you get a Hive query that fails to recover, or a rolling upgrade where a new AM cannot read an old task's status. The policy framing lives in Stage 11: Backward Compatibility and the compatibility mindset chapter; read both before this level's labs.

These two concerns are why committer review feels slow. It is not gatekeeping for its own sake; it is the cost of invariants that cannot be un-shipped once a release goes out and Hive upgrades into it.

Required Reading

Before the labs, read (or re-read) these, in order:

1. The Scheduler deep dive — the two-layer TaskSchedulerManager / TaskScheduler design, DagAwareYarnTaskScheduler vs YarnTaskSchedulerService, and how AMSchedulerEvents flow. Lab 9.1 assumes it.
2. Stage 10: Performance Improvements and Stage 11: Backward Compatibility — the issue-finding workflow and the policy for each concern.
3. The committer mindset, release voting, and PMC responsibilities chapters — the governance you are being measured against, not just the code.
4. The patch-quality and meritocracy chapters — why the path to committer is measured in years and how judgment is demonstrated.

How These Map to the Codebase

Two commands to orient yourself in the repo right now:

# The scheduler package — production and test side by side.
ls tez-dag/src/main/java/org/apache/tez/dag/app/rm/
ls tez-dag/src/test/java/org/apache/tez/dag/app/rm/

# Confirm the default scheduler the AM instantiates.
rg -n "TEZ_AM_YARN_SCHEDULER_CLASS_DEFAULT" \
  tez-api/src/main/java/org/apache/tez/dag/api/TezConfiguration.java
# -> "org.apache.tez.dag.app.rm.DagAwareYarnTaskScheduler"

Key Practices at the Committer Tier

Reviewing Others' Patches

By Level 9 you should be reviewing patches on JIRA and GitHub PRs, not only opening them. A committer review is a checklist applied in a fixed order, fastest-to-fail first:

1. Does it build and does Yetus pass? If the pre-commit build is red, stop — comment and move on.
2. Are the tests real? A scheduler fix with no test that drives the exact event sequence, or a bug fix whose test passes on the unpatched code, is not done.
3. Is it on a hot path? If the diff is inside TaskImpl, VertexImpl, the AM dispatcher, DefaultSorter, IFile, or a per-record I/O path, ask for numbers.
4. Does it touch the compatibility surface? Any change to tez-api, a .proto, a Writable, recovery data, or a config default is a compatibility review.
5. Is it the minimum-impact fix? A three-line targeted change beats a fifty-line refactor that also "cleans things up." Ask the contributor to split them.

Disagree without being a wall. The responding-to-feedback and community-interaction chapters cover the human side — how to say "this reorders a protobuf field and will break rolling upgrade" without making the contributor feel attacked.

Deliverables

You must demonstrate all of the following before attempting the capstone:

• Completed Lab 9.1: wrote a new scheduler behavior test on top of the real TestTaskSchedulerHelpers harness, driving a real scheduler against a mocked AMRMClient, and it passes under mvn test -pl tez-dag.
• Completed Lab 9.2: dissected a real perf commit, built a benchmark DAG, bisected a regression with git bisect run, and wrote one correct JMH microbenchmark against a real Tez class with before/after numbers (no fabricated results).
• A written review (JIRA-comment style) of one real Tez change that touches a hot path or the compatibility surface, applying the fixed-order checklist above.
• From memory: explain the difference between the two schedulers (DagAwareYarnTaskScheduler vs YarnTaskSchedulerService), and name three scheduler behaviors that deserve dedicated tests.
• A one-paragraph statement of which subsystem you would want to own as a committer and why, naming the JIRA component and one perf or compatibility invariant it carries.

Common Mistakes

How to Verify

You have completed Level 9 when every one of these passes:

cd ~/src/oss-repos/tez

# Your new scheduler test compiles and runs green.
mvn test -pl tez-dag -Dtest=TestDagAwareYarnTaskScheduler 2>&1 | tail -15

# You can point at the real harness pieces without a guide.
rg -n "class MockAMRMClient|setupMockTaskSchedulerContext|class TaskSchedulerContextDrainable" \
  tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestTaskSchedulerHelpers.java

# You can read the real perf case study from the diff.
git show 9aeb17b4b -- tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/TaskImpl.java

# Your benchmark DAG builds.
mvn package -DskipTests -pl tez-examples -am -q

If your scheduler test drives real events against a mocked RM and asserts on the RM calls; if you can explain TEZ‑4250 as an allocation-on-a-hot-path fix straight from the diff; and if you can bisect a regression to a single commit — you are ready.

PR Profile: Level 9 Graduate

You are now ready for the capstone — an end-to-end contribution where you select a real JIRA, reproduce it, find the root cause, implement and test the fix (with performance and compatibility discipline), prepare the patch, and write it up. Start at the Capstone Overview. The two reflexes you built here — did I prove the performance claim? and will this break Hive on upgrade? — are the ones the capstone evaluates hardest.

Note: Most engineers never reach this tier in an open-source project, not because they can't, but because they stop at "my patch is green." The distance from contributor to committer is the distance from "it works" to "it keeps working for every Hive cluster that upgrades into it." That is the entire content of Level 9.