Design via JIRA, Not PRs

Apache projects design in the open, but "the open" is not GitHub. In Tez it is the TEZ JIRA project and the dev@tez.apache.org mailing list. GitHub Issues is switched off for the repo entirely (features: issues: false in .asf.yaml); a pull request is a delivery mechanism for a patch, not where the design lives. The rationale, the alternatives considered, the "why not X" — those are in JIRA.

This chapter is about recovering that design intent from the JIRA trail, and about writing your own before a non-trivial patch. It is the highest-leverage reading skill on a maintenance-phase project, because the decisions that shaped the code you are reading were mostly made years ago by people who no longer commit — and every one of them is archived under a TEZ-NNNN.

Why the Design Isn't on the PR

If a discussion happens only on a PR and the PR is later force-closed or squashed away (Tez squash-merges — enabled_merge_buttons: squash: true), the rationale evaporates. JIRA and the mailing-list archive do not move. So the durable link from code to reason is the TEZ-NNNN in the commit message, not the (#NNN) PR number beside it.

The TEZ-NNNN Convention Is Your Index

Every substantive commit names its JIRA. Look at any slice of history:

cd ~/tez-src
git log --oneline | head -20

You will see the house format: TEZ-NNNN: <title> (#NNN) (<author> reviewed by <reviewer>). The TEZ-NNNN is the permanent key; the (#NNN) is the disposable PR. The code itself is peppered with the same keys as back-references:

grep -rn "TEZ-[0-9]" tez-dag/src/main/java | head

Some of those are TODO markers pointing at unfinished sub-work — for example the serviceplugins/api package still carries lines like // TODO TEZ-2003 (post) TEZ-2665. Move to the tez-api module. Each such comment is a live hyperlink from the code to a design conversation. When you leave your own non-obvious workaround, do the same: // TEZ-NNNN: <one line why> so the next reader finds your reasoning.

The Excavation Protocol: From a Class to Its "Why"

Given any surprising line of code, recover the decision behind it in four moves:

1. Blame the line to get the commit SHA:

   git blame -L <start>,<end> path/to/File.java
   

   If the last touch was a reformat (Spotless, a license-header normalization like TEZ-4711, an import cleanup like TEZ-4642), peel back to the real change with git log -L <start>,<end>:path/to/File.java.
2. Read the commit message for the TEZ-NNNN:

   git show <sha> | head -20
   

3. Open the JIRA at https://issues.apache.org/jira/browse/TEZ-NNNN. The Activity tab is the design conversation; the Attachments/Links tab has the design doc or the linked PR.
4. Follow the links on the JIRA — "is related to," "is a sub-task of," "blocks." Large features have an umbrella JIRA at the top of that link tree, and that umbrella is the project's own design document for the whole effort.

Worked Example: Reconstructing the DAG-Aware Task Scheduler

Let's do this for real, on a feature large enough to have a genuine design history: DagAwareYarnTaskScheduler, the component that decides which YARN containers to request and which task to run on each. It is the default task scheduler today, but it was not always, and the git trail tells a six-year story you can reconstruct entirely from commands.

Find the originating commit

cd ~/tez-src
git log --oneline -- \
  tez-dag/src/main/java/org/apache/tez/dag/app/rm/DagAwareYarnTaskScheduler.java | tail -1
git show a9b8bb5a6 --stat | head -20

The oldest commit is TEZ-3770 — "DAG-aware YARN task scheduler (jlowe)", by Jason Lowe, dated 2018-01-25. Its --stat is the first clue to the design's ambition:

• DagAwareYarnTaskScheduler.java — 2,064 lines, brand new
• TestDagAwareYarnTaskScheduler.java — 1,510 lines of tests
• ControlledScheduledExecutorService.java — 239 new lines
• touches TezConfiguration, DAGImpl, TaskSchedulerManager, TaskSchedulerContext

3,954 insertions in one commit. Two things jump out. First, the author shipped a pluggable scheduler — it slots in behind TaskSchedulerManager via the TEZ_AM_YARN_SCHEDULER_CLASS config, so it could coexist with the old one rather than replace it. Second, they built a ControlledScheduledExecutorService specifically so the tests could drive time deterministically — a strong signal that the scheduler's correctness is timing-dependent (container idle timeouts, request delays). That test harness is itself a design decision, and it tells you where the risk lives before you read a line of the scheduler.

Reconstruct the rationale from the follow-up trail

You do not need the original JIRA prose to recover the design pressures — the bug-fix commits that followed are the negative space around the design. List them:

git log --oneline -- \
  tez-dag/src/main/java/org/apache/tez/dag/app/rm/DagAwareYarnTaskScheduler.java

The trail, read newest-to-oldest, reconstructs the intent:

From four commit titles you have reconstructed the scheduler's design goals — reuse containers, respect locality, reason about the vertex dependency graph, and time out idle holds — without opening a browser. That is the excavation skill: the JIRA trail is the design doc, distributed across the commits that reference it.

Find the promotion decision

The most interesting design moment is not the introduction but the promotion. Ask when this scheduler became the default:

git log -S "DagAwareYarnTaskScheduler" --format='%ci %s' \
  -- tez-api/src/main/java/org/apache/tez/dag/api/TezConfiguration.java | tail -1

That returns TEZ-4553 — "Default task scheduler to DagAwareTaskScheduler to avoid hang in TEZ-3535", dated 2024-05-01. The full arc is now legible: introduced in 2018 as an opt-in expert setting, hardened by six years of bug fixes, and finally promoted to the default in 2024 specifically because the older scheduler could hang (the reason is right there in the commit title — TEZ-3535). If you were about to touch scheduling, this arc tells you which scheduler is live, why, and where its historical sharp edges are. Open TEZ-4553 and TEZ-3535 in JIRA and you have the human discussion behind the mechanical trail.

The Umbrella-JIRA Pattern

Big features don't live in one JIRA; they live under an umbrella with sub-tasks, often developed on a feature branch. The pluggable-services rework is the canonical Tez example. Look for its fingerprints:

git log --oneline master | grep "TEZ-2003"
git branch -r | grep TEZ-2003

You will find commits like TEZ-2139. Update version to 0.7.0-TEZ-2003-SNAPSHOT and TEZ-2626. ... consolidate TEZ-2003 TODOs, plus a long-lived origin/TEZ-2003 branch. The snapshot-version rename is the tell: the team versioned the whole branch after the umbrella JIRA while it baked, then merged it and cleaned up with follow-ups (TEZ-2708 renamed classes "post TEZ-2003 changes"). When you meet a cluster of commits sharing one JIRA number and a matching branch name, you have found an umbrella — read the umbrella JIRA first, top to bottom, before proposing anything in that area, because it is the project's own roadmap for that subsystem.

The TEZ JIRA Workflow (and Where PRs Fit Now)

A Tez JIRA moves through these statuses:

Open → In Progress → Patch Available → Resolved → Closed
                                    ↘ Reopened

You set only "Patch Available" yourself; everything below it requires a committer. How the patch is delivered has changed. In the classic era (which you will see all over old JIRAs) the patch was an attached file — TEZ-NNNN.001.patch, TEZ-NNNN.002.patch — re-rolled each review round and run through Apache Yetus. Since roughly late 2019 the delivery mechanism is a GitHub PR linked from the JIRA, squash-merged, with the reviewer named in the final commit (reviewed by Laszlo Bodor). The JIRA is still the system of record; the PR is just the modern attachment. Community Interaction covers the mechanics of both.

When to Open a JIRA Yourself

Open a JIRA before writing the patch when any of these is true:

You do not need a JIRA to ask a question on dev@ or user@, or to patch a private fork.

Writing a JIRA Description a Committer Can Act On

The skeleton, in order — a trivial fix may collapse the last two sections to one line each; a new API must expand them:

## Problem
(Two to four sentences. What is wrong. Who hits it.)

## Reproduction
(Steps, or a code sample. If a test reproduces it, name the test class.)

## Root Cause
(One paragraph. Cite file and method.)

## Proposed Fix
(One paragraph. What you intend to do. Alternatives considered.)

## Compatibility
(Wire? API? Config? "None." is a valid answer.)

## Test Plan
(Which tests pass after the change. Any new test added.)

Writing a Mini Design Note Before a Non-Trivial Patch

For anything larger than a single-file fix, attach a design note (Markdown or PDF) to the JIRA and announce it on dev@ with subject [DISCUSS] TEZ-NNNN: <short title>. The DagAwareYarnTaskScheduler history above shows why: a scheduler change that skips this step is how you get a TEZ-4027-style hang into a release. The skeleton:

# TEZ-NNNN: <short title>

## 1. Problem        — what is wrong today; why "do nothing" is unacceptable
## 2. Goals          — bulleted, testable ("survives a 10 MB DAGPlan without OOM")
## 3. Non-Goals      — what this explicitly will not address (stops scope creep)
## 4. Alternatives   — Option A / B / C; pros, cons, why rejected
## 5. Chosen Approach — architecture sketch (ASCII/Mermaid); cite files that change
## 6. Compatibility  — wire (.proto?), API (@Public touched?), config (new/renamed keys?)
## 7. Test Plan      — unit classes; MiniTezCluster scenarios; any manual verification
## 8. Rollout        — default off/on? opt-in like TEZ-3770 was? migration steps?

Note the rollout section, and re-read the scheduler story: Tez's own norm for a risky subsystem change is to ship it opt-in first and promote it to default only after it has proven itself in the field. TEZ-3770 (2018, opt-in) → TEZ-4553 (2024, default) is that norm in action across six years. Propose the same staging for anything comparably risky, and expect 1–2 weeks of asynchronous discussion on dev@ before consensus — on a low-traffic project, patience is not optional. Do not start patching until the design is at least loosely agreed; patches without design buy-in stall.

Validation Artifacts

After this chapter you should be able to produce:

1. The full excavation of one feature — the originating TEZ-NNNN, two follow-up bug JIRAs, and any promotion/default-change JIRA — reconstructed from git log alone, as done above.
2. The URLs of three TEZ-NNNN JIRAs cited from the source (grep -rn "TEZ-[0-9]"), with a one-line summary of each.
3. A draft JIRA description in ~/tez-notes/draft-jira.md for a real issue you have noticed.
4. One archived [DISCUSS] thread URL from lists.apache.org relevant to your area.

The next chapter — Community Interaction — covers how to actually post on dev@ and behave on JIRA without burning trust on day one.