Level 9: Advanced Maintainer / TSC-Level Contributor

This is the last level before the capstone, and it is the one that changes how you read every line of code. Up to now you have been a contributor: you found an issue, fixed it, wrote a test, and shipped a PR. A maintainer carries a different burden. A maintainer is the person who says no — who blocks a green, well-tested, genuinely useful PR because it adds a field to a transport response without a version guard, or because it allocates a HashMap per document in the search loop. The contributor optimizes for "does my change work." The maintainer optimizes for "does the project still work, on every supported version, at every scale, after my change merges and ten thousand clusters upgrade into it."

This curriculum will not hold your hand here. By Level 9 you can read the engine, trace a request from RestController to Lucene, write integration tests, and argue a design on a GitHub issue. What you build now is judgment — the two reflexes that every OpenSearch maintainer applies to every PR they review, including their own.

Learning Objectives

By the end of Level 9 you must be able to:

1. Read any one-line diff to a Writeable, *Request, *Response, or ClusterState component and immediately see its wire backward-compatibility implications — and prove safety with a round-trip and a qa/ BWC test.
2. Reason about index/Lucene format BWC (data on disk) as a distinct concern from wire BWC (nodes on the network), and know the N-1-major support window.
3. Treat performance as a correctness property on hot paths: profile before you change, write a JMH microbenchmark in :benchmarks, and confirm the macro impact with OpenSearch Benchmark before claiming a win.
4. Review another contributor's PR the way a maintainer does — checking BWC, allocation, test coverage, CHANGELOG.md, and the deprecation/backport story, not just whether the code compiles and the diff looks clean.
5. Own a subsystem: understand its MAINTAINERS.md, its area labels, its backport policy, and what "I am responsible for this not breaking" means.
6. Connect a code change to the release train — which branch it lands on, which Version constant gates it, whether it needs a backport 2.x, and whether it is a release blocker.

The Two Constant Maintainer Concerns

Strip away everything else and a maintainer's job reduces to defending two invariants on every change. Internalize these; they are the lens for the whole level.

flowchart TB
    PR[Incoming PR / your own change] --> BWC{Does it change<br/>the wire or on-disk<br/>format, or a public<br/>contract?}
    PR --> PERF{Is it on a hot<br/>path — per-document,<br/>per-request, per-shard,<br/>coordination?}
    BWC -->|yes| G1[Version-gate it, add a<br/>round-trip + qa/ BWC test,<br/>plan the deprecation cycle]
    PERF -->|yes| G2[Profile, JMH micro before/after,<br/>OSB macro, check allocation/GC]
    BWC -->|no| OK1[Standard review]
    PERF -->|no| OK2[Standard review]

Concern 1 — Backward compatibility (it stays working while it changes)

OpenSearch runs in clusters that upgrade node by node. During a rolling upgrade a cluster runs two versions at once, and every transport message between an old node and a new node must serialize correctly in both directions. Get this wrong and you do not get a red unit test — you get a corrupted cluster during a customer's upgrade, which is the single worst class of bug this project can ship. The full mechanism — Version-gated StreamInput/StreamOutput, why reordering writes silently corrupts a stream, NamedWriteableRegistry, and the index/Lucene format window — lives in the Serialization and Backward Compatibility deep dive. Re-read it before this level's first lab; the lab assumes it.

Concern 2 — Performance (it stays fast while it changes)

OpenSearch indexes and searches at scale. A path that runs once per document, once per request, once per shard, or inside the coordination layer is hot: a 5% allocation regression there compounds into real cost and real latency across every cluster running your code. Maintainers do not accept "it should be faster" — they accept "here is the JMH microbenchmark, here is the before/after, here is the OpenSearch Benchmark macro result that proves it matters and that no other workload regressed." The discipline is taught in Stage 10: Performance Improvements and drilled in this level's second lab.

These two concerns are why maintainer review feels slow. It is not gatekeeping for its own sake; it is the cost of an invariant that cannot be un-shipped once a release goes out.

How These Map to the Codebase

Two commands to orient yourself in the repo right now:

# Where the BWC QA suites live
ls qa/
# expect: rolling-upgrade  mixed-cluster  full-cluster-restart  smoke-test-* ...

# Where the JMH microbenchmarks live
ls benchmarks/src/main/java/org/opensearch/benchmark/
./gradlew :benchmarks:jmhJar -q   # builds the runnable JMH uber-jar

Key Practices at the Maintainer Tier

Reviewing Others' PRs

By Level 9 you should be reviewing PRs, not only opening them. A maintainer review is a checklist applied in a fixed order, fastest-to-fail first:

1. Does it build and pass CI? If not, stop — comment and move on.
2. Does it touch the wire or on-disk format? Any change to a Writeable, *Request/*Response, ClusterState component, Metadata custom, or NamedWriteable name is a BWC review. Look for out.getVersion().onOrAfter(...) and a matching qa/ or round-trip test. No gate, no merge.
3. Is it on a hot path? If the diff is inside QueryPhase, an Aggregator, IndexShard.applyIndexOperationOnPrimary, the coordination layer, or any per-document/per-request loop, ask for numbers.
4. Are the tests real? A test that asserts "no exception" is not a test. Look for round-trip, randomized, and (for BWC) cross-version coverage.
5. CHANGELOG, backport label, deprecation story. Is there a CHANGELOG.md entry? Does it need backport 2.x? Does it remove or rename anything a client relies on?

Disagree without being a wall. The GitHub review chapter and the responding-to-feedback mindset chapter cover the human side — Approve vs Request changes vs Comment, and how to say "this is a BWC break" without making the contributor feel attacked.

Owning a Subsystem

A maintainer is listed in a repo's MAINTAINERS.md and is accountable for an area: search, indexing, cluster coordination, a specific plugin. Ownership means:

• You know the area's area labels (Search:Performance, Storage:Durability, Cluster Manager, etc.) and you triage untriaged issues into them.
• You know the backport policy for that area and which Version constants gate in-flight features.
• You are the BWC and performance conscience for the area — when someone changes a serialized type you own, it is your job to catch the missing gate.
• You write down the non-obvious invariants so the next contributor doesn't relearn them by breaking a rolling upgrade. (This is itself a high-value contribution; see the maintainership mindset chapter.)

Deliverables

You must demonstrate all of the following before attempting the capstone:

• Completed Lab 9.1: added a version-gated field to a transport request/response, wrote the round-trip test, and ran a qa/ BWC task.
• Completed Lab 9.2: wrote a JMH benchmark in :benchmarks, captured before/after numbers, and interpreted an OSB workload.
• A written review (in the style of a GitHub review) of one real OpenSearch PR that touches a Writeable, explicitly assessing its wire-BWC safety.
• A one-paragraph statement of which subsystem you would own and why, naming its MAINTAINERS.md, its area labels, and one BWC or perf invariant it carries.
• From memory: explain the difference between wire BWC, index/Lucene format BWC, and REST BWC, with one failure mode and one mitigation for each.

Common Mistakes

Maintainer Profile: Level 9 Graduate

You are now ready for the capstone — an end-to-end contribution where you select a real issue, reproduce it, find the root cause, implement and test the fix (with BWC and performance discipline), open the PR, and write it up. Start at the Capstone Overview. The two reflexes you built here — will this break a rolling upgrade? and did I prove the performance claim? — are the ones the capstone evaluates hardest.

Note: Most engineers never reach this tier in an open-source project, not because they can't, but because they stop at "my PR is green." The distance from contributor to maintainer is the distance from "it works" to "it keeps working for everyone who upgrades into it." That is the entire content of Level 9.