How Maintainers Think About Compatibility

The single most common surprise for a new OpenSearch contributor is watching a maintainer block a five-line change. The code is correct. The tests pass. And the maintainer still says no, or "not like this," or "this needs a feature flag." This chapter explains the lens behind that reflex. It is the risk lens: a maintainer is not primarily evaluating whether your change is right — CI and the diff handle most of that — they are evaluating what happens when your change meets a fleet of running clusters that they cannot upgrade atomically.

This is the maintainer-side mirror of the contributor compatibility chapter, and it leans on the serialization-BWC deep dive for the mechanics. Read those for the how; this chapter is the why it dominates everything.

The Core Fact: Clusters Are Never Uniform During an Upgrade

OpenSearch clusters are upgraded node by node — a rolling upgrade. For the duration of the upgrade, the cluster is mixed-version: some nodes run 2.17.0, some run 2.18.0, and they must speak to each other correctly the entire time. The cluster manager (formerly master) might be the old version while a data node is new, or vice versa. Replication, coordination, and search all cross that version boundary live.

This single fact generates almost every compatibility rule a maintainer enforces:

A maintainer reads your diff and asks, for each hunk: which of these surfaces does this touch, and is it safe across a version boundary? If the answer is "wire" or "index" or "REST" and the change isn't carefully versioned, the change is blocked until it is.

Wire BWC: The Version Gate

Anything that crosses the transport layer is serialized through StreamOutput.writeX(...) and StreamInput.readX(...). When you add a field to a Writeable, you cannot just write it — old nodes don't know it's there and will misread the stream. The mechanism is a version check against the stream's version:

// Writing: only emit the new field to peers new enough to understand it.
@Override
public void writeTo(StreamOutput out) throws IOException {
    out.writeString(name);
    if (out.getVersion().onOrAfter(Version.V_3_1_0)) {
        out.writeOptionalString(newField);
    }
}

// Reading: only consume it if the sender was new enough to have written it.
public MyMessage(StreamInput in) throws IOException {
    this.name = in.readString();
    if (in.getVersion().onOrAfter(Version.V_3_1_0)) {
        this.newField = in.readOptionalString();
    }
}

Note: Class and version-constant names vary by branch. Find the pattern in the tree with grep -rn "getVersion().onOrAfter" server/src/main/java/org/opensearch | head and read a few real examples before you write your own. The deep dive serialization-bwc walks the full mechanism, including NamedWriteableRegistry for polymorphic types.

A maintainer reviewing a writeTo/reader change checks three things instantly: (1) is the new field guarded by a version check on both sides; (2) is the version constant correct (the version the field actually ships in, not a guess); (3) is there a round-trip serialization test — typically extending AbstractWireSerializingTestCase — that exercises both the new and an older bwcVersion. No version guard, or no BWC test, is an automatic block.

REST API Stability and the Deprecation Policy

The REST API is a public contract. Clients, Dashboards, dashboards plugins, and third-party tools depend on response shapes and parameter names. The rules a maintainer enforces:

• You may add new optional fields and new optional parameters. Additive is usually safe.
• You may not silently remove or rename an existing field, parameter, or endpoint. That breaks clients without warning.
• To change or remove, you go through the deprecation policy: the old form keeps working but emits a deprecation warning (the Warning HTTP header, via the deprecation logger), is documented as deprecated, and is removed only in a later major version.

# See how deprecation warnings are emitted in the codebase:
grep -rn "deprecationLogger\|DeprecationLogger" server/src/main/java/org/opensearch/rest | head

This is why a maintainer will block "just rename this response field to something clearer." Clearer is not worth breaking every consumer. The renamed field has to be added alongside the old one, the old one deprecated, and the removal deferred to a major release.

Blast Radius: Why "Small" Gets Heavy Scrutiny

The size of a diff is a terrible predictor of its risk. A maintainer judges blast radius: how many code paths, nodes, and clusters a change can affect if it is subtly wrong.

Estimate it yourself before you propose the change:

cd ~/OpenSearch
# Who calls the method you're about to change?
grep -rn "applyIndexOperationOnPrimary" server/src/main/java | wc -l
# Is this on a hot path (indexing/search/coordination)?
grep -rln "ThreadPool.Names.WRITE\|ThreadPool.Names.SEARCH" server/src/main/java | head

A change with wide blast radius gets scrutiny proportional to the radius, not the line count. This is not bureaucracy; it is the maintainer pricing in the cost of being wrong in a system that stores other people's data.

Feature Flags and the experimental Path

When a change is valuable but risky — a new replication mode, a new storage backend, a behavior the team isn't sure is right yet — the maintainer's tool is to ship it off by default:

• A feature flag / setting gates the new behavior so the default path is unchanged. Find the pattern with grep -rn "FeatureFlags" server/src/main/java/org/opensearch | head.
• An experimental label/annotation tells users the surface may change and is not yet covered by the usual BWC guarantees.

This lets the project gather real-world signal without betting the default behavior of every cluster on it. As a contributor proposing something ambitious, offering a feature flag proactively is a strong signal to maintainers that you understand the risk — and it is often the difference between "let's discuss for three months" and "merged behind a flag, iterate in the open."

The Cost of a Revert

The last item in a maintainer's risk lens is the exit cost: if this is wrong, how hard is it to undo? A revert is never free:

flowchart LR
    A[Bad change merged to main] --> B[Backported to 2.x]
    B --> C[Shipped in 2.18.0]
    C --> D[Users upgraded, data written in new format]
    D --> E{Revert now?}
    E -->|Wire/REST only| F[Revert + version-guard cleanup, painful]
    E -->|Index/data format| G[Cannot cleanly revert: on-disk data exists]

A revert of a behavioral change is annoying. A revert of a change that altered an on-disk format or a wire contract that nodes now depend on can be effectively impossible — because data has been written, or clusters have upgraded, in a way that assumes the change. This asymmetry is why maintainers front-load scrutiny: it is far cheaper to argue on the issue for an extra week than to live with an un-revertable mistake in a released line. It is also why format and contract changes are gated behind version checks and feature flags in the first place — those mechanisms preserve a revert path.

What This Means for You as a Contributor

Read your own change the way a maintainer will:

1. Classify the surface. Does it touch wire, index, REST, settings, or cluster state? If yes to any, you are in BWC territory; budget accordingly.
2. Version-guard and test it. Add the version check on both read and write; add a round-trip BWC test. Do this before you ask for review.
3. Estimate blast radius, not line count. A one-liner in an allocation decider deserves more justification than a 300-line self-contained new module.
4. Offer the safety valve. For risky-but-valuable work, propose a feature flag or experimental gate yourself.
5. Respect the deprecation policy. Add-then-deprecate-then-remove-in-a-major, never silent rename.

Do this and the "why is a maintainer blocking my five lines" mystery evaporates — because you will have already answered the questions they were going to ask.

Prove You Understand This

1. Why is every OpenSearch cluster mixed-version for a window, and what does that imply for a change to StreamOutput.writeTo?
2. Show the read and write halves of a version-guarded new field. Which test base class proves it round-trips, and against what versions?
3. A reviewer asks you to keep an old REST field while adding a clearer new one, instead of renaming. Cite the policy that requires this and explain when the old field can finally go.
4. Rank these by blast radius and justify: a new self-contained ingest processor; a one-line change to DiskThresholdDecider; a change to a cluster-state serialized field.
5. When is a feature flag the right answer, and why does offering one make a risky PR easier to merge?
6. Explain why a change to an on-disk index format is harder to revert than a change to a REST response, and how maintainers preserve a revert path in advance.

Next: The Release Process and Release Trains — how a compatibility-safe, reviewed change actually ships to users.