Compatibility, Stability, and Performance

OpenSearch runs in long-lived production clusters that hold petabytes of data and are upgraded in place, node by node, without downtime. That single operational fact — rolling upgrades on clusters that cannot be re-indexed from scratch — is the source of almost every "no" a maintainer will give you. This chapter is the maintainer's three constant concerns, and it is the knowledge behind the most common blocking review comments (see Responding to Feedback).

A maintainer reviewing your PR is silently asking three questions:

1. Wire / index backward compatibility — will this break a mixed-version cluster mid-upgrade, or make today's data unreadable by a future node (or vice versa)?
2. API stability — does this change a REST contract that users and clients depend on?
3. Performance — does this regress a hot path that benchmarks watch?

If you can answer all three before they ask, your PR is half-reviewed already.

Concern 1: Wire and Index Backward Compatibility

Why a "Small" Change Breaks a Rolling Upgrade

During a rolling upgrade, the cluster is mixed: some nodes run version N, some run N+1, and they talk to each other over the transport layer (:9300). A node serializes a request/response with StreamOutput and the peer deserializes it with StreamInput. If a newer node writes a field that an older node's readFrom does not expect, the older node throws while parsing the stream — and a node that cannot parse cluster traffic is a node that cannot stay in the cluster.

So the rule: any change to what a Writeable writes or reads is a wire-compatibility change, even adding "just one field." It is invisible in single-version tests and fatal in a mixed cluster.

Version-Gated Serialization

The mechanism OpenSearch uses is Version gating inside writeTo/readFrom. The stream carries the negotiated version of the peer, and you only write the new field if the peer understands it:

@Override
public void writeTo(StreamOutput out) throws IOException {
    super.writeTo(out);
    out.writeString(name);
    if (out.getVersion().onOrAfter(Version.V_3_0_0)) {   // only newer peers get it
        out.writeOptionalString(newField);
    }
}

public FooRequest(StreamInput in) throws IOException {
    super(in);
    this.name = in.readString();
    if (in.getVersion().onOrAfter(Version.V_3_0_0)) {     // mirror the gate exactly
        this.newField = in.readOptionalString();
    } else {
        this.newField = null;                             // sensible default for old peers
    }
}

The read and write gates must match exactly, and you must choose a sane default for the old-peer case. Find the Version constants and study the pattern:

cd ~/os-src
find libs/core -name 'Version.java'
grep -n "getVersion().onOrAfter\|getVersion().before" \
  server/src/main/java/org/opensearch/action/search/SearchRequest.java | head

The deep mechanics — NamedWriteableRegistry, Diffable cluster state, XContent BWC — live in the serialization-BWC deep dive. This chapter is the judgment: recognize that you touched the wire, then gate it.

Index Backward Compatibility

Index BWC is the other half. OpenSearch reads indices written by the previous major line, and a node must read segments and metadata written before it upgraded. You break index BWC by changing how something is written to disk (mapping serialization, a new codec, metadata format) without a read path for the old form. Index BWC violations are worse than wire violations because the data is already on disk and cannot be re-serialized by negotiation — there is no peer to gate against, only the old bytes.

The qa/ BWC Test Suites

OpenSearch proves BWC with dedicated suites under qa/. These are the tests maintainers expect to see touched when your change has any compatibility surface:

ls ~/os-src/qa/
# rolling-upgrade        — node-by-node upgrade across a mixed cluster
# mixed-cluster          — old + new nodes serving traffic simultaneously
# full-cluster-restart   — write on old, restart whole cluster on new, read back
# repository-multi-version, etc.

Run the rolling-upgrade suite against a previous version (bwcVersion wiring drives this):

cd ~/os-src
./gradlew :qa:rolling-upgrade:check          # may take a while; downloads BWC artifacts
./gradlew :qa:mixed-cluster:check
./gradlew :qa:full-cluster-restart:check

For the unit-level proof, serialization round-trip tests against an older Version are the cheap, fast signal a reviewer wants:

grep -rln "AbstractWireSerializingTestCase\|AbstractSerializingTestCase" server/src/test | head

Extend one of those with a case asserting that, at the old Version, your new field is not written and the round-trip still succeeds. This is exactly the test the worked review thread in Responding to Feedback demands.

Concern 2: API Stability

The REST API is a contract. Dashboards, the language clients (opensearch-java, opensearch-py, opensearch-js), Logstash/Beats-style ingest, and countless user scripts call it. Breaking it silently breaks tools the user never thinks about.

The REST contract is defined in rest-api-spec/ and exercised by REST-YAML tests:

ls ~/os-src/rest-api-spec/src/main/resources/rest-api-spec/api/ | head
sed -n '1,40p' ~/os-src/rest-api-spec/src/main/resources/rest-api-spec/api/search.json

Rules of thumb for a REST change:

• Adding an optional query parameter or response field is generally safe (additive).
• Removing or renaming a parameter, changing a response field's type, or changing a default is a breaking change — it needs deprecation first, and usually targets a major version.
• Deprecate, don't delete. OpenSearch surfaces deprecations through the Warning HTTP header and deprecation logs. Mark the parameter deprecated, keep it working, log a warning, and remove it only at a major boundary. The master → cluster_manager rename (see Design via GitHub) is the canonical example: old names live on as deprecated aliases rather than being deleted.

REST-YAML tests document version-specific behavior with skip blocks — when behavior changes by version, the YAML test encodes both:

- skip:
    version: " - 2.99.99"
    reason: "new response field added in 3.0"

When you change a REST endpoint, update its JSON spec, update/extend its YAML tests, and confirm:

./gradlew :rest-api-spec:yamlRestTest

Concern 3: Performance — No Regressions

OpenSearch is a performance product; a search or indexing regression that ships is a regression real users feel. Maintainers guard hot paths (search query/fetch, indexing, aggregation reduce, serialization) jealously. A change that is correct but slower on a hot path can still be a "no."

The tools, smallest to largest:

The microbenchmarks live in the :benchmarks project and use JMH:

cd ~/os-src
ls benchmarks/src/main/java/org/opensearch/benchmark/ | head
# Run a JMH benchmark (pattern matches the @Benchmark classes):
./gradlew :benchmarks:jmh -Pjmh.includes='.*YourBenchmark.*'

For end-to-end throughput/latency, OpenSearch Benchmark (opensearch-benchmark, the separate benchmarking client) drives standardized workloads against a real cluster:

# Installed separately (pip install opensearch-benchmark); illustrative run:
opensearch-benchmark execute-test \
  --target-hosts localhost:9200 \
  --workload geonames \
  --pipeline benchmark-only

The discipline a maintainer expects when your change is on a hot path:

1. Measure before (baseline on main).
2. Measure after (your branch), same workload, same hardware.
3. Report the delta in the PR — numbers, not adjectives. "No measurable regression on the geonames workload (p50 indexing 41.2k → 41.0k docs/s, within noise)" is a sentence that gets a PR merged. "Should be fine, it's a small change" is not.

If you cannot run the full macrobenchmark, say so and at least provide a JMH result for the changed method; a reviewer would rather have a microbenchmark than a guess.

Putting It Together: The Compatibility Triage

Before opening a PR, triage which surfaces you touched:

flowchart TD
    A[My change] --> B{Touches a Writeable<br/>read/write or serialized<br/>cluster state?}
    B -- Yes --> B1[Version-gate it + round-trip test<br/>+ qa/ rolling-upgrade or mixed-cluster]
    A --> C{Changes a REST path,<br/>param, or response?}
    C -- Yes --> C1[Update rest-api-spec + YAML<br/>deprecate, don't delete]
    A --> D{Writes a new on-disk<br/>mapping/metadata/codec?}
    D -- Yes --> D1[Old-form read path<br/>+ qa/full-cluster-restart]
    A --> E{On a hot path<br/>search/index/agg/serde?}
    E -- Yes --> E1[Benchmark before/after,<br/>report the delta]
    B -- No --> Z[Standard unit/IT tests]
    C -- No --> Z
    D -- No --> Z
    E -- No --> Z

Most PRs touch zero of these and need only ordinary tests. But the moment a PR touches one, the review bar rises, and the contributor who self-identifies the surface and brings the proving test or benchmark is the one whose PR sails through.

Where To Go Deeper

• Mechanics of Version gating, NamedWriteableRegistry, Diffable, XContent BWC, and the qa/ suite → the serialization-BWC deep dive.
• The replication/global-checkpoint interactions that make some serialized changes subtle → the replication deep dive.
• Hands-on BWC and performance labs → Level 9, where you build and run a mixed-cluster BWC test and a benchmark comparison end to end.
• How maintainers reason about these trade-offs at the project level → the maintainer-mindset chapter.

Validation: Prove You Understand This

1. Explain, to a colleague, why adding a single non-gated field to a TransportRequest can take down a node during a rolling upgrade — name the exact failure point.
2. Write the writeTo/readFrom pair for a new optional field, Version-gated on V_3_0_0, with the correct old-peer default.
3. Name the three qa/ suites and what each one proves, and give the ./gradlew task to run the rolling-upgrade suite.
4. For a REST change that renames a query parameter, describe the deprecation path (keep working, Warning header, remove at major) and the spec/YAML files you must update.
5. Describe the before/after performance discipline for a hot-path change, name the JMH project, and write the one-sentence delta report you would put in the PR.
6. Triage a hypothetical change ("add a min_score field to a custom query, serialized over the wire and in cluster state") across all three concerns and list every artifact you'd attach.

When you can triage any change across wire/index BWC, API stability, and performance — and bring the proving test or benchmark unprompted — you have the judgment maintainers most want to see. The next chapter — The Path to Maintainership — is how that judgment, shown repeatedly, turns a contributor into a maintainer.