Stage 11 — Backward Compatibility

What this stage teaches

OpenSearch clusters run mixed versions during rolling upgrades, persist indices written by older versions, and stream objects between nodes that may not be on the same release. A change that ignores this breaks upgrades — the most expensive class of bug, because it surfaces in production, not CI. Stage 11 drills the reflexes that keep the project upgradeable:

  • Wire BWC — gating StreamInput/StreamOutput reads/writes on Version so a new field is only sent to nodes that understand it, and an old node's stream is parsed correctly.
  • Index / Lucene format BWC — an index written by version N must open on N+1; you cannot drop or repurpose a stored field or codec without a read path for the old format.
  • The qa/ tests — qa/full-cluster-restart, qa/rolling-upgrade, qa/mixed-cluster exercise old↔new behaviour for real.
  • Serialization round-trip tests — AbstractWireSerializingTestCase and AbstractSerializingTestCase prove a Writeable/XContent type survives write-then-read, including across Version boundaries with bwcSerializationCheck.
  • The deprecation policy — how a setting/field is deprecated (warning header, @Deprecated, removal only on a major) rather than removed outright.

Prerequisite: Stage 4 (cluster state crosses the wire) and the serialization & BWC deep dive, plus the compatibility mindset. The current line is 3.x/main, maintenance is 2.x, legacy is 1.x.

The two BWC axes

flowchart TD
  subgraph Wire[Wire BWC: node <-> node]
    A[new node writes a field] -->|Version gate| B{peer >= V?}
    B -->|yes| C[write the field]
    B -->|no| D[omit / write a default]
  end
  subgraph Index[Index BWC: disk]
    E[index written by N] --> F[must open on N+1]
    F --> G[old codec/field read path retained]
  end
  • Wire BWC is per-message: every writeTo/constructor-StreamInput that adds a field must guard it with out.getVersion() / in.getVersion().
  • Index BWC is per-format: the read path for an older index/segment format cannot be deleted until the major that drops support for that version.

OpenSearch's Version (in server/libs/core) and the Lucene version it embeds are the gates. A node refuses to join a cluster, or to open an index, that is too old — but within the supported window, every cross-version path must work.

Finding Stage 11 issues

is:issue is:open label:bug no:assignee "rolling upgrade" in:body
is:issue is:open label:bug no:assignee "mixed cluster" in:body
is:issue is:open label:bug no:assignee "serialization" "version" in:body
is:issue is:open label:"backport 2.x" no:assignee

Fallback grep — find the Version-gated serialization patterns to learn the idiom and spot a field that crosses the wire without a guard:

grep -rn "out.getVersion().onOrAfter\|in.getVersion().onOrAfter\|out.getVersion().before" \
  server/src/main/java/org/opensearch/ | head
ls qa/                          # full-cluster-restart, rolling-upgrade, mixed-cluster, ...
grep -rln "extends AbstractWireSerializingTestCase" server/src/test/java/org/opensearch/ | head

Walked example — adding a new field to a Writeable correctly

Illustrative of the pattern. The grep finds a real Writeable; treat the field as a stand-in for "a new optional field on a response that crosses the wire."

Symptom: you are adding a new optional field (say a long timeoutMillis) to a transport response. A naive implementation writes it unconditionally — so a 3.2 node sends 9 bytes that a 3.1 node does not expect, corrupting the stream and crashing the older node during a rolling upgrade.

Locate the serialization

grep -rn "class SomeResponse" server/src/main/java/org/opensearch/action/
grep -n "writeTo\|StreamInput\|readFrom\|readVInt\|writeVLong" \
  server/src/main/java/org/opensearch/action/<path>/SomeResponse.java

The constructor-from-stream and writeTo:

public SomeResponse(StreamInput in) throws IOException {
    super(in);
    this.count = in.readVInt();
}

@Override
public void writeTo(StreamOutput out) throws IOException {
    super.writeTo(out);
    out.writeVInt(count);
}

Diff — gate the new field on Version

First, find the version the field is introduced in. New work targets main, which carries the next Version.V_3_x_0 (or Version.CURRENT). Use the actual constant on your branch:

grep -n "public static final Version V_3" server/src/main/java/org/opensearch/Version.java | tail

--- a/server/src/main/java/org/opensearch/action/<path>/SomeResponse.java
+++ b/server/src/main/java/org/opensearch/action/<path>/SomeResponse.java
@@
 public SomeResponse(StreamInput in) throws IOException {
     super(in);
     this.count = in.readVInt();
+    if (in.getVersion().onOrAfter(Version.V_3_2_0)) {
+        this.timeoutMillis = in.readVLong();
+    } else {
+        this.timeoutMillis = DEFAULT_TIMEOUT_MILLIS;   // sensible default for old peers
+    }
 }

 @Override
 public void writeTo(StreamOutput out) throws IOException {
     super.writeTo(out);
     out.writeVInt(count);
+    if (out.getVersion().onOrAfter(Version.V_3_2_0)) {
+        out.writeVLong(timeoutMillis);
+    }
 }

The rules:

  1. Read and write guards must use the same version constant. A mismatch desynchronises the stream — the cause of most BWC crashes.
  2. Old peers get a default, not garbage. When the field is absent, choose a default that preserves old behaviour (here, the pre-existing implicit timeout).
  3. Use the variable-length types (writeVLong/readVLong, writeOptionalString, etc.) that the surrounding code uses; do not switch encodings.

Prove the round-trip with AbstractWireSerializingTestCase

This base class round-trips your object through StreamInput/StreamOutput and, crucially, across a range of versions:

public class SomeResponseTests extends AbstractWireSerializingTestCase<SomeResponse> {
    @Override protected Writeable.Reader<SomeResponse> instanceReader() { return SomeResponse::new; }
    @Override protected SomeResponse createTestInstance() {
        return new SomeResponse(randomNonNegativeInt(), randomNonNegativeLong());
    }
    // Optional: assert behaviour when serialized to an OLD version drops the field.
    public void testSerializeToOldVersionDropsField() throws IOException {
        SomeResponse original = createTestInstance();
        SomeResponse roundTripped = copyWriteable(original, writableRegistry(),
            SomeResponse::new, Version.V_3_1_0);   // serialize as if to a 3.1 node
        assertEquals(DEFAULT_TIMEOUT_MILLIS, roundTripped.getTimeoutMillis());  // field defaulted
        assertEquals(original.getCount(), roundTripped.getCount());             // rest survives
    }
}

copyWriteable(..., Version.V_3_1_0) simulates serializing to an older node — the single most important BWC test you can write for a new field.

./gradlew :server:test --tests "*SomeResponseTests" -q

Run the qa/ cross-version tests

Unit round-trips are necessary but not sufficient; the qa/ suites bring up actual old-and-new clusters:

# Rolling upgrade: a real cluster upgraded node-by-node, exercising mixed-version traffic.
./gradlew :qa:rolling-upgrade:check -Dtests.bwc.version=<previous minor> -q
# Full cluster restart: write on old, restart onto new, assert data + behaviour survive.
./gradlew :qa:full-cluster-restart:check -q
# Mixed cluster: old and new nodes coexisting.
./gradlew :qa:mixed-cluster:check -q

These are slow and depend on a downloadable BWC distribution; consult TESTING.md for the exact bwc.version invocation on your branch. They are the proof that a 3.1 node and your 3.2 node actually coexist.

Build, CHANGELOG, PR

--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ ### Added
+- Add an optional `timeoutMillis` field to `SomeResponse`, version-gated for rolling-upgrade safety ([#NNNNN](...))

In the PR, state explicitly: the version constant you gated on, the default for old peers, and that you ran (or why CI runs) the relevant qa/ suite. BWC reviewers key off exactly those three.

Index / Lucene format BWC

Wire BWC is per-message; index BWC is per-format and longer-lived. The rules:

  • An index written by a supported older version must open on the current version. The read path for the old format cannot be removed until the major that drops that version's support.
  • You cannot repurpose a stored field, doc-values format, or codec name. Add a new one; read the old one for old segments.
  • qa/full-cluster-restart is the safety net: it writes data on the old version and asserts it reads back correctly on the new one.
grep -rn "IndexVersion\|MINIMUM_COMPATIBLE\|MINIMUM_INDEX_COMPATIBILITY" \
  server/src/main/java/org/opensearch/Version.java | head

The deprecation policy

When a setting, field, or API must go away, you deprecate, then remove on a major:

  1. Mark it @Deprecated in code and emit a deprecation warning (the _deprecation log / Warning response header) when it is used.
  2. Keep an alias/default so existing configs keep working (the master → cluster_manager rename is the canonical example — both work; master warns).
  3. Remove it only in a major release, with a migration note in the breaking-changes documentation and the CHANGELOG ### Removed/### Deprecated.
grep -rn "deprecationLogger\|@Deprecated\|addDeprecatedSetting\|deprecateAndAddAlias" \
  server/src/main/java/org/opensearch/common/settings/ | head

Pitfalls

  • Mismatched read/write version guards. The single most common BWC crash. Read and write must gate on the identical Version constant.
  • No default for old peers. An absent field must default to old behaviour, not zero or null-that-NPEs downstream.
  • Removing an old index/codec read path. You cannot delete the read path for a still- supported format. Add new, retain old.
  • Skipping the qa/ suites. Unit round-trips do not exercise a real mixed cluster. Run (or ensure CI runs) rolling-upgrade/full-cluster-restart.
  • Repurposing a serialized field's meaning. Even if the type is unchanged, changing what a field means across versions desynchronises semantics. Add a new field.
  • Removing a setting without deprecation. Deprecate with an alias and a warning; remove only on a major. Reviewers will block a hard removal on a minor.
  • Guessing the version constant. grep Version.java for the real next constant on your branch; do not hard-code a number you assume.

Exit criteria — when you're ready for Stage 12

  • One BWC-sensitive PR is merged with a Version-gated StreamInput/StreamOutput, an AbstractWireSerializingTestCase that round-trips to an older version, and a passing qa/ cross-version run.
  • You can explain the difference between wire BWC (per-message, Version-gated) and index BWC (per-format, retained read path), and the support window for each.
  • You default to deprecate-then-remove-on-major, with an alias and a warning, never a hard removal on a minor.
  • You always verify the version constant against Version.java rather than assuming it.

BWC judgment is the core of release triage. Stage 12 is where that judgment decides whether an issue blocks a release.