Lab 9.1: Write Backward-Compatibility (BWC) Tests

Lab type: Build It + Test It Estimated time: 4–6 hours

Background

OpenSearch clusters upgrade one node at a time. For the duration of a rolling upgrade — which on a large cluster can last hours — nodes of two different versions serve traffic in the same cluster and exchange transport messages. A 3.1 node must be able to send a request to a 3.0 node and read its response, and vice versa. The machinery that makes this possible is version-gated serialization: every object that crosses the wire implements Writeable, carries the peer's Version on its StreamInput/StreamOutput, and writes new fields only when the peer is new enough to understand them.

This lab makes that abstract. You will add a new field to a transport request/response pair, gate it correctly with a Version check, and then write the tests that prove an old node can still talk to a new node. You will run both the cheap round-trip test (AbstractWireSerializingTestCase) and the expensive but authoritative qa/ BWC suite (rolling-upgrade, mixed-cluster, full-cluster-restart).

Read the Serialization and Backward Compatibility deep dive first. This lab assumes you know why reordering stream writes corrupts a stream, what out.getVersion() returns during a rolling upgrade, and the three kinds of compatibility. This lab is the hands-on counterpart to that chapter.

Why This Lab Matters for Contributors

A missing version gate is the single most common reason a maintainer rejects an otherwise-good PR — and the most dangerous bug class to miss, because it produces no failing unit test. It surfaces only in a real mixed-version cluster, typically in production, during a customer's upgrade. Learning to write BWC tests is learning to make this invisible class of bug visible before merge. Every maintainer in the project carries this reflex; by the end of this lab, so do you.

Prerequisites

• You can build OpenSearch from source and run ./gradlew :server:test.
• You have read the serialization BWC deep dive and the transport layer deep dive.
• You understand Writeable: a writeTo(StreamOutput) method plus a StreamInput constructor, read in the same order they were written.
• You know the current version lines: 3.x/main is current, 2.x is the maintenance line, 1.x is legacy.

Step-by-Step Tasks

Step 1: Orient yourself in the version and BWC machinery

Find the Version constants — these are the levers for every wire gate — and the qa/ suites that prove BWC end to end.

# The named version constants and the ordering helpers you will gate on.
grep -n "public static final Version V_3_\|public static final Version CURRENT\|minimumCompatibilityVersion\|minimumIndexCompatibilityVersion" \
  server/src/main/java/org/opensearch/Version.java | head -40

# The methods you will use in gates.
grep -n "public boolean onOrAfter\|public boolean before\|public boolean onOrBefore\|public boolean after" \
  server/src/main/java/org/opensearch/Version.java

# The BWC QA suites.
ls qa/
# expect: rolling-upgrade  mixed-cluster  full-cluster-restart  smoke-test-* verify-version-constants ...

Note: The newest unreleased version constant on main is what you usually gate a brand-new field on. Pick it by reading Version.java on your branch — do not guess V_3_1_0 from memory. If CURRENT on your branch is 3.2.0, a field you add now ships in 3.2.0 and gates on Version.V_3_2_0. "Exact names vary by branch."

Step 2: Find a real, simple transport request/response to extend

You want a small Writeable request/response pair so the serialization is easy to read. Good candidates are simple stats or info responses. Find one and read both its writeTo and its StreamInput constructor.

# Find transport requests/responses with explicit version gates already in them —
# these are your worked examples of the correct pattern.
grep -rln "out.getVersion().onOrAfter\|in.getVersion().onOrAfter" \
  server/src/main/java/org/opensearch/action/ | head

# A concrete, readable example to study end to end:
grep -n "writeTo\|StreamInput\|getVersion().onOrAfter" \
  server/src/main/java/org/opensearch/action/admin/cluster/health/ClusterHealthRequest.java | head -40

For this lab we will use a worked, self-contained example so the pattern is crisp. Imagine a transport response NodeFeatureStatsResponse (your real target will be an existing class) that today serializes two fields:

public class NodeFeatureStatsResponse extends TransportResponse {
    private final long indexedDocs;
    private final String engineType;

    public NodeFeatureStatsResponse(long indexedDocs, String engineType) {
        this.indexedDocs = indexedDocs;
        this.engineType = engineType;
    }

    public NodeFeatureStatsResponse(StreamInput in) throws IOException {  // READ
        super(in);
        this.indexedDocs = in.readVLong();
        this.engineType = in.readString();
    }

    @Override
    public void writeTo(StreamOutput out) throws IOException {            // WRITE
        out.writeVLong(indexedDocs);
        out.writeString(engineType);
    }
}

Step 3: Add the new field — gated, appended, symmetric

We add an optional mergedSegmentCount (a long) that ships in the next release. Determine that version from Version.java (Step 1); here we assume V_3_2_0.

Three rules, all non-negotiable:

1. Append, never insert in the middle. The new write goes after every existing write; the new read goes after every existing read.
2. Gate symmetrically. The if (out.getVersion().onOrAfter(...)) on write and the if (in.getVersion().onOrAfter(...)) on read must be the identical condition.
3. Give the old peer a sensible default on read, since an old node never sent the field.

 public class NodeFeatureStatsResponse extends TransportResponse {
     private final long indexedDocs;
     private final String engineType;
+    private final long mergedSegmentCount;   // new in 3.2.0

-    public NodeFeatureStatsResponse(long indexedDocs, String engineType) {
+    public NodeFeatureStatsResponse(long indexedDocs, String engineType, long mergedSegmentCount) {
         this.indexedDocs = indexedDocs;
         this.engineType = engineType;
+        this.mergedSegmentCount = mergedSegmentCount;
     }

     public NodeFeatureStatsResponse(StreamInput in) throws IOException {  // READ
         super(in);
         this.indexedDocs = in.readVLong();
         this.engineType = in.readString();
+        if (in.getVersion().onOrAfter(Version.V_3_2_0)) {
+            this.mergedSegmentCount = in.readVLong();
+        } else {
+            this.mergedSegmentCount = 0L;        // old peer didn't send it
+        }
     }

     @Override
     public void writeTo(StreamOutput out) throws IOException {            // WRITE
         out.writeVLong(indexedDocs);
         out.writeString(engineType);
+        if (out.getVersion().onOrAfter(Version.V_3_2_0)) {
+            out.writeVLong(mergedSegmentCount);  // only send to 3.2+ peers
+        }
     }
 }

Warning: The two if conditions are the entire safety argument. If the read gate says V_3_2_0 and the write gate says V_3_1_0, then a 3.1 node will read a mergedSegmentCount that a 3.2 node never wrote (or worse, the reverse), and the stream desynchronizes — every field after it is garbage. They must be character-for- character the same condition.

When the field is a reference type, prefer the writeOptional*/readOptional* helpers so null round-trips cleanly:

// write side, gated:
out.writeOptionalString(maybeNullName);
// read side, gated:
this.maybeNullName = in.readOptionalString();

Step 4: Write the round-trip serialization test

This is the cheapest, highest-value BWC guard. AbstractWireSerializingTestCase<T> serializes your instance, deserializes it, asserts equality — and crucially also does this at random older Versions, which catches a missing or asymmetric gate immediately.

package org.opensearch.action.admin.cluster.stats;

import org.opensearch.Version;
import org.opensearch.core.common.io.stream.Writeable;
import org.opensearch.test.AbstractWireSerializingTestCase;

public class NodeFeatureStatsResponseTests
        extends AbstractWireSerializingTestCase<NodeFeatureStatsResponse> {

    @Override
    protected Writeable.Reader<NodeFeatureStatsResponse> instanceReader() {
        return NodeFeatureStatsResponse::new;     // the StreamInput ctor
    }

    @Override
    protected NodeFeatureStatsResponse createTestInstance() {
        return new NodeFeatureStatsResponse(
            randomNonNegativeLong(),
            randomFrom("internal", "nrt_replication", "read_only"),
            randomNonNegativeLong()                // mergedSegmentCount
        );
    }

    @Override
    protected NodeFeatureStatsResponse mutateInstance(NodeFeatureStatsResponse in) {
        // Optional but good practice: ensure equals/hashCode discriminate the new field.
        return new NodeFeatureStatsResponse(
            in.getIndexedDocs(),
            in.getEngineType(),
            in.getMergedSegmentCount() + 1
        );
    }
}

For the cross-version assertion to mean anything, your type must implement equals and hashCode over all fields, including the new one — otherwise the round-trip "passes" while silently dropping the field. The base class compares the deserialized instance to the original with equals.

Run it:

./gradlew :server:test --tests "org.opensearch.action.admin.cluster.stats.NodeFeatureStatsResponseTests"

If you forgot or mis-gated the field, this test fails at a randomly chosen old Version with a mismatch — often before you ever touch the slow qa/ suite. That is the point: make the gate fail in two seconds, not in a customer's upgrade.

Note: AbstractWireSerializingTestCase uses the randomized testing framework. A failure prints a -Dtests.seed=... line; re-run with that exact seed to reproduce the specific old Version that exposed the bug.

Step 5: Understand and run the qa/ BWC suites

The round-trip test proves your type round-trips at an older version in a single JVM. The qa/ suites prove that two real, separately-built clusters of different versions actually interoperate over a real network — the authoritative proof.

These are wired to run against specific old versions via bwcVersion. The build resolves the set of versions still in the compatibility window from the version catalog.

# See how the BWC versions are declared and wired into the qa/ projects.
grep -rn "bwcVersion\|bwc_version\|BWC_VERSION\|wireCompatVersions\|indexCompatVersions" \
  qa/ build.gradle settings.gradle gradle/ 2>/dev/null | head

# Run the rolling-upgrade suite (slow — it builds and boots two-version clusters).
./gradlew :qa:rolling-upgrade:check

# Run the mixed-cluster suite.
./gradlew :qa:mixed-cluster:check

# Run the full-cluster-restart suite (also exercises index/on-disk format BWC).
./gradlew :qa:full-cluster-restart:check

Warning: These tasks download and build the old version's artifacts and start multiple JVMs. They are slow (minutes to tens of minutes) and resource-hungry. Run the round-trip test in Step 4 on every change; run the qa/ suite before you open the PR and whenever you touch serialization of something that crosses the wire.

To scope a qa/ run while iterating, target a single BWC version or a single test:

# List the version-parameterized tasks the rolling-upgrade project generates.
./gradlew :qa:rolling-upgrade:tasks --all | grep -i bwc

# Run just one test class inside the suite.
./gradlew :qa:rolling-upgrade:check --tests "*Recovery*"

Step 6: Wire the new field into a real qa/ assertion (optional but ideal)

The strongest BWC test asserts the behavior the field enables across versions: a new node should still get a correct response from an old node (where the field is absent and defaults), and an old node should not choke on the new node's message (where the field is present but the old node skips it because the write gate suppressed it). In the rolling-upgrade suite these are typically REST-level assertions in the OLD/MIXED/UPGRADED task phases. Read an existing one:

grep -rln "MIXED\|UPGRADED\|TEST_STEP\|isRunningAgainstOldCluster" qa/rolling-upgrade/ | head
grep -rn "assertBusy\|client().performRequest" \
  qa/rolling-upgrade/src/test/java/org/opensearch/upgrades/ 2>/dev/null | head

Model your assertion on those: in the MIXED phase, hit the endpoint that returns your field and assert the cluster behaves correctly regardless of which node answers.

Index / Lucene Format BWC (know the difference)

Everything above is wire BWC — nodes on a network. There is a second, distinct kind: index/Lucene format BWC — data already written to disk. OpenSearch reads indices created by the previous major version (Lucene's N-1 codec back-compat), but not older; an index created two majors ago must be reindexed. The creation version is recorded as index.version.created in IndexMetadata, and the engine refuses to open an index whose format is too old.

grep -rn "index.version.created\|VERSION_CREATED\|minimumIndexCompatibilityVersion\|isCompatible" \
  server/src/main/java/org/opensearch/cluster/metadata/IndexMetadata.java \
  server/src/main/java/org/opensearch/Version.java | head

The qa:full-cluster-restart suite is where index-format BWC is exercised: it writes data on the old version, restarts on the new version, and asserts the index still opens and serves. If your change touches how a segment, a doc value, or index metadata is written, this is the suite that catches a format break. See the refresh/flush/merge deep dive for what actually lands on disk.

Pitfalls

Expected Output

A correct round-trip test run:

> Task :server:test
NodeFeatureStatsResponseTests > testSerialization PASSED
NodeFeatureStatsResponseTests > testEqualsAndHashcode PASSED

BUILD SUCCESSFUL

A failure from a missing/asymmetric gate looks like a mismatch at a random old version, with a reproducible seed:

NodeFeatureStatsResponseTests > testSerialization FAILED
    java.lang.AssertionError: expected ... but was ...
    REPRODUCE WITH: ./gradlew :server:test --tests "...NodeFeatureStatsResponseTests" \
      -Dtests.seed=DEADBEEFCAFE -Dtests.method="testSerialization"

A clean qa/ run ends with BUILD SUCCESSFUL after booting and tearing down the multi-version clusters.

Stretch Goals

• Add an enum field instead of a primitive. Enums are subtler: an old node cannot deserialize a value it doesn't know. Gate the producer so an old peer never receives the new value, and write a test that proves it.
• Make the new field a NamedWriteable and register it in NamedWriteableRegistry. Write a test that a node without the registration gets unknown NamedWriteable [x] — then add the registration and watch it pass. See the serialization BWC deep dive.
• Find a real merged PR on github.com/opensearch-project/OpenSearch that added a version-gated field. Read its writeTo/StreamInput diff and its test. Confirm the gates are symmetric and the test exercises old versions.
• Run qa:full-cluster-restart and trace how it asserts that an index written on the old version still opens on the new version.

Coding Exercises

BWC is a code skill, not a reading skill — the bug it prevents leaves no failing unit test unless you write one. Each exercise produces a serialization test that catches an asymmetric or missing version gate. Pick a real Writeable request/response with rg and study its existing gates before you touch anything.

1. (warm-up) A StreamInput/StreamOutput round-trip test. For a simple, real transport response, write an AbstractWireSerializingTestCase<T> with instanceReader, createTestInstance, and mutateInstance. Run it green, then confirm equals/hashCode cover every field (the base class compares by equals). Find a candidate and a worked-example test:

   grep -rln "extends AbstractWireSerializingTestCase" server/src/test/java/org/opensearch/action/ | head
   grep -rln "out.getVersion().onOrAfter\|in.getVersion().onOrAfter" server/src/main/java/org/opensearch/action/ | head
   

2. (core) Add a version-gated field and prove the gate symmetric. Append one optional long, gate write and read with the identical onOrAfter(...) condition (pick the version by reading Version.java on your branch — never from memory), default it for old peers, and extend createTestInstance/mutateInstance. The round-trip test serializes at random older versions, so a correct gate passes and a missing one fails.

   grep -n "public static final Version V_3_\|public static final Version CURRENT" \
     server/src/main/java/org/opensearch/Version.java | tail -10
   

3. (core) A serialization-BWC assertion at an explicit old version. Beyond random versions, write a test that serializes your instance with out.setVersion(Version.V_3_0_0) (a version before your field shipped), deserializes, and asserts the new field equals its default — proving the write gate suppressed it. Then serialize at CURRENT and assert the field round-trips intact. This pins both halves of the gate explicitly. Find the version-setting API:

   grep -rn "setVersion\|copyWriteable\|copyInstance" server/src/test/framework/ server/src/main/java/org/opensearch/core/common/io/stream/ 2>/dev/null | head
   

4. (core) The asymmetric-gate red test. Deliberately make the read gate V_3_2_0 and the write gate V_3_1_0 (or swap one for before(...)), run the round-trip test, and capture the failure + its -Dtests.seed=. Then restore symmetry. You now know exactly what a desynced stream looks like in CI — the thing the round-trip test exists to catch in two seconds instead of in a customer's upgrade.

5. (advanced) Advanced challenge — a NamedWriteable enum field with a producer gate, end to end. Add an enum field to a transport message, register it (or a new value) in NamedWriteableRegistry, and gate the producer so an old peer never receives a value it cannot deserialize. Write three tests: (a) a round-trip at CURRENT; (b) a node missing the registration gets unknown NamedWriteable [x]; (c) serializing to an old version omits/falls back the new enum value rather than sending it. Then wire one REST-level assertion into qa:rolling-upgrade so the MIXED phase proves a new node and an old node interoperate. This is the full version-gated-field contract — the reflex every maintainer carries.

   grep -rn "NamedWriteableRegistry\|registerWriteable\|getNamedWriteables" \
     server/src/main/java/org/opensearch/ | head
   grep -rln "MIXED\|isRunningAgainstOldCluster" qa/rolling-upgrade/ | head
   

Issues to Practice On

BWC and serialization work lives in opensearch-project/OpenSearch core; filter on backward-compat and the affected area (labels move; confirm with gh label list --repo opensearch-project/OpenSearch).

Representative patterns. (1) A "field added without a version gate / mixed-cluster deserialize failure" bug — reproduce by serializing across versions (or via qa:mixed-cluster), locate the ungated writeTo/StreamInput via rg "getVersion().onOrAfter", add the symmetric gate, and ship a versioned round-trip test. (2) A "flaky rolling-upgrade test" — often a real BWC break hiding behind intermittence; reproduce with the pinned seed, isolate the phase (OLD/MIXED/UPGRADED), and fix the gate or the assertion. Loop: reproduce → locate via rg → fix → test → PR with CHANGELOG + DCO sign-off.

Planted-bug drill. In a real Writeable you've added a gated field to, change only the read gate from onOrAfter(V_3_2_0) to onOrAfter(V_3_1_0) (leave the write gate alone) — an asymmetric gate. Run ./gradlew :server:test --tests "*<YourType>Tests" and watch the round-trip fail at a random old version with a reproducible seed. Restore symmetry, then add an explicit setVersion(V_3_1_0) assertion (Exercise 3) that would have caught it deterministically, not just under a lucky random seed. This is the safest way to see a silent stream desync without shipping one.

Etiquette: Claim the issue before working it, reproduce on main (and across versions) first, run :qa:rolling-upgrade:check before opening the PR, and every PR needs a test + a CHANGELOG.md entry + a DCO Signed-off-by (git commit -s). See community interaction, the compatibility chapter, and the level-2 PR lab.

Validation / Self-check

Answer all of these before marking the lab complete:

1. Write, from memory, a correct version-gated writeTo/StreamInput pair that adds one optional long field in the next release. State the single invariant that makes it safe in a mixed cluster.
2. With a byte-level sketch, explain why reordering two existing writeTo writes corrupts the stream silently rather than throwing.
3. During a 3.0→3.2 rolling upgrade, a 3.2 node sends your response to a 3.0 node. What does out.getVersion() return, and exactly which field "disappears" and why?
4. Why must your type implement equals/hashCode over the new field for AbstractWireSerializingTestCase to actually catch a missing gate?
5. Name the three qa/ suites and state what each one proves that the round-trip unit test cannot.
6. Distinguish wire BWC from index/Lucene format BWC. Which qa/ suite exercises the index-format kind, and how many major versions back can an index be read?
7. Your round-trip test is green but qa:rolling-upgrade is red. List two concrete causes and where you would look first.