Serialization and Backward Compatibility

This is the chapter that separates a contributor from a maintainer. Almost every other deep dive describes how OpenSearch works now; this one describes how OpenSearch keeps working while it changes — across a rolling upgrade, between a 3.0 node and a 3.1 node in the same cluster, between an old index written years ago and the engine reading it today. Get serialization BWC wrong and you don't get a failing unit test — you get a corrupted cluster during a customer's upgrade, the single worst class of bug this project can ship. Reviewers reject PRs over BWC more than over almost anything else, and learning to see the BWC implications of a one-line change is the skill this chapter builds.

It depends on The Transport Layer (the wire that carries Writeables) and connects to every chapter that defines a serialized type: ClusterState, Query DSL, Aggregations, Replication, and the Level-9 BWC lab.

After this chapter you can:

• Read and write version-gated Writeable serialization correctly.
• Explain why reordering stream writes silently corrupts a mixed-version cluster.
• Use Version, out.getVersion(), NamedWriteableRegistry/NamedXContentRegistry appropriately.
• Reason about wire BWC, index/Lucene format BWC, and how qa/ BWC tests prove it.

Three kinds of compatibility — don't conflate them

This chapter is mostly about the wire kind, with a section on index format. REST BWC is a policy concern covered in the compatibility mindset chapter.

The serialization primitives

Every object that crosses the transport wire implements Writeable: a writeTo(StreamOutput) method and a StreamInput constructor (the read side). The contract is brutally simple and brutally unforgiving: the read side must read fields in exactly the same order, of exactly the same type, as the write side wrote them.

public class MyState implements Writeable {
    private final String name;
    private final int count;

    public MyState(StreamInput in) throws IOException {  // READ
        this.name = in.readString();
        this.count = in.readVInt();
    }

    @Override
    public void writeTo(StreamOutput out) throws IOException {  // WRITE
        out.writeString(name);
        out.writeVInt(count);
    }
}

grep -n "interface Writeable\|writeTo\|class StreamInput\|class StreamOutput" \
  libs/core/src/main/java/org/opensearch/core/common/io/stream/Writeable.java
grep -n "public .*read\|public void write" \
  libs/core/src/main/java/org/opensearch/core/common/io/stream/StreamInput.java | head -40

StreamInput/StreamOutput carry a Version (the version of the node on the other end of the connection). That Version is the lever for all wire BWC.

The cardinal sin: reordering or retyping stream writes

A StreamOutput is a positional byte stream. There are no field names. If you write [string, vint] and read [vint, string], the reader doesn't error politely — it reads your string's length prefix as an int, then reads garbage bytes as a string, and you get corruption that may not surface until much later.

flowchart LR
    W["writeTo: writeString(name); writeVInt(count)"] --> B["bytes: [len][name...][count]"]
    B --> R1["correct read: readString(); readVInt() OK"]
    B --> R2["reordered read: readVInt(); readString() -> reads name-length as count, then garbage"]

Warning: Never reorder, insert-in-the-middle, retype, or remove a field in an existing writeTo/StreamInput pair without version-gating. This is the #1 BWC bug. New fields go at the end, behind a version check. The order in writeTo and the order in the StreamInput constructor must match each other and must match every version that's still in the mixed-cluster window.

Version gating: adding a field safely

When you add a field in version X, you must serialize it only when talking to a node that is on or after X — otherwise an older node reads bytes it doesn't expect. The pattern:

public MyState(StreamInput in) throws IOException {
    this.name = in.readString();
    this.count = in.readVInt();
    if (in.getVersion().onOrAfter(Version.V_3_1_0)) {
        this.newField = in.readOptionalString();   // only present from 3.1+
    } else {
        this.newField = null;                       // older peer didn't send it
    }
}

@Override
public void writeTo(StreamOutput out) throws IOException {
    out.writeString(name);
    out.writeVInt(count);
    if (out.getVersion().onOrAfter(Version.V_3_1_0)) {
        out.writeOptionalString(newField);          // only send to 3.1+ peers
    }
}

The symmetry is the whole game: the if condition on read and write must be identical, and the field must be appended after all pre-existing fields. The Version class defines named constants and ordering.

grep -n "public static final Version V_3\|onOrAfter\|before\|CURRENT\|minimumCompatibilityVersion" \
  server/src/main/java/org/opensearch/Version.java
grep -rn "out.getVersion().onOrAfter\|in.getVersion().onOrAfter" \
  server/src/main/java/org/opensearch/ | head

NamedWriteable: polymorphic types over the wire

A QueryBuilder, an Aggregation, an AllocationDecider decision — these are polymorphic. The stream needs to know which concrete class to instantiate. That's NamedWriteable: each type declares a stable string name (getWriteableName()), and a NamedWriteableRegistry maps name → reader. Core types and every plugin's types (see Plugin Architecture) register here.

flowchart LR
    O[writeNamedWriteable q] --> N["bytes: [name='bool'][BoolQueryBuilder fields]"]
    N --> R[readNamedWriteable: registry lookup 'bool' -> reader -> BoolQueryBuilder StreamInput ctor]

grep -n "interface NamedWriteable\|getWriteableName\|class NamedWriteableRegistry\|writeNamedWriteable\|readNamedWriteable" \
  libs/core/src/main/java/org/opensearch/core/common/io/stream/NamedWriteable.java \
  libs/core/src/main/java/org/opensearch/core/common/io/stream/NamedWriteableRegistry.java

The XContent analog is NamedXContentRegistry (name → fromXContent parser), used for JSON/YAML parsing of the same polymorphic types (see Query DSL). A plugin that adds a query must register in both: NamedWriteableRegistry for transport, NamedXContentRegistry for the REST body.

Note: Changing a getWriteableName() string is a wire break — old nodes registered the old name. The name is part of the contract, not an implementation detail.

Mixed clusters and the rolling upgrade

During a rolling upgrade, a cluster temporarily runs two versions at once. Every transport message between an old node and a new node must serialize correctly in both directions. The cluster's effective behavior is gated by the minimum node version in the cluster — the elected cluster manager (formerly master) won't enable features that older nodes can't speak.

sequenceDiagram
    participant N3 as Node v3.1 (new)
    participant N0 as Node v3.0 (old)
    Note over N3,N0: connection negotiates min(version) = 3.0
    N3->>N0: writeTo with out.getVersion()=3.0 -> SKIPS the 3.1 field
    N0->>N3: writeTo with out.getVersion()=3.1? No -> N0 has no 3.1 field; sends 3.0 layout
    N3->>N3: reads with in.getVersion()=3.0 -> does NOT read the 3.1 field

The negotiated Version on each connection is what makes out.getVersion() return the peer's version, so a 3.1 node automatically downshifts its wire format when talking to a 3.0 node. This only works if your version gate is correct.

grep -rn "minimumNodeVersion\|minimumCompatibilityVersion\|getMinNodeVersion\|smallestNonClientNodeVersion" \
  server/src/main/java/org/opensearch/cluster/node/DiscoveryNodes.java \
  server/src/main/java/org/opensearch/Version.java

Index and Lucene format BWC

Wire BWC is about nodes; index BWC is about data on disk. OpenSearch reads indices created by the previous major version (Lucene's N-1 codec back-compat), but not older than that — an index created two majors ago must be reindexed. IndexMetadata records the creation version (index.version.created), and the engine refuses to open an index whose format is too old.

grep -rn "index.version.created\|VERSION_CREATED\|minimumIndexCompatibilityVersion\|isCompatible" \
  server/src/main/java/org/opensearch/cluster/metadata/IndexMetadata.java \
  server/src/main/java/org/opensearch/Version.java

Note: This is why upgrade docs say "you can upgrade across one major, then reindex." A 1.x index can be read by 2.x; to use it on 3.x you reindex (or upgrade through 2.x). The _cat/indices + index.version.created tell you what you have.

Proving BWC: the tests that gate your PR

BWC is not a thing you eyeball — it's a thing you test, and OpenSearch has dedicated machinery.

The serialization round-trip test is the cheapest, highest-value BWC guard. It serializes your object, deserializes it, and asserts equality — and the bwc-aware variants do this at an older Version, catching a missing or asymmetric gate immediately.

public class MyStateTests extends AbstractWireSerializingTestCase<MyState> {
    @Override protected Writeable.Reader<MyState> instanceReader() { return MyState::new; }
    @Override protected MyState createTestInstance() {
        return new MyState(randomAlphaOfLength(8), randomIntBetween(0, 100));
    }
    // The framework also exercises serialization at random older versions.
}

grep -rn "class AbstractWireSerializingTestCase\|assertSerialization\|copyInstance\|VersionUtils.randomVersion" \
  test/framework/src/main/java/org/opensearch/test/AbstractWireSerializingTestCase.java
ls qa/
grep -rn "bwcVersion\|bwc_short_version\|BWC_VERSION" qa/ build.gradle settings.gradle 2>/dev/null | head

Run them:

# Round-trip serialization tests for your type
./gradlew :server:test --tests "org.opensearch.cluster.metadata.IndexMetadataTests"

# The real mixed-version / rolling-upgrade QA (slow, the actual proof)
./gradlew :qa:rolling-upgrade:check
./gradlew :qa:mixed-cluster:check

Reading exercise

# 1. The stream primitives and how Version rides along
grep -n "getVersion\|setVersion\|class StreamOutput" \
  libs/core/src/main/java/org/opensearch/core/common/io/stream/StreamOutput.java

# 2. A real version-gated writeTo in core (find live examples)
grep -rln "out.getVersion().onOrAfter" server/src/main/java/org/opensearch/cluster/ | head
# pick one and read both its writeTo and StreamInput ctor:
grep -n "out.getVersion().onOrAfter\|in.getVersion().onOrAfter\|writeTo\|StreamInput" \
  server/src/main/java/org/opensearch/cluster/metadata/IndexMetadata.java | head -40

# 3. NamedWriteable registration
grep -rn "getNamedWriteables\|new NamedWriteableRegistry.Entry" \
  server/src/main/java/org/opensearch/search/SearchModule.java | head

# 4. The round-trip test base
grep -n "copyInstance\|assertEqualInstances\|serialize\|deserialize" \
  test/framework/src/main/java/org/opensearch/test/AbstractWireSerializingTestCase.java

Answer:

1. Why does reordering two writeTo writes corrupt rather than error? Explain in terms of the positional byte stream and length prefixes.
2. Write the symmetric read/write pair for adding an optional long field in Version.V_3_2_0. What must be true of the two if conditions?
3. What does out.getVersion() return during a rolling upgrade when a 3.2 node talks to a 3.0 node, and how does that make the new field "disappear"?
4. Why must a plugin's custom query register in both NamedWriteableRegistry and NamedXContentRegistry? What does each one serve?
5. Explain index/Lucene format BWC: how many major versions back can an index be read, where the creation version is recorded, and what the upgrade path is for an index two majors old.
6. Which single test base class catches a missing version gate fastest, and what does its bwc-aware variant do that a plain round-trip doesn't?

Common bugs and symptoms

Validation: prove you understand this

1. From memory, write a correct version-gated writeTo/StreamInput pair that adds one new optional field in Version.V_3_1_0, and state the invariant that makes it safe in a mixed cluster.
2. Explain, with a byte-level sketch, why swapping the order of two writes corrupts the stream silently instead of throwing.
3. Describe what happens on each connection during a 3.0→3.1 rolling upgrade in terms of out.getVersion()/in.getVersion(), and why the negotiated version is the minimum of the two endpoints.
4. Distinguish wire BWC, index/Lucene format BWC, and REST BWC, giving one concrete failure mode and one mitigation for each.
5. Explain the role of NamedWriteableRegistry vs NamedXContentRegistry for a polymorphic type, and why renaming getWriteableName() is a breaking change.
6. Name the test base class for serialization round-trips, write a minimal subclass for a two-field type, and state what running ./gradlew :qa:rolling-upgrade:check proves that the unit test cannot.