The Transport Layer

Every interaction between nodes in an OpenSearch cluster — a search fan-out, a replication request, a join, a cluster-state publish — travels over the transport layer. It is OpenSearch's internal RPC fabric: a binary protocol on port 9300 (distinct from the REST/HTTP port 9200), built on Netty, with a typed request/response model and a custom serialization framework (StreamInput/StreamOutput, Writeable). This chapter explains TransportService and the Transport SPI, how request handlers are registered and invoked, the serialization primitives that make payloads cross the wire, and the version-aware hooks that keep a mixed-version cluster working.

After this chapter you should be able to: register a transport request handler; explain what Writeable, StreamInput/StreamOutput, and NamedWriteableRegistry each do; trace a TransportRequest from sender to handler to TransportResponse; and find where the wire format is version-gated for backward compatibility (the hook into serialization-bwc.md).

Note: Port 9200 is HTTP/REST (clients talk to it). Port 9300 is the transport protocol (nodes talk to each other). They are different stacks with different handlers. This chapter is about 9300; rest-layer.md is about 9200.

The classes

find server/src/main/java/org/opensearch/transport -name "TransportService.java"
find . -path "*transport-netty4*" -name "Netty4Transport.java"
grep -n "class TransportService\|registerRequestHandler\|sendRequest\|interface Transport" \
  server/src/main/java/org/opensearch/transport/TransportService.java

flowchart LR
    Comp[some component] -->|sendRequest| TS1[TransportService A]
    TS1 -->|Transport SPI| N4A[Netty4Transport A]
    N4A -->|"TCP 9300, framed bytes"| N4B[Netty4Transport B]
    N4B --> TS2[TransportService B]
    TS2 -->|dispatch by action name| H[registered handler]
    H -->|channel.sendResponse| TS2
    TS2 --> N4B
    N4B --> N4A
    N4A --> TS1
    TS1 -->|handleResponse| Comp

Registering and invoking a handler

Every transport interaction is keyed by an action name string (e.g. "indices:data/read/search[phase/query]"). A node that can serve an action registers a handler:

transportService.registerRequestHandler(
    ACTION_NAME,                       // String key
    ThreadPool.Names.SEARCH,           // which thread pool runs the handler
    MyRequest::new,                    // Writeable.Reader<MyRequest> (deserializer)
    (request, channel, task) -> {
        MyResponse resp = handle(request);
        channel.sendResponse(resp);    // or channel.sendResponse(exception)
    });

grep -n "registerRequestHandler" server/src/main/java/org/opensearch/transport/TransportService.java
# Real examples across the codebase:
grep -rn "registerRequestHandler" server/src/main/java | head -20

To call a remote (or local) action:

transportService.sendRequest(targetNode, ACTION_NAME, request,
    new ActionListenerResponseHandler<>(listener, MyResponse::new, ThreadPool.Names.SAME));

ActionListenerResponseHandler is the bridge between the transport world and the action world (action-framework.md): it deserializes the response and feeds it to an ActionListener.onResponse/onFailure.

Note: The thread-pool name in registerRequestHandler decides which pool runs the handler — pick it deliberately. A search handler runs on SEARCH, a write on WRITE, a lightweight metadata reply on SAME (the calling thread). Choosing the wrong pool causes rejections or starves other work; see threadpools-concurrency.md.

Serialization: Writeable, StreamInput, StreamOutput

Everything on the wire is serialized by hand through a binary stream. There is no reflection-based JSON here — it is explicit, ordered, byte-for-byte. The primitives live in libs/core:

find libs -name "Writeable.java" -o -name "StreamInput.java" -o -name "StreamOutput.java"
grep -n "interface Writeable\|void writeTo\|interface Reader" \
  libs/core/src/main/java/org/opensearch/core/common/io/stream/Writeable.java

The cardinal rule: read order must match write order, exactly. A TransportRequest typically looks like:

public MyRequest(StreamInput in) throws IOException {
    super(in);
    this.indexName = in.readString();
    this.size = in.readVInt();
    this.filter = in.readOptionalWriteable(Filter::new);
}
@Override
public void writeTo(StreamOutput out) throws IOException {
    super.writeTo(out);
    out.writeString(indexName);
    out.writeVInt(size);
    out.writeOptionalWriteable(filter);
}

Warning: A mismatch between writeTo and the reader is the single most common serialization bug. It does not fail at compile time; it manifests as a corrupt deserialization (EOFException, wrong values, or IllegalStateException) at runtime, often only against a different node version. The round-trip test base classes (AbstractWireSerializingTestCase) exist precisely to catch this — every new Writeable owes one. See serialization-bwc.md.

NamedWriteableRegistry: polymorphism on the wire

StreamInput can read a concrete type if it knows the reader. But many fields are polymorphic — a QueryBuilder could be any of dozens of subclasses, an Aggregation could be any aggregation. For those, OpenSearch writes a name and looks up the reader in a NamedWriteableRegistry.

find . -name "NamedWriteableRegistry.java" -o -name "NamedWriteable.java"
grep -n "writeNamedWriteable\|readNamedWriteable\|getNamedWriteableName\|class Entry" \
  libs/core/src/main/java/org/opensearch/core/common/io/stream/NamedWriteableRegistry.java

The registry is assembled at node start from core plus every plugin's getNamedWriteables() (see plugin-architecture.md). The XContent analog (NamedXContentRegistry) does the same for JSON parsing on the REST side — see rest-layer.md.

flowchart LR
    Q["QueryBuilder field"] -->|writeNamedWriteable| W["name 'bool' + bytes"]
    W -->|wire| R[reader node]
    R -->|"readNamedWriteable(QueryBuilder.class)"| Reg[NamedWriteableRegistry]
    Reg -->|"lookup 'bool'"| BB["BoolQueryBuilder(StreamInput)"]
    BB --> Obj[reconstructed QueryBuilder]

Connections, ports, and request options

TransportService opens multiple logical channels per node connection, grouped by purpose so that, say, a flood of bulk traffic cannot starve cluster-state pings.

grep -n "ConnectionProfile\|TransportRequestOptions\|Type\.\(BULK\|PING\|REG\|STATE\|RECOVERY\)" \
  server/src/main/java/org/opensearch/transport/TransportRequestOptions.java \
  server/src/main/java/org/opensearch/transport/ConnectionProfile.java 2>/dev/null

TransportRequestOptions lets a caller set the channel type and a per-request timeout. The port (default 9300, transport.port) and bind addresses come from NetworkModule/transport settings.

The version hook (BWC)

OpenSearch runs mixed-version clusters during rolling upgrades. The serialization layer must therefore be version-aware: a newer node writing to an older node must omit fields the older node cannot parse, and vice versa. Every StreamInput/StreamOutput carries a Version, and serialization code branches on it:

@Override
public void writeTo(StreamOutput out) throws IOException {
    out.writeString(name);
    if (out.getVersion().onOrAfter(Version.V_3_0_0)) {
        out.writeOptionalString(newFieldAddedIn3x);  // older nodes never see this
    }
}

grep -rn "out.getVersion()\|in.getVersion()\|onOrAfter\|before(" \
  server/src/main/java/org/opensearch/action/ | head

This is the seam where transport meets backward compatibility. The full rules, the Version constants, and the qa/ BWC test suite are in serialization-bwc.md. For the purposes of this chapter: any time you add or change a field on a TransportRequest/TransportResponse, you must version-gate it and add a BWC round-trip test.

Where the transport layer sits in a request

flowchart TD
    REST["RestSearchAction (9200)"] --> NC[NodeClient.execute]
    NC --> TA[TransportSearchAction]
    TA -->|"per-shard sub-requests"| TS[TransportService.sendRequest]
    TS -->|9300| Remote[data node]
    Remote --> H["search query-phase handler"]
    H --> SS[SearchService.executeQueryPhase]
    SS --> Resp["channel.sendResponse(QuerySearchResult)"]

The transport layer is the connective tissue: the action framework (action-framework.md) decides what to send and where; the transport layer moves the bytes; the thread pools (threadpools-concurrency.md) decide which thread runs each handler.

Reading exercise

# 1. The send/receive API surface.
grep -n "public .*sendRequest\|registerRequestHandler\|openConnection" \
  server/src/main/java/org/opensearch/transport/TransportService.java

# 2. A real handler registration (search query phase).
grep -rn "registerRequestHandler" server/src/main/java/org/opensearch/search/ | head

# 3. The serialization primitives.
grep -n "public .* read\|public void write" \
  libs/core/src/main/java/org/opensearch/core/common/io/stream/StreamInput.java | head -30

# 4. NamedWriteable in action for queries.
grep -rn "writeNamedWriteable\|readNamedWriteable" server/src/main/java/org/opensearch/index/query/ | head

# 5. Tests.
./gradlew :server:test --tests "org.opensearch.transport.TransportServiceTests"
./gradlew :server:test --tests "*AbstractWireSerializingTestCase*" 2>/dev/null || true

Answer:

1. What three things must you supply to registerRequestHandler, and what does the thread-pool-name argument control?
2. State the cardinal rule relating writeTo to the StreamInput constructor. What runtime error appears when it is violated?
3. When does a field use NamedWriteableRegistry instead of a plain Writeable reader? Give a concrete OpenSearch type that needs it and why.
4. What is ActionListenerResponseHandler for, and how does it connect the transport layer to the ActionListener world?
5. Why does the transport layer use multiple channel types (PING, STATE, BULK, …)? What failure does the separation prevent?
6. Find one if (out.getVersion().onOrAfter(...)) in the codebase. Explain what breaks in a rolling upgrade if that guard is removed.

Common bugs and symptoms

Validation: prove you understand this

1. Distinguish port 9200 from 9300 and name the top-level class that owns each stack.
2. Write, from memory, the skeleton of a TransportRequest with one string and one optional sub-object, including both writeTo and the reader constructor in the correct order.
3. Explain NamedWriteableRegistry versus a plain Writeable.Reader, and name two OpenSearch types that require the registry.
4. Trace a single sendRequest from caller through Netty4Transport to a remote handler and back to the caller's handleResponse.
5. Show one version-gated write and explain the exact rolling-upgrade failure that the guard prevents.
6. Given a registered handler that runs on ThreadPool.Names.GENERIC but does heavy CPU search work, explain the operational symptom and the fix.