Lab 3.2: Node Roles, Discovery, and the Transport Layer

In Lab 3.1 you traced a request within one node. Real clusters span many nodes, and the moment a request leaves a node it travels over the transport layer: a binary protocol on port 9300, with Writeable payloads and a NamedWriteableRegistry. This lab makes that concrete. You will start a 3-node cluster, give each node different roles, inspect roles and thread pools live, and read the transport classes that move messages between nodes.

Background

A node is one OpenSearch JVM. A cluster is a set of nodes that discovered each other and agreed on a cluster manager (formerly master). Three things make that work:

  1. Node roles — what each node is allowed to do (node.roles).
  2. Discovery — how nodes find each other and form a cluster (discovery.seed_hosts, cluster.initial_cluster_manager_nodes).
  3. The transport layer — how nodes talk once connected (TransportService, Netty4Transport).

Discovery and election themselves are Level 4 material; here you treat them as "the thing that wires the nodes together" and focus on roles and transport.

Deep-dive companions: transport-layer.md, discovery-coordination.md.

Why This Lab Matters for Contributors

Most distributed bugs are inter-node bugs: a request hits a coordinating node, fans out to data nodes, and something serializes wrong, lands on the wrong thread pool, or hangs. To debug those you must be able to (a) reason about which node does what, and (b) read the transport code that carries the message. Wire-format changes are also the single most common source of backward-compatibility breakage (Level 9) — and they all live in the transport layer.

Prerequisites

  • OpenSearch builds and ./gradlew run works (Lab 3.1).
  • curl and jq (or python3 -m json.tool) for reading JSON.
  • Two free terminals (one per node group) if you start nodes manually.

Step-by-Step Tasks

Step 1 (10 min) — Read the role definitions in source

Before starting anything, learn where roles live:

find server/src/main/java -name "DiscoveryNodeRole.java"
grep -n "DiscoveryNodeRole\|roleName\|CLUSTER_MANAGER_ROLE\|DATA_ROLE\|INGEST_ROLE\|REMOTE_CLUSTER_CLIENT_ROLE\|SEARCH_ROLE" \
  server/src/main/java/org/opensearch/cluster/node/DiscoveryNodeRole.java | head -40

DiscoveryNodeRole defines each built-in role as a static instance with a roleName (the token you put in node.roles) and a roleNameAbbreviation (the letter _cat/nodes prints). Note the deprecated master role:

grep -n "MASTER_ROLE\|master\|deprecat\|cluster_manager" \
  server/src/main/java/org/opensearch/cluster/node/DiscoveryNodeRole.java

Terminology — cluster manager vs master: OpenSearch renamed the master role to cluster manager. DiscoveryNodeRole.MASTER_ROLE remains as a deprecated alias of CLUSTER_MANAGER_ROLE; setting node.roles: [master] still works but logs a deprecation warning. Always prefer cluster_manager in new config and code.

Now see how a node reads its roles:

grep -rn "node.roles\|NODE_ROLES_SETTING\|getRolesFromSettings\|nodeRoles" \
  server/src/main/java/org/opensearch/cluster/node/DiscoveryNode.java | head

DiscoveryNode.getRolesFromSettings(Settings) parses node.roles; a DiscoveryNode (id, name, address, roles, version, attributes) is what gets serialized into DiscoveryNodes inside the cluster state.

Step 2 (10 min) — Start a 3-node cluster

You have two options. Option A is the fast path; Option B is closer to production.

Option A — ./gradlew run with multiple nodes:

# Three nodes in one invocation; gradle assigns roles and forms the cluster for you.
./gradlew run -Dtests.opensearch.cluster.initial_cluster_manager_nodes= \
  -PnumNodes=3

Check the task's own flags on your branch (they evolve):

grep -rn "numNodes\|numberOfNodes\|setNumberOfNodes" build.gradle */build.gradle 2>/dev/null | head
./gradlew help --task run

Option B — three nodes from a built distro, distinct roles:

./gradlew localDistro
DISTRO=$(find distribution/archives -type d -name "opensearch-*" | head -1)
# (Adjust DISTRO to the unpacked archive dir on your branch.)

# Node 1: cluster_manager + data
"$DISTRO/bin/opensearch" \
  -Ecluster.name=lab32 -Enode.name=n1 \
  -Enode.roles=cluster_manager,data \
  -Ehttp.port=9200 -Etransport.port=9300 \
  -Ediscovery.seed_hosts=127.0.0.1:9300,127.0.0.1:9301,127.0.0.1:9302 \
  -Ecluster.initial_cluster_manager_nodes=n1 &

# Node 2: data + ingest
"$DISTRO/bin/opensearch" \
  -Ecluster.name=lab32 -Enode.name=n2 \
  -Enode.roles=data,ingest \
  -Ehttp.port=9201 -Etransport.port=9301 \
  -Ediscovery.seed_hosts=127.0.0.1:9300,127.0.0.1:9301,127.0.0.1:9302 &

# Node 3: coordinating only (empty roles)
"$DISTRO/bin/opensearch" \
  -Ecluster.name=lab32 -Enode.name=n3 \
  -Enode.roles= \
  -Ehttp.port=9202 -Etransport.port=9302 \
  -Ediscovery.seed_hosts=127.0.0.1:9300,127.0.0.1:9301,127.0.0.1:9302 &

Note: cluster.initial_cluster_manager_nodes (deprecated alias: cluster.initial_master_nodes) is used only for the very first cluster bootstrap to seed the voting configuration. Set it on the cluster-manager-eligible nodes, and only on first formation. More in Level 4 Lab 4.1.

Step 3 (8 min) — Inspect roles live

curl -s 'localhost:9200/_cat/nodes?v&h=name,ip,node.role,cluster_manager,version'

Expected shape:

name ip         node.role cluster_manager version
n1   127.0.0.1  dm        *               3.x.x
n2   127.0.0.1  di        -               3.x.x
n3   127.0.0.1  -         -               3.x.x

The node.role column uses the abbreviations from DiscoveryNodeRole (d=data, m=cluster_manager, i=ingest, -=coordinating-only). The * in cluster_manager marks the elected manager. Now the structured view:

curl -s 'localhost:9200/_nodes?filter_path=nodes.*.name,nodes.*.roles&pretty'

Confirm n1 has [cluster_manager, data], n2 [data, ingest], n3 [].

Step 4 (7 min) — Inspect thread pools per node

curl -s 'localhost:9200/_cat/thread_pool?v&h=node_name,name,type,size,active,queue,rejected' \
  | grep -E 'node_name|search|write|generic|cluster_manager'

Notice that the coordinating-only node (n3) still has search and write pools — because any node can coordinate a request and reduce results — but it never runs the shard-local half of a write (it has no shards). The cluster_manager / master service pool is single-threaded and only busy on the elected manager. Cross-reference the table in the Level 3 overview.

Step 5 (15 min) — Read the transport layer

This is the reading core of the lab. Find the players:

find server/src/main/java -name "TransportService.java"
find modules/transport-netty4/src/main/java -name "Netty4Transport.java"
find libs/core/src/main/java -name "Writeable.java" -o -name "StreamInput.java" -o -name "StreamOutput.java"

TransportService — the front door for node-to-node messaging. Read these methods:

grep -n "public.*sendRequest\|registerRequestHandler\|connectToNode\|TransportRequestHandler" \
  server/src/main/java/org/opensearch/transport/TransportService.java | head -30
  • registerRequestHandler(action, executor, requestReader, handler) — every transport action registers a handler here under a string name (the ActionType.NAME you saw in Lab 3.1), on a specific thread pool (the executor argument — this is where the pool choice lives).
  • sendRequest(connection, action, request, options, responseHandler) — serialize request (Writeable.writeTo), ship it to the remote node, and invoke responseHandler when the TransportResponse returns.

The wire contract — read Writeable and a real request:

sed -n '1,60p' libs/core/src/main/java/org/opensearch/core/common/io/stream/Writeable.java
grep -n "writeTo\|StreamInput\|readFrom\|public ClusterHealthRequest" \
  server/src/main/java/org/opensearch/action/admin/cluster/health/ClusterHealthRequest.java

A Writeable type has two halves: a StreamInput-taking constructor (read) and writeTo(StreamOutput) (write). The order of reads must exactly mirror the order of writes — get it wrong and you corrupt every subsequent field. This symmetry is the #1 thing to verify when you touch a request class.

Polymorphism on the wire — NamedWriteableRegistry:

find server/src/main/java -name "NamedWriteableRegistry.java"
grep -n "register\|getReader\|NamedWriteable\|readNamedWriteable" \
  $(find . -name NamedWriteableRegistry.java | head -1)

When a field could be one of many types — e.g. which QueryBuilder is inside a SearchRequest — OpenSearch writes a name then the payload. The reader looks the name up in NamedWriteableRegistry to pick the right StreamInput constructor. Plugins register their own named writeables (you'll do this in Lab 4.3).

Step 6 (10 min) — Watch a request hop between nodes

Send a request to the coordinating-only node and reason about the hops:

# Hits n3 (coordinating only) over HTTP :9202
curl -s 'localhost:9202/_cluster/health?pretty' | jq .status

What happened:

  1. HTTP on :9202 reaches n3's RestController → RestClusterHealthAction → NodeClient.
  2. TransportClusterHealthAction on n3 sees it is not the cluster manager, so it TransportService.sendRequest(...) over transport :9300 to n1 (the manager).
  3. n1's registered handler for cluster:monitor/health runs clusterManagerOperation, builds a ClusterHealthResponse, and sends it back over transport.
  4. n3 serializes the response to JSON on the HTTP channel.

Make the hop visible by enabling transport tracing on the coordinating node:

curl -s -XPUT 'localhost:9202/_cluster/settings' -H 'Content-Type: application/json' -d '{
  "transient": {
    "transport.tracer.include": ["cluster:monitor/health*"],
    "transport.tracer.exclude": []
  }
}'
# Re-run the health curl, then watch the node logs:
#   [trace] [transport.tracer] [n3] ... sent request ... action [cluster:monitor/health] ... node [n1]
#   [trace] [transport.tracer] [n3] ... received response for ... action [cluster:monitor/health]

Find the tracer in source to understand the log lines:

grep -rn "transport.tracer\|TransportLogger\|tracerLog\|traceRequestSent\|traceReceivedResponse" \
  server/src/main/java/org/opensearch/transport/ | head

Log it in your reading log:

cat >> ~/opensearch-notes/reading-log-3.1.md <<'EOF'

## 3.2 — transport hop (health from coordinating node)
- n3 (coord) RestClusterHealthAction -> TransportClusterHealthAction
- not manager -> TransportService.sendRequest("cluster:monitor/health") -> n1 over :9300
- n1 handler (registered via registerRequestHandler on a thread pool) runs clusterManagerOperation
- ClusterHealthResponse (Writeable.writeTo) serialized back to n3, then to JSON
EOF

Reading Exercises

# 1. Which thread pool does the bulk-at-shard transport handler run on?
grep -n "registerRequestHandler\|ThreadPool.Names\|executor" \
  server/src/main/java/org/opensearch/action/support/replication/TransportReplicationAction.java | head

# 2. How does a node's role set get into the cluster state?
grep -rn "DiscoveryNodes\|getRoles\|DiscoveryNode(" \
  server/src/main/java/org/opensearch/cluster/node/DiscoveryNodes.java | head

# 3. Where is the default transport implementation chosen?
grep -rn "Netty4Transport\|TRANSPORT_TYPE\|getTransports" \
  modules/transport-netty4/src/main/java/org/opensearch/transport/Netty4Plugin.java

Answer:

  1. Roles & capability. A node has node.roles: [data, ingest]. Can the cluster manager assign a shard to it? Can it be elected cluster manager? Which DiscoveryNodeRole flag decides each?
  2. The executor argument. In registerRequestHandler(action, executor, ...), what is executor, and why is choosing it correctly a correctness issue, not just a performance one? (Hint: blocking the wrong pool.)
  3. Wire symmetry. In ClusterHealthRequest, the StreamInput constructor reads fields in some order. What invariant must writeTo satisfy, and what breaks if a new field is read before it is written?
  4. NamedWriteableRegistry. Why can't SearchRequest simply call new TermQueryBuilder(in) when reading its query off the wire? What does the registry give you that a direct constructor cannot?
  5. Coordinating-only nodes. n3 holds no data but has a write thread pool. What part of a bulk write would n3 ever execute, and what part can it never execute?

Expected Output

  • _cat/nodes shows three nodes with distinct role abbreviations and exactly one * cluster manager.
  • _nodes confirms each node's roles array matches what you configured.
  • Transport tracer log lines show the health request hopping from the coordinating node to the manager and back.
  • Your reading log gained a "transport hop" entry with a file path / class name for each hop.

Stretch Goals

  1. Kill the manager. Stop n1 and watch n2 (if you make it cluster_manager-eligible) get elected; _cat/nodes shows a new *. This is a preview of Level 4 Lab 4.1.
  2. Force a serialization round-trip in a test. Find an AbstractWireSerializingTestCase subclass (e.g. ClusterHealthRequestTests) and run it: ./gradlew :server:test --tests "*ClusterHealthRequestTests". These tests round-trip a Writeable through StreamOutput/StreamInput and are the cheapest way to learn the wire format. (Full treatment in Level 5.)
  3. Inspect connection counts. curl 'localhost:9200/_nodes/stats/transport?pretty' — find rx_count/tx_count and correlate with the requests you sent.

Coding Exercises

You read roles and the transport wire contract. Now write code that asserts role behavior and forces a real serialization round-trip — the two things that break in inter-node bugs. Locate every class with rg before you touch it.

  1. (warm-up) Assert role abbreviations and tokens. Write an OpenSearchTestCase that asserts DiscoveryNodeRole.DATA_ROLE.roleName() is "data" and its abbreviation is "d", and likewise for CLUSTER_MANAGER_ROLE ("cluster_manager" / "m"). Find the exact accessors: rg -n "roleName|roleNameAbbreviation" server/src/main/java/org/opensearch/cluster/node/DiscoveryNodeRole.java. This pins the _cat/nodes contract you saw in Step 3.

  2. (warm-up) Parse node.roles in a test. Build a Settings with node.roles: [data, ingest], call DiscoveryNode.getRolesFromSettings(settings) (confirm the name: rg -n "getRolesFromSettings" server/src/main/java/org/opensearch/cluster/node/DiscoveryNode.java), and assert the returned set contains DATA_ROLE and INGEST_ROLE but not CLUSTER_MANAGER_ROLE. Then add a case asserting node.roles: [master] still parses (to the deprecated alias) — proving the terminology note in Step 1.

  3. (core) A Writeable round-trip test for a real request. This is the cheapest way to learn the wire format. Mirror an existing AbstractWireSerializingTestCase (rg -l "extends AbstractWireSerializingTestCase" server/src/test/java | head) and write MyClusterHealthRequestRoundTripTest that round-trips a ClusterHealthRequest (with non-default timeout, waitForStatus, indices) through StreamOutput→StreamInput and asserts the reconstructed request equals the original. Run ./gradlew :server:test --tests "*MyClusterHealthRequest*".

  4. (core) Prove wire-symmetry breakage by hand. Write a Writeable toy class Pair with two String fields whose writeTo writes a then b but whose StreamInput constructor reads b then a (deliberately swapped). In a test, round-trip it through a BytesStreamOutput / StreamInput (find the helpers: rg -n "class BytesStreamOutput" libs/) and assert the fields come back swapped. Then fix the order and assert equality. You have now demonstrated the #1 transport bug in 30 lines — exactly what the wire-symmetry invariant in Step 5 protects against.

  5. (core) A custom TransportRequest round-trip with a custom field. Define EchoRequest extends TransportRequest (find the base: rg -n "class TransportRequest" server/src/main/java/org/opensearch/transport) carrying a String message and an int count, with a symmetric writeTo/read constructor. Write a round-trip test. Bonus: add a Version-gated field (write count only if (out.getVersion().onOrAfter(...))) and round-trip across two versions — your first taste of BWC (Level 9).

  6. (advanced challenge) A two-node integration test that observes a transport hop. Using OpenSearchIntegTestCase with @ClusterScope(numDataNodes = 2) (and a coordinating-only node via internalCluster().startCoordinatingOnlyNode(Settings.EMPTY) — confirm the helper: rg -n "startCoordinatingOnlyNode" test/framework/src/main/java), register a TransportService message listener (addMessageListener, rg -n "addMessageListener|TransportMessageListener" server/src/main/java/org/opensearch/transport) on the coordinating node, send GET /_cluster/health to it, and assert the listener observed a sent request for action cluster:monitor/health to the elected manager. This codifies the exact hop you watched via the transport tracer in Step 6 — turning tracer output into a permanent test.

Issues to Practice On

Transport and node-role issues are higher-stakes (they touch the wire and BWC) but small fixes here are very welcome. Target opensearch-project/OpenSearch.

# Coordination/distributed/BWC areas (labels move; confirm on the tracker):
gh issue list --repo opensearch-project/OpenSearch --label "distributed" --state open
gh issue list --repo opensearch-project/OpenSearch --label "Cluster Manager" --state open
gh issue list --repo opensearch-project/OpenSearch --label "flaky-test" --state open --search "transport"
gh issue list --repo opensearch-project/OpenSearch --label "good first issue" --state open
gh label list --repo opensearch-project/OpenSearch | rg -i "transport|cluster|coordination|backward"

Representative patterns:

  • A serialization/BWC bug: a field added without a Version guard, so an old node can't read a new node's message. Reproduce with a mixed-version AbstractWireSerializingTestCase (exercise 5), locate the writeTo that forgot the guard via rg, add the guard + a cross-version round-trip test.
  • A flaky transport/disruption test: reproduce with ./gradlew test --tests "*X*" -Dtests.iters=50, find the race, and stabilize it. Flaky-test fixes are some of the most appreciated first PRs.

Planted-bug drill. Open ClusterHealthRequest and, in writeTo, swap the write order of two adjacent fields without changing the read constructor (or simply read a field the writer never wrote). Run ./gradlew :server:test --tests "*ClusterHealthRequestTests*" and watch the round-trip test go red — observe how the corruption manifests (often a wrong value or EOFException, not the field you touched). Revert, then confirm your exercise-3 test would also have caught it.

Etiquette: claim before you work, reproduce first, and every PR needs a test + CHANGELOG.md entry + DCO Signed-off-by (git commit -s). Wire-format changes especially need a reviewer's eye — flag BWC impact in the PR. See community interaction and the Level 2 PR lab.

Validation / Self-check

  1. Name the setting that declares a node's roles and the two settings (current + deprecated alias) used to bootstrap the first cluster's voting configuration.
  2. Which class parses node.roles into a set of DiscoveryNodeRole, and which class carries those roles into the cluster state?
  3. Port 9200 vs 9300: which protocol, which audience, and which entry class for each?
  4. In TransportService.registerRequestHandler, which argument decides the thread pool the handler runs on, and why does the wrong choice cause stalls?
  5. Explain, in two sentences, the role of NamedWriteableRegistry and why plugins must register their own named writeables.
  6. A health request sent to a coordinating-only node still returns the right answer. Trace the exact sequence of HTTP and transport hops that makes that work.

When you can start a multi-role cluster, read its roles, and explain a single request's hops with class names, you've completed Lab 3.2. Continue to Lab 3.3: Build It — A Custom REST Action Plugin.