Lab 3.2: Node Roles, Discovery, and the Transport Layer

In Lab 3.1 you traced a request within one node. Real clusters span many nodes, and the moment a request leaves a node it travels over the transport layer: a binary protocol on port 9300, with Writeable payloads and a NamedWriteableRegistry. This lab makes that concrete. You will start a 3-node cluster, give each node different roles, inspect roles and thread pools live, and read the transport classes that move messages between nodes.

Background

A node is one OpenSearch JVM. A cluster is a set of nodes that discovered each other and agreed on a cluster manager (formerly master). Three things make that work:

1. Node roles — what each node is allowed to do (node.roles).
2. Discovery — how nodes find each other and form a cluster (discovery.seed_hosts, cluster.initial_cluster_manager_nodes).
3. The transport layer — how nodes talk once connected (TransportService, Netty4Transport).

Discovery and election themselves are Level 4 material; here you treat them as "the thing that wires the nodes together" and focus on roles and transport.

Deep-dive companions: transport-layer.md, discovery-coordination.md.

Why This Lab Matters for Contributors

Most distributed bugs are inter-node bugs: a request hits a coordinating node, fans out to data nodes, and something serializes wrong, lands on the wrong thread pool, or hangs. To debug those you must be able to (a) reason about which node does what, and (b) read the transport code that carries the message. Wire-format changes are also the single most common source of backward-compatibility breakage (Level 9) — and they all live in the transport layer.

Prerequisites

• OpenSearch builds and ./gradlew run works (Lab 3.1).
• curl and jq (or python3 -m json.tool) for reading JSON.
• Two free terminals (one per node group) if you start nodes manually.

Step-by-Step Tasks

Step 1 (10 min) — Read the role definitions in source

Before starting anything, learn where roles live:

find server/src/main/java -name "DiscoveryNodeRole.java"
grep -n "DiscoveryNodeRole\|roleName\|CLUSTER_MANAGER_ROLE\|DATA_ROLE\|INGEST_ROLE\|REMOTE_CLUSTER_CLIENT_ROLE\|SEARCH_ROLE" \
  server/src/main/java/org/opensearch/cluster/node/DiscoveryNodeRole.java | head -40

DiscoveryNodeRole defines each built-in role as a static instance with a roleName (the token you put in node.roles) and a roleNameAbbreviation (the letter _cat/nodes prints). Note the deprecated master role:

grep -n "MASTER_ROLE\|master\|deprecat\|cluster_manager" \
  server/src/main/java/org/opensearch/cluster/node/DiscoveryNodeRole.java

Terminology — cluster manager vs master: OpenSearch renamed the master role to cluster manager. DiscoveryNodeRole.MASTER_ROLE remains as a deprecated alias of CLUSTER_MANAGER_ROLE; setting node.roles: [master] still works but logs a deprecation warning. Always prefer cluster_manager in new config and code.

Now see how a node reads its roles:

grep -rn "node.roles\|NODE_ROLES_SETTING\|getRolesFromSettings\|nodeRoles" \
  server/src/main/java/org/opensearch/cluster/node/DiscoveryNode.java | head

DiscoveryNode.getRolesFromSettings(Settings) parses node.roles; a DiscoveryNode (id, name, address, roles, version, attributes) is what gets serialized into DiscoveryNodes inside the cluster state.

Step 2 (10 min) — Start a 3-node cluster

You have two options. Option A is the fast path; Option B is closer to production.

Option A — ./gradlew run with multiple nodes:

# Three nodes in one invocation; gradle assigns roles and forms the cluster for you.
./gradlew run -Dtests.opensearch.cluster.initial_cluster_manager_nodes= \
  -PnumNodes=3

Check the task's own flags on your branch (they evolve):

grep -rn "numNodes\|numberOfNodes\|setNumberOfNodes" build.gradle */build.gradle 2>/dev/null | head
./gradlew help --task run

Option B — three nodes from a built distro, distinct roles:

./gradlew localDistro
DISTRO=$(find distribution/archives -type d -name "opensearch-*" | head -1)
# (Adjust DISTRO to the unpacked archive dir on your branch.)

# Node 1: cluster_manager + data
"$DISTRO/bin/opensearch" \
  -Ecluster.name=lab32 -Enode.name=n1 \
  -Enode.roles=cluster_manager,data \
  -Ehttp.port=9200 -Etransport.port=9300 \
  -Ediscovery.seed_hosts=127.0.0.1:9300,127.0.0.1:9301,127.0.0.1:9302 \
  -Ecluster.initial_cluster_manager_nodes=n1 &

# Node 2: data + ingest
"$DISTRO/bin/opensearch" \
  -Ecluster.name=lab32 -Enode.name=n2 \
  -Enode.roles=data,ingest \
  -Ehttp.port=9201 -Etransport.port=9301 \
  -Ediscovery.seed_hosts=127.0.0.1:9300,127.0.0.1:9301,127.0.0.1:9302 &

# Node 3: coordinating only (empty roles)
"$DISTRO/bin/opensearch" \
  -Ecluster.name=lab32 -Enode.name=n3 \
  -Enode.roles= \
  -Ehttp.port=9202 -Etransport.port=9302 \
  -Ediscovery.seed_hosts=127.0.0.1:9300,127.0.0.1:9301,127.0.0.1:9302 &

Note: cluster.initial_cluster_manager_nodes (deprecated alias: cluster.initial_master_nodes) is used only for the very first cluster bootstrap to seed the voting configuration. Set it on the cluster-manager-eligible nodes, and only on first formation. More in Level 4 Lab 4.1.

Step 3 (8 min) — Inspect roles live

curl -s 'localhost:9200/_cat/nodes?v&h=name,ip,node.role,cluster_manager,version'

Expected shape:

name ip         node.role cluster_manager version
n1   127.0.0.1  dm        *               3.x.x
n2   127.0.0.1  di        -               3.x.x
n3   127.0.0.1  -         -               3.x.x

The node.role column uses the abbreviations from DiscoveryNodeRole (d=data, m=cluster_manager, i=ingest, -=coordinating-only). The * in cluster_manager marks the elected manager. Now the structured view:

curl -s 'localhost:9200/_nodes?filter_path=nodes.*.name,nodes.*.roles&pretty'

Confirm n1 has [cluster_manager, data], n2 [data, ingest], n3 [].

Step 4 (7 min) — Inspect thread pools per node

curl -s 'localhost:9200/_cat/thread_pool?v&h=node_name,name,type,size,active,queue,rejected' \
  | grep -E 'node_name|search|write|generic|cluster_manager'

Notice that the coordinating-only node (n3) still has search and write pools — because any node can coordinate a request and reduce results — but it never runs the shard-local half of a write (it has no shards). The cluster_manager / master service pool is single-threaded and only busy on the elected manager. Cross-reference the table in the Level 3 overview.

Step 5 (15 min) — Read the transport layer

This is the reading core of the lab. Find the players:

find server/src/main/java -name "TransportService.java"
find modules/transport-netty4/src/main/java -name "Netty4Transport.java"
find libs/core/src/main/java -name "Writeable.java" -o -name "StreamInput.java" -o -name "StreamOutput.java"

TransportService — the front door for node-to-node messaging. Read these methods:

grep -n "public.*sendRequest\|registerRequestHandler\|connectToNode\|TransportRequestHandler" \
  server/src/main/java/org/opensearch/transport/TransportService.java | head -30

• registerRequestHandler(action, executor, requestReader, handler) — every transport action registers a handler here under a string name (the ActionType.NAME you saw in Lab 3.1), on a specific thread pool (the executor argument — this is where the pool choice lives).
• sendRequest(connection, action, request, options, responseHandler) — serialize request (Writeable.writeTo), ship it to the remote node, and invoke responseHandler when the TransportResponse returns.

The wire contract — read Writeable and a real request:

sed -n '1,60p' libs/core/src/main/java/org/opensearch/core/common/io/stream/Writeable.java
grep -n "writeTo\|StreamInput\|readFrom\|public ClusterHealthRequest" \
  server/src/main/java/org/opensearch/action/admin/cluster/health/ClusterHealthRequest.java

A Writeable type has two halves: a StreamInput-taking constructor (read) and writeTo(StreamOutput) (write). The order of reads must exactly mirror the order of writes — get it wrong and you corrupt every subsequent field. This symmetry is the #1 thing to verify when you touch a request class.

Polymorphism on the wire — NamedWriteableRegistry:

find server/src/main/java -name "NamedWriteableRegistry.java"
grep -n "register\|getReader\|NamedWriteable\|readNamedWriteable" \
  $(find . -name NamedWriteableRegistry.java | head -1)

When a field could be one of many types — e.g. which QueryBuilder is inside a SearchRequest — OpenSearch writes a name then the payload. The reader looks the name up in NamedWriteableRegistry to pick the right StreamInput constructor. Plugins register their own named writeables (you'll do this in Lab 4.3).

Step 6 (10 min) — Watch a request hop between nodes

Send a request to the coordinating-only node and reason about the hops:

# Hits n3 (coordinating only) over HTTP :9202
curl -s 'localhost:9202/_cluster/health?pretty' | jq .status

What happened:

1. HTTP on :9202 reaches n3's RestController → RestClusterHealthAction → NodeClient.
2. TransportClusterHealthAction on n3 sees it is not the cluster manager, so it TransportService.sendRequest(...) over transport :9300 to n1 (the manager).
3. n1's registered handler for cluster:monitor/health runs clusterManagerOperation, builds a ClusterHealthResponse, and sends it back over transport.
4. n3 serializes the response to JSON on the HTTP channel.

Make the hop visible by enabling transport tracing on the coordinating node:

curl -s -XPUT 'localhost:9202/_cluster/settings' -H 'Content-Type: application/json' -d '{
  "transient": {
    "transport.tracer.include": ["cluster:monitor/health*"],
    "transport.tracer.exclude": []
  }
}'
# Re-run the health curl, then watch the node logs:
#   [trace] [transport.tracer] [n3] ... sent request ... action [cluster:monitor/health] ... node [n1]
#   [trace] [transport.tracer] [n3] ... received response for ... action [cluster:monitor/health]

Find the tracer in source to understand the log lines:

grep -rn "transport.tracer\|TransportLogger\|tracerLog\|traceRequestSent\|traceReceivedResponse" \
  server/src/main/java/org/opensearch/transport/ | head

Log it in your reading log:

cat >> ~/opensearch-notes/reading-log-3.1.md <<'EOF'

## 3.2 — transport hop (health from coordinating node)
- n3 (coord) RestClusterHealthAction -> TransportClusterHealthAction
- not manager -> TransportService.sendRequest("cluster:monitor/health") -> n1 over :9300
- n1 handler (registered via registerRequestHandler on a thread pool) runs clusterManagerOperation
- ClusterHealthResponse (Writeable.writeTo) serialized back to n3, then to JSON
EOF

Reading Exercises

# 1. Which thread pool does the bulk-at-shard transport handler run on?
grep -n "registerRequestHandler\|ThreadPool.Names\|executor" \
  server/src/main/java/org/opensearch/action/support/replication/TransportReplicationAction.java | head

# 2. How does a node's role set get into the cluster state?
grep -rn "DiscoveryNodes\|getRoles\|DiscoveryNode(" \
  server/src/main/java/org/opensearch/cluster/node/DiscoveryNodes.java | head

# 3. Where is the default transport implementation chosen?
grep -rn "Netty4Transport\|TRANSPORT_TYPE\|getTransports" \
  modules/transport-netty4/src/main/java/org/opensearch/transport/Netty4Plugin.java

Answer:

1. Roles & capability. A node has node.roles: [data, ingest]. Can the cluster manager assign a shard to it? Can it be elected cluster manager? Which DiscoveryNodeRole flag decides each?
2. The executor argument. In registerRequestHandler(action, executor, ...), what is executor, and why is choosing it correctly a correctness issue, not just a performance one? (Hint: blocking the wrong pool.)
3. Wire symmetry. In ClusterHealthRequest, the StreamInput constructor reads fields in some order. What invariant must writeTo satisfy, and what breaks if a new field is read before it is written?
4. NamedWriteableRegistry. Why can't SearchRequest simply call new TermQueryBuilder(in) when reading its query off the wire? What does the registry give you that a direct constructor cannot?
5. Coordinating-only nodes. n3 holds no data but has a write thread pool. What part of a bulk write would n3 ever execute, and what part can it never execute?

Expected Output

• _cat/nodes shows three nodes with distinct role abbreviations and exactly one * cluster manager.
• _nodes confirms each node's roles array matches what you configured.
• Transport tracer log lines show the health request hopping from the coordinating node to the manager and back.
• Your reading log gained a "transport hop" entry with a file path / class name for each hop.

Stretch Goals

1. Kill the manager. Stop n1 and watch n2 (if you make it cluster_manager-eligible) get elected; _cat/nodes shows a new *. This is a preview of Level 4 Lab 4.1.
2. Force a serialization round-trip in a test. Find an AbstractWireSerializingTestCase subclass (e.g. ClusterHealthRequestTests) and run it: ./gradlew :server:test --tests "*ClusterHealthRequestTests". These tests round-trip a Writeable through StreamOutput/StreamInput and are the cheapest way to learn the wire format. (Full treatment in Level 5.)
3. Inspect connection counts. curl 'localhost:9200/_nodes/stats/transport?pretty' — find rx_count/tx_count and correlate with the requests you sent.

Validation / Self-check

1. Name the setting that declares a node's roles and the two settings (current + deprecated alias) used to bootstrap the first cluster's voting configuration.
2. Which class parses node.roles into a set of DiscoveryNodeRole, and which class carries those roles into the cluster state?
3. Port 9200 vs 9300: which protocol, which audience, and which entry class for each?
4. In TransportService.registerRequestHandler, which argument decides the thread pool the handler runs on, and why does the wrong choice cause stalls?
5. Explain, in two sentences, the role of NamedWriteableRegistry and why plugins must register their own named writeables.
6. A health request sent to a coordinating-only node still returns the right answer. Trace the exact sequence of HTTP and transport hops that makes that work.

When you can start a multi-role cluster, read its roles, and explain a single request's hops with class names, you've completed Lab 3.2. Continue to Lab 3.3: Build It — A Custom REST Action Plugin.