Cluster and Node Model

An OpenSearch cluster is a set of nodes that share a single, agreed-upon view of the world — the ClusterState — and cooperatively store and search data. This chapter is the foundation: it defines what a node is as a Java process, what roles a node can play, how nodes identify each other (DiscoveryNode), and the data hierarchy every other chapter assumes (index → shard → Lucene segment, primary vs replica). It ends by walking the Node constructor, which is the single most important "wiring diagram" in the whole codebase — almost every service you will ever touch is instantiated there.

After this chapter you should be able to: name every node role and the setting that enables it; explain what a DiscoveryNode carries on the wire; draw the index→shard→segment hierarchy and place ShardRouting/ShardId/Index on it; and open Node.java and find where any major service (transport, cluster service, indices service, search service) is constructed.

What "node" means as a process

A node is a single JVM running the OpenSearch server. The entry point is short and worth reading top to bottom:

find server/src/main/java/org/opensearch -name "OpenSearch.java"
find server/src/main/java/org/opensearch -name "Bootstrap.java"
find server/src/main/java/org/opensearch -name "Node.java"

The three classes form a layered bring-up:

Read the ordering with:

grep -n "new Node\|node.start\|INSTANCE.start\|bootstrap(" \
  server/src/main/java/org/opensearch/bootstrap/Bootstrap.java

Note: The "bootstrap checks" in BootstrapChecks are production guardrails (max file descriptors, vm.max_map_count, heap size equality, etc.). They are enforced only when a node binds a non-loopback address — i.e. when it looks like production. This is why ./gradlew run on localhost skips them. Find them:

grep -n "class .*BootstrapCheck" server/src/main/java/org/opensearch/bootstrap/BootstrapChecks.java

Node roles

A node advertises a set of roles. Roles are modeled as DiscoveryNodeRole objects, each backed by a node setting. Find the catalog:

find server/src/main/java/org/opensearch -name "DiscoveryNodeRole.java"
grep -n "public static final .*Role\|roleName\|ROLES" \
  server/src/main/java/org/opensearch/cluster/node/DiscoveryNodeRole.java

The roles you must know:

Warning: "master" → "cluster manager" is more than cosmetic in the code. The role is named cluster_manager; the setting cluster.initial_master_nodes is a deprecated alias of cluster.initial_cluster_manager_nodes; many APIs accept both ?master_timeout and ?cluster_manager_timeout. When you read older code you will still see MASTER_ROLE, isMasterNode(), and TransportMasterNodeAction. Treat them as synonyms and prefer the new names in anything you write. Confirm the deprecation wiring:

grep -rn "initial_cluster_manager_nodes\|initial_master_nodes" server/src/main/java
grep -rn "cluster_manager\|CLUSTER_MANAGER_ROLE" server/src/main/java/org/opensearch/cluster/node/DiscoveryNodeRole.java

Roles are resolved at startup from node.roles (or, on old configs, from the legacy booleans node.master/node.data/node.ingest). The resolution logic:

grep -rn "NODE_ROLES_SETTING\|getRolesFromSettings\|node.roles" \
  server/src/main/java/org/opensearch/cluster/node/DiscoveryNode.java

DiscoveryNode and DiscoveryNodes

A DiscoveryNode is the wire-serializable identity of one node: its name, a persistent ephemeral ID, the transport address, its roles, attributes, and the Version it runs. It is a Writeable (see transport-layer.md) and travels inside every cluster state.

grep -n "private final\|public DiscoveryNode\|getId\|getEphemeralId\|getRoles" \
  server/src/main/java/org/opensearch/cluster/node/DiscoveryNode.java

DiscoveryNodes is the collection — an immutable view of every node in the cluster, plus convenience accessors: the elected cluster manager, the local node, lookups by ID, and filtered views (data nodes, cluster-manager-eligible nodes). It lives inside the cluster state:

grep -n "getMasterNode\|getClusterManagerNode\|getDataNodes\|getLocalNode\|class DiscoveryNodes" \
  server/src/main/java/org/opensearch/cluster/node/DiscoveryNodes.java

flowchart TD
    CS[ClusterState] --> DNs[DiscoveryNodes]
    DNs --> CM["cluster_manager node (1, elected)"]
    DNs --> DN1[data node A]
    DNs --> DN2[data node B]
    DNs --> CO[coordinating-only node]
    DNs --> Local[localNode = this process]

The data hierarchy: index → shard → segment

A logical index is a named collection of documents. Physically it is partitioned into shards, and each shard is a full, independent Lucene index composed of immutable segments.

flowchart TD
    IDX["Index 'orders' (logical)"] --> S0["Shard 0"]
    IDX --> S1["Shard 1"]
    S0 --> P0["Primary 0 (node A)"]
    S0 --> R0["Replica 0 (node B)"]
    P0 --> L0["Lucene index = Engine + IndexWriter"]
    L0 --> Seg1["segment _0"]
    L0 --> Seg2["segment _1"]
    L0 --> Seg3["segment _2 (newest)"]

The identity classes you must keep straight:

find server libs -name "Index.java" -path "*core/index*"
find server libs -name "ShardId.java"
find server -name "ShardRouting.java"
grep -n "primary\|UNASSIGNED\|INITIALIZING\|STARTED\|RELOCATING\|currentNodeId" \
  server/src/main/java/org/opensearch/cluster/routing/ShardRouting.java

Primary vs replica

Every shard has exactly one primary copy and zero or more replica copies. The primary is the source of truth for writes; replicas serve reads and provide redundancy. A write is applied on the primary first, then replicated — see replication.md and action-framework.md for TransportReplicationAction. The placement of every copy is described by a ShardRouting inside the RoutingTable (see cluster-state.md), and decided by the allocator (see shard-allocation.md).

ShardRouting carries a small state machine of its own (distinct from the IndexShardState of the actual shard object — see index-shard-lifecycle.md):

stateDiagram-v2
    [*] --> UNASSIGNED
    UNASSIGNED --> INITIALIZING: allocator assigns a node
    INITIALIZING --> STARTED: recovery completes
    STARTED --> RELOCATING: rebalance / drain
    RELOCATING --> STARTED: relocation completes
    STARTED --> UNASSIGNED: node leaves
    INITIALIZING --> UNASSIGNED: recovery fails

Note: Do not conflate the two state machines. ShardRouting.state() is the cluster manager's plan for the shard (a field in cluster state). IndexShard.state() is the data node's reality for the shard object it holds. They converge but can briefly disagree; many allocation bugs live in that gap.

How services are wired in the Node constructor

The Node constructor (Node(Environment, ...)) is long, deliberately sequential, and the best map of the engine you will find. It constructs every major subsystem in dependency order and registers them for lifecycle management. Read it slowly:

grep -n "new ThreadPool\|new TransportService\|new ClusterService\|new IndicesService\|\
new SearchService\|new NodeClient\|new ActionModule\|new Coordinator\|new GatewayMetaState\|\
new RepositoriesService\|new SnapshotsService\|new PluginsService" \
  server/src/main/java/org/opensearch/node/Node.java

The rough construction order (names/order vary by branch — verify with the grep above):

flowchart TD
    Env[Environment + Settings] --> PS[PluginsService loads plugins]
    PS --> TP[ThreadPool]
    TP --> NWR[NamedWriteableRegistry + NamedXContentRegistry]
    NWR --> TS["TransportService (Netty4Transport)"]
    TS --> CS[ClusterService = MasterService + ClusterApplierService]
    CS --> IS[IndicesService]
    IS --> NC[NodeClient]
    NC --> AM[ActionModule registers TransportActions + REST handlers]
    AM --> SS[SearchService]
    SS --> Coord[Coordinator wires discovery]
    Coord --> Gateway[GatewayMetaState persists state]

The constructor also collects each plugin's contributions (extra NamedWriteables, settings, actions, REST handlers) and folds them into the registries — this is the seam every plugin lab exploits. See plugin-architecture.md.

Most constructed services implement LifecycleComponent and are added to a list that Node.start() iterates. To see the lifecycle contract:

grep -n "lifecycle\|LifecycleComponent\|doStart\|doStop\|doClose" \
  server/src/main/java/org/opensearch/common/component/AbstractLifecycleComponent.java

Reading exercise

Bring up a node from source and inspect the live model.

# 1. Launch a single-node cluster from source.
./gradlew run
# (in another shell)

# 2. Who is in the cluster, and what roles do they hold?
curl -s "localhost:9200/_cat/nodes?v&h=name,node.role,master,ip"

# 3. The full DiscoveryNodes + roles as the cluster manager sees them.
curl -s "localhost:9200/_nodes?filter_path=nodes.*.name,nodes.*.roles" | python3 -m json.tool

# 4. Create an index and look at its shards / ShardRouting.
curl -s -X PUT "localhost:9200/orders?pretty" \
  -H 'Content-Type: application/json' \
  -d '{"settings":{"number_of_shards":2,"number_of_replicas":0}}'
curl -s "localhost:9200/_cat/shards/orders?v&h=index,shard,prirep,state,node,docs"

# 5. The Index UUID (proves name != identity).
curl -s "localhost:9200/orders/_settings?filter_path=*.settings.index.uuid"

Now answer:

1. Your single node from ./gradlew run — which roles does it hold, and why does a one-node dev cluster need to be both cluster_manager and data?
2. In DiscoveryNode, what is the difference between nodeId and ephemeralId, and which one would change if you restarted the same node?
3. Find where node.roles is parsed into a Set<DiscoveryNodeRole>. What happens if a user lists an unknown role string?
4. A coordinating-only node has node.roles: []. Trace, in DiscoveryNodes, how such a node is excluded from getClusterManagerNodes() and getDataNodes().
5. In Node.java, find the line where ClusterService is constructed. What two services does it compose, and which one runs only on the elected cluster manager?
6. Delete orders, recreate it with the same name, and re-read its UUID. Why does the engine treat the new index as a different Index despite the identical name?

Common bugs and symptoms

Validation: prove you understand this

1. From memory, list six node roles and the one-line job of each. Mark which roles make a node election-eligible vs which let it hold shards.
2. Draw the index → shard → segment hierarchy and annotate it with Index, ShardId, ShardRouting, primary/replica, and Engine/IndexWriter.
3. Explain the difference between ShardRouting.state() and IndexShard.state(). Give one realistic moment when they disagree.
4. Open Node.java. Without scrolling past the constructor, name the order in which ThreadPool, TransportService, ClusterService, and IndicesService are built, and explain why that order is forced by their dependencies.
5. Explain why Index carries a UUID and not just a name. Construct a one-line failure that occurs if code keys a cache on index name alone.
6. A teammate proposes a node with node.roles: [ingest] only. What can and cannot this node do? Will it ever be elected cluster manager? Will it ever hold a shard?