Discovery and Coordination

Before a cluster can do anything, its nodes must (1) find each other and (2) agree on who is in charge and what the cluster state is. That is the job of the coordination layer in org.opensearch.cluster.coordination. OpenSearch inherits the "Zen2" consensus design from Elasticsearch 7 — a custom, Raft-like algorithm with a single elected leader (the cluster manager), a quorum-based voting configuration, and explicit fault detectors. This chapter explains the Coordinator state machine (CANDIDATE → LEADER → FOLLOWER), the consensus core (CoordinationState), the join protocol, the election machinery, the fault detectors, and how the voting configuration mathematically prevents split brain.

After this chapter you should be able to: explain why a two-node cluster cannot safely tolerate the loss of either node; trace a node from "starting up" to "joined a leader"; name the class that runs each step of an election; and read the diagnostics OpenSearch prints when a cluster fails to form.

Note: "cluster manager" is the role formerly called "master." The coordination package still uses Master/master in many class and method names (MasterService, getMasterNode()); read them as "cluster manager."

The package map

find server/src/main/java/org/opensearch/cluster/coordination -name "*.java" | sort
find server/src/main/java/org/opensearch/discovery -name "*.java" | sort

The classes that matter:

Discovery (finding peers) is separate from coordination (agreeing):

grep -n "class Coordinator\|enum Mode\|CANDIDATE\|LEADER\|FOLLOWER" \
  server/src/main/java/org/opensearch/cluster/coordination/Coordinator.java

The Coordinator state machine

A Coordinator is always in exactly one of three modes:

stateDiagram-v2
    [*] --> CANDIDATE
    CANDIDATE --> LEADER: wins election (quorum of joins)
    CANDIDATE --> FOLLOWER: receives a valid leader publication
    LEADER --> CANDIDATE: loses quorum / FollowersChecker fails
    FOLLOWER --> CANDIDATE: LeaderChecker fails (leader lost)
    LEADER --> [*]: node shuts down
    FOLLOWER --> [*]: node shuts down

Trace mode transitions:

grep -n "becomeCandidate\|becomeLeader\|becomeFollower\|mode ==" \
  server/src/main/java/org/opensearch/cluster/coordination/Coordinator.java

Every node starts as a CANDIDATE. It either becomes the leader by winning an election, or finds the leader and becomes a follower.

Terms, votes, and CoordinationState

The consensus core is CoordinationState. Like Raft, it is built on a monotonically increasing term: each election bumps the term, and a node will only accept a leader whose term is at least its own. The key fields:

grep -n "currentTerm\|lastAcceptedState\|lastCommittedConfiguration\|\
lastPublishedConfiguration\|handleStartJoin\|handleJoin\|handlePublishRequest\|handleCommit" \
  server/src/main/java/org/opensearch/cluster/coordination/CoordinationState.java

An election proceeds as a StartJoin (a candidate, at term T) collecting Join votes. A candidate becomes leader when it has a quorum of joins from the current VotingConfiguration. The candidate then publishes a new cluster state (term T, version V+1) which followers accept and then commit — that is the two-phase publish covered in cluster-state-publishing.md.

A node from boot to "joined"

sequenceDiagram
    participant N as New node (CANDIDATE)
    participant PF as PeerFinder/SeedHostsProvider
    participant PV as PreVoteCollector
    participant ES as ElectionScheduler
    participant L as Existing leader
    N->>PF: probe discovery.seed_hosts
    PF-->>N: discovered cluster-manager-eligible peers
    alt a leader already exists
        N->>L: send Join (handleJoin)
        L-->>N: publish current cluster state
        Note over N: becomeFollower, start LeaderChecker
    else no leader
        N->>PV: run pre-vote round
        PV-->>N: quorum reachable?
        N->>ES: schedule election (randomized delay)
        ES->>N: startElection at term T
        N->>N: collect Joins from voting config
        Note over N: quorum reached -> becomeLeader
        N->>L: publish new state (term T)
    end

Read the join handshake:

grep -n "sendJoinRequest\|handleJoinRequest\|JoinCallback\|validateJoinRequest" \
  server/src/main/java/org/opensearch/cluster/coordination/JoinHelper.java

The pre-vote round (PreVoteCollector) is a politeness optimization: a node asks "if I started an election, could I plausibly win?" before bumping the term. This prevents a node that is merely partitioned from itself repeatedly disrupting a healthy cluster by forcing term increases.

grep -n "PreVoteRequest\|PreVoteResponse\|start(\|update(" \
  server/src/main/java/org/opensearch/cluster/coordination/PreVoteCollector.java

The voting configuration and split-brain prevention

The voting configuration is the set of cluster-manager-eligible node IDs whose votes count toward quorum. Quorum = strict majority of the voting configuration. This is the single rule that prevents split brain.

find server -name "VotingConfiguration.java"
grep -n "hasQuorum\|isEmpty\|getNodeIds" \
  server/src/main/java/org/opensearch/cluster/coordination/CoordinationMetadata.java

Worked example — why a strict majority is safe:

The reason two disjoint halves can never both elect a leader: each elected leader needs a strict majority of the same voting configuration, and two disjoint majorities of one set cannot exist. A partition that splits 5 eligible nodes into 3+2 lets only the side-of-3 form a cluster; the side-of-2 stays CANDIDATE and serves no writes. This is the formal split-brain guarantee.

Warning: Therefore deploy an odd number of cluster-manager-eligible nodes (3 or 5 for HA). Two such nodes is the worst case: quorum is 2, so the failure of either node halts the cluster manager. The Reconfigurator automatically shrinks/grows the voting config to keep it odd when nodes are added/removed, but it cannot fix a fundamentally even topology. Confirm:

grep -n "reconfigure\|ODD\|auto_shrink_voting_configuration" \
  server/src/main/java/org/opensearch/cluster/coordination/Reconfigurator.java

Bootstrapping the very first cluster

A brand-new cluster has no committed voting configuration yet, so a chicken/egg problem appears: you cannot elect a leader without a voting config, and you cannot commit a voting config without a leader. This is resolved by cluster bootstrapping: the operator lists the initial cluster-manager-eligible nodes once, and the cluster uses that list as the seed voting configuration.

grep -rn "INITIAL_CLUSTER_MANAGER_NODES\|initial_cluster_manager_nodes\|\
INITIAL_MASTER_NODES\|ClusterBootstrapService" \
  server/src/main/java/org/opensearch/cluster/coordination/

The setting:

Warning: cluster.initial_*_nodes is a bootstrap-only setting. Set it on the first formation, then remove it. Leaving it set, or setting it differently on different nodes, can cause two independent clusters to form from one set of nodes — a real split brain that bypasses the quorum guarantee.

Fault detection: LeaderChecker and FollowersChecker

A healthy cluster runs two heartbeat loops in opposite directions:

grep -n "leaderFailed\|handleLeaderCheck\|class LeaderChecker" \
  server/src/main/java/org/opensearch/cluster/coordination/LeaderChecker.java
grep -n "FollowerChecker\|onNodeFailure\|handleDisconnectedNode" \
  server/src/main/java/org/opensearch/cluster/coordination/FollowersChecker.java

These are governed by timeout/retry settings (cluster.fault_detection.*). Tuning them too aggressively causes spurious elections under transient network blips; too loosely delays recovery from real failures.

When the cluster will not form: ClusterFormationFailureHelper

If a node sits in CANDIDATE mode and cannot form/join a cluster, OpenSearch does not stay silent — ClusterFormationFailureHelper logs a structured diagnosis every few seconds. This message is the most valuable single artifact for debugging discovery problems.

grep -n "describeQuorum\|getDescription\|class ClusterFormationFailureHelper" \
  server/src/main/java/org/opensearch/cluster/coordination/ClusterFormationFailureHelper.java

A typical message names: the discovered peers, the nodes still needed for quorum, and whether bootstrapping has happened. Learn to read it; it tells you which of the failure modes below you are in.

Reading exercise

# 1. Read the three-mode machine.
grep -n "becomeCandidate\|becomeLeader\|becomeFollower" \
  server/src/main/java/org/opensearch/cluster/coordination/Coordinator.java

# 2. Find the consensus invariants.
grep -n "currentTerm\|handleJoin\|handlePublishRequest\|handleCommit" \
  server/src/main/java/org/opensearch/cluster/coordination/CoordinationState.java

# 3. Run the coordination unit tests and watch a simulated election.
./gradlew :server:test --tests "org.opensearch.cluster.coordination.CoordinatorTests"
./gradlew :server:test --tests "org.opensearch.cluster.coordination.CoordinationStateTests"

# 4. The "why won't it form" message.
grep -rn "cluster-manager not discovered\|master not discovered\|describeQuorum" \
  server/src/main/java/org/opensearch/cluster/coordination/

Answer:

1. In Coordinator, what condition causes becomeLeader to be called, and what is CoordinationState.currentTerm immediately afterward?
2. What does PreVoteCollector do, and what disruption does it prevent in a cluster where one node is flapping?
3. Given a 5-node cluster-manager-eligible voting configuration, exactly how many node failures can it tolerate while still electing a leader? Show the quorum arithmetic.
4. Why is cluster.initial_cluster_manager_nodes only needed on the very first formation? What goes wrong if it is left set and two nodes are isolated at startup?
5. A follower's LeaderChecker fails three times. Trace, in code, the resulting mode transition and what the node does next.
6. In Reconfigurator, find where the voting configuration is kept odd-sized. Why does evenness hurt availability?

Common bugs and symptoms

Validation: prove you understand this

1. Draw the CANDIDATE/LEADER/FOLLOWER state diagram and label every transition with the class that triggers it (ElectionScheduler, LeaderChecker, FollowersChecker, a valid publication).
2. Explain, with the quorum arithmetic, why a 2-eligible-node cluster is strictly worse for availability than a 1-eligible-node cluster and a 3-eligible-node cluster.
3. From memory, name the four classes that participate in winning an election (scheduling, pre-voting, vote collection, join accumulation) and the order they fire in.
4. A new node will not join an existing cluster. List the three pieces of information the ClusterFormationFailureHelper message gives you, and how each narrows the diagnosis.
5. Explain the role of currentTerm in preventing a stale ex-leader from continuing to publish after a partition heals.
6. Why is cluster.initial_cluster_manager_nodes dangerous to leave configured in steady state? Describe the concrete split-brain it can produce.