Lab 4.1: Read the Coordinator and Cluster-Manager Election

This is a deep code-reading lab over org.opensearch.cluster.coordination — the consensus layer that elects a cluster manager (formerly master) and keeps the cluster formed. It is the OpenSearch analog of reading Tez's VertexImpl state machine: dense, important, and not absorbable in one pass. You will read the Coordinator and its collaborators, then run a guided 90-minute trace from node start → election → first published cluster state, with a file citation for every hop.

You will not change code. You will end with a reading-log artifact and answers to a set of questions that prove you understood the protocol, not just the file names.

Background

OpenSearch's coordination layer implements Zen2 — a Raft-inspired consensus protocol that replaced the original "Zen" discovery. It guarantees that at most one node believes it is the elected cluster manager at any committed term, using quorum-based voting over a voting configuration. The same machinery handles node joins, leaves, and failure detection.

The core ideas:

• Term: a monotonically increasing election epoch. Every successful election bumps the term. A node only accepts a publication from a manager whose term is ≥ its own.
• Voting configuration: the set of cluster-manager-eligible nodes whose votes count. A quorum is a strict majority of it; quorum overlap is what prevents split-brain.
• Pre-vote → vote → join → publish: a candidate first runs a non-binding pre-vote, then a real election, collects join votes from a quorum, becomes leader, and publishes the first state of the new term.

Deep-dive companions: discovery-coordination.md, cluster-state-publishing.md. The Level 4 overview has the class table you should keep open beside this lab.

Why This Lab Matters for Contributors

Coordination bugs are the scariest issues in the tracker: "cluster won't form," "split-brain after a network partition," "node can't rejoin," "stuck in election." Maintainers triage these by reading the exact classes in this lab and the ClusterFormationFailureHelper diagnostics. You cannot meaningfully contribute to — or even reproduce — these issues without having read the Coordinator. This is also the foundation for Lab 4.2 (publication) and the disruption tests in Level 5.

Prerequisites

• OpenSearch building; ./gradlew run works (Level 3).
• You can trace REST → Transport → Action (Lab 3.1).
• A reading log:

mkdir -p ~/opensearch-notes
: > ~/opensearch-notes/reading-log-4.1.md

Note: Coordination class/method names are stable, but bodies are long and line numbers drift. Every step gives you a grep/find to locate code on your branch. The terms here (term, quorum, voting config) are protocol vocabulary — learn them, not line numbers.

Map the package first (10 min)

find server/src/main/java -path "*cluster/coordination*" -name "*.java" | sort

Pin this table; it's your map for the whole lab:

The guided 90-minute trace: node start → election → first published state

Budget the steps. Keep the reading log open and append after each step.

Step 1 (15 min) — Where the Coordinator is born and started

grep -rn "new Coordinator\|Coordinator(" server/src/main/java/org/opensearch/node/Node.java
find server/src/main/java -name "Coordinator.java"
grep -n "public Coordinator(\|void start(\|void startInitialJoin\|enum Mode\|becomeCandidate\|becomeLeader\|becomeFollower" \
  server/src/main/java/org/opensearch/cluster/coordination/Coordinator.java | head -30

Read:

• The Coordinator constructor — note its collaborators: PeerFinder/discovery, JoinHelper, PreVoteCollector, ElectionSchedulerFactory, FollowersChecker, LeaderChecker, PublicationTransportHandler, CoordinationState (lazily, once persisted state is read).
• start() and startInitialJoin() — on node start the Coordinator enters CANDIDATE mode and begins discovery (finding peers via discovery.seed_hosts).
• The Mode enum: CANDIDATE, LEADER, FOLLOWER, and the becomeCandidate/becomeLeader/becomeFollower transitions. This is the state machine at the heart of the file.

Log it:

cat >> ~/opensearch-notes/reading-log-4.1.md <<'EOF'
## Coordinator lifecycle
- Node.java constructs Coordinator with JoinHelper, PreVoteCollector, ElectionSchedulerFactory,
  Followers/LeaderChecker, PublicationTransportHandler, CoordinationState.
- start() -> CANDIDATE; startInitialJoin() begins peer discovery.
- Mode: CANDIDATE | LEADER | FOLLOWER ; becomeCandidate/becomeLeader/becomeFollower
EOF

Step 2 (15 min) — Discovery and the pre-vote round

grep -n "PeerFinder\|getFoundPeers\|onFoundPeersUpdated\|startElectionScheduler\|PreVoteCollector\|startPreVoting\|preVoteResponse" \
  server/src/main/java/org/opensearch/cluster/coordination/Coordinator.java | head
grep -n "PreVoteResponse\|PreVoteRequest\|handlePreVoteRequest\|update\|isElectionQuorum\|start(" \
  server/src/main/java/org/opensearch/cluster/coordination/PreVoteCollector.java | head

The flow:

1. The PeerFinder finds other cluster-manager-eligible nodes and reports them via onFoundPeersUpdated.
2. When the node has discovered enough peers to possibly form a quorum, it starts the ElectionScheduler, which periodically (with randomized backoff) triggers an election attempt.
3. Before a real election, PreVoteCollector runs a pre-vote: it asks peers "would you vote for me?" without bumping the term. Only if it gets a pre-vote quorum does it proceed. This avoids disruptive elections that would needlessly bump the term and unseat a healthy leader.

Why pre-vote? In Raft, a partitioned node can repeatedly time out and bump the term, then rejoin and force a re-election even though the cluster was healthy. The pre-vote round lets such a node discover it wouldn't win before it disrupts anything. Read the class Javadoc for the exact rationale.

Step 3 (15 min) — The real election: term bump and join votes

grep -n "startElection\|StartJoinRequest\|joinLeaderInTerm\|getCurrentTerm\|incrementTerm\|handleJoinRequest" \
  server/src/main/java/org/opensearch/cluster/coordination/Coordinator.java | head
grep -n "handleJoin\|JoinRequest\|Join \|getTargetNode\|sendJoinRequest\|joinLeaderInTerm" \
  server/src/main/java/org/opensearch/cluster/coordination/JoinHelper.java | head

The real election:

1. The candidate broadcasts a StartJoinRequest for a new term (currentTerm + 1).
2. Each recipient, via JoinHelper/Coordinator.joinLeaderInTerm, decides whether it can vote in that term (it must not have already voted in a higher term) and, if so, sends a Join.
3. The candidate collects Joins. The vote-counting lives in CoordinationState:

grep -n "handleJoin\|isElectionQuorum\|VoteCollection\|getLastAcceptedConfiguration\|getLastCommittedConfiguration\|electionWon" \
  server/src/main/java/org/opensearch/cluster/coordination/CoordinationState.java | head

CoordinationState.handleJoin(...) adds a vote to a VoteCollection; isElectionQuorum(...) checks whether the collected votes form a quorum of the last-committed voting configuration. When they do, the candidate calls becomeLeader(...).

Log it:

cat >> ~/opensearch-notes/reading-log-4.1.md <<'EOF'
## Election
- pre-vote (PreVoteCollector) -> if pre-vote quorum, ElectionScheduler triggers real election
- StartJoinRequest for term = currentTerm+1
- peers send Join (JoinHelper / joinLeaderInTerm)
- CoordinationState.handleJoin -> VoteCollection; isElectionQuorum() over last-committed voting config
- quorum reached -> Coordinator.becomeLeader -> LEADER mode
EOF

Step 4 (20 min) — The first published state and how a follower joins

A freshly-elected leader must publish a cluster state to make the election durable. Find the publish entry and the follower side:

grep -n "publish\|coordinationState.handleClientValue\|PublishRequest\|PublishResponse\|ApplyCommitRequest\|handlePublishRequest" \
  server/src/main/java/org/opensearch/cluster/coordination/Coordinator.java | head
grep -n "handlePublishRequest\|handleApplyCommit\|PublishWithJoinResponse\|sendPublishRequest\|sendApplyCommit" \
  server/src/main/java/org/opensearch/cluster/coordination/PublicationTransportHandler.java | head

The first published state of a new term carries the new DiscoveryNodes (including the new leader) and the updated CoordinationMetadata. It goes through the two-phase publish (publish → quorum ACK → commit) you will study in detail in Lab 4.2. For now, note:

• The leader builds a new ClusterState (via MasterService) and hands it to Coordinator.publish(...).
• PublicationTransportHandler sends a PublishRequest (a diff or full state) to every node.
• Followers validate it against their CoordinationState (term checks), ACK, and — on the commit phase — apply it.
• A follower that wasn't part of the cluster joins by sending a Join to the new leader, which adds it to DiscoveryNodes in a subsequent published state.

Now the failure-detection loops that keep the formed cluster alive:

grep -n "FollowersChecker\|handleWakeUp\|setCurrentNodes\|onNodeFailure\|FollowerChecker" \
  server/src/main/java/org/opensearch/cluster/coordination/FollowersChecker.java | head
grep -n "LeaderChecker\|handleWakeUp\|leaderFailed\|setCurrentNodes\|onLeaderFailure" \
  server/src/main/java/org/opensearch/cluster/coordination/LeaderChecker.java | head

FollowersChecker (leader→followers) removes followers that stop responding; LeaderChecker (followers→leader) triggers a fresh CANDIDATE/election if the leader disappears. Together they are the cluster's heartbeat.

Log it:

cat >> ~/opensearch-notes/reading-log-4.1.md <<'EOF'
## First publish + steady state
- becomeLeader -> MasterService computes first state of new term -> Coordinator.publish()
- PublicationTransportHandler: PublishRequest -> quorum ACK -> ApplyCommitRequest (commit) [see Lab 4.2]
- new followers send Join -> added to DiscoveryNodes in a later published state
- steady state: FollowersChecker (leader->followers), LeaderChecker (followers->leader) heartbeats
EOF

Step 5 (5 min) — Read the failure diagnostics

When a cluster won't form, this class produces the message users paste into issues:

grep -n "describeQuorum\/\|getDescription\|cluster-manager not discovered\|master not discovered\|initial_cluster_manager_nodes\|discovery.seed_hosts" \
  server/src/main/java/org/opensearch/cluster/coordination/ClusterFormationFailureHelper.java | head

Read getDescription(). It explains why no quorum is possible — e.g. "this node must discover cluster-manager-eligible nodes [...] to bootstrap a cluster" or "have discovered [...]; need N more." Learning to read this message saves hours on real "cluster won't form" issues.

Bootstrapping: cluster.initial_cluster_manager_nodes

The very first cluster has no committed voting configuration, so there is nothing to form a quorum over. Bootstrapping seeds it:

grep -rn "INITIAL_CLUSTER_MANAGER_NODES\|INITIAL_MASTER_NODES\|initial_cluster_manager_nodes\|ClusterBootstrapService" \
  server/src/main/java/org/opensearch/cluster/coordination/ | head
find server/src/main/java -name "ClusterBootstrapService.java"
grep -n "INITIAL_CLUSTER_MANAGER_NODES\|INITIAL_MASTER_NODES\|deprecat\|bootstrap" \
  server/src/main/java/org/opensearch/cluster/coordination/ClusterBootstrapService.java | head

Terminology — cluster manager vs master: cluster.initial_cluster_manager_nodes is the modern setting; cluster.initial_master_nodes is its deprecated alias (still honored, logs a deprecation warning). ClusterBootstrapService uses whichever is set to build the initial voting configuration. It is consulted only until the cluster has a committed state — set it once, on first formation, on the cluster-manager-eligible nodes, and never again. Setting it on a running cluster does nothing.

You can watch bootstrapping live:

./gradlew run -PnumNodes=3
# In the logs of the elected node, look for:
#   [INFO ][o.o.c.c.Coordinator] ... cluster-manager node changed {previous [], current [{n1}...]}
#   [INFO ][o.o.c.s.ClusterApplierService] ... cluster-manager node changed
curl -s 'localhost:9200/_cat/cluster_manager?v'
curl -s 'localhost:9200/_cluster/state/metadata/cluster_coordination?pretty'   # the committed voting config

Reading Exercises

# 1. Where exactly is "quorum" defined? Find the majority math.
grep -rn "isQuorum\|hasQuorum\|quorum\|VotingConfiguration" \
  server/src/main/java/org/opensearch/cluster/coordination/CoordinationState.java | head

# 2. How does a node refuse a stale leader?
grep -n "term\|currentTerm\|coordinationState.handlePublishRequest\|CoordinationStateRejectedException" \
  server/src/main/java/org/opensearch/cluster/coordination/CoordinationState.java | head

# 3. The election timer backoff.
grep -n "ELECTION_INITIAL_TIMEOUT\|ELECTION_BACK_OFF\|ELECTION_MAX_TIMEOUT\|backoff\|scheduleNextElection" \
  server/src/main/java/org/opensearch/cluster/coordination/ElectionSchedulerFactory.java | head

Validation / Self-check

Answer in your own words; cite the class for each:

1. What is a term, and what guarantees does bumping it provide during an election?
2. Define quorum in terms of the voting configuration. Why does quorum overlap prevent split-brain? Which method in CoordinationState enforces it?
3. What problem does the pre-vote round (PreVoteCollector) solve that a plain vote does not?
4. Trace the messages of one election: name the request that starts it, the response a peer sends to vote, and the moment the candidate decides it has won.
5. After winning, why must the new leader publish a state before it is truly the manager? What would be inconsistent if it skipped that?
6. FollowersChecker and LeaderChecker both exist. Why two checkers instead of one? What does each trigger on failure?
7. cluster.initial_cluster_manager_nodes — when is it read, when is it ignored, and what is its deprecated alias? What goes wrong if two operators set different values on different seed nodes?
8. A user reports "cluster won't form." Which class produces the diagnostic message, and what are two distinct reasons that message might give?

When you can answer all eight from your reading log and explain the election as a sequence of messages (not just class names), you've completed Lab 4.1. Continue to Lab 4.2: ClusterState, Publishing, and the Applier Model.