Lab DC1: Trace a Cluster-Manager Election

Background

You have read the intensive and the deep-dive discovery-coordination: an election is pre-vote → term bump → collect Joins from a quorum → becomeLeader → publish the first state of the new term. Names on a page are not the same as watching it happen. In this lab you bring up a 3-node cluster from source, turn on TRACE logging for org.opensearch.cluster.coordination, and watch:

1. a clean first election — pre-vote, election, the first published state;
2. a re-election — you kill the elected cluster manager (formerly master) and watch the survivors detect the loss, elect a new leader, and reconfigure the voting configuration.

Every interesting log line, you will map back to the exact class (Coordinator, PreVoteCollector, JoinHelper, CoordinationState, FollowersChecker, LeaderChecker) with a grep. You end with an annotated log and the answers that prove you traced the protocol, not just the file names.

Note: Class and method names are stable; log strings and line numbers drift across versions. Every step gives you a grep/find to locate the real code and the real log site on your branch. Do not memorize line numbers.

Why This Matters for Contributors

"Cluster won't form," "split-brain after a partition," "stuck in election," "new node can't join" — these are the scariest issues in the tracker, and maintainers triage them by reading exactly these classes and exactly these log lines. If you cannot bring up a cluster and read its coordination TRACE, you cannot reproduce, diagnose, or fix a coordination bug. This lab is the muscle: from a live cluster's logs to the line of code that printed them. It is the runnable companion to Level 4 Lab 4.1 (the code-reading lab) and the foundation for Lab DC2 (partitions).

Prerequisites

• OpenSearch builds and ./gradlew run works (Level 3). Java 21+.
• You have read the intensive election section.
• A scratch directory for logs:

mkdir -p ~/opensearch-notes/dc1
: > ~/opensearch-notes/dc1/election-log.md

You will run a 3-node cluster. Two paths — pick one:

This lab uses Path A for the main trace (you can curl it and read files), and shows the Path B skeleton at the end for a deterministic re-run.

Step-by-Step Tasks

Step 1 — Find where the log strings live, so you can recognize them

Before you generate logs, learn what to look for. The coordination package logs at INFO for state changes and at TRACE/DEBUG for the protocol messages.

cd ~/src/OpenSearch    # your checkout

# Mode transitions and "cluster-manager node changed" INFO lines:
grep -rn "becomeCandidate\|becomeLeader\|becomeFollower\|cluster-manager node changed\|master node changed" \
  server/src/main/java/org/opensearch/cluster/coordination/Coordinator.java | head

# Pre-vote round:
grep -n "PreVoteRequest\|PreVoteResponse\|starting election\|election scheduler" \
  server/src/main/java/org/opensearch/cluster/coordination/PreVoteCollector.java \
  server/src/main/java/org/opensearch/cluster/coordination/ElectionSchedulerFactory.java | head

# Join handshake:
grep -n "sendJoinRequest\|handleJoinRequest\|failed to join\|join validation" \
  server/src/main/java/org/opensearch/cluster/coordination/JoinHelper.java | head

# Vote counting / election won:
grep -n "handleJoin\|isElectionQuorum\|electionWon\|VoteCollection" \
  server/src/main/java/org/opensearch/cluster/coordination/CoordinationState.java | head

Paste the grep hits into your log file so you have the class→log-string map open while you read the logs:

cat >> ~/opensearch-notes/dc1/election-log.md <<'EOF'
## Class → log-string map (fill from greps above)
- Coordinator.becomeLeader  -> "cluster-manager node changed {previous [], current [...]}"
- ElectionSchedulerFactory  -> "starting election" (scheduled, randomized)
- PreVoteCollector          -> pre-vote round
- JoinHelper                -> join request/response
- CoordinationState         -> handleJoin / isElectionQuorum (TRACE)
EOF

Step 2 — Bring up a 3-node cluster with coordination TRACE on

./gradlew run reads logging config; the simplest way to get coordination TRACE is to pass the logger setting on the command line so all nodes inherit it:

cd ~/src/OpenSearch
./gradlew run -PnumNodes=3 \
  -Dtests.opensearch.logger.org.opensearch.cluster.coordination=TRACE \
  -Dtests.opensearch.logger.org.opensearch.cluster.service=DEBUG \
  | tee ~/opensearch-notes/dc1/run.out

Note: If the -Dtests.opensearch.logger.* form is not honored on your version, set it at runtime instead once the cluster is up:

curl -s -XPUT localhost:9200/_cluster/settings -H 'content-type: application/json' -d '{
  "persistent": {
    "logger.org.opensearch.cluster.coordination": "TRACE",
    "logger.org.opensearch.cluster.service": "DEBUG"
  }
}'

(You will only catch the first election this way if the setting is applied before a re-election; that's fine — you trigger a re-election yourself in Step 6.)

The gradlew run nodes write logs under build/testclusters/<name>/logs/. Find them:

find ~/src/OpenSearch -path "*testclusters*" -name "*.log" | sort
# tail one node's log live in another terminal:
tail -f $(find ~/src/OpenSearch -path "*testclusters*runTask-0*" -name "*.log" | head -1)

Step 3 — Confirm the cluster formed and find the cluster manager

curl -s 'localhost:9200/_cat/cluster_manager?v'
# id                     host       ip         node
# kf3...                 127.0.0.1  127.0.0.1  runTask-0

curl -s 'localhost:9200/_cat/nodes?v&h=name,node.role,cluster_manager,version'
# the node with a '*' under cluster_manager is the elected one

curl -s 'localhost:9200/_cluster/health?pretty' | grep -E 'status|number_of_nodes'

Record which node won the first election (e.g. runTask-0). Now look at the committed voting configuration — the set whose majority is quorum:

curl -s 'localhost:9200/_cluster/state/metadata?filter_path=metadata.cluster_coordination&pretty'
# {
#   "metadata" : { "cluster_coordination" : {
#       "term" : 1,
#       "last_committed_config" : [ "node-id-0", "node-id-1", "node-id-2" ],
#       "last_accepted_config"  : [ "node-id-0", "node-id-1", "node-id-2" ]
#   } } }

cat >> ~/opensearch-notes/dc1/election-log.md <<'EOF'
## First election result
- elected cluster manager: runTask-? (node id ...)
- term: 1
- committed voting config size: 3 → quorum = 2
EOF

Step 4 — Read the first election in the logs

In the cluster-manager node's log, find the boot→election→publish arc. The exact strings vary; these greps over the captured log find the signal:

L=$(find ~/src/OpenSearch -path "*testclusters*runTask-0*" -name "*.log" | head -1)

# (a) discovery: peers found
grep -nE "PeerFinder|found peers|setting initial configuration|cluster UUID" "$L" | head

# (b) election scheduled + pre-vote
grep -nE "starting election|election scheduler|pre.?vote|PreVote" "$L" | head

# (c) won election → became cluster manager (the INFO line)
grep -nE "cluster-manager node changed|master node changed|becomeLeader|elected-as-cluster-manager|elected-as-master" "$L" | head

# (d) first published state version
grep -nE "publishing|published cluster state|version \[1\]|term: ?1" "$L" | head

Map each to a class. A typical clean-election arc (your strings will differ slightly):

[INFO ][o.o.c.c.Coordinator        ] [runTask-0] setting initial configuration to VotingConfiguration{...}
[DEBUG][o.o.c.c.ElectionScheduler  ] [runTask-0] scheduleNextElection ... starting election
[TRACE][o.o.c.c.PreVoteCollector   ] [runTask-0] PreVoteResponse{...} from {runTask-1}
[INFO ][o.o.c.c.JoinHelper         ] [runTask-0] handleJoinRequest ... Join{term=1, ...}
[INFO ][o.o.c.s.ClusterManagerService      ] [runTask-0] cluster-manager node changed {previous [], current [{runTask-0}...]}, term: 1, version: 1
[INFO ][o.o.c.s.ClusterApplierService][runTask-1] cluster-manager node changed {previous [], current [{runTask-0}...]}

cat >> ~/opensearch-notes/dc1/election-log.md <<'EOF'
## First election — annotated log (paste your real lines)
1. Coordinator: setting initial configuration → ClusterBootstrapService seeded voting config
2. ElectionScheduler: scheduled election (randomized backoff)
3. PreVoteCollector: pre-vote responses → quorum plausible
4. JoinHelper: handleJoinRequest, Join{term=1}
5. CoordinationState.isElectionQuorum → true → Coordinator.becomeLeader
6. ClusterManagerService: "cluster-manager node changed" term:1 version:1 (first publish)
7. ClusterApplierService on followers: applied → they are FOLLOWERs
EOF

Step 5 — Prove the quorum/term facts from the live cluster

# term is 1 after one election; bump happens on the NEXT election
curl -s 'localhost:9200/_cluster/state/metadata?filter_path=metadata.cluster_coordination.term&pretty'

# the routing/version increments on every published state, not every election
curl -s 'localhost:9200/_cluster/state?filter_path=version,state_uuid&pretty'

Answer in your notes: with 3 eligible nodes, what is the quorum? How many can you lose and still elect? (2 / 1.) You are about to lose one.

Step 6 — Kill the elected cluster manager and watch re-election

Find the elected node's PID and kill it (simulating a crash):

# the gradlew run pids:
find ~/src/OpenSearch -path "*testclusters*" -name "*.pid" -exec sh -c 'echo "$1: $(cat "$1")"' _ {} \;

# kill the FIRST-elected node (e.g. runTask-0). Use the matching pid:
kill -9 <pid-of-runTask-0>

Immediately watch a surviving node's log:

L2=$(find ~/src/OpenSearch -path "*testclusters*runTask-1*" -name "*.log" | head -1)
grep -nE "LeaderChecker|leader \[.*\] failed|leader failed|becomeCandidate|starting election|cluster-manager node changed|master node changed|removed.*runTask-0|node-left" "$L2" | tail -30

You should see, in order:

[DEBUG][o.o.c.c.LeaderChecker      ] [runTask-1] leader [{runTask-0}...] failed, restarting discovery
[INFO ][o.o.c.c.Coordinator        ] [runTask-1] becoming candidate: onLeaderFailure
[DEBUG][o.o.c.c.ElectionScheduler  ] [runTask-1] starting election
[INFO ][o.o.c.s.ClusterManagerService      ] [runTask-1] cluster-manager node changed {previous [{runTask-0}...], current [{runTask-1}...]}, term: 2
[INFO ][o.o.c.s.ClusterManagerService      ] [runTask-1] node-left[{runTask-0}... reason: disconnected], term: 2, version: N

Map it:

Step 7 — Inspect the new term and reconfigured voting config

curl -s 'localhost:9200/_cat/cluster_manager?v'   # now a different node
curl -s 'localhost:9200/_cluster/state/metadata?filter_path=metadata.cluster_coordination&pretty'
# term is now 2; last_committed_config may have shrunk to the 2 survivors
# (Reconfigurator), depending on auto_shrink_voting_configuration.

Note whether the voting config shrank to 2 (then quorum would be 2 — meaning you can now lose zero more and keep a manager: the textbook even-node hazard).

cat >> ~/opensearch-notes/dc1/election-log.md <<'EOF'
## Re-election result
- new cluster manager: runTask-?  (term 2)
- LeaderChecker on survivors fired first → becomeCandidate
- voting config after Reconfigurator: [...]  (size ?, quorum ?)
- consequence: losing one more eligible node would now ...
EOF

Step 8 (Path B) — Deterministic re-run with InternalTestCluster

For a scriptable, deterministic version (no PIDs, no log files), drive it from a test. Skeleton:

// server/src/test/java/.../DC1ElectionTraceIT.java
package org.opensearch.cluster.coordination;

import org.opensearch.cluster.node.DiscoveryNode;
import org.opensearch.test.OpenSearchIntegTestCase;
import org.opensearch.test.OpenSearchIntegTestCase.ClusterScope;
import org.opensearch.test.OpenSearchIntegTestCase.Scope;

import static org.hamcrest.Matchers.equalTo;

@ClusterScope(scope = Scope.TEST, numDataNodes = 3, numClientNodes = 0)
public class DC1ElectionTraceIT extends OpenSearchIntegTestCase {

    public void testElectionThenReElection() throws Exception {
        ensureGreen();

        // who is the elected cluster manager right now?
        String cm = internalCluster().getMasterName();   // "master" = cluster manager
        logger.info("--> elected cluster manager: {}", cm);

        long termBefore = internalCluster()
            .clusterService()                 // on any node
            .state().term();
        logger.info("--> term before kill: {}", termBefore);

        // kill it and force a re-election
        internalCluster().stopCurrentMasterNode();

        // a NEW cluster manager must emerge among the survivors
        assertBusy(() -> {
            String newCm = internalCluster().getMasterName();
            assertNotNull(newCm);
            assertThat(newCm.equals(cm), equalTo(false));   // different node
        });

        long termAfter = internalCluster().clusterService().state().term();
        logger.info("--> term after re-election: {}", termAfter);
        assertThat(termAfter > termBefore, equalTo(true));  // term must bump
    }
}

Run it with coordination TRACE:

./gradlew :server:internalClusterTest \
  --tests "org.opensearch.cluster.coordination.DC1ElectionTraceIT" \
  -Dtests.logger.org.opensearch.cluster.coordination=TRACE

This asserts the two facts you observed by hand: a different node becomes cluster manager, and the term strictly increases.

Deliverables

• ~/opensearch-notes/dc1/election-log.md with: the class→log-string map, the annotated first election arc, the re-election arc, and both voting configs (before/after) with their quorum sizes.
• The output of _cat/cluster_manager before and after the kill (different node).
• The term before (1) and after (2) the re-election from _cluster/state/metadata.
• (Path B) A green run of DC1ElectionTraceIT.

Troubleshooting

Expected Output

A clean 3-node bring-up, _cat/cluster_manager showing one elected node, a coordination TRACE arc that goes pre-vote → Join → becomeLeader → first publish at term: 1. After kill -9 of the manager: LeaderChecker failure on survivors → becomeCandidate → new election → a different _cat/cluster_manager, term: 2, and a node-left for the dead node. The voting config either stays size-3 or shrinks toward the survivors via Reconfigurator.

Stretch Goals

• Kill a follower instead and confirm: no new election, no term bump, just a node-left and (if a replica lived there) shard reallocation. Contrast with killing the manager.
• Set logger.org.opensearch.cluster.coordination.CoordinationState=TRACE and find the exact handleJoin / isElectionQuorum lines that prove the quorum was reached.
• Bring up 2 eligible nodes instead of 3; kill one and watch the cluster halt (no quorum, ClusterFormationFailureHelper complaining). This is the even-node hazard from the intensive, live.
• Grep ElectionSchedulerFactory for the ELECTION_INITIAL_TIMEOUT / ELECTION_BACK_OFF / ELECTION_MAX_TIMEOUT settings and explain how the randomized backoff prevents dueling candidates.

Coding Exercises

Tracing is not enough — you must turn what you watched into code that asserts it. Each exercise below produces a test, an instrumentation patch, or a small variant. Locate every class with rg first (rg --files-with-matches "class FollowersChecker" server/src); never trust a line number you didn't just print.

1. (warm-up) Assert the term bump in a CoordinatorTests unit test. The coordination package ships org.opensearch.cluster.coordination.CoordinatorTests (find it: rg --files -g 'CoordinatorTests.java' server/src/test). Add a test method that builds a small Cluster, stabilises it, records the leader's term, stops the leader, re-stabilises, and asserts the new term is strictly greater and the leader is a different node. Reuse the existing cluster.stabilise(...) and cluster.getAnyLeader() helpers in that file rather than InternalTestCluster — this is the fast, deterministic harness the maintainers use. Verify: ./gradlew :server:test --tests "org.opensearch.cluster.coordination.CoordinatorTests".

2. (core) Write a FollowersChecker/LeaderChecker reaction test. In Step 6 you watched LeaderChecker fire first on survivors. Prove it with code: locate FollowersCheckerTests/LeaderCheckerTests (rg -l "class LeaderCheckerTests" server/src/test) and add a case that drives a LeaderChecker against a MockTransport, makes the leader's checks fail leader_check.retry_count + 1 times, and asserts the supplied onLeaderFailure callback ran exactly once. Read the existing tests in that file for the transport-failure scaffolding before you write yours.

3. (core) Assert quorum arithmetic in CoordinationState. Find CoordinationStateTests (rg -l "class CoordinationStateTests"). Add a test that builds a VotingConfiguration of three node ids, feeds handleJoin exactly two Joins for the same term, and asserts isElectionQuorum(...) flips to true at the second join and not the first. This encodes "quorum of 3 is 2" as an executable fact — the thing you computed by hand in Step 5.

4. (core) Instrument the first publish and capture it. Add a temporary logger.info in Coordinator.becomeLeader (locate with rg -n "becomeLeader" server/src/main/java/org/opensearch/cluster/coordination/Coordinator.java) that prints the term and the size of the committed voting config. Then write a MockLogAppender-based assertion (grep the codebase for MockLogAppender usages in coordination tests as a template) that captures exactly that line during an election in CoordinatorTests. Remove the logger.info afterward — the test that proves it fired is the deliverable, not the print.

5. (advanced) Implement and test a one-line Reconfigurator variant. Read Reconfigurator (rg -n "auto_shrink_voting_configuration|class Reconfigurator" server/src/main) and ReconfiguratorTests. Advanced challenge: add a JUnit test that, with auto_shrink_voting_configuration enabled, starts 5 cluster-manager-eligible nodes, removes one, and asserts the committed config shrinks to an odd size (3), never an even one — the protocol's even-node-avoidance rule. Then make the rule fail on purpose by reading how reconfigure rounds, predict which assertion would catch a regression that let it settle on 4, and add that assertion. Run: ./gradlew :server:test --tests "org.opensearch.cluster.coordination.ReconfiguratorTests".

Issues to Practice On

Coordination issues are high-stakes and well-labelled on opensearch-project/OpenSearch. Start here (labels move; confirm on the tracker):

gh issue list --repo opensearch-project/OpenSearch --label "good first issue" --state open
gh issue list --repo opensearch-project/OpenSearch --label "Cluster Manager" --state open
gh issue list --repo opensearch-project/OpenSearch --label "flaky-test"      --state open
# Discover the exact area labels first — taxonomies drift:
gh label list --repo opensearch-project/OpenSearch | grep -iE "cluster|coordination|distributed"

Representative patterns. (1) A flaky CoordinatorTests/...IT case in the coordination package: reproduce with ./gradlew ... -Dtests.iters=50, locate the racing assertion via rg, and fix timing with assertBusy/stabilise rather than Thread.sleep. (2) A "cluster won't form with config X" report: reproduce against a small InternalTestCluster, trace the failure through ClusterFormationFailureHelper, and add a regression test. Both follow the same arc: reproduce → locate via rg → fix → test → PR with a CHANGELOG entry and DCO sign-off.

Planted-bug drill. In CoordinationState.isElectionQuorum (locate it with rg -n "isElectionQuorum" server/src/main), change the majority check from votes.isQuorum(...) to accept one fewer vote (e.g. weaken the comparison). Run ./gradlew :server:test --tests "org.opensearch.cluster.coordination.CoordinationStateTests" and watch which test goes red — that test is the safety net for split-brain. Revert, then add an assertion in your Exercise 3 test that pins "two of three is the minimum" so any future loosening fails loudly.

Etiquette: claim an issue with a comment before working it, reproduce the bug first, and remember every PR needs a test, a CHANGELOG.md entry, and a DCO Signed-off-by (git commit -s). See community-interaction and the code-reading companion Level 4 Lab 4.1.

Validation / Self-check

Answer from your annotated log, citing the class for each:

1. Name the four classes that fire, in order, to win the first election. Which one prints the cluster-manager node changed line?
2. What was the term after the first election, and after the re-election? Which field in _cluster/state/metadata did you read it from, and why does it bump on election but version bump on every publish?
3. When you killed the manager, which detector fired first on the survivors, and what mode transition did it cause? (Cite the log line and the class.)
4. From the committed voting config before and after the kill, compute the quorum in each case. If Reconfigurator shrank it to 2, what is the new availability risk?
5. In Path B, which two assertions encode "an election really happened"? Why is "the cluster-manager name changed" not sufficient on its own without the term check?
6. Contrast killing the manager vs killing a follower (Stretch). Which one bumps the term, and why does the other not?

When you can replay the election as a sequence of log lines mapped to classes — and explain why the term bumped — you've completed Lab DC1. Continue to Lab DC2: Failure Detection and Network Partitions.