Lab DC2: Failure Detection and Network Partitions

Background

Lab DC1 made you watch an election by killing a node — a clean crash. Real distributed bugs are nastier: the node is alive but unreachable, the network is half-broken, packets are delayed but not dropped, or two halves of the cluster can each talk internally but not across. These are network partitions, and they are exactly where consensus protocols earn their keep — or break.

OpenSearch's test framework ships first-class disruption tooling so you can induce these conditions deterministically in a single JVM, then assert what the Coordinator, LeaderChecker, and FollowersChecker actually do. In this lab you will:

1. write an OpenSearchIntegTestCase that brings up 3 cluster-manager-eligible nodes;
2. install a NetworkDisruption (TwoPartitions + NetworkDisconnect) that isolates the elected cluster manager (formerly master) into the minority;
3. observe LeaderChecker/FollowersChecker reactions, the minority side stepping down, and the majority electing a new manager;
4. heal the partition and watch the old manager rejoin as a follower at the higher term;
5. understand why quorum makes two cluster managers impossible, and where the LinearizabilityChecker would catch a violation if one slipped through.

Note: This is the framework that the OpenSearch maintainers use to test coordination. The names (NetworkDisruption, MockTransportService, ServiceDisruptionScheme) are stable; APIs evolve. Grep to confirm the exact signatures on your branch — every step shows you how.

Why This Matters for Contributors

Almost every "split-brain"/"data loss after partition"/"node won't rejoin" issue is reproduced and regression-tested with this exact tooling. A PR that touches coordination, fault detection, or replication is expected to ship a disruption test. If you cannot write one, you cannot prove your fix or guard against the next regression. This lab is that skill, on the real test harness, against the real classes.

Prerequisites

• OpenSearch builds; ./gradlew :server:internalClusterTest runs (Level 3+).
• You read the intensive partition section and Lab DC1.
• Notes dir:

mkdir -p ~/opensearch-notes/dc2

Find the disruption toolbox first

cd ~/src/OpenSearch
find test -name "NetworkDisruption.java" -o -name "ServiceDisruptionScheme.java" \
     -o -name "MockTransportService.java" -o -name "LinearizabilityChecker.java"

# The disruption types you'll use:
grep -n "class NetworkDisconnect\|class NetworkDelay\|class NetworkLinkDisruptionType\|class Bridge\|class TwoPartitions\|class DisruptedLinks" \
  test/framework/src/main/java/org/opensearch/test/disruption/NetworkDisruption.java | head -30

# How a test installs a disruption:
grep -rn "setDisruptionScheme\|startDisrupting\|stopDisrupting\|ensureFullyConnected" \
  test/framework/src/main/java/org/opensearch/test/ | head

The toolbox:

Step-by-Step Tasks

Step 1 — A test that needs MockTransportService

NetworkDisruption works by intercepting transport at MockTransportService, so the test must request the mock transport. OpenSearchIntegTestCase does this when you add the right nodePlugins()/transport. Skeleton:

// server/src/test/java/.../DC2PartitionIT.java
package org.opensearch.cluster.coordination;

import org.opensearch.cluster.node.DiscoveryNodes;
import org.opensearch.plugins.Plugin;
import org.opensearch.test.OpenSearchIntegTestCase;
import org.opensearch.test.OpenSearchIntegTestCase.ClusterScope;
import org.opensearch.test.OpenSearchIntegTestCase.Scope;
import org.opensearch.test.disruption.NetworkDisruption;
import org.opensearch.test.disruption.NetworkDisruption.NetworkDisconnect;
import org.opensearch.test.disruption.NetworkDisruption.TwoPartitions;
import org.opensearch.test.transport.MockTransportService;

import java.util.Collection;
import java.util.Collections;
import java.util.HashSet;
import java.util.List;
import java.util.Set;

import static org.hamcrest.Matchers.equalTo;
import static org.hamcrest.Matchers.not;

// 3 cluster-manager-eligible nodes; quorum = 2.
@ClusterScope(scope = Scope.TEST, numDataNodes = 3, numClientNodes = 0)
public class DC2PartitionIT extends OpenSearchIntegTestCase {

    @Override
    protected Collection<Class<? extends Plugin>> nodePlugins() {
        // MockTransportService is required for NetworkDisruption to intercept sends.
        return Collections.singletonList(MockTransportService.TestPlugin.class);
    }

    @Override
    protected boolean addMockTransportService() {
        return true;
    }
    // ... tests below
}

Confirm the plugin/method names on your branch — they drift:

grep -rn "MockTransportService.TestPlugin\|addMockTransportService\|getMockPlugins" \
  test/framework/src/main/java/org/opensearch/test/OpenSearchIntegTestCase.java | head

Step 2 — Isolate the cluster manager into the minority

Build a TwoPartitions with the elected manager alone on one side and the other two together on the majority side, with NetworkDisconnect (a hard partition):

    public void testMinorityManagerStepsDownAndMajorityReElects() throws Exception {
        ensureGreen();

        final String originalCm = internalCluster().getMasterName(); // "master" = cluster manager
        final long termBefore = internalCluster()
            .getInstance(org.opensearch.cluster.service.ClusterService.class)
            .state().term();
        logger.info("--> original cluster manager [{}], term [{}]", originalCm, termBefore);

        // majority = the two nodes that are NOT the current manager
        final Set<String> majority = new HashSet<>(List.of(internalCluster().getNodeNames()));
        majority.remove(originalCm);

        // minority = just the current manager
        final Set<String> minority = Collections.singleton(originalCm);

        final TwoPartitions partitions = new TwoPartitions(minority, majority);
        final NetworkDisruption scheme =
            new NetworkDisruption(partitions, new NetworkDisconnect());

        internalCluster().setDisruptionScheme(scheme);
        logger.info("--> partitioning: minority {} | majority {}", minority, majority);
        scheme.startDisrupting();

        // The majority (2 of 3 = quorum) must elect a NEW manager, with a higher term.
        assertBusy(() -> {
            // ask a node on the MAJORITY side for its view
            String majorityNode = majority.iterator().next();
            String cmSeenByMajority = internalCluster().getInstance(
                    org.opensearch.cluster.service.ClusterService.class, majorityNode)
                .state().nodes().getMasterNodeId();
            assertNotNull("majority should have a cluster manager", cmSeenByMajority);

            long termOnMajority = internalCluster().getInstance(
                    org.opensearch.cluster.service.ClusterService.class, majorityNode)
                .state().term();
            assertThat("term must have bumped", termOnMajority > termBefore, equalTo(true));
        }, 30, java.util.concurrent.TimeUnit.SECONDS);

        // The MINORITY (the old manager, alone) must NOT still consider itself manager.
        assertBusy(() -> {
            DiscoveryNodes nodesOnMinority = internalCluster().getInstance(
                    org.opensearch.cluster.service.ClusterService.class, originalCm)
                .state().nodes();
            // it cannot reach quorum, so it must step down to CANDIDATE: no local master
            assertThat("minority must step down (no cluster manager locally)",
                nodesOnMinority.getMasterNodeId(), equalTo((String) null));
        }, 30, java.util.concurrent.TimeUnit.SECONDS);

        // ... heal in Step 4
    }

The two assertions encode the whole guarantee: the majority side (quorum = 2) does elect a new manager and bumps the term; the minority side (1 node, below quorum) steps down and has no local manager. There is never a moment when both sides claim a manager.

Step 3 — Watch the detectors react (logging)

Run the test with coordination + fault-detection TRACE to see which detector fired:

./gradlew :server:internalClusterTest \
  --tests "org.opensearch.cluster.coordination.DC2PartitionIT" \
  -Dtests.logger.org.opensearch.cluster.coordination=TRACE \
  -Dtests.logger.org.opensearch.cluster.coordination.LeaderChecker=TRACE \
  -Dtests.logger.org.opensearch.cluster.coordination.FollowersChecker=TRACE \
  2>&1 | tee ~/opensearch-notes/dc2/run.out

What you should see in the captured output, mapped to classes:

grep -nE "leader \[.*\] failed|becoming candidate|cluster-manager node changed|master node changed|cannot reach|not enough|quorum|FollowersChecker|LeaderChecker" \
  ~/opensearch-notes/dc2/run.out | head -40

Step 4 — Heal the partition; old manager rejoins as follower

Append to the test:

        // --- HEAL ---
        scheme.stopDisrupting();
        internalCluster().clearDisruptionScheme();
        ensureStableCluster(3);          // all 3 reconnect into one cluster
        ensureGreen();

        // Everyone now agrees on ONE manager (the one the majority elected),
        // and the old manager is NOT it anymore.
        String finalCm = internalCluster().getMasterName();
        assertThat("old manager must not regain leadership automatically",
            finalCm, not(equalTo(originalCm)));

        // The old manager rejoined as a FOLLOWER at the higher term.
        long finalTerm = internalCluster()
            .getInstance(org.opensearch.cluster.service.ClusterService.class)
            .state().term();
        assertThat(finalTerm > termBefore, equalTo(true));
        logger.info("--> healed. final cluster manager [{}], term [{}]", finalCm, finalTerm);

The healed cluster has exactly one manager, the old manager is a follower, and the term reflects the re-election. The old manager rejoined precisely because it saw a higher term and accepted the new leader's publication (term fencing, from the intensive).

Step 5 — Prove a NetworkDelay (flaky, not dropped) is different from a disconnect

A hard disconnect is unambiguous; a delay is the subtler, more realistic case — it can cause spurious failovers if fault-detection timeouts are too tight. Add a second test that delays cross-links and tunes the detector:

    public void testDelayCanCauseSpuriousFailoverIfTimeoutsTooTight() throws Exception {
        // Tighten fault detection so a delay LOOKS like a failure.
        internalCluster().client().admin().cluster().prepareUpdateSettings()
            .setPersistentSettings(org.opensearch.common.settings.Settings.builder()
                .put("cluster.fault_detection.leader_check.timeout", "1s")
                .put("cluster.fault_detection.leader_check.retry_count", 1)
                .build())
            .get();
        ensureGreen();

        String cm = internalCluster().getMasterName();
        Set<String> majority = new HashSet<>(List.of(internalCluster().getNodeNames()));
        majority.remove(cm);

        // Delay (don't drop) the manager's links by more than the check timeout.
        NetworkDisruption delay = new NetworkDisruption(
            new TwoPartitions(Collections.singleton(cm), majority),
            new NetworkDisruption.NetworkDelay(java.util.concurrent.TimeUnit.SECONDS.toMillis(3)));
        internalCluster().setDisruptionScheme(delay);
        delay.startDisrupting();

        // With 1s timeout / 1 retry, the delayed manager looks dead → failover.
        assertBusy(() -> assertThat(
            internalCluster().getInstance(
                    org.opensearch.cluster.service.ClusterService.class, majority.iterator().next())
                .state().term() > 1, equalTo(true)));

        delay.stopDisrupting();
        internalCluster().clearDisruptionScheme();
        ensureStableCluster(3);
    }

The lesson: with aggressive timeouts, a transient delay (or a GC pause!) causes a needless re-election. This is the real-world "frequent leader re-elections under load" symptom from the intensive — induced on purpose.

Step 6 — Where the LinearizabilityChecker fits

NetworkDisruption lets you induce the partition; the LinearizabilityChecker lets you verify that the resulting history of operations is still linearizable — i.e. the consensus never let two managers commit conflicting states. Read its shape:

grep -n "class LinearizabilityChecker\|interface SequentialSpec\|History\|isLinearizable\|class Event\|invoke\|response" \
  test/framework/src/main/java/org/opensearch/test/disruption/LinearizabilityChecker.java | head -30
# Real users in core: the coordination/cluster tests that record a history then assert linearizable.
grep -rln "LinearizabilityChecker\|isLinearizable" server/src/test test/framework | head

You feed it a History of (invoke, response) events and a SequentialSpec (the expected sequential semantics — e.g. a register with read/write); it searches for some sequential ordering consistent with the observed concurrency. If none exists, the history is not linearizable — a real safety bug. You do not write one in this lab, but you must know it exists and what it asserts: it is the oracle that turns "we induced a partition and nothing looked broken" into "we proved no two managers forked the state."

Step 7 — Why quorum makes two managers impossible (write the argument)

In ~/opensearch-notes/dc2/answer.md, write the proof in your own words, anchored to what you observed:

cat >> ~/opensearch-notes/dc2/answer.md <<'EOF'
# Why two cluster managers cannot both commit
- 3 eligible nodes, quorum = 2 (strict majority).
- To become leader of term T, a candidate needs a quorum of Joins for term T.
- A node votes at most once per term.
- Two strict majorities of the SAME set of 3 always share >= 1 node (pigeonhole).
- Therefore two leaders elected from this config cannot have the same term;
  the higher term wins, and the overlap node would have refused the lower-term vote.
- In my test: minority {cm} has size 1 < quorum 2 -> can never elect or commit;
  majority {a,b} has size 2 = quorum -> elects new manager at higher term.
- Observed: minority's getMasterNodeId() == null (stepped down);
  majority's term bumped and it has a manager. No overlap in time where both lead.
EOF

Deliverables

• DC2PartitionIT with the minority-step-down + majority-re-elect test, green.
• The TRACE run output showing which detector fired on each side (LeaderChecker on the majority, FollowersChecker/formation-failure on the minority).
• The heal assertions (one manager, old manager is a follower, term bumped).
• The NetworkDelay spurious-failover test (green), demonstrating tight-timeout sensitivity.
• answer.md with the quorum-overlap argument grounded in your observations.

Troubleshooting

Expected Output

The hard-partition test goes green: the majority elects a new manager at a higher term while the isolated old manager reports no local manager; on heal, the cluster reconverges to one manager (not the old one) at the bumped term. The TRACE log shows LeaderChecker failing on the majority and FollowersChecker/formation-failure on the minority. The delay test shows that tight timeouts turn a mere delay into a failover.

Stretch Goals

• Swap TwoPartitions for NetworkDisruption.Bridge: a bridge node both sides can reach but the two sides cannot reach each other. Predict, then observe, which side keeps quorum.
• Index documents during the partition into the majority side, heal, and verify with _cluster/health and a search that the writes survived and the minority caught up. (Touches the data plane — link replication.)
• Run an existing core disruption test to see the harness at production scale: grep -rl "NetworkDisruption" server/src/test | head then run one with ./gradlew :server:internalClusterTest --tests "...DisruptionIT".
• Sketch (or read) a LinearizabilityChecker usage: a register SequentialSpec, a History of concurrent reads/writes under a partition, and the isLinearizable assertion.

Coding Exercises

You already wrote DC2PartitionIT; these exercises harden it into a graded partition-test suite and push you into the framework the maintainers actually use. Confirm every signature with rg first (rg -n "void startDisrupting" test/framework).

1. (warm-up) Make the minority step-down assertion airtight. Your Step 2 test asserts getMasterNodeId() == null on the isolated manager. Strengthen it: add a second assertion that the minority node's Coordinator.getMode() is CANDIDATE (locate the enum with rg -n "enum Mode|getMode" server/src/main/java/org/opensearch/cluster/coordination/Coordinator.java). Reach the coordinator via internalCluster().getInstance(Coordinator.class, originalCm). Verify the whole test still goes green under :server:internalClusterTest.

2. (core) Add a Bridge partition test (Stretch goal, graded). Replace TwoPartitions with NetworkDisruption.Bridge (confirm its constructor with rg -n "class Bridge" test/framework/src/main/java/org/opensearch/test/disruption/NetworkDisruption.java): a bridge node both sides reach but the two wings cannot reach each other. Write a test that predicts in a comment, then asserts, which side retains a cluster manager (the side whose nodes — including the bridge — total a quorum) and that the other side has none. This is the canonical "is the bridge node decisive?" scenario.

3. (core) Parameterise the spurious-failover test. Turn Step 5's testDelayCanCauseSpuriousFailoverIfTimeoutsTooTight into a table-driven test: loop over delay/timeout pairs and assert that failover happens iff the delay exceeds leader_check.timeout * (retry_count + 1). This makes the timeout/retry arithmetic an executable contract instead of a single anecdotal case. Read the default values from rg -n "LEADER_CHECK_TIMEOUT_SETTING|LEADER_CHECK_RETRY_COUNT_SETTING" server/src/main/java/org/opensearch/cluster/coordination/LeaderChecker.java.

4. (core) Assert the data plane survived the partition (Stretch goal, graded). Extend the hard-partition test: index N documents into the majority side during the partition, heal, ensureGreen(), then assert a search on every node returns all N and that the formerly-isolated old manager (now a follower) serves the same count. This proves writes accepted by the quorum survived and the minority caught up via the higher-term publication. Link replication.

5. (advanced) A ClusterStatePublisher assertion under churn. Advanced challenge: instrument the publish path to prove that no state committed on the majority during the partition was ever applied on the minority. Find the publisher seam (rg -n "ClusterStatePublisher|interface ClusterStatePublisher" server/src/main/java/org/opensearch/cluster/coordination/) and, in your IT, capture the highest applied cluster-state version() on the isolated node before healing; assert it is strictly less than the majority's version at heal time, then equal after heal. Pair this with a MockLogAppender (grep coordination tests for the pattern) asserting the minority logged ClusterFormationFailureHelper "not enough nodes" while partitioned. The deliverable is one IT that demonstrates fencing end to end: minority stalls, majority advances, heal reconciles to the higher term.

Issues to Practice On

Partition and fault-detection bugs are the scariest entries on opensearch-project/OpenSearch, and most ship a NetworkDisruption regression test. Start here (labels move; confirm on the tracker):

gh issue list --repo opensearch-project/OpenSearch --label "Cluster Manager" --state open
gh issue list --repo opensearch-project/OpenSearch --label "flaky-test"      --state open
gh issue list --repo opensearch-project/OpenSearch --label "bug"             --state open --search "partition OR disruption OR FollowersChecker"
gh label list --repo opensearch-project/OpenSearch | grep -iE "cluster|coordination|distributed|flaky"

Representative patterns. (1) A flaky disruption IT — passes locally, fails in CI under timing: reproduce with -Dtests.iters=100, find the racy wait via rg "ensureStableCluster|assertBusy", and replace fixed timing with framework helpers. (2) A "node won't rejoin after partition heal" report: reproduce with a TwoPartitions + heal IT, trace the rejoin through JoinHelper/term fencing, fix, and ship the IT as the regression guard. Arc for both: reproduce → locate via rg → fix → test → PR with CHANGELOG + DCO.

Planted-bug drill. Find FollowersChecker's failure threshold (rg -n "FOLLOWER_CHECK_RETRY_COUNT_SETTING|retryCount|setFailureCountSupplier" server/src/main/java/org/opensearch/cluster/coordination/FollowersChecker.java) and change the retry-count comparison so it fails over after one missed check instead of the configured count. Run the coordination ITs / FollowersCheckerTests and watch which test flips. Revert, then add an assertion in your Exercise 3 test that pins the exact "delay must exceed timeout * (retries + 1)" boundary so this loosening would be caught.

Etiquette: claim the issue first, reproduce before you theorise, and every PR carries a disruption test + CHANGELOG.md entry + DCO Signed-off-by (git commit -s). See community-interaction.

Validation / Self-check

Cite a class or a log line for each:

1. What does MockTransportService provide that makes NetworkDisruption possible at all? Why can't you partition a normal TransportService?
2. In your hard-partition test, which detector fired on the majority side and which on the minority side? What mode transition did each cause?
3. State, with the quorum arithmetic for 3 eligible nodes, why the minority of 1 could neither stay manager nor elect a new one.
4. On heal, why did the old manager become a follower rather than reclaim leadership? Which fencing mechanism forced that?
5. In the delay test, you tightened cluster.fault_detection.leader_check.*. Explain how a non-dropping delay (or a GC pause) then produces a spurious failover, and what you'd change in production to avoid it.
6. What does the LinearizabilityChecker assert, and what would a failed check mean for the consensus protocol? Why is "we induced a partition and saw no obvious breakage" weaker than a linearizability check?

When you can induce a partition, predict which side keeps quorum, point to the detector that fires on each side, and explain why two managers are mathematically impossible — you've completed Lab DC2. Continue to Lab DC3: Cluster State Publish/Commit.