Lab 5.3: Build It — A Multi-Node Integration Test

Background

Unit tests prove logic; integration tests prove distributed behavior. The behavior that breaks in production — a primary dying, a replica getting promoted, shards reallocating, the cluster recovering to green — only shows up when you have multiple nodes and you take one away. This is what OpenSearchIntegTestCase and its in-JVM InternalTestCluster exist for.

In this Build-It lab you write a complete failover integration test from scratch: stand up a three-node cluster, index documents into an index with one replica, stop a random data node, and assert that OpenSearch promotes the replica and reallocates shards so the cluster returns to green with no data loss. You will do this deterministically — no Thread.sleep, only ensureGreen and assertBusy.

Note: This builds on Lab 5.1, which got you a multi-node cluster indexing data. Here we add the hard part: surviving a node death. The cluster manager (formerly master) is what drives replica promotion and reallocation under the hood — see the replication and shard allocation deep dives for the machinery you are testing.

Why This Lab Matters for Contributors

Failover/recovery is where the hardest OpenSearch bugs live. You cannot credibly fix one without a test that reproduces the failure scenario — and that test is exactly this shape.
Maintainers expect distributed changes to come with an *IT that runs under internalClusterTest. Knowing the idioms (internalCluster(), ensureGreen, assertBusy, disruption helpers) makes your PRs reviewable.
This is the difference between "it worked on my laptop" and "it survives a node going away under load" — the bar real clusters are held to.
Writing it teaches you determinism: why asserting on async state without polling produces flaky CI, the topic you will hunt in Lab 5.4.

Prerequisites

Lab 5.1 completed; you can run an *IT under internalClusterTest.
A clean build: ./gradlew :server:compileInternalClusterTestJava -q.
~6 GB free RAM (three in-JVM nodes plus the test runner).

# Confirm the integ-test source set and task exist.
ls server/src/internalClusterTest/java/org/opensearch | head
./gradlew :server:tasks --all | grep internalClusterTest

Step-by-Step Tasks

Step 1: Decide the topology and what "correct" means

You are testing the simplest non-trivial failover:

3 data nodes, 1 index, 1 primary + 1 replica per shard (number_of_replicas = 1)
        ┌─ node A: primary p0
        ├─ node B: replica r0   (eligible to be promoted)
        └─ node C: (spare capacity for reallocation)

Kill the node holding a copy → cluster must:
  1. promote the surviving copy to primary (if the primary died), AND
  2. allocate a NEW replica onto a remaining node, AND
  3. return to GREEN with the same document count (no data loss).

The invariants your test asserts:

Invariant	How you assert it
Cluster recovers to green	`ensureGreen(index)` after the node stop
No documents lost	search count equals what you indexed, via `assertBusy`
A replica was promoted / reallocated	health shows `active_shards == primaries + replicas`; `unassigned == 0`
It happened automatically	no manual reroute in the test

Step 2: Create the IT file (correct source set, correct suffix)

Integration tests are *IT.java under src/internalClusterTest — not src/test. Put it in the wrong place and Gradle silently never runs it.

mkdir -p server/src/internalClusterTest/java/org/opensearch/cluster/routing
$EDITOR server/src/internalClusterTest/java/org/opensearch/cluster/routing/ReplicaPromotionFailoverIT.java

Step 3: Write the test

Here is the complete, runnable test. Read every line — the comments explain why each call is there.

/*
 * SPDX-License-Identifier: Apache-2.0
 *
 * The OpenSearch Contributors require contributions made to
 * this file be licensed under the Apache-2.0 license or a
 * compatible open source license.
 */

package org.opensearch.cluster.routing;

import org.opensearch.action.admin.cluster.health.ClusterHealthResponse;
import org.opensearch.action.index.IndexRequestBuilder;
import org.opensearch.cluster.health.ClusterHealthStatus;
import org.opensearch.common.settings.Settings;
import org.opensearch.index.query.QueryBuilders;
import org.opensearch.test.OpenSearchIntegTestCase;
import org.opensearch.test.OpenSearchIntegTestCase.ClusterScope;
import org.opensearch.test.OpenSearchIntegTestCase.Scope;

import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.TimeUnit;

import static org.opensearch.index.query.QueryBuilders.matchAllQuery;
import static org.hamcrest.Matchers.equalTo;

/**
 * Stops a data node and verifies OpenSearch promotes a replica and reallocates shards
 * back to GREEN with no data loss. A failover test — Scope.TEST so node death never
 * leaks into another method's cluster.
 */
@ClusterScope(scope = Scope.TEST, numDataNodes = 0)   // we start nodes ourselves, explicitly
public class ReplicaPromotionFailoverIT extends OpenSearchIntegTestCase {

    private static final String INDEX = "failover-idx";

    public void testReplicaIsPromotedAndClusterRecoversAfterDataNodeStop() throws Exception {
        // 1) A dedicated cluster manager + three data nodes. Dedicated CM keeps election
        //    independent of the data node we are about to kill.
        internalCluster().startClusterManagerOnlyNode();
        final List<String> dataNodes = new ArrayList<>(internalCluster().startDataOnlyNodes(3));
        assertEquals(3, dataNodes.size());

        // 2) An index with 2 primaries and 1 replica each → 4 shard copies across 3 data nodes.
        createIndex(
            INDEX,
            Settings.builder()
                .put("index.number_of_shards", 2)
                .put("index.number_of_replicas", 1)
                .build()
        );
        ensureGreen(INDEX);   // block until all 4 copies are STARTED before we touch anything

        // 3) Index a known number of docs and make them durable+visible.
        final int docCount = scaledRandomIntBetween(20, 100);   // randomized, but bounded
        indexDocs(docCount);
        refresh(INDEX);
        assertHitCount(docCount);   // sanity: all docs are searchable before the failure

        // 4) Inject the failure: stop a random DATA node (could be holding a primary or a replica).
        internalCluster().stopRandomDataNode();

        // 5) The cluster must self-heal. ensureGreen blocks until promotion + reallocation finish.
        //    We allow a generous timeout because reallocation copies a shard.
        ensureGreen(TimeValueMinutes(1), INDEX);

        // 6) Health invariants: no unassigned shards, all copies active again.
        ClusterHealthResponse health = client().admin().cluster()
            .prepareHealth(INDEX)
            .setWaitForGreenStatus()
            .get();
        assertThat(health.getStatus(), equalTo(ClusterHealthStatus.GREEN));
        assertThat(health.getUnassignedShards(), equalTo(0));
        assertThat(health.getActiveShards(), equalTo(4));   // 2 primaries + 2 replicas

        // 7) No data loss. Search is async w.r.t. shard relocation, so poll with assertBusy.
        assertBusy(() -> assertHitCount(docCount), 30, TimeUnit.SECONDS);
    }

    // ----- helpers -----

    private void indexDocs(int count) throws InterruptedException {
        final List<IndexRequestBuilder> builders = new ArrayList<>(count);
        for (int i = 0; i < count; i++) {
            builders.add(
                client().prepareIndex(INDEX)
                    .setId(Integer.toString(i))
                    .setSource("field", randomAlphaOfLengthBetween(1, 20), "n", i)
            );
        }
        // indexRandom shuffles, batches, and refreshes for you — preferred over a manual loop.
        indexRandom(true, builders);
    }

    private void assertHitCount(long expected) {
        long actual = client().prepareSearch(INDEX)
            .setQuery(matchAllQuery())
            .setSize(0)
            .get()
            .getHits()
            .getTotalHits()
            .value();
        assertThat(actual, equalTo(expected));
    }

    private static org.opensearch.common.unit.TimeValue TimeValueMinutes(long m) {
        return org.opensearch.common.unit.TimeValue.timeValueMinutes(m);
    }
}

Note: indexRandom(true, builders) is the idiomatic way to index in integ tests: it randomizes order and batching (finding ordering bugs) and refreshes at the end so docs are searchable. scaledRandomIntBetween scales the upper bound down under nightly/CI pressure so the test stays fast.

Step 4: Run it under the integration test task

# The whole class
./gradlew :server:internalClusterTest --tests "org.opensearch.cluster.routing.ReplicaPromotionFailoverIT"

# One method, verbose
./gradlew :server:internalClusterTest \
  --tests "*ReplicaPromotionFailoverIT.testReplicaIsPromotedAndClusterRecoversAfterDataNodeStop" -i

It is slower than a unit test (it builds and tears down a real cluster), typically 20–90 seconds.

Step 5: Prove it's deterministic — hammer it

A failover test that passes once but flakes under load is worse than none. Run it many times with different seeds to flush out timing assumptions:

./gradlew :server:internalClusterTest --tests "*ReplicaPromotionFailoverIT" -Dtests.iters=50

If any iteration fails, copy the REPRODUCE WITH ... -Dtests.seed=... line and debug that exact seed. A flaky failure here almost always means you asserted on async state without polling — replace a direct assertion with assertBusy or add the missing ensureGreen.

Step 6 (optional): Use a disruption helper instead of a hard stop

Stopping a node is the simplest failure. To test a network partition (the node is alive but unreachable — a different and nastier failure mode), use the disruption framework. Override the test plugins to mock the transport, then apply a partition:

import org.opensearch.plugins.Plugin;
import org.opensearch.test.disruption.NetworkDisruption;
import org.opensearch.test.transport.MockTransportService;

import java.util.Collection;
import java.util.Collections;

@Override
protected Collection<Class<? extends Plugin>> nodePlugins() {
    return Collections.singletonList(MockTransportService.TestPlugin.class);
}

public void testPartitionThenHeal() throws Exception {
    internalCluster().startClusterManagerOnlyNode();
    internalCluster().startDataOnlyNodes(3);
    createIndex(INDEX, Settings.builder()
        .put("index.number_of_shards", 1).put("index.number_of_replicas", 1).build());
    ensureGreen(INDEX);

    // Isolate one node from the rest, hold it, then heal and require recovery.
    String victim = internalCluster().getRandomNodeName();   // pick a node to cut off
    NetworkDisruption partition = new NetworkDisruption(
        new NetworkDisruption.TwoPartitions(
            Collections.singleton(victim), otherNodes(victim)),
        NetworkDisruption.DISCONNECT);
    internalCluster().setDisruptionScheme(partition);
    partition.startDisrupting();

    // While partitioned, the cluster manager evicts the isolated node and reallocates its shards.
    ensureStableCluster(/* expectedNodes */ 3, internalCluster().getClusterManagerName());

    partition.stopDisrupting();
    internalCluster().clearDisruptionScheme();
    ensureGreen(TimeValueMinutes(1), INDEX);
}

Warning: Disruption tests are powerful but easy to get wrong — partition the wrong set and you either deadlock or test nothing. Start with the hard-stop version (Step 3); reach for partitions only when the bug specifically involves a node being unreachable rather than dead.

Determinism: `ensureGreen` and `assertBusy`, never `Thread.sleep`

Everything after a node stop is asynchronous: the cluster manager detects the departure, computes a new cluster state (promotion + a new replica), publishes it, and shards recover. You do not know when that finishes — only that it will.

Anti-pattern	Why it flakes	Deterministic replacement
`Thread.sleep(2000)` then assert	Too short under load → fail; too long → slow always	`ensureGreen(index)` blocks exactly until healthy
Assert hit count immediately after stop	Search may hit a relocating shard mid-flight	`assertBusy(() -> assertHitCount(n), 30, SECONDS)`
Assume the killed node held the primary	It might have held the replica; promotion may be a no-op	Assert the end state (green, 0 unassigned), not the path
`ensureGreen()` with the default short timeout during reallocation	Copying a shard can exceed it	Pass an explicit timeout: `ensureGreen(timeValueMinutes(1), index)`

ensureGreen polls cluster health until GREEN (all primaries and replicas STARTED) or it times out. assertBusy re-runs an assertion with backoff until it passes or times out. Together they let you assert on a converged state without guessing how long convergence takes. This is the determinism discipline that Lab 5.4 is entirely about.

Implementation Requirements / Deliverables

ReplicaPromotionFailoverIT.java in server/src/internalClusterTest/java/... with the SPDX header.
@ClusterScope(scope = Scope.TEST) (failure must not leak into other tests).
A 3-data-node cluster + dedicated cluster manager, index with number_of_replicas >= 1.
ensureGreen before indexing and after the node stop.
Documents indexed via indexRandom, with a count assertion before the failure.
internalCluster().stopRandomDataNode() (or a disruption scheme) as the injected failure.
Post-failure assertions: green status, unassignedShards == 0, correct activeShards, and a no-data-loss hit count via assertBusy.
Zero Thread.sleep calls anywhere in the test.
Survives -Dtests.iters=50.

Troubleshooting

Symptom	Likely cause	Fix
Test never runs / "0 tests"	File is in `src/test` or not named `*IT`	Move to `src/internalClusterTest`, name it `*IT.java`
`ensureGreen` times out after the stop	Only 1 data node left can't host both copies (same-shard decider)	Start enough data nodes (3) so a new replica has a home
Hit count is short right after stop	Asserted before relocation settled	Wrap in `assertBusy(...)`; ensure `refresh` ran
Flaky under `-Dtests.iters`	An async assertion without polling	Replace direct asserts with `ensureGreen`/`assertBusy`
`NoNodeAvailableException`	You stopped the node the client was pinned to	Use `client()` (round-robins), not a node-pinned client
Cluster won't form	No cluster-manager-eligible node started	Start `startClusterManagerOnlyNode()` first
OOM / very slow	Too many docs or `Scope.SUITE` leaking nodes	Use `scaledRandomIntBetween`; keep `Scope.TEST`

Expected Output

> Task :server:internalClusterTest
org.opensearch.cluster.routing.ReplicaPromotionFailoverIT >
  testReplicaIsPromotedAndClusterRecoversAfterDataNodeStop PASSED

BUILD SUCCESSFUL in 1m 12s
1 test completed

Open the report to inspect captured node logs (where the promotion/reallocation messages are):

open server/build/reports/tests/internalClusterTest/index.html   # macOS

In the captured logs you should see the cluster manager log the node leaving, then shard-started messages as the replica is promoted and a new replica recovers.

Stretch Goals

Assert the promotion explicitly. Before the stop, record which node holds the primary for each shard via client().admin().cluster().prepareState().get().getState().routingTable(). Stop the node holding a primary on purpose (not a random one) and assert the former replica's node now holds the primary.
Test the no-replica case. Set number_of_replicas = 0, stop the node holding a shard, and assert the cluster goes red and the index loses data — proving why replicas matter. (Use ensureRed/a health check; this is a negative test.)
Add a disruption variant. Implement the testPartitionThenHeal from Step 6 and confirm the cluster ejects then re-admits the isolated node.
Index during the failure. Start a background indexing thread, stop a node mid-stream, and assert no acknowledged write was lost — closer to a real-world workload and a great bug finder.
Tie it to a real issue. Find an open flaky-test or recovery bug in github.com/opensearch-project/OpenSearch/issues whose repro is "stop a node," and shape your test to match it.

Coding Exercises

These extend ReplicaPromotionFailoverIT. All run under :server:internalClusterTest, all use condition-waits only — zero Thread.sleep. Locate disruption/cluster helpers with rg (e.g. rg "stopRandomDataNode|restartRandomDataNode|getInstance\(IndexShard" test/framework/src/main/java) rather than guessing a signature.

(warm-up) Record placement, then kill the primary on purpose. Before the stop, read the routing table (client().admin().cluster().prepareState().get().getState().routingTable()), find the node holding p0's primary, and stop that specific node via internalCluster().stopRandomNodeNotClusterManager(...) or by node name (find the right helper with rg "stopRandomNode|stopNode|stopCurrentClusterManagerNode" test/framework/src/main/java). After ensureGreen, assert the former replica's node now holds the primary — a deterministic promotion assertion, not a random one.
(warm-up) Negative test: replicas=0 means data loss. Add a method with number_of_replicas = 0, index N docs, stop the node holding the only copy of a shard, and assert the cluster goes red (prepareHealth().get().getStatus() == RED) and the hit count drops below N. This proves why replicas exist. Use ensureRed/a health poll, found via rg "ensureRed|waitForRedStatus" test/framework/src/main/java.
(core) Recover-from-restart, not just from stop. Replace the hard stop with internalCluster().restartRandomDataNode() (a RestartCallback). Assert the cluster briefly leaves green and returns to green with the same doc count via assertBusy. This exercises the recovery path (translog replay on the returning node) rather than reallocation — a distinct code path worth a distinct test.
(core) Index during the failure (the real bug finder). Start a background indexing loop on a daemon thread that keeps issuing acknowledged writes; stop a data node mid-stream; then quiesce and assert that every acknowledged write survived (track returned _ids in a concurrent set, then assertBusy that a mget/search finds them all). Guard with a volatile boolean running and join the thread in a finally. This is the closest thing to a real workload and routinely surfaces replication races.
(core) Network partition instead of a hard stop. Implement the Step 6 testPartitionThenHeal for real: override nodePlugins() to add MockTransportService.TestPlugin, apply a NetworkDisruption.TwoPartitions with DISCONNECT, ensureStableCluster while partitioned, then heal and ensureGreen. Assert the isolated node is ejected then re-admitted (node count dips and recovers). Find the disruption API with rg "class NetworkDisruption|TwoPartitions|DISCONNECT" test/framework/src/main/java.
(advanced challenge) Two concurrent failures with an explicit no-data-loss + checkpoint invariant. Build a Scope.TEST IT with a dedicated cluster manager and four data nodes, 2 shards × 1 replica. Index a known set with indexRandom, then under a background indexing thread (Exercise 4) stop two different data nodes in succession (waiting for green between, so each shard always retains a live copy). After healing, assert: (a) getUnassignedShards() == 0 and green; (b) every acknowledged _id is still searchable; and (c) the global checkpoint has advanced past the last write on every shard — read it from shard stats / IndexShard via rg "getGlobalCheckpoint|seqNoStats|SeqNoStats" server/src/main/java test/framework/src/main/java, and tie it back to the durability discussion in Lab 6.2. Survive -Dtests.iters=30. This is a maintainer-grade resilience test: it proves convergence, no loss, and checkpoint progress under compound failure.

Issues to Practice On

Failover/recovery and the flaky tests around them are where the hardest OpenSearch bugs live — and the best-labeled on-ramps. Labels drift; list them with gh label list --repo opensearch-project/OpenSearch and confirm.

# Recovery / failover / flaky-test issues (labels move; confirm on the tracker).
gh issue list --repo opensearch-project/OpenSearch --label "flaky-test" --state open
gh issue list --repo opensearch-project/OpenSearch --label "distributed" --state open
gh issue list --repo opensearch-project/OpenSearch --label "Storage" --state open
gh issue list --repo opensearch-project/OpenSearch --search "recovery in:title is:open label:bug"
gh issue list --repo opensearch-project/OpenSearch --search "stop node OR node left in:title is:open"

Representative patterns:

"Flaky failover/recovery IT." The repro is literally "stop a node" or "restart a node," and the test asserts on async state without polling. Reproduce from the seed in the issue, replace the racing assertion with ensureGreen/assertBusy, and shape your test to match (Step 5 of Lab 5.4).
"No test covers failover for feature X." A distributed feature shipped without a node-death *IT. Mirror an existing failover test (rg -l "stopRandomDataNode" server/src/internalClusterTest), adapt it to the feature, and PR.

Planted bug: In your failover test, delete the post-stop ensureGreen(TimeValueMinutes(1), INDEX) and replace the no-data-loss check with a single immediate assertHitCount(docCount) (no assertBusy). Run -Dtests.iters=50: it will flake — the search races shard relocation. That flake is the bug. Restore ensureGreen and the assertBusy, then add a comment naming the race so a future "simplification" can't quietly reintroduce it. Confirm green across 50 iters before reverting nothing (this fix stays).

Etiquette: Claim the issue, reproduce the failure first, and ship a test + CHANGELOG.md + DCO Signed-off-by (git commit -s). See community interaction and the level-2 PR lab.

Validation / Self-check

Why is this an OpenSearchIntegTestCase and not an OpenSearchSingleNodeTestCase or a unit test? What specifically requires multiple real nodes?
Why @ClusterScope(scope = Scope.TEST) rather than the default Scope.SUITE for a test that kills a node?
After stopRandomDataNode(), exactly what does OpenSearch do to return to green? Name the two distinct actions and which component drives them.
Why must you call ensureGreen before indexing as well as after the stop?
Why is assertBusy(() -> assertHitCount(n), ...) correct but Thread.sleep(2000); assertHitCount(n) wrong? Give the two distinct failure modes of the sleep version.
You started 3 data nodes. What goes wrong if you start only 2, and which AllocationDecider explains it?
Run -Dtests.iters=50. Did any iteration fail? If so, what timing assumption did it expose and how did you make it deterministic?

When your failover test is green, deterministic across 50 iterations, and free of Thread.sleep, proceed to Lab 5.4: Fix It — Un-Mute a Flaky Test.

Open-Source Engineer & Contributor