Lab 5.3: Build It — A Multi-Node Integration Test

Background

Unit tests prove logic; integration tests prove distributed behavior. The behavior that breaks in production — a primary dying, a replica getting promoted, shards reallocating, the cluster recovering to green — only shows up when you have multiple nodes and you take one away. This is what OpenSearchIntegTestCase and its in-JVM InternalTestCluster exist for.

In this Build-It lab you write a complete failover integration test from scratch: stand up a three-node cluster, index documents into an index with one replica, stop a random data node, and assert that OpenSearch promotes the replica and reallocates shards so the cluster returns to green with no data loss. You will do this deterministically — no Thread.sleep, only ensureGreen and assertBusy.

Note: This builds on Lab 5.1, which got you a multi-node cluster indexing data. Here we add the hard part: surviving a node death. The cluster manager (formerly master) is what drives replica promotion and reallocation under the hood — see the replication and shard allocation deep dives for the machinery you are testing.

Why This Lab Matters for Contributors

• Failover/recovery is where the hardest OpenSearch bugs live. You cannot credibly fix one without a test that reproduces the failure scenario — and that test is exactly this shape.
• Maintainers expect distributed changes to come with an *IT that runs under internalClusterTest. Knowing the idioms (internalCluster(), ensureGreen, assertBusy, disruption helpers) makes your PRs reviewable.
• This is the difference between "it worked on my laptop" and "it survives a node going away under load" — the bar real clusters are held to.
• Writing it teaches you determinism: why asserting on async state without polling produces flaky CI, the topic you will hunt in Lab 5.4.

Prerequisites

• Lab 5.1 completed; you can run an *IT under internalClusterTest.
• A clean build: ./gradlew :server:compileInternalClusterTestJava -q.
• ~6 GB free RAM (three in-JVM nodes plus the test runner).

# Confirm the integ-test source set and task exist.
ls server/src/internalClusterTest/java/org/opensearch | head
./gradlew :server:tasks --all | grep internalClusterTest

Step-by-Step Tasks

Step 1: Decide the topology and what "correct" means

You are testing the simplest non-trivial failover:

3 data nodes, 1 index, 1 primary + 1 replica per shard (number_of_replicas = 1)
        ┌─ node A: primary p0
        ├─ node B: replica r0   (eligible to be promoted)
        └─ node C: (spare capacity for reallocation)

Kill the node holding a copy → cluster must:
  1. promote the surviving copy to primary (if the primary died), AND
  2. allocate a NEW replica onto a remaining node, AND
  3. return to GREEN with the same document count (no data loss).

The invariants your test asserts:

Step 2: Create the IT file (correct source set, correct suffix)

Integration tests are *IT.java under src/internalClusterTest — not src/test. Put it in the wrong place and Gradle silently never runs it.

mkdir -p server/src/internalClusterTest/java/org/opensearch/cluster/routing
$EDITOR server/src/internalClusterTest/java/org/opensearch/cluster/routing/ReplicaPromotionFailoverIT.java

Step 3: Write the test

Here is the complete, runnable test. Read every line — the comments explain why each call is there.

/*
 * SPDX-License-Identifier: Apache-2.0
 *
 * The OpenSearch Contributors require contributions made to
 * this file be licensed under the Apache-2.0 license or a
 * compatible open source license.
 */

package org.opensearch.cluster.routing;

import org.opensearch.action.admin.cluster.health.ClusterHealthResponse;
import org.opensearch.action.index.IndexRequestBuilder;
import org.opensearch.cluster.health.ClusterHealthStatus;
import org.opensearch.common.settings.Settings;
import org.opensearch.index.query.QueryBuilders;
import org.opensearch.test.OpenSearchIntegTestCase;
import org.opensearch.test.OpenSearchIntegTestCase.ClusterScope;
import org.opensearch.test.OpenSearchIntegTestCase.Scope;

import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.TimeUnit;

import static org.opensearch.index.query.QueryBuilders.matchAllQuery;
import static org.hamcrest.Matchers.equalTo;

/**
 * Stops a data node and verifies OpenSearch promotes a replica and reallocates shards
 * back to GREEN with no data loss. A failover test — Scope.TEST so node death never
 * leaks into another method's cluster.
 */
@ClusterScope(scope = Scope.TEST, numDataNodes = 0)   // we start nodes ourselves, explicitly
public class ReplicaPromotionFailoverIT extends OpenSearchIntegTestCase {

    private static final String INDEX = "failover-idx";

    public void testReplicaIsPromotedAndClusterRecoversAfterDataNodeStop() throws Exception {
        // 1) A dedicated cluster manager + three data nodes. Dedicated CM keeps election
        //    independent of the data node we are about to kill.
        internalCluster().startClusterManagerOnlyNode();
        final List<String> dataNodes = new ArrayList<>(internalCluster().startDataOnlyNodes(3));
        assertEquals(3, dataNodes.size());

        // 2) An index with 2 primaries and 1 replica each → 4 shard copies across 3 data nodes.
        createIndex(
            INDEX,
            Settings.builder()
                .put("index.number_of_shards", 2)
                .put("index.number_of_replicas", 1)
                .build()
        );
        ensureGreen(INDEX);   // block until all 4 copies are STARTED before we touch anything

        // 3) Index a known number of docs and make them durable+visible.
        final int docCount = scaledRandomIntBetween(20, 100);   // randomized, but bounded
        indexDocs(docCount);
        refresh(INDEX);
        assertHitCount(docCount);   // sanity: all docs are searchable before the failure

        // 4) Inject the failure: stop a random DATA node (could be holding a primary or a replica).
        internalCluster().stopRandomDataNode();

        // 5) The cluster must self-heal. ensureGreen blocks until promotion + reallocation finish.
        //    We allow a generous timeout because reallocation copies a shard.
        ensureGreen(TimeValueMinutes(1), INDEX);

        // 6) Health invariants: no unassigned shards, all copies active again.
        ClusterHealthResponse health = client().admin().cluster()
            .prepareHealth(INDEX)
            .setWaitForGreenStatus()
            .get();
        assertThat(health.getStatus(), equalTo(ClusterHealthStatus.GREEN));
        assertThat(health.getUnassignedShards(), equalTo(0));
        assertThat(health.getActiveShards(), equalTo(4));   // 2 primaries + 2 replicas

        // 7) No data loss. Search is async w.r.t. shard relocation, so poll with assertBusy.
        assertBusy(() -> assertHitCount(docCount), 30, TimeUnit.SECONDS);
    }

    // ----- helpers -----

    private void indexDocs(int count) throws InterruptedException {
        final List<IndexRequestBuilder> builders = new ArrayList<>(count);
        for (int i = 0; i < count; i++) {
            builders.add(
                client().prepareIndex(INDEX)
                    .setId(Integer.toString(i))
                    .setSource("field", randomAlphaOfLengthBetween(1, 20), "n", i)
            );
        }
        // indexRandom shuffles, batches, and refreshes for you — preferred over a manual loop.
        indexRandom(true, builders);
    }

    private void assertHitCount(long expected) {
        long actual = client().prepareSearch(INDEX)
            .setQuery(matchAllQuery())
            .setSize(0)
            .get()
            .getHits()
            .getTotalHits()
            .value();
        assertThat(actual, equalTo(expected));
    }

    private static org.opensearch.common.unit.TimeValue TimeValueMinutes(long m) {
        return org.opensearch.common.unit.TimeValue.timeValueMinutes(m);
    }
}

Note: indexRandom(true, builders) is the idiomatic way to index in integ tests: it randomizes order and batching (finding ordering bugs) and refreshes at the end so docs are searchable. scaledRandomIntBetween scales the upper bound down under nightly/CI pressure so the test stays fast.

Step 4: Run it under the integration test task

# The whole class
./gradlew :server:internalClusterTest --tests "org.opensearch.cluster.routing.ReplicaPromotionFailoverIT"

# One method, verbose
./gradlew :server:internalClusterTest \
  --tests "*ReplicaPromotionFailoverIT.testReplicaIsPromotedAndClusterRecoversAfterDataNodeStop" -i

It is slower than a unit test (it builds and tears down a real cluster), typically 20–90 seconds.

Step 5: Prove it's deterministic — hammer it

A failover test that passes once but flakes under load is worse than none. Run it many times with different seeds to flush out timing assumptions:

./gradlew :server:internalClusterTest --tests "*ReplicaPromotionFailoverIT" -Dtests.iters=50

If any iteration fails, copy the REPRODUCE WITH ... -Dtests.seed=... line and debug that exact seed. A flaky failure here almost always means you asserted on async state without polling — replace a direct assertion with assertBusy or add the missing ensureGreen.

Step 6 (optional): Use a disruption helper instead of a hard stop

Stopping a node is the simplest failure. To test a network partition (the node is alive but unreachable — a different and nastier failure mode), use the disruption framework. Override the test plugins to mock the transport, then apply a partition:

import org.opensearch.plugins.Plugin;
import org.opensearch.test.disruption.NetworkDisruption;
import org.opensearch.test.transport.MockTransportService;

import java.util.Collection;
import java.util.Collections;

@Override
protected Collection<Class<? extends Plugin>> nodePlugins() {
    return Collections.singletonList(MockTransportService.TestPlugin.class);
}

public void testPartitionThenHeal() throws Exception {
    internalCluster().startClusterManagerOnlyNode();
    internalCluster().startDataOnlyNodes(3);
    createIndex(INDEX, Settings.builder()
        .put("index.number_of_shards", 1).put("index.number_of_replicas", 1).build());
    ensureGreen(INDEX);

    // Isolate one node from the rest, hold it, then heal and require recovery.
    String victim = internalCluster().getRandomNodeName();   // pick a node to cut off
    NetworkDisruption partition = new NetworkDisruption(
        new NetworkDisruption.TwoPartitions(
            Collections.singleton(victim), otherNodes(victim)),
        NetworkDisruption.DISCONNECT);
    internalCluster().setDisruptionScheme(partition);
    partition.startDisrupting();

    // While partitioned, the cluster manager evicts the isolated node and reallocates its shards.
    ensureStableCluster(/* expectedNodes */ 3, internalCluster().getClusterManagerName());

    partition.stopDisrupting();
    internalCluster().clearDisruptionScheme();
    ensureGreen(TimeValueMinutes(1), INDEX);
}

Warning: Disruption tests are powerful but easy to get wrong — partition the wrong set and you either deadlock or test nothing. Start with the hard-stop version (Step 3); reach for partitions only when the bug specifically involves a node being unreachable rather than dead.

Determinism: ensureGreen and assertBusy, never Thread.sleep

Everything after a node stop is asynchronous: the cluster manager detects the departure, computes a new cluster state (promotion + a new replica), publishes it, and shards recover. You do not know when that finishes — only that it will.

ensureGreen polls cluster health until GREEN (all primaries and replicas STARTED) or it times out. assertBusy re-runs an assertion with backoff until it passes or times out. Together they let you assert on a converged state without guessing how long convergence takes. This is the determinism discipline that Lab 5.4 is entirely about.

Implementation Requirements / Deliverables

• ReplicaPromotionFailoverIT.java in server/src/internalClusterTest/java/... with the SPDX header.
• @ClusterScope(scope = Scope.TEST) (failure must not leak into other tests).
• A 3-data-node cluster + dedicated cluster manager, index with number_of_replicas >= 1.
• ensureGreen before indexing and after the node stop.
• Documents indexed via indexRandom, with a count assertion before the failure.
• internalCluster().stopRandomDataNode() (or a disruption scheme) as the injected failure.
• Post-failure assertions: green status, unassignedShards == 0, correct activeShards, and a no-data-loss hit count via assertBusy.
• Zero Thread.sleep calls anywhere in the test.
• Survives -Dtests.iters=50.

Troubleshooting

Expected Output

> Task :server:internalClusterTest
org.opensearch.cluster.routing.ReplicaPromotionFailoverIT >
  testReplicaIsPromotedAndClusterRecoversAfterDataNodeStop PASSED

BUILD SUCCESSFUL in 1m 12s
1 test completed

Open the report to inspect captured node logs (where the promotion/reallocation messages are):

open server/build/reports/tests/internalClusterTest/index.html   # macOS

In the captured logs you should see the cluster manager log the node leaving, then shard-started messages as the replica is promoted and a new replica recovers.

Stretch Goals

1. Assert the promotion explicitly. Before the stop, record which node holds the primary for each shard via client().admin().cluster().prepareState().get().getState().routingTable(). Stop the node holding a primary on purpose (not a random one) and assert the former replica's node now holds the primary.
2. Test the no-replica case. Set number_of_replicas = 0, stop the node holding a shard, and assert the cluster goes red and the index loses data — proving why replicas matter. (Use ensureRed/a health check; this is a negative test.)
3. Add a disruption variant. Implement the testPartitionThenHeal from Step 6 and confirm the cluster ejects then re-admits the isolated node.
4. Index during the failure. Start a background indexing thread, stop a node mid-stream, and assert no acknowledged write was lost — closer to a real-world workload and a great bug finder.
5. Tie it to a real issue. Find an open flaky-test or recovery bug in github.com/opensearch-project/OpenSearch/issues whose repro is "stop a node," and shape your test to match it.

Validation / Self-check

1. Why is this an OpenSearchIntegTestCase and not an OpenSearchSingleNodeTestCase or a unit test? What specifically requires multiple real nodes?
2. Why @ClusterScope(scope = Scope.TEST) rather than the default Scope.SUITE for a test that kills a node?
3. After stopRandomDataNode(), exactly what does OpenSearch do to return to green? Name the two distinct actions and which component drives them.
4. Why must you call ensureGreen before indexing as well as after the stop?
5. Why is assertBusy(() -> assertHitCount(n), ...) correct but Thread.sleep(2000); assertHitCount(n) wrong? Give the two distinct failure modes of the sleep version.
6. You started 3 data nodes. What goes wrong if you start only 2, and which AllocationDecider explains it?
7. Run -Dtests.iters=50. Did any iteration fail? If so, what timing assumption did it expose and how did you make it deterministic?

When your failover test is green, deterministic across 50 iterations, and free of Thread.sleep, proceed to Lab 5.4: Fix It — Un-Mute a Flaky Test.