Lab 5.1: OpenSearchIntegTestCase and InternalTestCluster

Background

Most of what makes OpenSearch OpenSearch — replication, allocation, recovery, failover, cross-node search — only happens when there is more than one node. You cannot prove that behavior with a unit test. The framework's answer is OpenSearchIntegTestCase: it stands up a real, multi-node cluster inside the test JVM via InternalTestCluster, hands you a client(), and lets you drive it exactly like a production cluster — except it is deterministic, fast to start, and torn down automatically.

In this lab you write a working integration test from scratch: start a multi-node cluster, create an index with replicas, index documents, wait for the cluster to go green, and assert a query returns what you indexed. Along the way you read the framework itself so the magic stops being magic.

This is the hands-on counterpart to the taxonomy in the Level 5 index. For the engine internals these tests exercise, see the engine-internals deep dive and index-shard-lifecycle deep dive.

Why This Lab Matters for Contributors

• A huge fraction of OpenSearch's test suite is *IT.java integ tests. To fix bugs in distributed behavior — and to prove the fix — you must be fluent in OpenSearchIntegTestCase.
• Maintainers reject PRs whose tests can't actually fail when the bug is present. Knowing how to size the cluster, choose @ClusterScope, and ensureGreen() correctly is the difference between a test that catches regressions and one that's theater.
• The same patterns (internalCluster(), ensureGreen(), assertBusy()) carry into every later level, including the failover test in Lab 5.3.

Prerequisites

• OpenSearch builds and ./gradlew :server:test passes on your machine (Level 1).
• You can run a single node from source (./gradlew run) and hit it with curl (Level 2).
• You've read the Level 5 index taxonomy and the @ClusterScope table.
• JDK 21, ~8 GB RAM free (an in-JVM 3-node cluster is real and uses real memory).

Step-by-Step Tasks

Step 1 (15 min) — Read the framework you're about to use

Before writing a test, read the two classes that do the work. You will refer back to these constantly.

# The base class: what client(), internalCluster(), ensureGreen, @ClusterScope come from.
find test/framework/src/main/java -name "OpenSearchIntegTestCase.java"
grep -n "public.*client()\|protected.*internalCluster()\|ensureGreen\|ensureYellow\|@interface ClusterScope\|numDataNodes\|Scope " \
  test/framework/src/main/java/org/opensearch/test/OpenSearchIntegTestCase.java | head -40

# The cluster itself: how nodes start/stop and how health is reached.
find test/framework/src/main/java -name "InternalTestCluster.java"
grep -n "startNodes\|startNode\|stopRandomDataNode\|getClusterManagerName\|client(\|ensureAtLeastNumDataNodes" \
  test/framework/src/main/java/org/opensearch/test/InternalTestCluster.java | head -40

Note three things:

1. client() returns a client that load-balances across the cluster (a random node), so your request goes through a real coordinating hop — exactly like production.
2. internalCluster() is your handle to the physical cluster: starting/stopping nodes, finding the cluster manager (formerly master), addressing a specific node.
3. ensureGreen(...) blocks until all primaries and replicas of the named indices are assigned and started. Without it, your assertions race the allocator.

Step 2 (10 min) — Find an existing integ test to mirror

Never write an integ test on a blank page. Copy the shape of a known-good one.

# Real integ tests live in src/internalClusterTest and end in *IT.java
find server/src/internalClusterTest/java -name "*IT.java" | head
# A small, readable example to model on:
grep -rln "extends OpenSearchIntegTestCase" server/src/internalClusterTest/java | head
grep -rln "ensureGreen\|internalCluster().startNodes" server/src/internalClusterTest/java | head

Open one and study how it: declares @ClusterScope, sizes the cluster, creates an index via client().admin().indices().prepareCreate(...), indexes via client().prepareIndex(...), and asserts via client().prepareSearch(...).

Step 3 (5 min) — Decide cluster scope and size

For this lab we want a fresh, isolated cluster so our assertions are clean, and three data nodes so replicas have somewhere to go:

Note: Scope.TEST is slower than the default Scope.SUITE because it rebuilds the cluster each method. For a teaching lab with one method that's fine. In real PRs, prefer Scope.SUITE and clean up after yourself unless the test is disruptive (kills nodes), in which case use Scope.TEST.

Step 4 (25 min) — Write the test

Create server/src/internalClusterTest/java/org/opensearch/integlab/IntegLabIT.java:

/*
 * SPDX-License-Identifier: Apache-2.0
 *
 * The OpenSearch Contributors require contributions made to
 * this file be licensed under the Apache-2.0 license or a
 * compatible open source license.
 */

package org.opensearch.integlab;

import org.opensearch.action.admin.cluster.health.ClusterHealthResponse;
import org.opensearch.action.search.SearchResponse;
import org.opensearch.cluster.health.ClusterHealthStatus;
import org.opensearch.common.settings.Settings;
import org.opensearch.index.query.QueryBuilders;
import org.opensearch.test.OpenSearchIntegTestCase;
import org.opensearch.test.OpenSearchIntegTestCase.ClusterScope;
import org.opensearch.test.OpenSearchIntegTestCase.Scope;

import static org.opensearch.test.hamcrest.OpenSearchAssertions.assertHitCount;
import static org.opensearch.test.hamcrest.OpenSearchAssertions.assertSearchHits;

@ClusterScope(scope = Scope.TEST, numDataNodes = 3)
public class IntegLabIT extends OpenSearchIntegTestCase {

    public void testIndexThenSearchOnAGreenMultiNodeCluster() throws Exception {
        final String index = "integ-lab";

        // 1. Create an index with 2 primaries and 1 replica each. With 3 data nodes,
        //    every shard copy can be assigned -> the cluster can reach GREEN.
        createIndex(index, Settings.builder()
            .put("index.number_of_shards", 2)
            .put("index.number_of_replicas", 1)
            .build());

        // 2. Wait until all primaries AND replicas are started. This is the gate that
        //    makes the rest of the test deterministic.
        ClusterHealthResponse health = ensureGreen(index);
        assertEquals(ClusterHealthStatus.GREEN, health.getStatus());
        // With 2 shards * (1 primary + 1 replica) = 4 active shards.
        assertEquals(4, health.getActiveShards());

        // 3. Index three documents. IMMEDIATE refresh makes them searchable now,
        //    so we don't have to wait for the periodic refresh.
        client().prepareIndex(index).setId("1")
            .setSource("title", "opensearch internals", "level", 5).get();
        client().prepareIndex(index).setId("2")
            .setSource("title", "lucene segments", "level", 6).get();
        client().prepareIndex(index).setId("3")
            .setSource("title", "opensearch testing", "level", 5).get();
        refresh(index); // open a new searcher so the docs are visible

        // 4. Assert a match-all sees all three docs.
        SearchResponse all = client().prepareSearch(index).get();
        assertHitCount(all, 3L);

        // 5. Assert a term query returns the right subset, by id.
        SearchResponse hits = client().prepareSearch(index)
            .setQuery(QueryBuilders.matchQuery("title", "opensearch"))
            .get();
        assertHitCount(hits, 2L);
        assertSearchHits(hits, "1", "3");

        // 6. Sanity: the cluster really does have 3 data nodes, and a single cluster manager.
        assertEquals(3, internalCluster().numDataNodes());
        assertNotNull(internalCluster().getClusterManagerName()); // formerly getMasterName()
    }
}

What each piece is doing:

• createIndex(name, settings) is an OpenSearchIntegTestCase helper around client().admin().indices().prepareCreate(...). With replicas: 1 and 3 data nodes, the replica has a legal home, so green is reachable.
• ensureGreen(index) blocks on a cluster-health request that waits for GREEN. If allocation is broken (e.g. a decider bug from Lab 4.4), this is where your test would hang/fail — that is exactly the signal you want.
• refresh(index) opens a new Lucene searcher so just-indexed docs become visible. Skipping this is the #1 cause of "my integ test sees 0 hits" (see Troubleshooting).
• assertHitCount / assertSearchHits come from OpenSearchAssertions — use these, not raw JUnit, because they produce far better failure messages for search responses.

Step 5 (10 min) — Run it and read the output

./gradlew :server:internalClusterTest --tests "org.opensearch.integlab.IntegLabIT"

Note: It's :server:internalClusterTest, not :server:test. The *IT.java suffix routes the class to the internalClusterTest source set/task. Run it under :server:test and it won't be picked up at all.

Open the report:

open server/build/reports/tests/internalClusterTest/index.html   # macOS
# or
xdg-open server/build/reports/tests/internalClusterTest/index.html

Step 6 (10 min) — Prove the test can actually fail

A test that can't fail is worthless. Temporarily break it, confirm it goes red, then revert:

// Change the expected count to something wrong:
assertHitCount(all, 99L);   // should be 3

Run again — it must fail with a clear message (expected 99 but was 3). Revert. This habit (deliberately falsifying the assertion once) catches tests that silently pass for the wrong reason.

Implementation Requirements / Deliverables

• A new IntegLabIT under server/src/internalClusterTest/java/... extending OpenSearchIntegTestCase.
• @ClusterScope(scope = Scope.TEST, numDataNodes = 3) (or your justified alternative).
• Creates an index with number_of_replicas: 1, then ensureGreen(...) and asserts the status is GREEN and the active-shard count is what you expect.
• Indexes ≥3 docs, refreshes, and asserts both a match-all count and a term-query subset (by id).
• Runs green under ./gradlew :server:internalClusterTest --tests "*IntegLabIT".
• You demonstrated it can fail (Step 6) before reverting.

Troubleshooting

Expected Output

> Task :server:internalClusterTest
org.opensearch.integlab.IntegLabIT > testIndexThenSearchOnAGreenMultiNodeCluster PASSED

BUILD SUCCESSFUL

The HTML report shows one test, status PASSED, with captured node logs under the stdout/stderr tab (a good place to see the cluster forming, the index being created, and shards going green).

Stretch Goals

1. Randomize the topology. Replace fixed numDataNodes = 3 with @ClusterScope(minNumDataNodes = 2, maxNumDataNodes = 5) and make the test size-agnostic (compute expected active shards from number_of_shards * (1 + replicas) and the actual node count). This is how real OpenSearch tests find size-dependent bugs.
2. Assert per-node placement. Use client().admin().cluster().prepareState().get().getState() .getRoutingTable() to assert no primary and its replica landed on the same node (the SameShardAllocationDecider invariant). Cross-reference shard-allocation deep dive.
3. Make it a SUITE-scoped test. Convert to Scope.SUITE, add a second test method, and add a teardown that deletes the index — proving you can keep a shared cluster clean between methods.
4. Watch it form under TRACE. Add @TestLogging("org.opensearch.cluster.service:TRACE") and read the cluster-state versions advancing in the report, connecting this lab to Level 4.

Validation / Self-check

1. Why is the task :server:internalClusterTest and not :server:test? What makes Gradle route IntegLabIT to the right source set?
2. What exactly does ensureGreen("idx") wait for, and what would make it never return?
3. Why must you refresh(index) (or use IMMEDIATE) before searching in an integ test? What is actually happening when you refresh?
4. With number_of_shards: 2 and number_of_replicas: 1 on 3 data nodes, how many active shards do you expect, and why?
5. What's the difference between client() and internalCluster().client(nodeName)? When would you need the second?
6. When should you choose Scope.TEST over the default Scope.SUITE, and what's the cost?
7. You changed an assertion to a wrong value and the test still passed. What does that tell you, and what would you check?

When your test stands up a green 3-node cluster, indexes and queries documents deterministically, and you've proven it can fail, move on to Lab 5.2: Add a Missing Unit Test.