Lab 5.1: OpenSearchIntegTestCase and InternalTestCluster

Background

Most of what makes OpenSearch OpenSearch — replication, allocation, recovery, failover, cross-node search — only happens when there is more than one node. You cannot prove that behavior with a unit test. The framework's answer is OpenSearchIntegTestCase: it stands up a real, multi-node cluster inside the test JVM via InternalTestCluster, hands you a client(), and lets you drive it exactly like a production cluster — except it is deterministic, fast to start, and torn down automatically.

In this lab you write a working integration test from scratch: start a multi-node cluster, create an index with replicas, index documents, wait for the cluster to go green, and assert a query returns what you indexed. Along the way you read the framework itself so the magic stops being magic.

This is the hands-on counterpart to the taxonomy in the Level 5 index. For the engine internals these tests exercise, see the engine-internals deep dive and index-shard-lifecycle deep dive.

Why This Lab Matters for Contributors

• A huge fraction of OpenSearch's test suite is *IT.java integ tests. To fix bugs in distributed behavior — and to prove the fix — you must be fluent in OpenSearchIntegTestCase.
• Maintainers reject PRs whose tests can't actually fail when the bug is present. Knowing how to size the cluster, choose @ClusterScope, and ensureGreen() correctly is the difference between a test that catches regressions and one that's theater.
• The same patterns (internalCluster(), ensureGreen(), assertBusy()) carry into every later level, including the failover test in Lab 5.3.

Prerequisites

• OpenSearch builds and ./gradlew :server:test passes on your machine (Level 1).
• You can run a single node from source (./gradlew run) and hit it with curl (Level 2).
• You've read the Level 5 index taxonomy and the @ClusterScope table.
• JDK 21, ~8 GB RAM free (an in-JVM 3-node cluster is real and uses real memory).

Step-by-Step Tasks

Step 1 (15 min) — Read the framework you're about to use

Before writing a test, read the two classes that do the work. You will refer back to these constantly.

# The base class: what client(), internalCluster(), ensureGreen, @ClusterScope come from.
find test/framework/src/main/java -name "OpenSearchIntegTestCase.java"
grep -n "public.*client()\|protected.*internalCluster()\|ensureGreen\|ensureYellow\|@interface ClusterScope\|numDataNodes\|Scope " \
  test/framework/src/main/java/org/opensearch/test/OpenSearchIntegTestCase.java | head -40

# The cluster itself: how nodes start/stop and how health is reached.
find test/framework/src/main/java -name "InternalTestCluster.java"
grep -n "startNodes\|startNode\|stopRandomDataNode\|getClusterManagerName\|client(\|ensureAtLeastNumDataNodes" \
  test/framework/src/main/java/org/opensearch/test/InternalTestCluster.java | head -40

Note three things:

1. client() returns a client that load-balances across the cluster (a random node), so your request goes through a real coordinating hop — exactly like production.
2. internalCluster() is your handle to the physical cluster: starting/stopping nodes, finding the cluster manager (formerly master), addressing a specific node.
3. ensureGreen(...) blocks until all primaries and replicas of the named indices are assigned and started. Without it, your assertions race the allocator.

Step 2 (10 min) — Find an existing integ test to mirror

Never write an integ test on a blank page. Copy the shape of a known-good one.

# Real integ tests live in src/internalClusterTest and end in *IT.java
find server/src/internalClusterTest/java -name "*IT.java" | head
# A small, readable example to model on:
grep -rln "extends OpenSearchIntegTestCase" server/src/internalClusterTest/java | head
grep -rln "ensureGreen\|internalCluster().startNodes" server/src/internalClusterTest/java | head

Open one and study how it: declares @ClusterScope, sizes the cluster, creates an index via client().admin().indices().prepareCreate(...), indexes via client().prepareIndex(...), and asserts via client().prepareSearch(...).

Step 3 (5 min) — Decide cluster scope and size

For this lab we want a fresh, isolated cluster so our assertions are clean, and three data nodes so replicas have somewhere to go:

Note: Scope.TEST is slower than the default Scope.SUITE because it rebuilds the cluster each method. For a teaching lab with one method that's fine. In real PRs, prefer Scope.SUITE and clean up after yourself unless the test is disruptive (kills nodes), in which case use Scope.TEST.

Step 4 (25 min) — Write the test

Create server/src/internalClusterTest/java/org/opensearch/integlab/IntegLabIT.java:

/*
 * SPDX-License-Identifier: Apache-2.0
 *
 * The OpenSearch Contributors require contributions made to
 * this file be licensed under the Apache-2.0 license or a
 * compatible open source license.
 */

package org.opensearch.integlab;

import org.opensearch.action.admin.cluster.health.ClusterHealthResponse;
import org.opensearch.action.search.SearchResponse;
import org.opensearch.cluster.health.ClusterHealthStatus;
import org.opensearch.common.settings.Settings;
import org.opensearch.index.query.QueryBuilders;
import org.opensearch.test.OpenSearchIntegTestCase;
import org.opensearch.test.OpenSearchIntegTestCase.ClusterScope;
import org.opensearch.test.OpenSearchIntegTestCase.Scope;

import static org.opensearch.test.hamcrest.OpenSearchAssertions.assertHitCount;
import static org.opensearch.test.hamcrest.OpenSearchAssertions.assertSearchHits;

@ClusterScope(scope = Scope.TEST, numDataNodes = 3)
public class IntegLabIT extends OpenSearchIntegTestCase {

    public void testIndexThenSearchOnAGreenMultiNodeCluster() throws Exception {
        final String index = "integ-lab";

        // 1. Create an index with 2 primaries and 1 replica each. With 3 data nodes,
        //    every shard copy can be assigned -> the cluster can reach GREEN.
        createIndex(index, Settings.builder()
            .put("index.number_of_shards", 2)
            .put("index.number_of_replicas", 1)
            .build());

        // 2. Wait until all primaries AND replicas are started. This is the gate that
        //    makes the rest of the test deterministic.
        ClusterHealthResponse health = ensureGreen(index);
        assertEquals(ClusterHealthStatus.GREEN, health.getStatus());
        // With 2 shards * (1 primary + 1 replica) = 4 active shards.
        assertEquals(4, health.getActiveShards());

        // 3. Index three documents. IMMEDIATE refresh makes them searchable now,
        //    so we don't have to wait for the periodic refresh.
        client().prepareIndex(index).setId("1")
            .setSource("title", "opensearch internals", "level", 5).get();
        client().prepareIndex(index).setId("2")
            .setSource("title", "lucene segments", "level", 6).get();
        client().prepareIndex(index).setId("3")
            .setSource("title", "opensearch testing", "level", 5).get();
        refresh(index); // open a new searcher so the docs are visible

        // 4. Assert a match-all sees all three docs.
        SearchResponse all = client().prepareSearch(index).get();
        assertHitCount(all, 3L);

        // 5. Assert a term query returns the right subset, by id.
        SearchResponse hits = client().prepareSearch(index)
            .setQuery(QueryBuilders.matchQuery("title", "opensearch"))
            .get();
        assertHitCount(hits, 2L);
        assertSearchHits(hits, "1", "3");

        // 6. Sanity: the cluster really does have 3 data nodes, and a single cluster manager.
        assertEquals(3, internalCluster().numDataNodes());
        assertNotNull(internalCluster().getClusterManagerName()); // formerly getMasterName()
    }
}

What each piece is doing:

• createIndex(name, settings) is an OpenSearchIntegTestCase helper around client().admin().indices().prepareCreate(...). With replicas: 1 and 3 data nodes, the replica has a legal home, so green is reachable.
• ensureGreen(index) blocks on a cluster-health request that waits for GREEN. If allocation is broken (e.g. a decider bug from Lab 4.4), this is where your test would hang/fail — that is exactly the signal you want.
• refresh(index) opens a new Lucene searcher so just-indexed docs become visible. Skipping this is the #1 cause of "my integ test sees 0 hits" (see Troubleshooting).
• assertHitCount / assertSearchHits come from OpenSearchAssertions — use these, not raw JUnit, because they produce far better failure messages for search responses.

Step 5 (10 min) — Run it and read the output

./gradlew :server:internalClusterTest --tests "org.opensearch.integlab.IntegLabIT"

Note: It's :server:internalClusterTest, not :server:test. The *IT.java suffix routes the class to the internalClusterTest source set/task. Run it under :server:test and it won't be picked up at all.

Open the report:

open server/build/reports/tests/internalClusterTest/index.html   # macOS
# or
xdg-open server/build/reports/tests/internalClusterTest/index.html

Step 6 (10 min) — Prove the test can actually fail

A test that can't fail is worthless. Temporarily break it, confirm it goes red, then revert:

// Change the expected count to something wrong:
assertHitCount(all, 99L);   // should be 3

Run again — it must fail with a clear message (expected 99 but was 3). Revert. This habit (deliberately falsifying the assertion once) catches tests that silently pass for the wrong reason.

Implementation Requirements / Deliverables

• A new IntegLabIT under server/src/internalClusterTest/java/... extending OpenSearchIntegTestCase.
• @ClusterScope(scope = Scope.TEST, numDataNodes = 3) (or your justified alternative).
• Creates an index with number_of_replicas: 1, then ensureGreen(...) and asserts the status is GREEN and the active-shard count is what you expect.
• Indexes ≥3 docs, refreshes, and asserts both a match-all count and a term-query subset (by id).
• Runs green under ./gradlew :server:internalClusterTest --tests "*IntegLabIT".
• You demonstrated it can fail (Step 6) before reverting.

Troubleshooting

Expected Output

> Task :server:internalClusterTest
org.opensearch.integlab.IntegLabIT > testIndexThenSearchOnAGreenMultiNodeCluster PASSED

BUILD SUCCESSFUL

The HTML report shows one test, status PASSED, with captured node logs under the stdout/stderr tab (a good place to see the cluster forming, the index being created, and shards going green).

Stretch Goals

1. Randomize the topology. Replace fixed numDataNodes = 3 with @ClusterScope(minNumDataNodes = 2, maxNumDataNodes = 5) and make the test size-agnostic (compute expected active shards from number_of_shards * (1 + replicas) and the actual node count). This is how real OpenSearch tests find size-dependent bugs.
2. Assert per-node placement. Use client().admin().cluster().prepareState().get().getState() .getRoutingTable() to assert no primary and its replica landed on the same node (the SameShardAllocationDecider invariant). Cross-reference shard-allocation deep dive.
3. Make it a SUITE-scoped test. Convert to Scope.SUITE, add a second test method, and add a teardown that deletes the index — proving you can keep a shared cluster clean between methods.
4. Watch it form under TRACE. Add @TestLogging("org.opensearch.cluster.service:TRACE") and read the cluster-state versions advancing in the report, connecting this lab to Level 4.

Coding Exercises

These all extend the IntegLabIT you wrote (or a sibling *IT). Each must compile and run under ./gradlew :server:internalClusterTest --tests "*...". Locate any class/method you touch with rg on your branch (e.g. rg "ensureStableCluster|assertAcked" test/framework/src/main/java) — never trust a line number.

1. (warm-up) Size-agnostic active-shard assertion. Replace the hard-coded assertEquals(4, health.getActiveShards()) with a computed expectation. Read number_of_shards and number_of_replicas back from the index settings via client().admin().indices().prepareGetSettings(index).get(), compute shards * (1 + replicas), and assert that. Prove it by changing the index to 3 shards and watching the test still pass with no edit to the assertion.
2. (warm-up) Use assertAcked and assertBusy correctly. Wrap your createIndex in assertAcked(...) (find it in OpenSearchAssertions with rg "static.*assertAcked" test/framework/src/main/java) so a non-acknowledged create fails loudly. Then add a node-count check that polls: assertBusy(() -> assertEquals(3, internalCluster().numDataNodes())). Explain in a comment why assertBusy is pointless here (the count is synchronous) — and where it would matter.
3. (core) Randomize the topology. Convert the lab's stretch goal #1 into a graded test: annotate with @ClusterScope(scope = Scope.TEST, minNumDataNodes = 2, maxNumDataNodes = 5), read the actual count from internalCluster().numDataNodes(), set number_of_replicas so green is always reachable (replicas <= dataNodes - 1), and assert the computed active-shard count. Run with -Dtests.iters=20 and confirm it is green across the size range.
4. (core) Assert the same-shard invariant. Add a second test method that, after ensureGreen, walks client().admin().cluster().prepareState().get().getState().getRoutingTable() and asserts no primary and its replica share a node (the SameShardAllocationDecider invariant from the shard-allocation deep dive). Then deliberately set number_of_replicas higher than dataNodes - 1, switch the gate to ensureYellow, and assert getUnassignedShards() > 0 — a negative test proving the decider refuses to co-locate copies.
5. (core) Cross-node coordinating hop. Index on a pinned node, search on a different pinned node. Use internalCluster().getNodeNames() to grab two names, then internalCluster().client(nodeA).prepareIndex(...) and internalCluster().client(nodeB).prepareSearch(...) with a refresh between. Assert the doc is visible from nodeB, proving the coordinating-node fan-out works regardless of where you index.
6. (advanced challenge) SUITE-scoped multi-method suite with a teardown contract. Convert to @ClusterScope(scope = Scope.SUITE, numDataNodes = 3) and write three methods that share one cluster: (a) create-index-and-green, (b) index-and-search, (c) a per-index alias test. Add an @After-style cleanup (override the framework hook found via rg "wipeIndices|cleanUp|protected void afterIndexsCreated|ensureClusterStateConsistency" test/framework/src/main/java) that deletes the indices each method created, then assert in a final method that client().admin().indices().prepareGetIndex().get().getIndices().length == 0. The point: prove you can keep a shared SUITE cluster clean — the discipline real OpenSearch suites depend on. Run the whole class under -Dtests.iters=10 and show no method leaks state into the next.

Issues to Practice On

The integ-test framework is exercised by nearly every distributed-behavior bug, so these labels are a broad on-ramp. Label taxonomies drift — run gh label list --repo opensearch-project/OpenSearch and confirm before relying on any one.

# Core integ-test / cluster-behavior issues (labels move; confirm on the tracker).
gh issue list --repo opensearch-project/OpenSearch --label "good first issue" --state open
gh issue list --repo opensearch-project/OpenSearch --label "flaky-test" --state open
gh issue list --repo opensearch-project/OpenSearch --label "Cluster Manager" --state open
gh issue list --repo opensearch-project/OpenSearch --label "distributed" --state open
gh issue list --repo opensearch-project/OpenSearch --search "test coverage in:title is:open"

Representative patterns to look for:

• "Add an integration test for X behavior." A change landed without an *IT that exercises the multi-node path. Reproduce the gap (delete the test mentally; what could regress?), write the OpenSearchIntegTestCase that asserts the end state, and PR it. Locate a sibling to mirror with rg -l "extends OpenSearchIntegTestCase" server/src/internalClusterTest/java.
• "Test assumes a fixed node count / fails on randomized cluster size." Find the hard-coded numDataNodes or active-shard count, make the assertion size-agnostic (Exercise 3), and add a seed that exposed it.

Planted bug: In your IntegLabIT, change the create call to number_of_replicas: 3 while keeping numDataNodes = 3. Run it: ensureGreen now hangs and times out (no legal home for the 4th copy — same-shard decider). That hang is the bug signal. Now fix it by either lowering replicas to 2 or starting a 4th data node, and add an explicit assertThat(health.getUnassignedShards(), equalTo(0)) after ensureGreen that would have caught a silent under-allocation. Revert when done.

Etiquette: Claim an issue with a comment before working it, reproduce the gap first, and ship a test plus a CHANGELOG.md entry, DCO-signed (git commit -s). See community interaction and the level-2 PR lab.

Validation / Self-check

1. Why is the task :server:internalClusterTest and not :server:test? What makes Gradle route IntegLabIT to the right source set?
2. What exactly does ensureGreen("idx") wait for, and what would make it never return?
3. Why must you refresh(index) (or use IMMEDIATE) before searching in an integ test? What is actually happening when you refresh?
4. With number_of_shards: 2 and number_of_replicas: 1 on 3 data nodes, how many active shards do you expect, and why?
5. What's the difference between client() and internalCluster().client(nodeName)? When would you need the second?
6. When should you choose Scope.TEST over the default Scope.SUITE, and what's the cost?
7. You changed an assertion to a wrong value and the test still passed. What does that tell you, and what would you check?

When your test stands up a green 3-node cluster, indexes and queries documents deterministically, and you've proven it can fail, move on to Lab 5.2: Add a Missing Unit Test.