Level 5: Testing and Debugging

Up to now you have read the engine and made small, surgical changes. From here on, no change ships without a test, and most of your time as a contributor is spent in the test framework, not in server/src/main. This level makes the OpenSearch test framework your home: its taxonomy, its randomization, its in-JVM clusters, and the debugging tooling you use when a test — or production — misbehaves.

This is the OpenSearch analog of learning how a mature project defends itself. A maintainer's first question on almost every PR is "where's the test?" The second is "is it deterministic?" By the end of this level you can answer both without being asked.

Note: OpenSearch inherited a famously rigorous, randomized test framework from Elasticsearch (which inherited the Carrotsearch Randomized Testing runner from Lucene). Tests run with a random seed by default; the same test exercises different inputs on every run. This finds bugs ordinary example-based tests miss — and it is also the source of "flaky" tests, which you will learn to fix rather than mute.

Learning Objectives

By the end of Level 5 you must be able to:

1. Name every base test class in the OpenSearch test framework and choose the right one for a given change (unit vs single-node vs multi-node vs REST vs serialization vs BWC).
2. Explain randomized testing: the RandomizedRunner, the seed, randomAlphaOfLength/ randomIntBetween, and how to reproduce a failure from a printed -Dtests.seed=....
3. Write and run an OpenSearchIntegTestCase that stands up a multi-node InternalTestCluster, indexes data, and asserts cluster health and query results.
4. Write a unit test, including an equals/hashCode contract test and a serialization round-trip for a Writeable via AbstractWireSerializingTestCase.
5. Run the right Gradle test task for each kind of test and read the HTML/.bin reports it produces.
6. Debug a test or a running node: --debug-jvm, TRACE logging on a single package, and assertBusy for asynchronous assertions.
7. Reproduce, diagnose, and correctly fix a flaky test — and explain why adding Thread.sleep is not a fix.

The Test Framework Taxonomy

Everything lives under test/framework/ (the published org.opensearch.test framework artifact) and in each module's src/test/java. The single most valuable skill in this level is picking the cheapest test class that still proves your change. Reach for a heavier base only when a lighter one cannot exercise the behavior.

find test/framework/src/main/java -name "OpenSearch*TestCase.java" | sort
find test/framework/src/main/java -name "InternalTestCluster.java" -o -name "MockNode.java"

Two more you must recognize even before you write them:

• REST-YAML tests (yamlRestTest). The shared, language-agnostic API tests under rest-api-spec/ (and per-module src/yamlRestTest/resources/rest-api-spec/test/...). They are YAML files of do:/match: steps run by a Java harness (OpenSearchClientYamlSuiteTestCase). Every OpenSearch client project reuses them. Run with ./gradlew :rest-api-spec:yamlRestTest or a module's :yamlRestTest.
• Backward-compatibility (BWC) tests in qa/. These start a cluster on an old version and an upgraded node, asserting data and APIs survive an upgrade. They key off bwcVersion. You will rarely write these early, but you must know they exist and gate the release.

flowchart TD
    Q{What does your change touch?} --> L1[Pure logic / parsing / equality]
    Q --> L2[A Writeable wire/XContent type]
    Q --> L3[One node: index/shard/engine]
    Q --> L4[Multiple nodes: replication/allocation/recovery]
    Q --> L5[The REST/API contract]
    Q --> L6[Analysis: tokenizer/filter/analyzer]
    L1 --> C1[OpenSearchTestCase]
    L2 --> C2[AbstractWireSerializingTestCase /<br/>AbstractSerializingTestCase]
    L3 --> C3[OpenSearchSingleNodeTestCase]
    L4 --> C4[OpenSearchIntegTestCase + @ClusterScope]
    L5 --> C5[OpenSearchRestTestCase + yamlRestTest]
    L6 --> C6[OpenSearchTokenStreamTestCase]

Randomized Testing: The Seed Is the Test

OpenSearch tests run under Carrotsearch RandomizedRunner (@RunWith is wired into the base classes). Every test method gets a deterministic source of randomness derived from a master seed. Instead of hand-picking inputs, you ask the framework for them:

String name   = randomAlphaOfLength(randomIntBetween(1, 20));   // varies every run
int shards    = randomIntBetween(1, 5);
boolean flag  = randomBoolean();
TimeValue ttl = randomTimeValue();
Version v     = VersionUtils.randomVersion(random());           // random known version (BWC)

The same method body exercises a different name, shards, flag on each run. Over CI's thousands of runs, the input space gets covered far more than any example you'd write by hand.

When a randomized test fails, the framework prints a reproduce line. It looks like:

REPRODUCE WITH: ./gradlew ':server:test' --tests "org.opensearch.cluster.SomeTests.testThing" \
  -Dtests.seed=ABCDEF0123456789 -Dtests.locale=de-DE -Dtests.timezone=America/New_York

Copy that line verbatim. The seed pins all randomness (the random* values, the iteration order, even the JVM locale/timezone) so the failure reproduces exactly. This is the single most important habit in OpenSearch testing: never debug a randomized failure without its seed.

Warning: Do not "fix" a randomized test by narrowing its inputs (e.g. replacing randomIntBetween(1, 100) with a constant) to dodge a failure. That hides the bug the randomization found. Either the production code is wrong, or your test made an unwarranted assumption — fix the real cause. See Lab 5.4.

InternalTestCluster and MockNode

OpenSearchIntegTestCase runs a real cluster inside the test JVM. The machinery is InternalTestCluster: it starts N Nodes (actually MockNodes — Node subclasses that swap in test implementations like MockTransportService, deterministic random, and bounded thread pools) and wires them into a working cluster you drive through internalCluster() and client().

grep -n "startNode\|startNodes\|stopRandomDataNode\|client()\|ensureGreen\|nodesInclude\|getMasterName\|getClusterManagerName" \
  test/framework/src/main/java/org/opensearch/test/InternalTestCluster.java | head -40

What you control from a test:

@ClusterScope controls cluster reuse, which is the single biggest lever on integ-test speed and isolation:

Covered hands-on in Lab 5.1 and pushed further (failover) in Lab 5.3.

Gradle Test Tasks

There is a task per kind of test. Picking the wrong one wastes minutes (or hours).

Scoping is mandatory for fast iteration:

# One class
./gradlew :server:test --tests "org.opensearch.cluster.ClusterStateTests"
# One method
./gradlew :server:test --tests "org.opensearch.index.engine.InternalEngineTests.testVersioningNewIndex"
# A glob
./gradlew :server:test --tests "*MaxRetry*"
# An integ test
./gradlew :server:internalClusterTest --tests "org.opensearch.cluster.SpecificClusterManagerNodesIT"

Note: Naming convention is load-bearing. *Tests.java are picked up by test; *IT.java by internalClusterTest. Put an integ test in src/test and it simply won't run; put a unit test named *IT and it won't run under test. Match the source set to the suffix.

Reading the Reports

Every Gradle test task writes machine- and human-readable reports. When CI is red and the console is a wall of text, the report is faster than scrolling.

# HTML report (open in a browser) — index, per-class pages, stdout/stderr captured
open server/build/reports/tests/test/index.html          # macOS
xdg-open server/build/reports/tests/test/index.html       # Linux

# Raw per-test XML (what CI parses)
ls server/build/test-results/test/*.xml

# Find the exact reproduce line from a failed run
grep -rn "REPRODUCE WITH" server/build/test-results/ server/build/reports/ 2>/dev/null | head

The HTML page for a failed test shows the assertion, the full stack trace, and the captured stdout/stderr — which is where the REPRODUCE WITH line and any test logging end up.

Debugging

Three tools cover almost every test/debug situation.

1. Attach a debugger to a running node — --debug-jvm:

# Pauses the node JVM waiting for a debugger on port 5005, then runs a single node from source.
./gradlew run --debug-jvm
# Attach IntelliJ "Remote JVM Debug" to localhost:5005. Set breakpoints in IndexShard,
# InternalEngine, TransportShardBulkAction, etc., then send a curl and step through.

To debug a test under a debugger, use the test-task variant:

./gradlew :server:test --tests "*InternalEngineTests.testSimple" --debug-jvm

2. TRACE logging on exactly the package you care about (cluster-wide, dynamically):

curl -s -XPUT 'localhost:9200/_cluster/settings' -H 'Content-Type: application/json' -d '{
  "transient": { "logger.org.opensearch.index.engine": "TRACE",
                 "logger.org.opensearch.index.translog": "TRACE" }
}'
# ...do the operation, read the logs, then turn it back off:
curl -s -XPUT 'localhost:9200/_cluster/settings' -H 'Content-Type: application/json' -d '{
  "transient": { "logger.org.opensearch.index.engine": null,
                 "logger.org.opensearch.index.translog": null }
}'

Inside a test, set it on the class with @TestLogging("org.opensearch.index.engine:TRACE").

3. assertBusy for asynchronous assertions. Most OpenSearch work is asynchronous (refresh, allocation, recovery, cluster-state apply). Do not assert immediately, and do not Thread.sleep. Poll until true (with a timeout):

assertBusy(() -> {
    long count = client().prepareSearch("idx").setSize(0).get().getHits().getTotalHits().value;
    assertEquals(3L, count);
}, 30, TimeUnit.SECONDS);

assertBusy retries the assertion with backoff until it passes or the timeout fires. It is the deterministic alternative to "sleep and hope" — the distinction at the heart of Lab 5.4.

Key Classes Quick Reference

The Labs

Deliverables

You must demonstrate all of the following before advancing to Level 6:

• An OpenSearchIntegTestCase you wrote that starts a multi-node InternalTestCluster, indexes docs, ensureGreens, and asserts a query result (Lab 5.1).
• A unit test for a previously untested class, including equals/hashCode coverage and a serialization round-trip via AbstractWireSerializingTestCase (Lab 5.2).
• A multi-node integ test that stops a data node and asserts replica promotion / shard reallocation with ensureGreen + assertBusy (Lab 5.3).
• A reproduced, diagnosed, and correctly fixed flaky test — un-muted, with the race explained (Lab 5.4).
• From memory: the base-class taxonomy table, the meaning of -Dtests.seed, and why assertBusy beats Thread.sleep.

Common Mistakes

How to Verify Success

# 1. You can scope and reproduce any test.
./gradlew :server:test --tests "*ClusterStateTests" -Dtests.seed=DEADBEEFDEADBEEF

# 2. You can run an integ test and read its report.
./gradlew :server:internalClusterTest --tests "*ClusterManager*IT"
open server/build/reports/tests/internalClusterTest/index.html

# 3. You can turn on TRACE for one package and back off again.
curl -s -XPUT localhost:9200/_cluster/settings -H 'Content-Type: application/json' \
  -d '{"transient":{"logger.org.opensearch.index.engine":"TRACE"}}'
curl -s -XPUT localhost:9200/_cluster/settings -H 'Content-Type: application/json' \
  -d '{"transient":{"logger.org.opensearch.index.engine":null}}'

When you can choose the right base class, reproduce any failure from its seed, run the matching Gradle task, and fix flakiness without sleeping, you are ready for Level 6: Indexing Path and Storage Engine.