Lab 1.2: Run Unit and Integration Tests

Background

OpenSearch has one of the most thorough test suites of any open-source distributed system, and a PR is not credible unless it passes the relevant slice of it. The suite is built on Randomized Testing (the same RandomizedRunner Lucene uses): every run picks a random seed, and the test exercises randomized inputs (field types, document counts, cluster topologies, serialization round-trips). This finds bugs ordinary fixed-input tests never would — and it means a failure must be reproduced with the seed it failed under, not dismissed as flaky.

This lab teaches you to run unit and integration tests, scope them tightly, reproduce a randomized failure deterministically, run the precommit and full-check gates, and read the HTML reports under build/reports/tests.

Why This Lab Matters for Contributors

• CI's gradle-check runs a superset of what you will run here; passing locally first turns a five-round review into one.
• Knowing how to scope tests with --tests is the difference between a 5-second feedback loop and a 90-minute one.
• Reproducing a randomized failure from its seed is the core debugging skill for OpenSearch — you will use it constantly in Level 5 and Level 8.
• Reading the test reports tells you what failed and why, not just that something did.

Prerequisites

• Lab 1.1 complete: a clean ./gradlew assemble.
• Familiarity with the project paths printed by ./gradlew projects.

Step-by-Step Tasks

Step 1: The Test Types

OpenSearch tests come in tiers. Know which tier you are running before you run it.

The two you will touch most at Level 1 are OpenSearchTestCase (fast, no cluster) and OpenSearchIntegTestCase (slow, real cluster). The distinction is covered in depth in Level 5.

# See how many of each style live in :server (rough proxy via base class):
grep -rln "extends OpenSearchTestCase"            server/src/test | wc -l
grep -rln "extends OpenSearchSingleNodeTestCase"  server/src/test | wc -l
grep -rln "extends OpenSearchIntegTestCase"       server/src/internalClusterTest 2>/dev/null | wc -l

Step 2: Run a Single Unit-Test Class

Never start with the whole :server:test task — it runs thousands of classes. Scope with --tests.

# A small, fast, dependency-free class — good first run:
./gradlew :server:test --tests "org.opensearch.common.UUIDTests"

--tests accepts a fully-qualified class name. You can also use wildcards and pick a single method:

# All test classes in a package:
./gradlew :server:test --tests "org.opensearch.cluster.*"

# A single method on a class:
./gradlew :server:test --tests "org.opensearch.cluster.ClusterStateTests.testToXContent"

# Every test class whose name ends in "Tests" under a subtree:
./gradlew :server:test --tests "org.opensearch.index.engine.*Tests"

Tip: ./gradlew :server:test --tests "X" will recompile changed sources first. If you only changed test code, the main-source compile is skipped — the loop stays fast.

Step 3: Run Integration (In-JVM Cluster) Tests

Integration tests spin up a real multi-node cluster inside the JVM via InternalTestCluster. They live under a separate source set and run via a separate task:

# One integration-test class (note the *IT naming convention):
./gradlew :server:internalClusterTest --tests "org.opensearch.cluster.SimpleClusterStateIT"

These are much slower (seconds-to-minutes per class) because each spins up nodes, allocates shards, and tears everything down. Run them scoped; never blanket-run :server:internalClusterTest.

Step 4: Randomized Testing and -Dtests.seed

Every test run uses a random seed. When a test fails, Gradle prints a reproduction line that pins the seed (and other randomization) so the failure is deterministic:

REPRODUCE WITH: ./gradlew ':server:test' --tests "org.opensearch.Foo.testBar" \
  -Dtests.seed=DEADBEEFCAFE -Dtests.locale=en-US -Dtests.timezone=UTC ...

To reproduce, copy that line verbatim:

./gradlew ':server:test' --tests "org.opensearch.SomeTest.testThing" -Dtests.seed=DEADBEEFCAFE

You can also force a seed to make a run deterministic, or iterate a flaky test to flush out the failing seed:

# Force a specific seed:
./gradlew :server:test --tests "org.opensearch.SomeTest" -Dtests.seed=DEADBEEFCAFE

# Run the same test many times with fresh random seeds to surface flakiness:
./gradlew :server:test --tests "org.opensearch.SomeTest" -Dtests.iters=50

Warning: A randomized failure is not automatically "flaky." If it reproduces under its seed, it is a real bug that only some inputs trigger. Only when it fails on one seed and passes on the same seed on re-run is it genuinely non-deterministic. Flaky-test handling — muting with @AwaitsFix(bugUrl=...), never @Ignore — is Level 5 material.

Step 5: Run the Precommit Gate

precommit is the static-analysis half of the gate: checkstyle, forbidden APIs, license/SPDX headers, dependency checks, loggerUsageCheck, and more. It is fast relative to the tests and CI runs it on every PR.

./gradlew precommit

Pair it with formatting:

./gradlew spotlessApply      # fix formatting
./gradlew spotlessJavaCheck  # verify formatting (what CI checks)

Step 6: The Full Check (Know What It Is, Run It Sparingly)

check is the everything gate: unit tests + precommit + integration tests for the project(s) you target. It is long-running. Scope it to a project, and only run it before a substantial PR.

# The full gate for the server project (long — minutes to tens of minutes):
./gradlew :server:check

# The whole repo (very long — reserve for pre-release / large changes):
# ./gradlew check

For most PRs you will run, in order: the affected --tests, then spotlessApply, then precommit. That mirrors what review will demand.

Step 7: Read the Test Reports

Whether a test passes or fails, Gradle writes an HTML report. For a failure you want the report, not just the console tail.

# Reports live under each project's build/reports/tests/<taskName>/index.html
find server/build/reports/tests -name index.html
# e.g. server/build/reports/tests/test/index.html
#      server/build/reports/tests/internalClusterTest/index.html

# Open it (macOS):
open server/build/reports/tests/test/index.html

The report gives you, per class and per method: pass/fail/skip counts, the full stack trace of each failure, and the captured stdout/stderr (where the test logged the reproduction line and any cluster diagnostics). The raw JUnit XML is alongside, under server/build/test-results/test/*.xml — useful for grepping or CI parsing.

# Quickly find which methods failed without opening a browser:
grep -rl 'testcase' server/build/test-results/test/*.xml >/dev/null 2>&1
grep -rE '<(failure|error)' server/build/test-results/test/*.xml | head

Implementation Requirements

This lab has no code to implement. Deliverables:

1. One OpenSearchTestCase-style unit class run green via --tests.
2. One *IT integration class run green via :server:internalClusterTest --tests.
3. A demonstrated seed reproduction: take any test's printed REPRODUCE WITH line and re-run it.
4. A clean ./gradlew precommit.
5. The path to your build/reports/tests/test/index.html and a one-line description of what it shows.

Troubleshooting

A test "passes locally but fails in CI" (or vice versa)

Almost always a seed difference. Reproduce CI's failure by copying the -Dtests.seed=... (and -Dtests.locale/-Dtests.timezone) from the CI log into your local command. Randomization includes locale and timezone — a locale-sensitive bug will only appear under certain -Dtests.locale values.

OutOfMemoryError during :server:test

Test JVMs are forked. Raise the daemon/test heap via org.gradle.jvmargs in gradle.properties (see Lab 1.1), and scope your run with --tests instead of running the whole suite.

Integration test hangs or leaks threads

OpenSearchIntegTestCase has aggressive leak detection; a hang at teardown usually means the test (or your change) left a thread/Closeable open. Read the report's captured output — the framework names the leaked resource. This is exactly the signal Level 5 teaches you to act on.

"Tests are UP-TO-DATE and won't re-run"

Gradle caches test results. Force a re-run:

./gradlew :server:test --tests "org.opensearch.common.UUIDTests" --rerun-tasks

precommit fails on a file you did not touch

Confirm it is pre-existing (some failures depend on branch state). Run on a clean checkout of main to establish the baseline:

git stash && ./gradlew precommit ; git stash pop

If main is clean and your change introduced it, the failure is yours to fix.

Expected Output

A passing scoped unit run:

> Task :server:test

org.opensearch.common.UUIDTests > testRandomUUID PASSED
org.opensearch.common.UUIDTests > testTimeBasedUUID PASSED

BUILD SUCCESSFUL in 38s

A failure prints the all-important reproduction line:

org.opensearch.cluster.SomeTest > testThing FAILED
    java.lang.AssertionError: expected:<3> but was:<2>
        at org.opensearch.cluster.SomeTest.testThing(SomeTest.java:91)

REPRODUCE WITH: ./gradlew ':server:test' --tests "org.opensearch.cluster.SomeTest.testThing" \
  -Dtests.seed=A1B2C3D4 -Dtests.locale=en-US -Dtests.timezone=UTC

BUILD FAILED in 41s

Stretch Goals

1. Find a serialization round-trip test and run it. These verify wire/XContent BWC and are the guardians of the protocol you must respect in Level 9:

   grep -rln "extends AbstractWireSerializingTestCase" server/src/test | head
   # pick one and run it with --tests
   

2. Stress a single test for flakiness. Run a chosen test 100 times with fresh seeds:

   ./gradlew :server:test --tests "org.opensearch.common.UUIDTests" -Dtests.iters=100
   

3. Run a module's REST-YAML tests and watch a black-box test of the REST API:

   ./gradlew :modules:reindex:yamlRestTest --tests "*" 2>&1 | tail -30
   

4. Diff two runs' reports. Run a test twice with two explicit different seeds and compare what the report captured — see the randomized inputs differ.

Coding Exercises

Running tests is half the skill; writing them is the other half, and it is what every PR demands. These exercises have you author real OpenSearchTestCase/OpenSearchIntegTestCase tests against existing :server code. Locate every class with find/rg first — never cite a line number you did not just see.

1. (warm-up) Add a method to an existing unit test. Find UUIDTests (find server -name "UUIDTests.java" -path "*/test/*") and add a testUUIDsAreUnique method that generates, say, 1000 UUIDs with the same generator the class already exercises and asserts the set size equals 1000. Use randomIntBetween/randomAlphaOfLength from the base class where natural. Verify: ./gradlew :server:test --tests "org.opensearch.common.UUIDTests.testUUIDsAreUnique" passes.

2. (warm-up) Make a randomized test reproducible, then assert on the random input. Take any OpenSearchTestCase you ran in Step 2 and add a method that pulls a value with randomLong(), prints the active seed via the framework, and asserts a property that holds for all inputs (e.g. Math.abs((double) x) is non-negative — pick a real invariant for the class under test). Force a seed with -Dtests.seed=DEADBEEF and confirm the test is deterministic across two runs. Verify: two runs with the same -Dtests.seed produce identical captured output in the HTML report.

3. (core) Parametrize a test over inputs. Pick a small pure function in :server or :libs:core (e.g. something in org.opensearch.common — locate with rg -n "public static" libs/core/src/main/java/org/opensearch/core/common | head). Write a single test* method that loops over a table of {input, expected} cases and asserts each, so one method covers many cases. Verify: the scoped --tests run is green; deliberately break one expected value and confirm the failure message names the offending case.

4. (core) Write a serialization round-trip test. Stretch Goal 1 had you find an AbstractWireSerializingTestCase; now write one. Pick a simple Writeable in :server (grep -rln "implements Writeable" server/src/main/java | head and choose one with a small, obvious constructor). Create FooTests extends AbstractWireSerializingTestCase<Foo> implementing createTestInstance() (build a random instance) and instanceReader() (the StreamInput constructor reference). Verify: ./gradlew :server:test --tests "*FooTests" passes — you have just proven the object survives a StreamOutput → StreamInput round-trip, the guard you will audit in Lab 2.4.

5. (core) Write a single-node integration test. Using OpenSearchSingleNodeTestCase (find server -name "*.java" -path "*/test/*" | xargs grep -l "extends OpenSearchSingleNodeTestCase" | head for an example to copy), write a test that creates an index, indexes one document, refreshes, and asserts _count (via the test client) returns 1. Verify: it runs green under :server:test. This is Lab 1.3's curl flow expressed as a deterministic test.

6. (advanced) Advanced challenge — reproduce a seed, then write a regression test that pins it. Run a randomized test with -Dtests.iters=200 on a class you choose until you can describe one specific input that exercises an interesting branch (or deliberately weaken an assertion to force a failure and capture its REPRODUCE WITH line). Then write a new, non-randomized test method that hard-codes that exact input and asserts the correct behavior — the way a real bug fix locks in a regression. Verify: the new method passes without any -Dtests.seed (it must be deterministic by construction), and document the seed that led you to it in the method's Javadoc. This is the exact muscle Level 5 and Level 8 demand: a flaky/seed failure becomes a permanent, deterministic guard.

Issues to Practice On

The most beginner-friendly and high-value testing work in OpenSearch is fixing flaky tests — real issues with a dedicated label. The right repo is opensearch-project/OpenSearch.

# Flaky tests are the canonical first testing contribution:
gh issue list --repo opensearch-project/OpenSearch --label "flaky-test" --state open

# And good-first-issues that are test-shaped:
gh issue list --repo opensearch-project/OpenSearch --label "good first issue" --state open

# Untriaged bugs often hide a missing-test gap:
gh issue list --repo opensearch-project/OpenSearch --label "bug" --label "untriaged" --state open

# Labels drift — confirm on the tracker:
gh label list --repo opensearch-project/OpenSearch | grep -iE "flaky|test"

Representative issue patterns for this subsystem:

• A flaky test. A flaky-test issue pastes a CI failure with a -Dtests.seed=... reproduction line. Your job: copy that seed (and -Dtests.locale/-Dtests.timezone) and reproduce locally; if it reproduces deterministically it is a real bug under that input — locate the racy/order-dependent code with rg and fix it; if it only fails intermittently, the test itself is at fault (often a Thread.sleep, a fixed timeout, or an ordering assumption — replace with assertBusy). Prove the fix with -Dtests.iters=200.
• A missing-coverage bug. A bug report describes behavior with no guarding test. Reproduce, write the failing test first, then fix. The test is the deliverable as much as the fix.

Planted-bug drill. Plant a flaky test, watch it fail under iteration, then harden it:

1. In a scratch test class, write a method that does assertEquals(0, System.nanoTime() % 7) (a pseudo-random failure). Run it with -Dtests.iters=50 and watch it fail on some iterations and pass on others — that is the signature of true non-determinism (versus a seed-stable bug).
2. Now plant a sleep-based race: write triggerSomethingAsync(); Thread.sleep(50); assertTrue(done), run it under load (-Dtests.iters=100) and watch it flake. Replace the Thread.sleep with assertBusy(() -> assertTrue(done)) and confirm it is stable. Add a comment naming why assertBusy fixed it — this is precisely the anti-pattern you will flag as a reviewer in Lab 2.4.

Etiquette: claim the issue before working it, reproduce first (paste the seed in your comment), and every PR needs the new/updated test, a CHANGELOG entry, and a DCO Signed-off-by (git commit -s). Mechanics in Lab 2.2; norms in community interaction.

Validation / Self-check

You are done when you can answer these without notes:

1. What does --tests accept, and how do you scope to a single method? A whole package?
2. Why is :server:internalClusterTest separate from :server:test, and which base class backs it?
3. What is -Dtests.seed for, and where do you find the value to reproduce a given failure?
4. When is a randomized failure a real bug, and when is it genuinely non-deterministic?
5. What does precommit check that the test task does not?
6. Where do the HTML report and the raw JUnit XML for a :server:test run live?
7. Which three commands, in order, would you run before opening a PR that touches :server?

See the testing internals in depth in Level 5. Next: Lab 1.3 — Launch a Single-Node Cluster and Index Data.