Step 2: Reproduction

A bug you cannot reproduce on demand is a bug you cannot fix with confidence. The reproducer is the most important artifact in the entire Capstone — it is what makes Step 4 (root cause) provable, what makes Step 5 (implementation) verifiable, and what a maintainer will look for first when they review your PR.

The goal of this step is a deterministic reproducer: something that fails on main without your change, every single run, and that you can re-run in seconds. Ideally it is a JUnit test you will ship in the PR. At minimum it is a recorded curl/REST-YAML sequence that misbehaves the same way every time.

The rule that governs this step: the reproducer must fail on main and pass after the fix. If it does not fail on main, you are testing the wrong thing. If it does not pass after the fix, you have not fixed the bug.

Two Kinds of Reproducer

Pick the lowest-cost level that reliably reproduces the bug.

Always start manual to see the failure, then promote it to code. A manual curl repro is fine for Step 2's exploration, but the PR needs an automated test (Step 6). Write the automated version as early as you can — it is your regression guard for the rest of the Capstone.

Manual Reproduction with ./gradlew run

Bring up a single node from source and reproduce by hand first. This is where you confirm the bug and capture the exact request/response.

# Launch a debuggable single-node cluster from source. REST on :9200.
./gradlew run

# In another shell:
curl -s localhost:9200/_cluster/health?pretty

Now run the exact sequence from the issue. For example, a hypothetical aggregation-edge bug:

curl -s -XPUT localhost:9200/repro -H 'Content-Type: application/json' -d '
{ "settings": { "number_of_shards": 1, "number_of_replicas": 0 } }'

curl -s -XPOST 'localhost:9200/repro/_doc?refresh=true' \
  -H 'Content-Type: application/json' -d '{ "value": 10 }'

curl -s 'localhost:9200/repro/_search?pretty' -H 'Content-Type: application/json' -d '
{
  "size": 0,
  "aggs": {
    "by_missing": {
      "terms": { "field": "absent_field", "missing": "N/A", "min_doc_count": 0 }
    }
  }
}'

Compare the response to what the issue (and the docs) say it should be. Capture both. If the response is wrong, you have a manual reproducer. Save the exact commands and outputs into capstone-work/repro.md — you will paste them into the PR.

Tip: ./gradlew run defaults to a fresh data directory each run, so your repro starts clean. To keep data across restarts, use ./gradlew run --preserve-data. To attach a debugger (you will, in Step 3), use ./gradlew run --debug-jvm and connect on port 5005.

Symptom vs. trigger conditions

While reproducing, separate two things you will need explicitly in your root-cause doc:

• Symptom — what the user observes. "The by_missing bucket count is 0 when it should be 1," or "the request returns HTTP 500 with a NullPointerException instead of HTTP 400."
• Trigger conditions — the precise circumstances under which the symptom appears. "Only with min_doc_count: 0 and a field that exists in the mapping but in zero documents," or "only on a single-shard index," or "only after a refresh but before a flush."

Bisect the trigger conditions experimentally. Remove missing. Add a second shard. Add a second document. Each variation that makes the symptom disappear teaches you a trigger condition, and trigger conditions are what your test must encode and what your fix must address. A reproducer that only works under your exact lucky setup is a fragile reproducer.

Promote to a JUnit Reproducer

Now write the failing test. Name it after the issue so the intent is obvious and so anyone can find it later: SomeComponentTests with a method testIssueNNNNRepro (or, for an aggregation, add a method to the existing ...AggregatorTests).

A unit-level reproducer (OpenSearchTestCase)

Find the existing test class for the component first — do not invent a new one if a home exists:

# Where do tests for this aggregation live?
find server/src/test -name "*Terms*AggregatorTests.java"
# Where is the production class?
grep -rn "class TermsAggregator" server/src/main/java

A minimal failing unit test reads like this (illustrative — match the real base class and helpers in the file you found):

/*
 * SPDX-License-Identifier: Apache-2.0
 * ... (keep the existing header in the file you are editing)
 */
public class TermsAggregatorTests extends AggregatorTestCase {

    public void testIssueNNNNMissingWithMinDocCountZero() throws Exception {
        // Arrange: a field present in the mapping but in zero matching docs,
        // with `missing` set and min_doc_count = 0 — the trigger conditions.
        MappedFieldType fieldType = new KeywordFieldMapper.KeywordFieldType("absent_field");
        TermsAggregationBuilder agg = new TermsAggregationBuilder("by_missing")
            .field("absent_field")
            .missing("N/A")
            .minDocCount(0);

        try (Directory directory = newDirectory();
             RandomIndexWriter w = new RandomIndexWriter(random(), directory)) {
            w.addDocument(List.of()); // a doc with no value for the field
            try (IndexReader reader = w.getReader()) {
                IndexSearcher searcher = newIndexSearcher(reader);
                StringTerms result =
                    searchAndReduce(searcher, new MatchAllDocsQuery(), agg, fieldType);

                // Assert the CORRECT behavior. This FAILS on main today.
                assertEquals(1, result.getBuckets().size());
                assertEquals("N/A", result.getBuckets().get(0).getKeyAsString());
            }
        }
    }
}

Run only that test:

./gradlew :server:test \
  --tests "org.opensearch.search.aggregations.bucket.terms.TermsAggregatorTests.testIssueNNNNMissingWithMinDocCountZero"

It should fail. That red bar is your Step 2 success criterion. If it passes, your assertion encodes the current (wrong) behavior, not the correct one — fix the assertion, not the code (yet).

An integration-level reproducer (OpenSearchIntegTestCase)

When the bug needs a real cluster — multiple shards, allocation, recovery, cluster-state propagation — use the in-JVM multi-node harness. These live under server/src/internalClusterTest/java/... and run via the :server:internalClusterTest task.

@OpenSearchIntegTestCase.ClusterScope(scope = OpenSearchIntegTestCase.Scope.TEST, numDataNodes = 2)
public class SomeBehaviorIT extends OpenSearchIntegTestCase {

    public void testIssueNNNNRepro() throws Exception {
        internalCluster().startClusterManagerOnlyNode();
        internalCluster().startDataNodes(2);

        createIndex("repro", Settings.builder()
            .put(IndexMetadata.SETTING_NUMBER_OF_SHARDS, 1)
            .put(IndexMetadata.SETTING_NUMBER_OF_REPLICAS, 1)
            .build());
        ensureGreen("repro");

        // ... drive the exact trigger conditions, then:
        assertBusy(() -> {
            ClusterHealthResponse health = client().admin().cluster()
                .prepareHealth("repro").get();
            assertThat(health.getStatus(), equalTo(ClusterHealthStatus.GREEN));
        });
    }
}

./gradlew :server:internalClusterTest --tests "org.opensearch.*.SomeBehaviorIT.testIssueNNNNRepro"

Warning: Never use Thread.sleep to wait for cluster state to settle. Use assertBusy(...), ensureGreen(...), ensureStableCluster(...), or a ClusterStateListener. A sleep-based test is non-deterministic by construction, and reviewers will (correctly) block it. See Step 6.

A REST-YAML reproducer

If the bug is in the REST contract (status code, response shape, error message), the most faithful reproducer is a REST-YAML test under rest-api-spec (run via :rest-api-spec:yamlRestTest):

---
"terms agg with missing and min_doc_count zero":
  - do:
      indices.create:
        index: repro
        body:
          settings:
            number_of_shards: 1
            number_of_replicas: 0
  - do:
      index:
        index: repro
        refresh: true
        body: { value: 10 }
  - do:
      search:
        index: repro
        body:
          size: 0
          aggs:
            by_missing:
              terms: { field: absent_field, missing: "N/A", min_doc_count: 0 }
  - match: { aggregations.by_missing.buckets.0.key: "N/A" }
  - match: { aggregations.by_missing.buckets.0.doc_count: 1 }

Pin Versions and the Random Seed

OpenSearch tests are randomized (RandomizedRunner). A test can pass on one seed and fail on another. For a reproducer, pin the seed so the failure is exact:

# Reproduce a specific failure deterministically.
./gradlew :server:test --tests "...TermsAggregatorTests.testIssueNNNNRepro" \
  -Dtests.seed=DEADBEEFDEADBEEF

When a randomized existing test already exposes the bug intermittently (a flaky-test), the failure log prints the exact reproduction line — copy it verbatim:

REPRODUCE WITH: ./gradlew ':server:test' --tests "...SomeTests.testThing" \
  -Dtests.seed=ABC123 -Dtests.locale=fr-FR -Dtests.timezone=America/Sao_Paulo

Pin the branch/tag too. State which ref you reproduced on. For the Capstone, reproduce on main. If the issue claims a regression from a released version, also reproduce on the relevant tag to confirm it was correct there:

git fetch origin
git checkout main && ./gradlew :server:test --tests "...testIssueNNNNRepro"   # fails
git checkout 2.11.0 && ./gradlew :server:test --tests "...testIssueNNNNRepro"  # passes? -> regression
git checkout main   # back to work

That last experiment is the seed of Step 4's git bisect. If main fails and 2.11.0 passes, you have a regression and a known-good/known-bad pair to bisect between.

Build the Reproduction Report

Save capstone-work/repro.md with everything a reviewer (or future you) needs to reproduce in 60 seconds:

# Reproduction: #NNNN

## Symptom
<one sentence — the observable wrong behavior>

## Trigger conditions
- <condition 1>
- <condition 2>

## Refs
- Fails on: `main` (commit <sha>)
- Passes on: `2.11.0` (regression) | n/a (never worked)
- Seed (if randomized): <seed>

## Manual repro (curl)
<the exact curl sequence + observed vs. expected output>

## Automated repro (the failing test)
- File: server/src/test/.../TermsAggregatorTests.java
- Method: testIssueNNNNMissingWithMinDocCountZero
- Command: ./gradlew :server:test --tests "...testIssueNNNNMissingWithMinDocCountZero"
- Result on main: FAIL (expected 1 bucket, got 0)

Deliverable for Step 2

• A failing automated test (unit, integration, or REST-YAML) that is red on main.
• The exact command to run just that test, recorded.
• Symptom and trigger conditions written down separately.
• The ref(s) you reproduced on, pinned (and the seed, if randomized).
• capstone-work/repro.md complete.

Validation / Self-check

Before advancing to Step 3:

1. Your reproducer fails on main, deterministically, every run.
2. You can run it in seconds with one ./gradlew ... --tests command.
3. You have listed the trigger conditions and verified at least one of them by making the symptom disappear when you remove it.
4. You chose the lowest-cost harness that reliably reproduces (unit over integration over manual).
5. If randomized, you have a pinned seed; if a regression, you have a known-good ref.
6. capstone-work/repro.md is written and someone else could follow it.
7. You did not use Thread.sleep anywhere in the reproducer.

Then go to Step 3: Execution Path Analysis.