Step 7: Validation

Your fix works and your tests are green on your machine. That is not the bar. The bar is: the full local gate is green before you push, so CI on the PR is green on the first run. A red CI run on your first push is not fatal, but it is a tell — it says you pushed before validating, and it costs a reviewer a round-trip.

This step runs the same checks CI will run, locally, first, in increasing order of cost, and produces a validation report you attach to the PR. The rule: you do not push until you have personally seen every gate go green.

What "Green" Means

OpenSearch CI ("gradle-check") is, in essence, ./gradlew check sharded across runners. "Green" is not "my one test passed." It means all of:

You run these from cheapest to most expensive so a fast failure stops you before you spend 40 minutes on check.

The Gate, in Order

1. Format (seconds)

./gradlew spotlessApply        # fix formatting
./gradlew spotlessJavaCheck    # verify; this is exactly what CI runs

If spotlessApply touched files you didn't mean to change, you have unrelated reformatting staged — strip it (see Step 5).

2. Precommit (a few minutes)

./gradlew precommit

precommit is where most first-time PRs fail CI, because it enforces rules you don't think about: an unregistered Setting, a forbidden API (System.currentTimeMillis() instead of the injected clock; raw java.util.Random), a missing SPDX header, a logger used without a guard. Read each failure literally — the task name tells you which rule (forbiddenApisMain, checkstyleMain, licenseHeaders, loggerUsageCheck).

Here is what a real precommit failure looks like. Say you reached for new Random() to generate a jitter value. forbiddenApisMain fails:

> Task :server:forbiddenApisMain FAILED
Forbidden method invocation: java.util.Random#<init>() [Use
org.opensearch.common.Randomness#get for reproducible sources of randomness]
  in org.opensearch.search.aggregations.bucket.terms.TermsAggregator
  (TermsAggregator.java)

BUILD FAILED
Execution failed for task ':server:forbiddenApisMain'.

The message is the fix: OpenSearch bans the JDK's non-reproducible randomness so tests stay deterministic under a seed. Replace it with the sanctioned helper:

-import java.util.Random;
-...
-long jitter = new Random().nextInt(100);
+import org.opensearch.common.Randomness;
+...
+long jitter = Randomness.get().nextInt(100);

The forbidden signatures are not magic — they live in buildSrc/src/main/resources/forbidden/ (e.g. opensearch-all-signatures.txt, jdk-signatures.txt), each entry pairing a banned method with the replacement message you saw in the failure. Confirm a rule with a grep when a message surprises you:

grep -rn "java.util.Random" buildSrc/src/main/resources/forbidden/

A checkstyleMain failure reads the same way but names a style rule — the most common is a wildcard import, which OpenSearch forbids outright:

> Task :server:checkstyleMain FAILED
[ERROR] TermsAggregator.java: Using the '.*' form of import should be avoided
  - org.opensearch.search.aggregations.* [AvoidStarImport]

Fix it by expanding the star import to the exact classes you use (your IDE's "optimize imports" does this) — never by suppressing the check.

3. Affected-module tests (minutes)

Don't run the whole world yet. Run the modules your diff touches. Find them from the diff:

git diff origin/main --stat        # which dirs changed?

Then scope to those Gradle projects:

# core search/agg change:
./gradlew :server:test --tests "org.opensearch.search.aggregations.*"

# if you touched a module/ or plugins/ project, run that project too, e.g.:
./gradlew :modules:reindex:test

Always include the class you changed and its whole package — your fix can break a sibling test that exercises the same code path.

There are two ways to scope a run, and you want both in your fingers:

• Gradle's --tests filter selects test classes/methods by glob. It is the cleaner syntax for "run this class" or "run this package":

  # one class:
  ./gradlew :server:test --tests "org.opensearch.search.aggregations.bucket.terms.TermsAggregatorTests"
  # one method:
  ./gradlew :server:test --tests "*.TermsAggregatorTests.testMissingWithMinDocCountZeroOnEmptyShard"
  # a whole package:
  ./gradlew :server:test --tests "org.opensearch.search.aggregations.*"
  

• The randomized-testing properties -Dtests.class / -Dtests.method are the OpenSearch test framework's own knobs, and they pair with -Dtests.seed and -Dtests.iters — which is exactly what you need to reproduce a CI failure or hammer a new test for flakiness:

  # reproduce a specific seed the framework printed:
  ./gradlew :server:test -Dtests.class="*.TermsAggregatorTests" \
    -Dtests.method="testMissingWithMinDocCountZeroOnEmptyShard" -Dtests.seed=DEADBEEF
  # run the new test 50 times to prove determinism (note the trailing glob on
  # the method — iterations are suffixed testFoo[0], testFoo[1], …):
  ./gradlew :server:test -Dtests.iters=50 -Dtests.class="*.TermsAggregatorTests" \
    -Dtests.method="testMissing*"
  

The :server:test prefix is the module target — the Gradle project path. ./gradlew test at the root would try to run every module's tests; naming the project (:server:test, :modules:reindex:test, :plugins:analysis-icu:test) runs only that project. Match the project to where your diff lives: a change under server/ is :server:test; a change under modules/reindex/ is :modules:reindex:test. Confirm the exact project path from the module's directory — the Gradle path mirrors the source tree with : for /.

4. Integration / REST tests (tens of minutes)

If your bug or fix touches cluster behavior or the REST contract:

./gradlew :server:internalClusterTest --tests "org.opensearch.cluster.*"
./gradlew :rest-api-spec:yamlRestTest

5. The full check scope (long — run once before push)

./gradlew check

This is the closest local mirror of CI. It is long (tens of minutes to over an hour depending on hardware) — run it once, after the cheaper gates are green, right before you push. Tips:

• Use --build-cache (and Gradle's local cache) so unchanged modules aren't rebuilt.
• If check fails in a module you never touched, suspect a flaky test before suspecting your change: re-run just that test with its printed -Dtests.seed=.... If it fails deterministically on main too (stash your fix and check), it's a pre-existing flake — note it; it is not yours to fix here.
• You can scope check to the heavy parts if hardware is limited: ./gradlew precommit :server:test :server:internalClusterTest is a strong proxy for most server-only fixes.

Note: OpenSearch CI on a PR is triggered by maintainers/automation (gradle-check) and reported as a status check on the PR. You cannot make CI green by wishing; you make it green by having run ./gradlew check locally first. The single best predictor of first-pass green CI is "I ran check and read every line of output."

Reading CI on the PR

How gradle-check is triggered

gradle-check is not a GitHub Action that runs on your laptop's schedule — it is a job on OpenSearch's public Jenkins CI (build.ci.opensearch.org) that runs ./gradlew check sharded across runners and reports back as the gradle-check status check on your PR. Two facts about when it runs shape your workflow:

• For a new or first-time contributor, CI does not auto-run on a stranger's code. A maintainer approves/triggers the run (a workflow-run approval, or a comment that kicks Jenkins). Until then gradle-check sits pending — that is not a failure, it is waiting for a human. A polite "ready for CI when you have a moment" on the PR is fair game.
• Once you are an established contributor, it runs automatically on push. Either way, you re-trigger it by pushing a new commit (or a maintainer re-runs it via the GitHub UI). There is no "retry" button you own as a contributor; a fresh push is the retry.

This is exactly why Step 7 exists: you cannot iterate on CI cheaply (each run is a maintainer round-trip or a full remote build), so you front-load every gate locally. The single best predictor of first-pass green CI is "I ran check and read every line."

When it's red

Once the PR is up (Step 8), the gradle-check status appears on the PR's Checks tab. When it's red:

1. Open the failing check's logs (click "Details" next to gradle-check).

2. Find the failing task and the REPRODUCE WITH: line the test framework prints — it includes the seed, locale, and timezone:

   REPRODUCE WITH: ./gradlew ':server:test' --tests "org.opensearch.x.YTests.testZ" \
     -Dtests.seed=ABC123 -Dtests.locale=fr-FR -Dtests.timezone=America/Sao_Paulo
   

3. Run that exact line locally. If it reproduces, it's yours — fix it. If it doesn't reproduce and the same test is flaky on main, it's a pre-existing flaky-test; comment on the PR linking the flaky-test issue and re-trigger CI (a maintainer can re-run, or a gradle-check retry).

Do not push speculative "maybe this fixes CI" commits. Reproduce locally with the seed first, then push a fix you've verified.

The Validation Report Artifact

Produce capstone-work/validation.md — a record of exactly what you ran and what you saw. This goes into (or is summarized in) the PR description; it is the single most reassuring thing a reviewer can read.

# Validation report: #NNNN

## Checkout
- Branch: fix/NNNN-terms-missing-min-doc-count
- Based on: origin/main @ <sha>

## Gates run (all green)
| Gate | Command | Result |
|---|---|---|
| Spotless | `./gradlew spotlessJavaCheck` | PASS |
| Precommit | `./gradlew precommit` | PASS |
| Unit (changed pkg) | `./gradlew :server:test --tests "...aggregations.bucket.terms.*"` | PASS (412 tests) |
| REST-YAML | `./gradlew :rest-api-spec:yamlRestTest` | PASS |
| Full check | `./gradlew check` | PASS (1h03m, --build-cache) |

## Reproducer status
- testMissingWithMinDocCountZeroOnEmptyShard: FAIL on main, PASS with fix
- Negative control testMissingWithMinDocCountOne...: PASS both (unchanged)

## Determinism
- 50 iters + 10 fresh seeds on the new tests: 0 flakes

## Diff scope
- `git diff origin/main --stat`: 2 files (TermsAggregator.java, TermsAggregatorTests.java) + 1 YAML
- No unrelated files touched.

## Notes
- One unrelated flake observed in qa:rolling-upgrade (issue #MMMM); not introduced by this PR.

Deliverable for Step 7

• spotlessJavaCheck, precommit, affected-module tests, and a full ./gradlew check all run locally and green.
• git diff origin/main --stat confirms only the bug's files changed.
• Any failure understood and either fixed or attributed (pre-existing flake, with the issue link).
• capstone-work/validation.md written with the exact commands and results.
• You have not pushed until every gate is green.

Validation / Self-check

Before advancing to Step 8:

1. You ran the gates in cost order and saw each one green — you did not skip precommit or check.
2. git diff origin/main --stat lists only files you intended to change.
3. You can explain every failing test you saw and whether it is yours or a pre-existing flake (with a link).
4. The new tests are deterministic across iters and seeds.
5. capstone-work/validation.md records the literal commands and results, not a vague "all tests passed."
6. You understand that local check green is what produces first-pass green CI; CI is not something you debug after pushing.
7. You did not push speculative fixes to chase CI without a local repro.

Then go to Step 8: Pull Request Preparation.