Lab DP4: Reproduce and Bisect a Bug

Background

The first three labs answered "slow," "hot," "stuck," and "OOM" — all performance questions you measure. This lab is the "why is it wrong?" workflow, which is a different discipline: you are not measuring, you are narrowing. A wrong count, a missing bucket, a mis-scored doc, a result that depends on which thread finished first — these are bugs you pin down by reproducing deterministically, then bisecting history to the commit that introduced them.

You will run the full end-to-end loop on real OpenSearch source:

1. Reproduce deterministically — build a failing OpenSearchIntegTestCase (or a curl script) and pin -Dtests.seed so it fails the same way every run.
2. Add TRACE logging to the suspect code path and read what it actually does.
3. Attach a debugger with ./gradlew run --debug-jvm (and --debug-jvm on a single test), set a breakpoint on port 5005, and inspect live state.
4. git bisect across tags (2.x..main) with the reproducer as the test, to find the introducing PR — then read its GitHub discussion to understand why the change was made.

This is the exact workflow the capstone root-cause step formalizes and that stage-9-flaky-tests applies to flaky tests specifically.

Why This Matters for Contributors

A correctness bug without a deterministic reproducer is unfixable and un-reviewable: a maintainer cannot verify your fix if they cannot see the failure. The single most valuable artifact you can attach to a correctness issue is a test that fails on main and passes after your change — and git bisect is how you turn "it regressed sometime" into "PR #12345 introduced it," which usually hands you the intent you must preserve while fixing. This lab builds the reflex: reproduce → narrow → attribute, every time.

Prerequisites

• An OpenSearch checkout with the build working: ./gradlew help succeeds, and you can run a test (./gradlew :server:test --tests "...SomeTest").
• Git history with tags fetched: git fetch --tags.
• A debugger that does JDWP attach (IntelliJ IDEA "Remote JVM Debug", or VS Code "Java: Attach"). Port 5005 free.
• Read the intensive section "Why is it wrong?".

Note: OpenSearch tests are intentionally randomized (random seeds, random settings, random doc orders) to find bugs you wouldn't think to test. That same randomness makes a failure look flaky until you pin the seed. Pinning the seed is the first move in every correctness investigation.

Step-by-Step Tasks

Step 1 — Locate the subsystem and a test to clone

Say the symptom is "a terms aggregation returns a wrong bucket count under concurrent segment search." Find where that code and its tests live:

cd ~/src/OpenSearch
# The aggregation under suspicion:
find server -name GlobalOrdinalsStringTermsAggregator.java
# Existing tests to clone the harness from:
find server test -name "*TermsAggregator*Tests.java" -o -name "*TermsIT.java" | head
grep -rln "extends AggregatorTestCase\|extends OpenSearchIntegTestCase" \
  server/src/test/java/org/opensearch/search/aggregations/bucket/terms/ | head

AggregatorTestCase runs an aggregation against an in-memory Lucene index with no cluster — fast, ideal for a unit-level reproducer. OpenSearchIntegTestCase spins a real in-process cluster — needed when the bug only shows across shards/nodes (e.g. a concurrency or reduce bug).

Step 2 — Write a deterministic reproducer (two ways)

Way A — a curl script (no build; good for a first capture and for the issue report). It creates a tiny index, indexes known docs, runs the query, and diffs against expected output:

cat > /tmp/repro.sh <<'SH'
#!/bin/sh
set -e
curl -s -XDELETE 'localhost:9200/repro' >/dev/null 2>&1 || true
curl -s -XPUT 'localhost:9200/repro' -H 'Content-Type: application/json' -d'
{ "settings": { "number_of_shards": 3, "number_of_replicas": 0 },
  "mappings": { "properties": { "cat": { "type": "keyword" } } } }' >/dev/null
# Known data: exactly 6 of cat=a, 4 of cat=b.
for i in $(seq 1 6); do curl -s -XPOST 'localhost:9200/repro/_doc' -H 'Content-Type: application/json' -d'{"cat":"a"}' >/dev/null; done
for i in $(seq 1 4); do curl -s -XPOST 'localhost:9200/repro/_doc' -H 'Content-Type: application/json' -d'{"cat":"b"}' >/dev/null; done
curl -s -XPOST 'localhost:9200/repro/_refresh' >/dev/null
# Run the agg and extract the buckets:
curl -s 'localhost:9200/repro/_search' -H 'Content-Type: application/json' -d'
{ "size":0, "aggs": { "by_cat": { "terms": { "field": "cat" } } } }' \
  | python3 -c 'import sys,json; b=json.load(sys.stdin)["aggregations"]["by_cat"]["buckets"]; print({x["key"]:x["doc_count"] for x in b})'
SH
chmod +x /tmp/repro.sh
/tmp/repro.sh        # expected: {'a': 6, 'b': 4}

If the printed map ever differs from {'a': 6, 'b': 4}, you have a reproducer.

Way B — a JUnit reproducer, which is what you actually attach to the PR. Skeleton based on AggregatorTestCase:

// In server/src/test/.../terms/MyReproTests.java
public class MyReproTests extends AggregatorTestCase {
  public void testTermsCountsAreExact() throws Exception {
    try (Directory dir = newDirectory();
         RandomIndexWriter w = new RandomIndexWriter(random(), dir)) {
      // 6 of "a", 4 of "b" -- the same known data as the curl repro.
      for (int i = 0; i < 6; i++) w.addDocument(doc("a"));
      for (int i = 0; i < 4; i++) w.addDocument(doc("b"));
      try (IndexReader reader = w.getReader()) {
        TermsAggregationBuilder agg = new TermsAggregationBuilder("by_cat")
            .field("cat").userValueTypeHint(ValueType.STRING);
        MappedFieldType ft = new KeywordFieldMapper.KeywordFieldType("cat");
        StringTerms result = searchAndReduce(reader, new MatchAllDocsQuery(), agg, ft);
        // The exact assertion that fails when the bug is present:
        assertEquals(6L, bucketCount(result, "a"));
        assertEquals(4L, bucketCount(result, "b"));
      }
    }
  }
  // helpers: doc(...) builds a Document with a "cat" keyword; bucketCount(...) reads a bucket.
}

Step 3 — Pin the seed so it fails every run

OpenSearch's test runner prints the seed on failure; reuse it to force the identical random path:

# Run once; on failure the output includes a reproduce line with -Dtests.seed=...
./gradlew :server:test --tests "org.opensearch.search.aggregations.bucket.terms.MyReproTests"

# Pin that seed so the SAME random index/order/settings recur every run:
./gradlew :server:test --tests "...MyReproTests" \
  -Dtests.seed=DEADBEEFCAFE0001 -Dtests.iters=1

# Stress it to confirm determinism (should fail/pass identically each iteration):
./gradlew :server:test --tests "...MyReproTests" -Dtests.seed=DEADBEEFCAFE0001 -Dtests.iters=20

# Where the seed plumbing lives, if you want to read it:
grep -rn "tests.seed\|RandomizedContext\|InternalTestCluster" \
  test/framework/src/main/java/org/opensearch/test/ | head

Note: A reproducer that fails 20/20 with a pinned seed is deterministic; one that fails 3/20 even with a pinned seed has a second nondeterminism source (real thread-timing — the concurrent-segment-search reduce bug from Lab CS2 is exactly this). That distinction tells you whether the bug is logic or a race.

Step 4 — Add TRACE logging to the suspect path

Before reaching for the debugger, let the code narrate. Enable TRACE on the aggregation logger at runtime (no rebuild) and re-run the curl reproducer:

curl -s -XPUT 'localhost:9200/_cluster/settings' -H 'Content-Type: application/json' -d'
{ "transient": { "logger.org.opensearch.search.aggregations": "TRACE" } }'

/tmp/repro.sh
tail -200 $(find . -name '*.log' -path '*logs*' | head -1)   # read the node log

# ALWAYS turn it back off -- TRACE is a firehose:
curl -s -XPUT 'localhost:9200/_cluster/settings' -H 'Content-Type: application/json' -d'
{ "transient": { "logger.org.opensearch.search.aggregations": null } }'

If the existing logging doesn't say enough, add a temporary TRACE line in the suspect method and rebuild:

// In the suspect collect()/reduce():
private static final Logger logger = LogManager.getLogger(GlobalOrdinalsStringTermsAggregator.class);
// ...
logger.trace("collect doc={} owningBucketOrd={} globalOrd={}", doc, owningBucketOrd, globalOrd);

grep -rn "LogManager.getLogger\|logger.trace\|logger.debug" \
  server/src/main/java/org/opensearch/search/aggregations/bucket/terms/GlobalOrdinalsStringTermsAggregator.java | head

Step 5 — Attach a debugger with ./gradlew run --debug-jvm

When you need to step through the code with live state, attach a real debugger.

Debug a running node:

# Starts a node that SUSPENDS and waits for a debugger on 5005:
./gradlew run --debug-jvm
# It prints: "Listening for transport dt_socket at address: 5005"

In IntelliJ: Run → Edit Configurations → + → Remote JVM Debug, host localhost, port 5005, Debug. Set a breakpoint in GlobalOrdinalsStringTermsAggregator's collect or in buildAggregations / the reduce path, then run /tmp/repro.sh from another terminal. Execution halts at your breakpoint; inspect globalOrd, owningBucketOrd, the bucket array — the live values that the TRACE log only sampled.

Debug a single test (often faster — no cluster):

# Suspends the test JVM on 5005; attach, then it runs your test to the breakpoint:
./gradlew :server:test --tests "...MyReproTests" --debug-jvm \
  -Dtests.seed=DEADBEEFCAFE0001

flowchart LR
    Repro["deterministic reproducer<br/>(curl or JUnit, pinned seed)"] --> Trace["TRACE logging:<br/>narrow to the method"]
    Trace --> Debug["./gradlew run --debug-jvm<br/>attach 5005, inspect state"]
    Debug --> Cause["root cause in current code"]
    Repro --> Bisect["git bisect:<br/>WHEN did it break?"]
    Bisect --> PR["introducing PR + its intent"]
    Cause --> Fix["fix that preserves the original intent"]
    PR --> Fix

Step 6 — git bisect to the introducing PR

If the bug is a regression — it worked in an older release — bisect finds the exact commit in O(log n) steps. Use the test (or a shell that runs the curl repro and exits nonzero on mismatch) as the automated verdict.

git fetch --tags
git bisect start
git bisect bad  main             # current HEAD reproduces the bug
git bisect good 2.11.0           # this tag did NOT (verify by checking it out first!)

# Automate: bisect checks out each midpoint, runs your test, reads the exit code.
git bisect run ./gradlew :server:test --tests "...MyReproTests" \
  -Dtests.seed=DEADBEEFCAFE0001

# bisect prints "<sha> is the first bad commit". Inspect it:
git show --stat <sha>
git log -1 --format='%H%n%s%n%b' <sha>
git bisect reset

Two practical notes:

• Verify the endpoints first. Manually check that 2.11.0 is actually good and main is bad before starting; a wrong endpoint makes bisect converge on nonsense.
• git bisect run needs a clean exit code: 0 = good, 1–124 (except 125) = bad, 125 = skip (un-buildable commit). A Gradle test failure already returns nonzero, so it slots right in. For the curl repro, wrap it so it exit 1s on a wrong count.

Step 7 — Read the PR and its discussion

The bad commit's message names its PR (e.g. (#12345)). Pull up the PR and its linked issue on GitHub and read the discussion:

# If you have the gh CLI authenticated:
gh pr view 12345 --repo opensearch-project/OpenSearch --comments | less
# Otherwise open in a browser:
#   https://github.com/opensearch-project/OpenSearch/pull/12345

What you're mining for: why the change was made (the original intent), what it meant to fix, and any reviewer concern that foreshadowed this regression. Your fix must preserve that intent — re-breaking what the PR fixed is how a fix gets reverted. This is the bridge to the capstone root-cause step, which asks for exactly this artifact: the introducing PR plus a fix that doesn't regress its purpose.

Deliverables

• A deterministic reproducer — both /tmp/repro.sh and a MyReproTests — that fails (or passes) identically across 20 pinned-seed iterations.
• The -Dtests.seed=... line that pins it, and a note on whether the bug is logic (fails 20/20) or a race (fails N/20 even pinned).
• TRACE-log output from the suspect path, and a screenshot/notes of the debugger stopped at a breakpoint with the offending live values.
• A git bisect transcript ending in "first bad commit," the PR number, and a two-sentence summary of the PR's original intent from its discussion.

Expected Output

# pinned-seed run (Step 3)
> Task :server:test
MyReproTests > testTermsCountsAreExact FAILED
    expected:<6> but was:<5>
Tests with failures: ... (reproduce with -Dtests.seed=DEADBEEFCAFE0001)

# bisect (Step 6)
Bisecting: 412 revisions left to test after this (roughly 9 steps)
...
e3f8a91... is the first bad commit
    Author: ...
    fix terms agg ordinal reuse (#12345)
bisect run success

# PR (Step 7)
PR #12345 "fix terms agg ordinal reuse" -- intended to dedupe ordinals across
slices; the dedupe dropped a bucket when a global ord appeared in only one slice.

Troubleshooting

Stretch Goals

• Bisect with the curl repro. Wrap /tmp/repro.sh so it exit 1s on a wrong count, start a node per step, and git bisect run it — proving bisect doesn't require a JUnit test.
• Bisect a performance regression. Use a good/bad threshold on a timed benchmark instead of a correctness check (exit 1 if p50 > X) to bisect a slowdown.
• Minimize the reproducer. Shrink MyReproTests to the fewest docs/settings that still fail — a minimal reproducer is what maintainers merge fastest.
• Add a regression test for the fix. Once you understand the cause, turn MyReproTests into a permanent regression test and note where it belongs in the suite (the capstone testing step covers this).
• Reverse-bisect a fix. Given a bug fixed upstream, bisect from a bad older tag to a good newer one to find the fixing commit and backport candidate.

Coding Exercises

This lab's deliverables are already code-shaped (a reproducer, a bisect script); these exercises make them graded — turn the manual loop into reusable, asserting tooling. Locate every class with rg/find; never paste a line number from this page.

1. (warm-up) A bisect-ready verdict wrapper. Turn /tmp/repro.sh into verdict.sh that runs the agg, parses the bucket map, and exit 0 on {'a':6,'b':4}, exit 1 on any other count, and exit 125 if the node isn't up (so git bisect run skips un-runnable commits instead of marking them bad). Test all three exit codes by hand. This is the contract git bisect run needs — getting the 125 skip right is what separates a clean bisect from a garbage one.

2. (core) Promote the curl repro to a JUnit reproducer. Flesh out the Step 2 MyReproTests skeleton into a compiling AggregatorTestCase test (find the helpers to crib from):

   rg -ln "extends AggregatorTestCase" server/src/test/java/org/opensearch/search/aggregations/bucket/terms/
   rg -n "searchAndReduce|new TermsAggregationBuilder|KeywordFieldType" \
     server/src/test/java/org/opensearch/search/aggregations/AggregatorTestCase.java
   

   Make it assert exact bucket counts. Confirm it fails identically across 20 pinned iterations (-Dtests.seed=... -Dtests.iters=20) — the determinism check that tells a logic bug from a race.

3. (core) A TRACE-capture assertion. Instead of tail-ing the node log by eye, write an OpenSearchSingleNodeTestCase that attaches a MockLogAppender (rg -ln "MockLogAppender" server/src/test), raises the aggregation logger to TRACE, runs the reproducer query, and asserts a specific TRACE line (e.g. one carrying globalOrd) was emitted. You're converting "read the firehose" into a precise, repeatable assertion — and learning the appender harness real PRs use.

4. (advanced) A self-contained bisect harness over a synthetic history. Don't wait for a real regression: create a throwaway git repo with ~12 commits where one introduces an off-by-one in a tiny count() function and a committed test asserts the right answer. Write bisect_demo.sh that runs git bisect start / bad HEAD / good <first> / run <your test> and prints the first-bad SHA. Verify bisect lands on the exact culprit commit in O(log n) steps. This rehearses the mechanics (good/bad endpoints, bisect run exit codes, reset) with zero build cost before you do it on OpenSearch.

5. (Advanced challenge) A performance-regression bisect gate. Combine Stretch Goal "bisect a performance regression" with real measurement. Write a script that, per bisect step: starts a node (or runs a benchmarks/ JMH harness — rg -l "@Benchmark" benchmarks), drives a fixed query N times, computes p50 latency, and exit 1 if p50 exceeds a threshold else exit 0 (and exit 125 on build failure). Drive it with git bisect run across two tags to attribute a slowdown to a commit. Bonus: emit a JSON {sha, p50, verdict} line per step so the run is auditable, and reuse your jfr_top.py from Lab DP2 at the first-bad commit to show which method the regression added. Guard against noise by averaging multiple runs and warming up first — a flaky threshold makes bisect converge on nonsense, the same failure mode as a wrong good endpoint.

Issues to Practice On

This is the workflow for every correctness and regression issue — a reproducer plus an attributed commit is the most valuable thing you can attach.

Confirm labels with gh label list --repo opensearch-project/OpenSearch before relying on any of them.

Representative patterns. (a) Flaky test — the highest-leverage entry point for this lab. Pin the seed; if it fails 20/20 it's logic, if N/20 even pinned it's a real race (route to stage-9-flaky-tests and Lab CS2). Either way you attach a deterministic reproducer. (b) "Result regressed between version A and B" — build the curl/JUnit reproducer, git bisect run from the good tag to bad main, name the introducing PR, read its discussion for the intent, and write a fix that preserves it — the capstone root-cause step.

Planted-bug drill. In the terms aggregator, find an ordinal/bucket bookkeeping line:

rg -n "collectExistingBucket|collectBucket|bucketOrd|incrementBucketDocCount|\\+\\+|--" \
  server/src/main/java/org/opensearch/search/aggregations/bucket/terms/GlobalOrdinalsStringTermsAggregator.java

Introduce an off-by-one (drop a +1, or change a <= to < in a bucket loop). Run your MyReproTests and watch expected:<6> but was:<5> appear. Then git bisect your own one-commit change to confirm bisect fingers the right line — and finally revert and keep MyReproTests as the permanent regression test (it belongs in the suite per the capstone testing step).

Etiquette: claim the issue before working it, reproduce first, and every PR needs a test + a CHANGELOG.md entry + DCO Signed-off-by (git commit -s). See community-interaction.

Validation / Self-check

• You can build a deterministic reproducer two ways (a curl script and an OpenSearchIntegTestCase/AggregatorTestCase) and pin it with -Dtests.seed.
• You can tell a logic bug (fails 20/20 pinned) from a race (fails N/20 pinned) and know which lab/stage each routes to.
• You can enable and scope TRACE logging at runtime and turn it back off safely.
• You can attach a debugger via ./gradlew run --debug-jvm (and --debug-jvm on a single test) on port 5005 and inspect live state at a breakpoint.
• You can run a full git bisect from a good tag to a bad main with an automated test and name the introducing PR.
• You can read the introducing PR's discussion to recover the change's intent and explain why your fix must preserve it — the handoff to the capstone root-cause step.

This closes the Debugging and Profiling masterclass. Back to the intensive, or apply the whole workflow in the capstone and on real flaky tests.