Lab P5: Reproducing Plugin/Core Integration Bugs

Background

P4 ends with "file the bug with a minimal repro in the right repo." This lab is about building that repro when the bug spans core and a plugin. A cross-repo repro is harder than a single-repo one for two reasons: it needs both pieces present at compatible versions, and it must be minimal and deterministic despite involving two independently-built artifacts. You will build both forms of repro: a black-box curl/REST-YAML script against a running distro with the plugin installed, and a white-box integration test in the plugin repo that boots an in-JVM cluster with the plugin loaded via nodePlugins().

You will also confront the thing that makes these bugs slippery: version sensitivity. A plugin must match the core version exactly, and an integration bug often only reproduces on a specific core↔plugin version pair. Getting the pinning right is half the work.

This depends on the build mechanic from the section index, the Plugin Architecture deep dive, and the testing framework you met in the testing levels (Level 7, Level 8). It connects to Serialization and BWC — the reason version skew bites.

Why This Lab Matters for Contributors

A bug report without a repro is a wish. A cross-repo bug report with a repro that only works on your laptop, with five plugins installed and a specific Dashboards build, is barely better — the maintainer can't run it. The contributor who ships a 12-line curl script or a self-contained OpenSearchIntegTestCase that the plugin's CI can run is the one whose bug gets fixed this release instead of next. Determinism (fixed seeds, fixed data, single shard) is what turns "sometimes wrong" into "always wrong here," which is what a maintainer needs to bisect.

Prerequisites

Two repro forms

flowchart LR
  BUG[Suspected core+plugin bug] --> A[Black-box repro]
  BUG --> B[White-box repro]
  A --> A1[Build a localDistro at version V]
  A1 --> A2[Install plugin built against V]
  A2 --> A3[curl / REST-YAML script]
  B --> B1[Plugin repo integ test]
  B1 --> B2[OpenSearchIntegTestCase + nodePlugins]
  B2 --> B3[Deterministic: fixed seed, 1 shard, fixed data]

Step-by-Step Tasks

Step 1 — Pin the versions across both repos

The cardinal rule: the plugin's opensearch.version must equal the distro's version. Decide one version V and drive everything from it.

# Discover the core version you're on
cd ~/OpenSearch
V=$(grep -m1 "opensearch[ =]" buildSrc/version.properties 2>/dev/null | sed 's/.*= *//')
echo "Building against core $V"

# Publish core artifacts at V to ~/.m2
./gradlew publishToMavenLocal
ls ~/.m2/repository/org/opensearch/opensearch/$V/ 2>/dev/null

Warning: This is the #1 source of false integration bugs. A "plugin won't load" report is usually a version mismatch, not a code bug — the descriptor says 3.1.0 and the node is 3.0.0. Always confirm _cat/plugins and _nodes/info versions agree before believing it's a real integration bug. See P6 for making that failure self-explanatory.

Step 2 — Build the black-box repro environment

# Build a runnable distro at V
cd ~/OpenSearch
./gradlew localDistro
DISTRO=$(ls -d distribution/archives/*/build/install/opensearch-* | head -1)

# Build the plugin against the SAME V and install it
cd ~/k-NN
./gradlew assemble -Dopensearch.version=$V
"$DISTRO/bin/opensearch-plugin" install --batch file://$(ls ~/k-NN/build/distributions/*$V*.zip)
"$DISTRO/bin/opensearch-plugin" list

# Start it
"$DISTRO/bin/opensearch" -d -p /tmp/os.pid
until curl -s localhost:9200 >/dev/null; do sleep 1; done
curl -s localhost:9200/_cat/plugins?v

Step 3 — Write the minimal, deterministic curl repro

Minimize ruthlessly: one shard, no replicas, a handful of documents, the smallest query that triggers the bug. A worked example — a knn query that should return a known nearest neighbor but doesn't on a specific param:

#!/usr/bin/env bash
# repro.sh — minimal cross-repo (core + k-NN) reproducer. Deterministic.
set -euo pipefail
H=localhost:9200
curl -s -XDELETE $H/r >/dev/null 2>&1 || true

# 1 shard, 0 replicas => no cross-shard nondeterminism
curl -s -XPUT $H/r -H 'Content-Type: application/json' -d '{
  "settings": { "index.knn": true, "number_of_shards": 1, "number_of_replicas": 0 },
  "mappings": { "properties": {
    "v": { "type": "knn_vector", "dimension": 2,
           "method": { "name": "hnsw", "engine": "faiss", "space_type": "l2" } } } }'

# Fixed data — neighbors are unambiguous
for id in 1 2 3; do
  curl -s -XPUT "$H/r/_doc/$id" -H 'Content-Type: application/json' \
    -d "{\"v\":[$id,$id]}" >/dev/null
done
curl -s -XPOST $H/r/_refresh >/dev/null

# The query under test — nearest to [1,1] should be doc 1
echo "Result:"
curl -s "$H/r/_search" -H 'Content-Type: application/json' -d '{
  "size": 1, "query": { "knn": { "v": { "vector": [1,1], "k": 1 } } } }' \
  | python3 -c "import sys,json;d=json.load(sys.stdin);print(d['hits']['hits'][0]['_id'])"
echo "Expected: 1"

The properties that make this a good repro: a fixed index name, single shard (eliminates the reduce as a variable), fixed integer vectors (the nearest neighbor is unarguable), and an explicit expected value printed next to the actual. Anyone can run it and see pass/fail in one line.

Step 4 — Promote it to a shareable REST-YAML test

For a bug you want the plugin's CI to guard against, write it as a REST-YAML test (the format under rest-api-spec in core; plugins have an equivalent yamlRestTest source set). It's the same repro, declarative and assertable:

# plugin-side yamlRestTest: knn nearest neighbor is deterministic
"knn returns the true nearest neighbor on a single shard":
  - do:
      indices.create:
        index: r
        body:
          settings: { index.knn: true, number_of_shards: 1, number_of_replicas: 0 }
          mappings:
            properties:
              v: { type: knn_vector, dimension: 2,
                   method: { name: hnsw, engine: faiss, space_type: l2 } }
  - do: { index: { index: r, id: "1", body: { v: [1, 1] }, refresh: true } }
  - do: { index: { index: r, id: "2", body: { v: [9, 9] }, refresh: true } }
  - do:
      search:
        index: r
        body: { size: 1, query: { knn: { v: { vector: [1, 1], k: 1 } } } }
  - match: { hits.hits.0._id: "1" }

This is the artifact that prevents regression: it lives in the plugin repo and runs against a real cluster on every PR.

Step 5 — The white-box repro: an integ test with nodePlugins()

When the bug needs internal state, or to run in the plugin's ./gradlew integTest without a separate distro, write an OpenSearchIntegTestCase that loads the plugin into the in-JVM InternalTestCluster:

// In the k-NN repo's integration test source set
public class KNNNearestNeighborIT extends OpenSearchIntegTestCase {

    @Override
    protected Collection<Class<? extends Plugin>> nodePlugins() {
        // This is the boundary: the in-JVM node loads core + this plugin
        return List.of(KNNPlugin.class);
    }

    public void testNearestNeighborIsDeterministic() throws Exception {
        String index = "r";
        createIndex(index, Settings.builder()
            .put("index.knn", true)
            .put("number_of_shards", 1)
            .put("number_of_replicas", 0)
            .build());
        // ... put the knn_vector mapping, index [1,1] and [9,9], refresh ...

        SearchResponse resp = client().prepareSearch(index)
            .setQuery(new KNNQueryBuilder("v", new float[]{1f, 1f}, 1))
            .setSize(1)
            .get();

        assertEquals("1", resp.getHits().getAt(0).getId());
    }
}

The load-bearing line is nodePlugins() returning the plugin class — that is what makes InternalTestCluster boot a node with the plugin's extension points wired in, so the test exercises the real core↔plugin integration, not a mock.

cd ~/k-NN
./gradlew integTest --tests "*KNNNearestNeighborIT*" -Dtests.seed=DEADBEEF

The -Dtests.seed=... pins the randomized-testing seed so the run is reproducible — paste that seed into the bug report so the maintainer gets your exact run. (See the testing levels for how randomized testing and seeds work.)

Why integration bugs are version-sensitive

A core↔plugin integration bug often reproduces only on a specific version pair, because the plugin compiles against core's internal APIs, which are not stable across versions (unlike the REST API). Three concrete ways version matters:

So a complete cross-repo repro states the exact versions:

core:   3.1.0-SNAPSHOT (commit abc1234)
plugin: opensearch-knn 3.1.0-SNAPSHOT (commit def5678)
java:   21

Without the version pair, a maintainer who can't reproduce will close it "cannot reproduce" — correctly, because on their version pair it may not happen.

Expected Output

• A localDistro at version V with a V-built plugin installed, verified via _cat/plugins.
• A repro.sh that is deterministic (single shard, fixed data) and prints actual-vs-expected in one line.
• A REST-YAML version of the same repro.
• An OpenSearchIntegTestCase skeleton using nodePlugins(), runnable with a pinned -Dtests.seed.
• A version block (core commit, plugin commit, JDK) ready to paste into a report.

Troubleshooting

Stretch Goals

1. Make the black-box repro reproduce only on one core version by checking out a different core commit, rebuilding, and confirming it disappears — then document the version window in the report.
2. Convert the OpenSearchIntegTestCase to also assert an internal counter (e.g. a k-NN stat) to show the bug's internal footprint, not just the REST result.
3. Add a second node to the integ test (@ClusterScope(numDataNodes = 2)) and confirm whether the bug is shard-count sensitive.

Coding Exercises

A repro is code — that's the whole lab. These exercises build the full repro ladder: deterministic curl, REST-YAML, white-box integ test, and a version-window proof. Locate real classes/source sets with rg/find; names and versions drift, so confirm them (grep -m1 "opensearch" ~/OpenSearch/buildSrc/version.properties).

1. (warm-up) Make repro.sh self-grading. Take the Step 3 script and extend it so it exits 0 on the expected neighbor and non-zero otherwise, prints the core/plugin versions it ran against (_cat/plugins, _nodes version), and is re-runnable (deletes the index first). A maintainer should be able to run it blind and read pass/fail from the exit code. This is the minimal shareable repro.

2. (core) Promote it to a yamlRestTest that runs in CI. Convert the curl repro to the Step 4 REST-YAML form, place it in the plugin's yamlRestTest source set (find it: find ~/k-NN -path "*yamlRestTest*" -name "*.yml" | head), and run ./gradlew yamlRestTest. The deliverable is a declarative, assertable regression guard living in the plugin repo — the artifact that prevents the bug from returning.

3. (core) Write the white-box OpenSearchIntegTestCase with nodePlugins(). Complete the Step 5 skeleton: implement the index creation, mapping, doc indexing, and the KNNQueryBuilder search, asserting the nearest neighbor. The load-bearing line is nodePlugins() returning the plugin class — confirm the pattern in a real test (rg -l "protected Collection<Class<. extends Plugin>> nodePlugins" ~/k-NN/src). Run with ./gradlew integTest --tests "*YourIT*" -Dtests.seed=DEADBEEF and put that seed in your would-be bug report.

4. (core) Add an internal-state assertion (Stretch 2, graded). Extend the integ test to also assert a plugin-internal counter — find a k-NN stat (rg -n "increment\(|KNNCounter|StatNames" ~/k-NN/src/main/java | head) — so the test proves the bug's internal footprint, not just the REST result. A repro that shows the internal counter is wrong is far more actionable than one that only shows a wrong hit.

5. (core) Prove the version window. Build the plugin against two different core commits/versions and run the same repro.sh against each, capturing that it reproduces on one and not the other (Stretch 1, graded). Write the result as the exact version block the lab specifies (core commit, plugin commit, JDK). This is the evidence that turns "cannot reproduce" into "reproduces on this pair."

6. (advanced challenge) Build a one-command cross-repo repro harness. Write a script that, given a target version V: pins core and the plugin to V (publishToMavenLocal → assemble -Dopensearch.version=$V), builds a localDistro, installs the plugin, runs the black-box repro.sh, and runs the white-box integ test with a pinned seed — emitting a single report containing the version block, the seed, the REST-YAML, and pass/fail. Then run a @ClusterScope(numDataNodes = 2) variant (Stretch 3) and record whether the bug is shard-count-sensitive. You have built the complete, shareable, deterministic cross-repo reproducer a maintainer can run unmodified.

Issues to Practice On

Practice on issues that lack a clean repro — turning them into one is the contribution this lab teaches, and it spans core and the plugin.

gh issue list --repo opensearch-project/OpenSearch --label "flaky-test" --state open
gh issue list --repo opensearch-project/OpenSearch --label "bug" --label "untriaged" --state open
gh issue list --repo opensearch-project/k-NN --label "bug" --state open
gh issue list --repo opensearch-project/k-NN --label "good first issue" --state open
gh label list --repo opensearch-project/k-NN | grep -iE "bug|test|build"

Labels move; confirm on each tracker before relying on a name.

Two representative patterns:

• "Plugin won't load / incompatible version" (the false integration bug). Almost always version skew, not a code bug. Reproduce, run the triage triad (_cat/plugins, node version, _nodes/plugins), and either close it as user-error with the fix, or — if the versions do match — capture a genuine minimal repro. This is the P6 failure made concrete.
• A bug with a vague repro. Reproduce locally, then minimize ruthlessly: single shard, fixed data, smallest query, explicit expected value. Attach the deterministic repro.sh + the version block. A maintainer can now bisect, and the issue gets fixed this release instead of next.

Planted-bug drill. In your integ test, introduce nondeterminism: bump the index to number_of_shards: 3 and remove the fixed-seed flag, then re-run several times — watch it pass sometimes and fail others. This is the exact "passes for you, fails in CI" trap from the Troubleshooting table. Now fix it the lab's way (1 shard, -Dtests.seed, fixed data) and confirm it's stable across 20 runs (./gradlew integTest --tests "*YourIT*" -Dtests.iters=20). You have felt — and removed — the nondeterminism that makes cross-repo bugs un-bisectable.

Etiquette: Claim an issue before working it; reproduce first; minimize before you file; state the exact version pair and seed; every PR needs a test + a CHANGELOG entry + a DCO Signed-off-by (git commit -s). See community interaction and the PR-review lab.

Validation / Self-check

1. Why must the plugin's opensearch.version equal the node version, and what's the failure mode when it doesn't?
2. What three things make a curl repro deterministic, and why does each matter?
3. What does nodePlugins() do, and why is it the load-bearing line in a white-box integration test?
4. Why are core↔plugin integration bugs version-sensitive in a way that pure REST API behavior is not?
5. What exactly do you paste into the bug report so a maintainer can reproduce your randomized integ-test run?
6. Given "plugin won't load," what do you check before concluding it's a code bug?