Lab P5: Reproducing Plugin/Core Integration Bugs

Background

P4 ends with "file the bug with a minimal repro in the right repo." This lab is about building that repro when the bug spans core and a plugin. A cross-repo repro is harder than a single-repo one for two reasons: it needs both pieces present at compatible versions, and it must be minimal and deterministic despite involving two independently-built artifacts. You will build both forms of repro: a black-box curl/REST-YAML script against a running distro with the plugin installed, and a white-box integration test in the plugin repo that boots an in-JVM cluster with the plugin loaded via nodePlugins().

You will also confront the thing that makes these bugs slippery: version sensitivity. A plugin must match the core version exactly, and an integration bug often only reproduces on a specific core↔plugin version pair. Getting the pinning right is half the work.

This depends on the build mechanic from the section index, the Plugin Architecture deep dive, and the testing framework you met in the testing levels (Level 7, Level 8). It connects to Serialization and BWC — the reason version skew bites.

Why This Lab Matters for Contributors

A bug report without a repro is a wish. A cross-repo bug report with a repro that only works on your laptop, with five plugins installed and a specific Dashboards build, is barely better — the maintainer can't run it. The contributor who ships a 12-line curl script or a self-contained OpenSearchIntegTestCase that the plugin's CI can run is the one whose bug gets fixed this release instead of next. Determinism (fixed seeds, fixed data, single shard) is what turns "sometimes wrong" into "always wrong here," which is what a maintainer needs to bisect.

Prerequisites

Two repro forms

flowchart LR
  BUG[Suspected core+plugin bug] --> A[Black-box repro]
  BUG --> B[White-box repro]
  A --> A1[Build a localDistro at version V]
  A1 --> A2[Install plugin built against V]
  A2 --> A3[curl / REST-YAML script]
  B --> B1[Plugin repo integ test]
  B1 --> B2[OpenSearchIntegTestCase + nodePlugins]
  B2 --> B3[Deterministic: fixed seed, 1 shard, fixed data]

Step-by-Step Tasks

Step 1 — Pin the versions across both repos

The cardinal rule: the plugin's opensearch.version must equal the distro's version. Decide one version V and drive everything from it.

# Discover the core version you're on
cd ~/OpenSearch
V=$(grep -m1 "opensearch[ =]" buildSrc/version.properties 2>/dev/null | sed 's/.*= *//')
echo "Building against core $V"

# Publish core artifacts at V to ~/.m2
./gradlew publishToMavenLocal
ls ~/.m2/repository/org/opensearch/opensearch/$V/ 2>/dev/null

Warning: This is the #1 source of false integration bugs. A "plugin won't load" report is usually a version mismatch, not a code bug — the descriptor says 3.1.0 and the node is 3.0.0. Always confirm _cat/plugins and _nodes/info versions agree before believing it's a real integration bug. See P6 for making that failure self-explanatory.

Step 2 — Build the black-box repro environment

# Build a runnable distro at V
cd ~/OpenSearch
./gradlew localDistro
DISTRO=$(ls -d distribution/archives/*/build/install/opensearch-* | head -1)

# Build the plugin against the SAME V and install it
cd ~/k-NN
./gradlew assemble -Dopensearch.version=$V
"$DISTRO/bin/opensearch-plugin" install --batch file://$(ls ~/k-NN/build/distributions/*$V*.zip)
"$DISTRO/bin/opensearch-plugin" list

# Start it
"$DISTRO/bin/opensearch" -d -p /tmp/os.pid
until curl -s localhost:9200 >/dev/null; do sleep 1; done
curl -s localhost:9200/_cat/plugins?v

Step 3 — Write the minimal, deterministic curl repro

Minimize ruthlessly: one shard, no replicas, a handful of documents, the smallest query that triggers the bug. A worked example — a knn query that should return a known nearest neighbor but doesn't on a specific param:

#!/usr/bin/env bash
# repro.sh — minimal cross-repo (core + k-NN) reproducer. Deterministic.
set -euo pipefail
H=localhost:9200
curl -s -XDELETE $H/r >/dev/null 2>&1 || true

# 1 shard, 0 replicas => no cross-shard nondeterminism
curl -s -XPUT $H/r -H 'Content-Type: application/json' -d '{
  "settings": { "index.knn": true, "number_of_shards": 1, "number_of_replicas": 0 },
  "mappings": { "properties": {
    "v": { "type": "knn_vector", "dimension": 2,
           "method": { "name": "hnsw", "engine": "faiss", "space_type": "l2" } } } }'

# Fixed data — neighbors are unambiguous
for id in 1 2 3; do
  curl -s -XPUT "$H/r/_doc/$id" -H 'Content-Type: application/json' \
    -d "{\"v\":[$id,$id]}" >/dev/null
done
curl -s -XPOST $H/r/_refresh >/dev/null

# The query under test — nearest to [1,1] should be doc 1
echo "Result:"
curl -s "$H/r/_search" -H 'Content-Type: application/json' -d '{
  "size": 1, "query": { "knn": { "v": { "vector": [1,1], "k": 1 } } } }' \
  | python3 -c "import sys,json;d=json.load(sys.stdin);print(d['hits']['hits'][0]['_id'])"
echo "Expected: 1"

The properties that make this a good repro: a fixed index name, single shard (eliminates the reduce as a variable), fixed integer vectors (the nearest neighbor is unarguable), and an explicit expected value printed next to the actual. Anyone can run it and see pass/fail in one line.

Step 4 — Promote it to a shareable REST-YAML test

For a bug you want the plugin's CI to guard against, write it as a REST-YAML test (the format under rest-api-spec in core; plugins have an equivalent yamlRestTest source set). It's the same repro, declarative and assertable:

# plugin-side yamlRestTest: knn nearest neighbor is deterministic
"knn returns the true nearest neighbor on a single shard":
  - do:
      indices.create:
        index: r
        body:
          settings: { index.knn: true, number_of_shards: 1, number_of_replicas: 0 }
          mappings:
            properties:
              v: { type: knn_vector, dimension: 2,
                   method: { name: hnsw, engine: faiss, space_type: l2 } }
  - do: { index: { index: r, id: "1", body: { v: [1, 1] }, refresh: true } }
  - do: { index: { index: r, id: "2", body: { v: [9, 9] }, refresh: true } }
  - do:
      search:
        index: r
        body: { size: 1, query: { knn: { v: { vector: [1, 1], k: 1 } } } }
  - match: { hits.hits.0._id: "1" }

This is the artifact that prevents regression: it lives in the plugin repo and runs against a real cluster on every PR.

Step 5 — The white-box repro: an integ test with nodePlugins()

When the bug needs internal state, or to run in the plugin's ./gradlew integTest without a separate distro, write an OpenSearchIntegTestCase that loads the plugin into the in-JVM InternalTestCluster:

// In the k-NN repo's integration test source set
public class KNNNearestNeighborIT extends OpenSearchIntegTestCase {

    @Override
    protected Collection<Class<? extends Plugin>> nodePlugins() {
        // This is the boundary: the in-JVM node loads core + this plugin
        return List.of(KNNPlugin.class);
    }

    public void testNearestNeighborIsDeterministic() throws Exception {
        String index = "r";
        createIndex(index, Settings.builder()
            .put("index.knn", true)
            .put("number_of_shards", 1)
            .put("number_of_replicas", 0)
            .build());
        // ... put the knn_vector mapping, index [1,1] and [9,9], refresh ...

        SearchResponse resp = client().prepareSearch(index)
            .setQuery(new KNNQueryBuilder("v", new float[]{1f, 1f}, 1))
            .setSize(1)
            .get();

        assertEquals("1", resp.getHits().getAt(0).getId());
    }
}

The load-bearing line is nodePlugins() returning the plugin class — that is what makes InternalTestCluster boot a node with the plugin's extension points wired in, so the test exercises the real core↔plugin integration, not a mock.

cd ~/k-NN
./gradlew integTest --tests "*KNNNearestNeighborIT*" -Dtests.seed=DEADBEEF

The -Dtests.seed=... pins the randomized-testing seed so the run is reproducible — paste that seed into the bug report so the maintainer gets your exact run. (See the testing levels for how randomized testing and seeds work.)

Why integration bugs are version-sensitive

A core↔plugin integration bug often reproduces only on a specific version pair, because the plugin compiles against core's internal APIs, which are not stable across versions (unlike the REST API). Three concrete ways version matters:

So a complete cross-repo repro states the exact versions:

core:   3.1.0-SNAPSHOT (commit abc1234)
plugin: opensearch-knn 3.1.0-SNAPSHOT (commit def5678)
java:   21

Without the version pair, a maintainer who can't reproduce will close it "cannot reproduce" — correctly, because on their version pair it may not happen.

Expected Output

• A localDistro at version V with a V-built plugin installed, verified via _cat/plugins.
• A repro.sh that is deterministic (single shard, fixed data) and prints actual-vs-expected in one line.
• A REST-YAML version of the same repro.
• An OpenSearchIntegTestCase skeleton using nodePlugins(), runnable with a pinned -Dtests.seed.
• A version block (core commit, plugin commit, JDK) ready to paste into a report.

Troubleshooting

Stretch Goals

1. Make the black-box repro reproduce only on one core version by checking out a different core commit, rebuilding, and confirming it disappears — then document the version window in the report.
2. Convert the OpenSearchIntegTestCase to also assert an internal counter (e.g. a k-NN stat) to show the bug's internal footprint, not just the REST result.
3. Add a second node to the integ test (@ClusterScope(numDataNodes = 2)) and confirm whether the bug is shard-count sensitive.

Validation / Self-check

1. Why must the plugin's opensearch.version equal the node version, and what's the failure mode when it doesn't?
2. What three things make a curl repro deterministic, and why does each matter?
3. What does nodePlugins() do, and why is it the load-bearing line in a white-box integration test?
4. Why are core↔plugin integration bugs version-sensitive in a way that pure REST API behavior is not?
5. What exactly do you paste into the bug report so a maintainer can reproduce your randomized integ-test run?
6. Given "plugin won't load," what do you check before concluding it's a code bug?