Lab P6: Writing Diagnostics for Integration Bugs

Background

The previous five labs were about consuming diagnostics: reading stack traces (P3), attributing symptoms (P4), reproducing bugs (P5). This lab is about producing them. The highest-leverage cross-repo contribution you can make is rarely the fix itself — it is making the next person's attribution trivial. A boundary error that names the owning component, a log line that says which plugin and which version, a clearer message when a plugin and core disagree on versions: each of these saves dozens of future debugging hours across the ecosystem.

You will inventory the diagnostics OpenSearch already exposes at the core↔plugin boundary, then write a worked example: a clearer error when a plugin and core version mismatch. The change is small; the leverage is enormous, because version mismatch is the single most common cross-repo failure (P5), and a confusing message there sends people to the wrong repo.

This builds on the Plugin Architecture deep dive (where PluginsService validates versions), the error-messages roadmap stage, and Stage 8.

Why This Lab Matters for Contributors

Diagnostics are the contribution maintainers love and reviewers approve fastest, because they are low-risk (no behavior change to the happy path) and high-value. A one-line improvement to a boundary exception message can deflect a stream of mis-filed issues. And the skill compounds: once you've added a good diagnostic at one boundary, you see every other boundary's missing diagnostics. This is how contributors become maintainers — by making the system explain itself.

Prerequisites

The diagnostics OpenSearch already gives you

Before adding new diagnostics, master the existing ones. These are your cross-cutting visibility tools at the boundary.

# The version-mismatch triage triad — run these first on any "plugin broken" report
curl -s localhost:9200/_cat/plugins?v
curl -s 'localhost:9200/_nodes?filter_path=nodes.*.version'
curl -s 'localhost:9200/_nodes/plugins?filter_path=nodes.*.plugins.name,nodes.*.plugins.opensearch_version'

If the plugin's opensearch_version and the node version differ, you've diagnosed the bug in three curls — and that's exactly the failure whose error message you're about to improve.

Anatomy of a good boundary diagnostic

A diagnostic at the core↔plugin seam should answer, in the message itself, three questions a confused user has:

1. Which component owns this? Name the plugin and core, so the reader knows it's a boundary issue, not a pure-core one.
2. What are the conflicting facts? Print both versions/values, not just "they don't match."
3. What's the fix? State the action ("rebuild the plugin for version X").

A message that does all three turns a P4 attribution from minutes into seconds, and stops the issue being filed on the wrong repo.

Worked Example — a clearer plugin/core version-mismatch error

Step 1 — Find where core enforces version compatibility

PluginsService validates each plugin's descriptor at load. The version check compares the descriptor's opensearch.version to the running node's Version.

grep -n "verifyCompatibility\|isIncompatible\|opensearch.version\|Version.CURRENT" \
  ~/OpenSearch/server/src/main/java/org/opensearch/plugins/PluginsService.java \
  ~/OpenSearch/server/src/main/java/org/opensearch/plugins/PluginInfo.java

You'll find the place that throws when the versions don't match. The current message is typically along the lines of:

plugin [opensearch-knn] is incompatible with version [3.0.0]; was designed for version [3.1.0]

That's okay — it names both versions — but it doesn't say what to do, doesn't distinguish "rebuild the plugin" from "use a different node," and doesn't point at the plugin's repo. We can make it actionable.

Step 2 — Improve the message

Edit the exception construction to add the action and the boundary framing. A representative diff (exact field/method names vary by branch — confirm with the grep above):

--- a/server/src/main/java/org/opensearch/plugins/PluginsService.java
+++ b/server/src/main/java/org/opensearch/plugins/PluginsService.java
@@ public final class PluginsService {
-            throw new IllegalArgumentException(
-                "plugin ["
-                    + info.getName()
-                    + "] is incompatible with version ["
-                    + Version.CURRENT
-                    + "]; was designed for version ["
-                    + info.getOpenSearchVersion()
-                    + "]");
+            throw new IllegalArgumentException(
+                "plugin ["
+                    + info.getName()
+                    + "] was built for OpenSearch ["
+                    + info.getOpenSearchVersion()
+                    + "] but this node is OpenSearch ["
+                    + Version.CURRENT
+                    + "]. Plugins are version-locked to the node because they "
+                    + "compile against internal APIs that are not stable across "
+                    + "versions. Rebuild the plugin against ["
+                    + Version.CURRENT
+                    + "] (set opensearch.version=" + Version.CURRENT + " when "
+                    + "building the plugin) and reinstall it, or run the plugin "
+                    + "on an OpenSearch [" + info.getOpenSearchVersion() + "] node.");

What changed, against the three-question rule:

Step 3 — Add a CHANGELOG entry (every core PR needs one)

--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ ## [Unreleased 3.x]
 ### Changed
+- Make the plugin/core version-mismatch error actionable by naming both versions
+  and the rebuild step ([#NNNNN](https://github.com/opensearch-project/OpenSearch/pull/NNNNN))

Step 4 — Test the new message

A plugin-version-compatibility check has an existing unit test; extend it (don't just eyeball the string). Find it and add an assertion on the new wording:

grep -rln "verifyCompatibility\|incompatible with version\|PluginsServiceTests" \
  ~/OpenSearch/server/src/test/java/org/opensearch/plugins/

// In PluginsServiceTests (illustrative)
IllegalArgumentException e = expectThrows(IllegalArgumentException.class,
    () -> PluginsService.verifyCompatibility(infoBuiltForDifferentVersion));
assertThat(e.getMessage(), containsString("was built for OpenSearch"));
assertThat(e.getMessage(), containsString("Rebuild the plugin"));

cd ~/OpenSearch
./gradlew :server:test --tests "*PluginsServiceTests*"
./gradlew precommit   # checkstyle, headers, forbidden-APIs — required before a PR

Step 5 — Prove it end to end

Reproduce the original confusing failure and confirm the new message:

# Build the plugin against a DIFFERENT version than the distro, then try to install
cd ~/k-NN && ./gradlew assemble -Dopensearch.version=3.1.0   # mismatched on purpose
"$DISTRO/bin/opensearch-plugin" install --batch file://$(ls build/distributions/*3.1.0*.zip) || true
# Start the node (or check the install-time validation) and read the message

The reader of that message now knows it's a version lock, sees both versions, and knows to rebuild — without opening a single source file or filing an issue on the wrong repo.

Other high-leverage boundary diagnostics to consider

The version-mismatch message is one example. The same three-question discipline applies all over the seam:

Each is a small PR; each deflects a category of mis-filed issues.

Why diagnostics are the highest-leverage cross-repo contribution

A contributor who fixes one bug helps one user. A contributor who makes the boundary explain itself helps everyone who hits that boundary, forever. That is why diagnostics, not features, are the fastest route from contributor to trusted maintainer in a multi-repo project.

Expected Output

• The output of the version-mismatch triage triad (_cat/plugins, node version, _nodes/plugins) on your cluster.
• A diff improving the plugin/core version-mismatch message against the three-question rule.
• A CHANGELOG entry and a unit-test assertion on the new wording, with :server:test and precommit passing.
• An end-to-end demonstration: a deliberately mismatched plugin install showing the new message.

Stretch Goals

1. Pick one row from the "other high-leverage diagnostics" table, implement it as a real patch in the relevant repo (core or the plugin), and write the test.
2. Add a DEBUG log line in the security action-filter path (in a security checkout) that names the action and the role evaluated on denial — then verify it appears with security TRACE on (P3).
3. Propose (as a GitHub issue) adding the plugin's git commit, not just version, to _nodes/plugins, and argue why it helps cross-repo repro (P5).

Validation / Self-check

1. State the three questions a good boundary diagnostic must answer in its own message text.
2. Run the version-mismatch triage triad and explain what each of the three calls tells you.
3. Show the before/after of the version-mismatch message and map each change to one of the three questions.
4. Why is a diagnostics patch lower-risk and faster to merge than a behavior fix?
5. Why is improving a boundary message a cross-repo contribution even though you only edited core?
6. Name two boundary diagnostics, other than the version message, that would deflect mis-filed issues, and which repo each lives in.