Stage 8 — Plugin and Extension Compatibility

What this stage teaches

Stage 8 is the OpenSearch analog of the Tez "Hive-on-Tez" attribution skill: most of OpenSearch's value (security, k-NN, SQL/PPL, alerting, ml-commons, index-management, cross-cluster-replication) lives in separate repositories that build against the core engine through extension points. A change to a core extension point can break those plugins without a single test failing in the core repo. The skill is:

• Reasoning about the plugin SPI: org.opensearch.plugins.Plugin and the interfaces a plugin implements (ActionPlugin, SearchPlugin, MapperPlugin, AnalysisPlugin, IngestPlugin, ClusterPlugin, EnginePlugin, NetworkPlugin, RepositoryPlugin, ExtensiblePlugin, …).
• Recognising when a core change is source- or binary-incompatible for plugins (a method signature change, a removed registration helper, a new abstract method) versus a safe, additive change.
• Attributing a reported break correctly: is the fault in core (it broke the contract), in the plugin (it relied on an internal it should not have), or in the build (a version mismatch)? Filing in the right repo is half the work.
• Testing a core change against a real plugin by building the plugin against your local core snapshot.

Prerequisite: whichever of Stages 4–7 covers the subsystem the extension point belongs to, plus the plugin architecture deep dive and the plugin labs. This stage assumes you know how PluginsService loads a plugin and gives it an isolated classloader.

The boundary

flowchart LR
  subgraph Core[opensearch-project/OpenSearch]
    SPI[Plugin + ActionPlugin/SearchPlugin/...<br/>extension-point interfaces]
    REG[ActionModule / SearchModule<br/>registries]
  end
  subgraph Plugins[separate repos]
    SEC[security]
    KNN[k-NN]
    SQL[sql]
    ML[ml-commons]
  end
  SPI -. implemented by .-> SEC
  SPI -. implemented by .-> KNN
  SPI -. implemented by .-> SQL
  SPI -. implemented by .-> ML
  Plugins -->|build against published<br/>org.opensearch:opensearch| Core

The contract is the set of public/protected signatures on the Plugin interfaces and the registration types they return (ActionHandler, RestHandler, QuerySpec, AggregationSpec, Mapper.TypeParser, …). Anything a plugin can see is part of the contract — including a return type you "just" widened.

Finding Stage 8 issues

is:issue is:open label:bug no:assignee "plugin" "compatibility" in:body
is:issue is:open label:bug no:assignee "extension point" in:body
is:issue is:open label:"help wanted" no:assignee label:"Plugins"

Cross-repo breaks are also reported as issues in the plugin repos themselves ("k-NN fails to build against OpenSearch 3.x main"). Search those repos too:

repo:opensearch-project/k-NN is:issue is:open "build against" OR "incompatible"
repo:opensearch-project/security is:issue is:open "API change" OR "removed"

Fallback grep — find the extension-point interfaces and the registries that back them:

ls libs/*/src/main/java/org/opensearch/plugins/ server/src/main/java/org/opensearch/plugins/ 2>/dev/null
grep -rln "interface .*Plugin" server/src/main/java/org/opensearch/plugins/
grep -n "registerQuery\|registerAggregation\|QuerySpec\|AggregationSpec" \
  server/src/main/java/org/opensearch/search/SearchModule.java | head

Walked example — a core change that forces a plugin registration update

Illustrative of the pattern. The grep finds the real registration method; treat the details as a stand-in for "an extension-point signature evolved."

Symptom: a core PR adds a required parameter to the SearchPlugin.QuerySpec constructor (say, to thread a NamedXContentRegistry or a new reader). Core compiles and its tests pass — but every out-of-repo SearchPlugin that constructs a QuerySpec now fails to compile against the new snapshot. The k-NN plugin's nightly build against main goes red.

Locate the contract that changed

grep -rn "class QuerySpec\|QuerySpec(" server/src/main/java/org/opensearch/plugins/SearchPlugin.java
git log --oneline -n 5 -- server/src/main/java/org/opensearch/plugins/SearchPlugin.java
# Where core itself registers queries (your in-repo callers to update):
grep -rn "new QuerySpec\|getQueries" server/src/main/java/org/opensearch/ modules/ plugins/ | head

Decide: is this break necessary?

The first question is not "how do I fix the plugin" but "should core have broken the contract at all?" Two options:

1. Additive (preferred). Add a new constructor/overload and deprecate the old one, so existing plugins keep compiling and migrate at their own pace.
2. Breaking (only if justified). If the old signature cannot be kept, the change must be announced (a breaking label, a CHANGELOG ### Changed with a migration note) and the plugin repos must be updated in lockstep.

The additive diff:

--- a/server/src/main/java/org/opensearch/plugins/SearchPlugin.java
+++ b/server/src/main/java/org/opensearch/plugins/SearchPlugin.java
@@  public static class QuerySpec<T extends QueryBuilder> {
+        /**
+         * @deprecated use {@link #QuerySpec(ParseField, Writeable.Reader, ContextParser)}
+         *             which threads the parse context. Retained for plugin source compatibility.
+         */
+        @Deprecated
         public QuerySpec(String name, Writeable.Reader<T> reader, QueryParser<T> parser) {
-            this(new ParseField(name), reader, parser);
+            this(new ParseField(name), reader, (p, c) -> parser.fromXContent(p));
         }
+        public QuerySpec(ParseField name, Writeable.Reader<T> reader, ContextParser<Object, T> parser) {
+            // new canonical constructor
+        }

Warning: "Just deprecate, don't delete" is the default posture at the core↔plugin boundary. Deleting a public method that a downstream plugin uses is a breaking change that needs maintainer sign-off and a coordinated migration. When in doubt, keep the old path.

Update in-repo callers and the CHANGELOG

grep -rn "new QuerySpec(" server/ modules/ plugins/
# update each in-repo caller to the new constructor where appropriate
./gradlew :server:compileJava :modules:compileJava -q

--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ ### Changed
+- Add a context-aware `SearchPlugin.QuerySpec` constructor and deprecate the legacy one ([#NNNNN](...))

Test the core change against a real plugin

This is the Stage 8 skill that no in-repo test gives you. Build core locally, publish it to your local Maven, then build a plugin against it:

# 1. In the core repo: publish your branch as a local snapshot.
./gradlew publishToMavenLocal -Dbuild.version_qualifier=SNAPSHOT
# note the version it published, e.g. 3.2.0-SNAPSHOT

# 2. In a plugin repo (e.g. k-NN), point it at your local snapshot and build.
git clone https://github.com/opensearch-project/k-NN.git
cd k-NN
./gradlew build -Dopensearch.version=3.2.0-SNAPSHOT

If the plugin builds with the deprecated path intact, your additive change is safe. If you had to make it breaking, this build is where you discover every call site the plugin must change — which you then file as a coordinated issue/PR in the plugin repo, linked from the core PR.

Note: Plugins consume core via -Dopensearch.version=... and the opensearch.opensearchplugin Gradle plugin; the exact flag name varies by plugin repo — check that repo's DEVELOPER_GUIDE.md. The principle is identical: publish core locally, resolve the plugin against it.

Open the PR(s)

• The core PR carries the additive change, the in-repo caller updates, the CHANGELOG note, and a line in the description: "verified against k-NN built with -Dopensearch.version=<snapshot>."
• If breaking, a companion plugin PR (or a tracking issue in the plugin repo) migrates the plugin. Link them. Maintainers will not merge a knowingly-breaking core change until the downstream migration path exists.

Attribution — whose bug is it?

When someone reports "feature X broke after upgrading," the most valuable thing you can do is attribute it correctly. Walk the boundary:

The decisive grep when you suspect overreach is whether the symbol the plugin used is part of the public surface:

grep -rn "@PublicApi\|@InternalApi\|@DeprecatedApi" \
  libs/core/src/main/java/org/opensearch/core/common/ | head   # annotation usage
grep -rn "class TheSymbolThePluginUsed" server/src/main/java/  # is it even public?

If the plugin imported a non-public core internal, the fix is in the plugin (or a request to core to promote the symbol to public API) — not a revert of the core change.

Pitfalls

• Breaking a public extension point without deprecation. The default is additive + @Deprecated. Deletion needs maintainer sign-off and a coordinated downstream migration.
• Assuming "core tests green" means "plugins fine." Core has no visibility into k-NN or security. The publishToMavenLocal → plugin-build loop is the only proof.
• Filing the bug in the wrong repo. A plugin that overreached into a core internal is a plugin bug. A core change that broke a public method is a core bug. Attribute before you patch.
• Forgetting the descriptor version pin. Plugins encode a compatible OpenSearch version in plugin-descriptor.properties; a core version bump can require a coordinated descriptor bump in every plugin repo.
• Ignoring the Extensions SDK. OpenSearch 2.10+ has an experimental out-of-process Extensions model (opensearch-sdk-java). It is forward-looking; plugins remain the mainstream model, but if your change touches ExtensiblePlugin/loadExtensions, mention the extensions implications.
• Mentioning a plugin maintainer without doing the build first. Bring evidence (the plugin build log against your snapshot), not a guess.

Exit criteria — when you're ready for Stage 9

• One core change at the plugin boundary is merged, made additive where possible, and you verified it against a real plugin via publishToMavenLocal + a plugin build.
• You can attribute a reported break to core, plugin, or build, with a grep that proves whether the disputed symbol is public API.
• You defaulted to deprecate-don't-delete and only went breaking with a maintainer-approved, coordinated downstream migration.
• You know how a plugin resolves core (-Dopensearch.version, the version catalog, the descriptor) well enough to reproduce its build.

You have worked the core↔plugin seam. Stage 9 turns to the tests themselves — the flaky ones that erode trust in every PR's CI.