Lab P2: Inspecting a Plugin's Extension Points

Background

The Plugin Architecture deep dive names the extension interfaces in the abstract. This lab makes them concrete: you will open a real, large out-of-repo plugin, find every place it plugs into core, and map each plugin method to the core SPI interface it implements. When you can read a plugin and immediately list "this is a MapperPlugin because of getMappers(), a SearchPlugin because of getQueries(), and an EnginePlugin because of getEngineFactory()," you can reason about what a core change to any of those interfaces would do to the plugin — which is the Stage 8 skill.

The primary subject is k-NN, because it is the most instructive: it implements four core extension interfaces at once and exercises the full stack (field type, query, engine, REST). The secondary subject is security, which is interesting precisely because it does not add a field type or a query — it implements ActionPlugin and NetworkPlugin to wrap every request in an authorization filter.

Why This Lab Matters for Contributors

Every cross-repo bug attribution starts with "what does this plugin actually extend?" If a knn query returns bad results, you need to know that k-NN owns the query builder (SearchPlugin.getQueries), the field type (MapperPlugin.getMappers), and the engine that stores the vectors (EnginePlugin.getEngineFactory) — so the bug could be in any of three plugin subsystems, or below them in Lucene. If you don't know the plugin↔SPI map, you cannot bisect. This lab builds that map by hand, the only way it sticks.

Prerequisites

git clone https://github.com/opensearch-project/k-NN.git ~/k-NN
git clone https://github.com/opensearch-project/security.git ~/security   # optional

The extension interfaces you will map

# In the core checkout — the SPI surface a plugin implements
ls ~/OpenSearch/server/src/main/java/org/opensearch/plugins/
grep -rln "interface .*Plugin" ~/OpenSearch/server/src/main/java/org/opensearch/plugins/

# Read the method signatures you will look for in the plugin
grep -n "getMappers"        ~/OpenSearch/server/src/main/java/org/opensearch/plugins/MapperPlugin.java
grep -n "getQueries\|getAggregations\|QuerySpec" \
                            ~/OpenSearch/server/src/main/java/org/opensearch/plugins/SearchPlugin.java
grep -n "getEngineFactory"  ~/OpenSearch/server/src/main/java/org/opensearch/plugins/EnginePlugin.java
grep -n "getActionFilters\|getRestHandlers\|getActions" \
                            ~/OpenSearch/server/src/main/java/org/opensearch/plugins/ActionPlugin.java

Step-by-Step Tasks

Step 1 — Find the k-NN plugin entrypoint and its implements clause

The classname in the plugin's descriptor names the Plugin subclass. Find it and read what interfaces it declares — that list is the extension-point map.

# The entrypoint class
find ~/k-NN -name "KNNPlugin.java"
grep -n "class KNNPlugin\|implements\|extends Plugin" \
  $(find ~/k-NN -name "KNNPlugin.java")

You should see it declare (names may vary slightly by branch) something like extends Plugin implements MapperPlugin, SearchPlugin, ActionPlugin, EnginePlugin, ScriptPlugin, SystemIndexPlugin. Each interface there is a contract with core.

Step 2 — Map MapperPlugin.getMappers → the knn_vector field type

grep -n "getMappers" $(find ~/k-NN -name "KNNPlugin.java")
# the field type implementation it registers
find ~/k-NN -name "KNNVectorFieldMapper.java"
grep -n "class KNNVectorFieldMapper\|CONTENT_TYPE\|TypeParser\|MappedFieldType" \
  $(find ~/k-NN -name "KNNVectorFieldMapper.java")

getMappers() returns a map whose key is the content type string ("knn_vector") and whose value is the TypeParser that builds the mapper from mapping JSON. This is exactly the SPI: core's MapperRegistry merges plugin mappers with built-in ones, which is why you can write "type": "knn_vector" in a mapping.

# Core side: where plugin mappers are collected
grep -n "getMappers\|MapperRegistry\|registerMappers" \
  ~/OpenSearch/server/src/main/java/org/opensearch/indices/IndicesModule.java

Step 3 — Map SearchPlugin.getQueries → the knn query

grep -n "getQueries" $(find ~/k-NN -name "KNNPlugin.java")
find ~/k-NN -name "KNNQueryBuilder.java"
grep -n "class KNNQueryBuilder\|NAME\|doToQuery\|fromXContent" \
  $(find ~/k-NN -name "KNNQueryBuilder.java")

getQueries() returns a list of QuerySpec entries; each binds a query name ("knn"), the QueryBuilder class, its Writeable reader (for transport serialization), and its fromXContent parser (for the REST DSL). This is why {"query": {"knn": {...}}} is understood. Note the three registrations per query — name, wire reader, XContent parser — because a query crosses the wire and the JSON boundary.

# Core side: where plugin queries are merged into the registry
grep -n "getQueries\|registerQuery\|QuerySpec" \
  ~/OpenSearch/server/src/main/java/org/opensearch/search/SearchModule.java

Step 4 — Map EnginePlugin.getEngineFactory → the vector store path

This is the subtle one. For native engines (faiss/nmslib), k-NN does not just add a query and field — it can supply a custom EngineFactory so that matching indices store vectors in a way the query can search.

grep -n "getEngineFactory" $(find ~/k-NN -name "KNNPlugin.java")
# the codec / engine glue
grep -rln "EngineFactory\|KNNCodec\|KNN80\|NativeMemory" ~/k-NN/src/main/java | head

getEngineFactory(IndexSettings) returns Optional<EngineFactory>; core calls it per index and, if present, uses that factory instead of the default InternalEngine (see Engine Internals). This is why a k-NN bug can live below the query layer entirely — in how vectors are written and read.

Step 5 — Map ActionPlugin → REST routes and transport actions

grep -n "getRestHandlers\|getActions" $(find ~/k-NN -name "KNNPlugin.java")
# the stats/warmup endpoints k-NN adds
grep -rln "RestKNNStatsHandler\|RestKNNWarmupHandler\|BaseRestHandler" ~/k-NN/src/main/java | head

getActions() registers ActionType → TransportAction pairs (e.g. k-NN stats); getRestHandlers() registers the REST routes that call them. This is the same machinery P1 traced for core's RestSearchAction, just contributed by a plugin.

Step 6 — The contrast: security as ActionPlugin + NetworkPlugin

security adds no field type and no query. It implements ActionPlugin to inject an action filter that runs before every transport action and can deny it, and NetworkPlugin to wrap transport/HTTP with TLS and authentication.

find ~/security -name "OpenSearchSecurityPlugin.java"
grep -n "getActionFilters\|getTransportInterceptors\|getRestHandlers\|implements" \
  $(find ~/security -name "OpenSearchSecurityPlugin.java")
# the filter that authorizes/denies actions
grep -rln "implements ActionFilter\|class SecurityFilter\|apply(.*Task" ~/security/src/main/java | head

# Core side: where action filters are collected and where they run
grep -n "getActionFilters\|ActionFilters\|ActionFilterChain" \
  ~/OpenSearch/server/src/main/java/org/opensearch/action/support/ActionFilters.java

ActionFilter.apply(...) sits in a chain in front of every TransportAction. A 403 you debug in P3 and P4 almost always originates exactly here — in security's filter deciding the principal lacks permission, before core's action logic even runs.

The plugin↔SPI map you produce

Fill this in from your own greps — it is the deliverable:

Step-by-Step: build the plugin against local core

This is the mechanic from the section index, exercised end to end. It proves you can take an unreleased core and test a plugin against it — the core of Stage 8.

# 1. Publish your local core snapshot to ~/.m2
cd ~/OpenSearch
./gradlew publishToMavenLocal
ls ~/.m2/repository/org/opensearch/opensearch/   # see the snapshot version dir

# 2. Build k-NN against that exact version
cd ~/k-NN
./gradlew assemble -Dopensearch.version=$(cat ~/OpenSearch/build.gradle | grep -m1 version || echo 3.1.0-SNAPSHOT)
ls build/distributions/   # opensearch-knn-*.zip

# 3. Build a runnable distro from the SAME core and install the plugin
cd ~/OpenSearch
./gradlew localDistro
DISTRO=$(ls -d distribution/archives/*/build/install/opensearch-* | head -1)
"$DISTRO/bin/opensearch-plugin" install file://$(ls ~/k-NN/build/distributions/*.zip)
"$DISTRO/bin/opensearch-plugin" list

Warning: The plugin's opensearch.version must equal the distro's version exactly. A mismatch fails at load with a compatibility error — this is the single most common cross-repo build failure, covered in P5 and P6.

Then confirm the extension points are live:

curl -s localhost:9200/_cat/plugins?v        # k-NN listed
# the field type from MapperPlugin:
curl -s -XPUT localhost:9200/vec -H 'Content-Type: application/json' -d '{
  "settings": { "index.knn": true },
  "mappings": { "properties": { "v": { "type": "knn_vector", "dimension": 3 } } } }'
# the query from SearchPlugin (after indexing some vectors):
curl -s localhost:9200/vec/_search -H 'Content-Type: application/json' -d '{
  "query": { "knn": { "v": { "vector": [1,2,3], "k": 2 } } } }'

If knn_vector is accepted as a type and knn is accepted as a query, you have proven both getMappers() and getQueries() registered correctly against your local core.

grep exercises across the SPI

Run these and answer in one line each:

# Which extension interfaces does k-NN implement? (count them)
grep -o "implements .*" $(find ~/k-NN -name "KNNPlugin.java")

# How many queries does it register, and what are their NAMEs?
grep -rn "public static final.*NAME\s*=" ~/k-NN/src/main/java | grep -i query

# Does security register any query or mapper? (should be none)
grep -rn "getQueries\|getMappers" ~/security/src/main/java | head

# For each SPI method, where does CORE call it during startup?
grep -rn "\.getMappers()\|\.getQueries()\|\.getEngineFactory(\|\.getActionFilters()" \
  ~/OpenSearch/server/src/main/java | head

Expected Output

• The k-NN implements clause, with each interface annotated by which method and which concrete class implements it.
• The filled-in plugin↔SPI map table.
• A running distro with k-NN installed, knn_vector accepted as a type, and a knn query returning hits.
• A one-paragraph contrast: why security is an ActionPlugin/NetworkPlugin and not a SearchPlugin/MapperPlugin.

Stretch Goals

1. Find one more SPI interface k-NN implements that this lab didn't cover (e.g. ScriptPlugin or SystemIndexPlugin) and explain what it contributes.
2. In the SQL plugin, find which extension interfaces it uses (hint: ActionPlugin for its _sql/_ppl REST endpoints) and contrast with k-NN.
3. Trace EnginePlugin.getEngineFactory from the plugin into core's IndexModule/IndexService to see exactly where core decides whether to use the plugin's engine.

Coding Exercises

Mapping a plugin's extension points by reading is the skill; the proof is writing a plugin that plugs into the same SPI. These exercises build, smallest-first, up to your own multi-interface plugin tested against local core. Locate the real SPI signatures with grep first — they drift by branch, so confirm in your checkout.

1. (warm-up) Annotate the k-NN implements clause into the map, with proof. For each interface k-NN declares (grep -o "implements .*" $(find ~/k-NN -name "KNNPlugin.java")), grep the one method that satisfies it and the concrete class it returns, and write a JUnit test or a bash assertion script that fails if any interface is dropped from the clause. This turns the deliverable table into something a CI run can verify.

2. (core) Build a one-interface MapperPlugin from scratch. In a fresh plugin skeleton (copy the structure of any :modules plugin in core), implement getMappers() to register a trivial field type — e.g. "reversed_keyword" whose TypeParser builds a KeywordFieldMapper-like type that stores the reversed string. Build against local core (publishToMavenLocal → assemble), install on a localDistro, and prove it with curl -XPUT localhost:9200/t -d '{"mappings":{"properties":{"k":{"type":"reversed_keyword"}}}}' returning acknowledged:true. You have now produced the exact getMappers() contract you traced in Step 2.

3. (core) Add a SearchPlugin.getQueries() registration and a unit test. Extend your plugin to implement SearchPlugin and register a tiny custom query (a QueryBuilder with NAME, a Writeable reader, and fromXContent). Write an AbstractQueryTestCase<YourQueryBuilder> (find an example: rg -l "extends AbstractQueryTestCase" ~/OpenSearch/server/src/test) that proves the three registrations — name, wire reader, XContent parser — all work. Run with ./gradlew test. This is the "three registrations per query" point made executable.

4. (core) Add a REST route + TransportAction (the ActionPlugin path). Implement getActions() and getRestHandlers() so your plugin exposes a tiny endpoint (e.g. GET /_plugins/_yours/echo) whose BaseRestHandler calls a TransportAction you register. Find the SPI shape in ~/OpenSearch/server/src/main/java/org/opensearch/plugins/ActionPlugin.java and a real handler to copy (rg -l "extends BaseRestHandler" ~/k-NN/src/main/java). Prove it with curl -s localhost:9200/_plugins/_yours/echo. This is the same machinery P1 traced for RestSearchAction.

5. (core) Cross-plugin TransportAction call. Have your transport action invoke a core action across the SPI seam — e.g. call NodeClient.execute with the SearchAction (or ClusterStatsAction) and return a slice of the result through your REST route. Assert the wiring in an OpenSearchIntegTestCase with nodePlugins() returning your plugin class (the load-bearing line from P5). This proves a plugin action calling a core action — the exact cross-plugin path the whole section is about.

6. (advanced challenge) Ship a multi-SPI plugin and prove every extension point live. Combine the above into one Plugin that implements MapperPlugin, SearchPlugin, and ActionPlugin — your own miniature k-NN. Build against local core, install on a localDistro, and write a yamlRestTest (or an integ test) that exercises all three in one run: create an index with your field type, index a doc, run your custom query, and hit your REST endpoint — asserting each. Then grep where core calls each SPI method during startup (rg -n "\.getMappers\(\)|\.getQueries\(\)|\.getActions\(\)" ~/OpenSearch/server/src/main/java) and write one sentence per method linking your registration to its core call site. This is the Stage 8 skill end to end.

Issues to Practice On

Extension-point work spans core (the SPI) and the plugin (its implementation), so practice on both trackers.

# Core — the SPI surface and plugin infrastructure:
gh issue list --repo opensearch-project/OpenSearch --label "Plugins" --state open
gh issue list --repo opensearch-project/OpenSearch --label "good first issue" --state open
# k-NN — a heavy multi-interface consumer of the SPI:
gh issue list --repo opensearch-project/k-NN --label "good first issue" --state open
gh issue list --repo opensearch-project/k-NN --label "bug" --state open
gh label list --repo opensearch-project/OpenSearch | grep -iE "plugin|extensib"

Labels move; confirm on each tracker before relying on a name.

Two representative patterns:

• "Plugin X stopped registering Y after a core change." A getQueries()/ getMappers() contract drift between core minors. Reproduce by building the plugin against the new core (Step "build against local core"), watch the registration fail or throw NoSuchMethodError, locate the changed SPI signature (rg in core's plugins/), and either fix the plugin or propose a back-compatible core signature. This is the Stage 8 bread-and-butter contribution.
• "New extension point requested" (enhancement). A plugin needs a hook core doesn't yet expose. Approach: propose the SPI method as an RFC/issue first, then implement it minimally in core with a default that preserves existing behavior, plus a unit test that a plugin-supplied implementation is invoked.

Planted-bug drill. In your plugin from the exercises (or a k-NN checkout), break one of the three query registrations: drop the Writeable reader from the QuerySpec in getQueries() (grep -n "getQueries\|QuerySpec\|new QuerySpec" $(find ~/k-NN -name "KNNPlugin.java")). Rebuild and run a query in a 2-node integ test — watch the transport deserialization fail when the query crosses the wire, while a single-node test may stay green. Note which test goes red, then restore the reader and add an integ-test assertion (@ClusterScope(numDataNodes = 2)) that forces the query across the wire so the missing reader would always be caught.

Etiquette: Claim an issue before working it; reproduce first; every PR needs a test + a CHANGELOG entry + a DCO Signed-off-by (git commit -s). SPI changes in core usually want an RFC/discuss issue before the PR. See community interaction and the PR-review lab.

Validation / Self-check

1. From memory, name the SPI interface for each of: a custom field type, a custom query, a custom engine, a custom REST endpoint, and an authorization filter.
2. For the knn query, why are there three registrations (name, wire reader, XContent parser) and what would break if one were omitted?
3. Why is security an ActionPlugin but not a SearchPlugin? What does that tell you about where a 403 comes from?
4. Walk publishToMavenLocal → plugin assemble → opensearch-plugin install and say why the version must match at each step.
5. Given "a knn query returns wrong neighbors," list the three plugin subsystems (field type, query, engine) that could own it and how you'd narrow down.
6. Where in core is each SPI method called during node startup? Name one file.