Lab P2: Inspecting a Plugin's Extension Points

Background

The Plugin Architecture deep dive names the extension interfaces in the abstract. This lab makes them concrete: you will open a real, large out-of-repo plugin, find every place it plugs into core, and map each plugin method to the core SPI interface it implements. When you can read a plugin and immediately list "this is a MapperPlugin because of getMappers(), a SearchPlugin because of getQueries(), and an EnginePlugin because of getEngineFactory()," you can reason about what a core change to any of those interfaces would do to the plugin — which is the Stage 8 skill.

The primary subject is k-NN, because it is the most instructive: it implements four core extension interfaces at once and exercises the full stack (field type, query, engine, REST). The secondary subject is security, which is interesting precisely because it does not add a field type or a query — it implements ActionPlugin and NetworkPlugin to wrap every request in an authorization filter.

Why This Lab Matters for Contributors

Every cross-repo bug attribution starts with "what does this plugin actually extend?" If a knn query returns bad results, you need to know that k-NN owns the query builder (SearchPlugin.getQueries), the field type (MapperPlugin.getMappers), and the engine that stores the vectors (EnginePlugin.getEngineFactory) — so the bug could be in any of three plugin subsystems, or below them in Lucene. If you don't know the plugin↔SPI map, you cannot bisect. This lab builds that map by hand, the only way it sticks.

Prerequisites

git clone https://github.com/opensearch-project/k-NN.git ~/k-NN
git clone https://github.com/opensearch-project/security.git ~/security   # optional

The extension interfaces you will map

# In the core checkout — the SPI surface a plugin implements
ls ~/OpenSearch/server/src/main/java/org/opensearch/plugins/
grep -rln "interface .*Plugin" ~/OpenSearch/server/src/main/java/org/opensearch/plugins/

# Read the method signatures you will look for in the plugin
grep -n "getMappers"        ~/OpenSearch/server/src/main/java/org/opensearch/plugins/MapperPlugin.java
grep -n "getQueries\|getAggregations\|QuerySpec" \
                            ~/OpenSearch/server/src/main/java/org/opensearch/plugins/SearchPlugin.java
grep -n "getEngineFactory"  ~/OpenSearch/server/src/main/java/org/opensearch/plugins/EnginePlugin.java
grep -n "getActionFilters\|getRestHandlers\|getActions" \
                            ~/OpenSearch/server/src/main/java/org/opensearch/plugins/ActionPlugin.java

Step-by-Step Tasks

Step 1 — Find the k-NN plugin entrypoint and its implements clause

The classname in the plugin's descriptor names the Plugin subclass. Find it and read what interfaces it declares — that list is the extension-point map.

# The entrypoint class
find ~/k-NN -name "KNNPlugin.java"
grep -n "class KNNPlugin\|implements\|extends Plugin" \
  $(find ~/k-NN -name "KNNPlugin.java")

You should see it declare (names may vary slightly by branch) something like extends Plugin implements MapperPlugin, SearchPlugin, ActionPlugin, EnginePlugin, ScriptPlugin, SystemIndexPlugin. Each interface there is a contract with core.

Step 2 — Map MapperPlugin.getMappers → the knn_vector field type

grep -n "getMappers" $(find ~/k-NN -name "KNNPlugin.java")
# the field type implementation it registers
find ~/k-NN -name "KNNVectorFieldMapper.java"
grep -n "class KNNVectorFieldMapper\|CONTENT_TYPE\|TypeParser\|MappedFieldType" \
  $(find ~/k-NN -name "KNNVectorFieldMapper.java")

getMappers() returns a map whose key is the content type string ("knn_vector") and whose value is the TypeParser that builds the mapper from mapping JSON. This is exactly the SPI: core's MapperRegistry merges plugin mappers with built-in ones, which is why you can write "type": "knn_vector" in a mapping.

# Core side: where plugin mappers are collected
grep -n "getMappers\|MapperRegistry\|registerMappers" \
  ~/OpenSearch/server/src/main/java/org/opensearch/indices/IndicesModule.java

Step 3 — Map SearchPlugin.getQueries → the knn query

grep -n "getQueries" $(find ~/k-NN -name "KNNPlugin.java")
find ~/k-NN -name "KNNQueryBuilder.java"
grep -n "class KNNQueryBuilder\|NAME\|doToQuery\|fromXContent" \
  $(find ~/k-NN -name "KNNQueryBuilder.java")

getQueries() returns a list of QuerySpec entries; each binds a query name ("knn"), the QueryBuilder class, its Writeable reader (for transport serialization), and its fromXContent parser (for the REST DSL). This is why {"query": {"knn": {...}}} is understood. Note the three registrations per query — name, wire reader, XContent parser — because a query crosses the wire and the JSON boundary.

# Core side: where plugin queries are merged into the registry
grep -n "getQueries\|registerQuery\|QuerySpec" \
  ~/OpenSearch/server/src/main/java/org/opensearch/search/SearchModule.java

Step 4 — Map EnginePlugin.getEngineFactory → the vector store path

This is the subtle one. For native engines (faiss/nmslib), k-NN does not just add a query and field — it can supply a custom EngineFactory so that matching indices store vectors in a way the query can search.

grep -n "getEngineFactory" $(find ~/k-NN -name "KNNPlugin.java")
# the codec / engine glue
grep -rln "EngineFactory\|KNNCodec\|KNN80\|NativeMemory" ~/k-NN/src/main/java | head

getEngineFactory(IndexSettings) returns Optional<EngineFactory>; core calls it per index and, if present, uses that factory instead of the default InternalEngine (see Engine Internals). This is why a k-NN bug can live below the query layer entirely — in how vectors are written and read.

Step 5 — Map ActionPlugin → REST routes and transport actions

grep -n "getRestHandlers\|getActions" $(find ~/k-NN -name "KNNPlugin.java")
# the stats/warmup endpoints k-NN adds
grep -rln "RestKNNStatsHandler\|RestKNNWarmupHandler\|BaseRestHandler" ~/k-NN/src/main/java | head

getActions() registers ActionType → TransportAction pairs (e.g. k-NN stats); getRestHandlers() registers the REST routes that call them. This is the same machinery P1 traced for core's RestSearchAction, just contributed by a plugin.

Step 6 — The contrast: security as ActionPlugin + NetworkPlugin

security adds no field type and no query. It implements ActionPlugin to inject an action filter that runs before every transport action and can deny it, and NetworkPlugin to wrap transport/HTTP with TLS and authentication.

find ~/security -name "OpenSearchSecurityPlugin.java"
grep -n "getActionFilters\|getTransportInterceptors\|getRestHandlers\|implements" \
  $(find ~/security -name "OpenSearchSecurityPlugin.java")
# the filter that authorizes/denies actions
grep -rln "implements ActionFilter\|class SecurityFilter\|apply(.*Task" ~/security/src/main/java | head

# Core side: where action filters are collected and where they run
grep -n "getActionFilters\|ActionFilters\|ActionFilterChain" \
  ~/OpenSearch/server/src/main/java/org/opensearch/action/support/ActionFilters.java

ActionFilter.apply(...) sits in a chain in front of every TransportAction. A 403 you debug in P3 and P4 almost always originates exactly here — in security's filter deciding the principal lacks permission, before core's action logic even runs.

The plugin↔SPI map you produce

Fill this in from your own greps — it is the deliverable:

Step-by-Step: build the plugin against local core

This is the mechanic from the section index, exercised end to end. It proves you can take an unreleased core and test a plugin against it — the core of Stage 8.

# 1. Publish your local core snapshot to ~/.m2
cd ~/OpenSearch
./gradlew publishToMavenLocal
ls ~/.m2/repository/org/opensearch/opensearch/   # see the snapshot version dir

# 2. Build k-NN against that exact version
cd ~/k-NN
./gradlew assemble -Dopensearch.version=$(cat ~/OpenSearch/build.gradle | grep -m1 version || echo 3.1.0-SNAPSHOT)
ls build/distributions/   # opensearch-knn-*.zip

# 3. Build a runnable distro from the SAME core and install the plugin
cd ~/OpenSearch
./gradlew localDistro
DISTRO=$(ls -d distribution/archives/*/build/install/opensearch-* | head -1)
"$DISTRO/bin/opensearch-plugin" install file://$(ls ~/k-NN/build/distributions/*.zip)
"$DISTRO/bin/opensearch-plugin" list

Warning: The plugin's opensearch.version must equal the distro's version exactly. A mismatch fails at load with a compatibility error — this is the single most common cross-repo build failure, covered in P5 and P6.

Then confirm the extension points are live:

curl -s localhost:9200/_cat/plugins?v        # k-NN listed
# the field type from MapperPlugin:
curl -s -XPUT localhost:9200/vec -H 'Content-Type: application/json' -d '{
  "settings": { "index.knn": true },
  "mappings": { "properties": { "v": { "type": "knn_vector", "dimension": 3 } } } }'
# the query from SearchPlugin (after indexing some vectors):
curl -s localhost:9200/vec/_search -H 'Content-Type: application/json' -d '{
  "query": { "knn": { "v": { "vector": [1,2,3], "k": 2 } } } }'

If knn_vector is accepted as a type and knn is accepted as a query, you have proven both getMappers() and getQueries() registered correctly against your local core.

grep exercises across the SPI

Run these and answer in one line each:

# Which extension interfaces does k-NN implement? (count them)
grep -o "implements .*" $(find ~/k-NN -name "KNNPlugin.java")

# How many queries does it register, and what are their NAMEs?
grep -rn "public static final.*NAME\s*=" ~/k-NN/src/main/java | grep -i query

# Does security register any query or mapper? (should be none)
grep -rn "getQueries\|getMappers" ~/security/src/main/java | head

# For each SPI method, where does CORE call it during startup?
grep -rn "\.getMappers()\|\.getQueries()\|\.getEngineFactory(\|\.getActionFilters()" \
  ~/OpenSearch/server/src/main/java | head

Expected Output

• The k-NN implements clause, with each interface annotated by which method and which concrete class implements it.
• The filled-in plugin↔SPI map table.
• A running distro with k-NN installed, knn_vector accepted as a type, and a knn query returning hits.
• A one-paragraph contrast: why security is an ActionPlugin/NetworkPlugin and not a SearchPlugin/MapperPlugin.

Stretch Goals

1. Find one more SPI interface k-NN implements that this lab didn't cover (e.g. ScriptPlugin or SystemIndexPlugin) and explain what it contributes.
2. In the SQL plugin, find which extension interfaces it uses (hint: ActionPlugin for its _sql/_ppl REST endpoints) and contrast with k-NN.
3. Trace EnginePlugin.getEngineFactory from the plugin into core's IndexModule/IndexService to see exactly where core decides whether to use the plugin's engine.

Validation / Self-check

1. From memory, name the SPI interface for each of: a custom field type, a custom query, a custom engine, a custom REST endpoint, and an authorization filter.
2. For the knn query, why are there three registrations (name, wire reader, XContent parser) and what would break if one were omitted?
3. Why is security an ActionPlugin but not a SearchPlugin? What does that tell you about where a 403 comes from?
4. Walk publishToMavenLocal → plugin assemble → opensearch-plugin install and say why the version must match at each step.
5. Given "a knn query returns wrong neighbors," list the three plugin subsystems (field type, query, engine) that could own it and how you'd narrow down.
6. Where in core is each SPI method called during node startup? Name one file.