Lab K4: Build It — A Custom k-NN Feature

Background

You have read k-NN architecture: one KNNPlugin class implements eleven extension interfaces, and ActionPlugin is the one that registers REST handlers and transport actions — the warmup API, the stats API, the train API, model CRUD. Each of those is the same three-part shape: a RestHandler that parses the HTTP request, a TransportAction that does the work (often fanned out to data nodes), and a *Request/*Response pair that serialize across the transport layer. Once you have seen that shape once, every k-NN action is a variation on it.

This is a build-it lab. You will add a brand-new REST endpoint to the k-NN plugin end to end, with real Java: a *Request/*Response, a TransportNodesAction that collects per-node native-memory cache statistics, a RestHandler, and the registration lines in KNNPlugin. You will build the plugin (linking lab-k1-build-knn-from-source), install it into a running OpenSearch, and curl your endpoint. You will write a unit test. By the end you will have done the single most transferable thing in OpenSearch plugin work: shipped a new action through the full REST → transport → node fan-out → response machinery.

Note on terminology: the cluster manager (formerly master) owns cluster state, but the metric you are exposing — native-memory cache stats — is a per-data-node, per-shard concern. Your action is therefore a nodes action (fan out to every node, each answers locally, the coordinating node aggregates), not a cluster-manager action. This is the same fan-out shape as GET /_plugins/_knn/stats and the warmup API.

Why This Matters for Contributors

The REST → transport → nodes-action triad is the most common kind of feature PR in OpenSearch and its plugins — a new stat, admin endpoint, or diagnostic is always this shape. Learn it once, reuse it forever.
k-NN's native memory is invisible to every JVM tool (see native integration and memory), so adding a stat is genuinely useful, mergeable work, not a toy.
You touch KNNPlugin registration directly (getActions() / getRestHandlers()), which demystifies the whole plugin architecture, and you must get serialization right — the foundation of backward compatibility.

Pick Your Build

Pick one and implement it end to end. They share the same skeleton; they differ in where the data comes from. The walkthrough below implements option A in full; options B and C are sketched so you can choose your own and reuse the scaffolding.

Option	Endpoint	What it returns	Data source
A (walkthrough)	`GET /_plugins/_knn/cachestats`	per-node native-memory cache size, hit/miss/eviction counts, capacity	`NativeMemoryCacheManager` on each node
B	a new `SpaceType` / scoring function	a custom distance (e.g. a weighted-L2) usable in `space_type` or `knn_score` script	`org.opensearch.knn.index.SpaceType` + the script scoring path
C	`GET /_plugins/_knn/graphinfo/<index>`	per-segment graph metadata (engine, method, vector count) for an index	the k-NN codec's per-segment files / field info

Warning: Do not name your endpoint /_plugins/_knn/stats — that already exists (KNNStatsAction). Pick a distinct path (cachestats, graphinfo) so you do not collide with a registered route and get a startup failure. Grep the existing routes first (below) to be sure.

Prerequisites

lab-k1-build-knn-from-source completed — you can build the plugin (Java + the jni/ native libs via CMake) and have a checkout.
You have read k-NN architecture (especially getActions() / getRestHandlers()) and native integration and memory (what NativeMemoryCacheManager tracks).
A local OpenSearch you can install the plugin into, or the k-NN repo's run task.
JDK 21, Gradle, a C++ toolchain + CMake (for the native step).

# Orient yourself: where the existing actions and rest handlers live.
cd ~/src/oss-repos/k-NN
ls src/main/java/org/opensearch/knn/plugin/transport   # *TransportAction, *Request, *Response
ls src/main/java/org/opensearch/knn/plugin/rest        # Rest*Handler
# Confirm the registration methods and the existing routes (names drift — grep, don't trust):
grep -n "getActions\|getRestHandlers\|registerHandler\|new Route" \
  src/main/java/org/opensearch/knn/plugin/KNNPlugin.java
grep -rn "_plugins/_knn" src/main/java/org/opensearch/knn/plugin/rest

Step-by-Step (Option A: a native-memory cache-stats endpoint)

The endpoint fans a request out to every node, each node reads its own NativeMemoryCacheManager, and the coordinating node concatenates the per-node answers. This is OpenSearch's TransportNodesAction pattern. Five files, then two registration lines.

Step 1: The per-node `Request`/`Response`

A nodes action has a top-level request/response (the whole operation) and a per-node request/response. Start with the per-node response — the actual payload one node reports.

/*
 * SPDX-License-Identifier: Apache-2.0
 *
 * The OpenSearch Contributors require contributions made to
 * this file be licensed under the Apache-2.0 license or a
 * compatible open source license.
 */

package org.opensearch.knn.plugin.transport;

import org.opensearch.action.support.nodes.BaseNodeResponse;
import org.opensearch.cluster.node.DiscoveryNode;
import org.opensearch.core.common.io.stream.StreamInput;
import org.opensearch.core.common.io.stream.StreamOutput;
import org.opensearch.core.xcontent.ToXContentFragment;
import org.opensearch.core.xcontent.XContentBuilder;

import java.io.IOException;

/** One node's view of the native-memory cache. */
public class KNNCacheStatsNodeResponse extends BaseNodeResponse implements ToXContentFragment {

    private final long graphMemoryKb;
    private final long hitCount;
    private final long missCount;
    private final long evictionCount;
    private final long cacheCapacityKb;

    public KNNCacheStatsNodeResponse(
        DiscoveryNode node, long graphMemoryKb, long hitCount,
        long missCount, long evictionCount, long cacheCapacityKb) {
        super(node);
        this.graphMemoryKb = graphMemoryKb;
        this.hitCount = hitCount;
        this.missCount = missCount;
        this.evictionCount = evictionCount;
        this.cacheCapacityKb = cacheCapacityKb;
    }

    /** Wire-format read. Field ORDER must match writeTo exactly. */
    public KNNCacheStatsNodeResponse(StreamInput in) throws IOException {
        super(in);
        this.graphMemoryKb = in.readLong();
        this.hitCount = in.readLong();
        this.missCount = in.readLong();
        this.evictionCount = in.readLong();
        this.cacheCapacityKb = in.readLong();
    }

    @Override
    public void writeTo(StreamOutput out) throws IOException {
        super.writeTo(out);
        out.writeLong(graphMemoryKb);
        out.writeLong(hitCount);
        out.writeLong(missCount);
        out.writeLong(evictionCount);
        out.writeLong(cacheCapacityKb);
    }

    @Override
    public XContentBuilder toXContent(XContentBuilder builder, Params params) throws IOException {
        builder.startObject(getNode().getId());
        builder.field("node_name", getNode().getName());
        builder.field("graph_memory_usage_kb", graphMemoryKb);
        builder.field("cache_capacity_kb", cacheCapacityKb);
        builder.field("hit_count", hitCount);
        builder.field("miss_count", missCount);
        builder.field("eviction_count", evictionCount);
        return builder.endObject();
    }

    long getGraphMemoryKb()  { return graphMemoryKb; }
    long getHitCount()       { return hitCount; }
    long getMissCount()      { return missCount; }
    long getEvictionCount()  { return evictionCount; }
    long getCacheCapacityKb(){ return cacheCapacityKb; }
}

Warning: The order of reads in the StreamInput constructor must match the order of writes in writeTo, field for field. A mismatch does not fail at compile time — it corrupts the stream at runtime and only shows up as a garbled response or a deserialization exception between nodes. This is the #1 mistake in transport code; see serialization & BWC.

The top-level request/response wrap the per-node ones. The request is a BaseNodesRequest (it carries the node selection); the response is a BaseNodesResponse<KNNCacheStatsNodeResponse> that holds the list and renders XContent.

package org.opensearch.knn.plugin.transport;

import org.opensearch.action.support.nodes.BaseNodesRequest;
import org.opensearch.core.common.io.stream.StreamInput;
import org.opensearch.core.common.io.stream.StreamOutput;
import java.io.IOException;

/** Top-level request: which nodes to ask. Empty body — node selection is in the base. */
public class KNNCacheStatsRequest extends BaseNodesRequest<KNNCacheStatsRequest> {
    public KNNCacheStatsRequest(String... nodeIds) { super(nodeIds); }
    public KNNCacheStatsRequest(StreamInput in) throws IOException { super(in); }
    @Override public void writeTo(StreamOutput out) throws IOException { super.writeTo(out); }
}

package org.opensearch.knn.plugin.transport;

import org.opensearch.action.FailedNodeException;
import org.opensearch.action.support.nodes.BaseNodesResponse;
import org.opensearch.cluster.ClusterName;
import org.opensearch.core.common.io.stream.StreamInput;
import org.opensearch.core.common.io.stream.StreamOutput;
import org.opensearch.core.xcontent.ToXContentObject;
import org.opensearch.core.xcontent.XContentBuilder;

import java.io.IOException;
import java.util.List;

/** Top-level response: the per-node answers, plus any node failures, as XContent. */
public class KNNCacheStatsResponse
        extends BaseNodesResponse<KNNCacheStatsNodeResponse> implements ToXContentObject {

    public KNNCacheStatsResponse(
        ClusterName clusterName, List<KNNCacheStatsNodeResponse> nodes,
        List<FailedNodeException> failures) {
        super(clusterName, nodes, failures);
    }

    public KNNCacheStatsResponse(StreamInput in) throws IOException { super(in); }

    @Override
    protected List<KNNCacheStatsNodeResponse> readNodesFrom(StreamInput in) throws IOException {
        return in.readList(KNNCacheStatsNodeResponse::new);
    }

    @Override
    protected void writeNodesTo(StreamOutput out, List<KNNCacheStatsNodeResponse> nodes) throws IOException {
        out.writeList(nodes);
    }

    @Override
    public XContentBuilder toXContent(XContentBuilder builder, Params params) throws IOException {
        builder.startObject();
        builder.startObject("nodes");
        for (KNNCacheStatsNodeResponse node : getNodes()) {
            node.toXContent(builder, params);
        }
        builder.endObject();
        return builder.endObject();
    }
}

Step 2: The `TransportNodesAction`

This is the engine of the action. TransportNodesAction handles the fan-out: it sends a per-node request to each selected node, calls nodeOperation on each, and gathers the KNNCacheStatsNodeResponses into a KNNCacheStatsResponse. You implement the four abstract methods.

package org.opensearch.knn.plugin.transport;

import org.opensearch.action.FailedNodeException;
import org.opensearch.action.support.ActionFilters;
import org.opensearch.action.support.nodes.BaseNodeRequest;
import org.opensearch.action.support.nodes.TransportNodesAction;
import org.opensearch.cluster.service.ClusterService;
import org.opensearch.common.inject.Inject;
import org.opensearch.core.common.io.stream.StreamInput;
import org.opensearch.core.common.io.stream.StreamOutput;
import org.opensearch.knn.index.memory.NativeMemoryCacheManager;
import org.opensearch.threadpool.ThreadPool;
import org.opensearch.transport.TransportRequest;
import org.opensearch.transport.TransportService;

import java.io.IOException;
import java.util.List;

public class KNNCacheStatsTransportAction extends TransportNodesAction<
        KNNCacheStatsRequest,
        KNNCacheStatsResponse,
        KNNCacheStatsTransportAction.NodeRequest,
        KNNCacheStatsNodeResponse> {

    @Inject
    public KNNCacheStatsTransportAction(
        ThreadPool threadPool, ClusterService clusterService,
        TransportService transportService, ActionFilters actionFilters) {
        super(
            KNNCacheStatsAction.NAME, threadPool, clusterService, transportService, actionFilters,
            KNNCacheStatsRequest::new, NodeRequest::new,
            ThreadPool.Names.MANAGEMENT, KNNCacheStatsNodeResponse.class);
    }

    @Override
    protected KNNCacheStatsResponse newResponse(
        KNNCacheStatsRequest request, List<KNNCacheStatsNodeResponse> nodes,
        List<FailedNodeException> failures) {
        return new KNNCacheStatsResponse(clusterService.getClusterName(), nodes, failures);
    }

    @Override
    protected NodeRequest newNodeRequest(KNNCacheStatsRequest request) {
        return new NodeRequest();
    }

    @Override
    protected KNNCacheStatsNodeResponse newNodeResponse(StreamInput in) throws IOException {
        return new KNNCacheStatsNodeResponse(in);
    }

    /** Runs on EACH node: read this node's native-memory cache and report it. */
    @Override
    protected KNNCacheStatsNodeResponse nodeOperation(NodeRequest request) {
        NativeMemoryCacheManager cm = NativeMemoryCacheManager.getInstance();
        // Method names below are illustrative — grep NativeMemoryCacheManager for the
        // real accessors in your version (getCacheSizeInKilobytes, getCacheStats, etc.).
        long graphMemKb     = cm.getCacheSizeInKilobytes();
        long maxCacheKb     = cm.getMaxCacheSizeInKilobytes();
        long hits           = cm.getHitCount();
        long misses         = cm.getMissCount();
        long evictions      = cm.getEvictionCount();
        return new KNNCacheStatsNodeResponse(
            clusterService.localNode(), graphMemKb, hits, misses, evictions, maxCacheKb);
    }

    /** Per-node request — empty; node selection lives in the top-level request. */
    public static class NodeRequest extends BaseNodeRequest {
        public NodeRequest() {}
        public NodeRequest(StreamInput in) throws IOException { super(in); }
        @Override public void writeTo(StreamOutput out) throws IOException { super.writeTo(out); }
    }
}

The ActionType that names the action (used in registration and to call it):

package org.opensearch.knn.plugin.transport;

import org.opensearch.action.ActionType;

public class KNNCacheStatsAction extends ActionType<KNNCacheStatsResponse> {
    public static final KNNCacheStatsAction INSTANCE = new KNNCacheStatsAction();
    public static final String NAME = "cluster:admin/knn_cache_stats_action";
    private KNNCacheStatsAction() { super(NAME, KNNCacheStatsResponse::new); }
}

Note: The action name cluster:admin/... is the transport-level identifier, not the HTTP route. The cluster:admin/ prefix tells the security layer this is an admin action. Keep it consistent with the existing k-NN actions — grep transport/KNN*Action.java for public static final String NAME.

Step 3: The `RestHandler`

The REST handler maps GET /_plugins/_knn/cachestats to your transport action.

package org.opensearch.knn.plugin.rest;

import org.opensearch.client.node.NodeClient;
import org.opensearch.knn.plugin.transport.KNNCacheStatsAction;
import org.opensearch.knn.plugin.transport.KNNCacheStatsRequest;
import org.opensearch.rest.BaseRestHandler;
import org.opensearch.rest.RestRequest;
import org.opensearch.rest.action.RestActions.NodesResponseRestListener;

import java.util.List;

import static java.util.Collections.singletonList;
import static org.opensearch.rest.RestRequest.Method.GET;

public class RestKNNCacheStatsHandler extends BaseRestHandler {

    private static final String NAME = "knn_cache_stats_action";

    @Override
    public String getName() { return NAME; }

    @Override
    public List<Route> routes() {
        return singletonList(new Route(GET, "/_plugins/_knn/cachestats"));
    }

    @Override
    protected RestChannelConsumer prepareRequest(RestRequest request, NodeClient client) {
        // null nodeIds => all nodes
        KNNCacheStatsRequest knnRequest = new KNNCacheStatsRequest((String[]) null);
        return channel -> client.execute(
            KNNCacheStatsAction.INSTANCE, knnRequest, new NodesResponseRestListener<>(channel));
    }
}

Step 4: Register both in `KNNPlugin`

Two lines. getActions() binds the ActionType to its TransportAction; getRestHandlers() adds your handler to the REST routing table.

// In src/main/java/org/opensearch/knn/plugin/KNNPlugin.java — add to the existing lists.

// getActions(): bind ActionType -> TransportAction
@Override
public List<ActionHandler<? extends ActionRequest, ? extends ActionResponse>> getActions() {
    return Arrays.asList(
        // ... existing handlers (KNNStatsAction, KNNWarmupAction, TrainingModelAction, ...) ...
        new ActionHandler<>(KNNCacheStatsAction.INSTANCE, KNNCacheStatsTransportAction.class)
    );
}

// getRestHandlers(): add the REST endpoint
@Override
public List<RestHandler> getRestHandlers(/* ...existing params... */) {
    return ImmutableList.of(
        // ... existing handlers (RestKNNStatsHandler, RestKNNWarmupHandler, ...) ...
        new RestKNNCacheStatsHandler()
    );
}

# Confirm the exact existing signatures and the import surface before editing — they drift:
grep -n "getActions\|getRestHandlers\|ActionHandler\|RestHandler" \
  src/main/java/org/opensearch/knn/plugin/KNNPlugin.java

Step 5: Build, install, and curl

Build the plugin (this links lab-k1 — the native jni/ step plus the Java compile), then run it.

# Full build (Java + native libs via CMake). See lab-k1 for the native prerequisites.
./gradlew assemble

# Fastest inner loop: run OpenSearch with the plugin already installed.
./gradlew run                  # boots a single node with k-NN + your new action

# In another shell, hit your endpoint:
curl -s 'localhost:9200/_plugins/_knn/cachestats?pretty'

To prove it reflects real native memory, create a faiss index, load it via warmup (see native integration and memory), then re-curl:

curl -XPUT 'localhost:9200/v' -H 'Content-Type: application/json' -d '
{ "settings": { "index.knn": true },
  "mappings": { "properties": { "e": { "type": "knn_vector", "dimension": 4,
    "method": { "name": "hnsw", "engine": "faiss", "space_type": "l2" } } } } }'

for i in 1 2 3 4 5; do
  curl -s -XPOST 'localhost:9200/v/_doc' -H 'Content-Type: application/json' \
    -d "{\"e\":[$i.0,$i.1,$i.2,$i.3]}" >/dev/null
done
curl -s -XPOST 'localhost:9200/v/_refresh' >/dev/null
curl -s -XPOST 'localhost:9200/_plugins/_knn/warmup/v?pretty' >/dev/null

# graph_memory_usage_kb should now be non-zero on the node holding the shard:
curl -s 'localhost:9200/_plugins/_knn/cachestats?pretty'

Step 6: A unit test

Test the serialization round-trip of the per-node response — the part most likely to break silently. k-NN's tests subclass KNNTestCase.

/*
 * SPDX-License-Identifier: Apache-2.0
 * ...license header...
 */
package org.opensearch.knn.plugin.transport;

import org.opensearch.Version;
import org.opensearch.cluster.node.DiscoveryNode;
import org.opensearch.common.io.stream.BytesStreamOutput;
import org.opensearch.core.common.io.stream.StreamInput;
import org.opensearch.core.common.transport.TransportAddress;
import org.opensearch.knn.KNNTestCase;

import java.net.InetAddress;
import java.util.Collections;

public class KNNCacheStatsNodeResponseTests extends KNNTestCase {

    public void testSerializationRoundTrip() throws Exception {
        DiscoveryNode node = new DiscoveryNode(
            "node-1", new TransportAddress(InetAddress.getLoopbackAddress(), 9300),
            Collections.emptyMap(), Collections.emptySet(), Version.CURRENT);

        KNNCacheStatsNodeResponse original =
            new KNNCacheStatsNodeResponse(node, 2048L, 17L, 3L, 1L, 51200L);

        // Write to a buffer, read it back — the wire round-trip.
        BytesStreamOutput out = new BytesStreamOutput();
        original.writeTo(out);
        StreamInput in = out.bytes().streamInput();
        KNNCacheStatsNodeResponse restored = new KNNCacheStatsNodeResponse(in);

        assertEquals(original.getGraphMemoryKb(),  restored.getGraphMemoryKb());
        assertEquals(original.getHitCount(),       restored.getHitCount());
        assertEquals(original.getMissCount(),      restored.getMissCount());
        assertEquals(original.getEvictionCount(),  restored.getEvictionCount());
        assertEquals(original.getCacheCapacityKb(),restored.getCacheCapacityKb());
        assertEquals(node.getId(),                 restored.getNode().getId());
    }
}

./gradlew test --tests "org.opensearch.knn.plugin.transport.KNNCacheStatsNodeResponseTests"

Options B and C, in brief

Option B — a custom SpaceType / scoring function. space_type values map to org.opensearch.knn.index.SpaceType enum constants, each carrying a distance/score function. A custom space (e.g. a weighted L2) means adding an enum constant with its getDistance/scoreTranslation, wiring it through validation, and — for the painless knn_score script — registering it on the script scoring path. Option C — GET /_plugins/_knn/graphinfo/<index> reuses option A's nodes-action skeleton, but nodeOperation reads the k-NN codec's per-segment field info (engine, method, vector count) for the index's local shards instead of the cache.

# Option B: the space/scoring surface.
grep -rn "enum SpaceType\|getDistance\|scoreTranslation\|KNNScoringSpace" \
  src/main/java/org/opensearch/knn/index/SpaceType.java src/main/java/org/opensearch/knn/plugin/script
# Option C: reaching per-segment metadata from a transport action.
grep -rn "KNN990Codec\|KNNCodecVersion\|FieldInfo\|perFieldKnnVectors\|getEngine" \
  src/main/java/org/opensearch/knn/index/codec

Implementation Requirements / Deliverables

One option chosen and implemented end to end with real Java.
*Request, *Response, per-node response, TransportNodesAction, ActionType, and RestHandler all present (option A) — or the equivalent files for B/C.
Registration lines added to KNNPlugin.getActions() and getRestHandlers(), confirmed by grep against your checkout.
Plugin builds (./gradlew assemble) including the native step.
Endpoint reachable: a pasted curl of your route returning a sensible body, with a non-zero metric after warmup (option A).
A passing unit test (serialization round-trip, or the equivalent for B/C).
./gradlew spotlessApply && ./gradlew precommit clean.

Troubleshooting

Symptom	Likely cause	Fix
Node fails to start: "duplicate route" / handler registration error	your route collides with an existing one (e.g. `stats`)	pick a distinct path; grep `_plugins/_knn` routes in `plugin/rest`
`ClassNotFoundException` / `NoSuchMethodError` for your action at startup	registration line wrong, or `ActionType` `NAME` mismatched	re-grep `getActions()`; ensure `ActionType` `NAME` is unique and matches the `super(NAME, ...)` call
Response body garbled or deserialization exception between nodes	`writeTo`/`StreamInput` field order mismatch	make the read order in the `StreamInput` ctor exactly match `writeTo`; see serialization & BWC
`graph_memory_usage_kb` always 0	no faiss graph loaded yet	index docs, refresh, then `POST /_plugins/_knn/warmup/<index>` before re-curling
`cm.getInstance()` NPE in a unit test	`NativeMemoryCacheManager` is a node singleton, not available in a plain unit test	test the request/response serialization (as above), not `nodeOperation`; cover `nodeOperation` in an integ test instead
`prepareRequest` won't compile against `NodesResponseRestListener`	import/signature drift across versions	grep an existing nodes-based `Rest*Handler` (e.g. the stats handler) and copy its listener wiring
Build fails in the native step	missing CMake/toolchain/submodule	see lab-k1 and native integration — `git submodule update --init --recursive`

Expected Output

A fresh node, no vectors loaded:

{
  "nodes" : {
    "kP3mGZ...node-1" : {
      "node_name" : "runTask-0",
      "graph_memory_usage_kb" : 0,
      "cache_capacity_kb" : 16777216,
      "hit_count" : 0,
      "miss_count" : 0,
      "eviction_count" : 0
    }
  }
}

After indexing a faiss field and warming it up, graph_memory_usage_kb and miss_count become non-zero on the node holding the shard, and a second query bumps hit_count — exactly mirroring what GET /_plugins/_knn/stats reports, which is your correctness oracle.

Stretch Goals

Cross-check against the real stats API. Confirm your graph_memory_usage_kb matches the graph_memory_usage field of GET /_plugins/_knn/stats. If they differ, you are reading the cache differently than KNNStats does — reconcile.
Add an integ test. Write a *IT (subclassing the k-NN integration base) that boots a node, indexes a faiss field, warms it, calls your endpoint over REST, and asserts a non-zero metric. This covers nodeOperation, which the unit test cannot.
Make it filterable. Accept ?nodeId=... to scope the fan-out to specific nodes — the BaseNodesRequest already supports node selection; wire it through the handler.
Open a real PR-shaped change. Search https://github.com/opensearch-project/k-NN/issues?q=is%3Aissue+label%3Aenhancement+stats for an actually-wanted stat or endpoint, and propose your feature against a real need — with a CHANGELOG.md entry and DCO sign-off.

Coding Exercises

The walkthrough built one endpoint. These exercises harden it, add a stat and a method parameter, and culminate in a second transport action wired through KNNPlugin — the full feature loop, graded. Confirm every signature with rg against your checkout before editing; the registration methods and listener wiring drift between versions.

(warm-up) Round-trip tests for every wire type you added. You wrote one serialization test for the per-node response. Extend it to a parameterized test that round-trips the per-node response with random values (use randomLong() from the test base), AND a separate test that round-trips the top-level KNNCacheStatsResponse holding a list of two node responses plus a FailedNodeException, asserting the node count, the failure count, and that readNodesFrom/writeNodesTo preserve order. Wire-order bugs are silent at compile time (the lab's #1 warning) — this is the net that catches them.
(warm-up) Assert the route and action registration in a test. Write a unit test that instantiates RestKNNCacheStatsHandler and asserts routes() contains exactly GET /_plugins/_knn/cachestats and getName() is your unique name — then assert it does NOT collide with an existing route (load the existing handlers' routes via rg and compare). A second assertion: that KNNCacheStatsAction.NAME is unique among the actions in transport/ (grep them). This catches the "duplicate route → startup failure" class before you ever boot a node.
(core) Add a new stat to the payload. Extend KNNCacheStatsNodeResponse with one more real metric from NativeMemoryCacheManager — e.g. a load-success/load-time/total-weight figure (rg -n "public (long|double|int) get" src/main/java/org/opensearch/knn/index/memory/NativeMemoryCacheManager.java for what's actually exposed). Add the field to the constructor, writeTo, the StreamInput ctor (correct order!), and toXContent; update nodeOperation. Update your round-trip test to cover the new field. Then cross-check it against GET /_plugins/_knn/stats on a live node (Stretch Goal 1) so your number agrees with the project's own oracle.
(core) Add a method parameter / new SpaceType (Option B path). Implement a custom distance — a weighted L2 is the canonical example. Add the enum constant to org.opensearch.knn.index.SpaceType with its getDistance/scoreTranslation (rg -n "enum SpaceType\|getDistance\|scoreTranslation" src/main/java/org/opensearch/knn/index/SpaceType.java), wire it through the mapper's space validation, and register it on the knn_score script scoring path (rg -rn "KNNScoringSpace\|knn_score" src/main/java/org/opensearch/knn/plugin/script). Write a unit test asserting your distance/score function returns the expected value for a hand-computed pair (do the arithmetic by hand using vector-math foundations), and a YAML/integration test that uses it in a knn_score query. This is a real, mergeable kind of feature — a new metric.
(core) A second transport action: GET /_plugins/_knn/graphinfo/<index> (Option C). Reuse the Option A nodes-action skeleton, but have nodeOperation read per-segment graph metadata (engine, method, vector count) from the k-NN codec's field info for the index's local shards (rg -rn "KNN990Codec\|FieldInfo\|getEngine\|perFieldKnnVectors" src/main/java/org/opensearch/knn/index/codec). Parse the index name from the path in the REST handler and thread it into the request. Deliverables: the five files + two registration lines, a serialization round-trip test, and a curl showing per-segment metadata for a faiss index you built. This proves you can take the triad to a different data source, which is the whole transferable skill.
(advanced challenge) A nodes-action with an integration test that exercises nodeOperation. Your unit tests can't cover nodeOperation (the cache is a node singleton). Write an *IT extending the k-NN integration base that: boots a node, creates a faiss field, indexes + refreshes + warms it, calls your endpoint over REST, and asserts the live metric is non-zero AND matches the corresponding field of GET /_plugins/_knn/stats within tolerance. Then add a node-failure sub-case: target a non-existent node id and assert the response surfaces a FailedNodeException rather than throwing. Finally, scope the fan-out with ?nodeId=... (Stretch Goal 3) and assert only the named node answers. This is the end-to-end proof — fan-out, aggregation, partial failure, and node selection — that turns a toy endpoint into a feature a maintainer would merge. The native-memory numbers it reports are the same ones you'll measure under load in lab-k6 and that the SIMD compute paths in native SIMD and faiss kernels ultimately feed.

Issues to Practice On

New stats, admin endpoints, and diagnostics — the exact triad this lab teaches — are among the most common k-NN feature PRs. The repo is opensearch-project/k-NN.

Goal	Command
Beginner feature/endpoint issues	`gh issue list --repo opensearch-project/k-NN --label "good first issue" --state open`
Stats/API enhancements	`gh issue list --repo opensearch-project/k-NN --label "enhancement" --search "stats OR metric OR endpoint OR api"`
Transport/serialization bugs	`gh issue list --repo opensearch-project/k-NN --label "bug" --search "stats OR transport OR serialization OR nodes"`
Roadmap (new capabilities/APIs)	`gh issue list --repo opensearch-project/k-NN --label "Roadmap"`

Labels drift — list and pick (gh label list --repo opensearch-project/k-NN).

Representative issue patterns. (1) "Expose stat X / add a diagnostic for the native cache" — a clean nodes-action PR. Approach: confirm the metric exists on NativeMemoryCacheManager/KNNStats via rg, add it to the response, register the route, add a round-trip test + an *IT, PR with CHANGELOG + DCO. (2) "New space_type / scoring function" — the Option B shape: a distance enum constant wired through validation and the script path, with a hand-verified arithmetic test. Both are enhancement-labeled and reviewer- friendly because they're additive and serialization-tested.

Planted-bug exercise. In KNNCacheStatsNodeResponse, swap two fields between writeTo and the StreamInput constructor (e.g. write hitCount then missCount but read missCount then hitCount). It compiles fine. Run your Exercise 1 round-trip test and watch it fail with mismatched values — the only thing that catches this class of bug. Now fix the order and add an assertion that distinct values can't be silently swapped (give each field a distinct sentinel). The lesson the lab states outright: read order must mirror write order, and a round-trip test is the one guard that enforces it. See serialization & BWC.

Etiquette. Claim the issue (and confirm the maintainers want the endpoint — propose in the issue first), reproduce/justify the need, and every PR needs a test, a CHANGELOG.md entry, and a DCO sign-off (git commit -s). See community interaction before opening a k-NN feature PR.

Validation / Self-check

Name the five+ classes a TransportNodesAction-based endpoint needs and the job of each. Which two carry the per-node request/response vs the top-level ones?
Where exactly in KNNPlugin does an action get registered, and where does a REST route? What does each registration call return?
Why must the read order in the StreamInput constructor match writeTo? What kind of failure does a mismatch produce, and at what time (compile/startup/runtime)?
Why is native-memory cache stats a nodes action and not a cluster-manager action? What does that say about where the data lives?
Walk a request through your endpoint: REST handler → action → fan-out → nodeOperation → aggregate → XContent. Where does each piece run (coordinating node vs every node)?
Why does graph_memory_usage_kb read 0 until you warm up or query a faiss field? (Tie this to lazy native loading in native integration.)
What does your unit test actually prove, and what does it not cover that an integ test would?

When you can answer all seven and your endpoint returns a non-zero metric after warmup, you have built a real k-NN feature through the full REST/transport machinery. Next, take the same muscles to a real defect in lab-k5: Fix It — a Real k-NN Issue, or revisit k-NN architecture to see how your action sits beside the ten other extension points KNNPlugin registers.

Open-Source Engineer & Contributor