Lab K1: Build the k-NN Plugin from Source

Background

Every other lab in this section assumes you can build and run the k-NN plugin. This lab earns you that. The k-NN repo (opensearch-project/k-NN) is the most structurally unusual project in the OpenSearch world: it has two build systems that must agree — a Gradle build that compiles the Java (src/main/java/org/opensearch/knn/...) and a CMake build under jni/ that compiles C++ glue and links the bundled faiss and nmslib libraries into shared objects (libopensearchknn_faiss, libopensearchknn_nmslib, libopensearchknn_common, and more). Gradle drives CMake; if either side is misconfigured, you get cryptic failures that look nothing like a normal Java compile error.

On top of that, k-NN is an out-of-repo plugin: it does not live inside the OpenSearch core tree, so it builds against published core artifacts. Its build.gradle pins an opensearch_version (e.g. 3.6.0-SNAPSHOT), and if no matching artifact exists in your local Maven cache, you must publish core yourself first.

In this lab you clone the repo, initialize the native submodules, build the JNI libraries with CMake, build the plugin with Gradle, run its unit and integration tests, and install the result into a local OpenSearch distribution. By the end you will have a node running with a plugin you compiled — native code and all.

Note: The cluster manager (formerly master) is the node that owns cluster state. It is mostly off-stage in this lab — building and installing a plugin is a per-node concern — but it reappears the moment you start a multi-node cluster, because every node must run the same plugin version or the cluster will not form.

Read k-NN architecture and native integration and memory alongside this lab; they are the conceptual map for the two trees you are about to build. The core analogue is Lab 1.1: Build OpenSearch from Source — do that first if you have not; the JDK/toolchain/Gradle-wrapper mechanics carry over.

Why This Lab Matters for Contributors

• You cannot fix a k-NN issue, run its tests, or reproduce a bug report without a working from-source build. This is the gate to every other contribution.
• The dual Gradle+CMake build is the single biggest source of "I can't even get it to compile" pain for new k-NN contributors. Understanding which build owns which failure is half the battle.
• The out-of-repo-plugin model (build against published core artifacts, publishToMavenLocal when there is no published snapshot) is how every OpenSearch plugin outside the core repo is built — k-NN, SQL, security, anomaly-detection. Learn it once here.
• Installing your own build into a distribution and seeing it in _cat/plugins is the end-to-end loop you will run hundreds of times.

Prerequisites

• A clean OpenSearch core build (you have completed Lab 1.1 and ./gradlew assemble works in the core repo).
• JDK 21 (the k-NN build, like core, targets JDK 21). The repo's bundled JDK is fine.
• A C++ toolchain: gcc/g++ or clang/clang++, plus make.
• CMake ≥ 3.24 (the jni/CMakeLists.txt declares cmake_minimum_required(VERSION 3.24.0) — confirm with cmake --version).
• git with submodule support, and enough disk (the faiss/nmslib submodules plus their build artifacts are several GB).
• On macOS: libomp (OpenMP) for faiss; on Linux: a BLAS/LAPACK implementation may be pulled in by faiss. Check jni/CMakeLists.txt and the repo's DEVELOPER_GUIDE.md for the exact platform prerequisites — they drift between versions.

# Verify the toolchain before you start. Any missing tool here = a build failure later.
java -version          # want 21.x
cmake --version        # want >= 3.24
g++ --version || clang++ --version
git --version

Step-by-Step Tasks

Step 1: Clone the repo WITH its submodules

This is the number-one "k-NN won't build" cause, so get it right first. faiss and nmslib are git submodules under jni/external/. A plain git clone leaves them empty, the Java side builds fine, and then the CMake step dies with errors about missing faiss headers.

mkdir -p ~/src/oss-repos && cd ~/src/oss-repos

# Clone AND initialize submodules in one shot:
git clone --recursive https://github.com/opensearch-project/k-NN.git
cd k-NN

# If you already cloned without --recursive, fix it now:
git submodule update --init --recursive

# Confirm the externals are actually populated (not empty directories):
ls jni/external/                 # expect: faiss  nmslib  (plus possibly gtest)
ls jni/external/faiss | head     # expect faiss's own source tree, NOT an empty dir

Warning: jni/external/faiss being an empty directory is the signature of the missing-submodule failure. The CMake config step also tries to self-heal — the jni/cmake/init-faiss.cmake script runs git submodule update --init -- external/faiss if it can't find faiss — but do not rely on that; initialize submodules explicitly.

Step 2: Read the version contract

k-NN builds against a specific OpenSearch core version. Find it before anything else; a version mismatch here is the second-most-common build failure.

# The pinned core version (e.g. "3.6.0-SNAPSHOT"):
grep -n "opensearch_version" build.gradle | head
#   opensearch_version = System.getProperty("opensearch.version", "3.6.0-SNAPSHOT")

# The build pulls build-tools and core from this version's published artifacts:
grep -n "build-tools\|opensearch_group\|repositories" build.gradle | head

The rule: the plugin's opensearch_version must match a core artifact that exists in your Maven repositories. For a released version, the public Maven repo has it. For a -SNAPSHOT, you usually need to publish core locally (Step 3).

Step 3: Publish core to Maven local (only if needed)

If opensearch_version is a -SNAPSHOT that isn't on the OpenSearch snapshot Maven repo, build it from the core checkout that matches and publish it to your local ~/.m2:

# In your OpenSearch CORE checkout (the one from Lab 1.1), on the matching version/branch:
cd ~/src/oss-repos/OpenSearch
./gradlew publishToMavenLocal -Dbuild.snapshot=true

# This installs org.opensearch:opensearch:<version>, build-tools, etc. into ~/.m2/repository.
ls ~/.m2/repository/org/opensearch/opensearch/ 2>/dev/null

Then point the k-NN build at the local artifacts. The build already adds mavenLocal() in most versions; if you need to override the version explicitly:

cd ~/src/oss-repos/k-NN
./gradlew assemble -Dopensearch.version=3.6.0-SNAPSHOT

Note: You only need this if the plugin can't resolve core. Try the build first (Step 5); if Gradle fails with Could not resolve org.opensearch:opensearch:<version>, come back and publish core to Maven local. Releases-against-released-core don't need this.

Step 4: Build the native JNI libraries with CMake

The native build is a CMake project under jni/. In the normal flow Gradle drives CMake for you (Step 5), but build it standalone once so you understand what Gradle is doing and can diagnose native failures in isolation.

The Gradle task cmakeJniLib runs the configure step and buildJniLib runs the compile step. Read exactly what they invoke:

grep -n "cmakeJniLib\|buildJniLib\|KNN_PLUGIN_VERSION\|jni/build\|--target" build.gradle | head

You will see the configure command is essentially:

# Configure: -S = source dir, -B = build dir. KNN_PLUGIN_VERSION must match opensearch_version.
cmake -S jni -B jni/build \
  -DKNN_PLUGIN_VERSION=3.6.0-SNAPSHOT \
  -DCMAKE_POLICY_VERSION_MINIMUM=3.5

...and the build command targets the named libraries:

# Compile only the native libs, in parallel:
cmake --build jni/build \
  --target opensearchknn_faiss opensearchknn_common opensearchknn_nmslib opensearchknn_simd \
  --parallel "$(nproc 2>/dev/null || sysctl -n hw.ncpu)"

Or, far simpler, let Gradle do it:

./gradlew buildJniLib       # runs cmakeJniLib (configure) then the native compile

When it finishes, find the artifacts. The shared-library suffix is platform-specific (.so on Linux, .dylib on macOS, .dll on Windows):

find jni/build -name "libopensearchknn_*.so"  -o -name "libopensearchknn_*.dylib" 2>/dev/null
#   .../libopensearchknn_faiss.{so,dylib}
#   .../libopensearchknn_common.{so,dylib}
#   .../libopensearchknn_nmslib.{so,dylib}
#   .../libopensearchknn_util.{so,dylib}   (a shared utility lib the others link)
#   .../libopensearchknn_simd.{so,dylib}   (SIMD compute helpers)

Note: The verified-facts shorthand for this build is "three libraries" (faiss/nmslib/common). The real CMake build emits more (util, simd), and faiss/nmslib are gated by AVX2_ENABLED/AVX512_ENABLED/feature flags. Always grep "set(TARGET_LIB" jni/CMakeLists.txt to see the current target list rather than trusting any doc — including this one.

Step 5: Build the whole plugin

Now the full build — Gradle compiles the Java, invokes the CMake native build, and packages the installable zip:

./gradlew assemble          # Java + native + the plugin zip (no tests)
# OR the fuller build that also runs checks:
./gradlew build             # assemble + precommit + tests

# Find the installable plugin zip:
find build/distributions -name "opensearch-knn-*.zip"
#   build/distributions/opensearch-knn-3.6.0-SNAPSHOT.zip

The fastest inner loop — a node with the plugin already loaded, no manual install:

./gradlew run               # builds + starts a single node with k-NN on the classpath
# In another terminal:
curl -s 'localhost:9200/_cat/plugins?v'
#   name   component  version
#   ...    opensearch-knn  3.6.0-SNAPSHOT

Step 6: Run the unit tests

# All Java unit tests:
./gradlew test

# A focused run (much faster while iterating):
./gradlew test --tests "org.opensearch.knn.index.mapper.KNNVectorFieldMapperTests"
./gradlew test --tests "*KNNQueryBuilderTests"

# Reproduce a flaky/seeded failure exactly (OpenSearch/Lucene test framework uses seeds):
./gradlew test --tests "*KNNQueryTests" -Dtests.seed=DEADBEEF

There are also native (C++) tests built and run via a separate task — the JNI glue has its own gtest suite:

grep -n "buildJniTest\|jniTest\|gtest" build.gradle | head
./gradlew buildJniTest      # builds the native libs, then runs the C++ JNI tests

Step 7: Run the integration tests

Integration tests spin up a real testClusters node with the plugin installed and exercise it over REST. They depend on buildJniLib, so the native libraries must build first.

# The main integ suite (starts a real node):
./gradlew integTest

# k-NN also has multi-node and remote-index-build integ variants — grep to see them all:
grep -n "RestIntegTestTask\|integTest\b\|integTestMultiNode\|integTestRemoteIndexBuild" build.gradle | head

# Run just one integ test class while iterating:
./gradlew integTest --tests "*KNNRestTestCase*" 2>/dev/null || \
  ./gradlew integTest --tests "org.opensearch.knn.*IT"

Note: Integration tests are slow (they boot a JVM cluster) and need the native libs. If integTest fails immediately with an UnsatisfiedLinkError, your buildJniLib step didn't produce the libraries for your platform — fix Step 4 first.

Step 8: Install the built plugin into a local distribution

./gradlew run is the dev loop, but to mimic a real deployment, install your zip into an actual OpenSearch distribution with bin/opensearch-plugin:

# 1. Produce (or locate) an OpenSearch distribution of the MATCHING version.
#    From your core checkout:
cd ~/src/oss-repos/OpenSearch
./gradlew localDistro
DISTRO=$(find distribution/archives -maxdepth 2 -name "opensearch-*" -type d | head -1)
echo "Distribution at: $DISTRO"

# 2. Install the k-NN zip you built (use a file:// URL to a local zip):
ZIP=$(find ~/src/oss-repos/k-NN/build/distributions -name "opensearch-knn-*.zip" | head -1)
"$DISTRO/bin/opensearch-plugin" install "file://$ZIP"
#   -> "Installing file://.../opensearch-knn-....zip"
#   -> "Continue with installation? [y/N]"  (the plugin has native code; answer y)

# 3. Start the node and confirm the plugin loaded:
"$DISTRO/bin/opensearch" &
sleep 20
curl -s 'localhost:9200/_cat/plugins?v'
#   name   component       version
#   node-0 opensearch-knn  3.6.0-SNAPSHOT

Step 9: Smoke-test the install end to end

Prove the native path works, not just that the plugin loaded:

# Create a tiny k-NN index, index two vectors, query, and warm up.
curl -s -XPUT 'localhost:9200/k1demo?pretty' -H 'Content-Type: application/json' -d '{
  "settings": { "index.knn": true },
  "mappings": { "properties": {
    "v": { "type": "knn_vector", "dimension": 2,
           "method": { "name": "hnsw", "engine": "faiss", "space_type": "l2" } }
  }}
}'

curl -s -XPOST 'localhost:9200/k1demo/_bulk?refresh=true' -H 'Content-Type: application/x-ndjson' -d '
{ "index": { "_id": "1" } }
{ "v": [1.0, 1.0] }
{ "index": { "_id": "2" } }
{ "v": [9.0, 9.0] }
'

# Load the faiss graph into native memory, then query it:
curl -s -XPOST 'localhost:9200/_plugins/_knn/warmup/k1demo?pretty'
curl -s -XPOST 'localhost:9200/k1demo/_search?pretty' -H 'Content-Type: application/json' -d '{
  "size": 1,
  "query": { "knn": { "v": { "vector": [1.1, 0.9], "k": 1 } } }
}'
# Expect _id "1" as the nearest neighbor.

# Confirm native memory accounting works (faiss graph is OFF-heap):
curl -s 'localhost:9200/_plugins/_knn/stats?pretty' | grep -E 'graph_memory|hit_count|miss_count|load'

Implementation Requirements / Deliverables

• A k-NN clone with jni/external/faiss and jni/external/nmslib populated (submodules initialized).
• The native libraries built: libopensearchknn_faiss, libopensearchknn_common, libopensearchknn_nmslib (plus util/simd) found under jni/build.
• ./gradlew assemble produces build/distributions/opensearch-knn-<version>.zip.
• ./gradlew test passes (or you can name and explain any pre-existing flaky failure).
• ./gradlew integTest runs a real node and passes its suite.
• The plugin installed into a localDistro via bin/opensearch-plugin install, listed in _cat/plugins.
• A working faiss query against a 2-D index, with _plugins/_knn/stats showing non-zero graph_memory_usage after warmup.
• You can state which build (Gradle vs CMake) owns a given failure mode.

Troubleshooting

Rule of thumb for which build owns a failure: a stack trace with org.opensearch.knn.* Java frames and a Gradle task name → the Gradle/Java side. A message mentioning cmake, make, faiss/nmslib headers, .so/.dylib, OpenMP, BLAS, or UnsatisfiedLinkError → the CMake/native side. Diagnose them with different tools.

Expected Output

# After ./gradlew buildJniLib
> Task :cmakeJniLib
> Task :buildJniLib
BUILD SUCCESSFUL

# Native artifacts (macOS example)
jni/build/release/libopensearchknn_faiss.dylib
jni/build/release/libopensearchknn_common.dylib
jni/build/release/libopensearchknn_nmslib.dylib

# After ./gradlew assemble
build/distributions/opensearch-knn-3.6.0-SNAPSHOT.zip

# _cat/plugins on the node with your build installed
name   component       version
node-0 opensearch-knn  3.6.0-SNAPSHOT

# Smoke query result
"hits": [ { "_id": "1", "_score": 0.83..., "_source": { "v": [1.0, 1.0] } } ]

# _plugins/_knn/stats after warmup (off-heap memory is non-zero)
"graph_memory_usage": 1,
"hit_count": 1,
"miss_count": 1

Stretch Goals

1. Build a single engine. The build supports -Pknn_libs=... to limit the native targets. Build only faiss (skip nmslib) and confirm the nmslib .so is absent. grep -n "knn_libs" build.gradle to find the property.
2. Toggle SIMD. Re-run cmakeJniLib with -Davx2.enabled=false (or the property the build reads — grep for avx2_enabled) and observe the configure flags change. Note that PlatformUtils.isAVX2SupportedBySystem() is what selects the runtime path.
3. Inspect the JNI glue. ls jni/src and read faiss_wrapper.cpp's query function. Match its Java_org_opensearch_knn_jni_FaissService_* name to the native method declared in src/main/java/org/opensearch/knn/jni/FaissService.java.
4. Run the C++ tests directly. After buildJniTest, find the gtest binary under jni/build and run it standalone to see the native test output without Gradle in the way.
5. Diff the published vs local artifact. After publishToMavenLocal, compare the core jar your plugin resolved against the one in the distribution you installed into — confirm they are the same version, which is why the plugin loads.
6. Time the build. Run ./gradlew assemble --profile and read the generated report. Notice how much of the wall-clock is the CMake/native step versus Java compilation.

Validation / Self-check

1. Name the two build systems k-NN uses, what each is responsible for, and how Gradle and CMake relate (which one invokes the other, and via which tasks).
2. Why does a fresh git clone (without --recursive) build the Java but fail the native step? Which directory tells you submodules are missing?
3. What does opensearch_version in build.gradle control, and when do you need publishToMavenLocal in the core repo before the plugin will build?
4. List the shared libraries the CMake build emits and what each wraps. Why are faiss and nmslib kept in separate libraries rather than one?
5. Given a failure message, decide whether the Gradle/Java build or the CMake/native build owns it. Give one concrete example of each.
6. What is the difference between ./gradlew run and installing the zip with bin/opensearch-plugin install? When would you use each?
7. After warming up a faiss index, which API proves the native graph is resident in off-heap memory, and which field do you read?

When you can build both trees, run all the test tiers, install your own zip, and serve a faiss query, you are ready to read code that actually runs. Continue to Lab K2: Trace a k-NN Query, and keep k-NN architecture and native integration and memory open as your map.