Native Integration, JNI, and Memory

Two of the k-NN plugin's three engines — faiss and the deprecated nmslib — are not Java. They are mature C++ libraries (Facebook's faiss, the nmslib similarity-search library) that the plugin calls across the Java Native Interface (JNI). That single fact drives a large fraction of k-NN's operational behaviour: the graphs live in native memory outside the JVM heap, they are loaded lazily, they are guarded by a separate circuit breaker that the core's heap circuit breakers know nothing about, and when something goes wrong you get C++ symptoms (segfaults, the Linux OOM-killer) rather than Java stack traces.

This chapter is the boundary chapter. It explains how the C++ is built (jni/ + CMake → three shared libraries), how Java reaches it (JNIService/FaissService/NmslibService), where the indexes actually live (native memory, the NativeMemoryCacheManager Guava cache), how warmup and the native-memory circuit breaker work, and how all of that differs from the JVM-heap breaker hierarchy you already know. It assumes you have read k-NN architecture, engines, and the algorithms chapter; the Lucene engine, which has no native code and rides Lucene's own HNSW format, is the deliberate contrast throughout.

After this chapter you can:

• Name the three native shared libraries the jni/ CMake build produces and what each one wraps.
• Trace a call from Java (JNIService) across the JNI boundary into faiss/nmslib and back.
• Explain where faiss/nmslib indexes live at runtime, when they load, and how they are evicted.
• Distinguish the native-memory circuit breaker from the JVM-heap parent breaker, and say which one a given OOM symptom implicates.
• Build the native libraries locally with CMake and diagnose a segfault or OOM-kill.

Note on terminology: the cluster manager (formerly master) is the node that owns cluster state. It is mostly off-stage in this chapter — native memory is a per-data-node, per-shard concern — but it reappears for the warmup/stats coordination at the end.

Why there is a native layer at all

faiss and nmslib are written in C++ for the same reason BLAS is: tight loops over contiguous float arrays with SIMD, no GC pauses, no object headers, no bounds-checks on the hot path. Re-implementing HNSW/IVF/PQ in Java to faiss's level of optimization would be a multi-year effort and would still lose to hand-tuned C++ plus AVX intrinsics. (The Lucene engine does implement HNSW in Java — and gets its SIMD through the Panama Vector API — which is exactly why the Lucene engine is the "no native code, slightly different perf envelope" alternative.)

The cost of that performance is the boundary. Everything below is bookkeeping around the fact that the actual index is a C++ object on the native heap, reachable from Java only through an opaque long pointer and a set of native methods.

Warning: nmslib is deprecated since 2.19; creating new nmslib indices is blocked in 3.0. Existing nmslib indices are still read for backward compatibility. New work targets faiss (and lucene). Everything this chapter says about JNI/native memory applies to both, but write your code and your mental model around faiss. Meta issue tracking the engine direction: k-NN #2605.

The jni/ build: three shared libraries from CMake

The plugin's C++ glue lives in the jni/ directory of the k-NN repo. It is a CMake project, separate from the Gradle build that compiles the Java. Gradle invokes it; CMake compiles the C++ and links against bundled faiss/nmslib to produce three shared libraries.

# In a k-NN checkout, orient yourself in the native tree:
ls jni/                       # CMakeLists.txt, src/, include/, external/ (faiss, nmslib, gtest)
ls jni/src jni/include        # the C++ JNI glue
grep -n "add_library\|target_link_libraries\|opensearchknn" jni/CMakeLists.txt

The split is deliberate: faiss and nmslib are independent third-party codebases with their own headers and their own ABI. Keeping each behind its own .so/.dylib means a faiss-only deployment never has to load nmslib's code, and a build can succeed for one engine even if the other's source is unavailable. The actual file names carry the platform suffix (.so on Linux, .dylib on macOS) and live under the plugin's lib/ directory in the distribution.

# Confirm the library names the build emits (don't trust this doc — grep):
grep -rn "libopensearchknn\|add_library" jni/CMakeLists.txt
# In an installed/built distribution, find the actual artifacts:
find . -name "libopensearchknn_*.so" -o -name "libopensearchknn_*.dylib" 2>/dev/null

Building the native libraries locally

You build the native libraries before (or as part of) building the plugin. The exact flags drift between versions — read jni/README.md and DEVELOPER_GUIDE.md in the repo — but the shape is always: pull the faiss/nmslib submodules, run CMake to configure, then make.

# 1. Pull the vendored native deps (faiss, nmslib, gtest live as submodules/externals).
git submodule update --init --recursive

# 2. Configure + build the native libs with CMake (run from the repo root or jni/).
#    KNN_PLUGIN_VERSION etc. are passed through by Gradle in the normal flow.
cd jni
cmake . -DCMAKE_BUILD_TYPE=Release
make -j$(nproc 2>/dev/null || sysctl -n hw.ncpu)

# 3. The full plugin build (Gradle drives CMake for you and runs the JNI tests):
cd ..
./gradlew build              # compiles Java + invokes the jni/ CMake build + tests
./gradlew :buildJniLib       # (name varies by version) just the native step — grep build.gradle
grep -n "cmake\|jni\|buildJniLib\|CMakeBuild" build.gradle

Note: The native build needs a C++ toolchain (gcc/clang), CMake, and the platform's build tools. A common first-build failure is a missing submodule (jni/external/faiss empty) — git submodule update --init --recursive fixes it. faiss itself may pull in a BLAS/LAPACK dependency; check jni/CMakeLists.txt and the developer guide for the exact prerequisites on your platform.

There is a from-source walkthrough in the lab lab-k1-build-knn-from-source; this chapter is the conceptual map.

The Java↔native boundary: JNIService and the engine services

On the Java side, the boundary is a thin set of classes with native methods. The canonical entry point is JNIService, which dispatches to engine-specific services (FaissService, NmslibService) — grep to confirm the exact names and package in your version, because this layer has been refactored more than once.

# Find the native-method declarations and the service classes (names vary by version):
grep -rln "class JNIService\|class FaissService\|class NmslibService\|native " \
  src/main/java/org/opensearch/knn/jni
grep -rn "public static native\|System.loadLibrary\|loadLibraries\|JNILibraryLoader" \
  src/main/java/org/opensearch/knn/jni

A native method in Java has no body — it is implemented in C++. The signature on the Java side and the function name on the C++ side are linked by the JNI naming convention (Java_<package>_<Class>_<method>), or registered explicitly. A typical declaration looks like:

// org.opensearch.knn.jni.FaissService (illustrative — grep for the real signatures)
public final class FaissService {
    // Load a serialized faiss index from a file into native memory; return a pointer to it.
    public static native long loadIndex(String indexPath);

    // Run a k-NN query against the loaded index; return the top-k as id/score pairs.
    public static native KNNQueryResult[] queryIndex(long indexPointer, float[] query, int k);

    // Free the native index. Java MUST call this or the off-heap memory leaks.
    public static native void free(long indexPointer);
}

The C++ side (in jni/src) implements the matching function. A faiss query implementation is roughly:

// jni/src/faiss_wrapper.cpp (illustrative — grep the real file/signature)
JNIEXPORT jobjectArray JNICALL
Java_org_opensearch_knn_jni_FaissService_queryIndex(
    JNIEnv* env, jclass cls, jlong indexPointer, jfloatArray queryJ, jint k) {

    // Reinterpret the opaque Java long as the real faiss index pointer.
    auto* index = reinterpret_cast<faiss::Index*>(indexPointer);

    // Copy the query out of the JVM (GetFloatArrayElements pins/copies the array).
    jfloat* query = env->GetFloatArrayElements(queryJ, nullptr);

    std::vector<float>   distances(k);
    std::vector<int64_t> labels(k);
    index->search(/*n=*/1, query, k, distances.data(), labels.data());  // <-- the C++ hot path

    env->ReleaseFloatArrayElements(queryJ, query, JNI_ABORT);
    // ... marshal labels+distances back into a Java KNNQueryResult[] (jobjectArray) ...
}

The three things to internalize about this boundary:

1. The index is an opaque long. Java holds a 64-bit pointer to a C++ object it cannot inspect; all lifecycle (load, query, free) crosses by passing it back down.
2. Crossing the boundary is not free. Each call marshals arguments (copying or pinning the float[] query), enters native code, and marshals results back. k-NN amortizes this with real work per call (a whole top-k search), not chatty per-vector calls.
3. Java owns the lifetime, C++ owns the memory. If Java never calls free, the native index leaks — the GC has no idea it exists. This is exactly why the NativeMemoryCacheManager (below) exists: to give those pointers a managed lifecycle.

# The C++ implementations and the result marshalling:
ls jni/src
grep -rn "JNIEXPORT\|reinterpret_cast<faiss\|GetFloatArrayElements\|KNNQueryResult" jni/src | head

Where the index lives: native memory, off the JVM heap

When a faiss field is first queried (or warmed), the plugin reads the serialized faiss index out of its Lucene segment file (k-NN writes the faiss graph as a custom codec format — .faiss/.vec-style files, see architecture and the query path), hands the path to FaissService.loadIndex, and faiss allocates the in-memory graph on the native heap — the process's C/C++ allocator, completely separate from the -Xmx JVM heap.

This has sharp consequences:

• -Xmx does not bound it. You can have a 32 GB heap and still OOM the machine because faiss allocated 100 GB of native memory for HNSW graphs.
• JVM tools don't see it. A heap dump, jmap, GC logs — none show the faiss index. You see it in RSS (resident set size) at the OS level and in the k-NN stats API.
• The core circuit breakers don't account for it. The HierarchyCircuitBreakerService tracks heap; faiss native memory is invisible to it — exactly why k-NN ships its own native-memory breaker.

# Off-heap reality check on a running node: RSS far exceeds -Xmx when faiss indexes load.
curl -s 'localhost:9200/_plugins/_knn/stats?pretty' | grep -E 'graph_memory|cache|hit_count|miss_count|eviction'
ps -o rss= -p <opensearch_pid>   # resident memory includes native faiss graphs; heap is a subset

Loading and eviction: NativeMemoryCacheManager

Native indexes are not loaded at startup. They load lazily on first query for a segment, or eagerly via the warmup API. Once loaded, they are held by the NativeMemoryCacheManager — a Guava Cache keyed by the index/segment file, whose values are NativeMemoryAllocation objects each wrapping one native pointer.

grep -rln "class NativeMemoryCacheManager\|NativeMemoryAllocation\|NativeMemoryEntryContext\|CacheBuilder\|RemovalListener" \
  src/main/java/org/opensearch/knn/index/memory
grep -rn "maximumWeight\|weigher\|invalidate\|getIndexAllocation\|removalListener" \
  src/main/java/org/opensearch/knn/index/memory

The cache is weight-bounded by the native-memory circuit-breaker limit (below), not by entry count. Each entry's "weight" is the index's native size in kilobytes. When loading a new index would push total weight over the limit, Guava evicts the least-recently-used entries; the cache's RemovalListener is where the crucial step happens — it calls the engine's free across JNI so the native memory is actually returned to the OS. Without that listener, eviction would drop the Java wrapper but leak the C++ allocation.

flowchart TD
    Q["k-NN query hits segment S<br/>(or warmup API)"] --> C{"NativeMemoryCacheManager:<br/>S already cached?"}
    C -->|hit| P["reuse cached native pointer"]
    C -->|miss| CB{"would loading S exceed<br/>native CB limit?"}
    CB -->|yes| EV["evict LRU entries:<br/>RemovalListener -> JNI free()"]
    EV --> L
    CB -->|no| L["FaissService.loadIndex(path)<br/>-> native malloc, return long ptr"]
    L --> W["wrap in NativeMemoryAllocation,<br/>put in Guava cache (weight = size KB)"]
    W --> P
    P --> S["JNIService.queryIndex(ptr, vec, k)<br/>-> top-k across the boundary"]

Note: Eviction is cooperative with the OS allocator, not magic. faiss free returns the memory to the C++ allocator, which may or may not immediately return it to the OS (glibc's malloc keeps arenas). RSS can therefore lag eviction. This is a frequent source of "I evicted but RSS didn't drop" confusion. The cache accounting is correct even when RSS is sticky.

The warmup API — preload instead of paying on the first query

The first query against a cold segment pays the full loadIndex cost (read-from-disk + native build/deserialize), which can be hundreds of milliseconds to seconds for a large graph. The warmup API moves that cost out of the query path by loading every segment's native index for an index up front:

# Preload all faiss/nmslib graphs for these indices into native memory (and the cache):
curl -XPOST 'localhost:9200/_plugins/_knn/warmup/products,reviews?pretty'

# Verify they're resident: cache_capacity_reached / graph_memory_usage in stats.
curl -s 'localhost:9200/_plugins/_knn/stats?pretty' | grep -E 'graph_memory|cache_capacity|load'

Warmup is a transport action fanned out to the data nodes holding the shards (KNNWarmupTransportAction/KNNWarmupRequest, names vary — grep to confirm):

grep -rln "Warmup\|warmup" src/main/java/org/opensearch/knn/plugin/transport \
  src/main/java/org/opensearch/knn/plugin/rest

The native-memory circuit breaker — a separate accountant

This is the single most important way k-NN's memory model differs from core's. The core breakers (Circuit Breakers and Memory) guard the JVM heap: parent, fielddata, request, in_flight_requests, accounting. faiss/nmslib native memory is off-heap and invisible to all of them. So k-NN runs its own breaker over the NativeMemoryCacheManager's total weight.

# Inspect / set the native-memory breaker (cluster settings, not index settings):
curl -s 'localhost:9200/_cluster/settings?include_defaults=true&flat_settings=true&pretty' \
  | grep -E 'knn\.memory\.circuit_breaker|knn\.circuit_breaker'

curl -XPUT 'localhost:9200/_cluster/settings' -H 'Content-Type: application/json' -d '
{ "persistent": { "knn.memory.circuit_breaker.limit": "60%" } }'

The behaviour differs from the heap breakers in a subtle, important way:

The mental model: the heap breaker is a hard stop that throws to protect the JVM; the native breaker is more like a cache governor — it bounds resident graph memory by evicting LRU entries, and it refuses to load more once you are over the line. Both exist to stop one greedy actor from killing the node; they just guard different memory regions with different mechanisms.

# Where the native breaker logic lives (grep — names vary by version):
grep -rln "KNNCircuitBreaker\|circuit_breaker.limit\|CircuitBreaker\|isTripped\|KNNSettings" \
  src/main/java/org/opensearch/knn/index/memory \
  src/main/java/org/opensearch/knn/plugin/stats \
  src/main/java/org/opensearch/knn/index/KNNSettings.java 2>/dev/null

Rearchitecture and a real config bug

The native breaker has accumulated known rough edges, and there is an active discussion to rearchitect it — read it before you touch this code:

• Rearchitecture discussion: k-NN #1582 — Investigate rearchitecture of the native memory circuit breaker. The crux: the breaker is entangled with the cache and with cluster-settings plumbing, the "tripped/un-tripped" hysteresis is coarse, and the relationship between eviction and rejection is not as clean as it should be.
• A real config bug: k-NN #585 — a concrete defect in how the circuit-breaker limit setting was handled. Reading it is a good way to see how a settings-plumbing bug manifests in this subsystem and what a fix looks like in review.

When you contribute here, the durable way to find current work is a GitHub search rather than guessing issue numbers:

repo:opensearch-project/k-NN is:issue label:"Memory" circuit breaker
repo:opensearch-project/k-NN is:issue native memory cache eviction

How this differs from the JVM-heap circuit breakers (the full contrast)

You already know the heap breaker hierarchy from Circuit Breakers and Memory. Here is the explicit bridge, because conflating the two is the #1 conceptual error when debugging k-NN memory:

flowchart TB
    subgraph JVM["JVM heap (-Xmx) — core's territory"]
      P["parent breaker (real-memory)"] --> FD[fielddata]
      P --> RQ[request]
      P --> IF[in_flight_requests]
      P --> AC[accounting]
    end
    subgraph NATIVE["Native memory (off-heap) — k-NN's territory"]
      NB["knn.memory.circuit_breaker"] --> CM["NativeMemoryCacheManager (Guava)"]
      CM --> F1["faiss index ptr (segment 1)"]
      CM --> F2["faiss index ptr (segment 2)"]
    end
    OS["OS RSS = heap + native + page cache + ..."]
    JVM -.contributes to.-> OS
    NATIVE -.contributes to.-> OS

• A CircuitBreakingException: [parent] Data too large is a heap event — aggregations, fielddata, big requests. faiss graphs did not cause it.
• A k-NN query that suddenly slows down, or a node whose RSS balloons past -Xmx with no heap pressure, is a native event — look at GET /_plugins/_knn/stats and knn.memory.circuit_breaker.*, not _nodes/stats/breaker.
• The Linux OOM-killer killing the OpenSearch process while the heap looked healthy is the classic native-memory failure: native graphs + heap + page cache exceeded physical RAM, and the kernel reaped the process. Raising -Xmx makes this worse (less RAM left for native). The fix is lowering the native CB limit, reducing graph memory (quantization — see quantization and disk-ANN), or adding RAM.

Warning: A frequent and dangerous misconfiguration: setting -Xmx to ~50% of RAM (the usual heap guidance) and knn.memory.circuit_breaker.limit to a high percentage of RAM. The two budgets overlap on the same physical RAM. Heap + native graphs + OS page cache for mmap'd segments must all fit in physical memory. Budget them together, not independently.

Common bugs and symptoms

Validation: prove you understand this

1. Name the three shared libraries the jni/ CMake build produces, what each wraps, and why faiss and nmslib are kept in separate libraries instead of one.
2. Walk a single queryIndex call across the JNI boundary: what is the opaque long, what gets copied where, who runs the actual top-k search, and who is responsible for freeing the native memory afterward.
3. A node has -Xmx16g, 32 GB of RAM, and gets OOM-killed by the kernel while a heap dump shows the heap at 6 GB. Explain what happened, which budget was the problem, and why raising -Xmx would make it worse.
4. Contrast the core parent heap breaker with knn.memory.circuit_breaker on four axes: what each measures, where its limit comes from, what "tripping" does, and which API exposes its state.
5. Trace a cold k-NN query through NativeMemoryCacheManager: cache miss → CB check → possible eviction → loadIndex → cache put. What does the Guava RemovalListener do on eviction, and why does forgetting it leak native memory?
6. Explain what the warmup API buys you, when you would call it, and how you would verify (via stats) that it worked.
7. Read k-NN #1582 and k-NN #585. Summarize, in two sentences each, what is wrong and what a fix would touch.

When you can do all seven, move to the k-NN query path to see these native indexes actually serve a query end-to-end, and to quantization and disk-ANN for how to make the native graphs small enough that the breaker stops being your enemy. For the pure-Java contrast with no JNI at all, re-read HNSW in Lucene and SIMD and the Vector API.