Lab VI2: Faiss Index Types and JNI

Background

Lab VI1 built an HNSW graph with Lucene — pure Java, inside the JVM. This lab crosses to the other k-NN engine: faiss, a C++ library reached over JNI. You will create OpenSearch k-NN indices with engine: faiss and different method (hnsw vs ivf), run the _train flow that builds IVF centroids and PQ codebooks (the model system index), and then look at the boundary itself — grep JNIService/FaissService on the Java side and the jni/ CMake C++ on the native side. Finally you watch native memory grow, use the warmup API, and compare build/search/memory across IndexHNSWFlat, IndexIVFFlat, and IndexIVFPQ — the faiss index types from the intensive Part 2.

The intensive told you which C++ object sits behind the opaque long pointer for each mapping; this lab makes you create each one and measure the difference.

Why This Matters for Contributors

Most hard k-NN bugs are at this boundary: an UnsatisfiedLinkError because the native lib didn't build, a kernel OOM-kill because faiss allocated off-heap past the native-memory circuit breaker, a training failure because the sample was too small for nlist, or a segfault in hs_err_pid with a faiss frame. To fix any of these you must be able to map an OpenSearch mapping to the faiss index type it produces, run the _train workflow, find the JNI function that crashed, and read native-memory stats. This lab builds that muscle on a single local node.

Prerequisites

• A running OpenSearch with the k-NN plugin (a dev build from an opensearch-project/k-NN checkout, or a distribution that bundles it). curl localhost:9200/_cat/plugins | grep knn.
• A k-NN source checkout to grep the Java and C++ (jni/).
• You've read the intensive Part 2 (faiss index types), the native-memory chapter, and k-NN engines.
• jq for reading JSON responses (optional but convenient).

Note on terminology: training is coordinated by the cluster manager (formerly master) — it creates the model document in .opensearch-knn-models and dispatches the training task. The native indexes themselves are a per-data-node concern. On a single-node dev cluster both roles are the same node, but keep the distinction in mind when you read the algorithms § training sequence diagram.

# Confirm the native libraries actually loaded (the JNI boundary is live):
find $(dirname $(readlink -f $(which opensearch 2>/dev/null) 2>/dev/null) 2>/dev/null)/.. \
  -name 'libopensearchknn_*' 2>/dev/null
# Or in a distribution:
find . -name 'libopensearchknn_faiss.*' -o -name 'libopensearchknn_common.*' 2>/dev/null

Step-by-Step Tasks

Step 1 — Read the JNI boundary, both sides

Before creating indices, see where Java becomes C++. The intensive's class map points here:

cd ~/src/k-NN     # a k-NN checkout
# Java side: the native-method declarations and the service classes (names vary by version).
grep -rln "class JNIService\|class FaissService\|native " src/main/java/org/opensearch/knn/jni
grep -rn "public static native\|loadIndex\|queryIndex\|createIndex\|trainIndex\|free" \
  src/main/java/org/opensearch/knn/jni

# C++ side: the matching JNI functions and the faiss index construction.
ls jni/src jni/include
grep -rn "JNIEXPORT\|reinterpret_cast<faiss\|IndexHNSWFlat\|IndexIVFFlat\|IndexIVFPQ\|index_factory\|index_description" jni/src | head -25

# The CMake build that produces the three shared libraries:
grep -n "add_library\|target_link_libraries\|opensearchknn" jni/CMakeLists.txt

In your notes, answer from the source you just read:

• Which Java native method loads an index, which runs a query, which frees it?
• On the C++ side, what does reinterpret_cast<faiss::Index*>(indexPointer) recover, and who calls index->search(...)?
• How is the faiss index type chosen — is there an index_factory / index-description string assembled from the method/encoder?

This is the boundary the native-memory chapter diagrams; you are now reading the real functions.

Step 2 — IndexHNSWFlat: train-free faiss HNSW

The simplest faiss index — HNSW over float32 vectors, no training. Create the field and index a few vectors:

curl -XPUT 'localhost:9200/vi2-hnsw' -H 'Content-Type: application/json' -d '
{
  "settings": { "index.knn": true },
  "mappings": { "properties": { "vec": {
    "type": "knn_vector", "dimension": 8, "space_type": "l2",
    "method": { "name": "hnsw", "engine": "faiss",
                "parameters": { "m": 16, "ef_construction": 128 } }
  } } }
}'

# Index a handful of vectors (no _train needed — HNSW builds incrementally).
for i in $(seq 1 200); do
  V=$(python3 -c "import random;print(','.join(str(round(random.random(),3)) for _ in range(8)))")
  curl -s -XPOST "localhost:9200/vi2-hnsw/_doc" -H 'Content-Type: application/json' \
    -d "{\"vec\":[$V]}" >/dev/null
done
curl -s -XPOST 'localhost:9200/vi2-hnsw/_refresh' >/dev/null

# Query it.
curl -s -XPOST 'localhost:9200/vi2-hnsw/_search' -H 'Content-Type: application/json' -d '
{ "size": 3, "query": { "knn": { "vec": { "vector": [0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8], "k": 3 } } } }' | jq '.hits.hits[]._score'

That mapping produces a faiss IndexHNSWFlat (intensive §2.2). It is train-free — note you indexed and searched without any _train call.

Step 3 — IndexIVFFlat + IndexIVFPQ: the _train flow

IVF and PQ are not train-free — the centroids and codebooks must be learned first (intensive §2.3–2.5). The workflow: index a representative sample, train a model, then create the real index referencing it by model_id.

# (a) Index a training sample (needs >> nlist points; tens per centroid minimum).
curl -XPUT 'localhost:9200/vi2-train' -H 'Content-Type: application/json' -d '
{ "settings": { "index.knn": true },
  "mappings": { "properties": { "vec": { "type": "knn_vector", "dimension": 8 } } } }'
for i in $(seq 1 4000); do
  V=$(python3 -c "import random;print(','.join(str(round(random.random(),3)) for _ in range(8)))")
  curl -s -XPOST "localhost:9200/vi2-train/_doc" -H 'Content-Type: application/json' \
    -d "{\"vec\":[$V]}" >/dev/null
done
curl -s -XPOST 'localhost:9200/vi2-train/_refresh' >/dev/null

# (b) Train an IVF+PQ model (k-means for nlist centroids AND per-subspace PQ codebooks).
curl -XPOST 'localhost:9200/_plugins/_knn/models/vi2-ivfpq/_train' -H 'Content-Type: application/json' -d '
{
  "training_index": "vi2-train",
  "training_field": "vec",
  "dimension": 8,
  "description": "IVF + PQ over 8-dim synthetic",
  "method": {
    "name": "ivf", "engine": "faiss", "space_type": "l2",
    "parameters": {
      "nlist": 16,
      "encoder": { "name": "pq", "parameters": { "m": 4, "code_size": 8 } }
    }
  }
}'

# (c) Poll until state == created (training runs k-means; coordinated by the cluster manager).
curl -s 'localhost:9200/_plugins/_knn/models/vi2-ivfpq?filter_path=model_id,state' | jq

# (d) Create the real index pointing at the trained model.
curl -XPUT 'localhost:9200/vi2-ivfpq' -H 'Content-Type: application/json' -d '
{ "settings": { "index.knn": true },
  "mappings": { "properties": { "vec": {
    "type": "knn_vector", "model_id": "vi2-ivfpq" } } } }'

Warning: dimension % PQ_m == 0 is required — here 8 % 4 == 0. The PQ m (4 subvectors) is not the HNSW m; they collide by name only (intensive §3.2 and k-NN algorithms § PQ params). Also: nlist k-means needs enough training points — tens per centroid is the faiss rule of thumb. nlist=16 over 4000 points is comfortable; nlist=4096 over 4000 is not and will fail or produce bad cells.

The model document lives in the .opensearch-knn-models system index. Read the training code:

grep -rln "TrainingJob\|ModelDao\|class Model\b\|MODEL_INDEX_NAME\|ModelState" src/main/java/org/opensearch/knn
grep -rn "_train\|TrainingModelRequest\|TrainingModelTransportAction" \
  src/main/java/org/opensearch/knn/plugin/rest src/main/java/org/opensearch/knn/plugin/transport

For an IVF-without-PQ model (IndexIVFFlat), drop the encoder block and re-train under a new model_id — train() then computes only the centroids, and the cell contents stay float32 (intensive §2.3).

Step 4 — Watch native memory grow and use warmup

faiss indexes live in native memory off the JVM heap — invisible to -Xmx, the heap breaker, and heap dumps (the whole point of the native-memory chapter). Observe it:

# Baseline k-NN stats (cache empty before first query/warmup).
curl -s 'localhost:9200/_plugins/_knn/stats?pretty' \
  | jq '.nodes[] | {graph_memory_usage, cache_capacity_reached, hit_count, miss_count, eviction_count, load_success_count}'

# RSS of the process (includes native faiss graphs; heap is a subset).
PID=$(pgrep -f 'org.opensearch.bootstrap.OpenSearch' | head -1); ps -o rss= -p "$PID"

# Cold first query pays loadIndex (native deserialize). Run it, then re-check stats:
curl -s -XPOST 'localhost:9200/vi2-hnsw/_search' -H 'Content-Type: application/json' \
  -d '{ "size":3,"query":{"knn":{"vec":{"vector":[0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8],"k":3}}}}' >/dev/null
curl -s 'localhost:9200/_plugins/_knn/stats?pretty' | jq '.nodes[] | {graph_memory_usage, miss_count, load_success_count}'

# Warmup: preload all segments' native graphs up front (move loadIndex out of the query path).
curl -s -XPOST 'localhost:9200/_plugins/_knn/warmup/vi2-hnsw,vi2-ivfpq?pretty' | jq
curl -s 'localhost:9200/_plugins/_knn/stats?pretty' | jq '.nodes[] | {graph_memory_usage, hit_count, cache_capacity_reached}'

You should see graph_memory_usage and load_success_count rise after the cold query / warmup, and hit_count rise on subsequent queries. RSS climbs by roughly the graph size, independent of -Xmx. The native-memory circuit breaker governs this cache:

# Inspect the native breaker (a cluster setting, NOT the heap breaker):
curl -s 'localhost:9200/_cluster/settings?include_defaults=true&flat_settings=true' \
  | jq -r 'to_entries[]|.value|to_entries[]?|select(.key|test("knn.*circuit_breaker"))|"\(.key)=\(.value)"' 2>/dev/null

Step 5 — Compare the three index types

Build the same vectors under IndexHNSWFlat, IndexIVFFlat, and IndexIVFPQ and compare. After indexing the same sample into each and warming them, record graph memory, a query latency (use "profile": true or wall-clock the curl), and recall against an exact script_score ground truth:

# Exact ground truth for recall: brute-force L2 via script_score (the rescore baseline).
curl -s -XPOST 'localhost:9200/vi2-hnsw/_search' -H 'Content-Type: application/json' -d '
{ "size": 10, "query": { "script_score": {
    "query": { "match_all": {} },
    "script": { "source": "1 / (1 + l2Squared(params.q, doc[\"vec\"]))",
                "params": { "q": [0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8] } } } } }' \
  | jq '[.hits.hits[]._id]'

The intensive's §2.5 table predicts the shape: HNSW the most memory and highest recall; IVFPQ the least memory (PQ codes instead of float32) and the lowest pre-rescore recall. Tie nprobe to recall for the IVF rows the way ef_search ties to recall for HNSW (algorithms § HNSW vs IVF).

Step 6 — Find the C++ that builds each index type

Close the loop: locate where the index-description / index type is actually constructed in C++, so you can see IndexHNSWFlat vs IndexIVFFlat vs IndexIVFPQ chosen from your mapping.

grep -rn "index_factory\|index_description\|IndexHNSW\|IndexIVF\|IndexPQ\|read_index\|write_index" jni/src | head
# The method->faiss string assembly on the Java side:
grep -rn "index_description\|indexDescription\|HNSW\|IVF\|PQ\|methodAsMap\|getIndexDescription" \
  src/main/java/org/opensearch/knn/index/engine 2>/dev/null | head

Deliverables

• Step-1 notes: the load/query/free native methods, the reinterpret_cast target, and how the faiss index type is selected.
• A working IndexHNSWFlat index (Step 2) and a trained IndexIVFPQ index reaching model state created (Step 3).
• Before/after k-NN stats showing graph_memory_usage and load_success_count rising on cold query / warmup (Step 4).
• The Step-5 comparison table filled with your numbers.
• The C++ site (file + line) where the faiss index type is built from the mapping.

Troubleshooting

Expected Output

# k-NN stats after warmup (shape, not exact values):
{ "graph_memory_usage": 3.1, "cache_capacity_reached": false, "hit_count": 12,
  "miss_count": 2, "eviction_count": 0, "load_success_count": 2 }

# Model after training:
{ "model_id": "vi2-ivfpq", "state": "created" }

# Index-type comparison (your numbers): IVFPQ smallest graph_memory, lowest pre-rescore
# recall; HNSW largest memory, highest recall; IVFFlat in between.

Stretch Goals

• Trip the native breaker. Lower knn.memory.circuit_breaker.limit to a tiny value, index/warm several segments, and watch eviction (eviction_count) and the knn.circuit_breaker.triggered stat — the cache-governor behaviour from the native chapter.
• IVF nprobe sweep. For the IVF index, vary nprobe (query param / method_parameters) and tabulate recall vs latency — the IVF analogue of VI1's ef_search sweep.
• Add a rescore pass. Configure rescoring / oversample_factor on the IVFPQ index and show recall recovers toward the float32 ground truth — the production answer to PQ's lossy distance (query path).
• Read a real boundary bug. Read k-NN #585 (a circuit-breaker config defect) and k-NN #1582 (rearchitecture discussion); summarize what each would touch.

Coding Exercises

These make you write across the JNI boundary you just read — Java tests on one side, native C++ probes on the other — instead of only curl-ing the REST API.

1. (warm-up) Assert the index-type-from-mapping in a unit test. Find the Java method that assembles the faiss index-description / type string from the method map (Step 6: grep -rn "getIndexDescription\|indexDescription\|methodAsMap" src/main/java/org/opensearch/knn/index/engine). Write a JUnit/OpenSearchTestCase test that feeds it three method maps — hnsw, ivf (no encoder), ivf+pq — and asserts the resulting description string contains HNSW, IVF/Flat, and IVF+PQ respectively. This pins the mapping→index_factory contract you traced in Step 1 so a refactor can't silently flip an index type.

2. (warm-up) A standalone faiss C++ program that builds all three index types. Outside OpenSearch, write faiss_types.cpp linking the vendored faiss (find jni -path '*faiss*' -name 'IndexHNSW*.h') that constructs IndexHNSWFlat, IndexIVFFlat, and IndexIVFPQ over the same n×d random matrix, adds, trains where required, searches one query, and prints each index's serialized size via write_index to a temp file + stat. You will see IVFPQ's file is smallest — the §2.5 memory ordering, measured in bytes you produced, not stats you read.

3. (core) A native SIMD distance kernel, validated against scalar. Write dot_kernel.cpp implementing dot_scalar and one SIMD variant (dot_avx2 on x86 or dot_neon on Graviton) exactly as in native SIMD: faiss distance kernels, plus a main that asserts they agree within a relative tolerance (never == — FP add is not associative; see that chapter's warning) over 1000 random vectors, and times both to print the SIMD speedup. Then locate faiss's real kernel (grep -rn "fvec_inner_product\|fvec_L2sqr" jni) and compare your structure to it. This is the C++ twin of VI1's Java work and the exact code path a space_type query runs in the hot loop.

4. (core) Integration test: warmup moves loadIndex out of the query path. Write an OpenSearchIntegTestCase/InternalTestCluster test (or a scripted curl harness with hard assertions) that: creates the IndexHNSWFlat index, snapshots _plugins/_knn/stats (miss_count, load_success_count, hit_count), runs the warmup API, and asserts load_success_count rose and a subsequent query's miss_count did not. This is Step 4 turned into a pass/fail gate — the two stats from Self-check Q5, asserted.

5. (advanced) Patch the JNI layer to log the chosen index type, capture it in a test. As a temporary instrumentation patch, add a log line (or a test-only counter) in the C++ JNI createIndex/trainIndex function right where the index-description string is consumed (Step 1/6), printing the description and the resolved faiss class. Build jni/ with CMake, create one index of each type, and grep the OpenSearch log to confirm IndexHNSWFlat vs IndexIVFFlat vs IndexIVFPQ were chosen from your mappings. Capture the three log lines as your deliverable — you have proven the boundary end to end. Revert the patch after.

6. (advanced challenge) An IVF nprobe recall-frontier harness with a rescore pass. Build the IndexIVFPQ index, then write a Java program (using the REST client or InternalTestCluster) that sweeps nprobe ∈ {1,2,4,8,16,nlist} and, against a script_score brute-force ground truth (Step 5), tabulates recall@10 and latency to CSV — the IVF analogue of VI1's ef_search frontier. Then add an oversample_factor/rescore pass and a second CSV column showing recall recover toward float32. Assert in a test that recall is non-decreasing in nprobe and that rescore at nprobe=8 clears your float32 baseline minus a small delta. Feed both CSVs to Lab VI4's frontier plot to overlay PQ-vs-rescored curves. The PQ distance you are sweeping is exactly the fast-scan shuffle described in native SIMD § PQ fast-scan.

Issues to Practice On

The JNI boundary and faiss index types are where the gnarliest, best-scoped k-NN bugs live — native-build, memory, and training failures. The repo is opensearch-project/k-NN.

Labels drift — list and pick (gh label list --repo opensearch-project/k-NN). The anchor issues from this lab's Stretch Goals are #585 (circuit-breaker config) and #1582 (rearchitecture).

Representative issue patterns. (1) "UnsatisfiedLinkError / segfault on first faiss op" — reproduce by exercising a faiss query, confirm the libopensearchknn_faiss.* artifact exists and matches your arch, rg the failing native declaration to its JNI function (grep -rn "JNIEXPORT" jni/src), and check whether the SIMD variant that loaded is the one built (see how to verify which kernel runs). (2) "_train fails / produces bad IVF cells" — reproduce with too-small a sample for nlist, locate TrainingJob/ModelDao (rg "TrainingJob\|ModelState" src/main/java), and fix with a validation message or sample-size guard plus a unit test on the model-state machine.

Planted-bug exercise. In the C++ JNI createIndex path (Step 1/6), force the index-description assembly to always emit the HNSW string regardless of method.name (e.g. hard-code it past the index_factory selection). Rebuild jni/, then run Exercise 1's mapping→type test: it goes red for the ivf and ivf+pq cases because the description no longer matches the mapping. Note that no curl smoke test would have caught this — an HNSW index still answers queries; only the type assertion fails. Restore the selection, then keep Exercise 1 as the regression test that would have caught it.

Etiquette. Claim an issue before working it, reproduce first (a native OOM is a kernel OOM-kill, not a Java exception — watch _plugins/_knn/stats, not _nodes/stats/breaker), and every PR needs a test, a CHANGELOG.md entry, and a DCO sign-off (git commit -s). See community interaction.

Validation / Self-check

1. Trace one knn query from the REST layer to index->search(...) in C++: name the Java native method, what the opaque long is, what gets copied across, and who frees the native memory.
2. Why is IndexHNSWFlat train-free while IndexIVFFlat and IndexIVFPQ are not? State exactly what train() computes for each.
3. From your Step-5 table: which index used the least native memory, and why (what is stored per vector)? Which had the lowest pre-rescore recall, and what fixes it?
4. Explain why a faiss OOM is a kernel OOM-kill, not a Java CircuitBreakingException, and which API (not _nodes/stats/breaker) you watch to see it coming.
5. What does warmup buy you, and which two stats prove it worked?
6. Map nlist/nprobe/PQ-m/code_size to what each controls and at what time (train vs query) each is fixed.

When this holds, go to Lab VI3: The Quantization Math to implement the SQ/PQ/BQ math the IVFPQ index used, and re-read the native integration chapter and k-NN engines for the runtime and engine-choice context. For the pure-Java contrast with no JNI, return to Lab VI1.