Engines: Faiss, Lucene, and NMSLIB

The architecture chapter ended on one load-bearing sentence: faiss/nmslib write native graphs through k-NN's codec into off-heap memory; lucene writes Lucene graphs through Lucene's codec onto the JVM heap. That fork is the engine mapping parameter. It is the single most consequential choice you make when you map a knn_vector field, and it decides almost everything downstream: which algorithms you can use, whether vectors live on or off the heap, whether filtering is exact or post-hoc, whether you can quantize, whether you need to train, and which segment files end up on disk.

This chapter is the comparison. It does not re-derive the plugin wiring (that is architecture) or the algorithms themselves (that is algorithms — HNSW, IVF, PQ). It answers a narrower, sharper question: given three engines that all do "approximate nearest neighbor," why are there three, what does each actually give you, and how do you choose. By the end you can fill in the capability matrix from memory, justify a default, and read an engine-related issue without guessing.

Note: "engine" here is k-NN's word, set per field as method.engine in the knn_vector mapping. It is unrelated to OpenSearch's Engine/InternalEngine (the shard write abstraction in engine internals). Same word, different layer. When this chapter says "engine" it means faiss / lucene / nmslib.

After this chapter you can:

• Name the three engines, what each is, and the one-line reason each exists.
• Fill in the capability matrix (algorithms, quantization, filtering, memory location, training, byte/binary vectors) without looking.
• Choose an engine for a given workload and defend the choice.
• Explain how the engine value flows through the custom codec into segment files.
• Read the "new vector engine" meta-issue and place today's engines on its roadmap.

Why three engines at all

A reasonable question on your first day: if all three implement HNSW, why does k-NN ship three of them? The answer is history plus a genuine three-way trade.

• nmslib came first. It is a mature C++ ANN library (Non-Metric Space Library) that does HNSW and nothing else here. It was k-NN's original engine and proved the whole idea: load a graph off-heap, search it via JNI, return doc IDs. It has now served its purpose and is deprecated — more below.
• faiss is Meta's vector library. It does HNSW and IVF, and — crucially — it is where the memory story lives: Product Quantization, scalar/FP16 quantization, binary quantization, and the disk-based ANN modes all run through faiss. It is also where the future is being built (GPU and remote index build). It is the default.
• lucene is not an external library at all. It is Lucene's own native HNSW vector format, the same one described in HNSW in Lucene. Choosing it means k-NN gets out of the native-memory business entirely for that field: vectors live on the heap, in Lucene's own segment files, searched by Lucene's own code, with exact pre-filtering that the native engines cannot match.

So the three exist because they sit at different points of one triangle:

flowchart TD
    F["faiss<br/>algorithms + quantization + future (GPU/remote)<br/>off-heap, native JNI"]
    L["lucene<br/>exact filtering + heap-managed + no JNI<br/>on-heap, Lucene format"]
    N["nmslib<br/>HNSW only, deprecated<br/>off-heap, native JNI"]
    F --- L
    L --- N
    N --- F

• faiss optimizes for capability and scale (every algorithm, every quantization, GPU on the horizon) at the cost of off-heap complexity.
• lucene optimizes for operational simplicity and filtering quality at the cost of features (no IVF, no PQ, on-heap memory pressure).
• nmslib optimized for being first and simple — and is now a legacy read path.

faiss — the default

The faiss engine builds a faiss index (HNSW or IVF) per segment, in C++, via JNI, and loads it off the JVM heap into native memory. Everything in the native integration and memory chapter — the NativeMemoryCacheManager, the warmup API, the off-heap circuit breaker — is primarily about faiss.

What faiss gives you that the others do not:

• Two algorithms. HNSW (graph, train-free) and IVF (inverted-file / cluster-based, needs training). See algorithms for the mechanics.
• The whole quantization menu. Product Quantization (PQ) for IVF and HNSW, scalar/FP16 quantization, Binary Quantization (BQ), and disk-based vector search (on_disk mode with compression_level 1x/2x/4x/8x/16x/32x and rescoring on full-precision vectors). This is the entire quantization and disk-ANN chapter, and it is almost entirely a faiss story.
• byte and binary vectors (data_type: byte from 2.17+; binary via the hamming space).
• SIMD on the C++ side. faiss picks an AVX2/AVX-512 path at runtime (PlatformUtils probes the CPU — the same vectorization theme as SIMD and the Vector API, but in C++, not the JDK Panama API).

cd ~/src/oss-repos/k-NN
# The faiss engine definition and its declared methods/capabilities.
ls src/main/java/org/opensearch/knn/index/engine/faiss/
grep -rn "class Faiss\|HNSW\|IVF\|encoder\|pq\|sq" \
  src/main/java/org/opensearch/knn/index/engine/faiss/ | head -20

# The C++ side it bridges to.
ls jni/src/                 # faiss_wrapper.cpp ...
grep -rn "native " src/main/java/org/opensearch/knn/jni/FaissService.java | head

Where faiss is going: GPU and remote build

The two most exciting live RFCs in the whole plugin are both about moving faiss index construction off the indexing node. Building an HNSW or IVF graph is the expensive part of indexing vectors; the search is comparatively cheap. So:

• GPU build. [RFC] Boosting OpenSearch Vector Engine Performance using GPUs — k-NN #2293 proposes building vector indexes on NVIDIA GPUs via cuVS and the CAGRA graph algorithm (FP32), then serving them. Graph construction is embarrassingly parallel; a GPU eats it.
• Remote build. [RFC] Remote Vector Index Build — k-NN #2294 proposes offloading segment vector-index construction to a separate remote fleet (CPU or GPU) instead of doing it inline on the data node during flush/merge. The data node ships the raw vectors out, gets a built graph back, and writes it as a segment file.

Both are faiss-engine features. They are also a clean example of an architecture you have already met: separating a write-heavy phase from the node serving reads — the same instinct behind reader/writer separation in core. Read both issues; they are the current frontier of this plugin and exactly the kind of work the capstone projects point at.

lucene — the heap-managed, filter-friendly engine

The lucene engine is the architectural odd one out, and the reason is worth saying precisely: it does not use k-NN's native codec or JNI at all. When you map a field with engine: lucene, k-NN steps aside and lets Lucene store the vectors with its own KnnFloatVectorField/KnnByteVectorField and HNSW format (Lucene99HnswVectorsFormat and friends), exactly as in HNSW in Lucene. The graph lives in Lucene's .vec/.vex/.vem segment files, on the JVM heap (memory-mapped, managed by Lucene), not in k-NN's off-heap native cache.

That single fact produces every difference that matters:

# The lucene engine and how it hands off to Lucene's own format.
ls src/main/java/org/opensearch/knn/index/engine/lucene/
grep -rn "Lucene\|KnnFloatVectorField\|HnswVectorsFormat\|ScalarQuantized" \
  src/main/java/org/opensearch/knn/index/engine/lucene/ | head -20

The headline reason to pick lucene is filtering. With faiss, a filtered k-NN query has historically meant a trade-off: filter-then-search can starve the graph of candidates; search-then-filter can return fewer than k. Lucene's HNSW applies the filter during traversal, so a filtered query is both exact and efficient. If your workload is "find the nearest vectors among documents matching this filter" and your vectors fit comfortably in heap, lucene is often the right answer even though faiss is the default.

nmslib — deprecated, read-only future

nmslib is the original engine and now the cautionary tale. It does HNSW only, off-heap, via JNI — the same shape as faiss but with none of faiss's algorithms, quantization, or roadmap.

Its status is the important part:

• Deprecated since 2.19. Mapping a new field with engine: nmslib emits a deprecation warning.
• Blocked for new indices in 3.0. You cannot create a new nmslib index in 3.0.
• Still read for backward compatibility. Existing nmslib segments from older indices are still loaded and queried, through the codec's backward_codecs/, so an upgraded cluster keeps serving them.

# The deprecation / block lives in the engine + mapper validation. Grep to find it.
grep -rn "nmslib\|NMSLIB\|deprecat\|isCreate\|3.0\|blocked" \
  src/main/java/org/opensearch/knn/index/engine/ | grep -i nmslib | head
# Backward codecs keep reading old nmslib/faiss segments after upgrade.
ls src/main/java/org/opensearch/knn/index/codec/backward_codecs/

Warning: "deprecated" and "blocked for new indices" are not "removed." If you touch nmslib code, you are almost always working on the read/BWC path — keep existing segments readable forever (or until an explicit major-version drop). A PR that breaks reading an old nmslib segment is a backward-compatibility regression, which is the compatibility failure mode the maintainers most reliably reject. Do not "clean up" nmslib by deleting its read path.

The practical rule: never reach for nmslib on a new index. If you find yourself wanting HNSW-off-heap, that is faiss. nmslib exists today only so that yesterday's indices keep working.

The capability matrix

Memorize this table. It is the chapter. When an issue or a user asks "can engine X do Y," this is the lookup.

Note: exact capability boundaries shift release to release (byte-vector support, which spaces a quantizer allows, radial support). The matrix above is the shape; always confirm the boundary in your checkout by grepping the engine's declared methods and the mapper's validation, rather than trusting a table — including this one. The validation lives in the field mapper and the per-engine method definitions.

# Confirm a capability boundary instead of trusting the table.
grep -rn "supportedDataTypes\|validateMethod\|isFilterSupported\|trainingRequired\|SpaceType" \
  src/main/java/org/opensearch/knn/index/engine/ | head -20
# The mapper rejects illegal combinations (e.g. IVF on the lucene engine) here.
grep -rn "validate\|UnsupportedOperation\|not supported" \
  src/main/java/org/opensearch/knn/index/mapper/KNNVectorFieldMapper.java | head

When to choose which

A decision procedure, not a religion. Walk it top to bottom; take the first match.

The two-line summary to carry around:

• Filtering + heap-friendly → lucene.
• Everything else, especially scale and quantization → faiss.
• nmslib → never new; legacy reads only.

flowchart TD
    Start["mapping a knn_vector field"] --> Q1{"need to filter, and<br/>vectors fit in heap?"}
    Q1 -->|yes| LUC["engine: lucene<br/>(exact pre-filter, on-heap)"]
    Q1 -->|no| Q2{"need IVF / PQ / BQ /<br/>disk-ANN / binary / GPU?"}
    Q2 -->|yes| FAI["engine: faiss"]
    Q2 -->|no| Q3{"brand-new index?"}
    Q3 -->|yes| FAI2["engine: faiss (the default)"]
    Q3 -->|reading old index| NMS["nmslib (BWC read only)"]

How engine choice flows through the codec to segment files

This is where the abstraction becomes files on disk, and it is the part new contributors most often hold only fuzzily. The engine value chosen in the mapping decides, at flush and merge time, which writer runs and which files appear.

flowchart TD
    MAP["knn_vector mapping<br/>method.engine = faiss | lucene | nmslib"] --> FT["KNNVectorFieldType<br/>(carries the KNNEngine)"]
    FT --> FLUSH["flush / merge<br/>(segment becomes immutable)"]
    FLUSH --> SEL{"which KnnVectorsFormat<br/>does the codec pick?"}
    SEL -->|faiss / nmslib| NEF["NativeEngines990KnnVectorsFormat<br/>(k-NN's KNN10xxCodec)"]
    SEL -->|lucene| LF["Lucene's Lucene9xHnswVectorsFormat<br/>(Lucene's default codec path)"]
    NEF -->|JNIService -> FaissService/NmslibService| NATIVE["build native graph in C++"]
    NATIVE --> NFILES["k-NN native graph files<br/>written as EXTRA segment files,<br/>loaded OFF-HEAP at query time"]
    LF -->|Lucene writes directly| LFILES[".vec / .vex / .vem<br/>(Lucene's own format, ON-HEAP)"]

Trace it as four facts:

1. The mapper records the chosen KNNEngine on the KNNVectorFieldType. This is the only place the decision is made; everything downstream just reads it.
2. At flush/merge (segments are immutable — see refresh/flush/merge), the custom codec consults the engine:
   • faiss/nmslib → NativeEngines990KnnVectorsFormat, which calls JNIService → FaissService/NmslibService to build the whole graph in C++ and write it as extra segment files beside Lucene's normal .cfs/.fdt/ .tim. These load off-heap at query time.
   • lucene → no JNI; Lucene's own KnnVectorsFormat writes .vec/.vex/.vem directly, on-heap.
3. Because the engine is recorded per field and the graph is built per segment, a single index can in principle host faiss and lucene fields side by side; the codec dispatches per field.
4. After a version upgrade, old faiss/nmslib segments are read by backward_codecs/; lucene-engine segments are read by Lucene's own backward codec machinery. This is why deleting a backward codec is a BWC break.

# Prove which files an engine produces. Build and run an index per engine, then:
find /path/to/data/nodes/0/indices/*/0/index -type f | sed 's/.*\.//' | sort | uniq -c
# A faiss field adds k-NN's native files; a lucene field adds .vec/.vex/.vem.
# Find the codec's per-engine dispatch in the writer:
grep -rn "NativeEngines990KnnVectorsFormat\|getKnnVectorsFormatForField\|engine" \
  src/main/java/org/opensearch/knn/index/codec/ | head

The "new vector engine" meta

Three engines is not a stable equilibrium — it is a snapshot. The forward-looking view is captured in [META] Supporting New Vector Engine in OpenSearch — k-NN #2605. Read it for two reasons.

First, it frames the engines not as fixed products but as pluggable backends behind one KNNEngine abstraction — which is exactly why adding a new one (or deprecating nmslib) is a contained change rather than a rewrite. The engine param, the per-engine method/capability declarations, the codec dispatch you just traced: all of it is the seam designed to let engines come and go.

Second, it is where the roadmap lives: where GPU/remote build (#2293, #2294) fit, how quantization evolves, and what "next engine" would even mean. If you want to contribute somewhere genuinely open, the meta-issue and the RFCs it links are the map. Use a GitHub search like repo:opensearch-project/k-NN is:issue label:Enhancement engine to find the live sub-tasks; do not assume the numbers above are the whole list.

Note: Treat #2605 as the index and #2293/#2294 as the current frontier chapters. The durable skill is reading a meta-issue, following its task list, and finding the one unclaimed, well-scoped sub-task — see design via GitHub.

Common bugs and symptoms

Validation: prove you understand this

1. Name the three engines and give the one-line reason each exists. Which is the default, and which is blocked for new indices in 3.0 (and since which version was it deprecated)?
2. Reproduce the capability matrix from memory for at least these rows: algorithms, PQ, filtering, memory location (on/off heap), training, and JNI usage.
3. Explain why the lucene engine has exact filtering but no PQ, in terms of whose format and code it uses. Why does a lucene-engine field never appear in the off-heap native-memory stats?
4. A user has 2B vectors, a tight RAM budget, and no filtering. Pick an engine and a storage strategy, and name the chapter that covers the strategy. Now they add a strict per-query filter and the vectors shrink to fit heap — does your choice change, and why?
5. Trace how the engine value flows from the mapping to segment files: which class records it, when the graph is built, which codec/format runs for faiss vs lucene, and which files appear on disk (and whether they load on- or off-heap).
6. Read k-NN #2293 and #2294. In one sentence each: what is being offloaded, where to, and which phase of indexing it targets. Why are both faiss-engine features and not lucene ones?
7. Explain why deleting the nmslib read path / its backward codec would be a regression, citing the kind of failure it is and the contributor-mindset chapter that names it.

When you can do all seven, go to algorithms — HNSW, IVF, PQ for the mechanics behind each method, then quantization and disk-ANN for the faiss memory story in full. For the off-heap subsystem the faiss/nmslib engines depend on, see native integration and memory. For the engine roadmap and how to find unclaimed work, read [META] #2605 and real issues and RFCs.