Quantization and Disk-Based ANN

A million 768-dimensional float32 vectors is ~3 GB of raw vectors before you add the HNSW graph. A billion is ~3 TB. The native-memory circuit breaker exists precisely because this number gets out of hand, and the algorithms chapter's headline trade-off — HNSW pays memory for speed — becomes the binding constraint at scale. Quantization is the answer: store each vector in fewer bits, accept a little recall loss, and recover most of it with a full-precision rescore pass.

This chapter is the memory story. It walks the compression spectrum k-NN offers — byte vectors, FP16 scalar quantization, Product Quantization (PQ), Binary Quantization (BQ) — and then disk-based vector search (on_disk mode with compression_level), which is the productized "store the compressed graph in RAM, keep full-precision on disk, rescore from disk" pattern. It maps each to its Lucene equivalent, and it points at the cutting-edge GPU / remote-index-build RFCs that make building these compressed indexes fast. It assumes algorithms (especially PQ), the query path (especially rescoring), and native integration (the memory the compression is fighting).

After this chapter you can:

• Place byte/FP16/PQ/BQ on a memory-vs-recall spectrum and pick one for a constraint.
• Explain on_disk mode and compression_level (1x → 32x) and how rescoring makes it usable.
• Map k-NN quantization to Lucene's Lucene99/Lucene104 scalar-quantized formats.
• Reason about why a quantized index lost recall and how rescoring/oversampling recovers it.
• Say what GPU and remote index build buy you and cite the RFCs.

Note on terminology: the cluster manager (formerly master) coordinates the model system index and (in the RFCs below) the remote-build component, but quantization itself is a per-segment, per-shard concern that lives in the codec and the native/Lucene vector format.

The problem, in bytes

(The HNSW graph's neighbour lists add to all of these; quantization shrinks the vectors, not the graph edges.) The whole game is buying down the per-vector cost without buying too much recall loss — and the universal recovery mechanism is rescore against full precision, which is why this chapter and the query path's rescoring section are inseparable.

Byte vectors (int8) — the cheap, lossy-but-simple win

Since 2.17, both the faiss and lucene engines support a knn_vector whose data_type is byte: each component is a signed 8-bit integer in [-128, 127] instead of a float. That is a flat 4× reduction with no codebooks and no training — the simplest compression on the menu.

PUT /products-byte
{
  "settings": { "index.knn": true },
  "mappings": {
    "properties": {
      "embedding": {
        "type": "knn_vector",
        "dimension": 768,
        "data_type": "byte",
        "space_type": "l2",
        "method": { "name": "hnsw", "engine": "faiss", "parameters": { "m": 16, "ef_construction": 128 } }
      }
    }
  }
}

The catch: you must quantize the vectors to [-128, 127] before indexing (or rely on the engine's quantization where supported). Byte vectors are exact as bytes — no approximation in the storage — but the act of squeezing a float embedding into 8 bits per dim is itself lossy unless your embeddings were already in that range. The distance math runs directly on bytes (faster than float on many CPUs), so byte vectors are a latency win as well as a memory win.

grep -rn "data_type\|VectorDataType\|BYTE\|FLOAT\|KNNVectorFieldMapper" \
  src/main/java/org/opensearch/knn/index/mapper \
  src/main/java/org/opensearch/knn/common/KNNConstants.java

FP16 scalar quantization — half the bytes, near-zero recall loss

FP16 (half-precision) scalar quantization stores each component as a 16-bit float — a 2× reduction with very little recall impact, because IEEE-754 half-precision still captures the magnitude and most of the precision of typical normalized embeddings. In faiss this is the fp16 encoder (a scalar quantizer, SQ); the engine handles the float→half conversion, so unlike raw byte vectors you do not pre-quantize.

"method": {
  "name": "hnsw",
  "engine": "faiss",
  "parameters": {
    "encoder": { "name": "sq", "parameters": { "type": "fp16" } }
  }
}

FP16 is the conservative choice: when you want to roughly halve memory and are not willing to risk recall, FP16 with no rescoring is usually safe. It is also the gentlest step on the spectrum — if FP16 already meets your memory budget, you do not need PQ/BQ or disk mode.

grep -rn "fp16\|ENCODER_SQ\|ScalarQuantization\|SQType\|FP16\|clip\|minScore" \
  src/main/java/org/opensearch/knn/common/KNNConstants.java \
  src/main/java/org/opensearch/knn/index/util 2>/dev/null

Product Quantization (PQ) — aggressive, trained, faiss-only

PQ is covered mechanically in the algorithms chapter: split each vector into m sub-vectors, learn a codebook per sub-space, store the nearest-centroid id per sub-vector. At d=768, m=96, nbits=8 that is 32× compression (3072 B → 96 B). The two things to remember here, in the memory story:

• PQ is the most aggressive float-based compression and the only one requiring training (the codebooks are learned k-means — needs the _train API and the .opensearch-knn-models system index; see algorithms § Training).
• PQ distances are computed between reconstructed (codebook-centroid) approximations, so PQ demands rescoring for production recall. Use PQ for the fast first pass, then rescore against full-precision vectors (see query path § rescoring).

PQ composes with both HNSW and IVF (the classic IVF+PQ billion-scale faiss recipe). It is faiss-only — the lucene engine has no PQ; its compression story is scalar quantization, below.

Binary Quantization (BQ) — one bit per dimension, the extreme

Binary Quantization reduces each component to a single bit (typically: is this dimension above or below a threshold?), and compares vectors with Hamming distance (population count of XOR) — an operation modern CPUs do in one instruction over 64 bits at a time. At 1 bit/dim that is a 32× reduction (same footprint as PQ-96×8 above, but train-free and far cheaper to compare).

BQ's recall loss is the steepest of the spectrum — you have thrown away almost all magnitude information — so BQ always pairs with rescoring, and often with oversampling factors larger than PQ's. It shines as the first stage of a funnel: binary Hamming to cheaply shortlist a large candidate set, then full-precision rescore to rank. The space_type for binary vectors is hamming.

grep -rn "hamming\|BINARY\|BinaryQuant\|bitcount\|popcount\|VectorDataType.BINARY" \
  src/main/java/org/opensearch/knn/common/KNNConstants.java \
  src/main/java/org/opensearch/knn/index/mapper

The compression spectrum at a glance

Disk-based ANN: on_disk mode and compression_level

Quantization shrinks the in-memory footprint. Disk-based vector search goes further: keep the compressed graph in RAM (small, fast to traverse) and keep the full-precision vectors on disk, reading them only to rescore the handful of candidates the first pass produced. This is the on_disk mode, and it is the productized version of "PQ/BQ retrieve + float32 rescore" with the storage tiering made explicit.

PUT /products-disk
{
  "settings": { "index.knn": true },
  "mappings": {
    "properties": {
      "embedding": {
        "type": "knn_vector",
        "dimension": 768,
        "space_type": "innerproduct",
        "mode": "on_disk",
        "compression_level": "32x"
      }
    }
  }
}

You set mode: on_disk and a compression_level — one of 1x, 2x, 4x, 8x, 16x, 32x — and k-NN picks the appropriate quantization (e.g. higher levels use binary/aggressive quantization) to hit that ratio, builds the small in-RAM graph over the compressed vectors, and automatically enables rescoring against the full-precision vectors kept on disk. 1x is effectively "in-memory full precision"; 32x is the most aggressive, smallest-RAM, most-rescore-dependent setting.

flowchart TD
    subgraph RAM["in RAM (small)"]
      G["compressed HNSW graph<br/>(quantized vectors per compression_level)"]
    end
    subgraph DISK["on disk (large)"]
      FP["full-precision float32 vectors"]
    end
    Q[query vector] --> G
    G --> CAND["pass 1: ANN over compressed graph<br/>-> oversample*k candidates"]
    CAND --> RD["read candidates' float32 from disk"]
    RD --> RS["pass 2: full-precision rescore"]
    FP --> RD
    RS --> TK[true top-k]

Warning: Disk mode trades memory for latency. The rescore pass reads full-precision vectors from disk (or OS page cache); under cache misses that is a random-read penalty per query. The win is fitting a corpus in a fraction of the RAM; the cost is tail latency on cold reads. Tune oversample_factor (more candidates → better recall, more disk reads) against your latency budget. See the query path's rescoring section.

grep -rn "on_disk\|compression_level\|CompressionLevel\|Mode\b\|MODE_ON_DISK\|ON_DISK" \
  src/main/java/org/opensearch/knn/index \
  src/main/java/org/opensearch/knn/common/KNNConstants.java

How it maps to Lucene scalar quantization

The lucene engine does not call faiss; it rides Lucene's own quantized vector formats, covered in HNSW Vector Search in Lucene. When you ask the lucene engine for quantization, you are configuring Lucene's KnnVectorsFormat:

So the same idea (scalar-quantize the vectors, optionally rescore with full precision) shows up twice in k-NN: once as the faiss sq/fp16 encoder + native graph, and once as Lucene's Lucene99*/Lucene104* quantized formats for the lucene engine. The int8 scalar-quantization lineage in Lucene traces to LUCENE-10577 / apache/lucene #11613, and the scalar-quantization codec to apache/lucene #12497. Lucene also gets its SIMD distance math from the Panama Vector API, so a Lucene104* int8 distance is a hardware VPDPBUSD-style dot product when the CPU and JDK support it.

# Confirm which Lucene quantized formats your bundled Lucene ships (run in a Lucene checkout):
grep -rln "Lucene9.*ScalarQuantizedVectorsFormat\|Lucene10.*ScalarQuantizedVectorsFormat" \
  lucene/core/src/java
# And which one k-NN's lucene engine wires up:
grep -rn "ScalarQuantized\|Lucene9\|Lucene10\|KnnVectorsFormat\|confidenceInterval\|bits" \
  src/main/java/org/opensearch/knn/index/codec 2>/dev/null

Building compressed indexes fast: GPU and remote index build

Quantization shrinks the stored index; it does not make building the index cheap. Constructing an HNSW graph (even over compressed vectors) is CPU-heavy and happens during indexing and every merge — it is often the dominant cost of a large k-NN ingest. Two active, cutting-edge directions attack this, and both are real RFCs worth reading before you touch index-build code:

• GPU-accelerated build — k-NN #2293, [RFC] Boosting OpenSearch Vector Engine Performance using GPUs. Uses NVIDIA cuVS and the CAGRA graph-build algorithm (a GPU-native ANN graph) to construct the vector index in FP32 on the GPU, then serve it on CPU. Graph build is embarrassingly parallel; a GPU can build in a fraction of the wall-clock time.
• Remote vector index build — k-NN #2294, [RFC] Remote Vector Index Build and the component meta k-NN #2391, [Meta] Remote Vector Index Build Component. The idea: offload the per-segment graph construction from the data node to a remote GPU/CPU fleet — the data node ships the segment's vectors out, a remote builder constructs the graph, and the result is pulled back and written into the segment. This decouples index-build CPU from the search cluster and lets you use specialized (GPU) hardware for builds only. Benchmarking of these lives in k-NN #2595.

flowchart LR
    subgraph DN["data node (search cluster)"]
      SEG["segment vectors (flush/merge)"]
    end
    subgraph REMOTE["remote build fleet (GPU)"]
      B["cuVS / CAGRA graph build (FP32)"]
    end
    SEG -->|ship vectors| B
    B -->|return built graph| SEG2["write graph into segment file"]
    SEG2 --> SRV["served on CPU like any k-NN segment"]

These are the frontier. They do not change the query path (a remotely-built graph queries identically to a locally-built one); they change who pays the build cost and on what hardware. For current status, search rather than assume:

repo:opensearch-project/k-NN is:issue GPU cuVS CAGRA remote build
repo:opensearch-project/k-NN is:issue label:"Roadmap" remote index build

Choosing: a decision guide

The universal rule: start gentle (FP16/byte), measure recall against an exact script_score ground truth (see query path), and only escalate to PQ/BQ/disk when memory forces you to — adding a rescore pass each time you do. Never ship an aggressively-quantized index without measuring post-rescore recall against ground truth; the compression ratio in the table is meaningless if recall is unacceptable for your task.

Common bugs and symptoms

Validation: prove you understand this

1. Compute per-vector storage for d=1024 under float32, byte, FP16, PQ(m=128, nbits=8), and binary. State each compression ratio and which require training.
2. Explain why quantized retrieval needs rescoring and what the rescore pass computes. For which methods is rescoring optional vs mandatory?
3. Walk an on_disk query end to end: what is in RAM, what is on disk, what pass 1 and pass 2 do, and what compression_level and oversample_factor each trade off.
4. A team needs 1B vectors on a memory-bounded cluster and can tolerate a training step and some tail latency. Specify a concrete mapping (engine, algorithm, compression, rescore) and justify each choice.
5. Map FP16/int8/PQ to their k-NN configuration and their Lucene-engine equivalent. Which compression methods have no Lucene-engine analogue, and why?
6. Explain what GPU build (cuVS/CAGRA, #2293) and remote index build (#2294/#2391) change about a k-NN deployment — and what they deliberately do not change about the query path.
7. You are handed an aggressively-quantized index with disappointing recall. Give the ordered diagnostic steps (measure against what ground truth; which knobs first; when to back off the compression level).

When you can do all seven, you have closed the loop: the algorithms decide the structure, this chapter decides how cheaply you store it, the query path decides how it serves a query, and native integration and memory decides where it all lives at runtime. For the Lucene-layer quantized formats these map onto, read HNSW in Lucene and SIMD and the Vector API. Then benchmark it for real in lab-k6-benchmark-recall-latency.