Project 1: A Disk-Based Quantization Mode for k-NN

Vectors are expensive. A single 768-dimensional float[] is 3 KB; a hundred million of them is ~300 GB of raw data, and the faiss engine loads its HNSW graph outside the JVM heap into native memory before it can answer a query. The whole "quantization and disk-ANN" story in the k-NN plugin exists to make that affordable: store vectors in fewer bytes, keep less in RAM, and pay back the lost precision with a rescoring pass. This project asks you to implement — or meaningfully extend — a quantization / disk-ANN capability in the k-NN plugin, starting from a slice you can finish in a weekend and ending at something a maintainer would review.

This is the natural first portfolio project if you came through the vector chapters. It is also the one with the cleanest "land a scoped slice upstream" path, because the validation and stats surface is small, self-contained, and exactly the kind of thing maintainers merge.

Note: Read Quantization and Disk-ANN and k-NN Native JNI and Memory before you start. This brief assumes you know what on_disk mode, compression_level, the FP16/PQ/BQ knobs, and the native-memory circuit breaker are. It will not re-derive them.

Problem & motivation

The k-NN plugin already ships a rich storage menu: byte vectors (2.17+), FP16 scalar quantization, Product Quantization (PQ), Binary Quantization (BQ), and a disk-based mode (mode: on_disk) with compression_level ∈ {1x, 2x, 4x, 8x, 16x, 32x} and rescoring on the full-precision vectors. The memory story is the whole reason vectors are usable at scale.

But "rich menu" hides sharp edges:

• Validation is thin and scattered. Which (engine, space_type, mode, compression_level, data_type) combinations are actually legal? Try a few illegal ones (on_disk with an engine that does not support it, a compression_level the engine cannot honour, a binary data_type with a quantization that assumes floats) and you will get inconsistent errors — sometimes a clean 400, sometimes a confusing native-side failure on first query, sometimes silent acceptance of a combination that quietly degrades recall.
• The cost of a mode is invisible. A knn_vector field in on_disk mode at 8x compression has a wildly different native-memory footprint than the same field at 1x. The GET /_plugins/_knn/stats surface does not cleanly tell an operator "this field is quantized this way, costing roughly this much."
• Adding a new mode is a deep, multi-file change that most contributors never attempt, because the field mapper, the engine/method registration, the codec, and the native build are all involved and there is no single "here is how a mode flows end to end" map.

This project makes one of those better. The phased plan starts at validation + a stats field (genuinely mergeable, low blast radius) and builds toward a real feature.

Real-world grounding

The grounding issue is the meta-issue for the vector engine's evolution and the quantization roadmap:

• [META] Supporting a New Vector Engine in OpenSearch — k-NN #2605: https://github.com/opensearch-project/k-NN/issues/2605

That meta-issue frames how new storage/engine capabilities get added — exactly the surface you are touching. Around it sits the broader memory and disk-ANN work:

• The native-memory circuit-breaker rearchitecture discussion — k-NN #1582: https://github.com/opensearch-project/k-NN/issues/1582
• A real, concrete circuit-breaker config bug — k-NN #585: https://github.com/opensearch-project/k-NN/issues/585

Citation discipline: the disk-ANN / compression-level feature set evolves fast. Do not cite an issue number you have not opened. Instead, run a live search before you scope: is:issue is:open label:"vector indexing" disk OR quantization OR compression and is:issue is:open on_disk in the opensearch-project/k-NN repo. Link the current issue in your design note. The meta-issue above is the durable anchor.

Subsystems you'll touch

Deep dives that cover the surrounding ground: k-NN engines · algorithms (HNSW/IVF/PQ) · native JNI and memory · quantization and disk-ANN · k-NN query path · the Lucene HNSW format the lucene engine reuses.

Phased plan

The discipline of this project is that Phase 1 is independently mergeable. You do not need to finish Phases 3–4 to have shipped something real.

Phase 0 — Build it and map the mode path (½ day)

Build k-NN from source (CMake JNI build + Gradle) per lab-k1-build-knn-from-source, then trace how a mode/compression_level mapping turns into encoder parameters.

# In your k-NN clone:
./gradlew build -x test          # full build incl. native (or buildKNNLib first; see lab-k1)
./gradlew run                    # single node with the plugin installed, REST on :9200

# Create a disk-mode field and watch it work:
curl -s -X PUT localhost:9200/disk-test -H 'Content-Type: application/json' -d '{
  "settings": { "index.knn": true },
  "mappings": { "properties": { "v": {
    "type": "knn_vector", "dimension": 8,
    "space_type": "l2",
    "mode": "on_disk", "compression_level": "8x"
  }}}
}' | python3 -m json.tool

Grep the path from mapping to encoder:

grep -rn "on_disk\|CompressionLevel\|compression_level\|\bMode\b" src/main/java/org/opensearch/knn/index | head -40
grep -rn "class KNNVectorFieldMapper\|parseCreateField\|TypeParser" src/main/java/org/opensearch/knn/index/mapper
grep -rn "supportedMethods\|MethodComponent\|Encoder\b" src/main/java/org/opensearch/knn/index/engine | head -40

Write a one-page capstone-work/mode-path.md: mapping JSON → KNNVectorFieldMapper.Builder → method/encoder resolution → codec field config → native build. With file:line citations. This is your execution-path-mastery artifact and you cannot skip it.

Phase 1 — Tighten validation (the scoped, mergeable slice)

Pick one illegal (engine, mode, compression_level, data_type, space_type) combination that currently fails late, silently, or confusingly, and make it fail early, clearly, and deterministically at mapping-creation time with a 400 and a precise message.

// In the resolver/validation path (grep for where the resolved config is checked):
if (mode == Mode.ON_DISK && !knnEngine.supportsMode(Mode.ON_DISK)) {
    throw new MapperParsingException(
        "Engine [" + knnEngine.getName() + "] does not support mode [on_disk]; "
        + "supported modes: " + knnEngine.getSupportedModes());
}
if (compressionLevel != CompressionLevel.x1
        && !knnEngine.supportsCompression(compressionLevel, dataType)) {
    throw new MapperParsingException(
        "Engine [" + knnEngine.getName() + "] does not support compression_level ["
        + compressionLevel.getName() + "] for data_type [" + dataType + "]");
}

Then a unit test that asserts the bad mapping is rejected and a good one is accepted:

// KNNVectorFieldMapperTests (OpenSearchTestCase-based)
public void testOnDiskUnsupportedEngineRejected() {
    XContentBuilder mapping = mapping(b -> b.startObject("v")
        .field("type", "knn_vector").field("dimension", 8)
        .field("mode", "on_disk")
        .startObject("method").field("name","hnsw").field("engine","nmslib").endObject()
        .endObject());
    MapperParsingException e = expectThrows(MapperParsingException.class,
        () -> createDocumentMapper(mapping));
    assertThat(e.getMessage(), containsString("does not support mode [on_disk]"));
}

Run the gates:

./gradlew spotlessApply
./gradlew :test --tests "org.opensearch.knn.index.mapper.KNNVectorFieldMapperTests"
./gradlew check -x integTest    # the broad gate; integTest needs the native lib

Why this is the right Phase 1: it is a minimum diff, it is exactly the kind of hardening maintainers merge without an RFC, and it forces you to read the resolver for real. A clean validation PR with a deterministic test is a credible first k-NN contribution.

Phase 2 — Surface the cost in stats

Add a per-field (or per-index) view to GET /_plugins/_knn/stats (or a _cat-style helper) that reports the quantization mode and an estimated native-memory footprint for each knn_vector field, so an operator can see what a field costs.

grep -rn "class KNNStats\|StatNames\|enum.*Stat\|register" src/main/java/org/opensearch/knn/plugin/stats
grep -rn "graphMemoryUsage\|NativeMemoryAllocation\|getSizeInKB\|cacheStats" src/main/java/org/opensearch/knn/index/memory

Add a stat name, populate it from NativeMemoryCacheManager, and round-trip it through the existing stats transport action. Test with KNNStatsTests (or the stats response serialization test) and a small integration test that creates two fields at different compression levels and asserts the reported footprints differ in the expected direction.

Phase 3 — A real mode/encoder extension

Now the feature. Choose one, scoped tightly, and write a design note first:

Whatever you pick: round-trip it through a real index, and prove recall (next phase).

Phase 4 — Prove recall and latency

A quantization change is meaningless without a recall number. Use the k-NN benchmark approach: index a known dataset, run a fixed query set, compute recall@k against an exact (1x / flat) baseline, and report before/after for your change.

config            recall@10   p50 query (ms)   native mem (MB)
1x (baseline)        1.000          18              3100
on_disk 8x (old)     0.91x          21               420
on_disk 8x (yours)   0.9xx          2x               4xx

Recall is the headline metric for any ANN change. A latency win that drops recall below the mode's documented floor is not a win — it is a bug.

Deliverables

• capstone-work/mode-path.md — the mapping → encoder → codec → native trace, with citations
• capstone-work/design.md — the gap, the chosen slice, the combinations you considered and rejected
• Phase 1: a validation tightening on a feature branch, with a unit test (red without the fix)
• Phase 2: a stats field exposing per-field quantization mode + footprint, with a test
• (Phase 3) a scoped mode/encoder extension behind clean validation
• (Phase 4) a recall@k + latency + native-memory before/after table on a real dataset
• capstone-work/validation.md — ./gradlew spotlessApply check output, test commands, seeds, dataset
• A CHANGELOG.md entry under the k-NN repo's ## [Unreleased]
• An upstreaming decision: a DCO-signed k-NN PR for the Phase-1/2 slice, or a written scoped proposal
• A 500–1000 word write-up: the gap, the path you traced, the trade-off you made

Difficulty & time

The native JNI build is the gate most people underestimate. Budget a full session for lab-k1-build-knn-from-source before you scope.

Stretch goals

• Wire the new stat into a _cat/knn style endpoint so footprint is visible at a glance.
• Add a mapping-time warning (not error) when a chosen (mode, compression_level) is legal but historically recall-risky for the given dimension, pointing at the rescore knobs.
• Extend the warmup API accounting so a warmed disk-mode field reports its loaded footprint, closing the loop between Phase 2's estimate and reality.
• Reproduce the spirit of k-NN #585 in a test: assert that the circuit breaker accounts for the quantized footprint, not the full-precision one.

Evaluation

Self-grade against the 100-point rubric. For this project:

A finished Phase 1–2 at 90+ is a real, merged k-NN contribution in the memory subsystem.

How to turn this into a real contribution

1. Start from validation, not the feature. Phase 1 is the upstream target. A 400-on-bad-mapping PR with a deterministic test is mergeable on its own merits and needs no RFC.
2. Comment before you code. Find the current disk/quantization issue (search query above), comment that you are tightening validation / adding a footprint stat, and confirm the maintainers want it. Link the [META] #2605 thread for context.
3. One slice per PR. Validation, then stats, then (if negotiated) the mode extension. Do not bundle a new mode with the validation cleanup — reviewers will (correctly) ask you to split it.
4. Bring a recall number to any encoder/mode change. The k-NN maintainers will ask for it; arrive with the table already in the PR description.
5. DCO applies. k-NN is GitHub-native like core: git commit -s, CHANGELOG.md, and the backport bot. Get the sign-off email right locally before you push.

If only Phase 1 lands, you have still shipped a real fix in one of OpenSearch's hardest plugins. That is the point of scoping from the small slice up.