Lab K3: The knn_vector Field Type

Background

Every k-NN index begins with one mapping decision: a field of type knn_vector. That single type declaration is where the plugin's most consequential choices are made — the dimension, the distance space, the engine (faiss / lucene / nmslib), the algorithm (hnsw / ivf), the data type (float / byte / binary), and the on-disk mode/compression story. Get the mapping right and indexing and querying "just work"; get it wrong and you get rejected documents, the wrong engine, exact scans instead of ANN, or a node that OOMs on native memory. The class that owns all of this is KNNVectorFieldMapper (with its companion KNNVectorFieldType), and reading it is the fastest way to understand how a high-level mapping JSON turns into low-level engine behavior.

In this Build-It lab you first read the mapper deeply — the ParametrizedFieldMapper Builder and its Parameter<> fields, the validation that happens at mapping time, and the wiring from the mapping into the KNNEngine and the custom codec. Then you make a small, real change: you add a validation guard to the dimension parameter (reject dimensions above a configurable ceiling with a clear error), and you write a KNNVectorFieldMapperTests-style unit test that proves it. The change is illustrative — the value is in landing a real edit in a real mapper and testing it the way the project does.

Note: The cluster manager (formerly master) publishes the index metadata that includes this mapping, but the mapper itself runs on the data nodes that index and search the field. Mapping validation happens when the mapping is parsed (on the node creating or updating the index); per-document parsing happens on the primary shard. Keep those two moments distinct — your validation guard fires at the first.

Read these alongside the lab:

• k-NN architecture — where the field mapper sits in the index path and how it reaches the codec.
• Engines — the faiss/lucene/nmslib capability fork your mapping selects.
• Mapping and analysis — the core ParametrizedFieldMapper/Mapper/FieldMapper machinery k-NN's mapper is built on; this lab does not re-derive it.

Why This Lab Matters for Contributors

• Mapping bugs and feature requests are a steady stream of k-NN issues ("dimension X rejected", "why can't I use filter with this engine", "add a new compression level"). All of them route through KNNVectorFieldMapper. Knowing it makes those issues legible.
• The mapper is the single point where a user's intent (engine, space, method, data type, mode) is validated against what the chosen KNNEngine supports — exactly the kind of small, well-scoped change a new contributor can own.
• Writing a ParametrizedFieldMapper Builder with Parameter<> validators is a transferable OpenSearch skill — every field type in core and in plugins uses this exact pattern. You learn it once, on the most interesting field type in the ecosystem.
• The project's mapper-test idiom (TypeParser.parse(...) + expectThrows) is how you prove a mapping change without booting a cluster. It is fast, deterministic, and reviewer-friendly.

Prerequisites

• A from-source k-NN build (Lab K1); ./gradlew test works.
• You have traced the query path (Lab K2) and read k-NN architecture.
• Comfort with the core mapping & analysis deep dive — specifically the ParametrizedFieldMapper/Parameter model.

cd ~/src/oss-repos/k-NN
ls src/main/java/org/opensearch/knn/index/mapper/
#   KNNVectorFieldMapper.java  KNNVectorFieldType.java  KNNMappingConfig.java
#   EngineFieldMapper.java  FlatVectorFieldMapper.java  ModelFieldMapper.java
#   CompressionLevel.java  Mode.java  VectorValidator.java  PerDimensionValidator.java ...

Step-by-Step Tasks

Step 1: Map the mapping parameters to the source

The knn_vector field type accepts a fixed set of mapping parameters. Find where each is declared — they are Parameter<> fields on KNNVectorFieldMapper.Builder.

grep -n "class Builder\|Parameter<\|KNNConstants\.\|restrictedStringParam\|stringParam" \
  src/main/java/org/opensearch/knn/index/mapper/KNNVectorFieldMapper.java | head -40

# The constant names the parameters use:
grep -n "DIMENSION\|KNN_METHOD\|KNN_ENGINE\|VECTOR_DATA_TYPE_FIELD\|MODE_PARAMETER\|COMPRESSION_LEVEL\|MODEL_ID\|TOP_LEVEL_PARAMETER_SPACE_TYPE" \
  src/main/java/org/opensearch/knn/common/KNNConstants.java | head

You should be able to reconstruct this table from the source (the Builder.getParameters() list — there are 11, confirmed by testBuilder_getParameters):

Note: space_type and engine can appear either at the top level of the mapping or nested under method — k-NN supports both spellings (the topLevelSpaceType/topLevelEngine parameters). Grep TOP_LEVEL_PARAMETER_SPACE_TYPE; it matters when you read validation, because both spellings must be reconciled.

Step 2: Read the validation — where bad mappings are rejected

The mapper validates the combination of parameters against the chosen engine's capabilities. This is the heart of the field type and where most of its bugs live.

# Per-element validators (e.g. byte range, fp16 range):
grep -n "class\|validate\|FloatVectorValidator\|ByteVectorValidator" \
  src/main/java/org/opensearch/knn/index/mapper/PerDimensionValidator.java \
  src/main/java/org/opensearch/knn/index/mapper/VectorValidator.java

# Whole-vector validators (e.g. space-specific constraints):
grep -n "class\|validateVector" \
  src/main/java/org/opensearch/knn/index/mapper/SpaceVectorValidator.java

# Engine-capability checks at build time (e.g. nmslib has no filter, lucene/ivf rules):
grep -rn "validateMethod\|isTrainingRequired\|validate\|UnsupportedOperationException\|addValidationError" \
  src/main/java/org/opensearch/knn/index/mapper/KNNVectorFieldMapper.java | head -20

The dimension parameter is the cleanest example. Read its declaration — a Parameter<Integer> whose (name, context, value) -> { ... } lambda parses and validates:

grep -n "Parameter<Integer> dimension" src/main/java/org/opensearch/knn/index/mapper/KNNVectorFieldMapper.java

// src/main/java/org/opensearch/knn/index/mapper/KNNVectorFieldMapper.java (illustrative — grep for the real block)
protected final Parameter<Integer> dimension = new Parameter<>(
    KNNConstants.DIMENSION,
    false,
    () -> UNSET_MODEL_DIMENSION_IDENTIFIER,
    (n, c, o) -> {                                   // n = name, c = context, o = raw value
        if (o == null) {
            throw new IllegalArgumentException("Dimension cannot be null");
        }
        int value;
        try {
            value = XContentMapValues.nodeIntegerValue(o);
        } catch (Exception exception) {
            throw new IllegalArgumentException(
                String.format(Locale.ROOT, "Unable to parse [dimension] from provided value [%s] for vector [%s]", o, name));
        }
        if (value <= 0) {                            // <-- the existing lower-bound guard
            throw new IllegalArgumentException(
                String.format(Locale.ROOT, "Dimension value must be greater than 0 for vector: %s", name));
        }
        return value;
    },
    m -> toType(m).originalMappingParameters.getDimension()
);

This lower-bound guard is exactly the shape of the change you will make: add an upper-bound guard beside it.

Step 3: Wire the mapping to the engine and codec (read-only)

Before changing anything, confirm how the validated mapping reaches the engine and the codec — this is what your validation is protecting.

# The field type holds the resolved mapping config used at index/query time:
grep -n "class KNNVectorFieldType\|getKnnMappingConfig\|KNNMappingConfig\|getKnnEngine\|SpaceType" \
  src/main/java/org/opensearch/knn/index/mapper/KNNVectorFieldType.java \
  src/main/java/org/opensearch/knn/index/mapper/KNNMappingConfig.java | head

# The engine the mapping selects (faiss/lucene/nmslib) and its capabilities:
grep -rn "enum KNNEngine\|FAISS\|LUCENE\|NMSLIB\|getMaxDimension\|supports" \
  src/main/java/org/opensearch/knn/index/engine/KNNEngine.java | head

# How the codec gets the native-vs-lucene format from the engine (the index path):
grep -rn "NativeEngines990KnnVectorsFormat\|KnnVectorsFormat\|getVectorsFormat\|EngineFieldMapper" \
  src/main/java/org/opensearch/knn/index/codec/ \
  src/main/java/org/opensearch/knn/index/mapper/EngineFieldMapper.java | head

The picture: KNNVectorFieldMapper.Builder validates → produces a KNNVectorFieldType holding a KNNMappingConfig (dimension, SpaceType, KNNEngine, method/model) → at flush/merge the engine selects the vectors format (native NativeEngines990KnnVectorsFormat for faiss/nmslib, Lucene's own format for lucene) → the graph is written as segment files. Your validation runs before any of this, at mapping-parse time, so a bad dimension never reaches the engine.

Map params → engine behavior, the version you should be able to defend:

Step 4: Make the change — add an upper-bound dimension guard

Add a configurable maximum-dimension check beside the existing value <= 0 guard. The ceiling should be read from a constant so it is one obvious place to change, and the error must name the field and the limit (good mapper errors are specific). First, find the real site:

grep -n "Dimension value must be greater than 0" \
  src/main/java/org/opensearch/knn/index/mapper/KNNVectorFieldMapper.java

Add a constant (near the other KNNConstants, or as a static final on the mapper — grep to match the project's placement convention; an illustrative inline constant is shown here):

// In KNNVectorFieldMapper (or KNNConstants — match the project's convention).
// Illustrative ceiling; real engine maxima live on KNNEngine (grep getMaxDimension).
public static final int LAB_MAX_DIMENSION = 16_000;

Then extend the dimension parameter's validation lambda, immediately after the existing lower-bound check:

        if (value <= 0) {
            throw new IllegalArgumentException(
                String.format(Locale.ROOT, "Dimension value must be greater than 0 for vector: %s", name));
        }
        // --- Lab K3 addition: upper-bound guard ---
        if (value > LAB_MAX_DIMENSION) {
            throw new IllegalArgumentException(
                String.format(
                    Locale.ROOT,
                    "Dimension value [%d] exceeds the maximum supported dimension [%d] for vector: %s",
                    value, LAB_MAX_DIMENSION, name));
        }
        // --- end addition ---
        return value;

Warning: This is deliberately illustrative. In a real PR you would not hardcode a ceiling — you would derive it from the chosen KNNEngine (each engine has its own maximum; grep getMaxDimension src/main/java/org/opensearch/knn/index/engine/KNNEngine.java), because faiss, lucene, and nmslib do not share a limit. Note that as the production-correct version in your PR description, and reference the per-engine maxima. The point of the lab is the mechanics of adding and testing a mapper validation, not this specific number.

Step 5: Write the unit test — KNNVectorFieldMapperTests-style

The project tests the mapper without a cluster: build a mapping with XContentBuilder, run it through KNNVectorFieldMapper.TypeParser.parse(...), and assert. Use expectThrows for the rejection path. Add this to src/test/java/org/opensearch/knn/index/mapper/KNNVectorFieldMapperTests.java:

public void testDimension_aboveMaximum_thenThrows() throws IOException {
    ModelDao modelDao = mock(ModelDao.class);
    KNNVectorFieldMapper.TypeParser typeParser = new KNNVectorFieldMapper.TypeParser(() -> modelDao);

    // A mapping with a dimension just over the lab ceiling.
    XContentBuilder tooBig = XContentFactory.jsonBuilder()
        .startObject()
        .field(TYPE_FIELD_NAME, KNN_VECTOR_TYPE)
        .field(DIMENSION_FIELD_NAME, KNNVectorFieldMapper.LAB_MAX_DIMENSION + 1)
        .endObject();

    IllegalArgumentException ex = expectThrows(
        IllegalArgumentException.class,
        () -> typeParser.parse(
            TEST_FIELD_NAME,
            xContentBuilderToMap(tooBig),
            buildParserContext(TEST_INDEX_NAME, settings)   // settings is the test base helper
        )
    );
    assertTrue(ex.getMessage(), ex.getMessage().contains("exceeds the maximum supported dimension"));
}

public void testDimension_atMaximum_thenSucceeds() throws IOException {
    ModelDao modelDao = mock(ModelDao.class);
    KNNVectorFieldMapper.TypeParser typeParser = new KNNVectorFieldMapper.TypeParser(() -> modelDao);

    // Exactly at the ceiling must still parse (boundary case).
    XContentBuilder atMax = XContentFactory.jsonBuilder()
        .startObject()
        .field(TYPE_FIELD_NAME, KNN_VECTOR_TYPE)
        .field(DIMENSION_FIELD_NAME, KNNVectorFieldMapper.LAB_MAX_DIMENSION)
        .endObject();

    KNNVectorFieldMapper.Builder builder = (KNNVectorFieldMapper.Builder) typeParser.parse(
        TEST_FIELD_NAME,
        xContentBuilderToMap(atMax),
        buildParserContext(TEST_INDEX_NAME, settings)
    );
    assertEquals(KNNVectorFieldMapper.LAB_MAX_DIMENSION, (int) builder.getOriginalParameters().getDimension());
}

Note: Helpers like buildParserContext, the settings field, xContentBuilderToMap, and getOriginalParameters() come from the existing test class (extends KNNTestCase). Copy the current signatures from an existing testTypeParser_* test rather than trusting these; the base class is KNNTestCase, not the generic OpenSearchTestCase.

Step 6: Build, run the test, and verify against a live node

# Compile + run only your new tests (fast inner loop):
./gradlew test --tests "org.opensearch.knn.index.mapper.KNNVectorFieldMapperTests" \
  --tests "*testDimension_aboveMaximum_thenThrows" \
  --tests "*testDimension_atMaximum_thenSucceeds"

# Run the whole mapper test class to confirm you didn't break a sibling test:
./gradlew test --tests "org.opensearch.knn.index.mapper.KNNVectorFieldMapperTests"

Then prove it end to end on a node (./gradlew run with your change):

# Rejected: dimension over the ceiling -> 400 with your specific message.
curl -s -XPUT 'localhost:9200/dim_too_big?pretty' -H 'Content-Type: application/json' -d '{
  "settings": { "index.knn": true },
  "mappings": { "properties": {
    "v": { "type": "knn_vector", "dimension": 16001,
           "method": { "name": "hnsw", "engine": "faiss", "space_type": "l2" } }
  }}
}'
# Expect: "exceeds the maximum supported dimension [16000]"

# Accepted: a normal dimension still works.
curl -s -XPUT 'localhost:9200/dim_ok?pretty' -H 'Content-Type: application/json' -d '{
  "settings": { "index.knn": true },
  "mappings": { "properties": {
    "v": { "type": "knn_vector", "dimension": 128,
           "method": { "name": "hnsw", "engine": "faiss", "space_type": "l2" } }
  }}
}'
# Expect: "acknowledged": true

Step 7: Precommit and a clean commit

./gradlew spotlessApply
./gradlew precommit
git checkout -b lab/k3-dimension-guard
git add src/main/java/org/opensearch/knn/index/mapper/KNNVectorFieldMapper.java \
        src/test/java/org/opensearch/knn/index/mapper/KNNVectorFieldMapperTests.java
git commit -s -m "Lab K3: add upper-bound dimension validation to knn_vector mapper"

Implementation Requirements / Deliverables

• You can reconstruct the knn_vector parameter table from Builder.getParameters() (all 11 params) and explain what each one drives in the engine.
• The new upper-bound guard added to the dimension Parameter validation lambda, with a specific, field-naming error message.
• A KNNVectorFieldMapperTests test for the reject path (expectThrows, IllegalArgumentException) and the boundary accept path.
• ./gradlew test --tests "*KNNVectorFieldMapperTests" passes, including your new tests and all pre-existing ones.
• A live node rejects an over-ceiling mapping with your message and accepts a normal one.
• precommit passes; a DCO-signed commit; SPDX headers intact on every touched file.
• In your write-up: the production-correct version of this change (per-engine getMaxDimension) and why hardcoding a single ceiling is wrong.

Troubleshooting

Expected Output

# Unit tests
> Task :test
KNNVectorFieldMapperTests > testDimension_aboveMaximum_thenThrows PASSED
KNNVectorFieldMapperTests > testDimension_atMaximum_thenSucceeds  PASSED
BUILD SUCCESSFUL

# Live reject (dimension 16001)
{
  "error": {
    "root_cause": [ { "type": "illegal_argument_exception",
      "reason": "Dimension value [16001] exceeds the maximum supported dimension [16000] for vector: v" } ],
    "type": "illegal_argument_exception"
  },
  "status": 400
}

# Live accept (dimension 128)
{ "acknowledged": true, "shards_acknowledged": true, "index": "dim_ok" }

Stretch Goals

1. Make it per-engine and correct. Replace LAB_MAX_DIMENSION with the chosen KNNEngine's real maximum (grep getMaxDimension on KNNEngine), so faiss, lucene, and nmslib each enforce their own ceiling. Update the error to name the engine, and add a test per engine.
2. Validate a parameter combination. Add a check that rejects space_type: hamming unless data_type: binary (grep SpaceType/VectorDataType to find where these meet), with a test. This teaches cross-parameter validation, the harder and more common case.
3. Trace the value into the codec. With your guard in place, index a vector and confirm (via Lab K2's technique) that the validated dimension flows into KNNMappingConfig and the vectors format. Where would a wrong dimension have surfaced if validation hadn't caught it?
4. Add a deprecation-style warning. Instead of rejecting, emit a deprecation log for a discouraged-but-allowed dimension range, using the deprecation logger (grep core for DeprecationLogger). Note when a warning beats a hard error.
5. Write a YAML REST test. Add a do: indices.create + catch: bad_request test under src/yamlRestTest/... asserting the over-ceiling mapping is rejected, and run the yamlRestTest task.

Coding Exercises

The lab walked you through one mapper change end to end. These exercises broaden that into a graded ladder: more parameters, more validators, a field-type variant, and finally a new mapping parameter wired all the way to the engine. Locate every Parameter<>, validator, and constant with rg before editing — the mapper is refactored often and line numbers lie.

1. (warm-up) A boundary-table test for the existing dimension guard. Before adding anything, pin the current behavior. Write a parameterized KNNVectorFieldMapperTests method that runs TypeParser.parse(...) across a table of dimensions — 0, 1, a normal value, and (after you add the Step-4 ceiling) LAB_MAX_DIMENSION and +1 — asserting reject/accept per row. This is the regression net the project expects around any validation change, and it makes the guard's contract explicit.

2. (core) A whole-vector validator test. Find the per-element/whole-vector validators (rg -n "class .*Validator" src/main/java/org/opensearch/knn/index/mapper/). Pick one — e.g. the byte-range or fp16-range check — and write a unit test that feeds an out-of-range element through the document-parsing path and asserts the precise rejection. Where the validator isn't easily reachable, document the seam. This teaches the per-document parsing moment (vs the mapping-time moment your dimension guard touched), which the lab's opening Note draws a sharp line around.

3. (core) Cross-parameter validation: hamming requires binary. Implement Stretch Goal 2 properly: add a check that rejects space_type: hamming unless data_type: binary (find where SpaceType and VectorDataType meet — rg -n "SpaceType\|VectorDataType\|hamming" src/main/java/org/opensearch/knn/index/mapper/), with a field-naming error message. Write two tests: the reject path (expectThrows) and a valid binary+hamming accept path. Cross-parameter constraints are the harder, more common mapper bug class, and the distance semantics behind hamming vs l2/cosine are exactly the territory of vector-math foundations.

4. (core) A field-type variant: a read-only max_dimension introspection on the FieldType. Add a method to KNNVectorFieldType (or its KNNMappingConfig) that returns the engine-resolved maximum dimension for the field (deriving it from the chosen KNNEngine's getMaxDimension — rg -n "getMaxDimension" src/main/java/org/opensearch/knn/index/engine/KNNEngine.java). Write a test that builds a faiss field and a lucene field via TypeParser.parse and asserts the two report their engine-specific maxima — proving the per-engine truth the lab insists is the production-correct design.

5. (core) Make the ceiling per-engine and correct. Promote your LAB_MAX_DIMENSION to the real, per-engine maximum from Exercise 4: the dimension validator (or a build-time check) must reject a dimension above the chosen engine's limit and name the engine in the error. Add a test per engine (faiss/lucene) asserting each enforces its own ceiling. Run the full KNNVectorFieldMapperTests class to confirm no sibling test regressed. This is the version you would actually defend in a PR.

6. (advanced challenge) Add a brand-new mapping parameter end to end. Add an optional mapping parameter — say description (a free-text annotation) or, more ambitiously, a numeric dimension_warn_threshold that emits a deprecation-style warning (grep core for DeprecationLogger) rather than rejecting. Wire it as a Parameter<> on the Builder, add it to getParameters(), thread it into KNNVectorFieldType/KNNMappingConfig so it round-trips through toXContent (the mapping serializes back identically on a GET _mapping), and validate it. Deliverables: a KNNVectorFieldMapperTests test asserting the parameter parses, defaults correctly, and survives a parse→serialize→parse round-trip; a YAML REST test under src/yamlRestTest/... creating an index with the parameter and reading it back; and a note on whether the parameter belongs on the field type or the codec. This is the full field-mapper feature loop — the most transferable plugin skill in the ecosystem, and the foundation for the codec-side work in native SIMD and faiss kernels.

Issues to Practice On

Mapping bugs and feature requests are a steady stream in k-NN, and almost all of them route through KNNVectorFieldMapper. The repo is opensearch-project/k-NN.

Labels drift — list them and pick (gh label list --repo opensearch-project/k-NN).

Representative issue patterns. (1) "Mapping param X should be validated / gives a confusing error" — a classic well-scoped first PR. Approach: reproduce the bad mapping, find the Parameter<> and its validator via rg, add or sharpen the check + a field-naming message, add a KNNVectorFieldMapperTests reject/accept pair, PR with CHANGELOG + DCO. (2) "Support a new combination (engine × data_type × mode)" — an enhancement that lives in the cross-parameter validation and the engine-capability table. Approach: map the constraint to the existing capability checks (rg -n "validateMethod\|isTrainingRequired\|supports"), extend them, and test every (allowed, rejected) pair.

Planted-bug exercise. In the dimension Parameter<Integer> validator (find it with rg -n "greater than 0" src/main/java/org/opensearch/knn/index/mapper/KNNVectorFieldMapper.java), flip the lower-bound condition from value <= 0 to value < 0 so a dimension: 0 mapping is wrongly accepted. Rebuild, run ./gradlew test --tests "*KNNVectorFieldMapperTests", and watch which test goes red (if none does, that's a coverage gap — note it). Now fix it and add the boundary assertion from Exercise 1 that pins dimension: 0 as rejected. The lesson: an off-by-one in a validator is invisible until a boundary test exists — so write the boundary test.

Etiquette. Claim the issue before working it, reproduce the bad mapping first, and every PR needs a test, a CHANGELOG.md entry, an intact SPDX header on touched files, and a DCO sign-off (git commit -s). See community interaction before your first k-NN PR.

Validation / Self-check

1. List the knn_vector mapping parameters and, for each, name the KNNEngine behavior it drives. Which two can be written either top-level or nested under method, and why does that matter for validation?
2. What is the difference between mapping-time validation and per-document parsing in KNNVectorFieldMapper? On which node does each happen, and which one did your guard touch?
3. Walk the path from a validated mapping to a written graph: Builder → KNNVectorFieldType / KNNMappingConfig → engine → vectors format → segment files. Where does faiss diverge from lucene?
4. Why is hardcoding a single LAB_MAX_DIMENSION the wrong production design, and what is the correct source of the limit?
5. In the unit test, what does KNNVectorFieldMapper.TypeParser.parse(...) give you that a live curl mapping does not, and why is expectThrows the right assertion here?
6. How does a model_id field get its dimension, and why does your inline-dimension guard not apply to it?
7. Name one cross-parameter constraint the mapper enforces (e.g. engine vs filter, space vs data_type) and where in the source it lives.

When your guard rejects bad dimensions with a clear message, your KNNVectorFieldMapperTests tests pass, and you can defend the per-engine-correct version of the change, you understand the knn_vector field type from mapping JSON to engine behavior. Revisit k-NN architecture for how the mapper sits in the index path, and engines for the capability fork your mapping selects; the core machinery your mapper is built on is in mapping and analysis.