Vectorization and Embeddings — Intensive

The word vectorization is overloaded in OpenSearch, and the overload is not an accident — both meanings live in the same vector-search stack, one stacked on the other. This masterclass nails down both, because a contributor who only understands one of them will misdiagnose the other.

• (a) SIMD vectorization — the hardware meaning. A SIMD vector is a CPU register that holds 8 or 16 floats and operates on all of them in one instruction. This is the layer that makes a dot product fast. It is the single hottest loop in all of vector search, and OpenSearch inherits it from Lucene's VectorUtil via the Panama Vector API.
• (b) Embeddings / semantic vectors — the machine-learning meaning. An embedding vector is a float[768] that a model produced from a piece of text, such that semantically-similar text lands close together in the space. This is the layer that makes search understand meaning instead of matching keywords. It lives in two plugins — ml-commons (serves the model) and neural-search (wires the model into ingest and query) — and ultimately lands on the k-NN plugin's knn_vector field.

The connection is direct: the embeddings layer (b) produces the float[]s, and the SIMD layer (a) compares them. A neural query at the top of the stack becomes a k-NN query in the middle becomes millions of SIMD dot products at the bottom. You need both mental models to reason about a latency report ("is it the model inference, the graph walk, or the scalar fallback?") or a recall report ("is it the embedding model, the HNSW parameters, or the normalization?").

This chapter extends three existing deep-dives — read them as prerequisites:

• SIMD and the Panama Vector API — the SIMD kernel. We go deeper here on FMA, lanes, and the JIT mapping, and a full microbench lands in Lab VE4.
• The k-NN Query Path — where a dense neural query lands.
• Algorithms: HNSW, IVF, and PQ — the index structure the SIMD kernel walks.

Note on terminology: the cluster manager (formerly master) owns cluster state, including the .plugins-ml-model system index and model-deployment routing. It is off the per-query hot path: once a model is deployed onto data nodes, embedding and k-NN run shard-local, coordinated by the same coordinating node as any _search.

After this masterclass you can: explain why distance is the hot loop and write a Panama-vectorized dot product; register and deploy a model in ml-commons (local DJL/ONNX or a remote connector); build an end-to-end semantic-search pipeline with a text_embedding ingest processor and a neural query; fuse BM25 and neural scores with a hybrid query and the normalization-processor; stand up SPLADE-style neural sparse search on a rank_features field; and reason about dense-vs-sparse-vs-hybrid trade-offs.

Part A — SIMD vectorization (the hardware meaning)

The SIMD chapter introduced this. The masterclass-level summary, then the extensions.

Why vector distance is the HNSW hot loop

A dot product over d dimensions is d multiplies and d-1 adds. Now count how often HNSW runs it:

Every greedy hop on a sparse HNSW layer, every candidate in the layer-0 beam, every node inserted during a merge costs one distance computation. Profilers of vector workloads light up almost entirely on dotProduct / squareDistance / cosine. The graph-traversal bookkeeping — priority queues, visited-bitsets — is noise by comparison. Make the kernel 4× faster and you make vector search ~4× faster. That is the entire reason SIMD matters here.

A scalar Java loop processes one float per instruction. A SIMD loop processes 8 (AVX2, 256-bit) or 16 (AVX-512, 512-bit) floats per instruction. That 8–16× per-lane parallelism is the prize.

The Panama Vector API in one screen

Java historically could not emit SIMD reliably — you got whatever the JIT's auto-vectorizer managed, which for a reduction like a dot product was unreliable. Project Panama added the Vector API (jdk.incubator.vector), an incubator module that exposes SIMD as portable Java. Because it is an incubator, it is not on the module path by default; you opt in:

java --add-modules jdk.incubator.vector ...     # required to load the module

OpenSearch's bundled jvm.options already adds this on JDK 21, so a running node has it. Lucene probes for the module at startup and, if absent, transparently uses its scalar fallback rather than crashing.

The four Panama types you actually touch:

How dot-product / squared-distance / cosine vectorize

All three share the same skeleton: a lane-striding main loop that accumulates into a SIMD register with fma, a horizontal reduce at the end, and a scalar tail for the length % laneCount remainder. The differences are only in what you accumulate.

import jdk.incubator.vector.FloatVector;
import jdk.incubator.vector.VectorOperators;
import jdk.incubator.vector.VectorSpecies;

static final VectorSpecies<Float> SP = FloatVector.SPECIES_PREFERRED;

/** Dot product: sum(a[i]*b[i]) — one accumulator. */
static float dot(float[] a, float[] b) {
    FloatVector acc = FloatVector.zero(SP);
    int i = 0, bound = SP.loopBound(a.length);
    for (; i < bound; i += SP.length()) {
        FloatVector va = FloatVector.fromArray(SP, a, i);
        FloatVector vb = FloatVector.fromArray(SP, b, i);
        acc = va.fma(vb, acc);                   // acc += va * vb, lane-wise, one instruction
    }
    float sum = acc.reduceLanes(VectorOperators.ADD);
    for (; i < a.length; i++) sum += a[i] * b[i];
    return sum;
}

/** Squared L2 distance: sum((a[i]-b[i])^2) — subtract, then fma the diff with itself. */
static float squareDistance(float[] a, float[] b) {
    FloatVector acc = FloatVector.zero(SP);
    int i = 0, bound = SP.loopBound(a.length);
    for (; i < bound; i += SP.length()) {
        FloatVector va = FloatVector.fromArray(SP, a, i);
        FloatVector vb = FloatVector.fromArray(SP, b, i);
        FloatVector diff = va.sub(vb);
        acc = diff.fma(diff, acc);               // acc += diff * diff
    }
    float sum = acc.reduceLanes(VectorOperators.ADD);
    for (; i < a.length; i++) { float d = a[i] - b[i]; sum += d * d; }
    return sum;
}

/** Cosine numerator + both norms: three accumulators, one pass. */
static float cosine(float[] a, float[] b) {
    FloatVector dot = FloatVector.zero(SP), na = FloatVector.zero(SP), nb = FloatVector.zero(SP);
    int i = 0, bound = SP.loopBound(a.length);
    for (; i < bound; i += SP.length()) {
        FloatVector va = FloatVector.fromArray(SP, a, i);
        FloatVector vb = FloatVector.fromArray(SP, b, i);
        dot = va.fma(vb, dot);
        na  = va.fma(va, na);                    // |a|^2
        nb  = vb.fma(vb, nb);                    // |b|^2
    }
    float d = dot.reduceLanes(VectorOperators.ADD);
    float aa = na.reduceLanes(VectorOperators.ADD);
    float bb = nb.reduceLanes(VectorOperators.ADD);
    for (; i < a.length; i++) { d += a[i]*b[i]; aa += a[i]*a[i]; bb += b[i]*b[i]; }
    return (float) (d / Math.sqrt((double) aa * bb));
}

Two facts to internalize:

1. fma (fused multiply-add) computes a*b+c in one instruction with one rounding. It is both faster (one op instead of two) and more accurate (no intermediate rounding) than a separate multiply and add. It is the single most important primitive in these kernels — cosine above does three FMAs per lane per iteration, doing the entire numerator-and-both-norms in one pass over memory.
2. The scalar tail is mandatory. Array length is rarely an exact multiple of 8 or 16. SP.loopBound(len) gives the largest multiple of the lane count ≤ len; the tail loop mops up the remainder. Forget it and you silently drop dimensions → wrong distance → wrong neighbours.

How the JIT maps lanes to AVX / NEON

FloatVector.SPECIES_PREFERRED resolves at runtime to the host's widest shape. The HotSpot JIT then intrinsifies the Panama ops to the matching machine instructions:

The same Java source runs on all of them. This portability is exactly why a library like Lucene — which must run on x86 and ARM/Graviton — chose the Vector API over hand-written intrinsics.

flowchart TD
    SRC["one Panama source: va.fma(vb, acc)"] --> JIT["HotSpot JIT intrinsifies at runtime"]
    JIT -->|AVX-512 host| Z["vfmadd231ps on zmm (16 lanes)"]
    JIT -->|AVX2 host| Y["vfmadd231ps on ymm (8 lanes)"]
    JIT -->|ARM NEON host| N["fmla (4 lanes)"]
    JIT -->|module absent| S["scalar FP (1 lane)"]

Warning: Floating-point addition is not associative. A SIMD reduction sums in a different order than a scalar loop, so the two can produce slightly different sums. This is expected and within float tolerance — never assert exact equality between a scalar and a vectorized score in a test; assert within delta. This bites people who golden-test embedding scores.

Lucene's VectorUtil + VectorizationProvider — the selection boundary

Lucene does not sprinkle Panama calls everywhere. All vector arithmetic funnels through org.apache.lucene.util.VectorUtil (dotProduct, squareDistance, cosine, plus int8/byte variants). VectorUtil delegates to a VectorUtilSupport chosen once at class-load by a VectorizationProvider:

flowchart TD
    Start["VectorizationProvider.lookup() — once, at class init"] --> Mod{"jdk.incubator.vector present<br/>AND JDK version supported<br/>AND CPU has SIMD?"}
    Mod -->|yes| Panama["PanamaVectorizationProvider<br/>→ PanamaVectorUtilSupport (SIMD)"]
    Mod -->|no| Scalar["DefaultVectorizationProvider<br/>→ DefaultVectorUtilSupport (scalar)"]
    Panama --> VU["VectorUtil.dotProduct / squareDistance / cosine"]
    Scalar --> VU
    VU --> HNSW["HnswGraphSearcher / VectorScorer / merge"]

Because the choice is made once, at the VectorUtil boundary, there is no per-call branch in the hot loop. Every consumer — HNSW search, exact scoring, graph construction, merge — gets SIMD when available and scalar when not, for free. An OpenSearch Lucene upgrade that improves VectorUtil improves k-NN's lucene engine with zero plugin changes. Modern Lucene also reads stored vectors through MemorySegment, letting the Vector API load lanes directly from the mapped .vec region with no copy into a float[] — a meaningful win on indices that do not fit in heap.

The payoff and where it lands in OpenSearch

Faster distance helps twice: at query time it lowers per-comparison cost (lower HNSW latency); at index/merge time it speeds graph construction (which is distance-bound). A faster kernel plus an improved HNSW graph merger produced ~25% indexing speedups in Lucene's nightly benchmarks — a combination, with SIMD load-bearing. In OpenSearch:

• k-NN lucene engine uses Lucene's HNSW → Lucene's VectorUtil → Panama SIMD, provided the node runs with --add-modules jdk.incubator.vector (it does on JDK 21).
• k-NN faiss engine does not use Java SIMD; its native C++ is compiled with AVX2/AVX-512 (and NEON on ARM) kernels — same hardware idea, achieved in C++. See k-NN native integration and memory.

Lab VE4 makes this concrete: a scalar vs Panama dot product over 768-dim vectors, run with and without --add-modules, with a results table.

Part B — Embeddings and semantic vectors (the ML meaning)

Now the other "vector." Keyword search (BM25, the inverted index) matches tokens: a query for "automobile" misses a document that only says "car." Semantic search fixes that by turning text into an embedding — a dense float[] from a neural model — such that "car" and "automobile" land close in the space. Then "find documents about X" becomes "find the nearest vectors to embed(X)," i.e. a k-NN query.

Two plugins divide the work:

The dense path then lands on the k-NN plugin's knn_vector field and knn query — everything you already know from the k-NN chapters.

flowchart LR
    subgraph mlc["ml-commons (serves the model)"]
      MODEL["all-MiniLM-L6-v2 (local DJL/ONNX)<br/>or remote connector (OpenAI/Bedrock/...)"]
    end
    subgraph ns["neural-search (wires it in)"]
      TEP["text_embedding ingest processor"]
      NQ["neural query"]
      HQ["hybrid query + normalization-processor"]
      NS["neural_sparse query"]
    end
    subgraph knn["k-NN (stores & searches dense)"]
      KV["knn_vector field + HNSW graph"]
    end
    TEP -->|embeds at index time| KV
    NQ -->|embeds at query time| KV
    MODEL -.model_id.-> TEP
    MODEL -.model_id.-> NQ
    MODEL -.model_id.-> NS
    NS -->|token:weight| RF["rank_features field (inverted index)"]

ml-commons: model serving

ml-commons distinguishes local models (run in-process on the node, via DJL with an ONNX Runtime / PyTorch backend) from remote models (a connector calls out to an external inference service).

Local model — register + deploy. The classic dense encoder is huggingface/sentence-transformers/all-MiniLM-L6-v2 (384 dimensions, mean-pooled). You register it into a model group, then deploy it onto nodes:

# 1. Create a model group (a permissions/versioning container).
curl -XPOST 'localhost:9200/_plugins/_ml/model_groups/_register' \
  -H 'Content-Type: application/json' -d '{
  "name": "semantic_search_models",
  "description": "dense encoders for semantic search"
}'
# -> { "model_group_id": "<MG_ID>", "status": "CREATED" }

# 2. Register a pretrained local model into that group (async -> returns a task_id).
curl -XPOST 'localhost:9200/_plugins/_ml/models/_register' \
  -H 'Content-Type: application/json' -d '{
  "name": "huggingface/sentence-transformers/all-MiniLM-L6-v2",
  "version": "1.0.1",
  "model_group_id": "<MG_ID>",
  "model_format": "TORCH_SCRIPT"
}'
# -> { "task_id": "<TASK_ID>", "status": "CREATED" }

# 3. Poll the task until state == COMPLETED; it carries the model_id.
curl -s 'localhost:9200/_plugins/_ml/tasks/<TASK_ID>?pretty'
# -> { "model_id": "<MODEL_ID>", "state": "COMPLETED", ... }

# 4. Deploy the model onto data nodes (loads it into memory; async task again).
curl -XPOST 'localhost:9200/_plugins/_ml/models/<MODEL_ID>/_deploy'
curl -s 'localhost:9200/_plugins/_ml/models/<MODEL_ID>?pretty'   # model_state: DEPLOYED

Note: Registering pretrained models from the public URL requires plugins.ml_commons.allow_registering_model_via_url: true (or the model-via-local options) in cluster settings on many setups, and on a non-dedicated-ML-node cluster you may need plugins.ml_commons.only_run_on_ml_node: false. Set these via PUT /_cluster/settings before step 2 if registration is rejected.

Remote model — connector. Instead of running inference in-process, a connector POSTs to an external service (OpenAI, Amazon Bedrock, SageMaker, Cohere). You create a connector describing the endpoint and request/response mapping, then register a model that uses it:

# Create a connector to a remote embedding endpoint (Bedrock Titan shown).
curl -XPOST 'localhost:9200/_plugins/_ml/connectors/_create' \
  -H 'Content-Type: application/json' -d '{
  "name": "Bedrock Titan Embeddings",
  "description": "remote embedding connector",
  "version": 1,
  "protocol": "aws_sigv4",
  "parameters": { "region": "us-east-1", "service_name": "bedrock" },
  "credential": { "access_key": "...", "secret_key": "..." },
  "actions": [{
    "action_type": "predict",
    "method": "POST",
    "url": "https://bedrock-runtime.us-east-1.amazonaws.com/model/amazon.titan-embed-text-v1/invoke",
    "request_body": "{ \"inputText\": \"${parameters.input}\" }",
    "pre_process_function": "connector.pre_process.bedrock.embedding",
    "post_process_function": "connector.post_process.bedrock.embedding"
  }]
}'
# -> { "connector_id": "<CONNECTOR_ID>" }

# Register + deploy a model bound to that connector.
curl -XPOST 'localhost:9200/_plugins/_ml/models/_register?deploy=true' \
  -H 'Content-Type: application/json' -d '{
  "name": "bedrock-titan-embed",
  "function_name": "remote",
  "model_group_id": "<MG_ID>",
  "connector_id": "<CONNECTOR_ID>"
}'

Either way you end up with a model_id. The model and its metadata live in the .plugins-ml-model system index (chunked into the .plugins-ml-model-* indices for large artifacts); deployment routing and model-group ACLs are cluster-state coordinated by the cluster manager. You can sanity-check inference directly:

curl -XPOST 'localhost:9200/_plugins/_ml/_predict/text_embedding/<MODEL_ID>' \
  -H 'Content-Type: application/json' -d '{ "text_docs": ["a fast red car"] }'
# -> inference_results[0].output[0].data : the 384-float embedding

neural-search: the four building blocks

1. The text_embedding ingest processor turns text into a vector at index time. You put it in an ingest pipeline, point it at a model_id, and map source text fields to destination knn_vector fields. Every document that flows through the pipeline gets its embedding computed and stored automatically — you never send vectors over the wire.

PUT /_ingest/pipeline/nlp-ingest-pipeline
{
  "description": "embed `text` into `text_embedding` (knn_vector)",
  "processors": [
    { "text_embedding": {
        "model_id": "<MODEL_ID>",
        "field_map": { "text": "text_embedding" }
    }}
  ]
}

2. The neural query embeds the query text at search time, then runs k-NN under the hood. You give it the text, the field, the model_id, and k; neural-search calls ml-commons to embed, then builds a knn query against the knn_vector field:

{ "query": { "neural": {
  "text_embedding": {
    "query_text": "fast red sports car",
    "model_id": "<MODEL_ID>",
    "k": 10
  }
}}}

This is where Part B hands off to the k-NN chapters: the neural query becomes a KNNQueryBuilder → KNNQuery → per-segment HNSW search → SIMD distance. The whole k-NN query path runs underneath, and the HNSW parameters (ef_search, m) tune it.

3. The hybrid query + normalization-processor fuse BM25 and neural. BM25 scores and cosine/L2 neural scores live on incomparable scales (BM25 is unbounded and corpus- dependent; cosine is roughly [0,1]) — you cannot just add them. A search pipeline with the normalization-processor normalizes each clause's scores (min-max or L2) and then combines them (arithmetic / harmonic / geometric mean, with optional weights):

PUT /_search/pipeline/nlp-search-pipeline
{
  "phase_results_processors": [
    { "normalization-processor": {
        "normalization": { "technique": "min_max" },
        "combination": {
          "technique": "arithmetic_mean",
          "parameters": { "weights": [0.3, 0.7] }
        }
    }}
  ]
}

The full mechanics — why normalization is mandatory, how to tune weights — are Lab VE2.

4. Neural sparse (2.13+) is the dark-horse third option. Instead of a dense float[768], a SPLADE-style encoder produces a sparse vector: a small set of token:weight pairs (the model expands the text into related vocabulary terms with learned weights). Those pairs are stored in a rank_features field, which rides the inverted index — so a neural-sparse query is scored by the same postings machinery as BM25 and is roughly as efficient as BM25, while capturing semantics. The full setup is Lab VE3.

Dense vs sparse vs hybrid — the trade-off table

¹ doc-only neural-sparse mode skips query-side inference (see below), trading a little recall for BM25-equivalent query cost.

The practical rule: dense wins on paraphrase/cross-lingual recall; sparse wins on rare/exact terms and memory; hybrid (dense or sparse + BM25) wins overall on most benchmarks because it covers each method's blind spots. Start with hybrid if you can afford it.

The end-to-end semantic-search pipeline

Putting ingest and query together for the dense path:

flowchart TD
    subgraph ingest["Index time"]
      D["incoming doc { text: '...' }"] --> P["ingest pipeline:<br/>text_embedding processor"]
      P -->|ml-commons _predict| EMB["embed → float[384]"]
      EMB --> STORE["store text + text_embedding (knn_vector)<br/>HNSW graph updated"]
    end
    subgraph query["Query time"]
      QT["neural query: query_text + model_id + k"] --> QE["embed query → float[384]<br/>(ml-commons)"]
      QE --> KNN["knn query on text_embedding<br/>per-segment HNSW walk (SIMD distance)"]
      KNN --> RED["coordinator reduce → global top-k"]
      RED --> FETCH["fetch _source → results"]
    end

Index: doc → ingest pipeline embeds text → store text + knn_vector (graph updated). Query: neural query embeds the query text → k-NN over the knn_vector field → reduce → fetch. The model serves both halves; the same model_id must be used at ingest and query, or the query embedding lands in a different space than the stored embeddings and recall collapses. Lab VE1 builds exactly this.

RAG via ml-commons (brief)

Retrieval-Augmented Generation reuses everything above and adds a generation step. A search pipeline response processor — the retrieval_augmented_generation (RAG) processor — takes the retrieved hits, stuffs them into a prompt template, and calls a remote LLM (registered in ml-commons as a remote model via a connector, exactly like the embedding connector above) to synthesize an answer with citations. The retrieval half is the semantic/hybrid pipeline you already built; the generation half is one more remote model plus a generative_qa search-pipeline processor. ml-commons also ships agents and memory (conversation history) for multi-turn RAG. The contributor takeaway: RAG is not a new subsystem — it is retrieval (this masterclass) + one remote LLM model + a response processor.

Provider note: when the remote LLM behind RAG is Anthropic Claude, you wire it as a standard ml-commons remote connector (Bedrock anthropic.claude-* or the Anthropic Messages API) — the same connector shape as the embedding example, with a different url, request_body, and post-process function. The retrieval pipeline is unchanged.

Common bugs and symptoms

Validation: prove you understand this

1. State the two meanings of "vectorization" in OpenSearch and how they connect in a single neural query (trace it from REST to AVX instruction).
2. Why is vector distance the HNSW hot loop? Quantify roughly how many distance computations one query vs one 1M-vector merge runs, and why that makes the kernel high-leverage.
3. Write, from memory, the Panama dot-product skeleton: the lane-striding main loop, the fma, the reduce, and the scalar tail. Why is each piece necessary?
4. Trace how VectorUtil.dotProduct chooses SIMD vs scalar. When is the decision made, and why is there no per-call branch in the hot loop?
5. Walk the ml-commons local-model flow: register → poll task → deploy → predict. Where does the serialized model live, and which node coordinates deployment?
6. Explain the dense semantic-search pipeline end to end: which processor embeds at index time, which query embeds at search time, what the neural query becomes underneath, and why the same model_id must be used in both halves.
7. Why can't you simply add a BM25 score and a cosine score? What does the normalization-processor do, and what are the three combination techniques?
8. Contrast dense, sparse (neural-sparse / SPLADE), and lexical (BM25) search on representation, index structure, semantics, latency, and memory. When is each best?
9. Give two independent ways to prove SIMD is actually active on a running OpenSearch node, and three ways it can be silently disabled.

When you can do all nine, you own both meanings of vectorization. Now make it concrete: Lab VE1 builds semantic search end to end, Lab VE2 fuses it with BM25, Lab VE3 does the sparse variant, and Lab VE4 proves the SIMD speedup at the bottom of the stack. For where the dense path lands, return to the k-NN query path and algorithms; for the kernel itself, re-read SIMD and the Panama Vector API.