Lab VE1: Semantic Search End-to-End

You will stand up a complete dense semantic-search pipeline on a local cluster, with no vectors ever leaving your terminal as raw floats. You register and deploy an embedding model in ml-commons, create an ingest pipeline whose text_embedding processor embeds text into a knn_vector field at index time, index a small corpus, then run a neural query and compare its ranking against a plain BM25 match. By the end you will have seen — and verified — every hop from text to stored vector to nearest-neighbour result.

This is the concrete build of the dense pipeline diagrammed in the masterclass index. It sits on top of the k-NN query path (which the neural query becomes) and HNSW algorithms (the index it walks).

Background

Keyword search matches tokens; semantic search matches meaning. The trick is to precompute an embedding for every document and, at query time, embed the query and find the nearest document vectors. OpenSearch does both halves for you: the text_embedding ingest processor calls ml-commons to embed documents as they index, and the neural query calls ml-commons to embed the query text and then runs a k-NN search. Your job is to wire the three plugins — ml-commons (serves the model), neural-search (the processor and query), k-NN (the knn_vector field) — into one working flow.

Why This Matters for Contributors

• Semantic search is the headline use case for OpenSearch's vector stack; bugs reported against it usually turn out to be in one specific hop (model not deployed, dimension mismatch, wrong model_id at query time). Knowing the hops lets you triage in minutes.
• The neural query is a thin wrapper that becomes a KNNQueryBuilder. Once you have run this lab you can connect a user's "my neural query is slow/empty" to the exact k-NN internals in the query path chapter.
• You will learn to inspect the generated vectors, which is the single most useful debugging skill for this subsystem.

Prerequisites

• A local OpenSearch 2.13+ (or 3.x) single-node cluster with the opensearch-knn, opensearch-ml, and neural-search plugins (all bundled in the default distribution). bin/opensearch running, REST on localhost:9200.
• curl and (optionally) jq for reading JSON.
• You have read the masterclass index, Part B.
• ~2 GB free RAM for the local model (MiniLM is small; this is comfortable).

Note: This lab uses a local model (all-MiniLM-L6-v2, 384 dims) so you need no external API keys. The remote-connector path from the index is identical from the ingest/query side — only the _register body differs.

Step-by-Step Tasks

Step 1 — Relax ML settings for a single-node dev cluster

On a one-node dev box, ML models would otherwise refuse to run (they want a dedicated ML node) and URL registration is off by default. Enable both:

curl -XPUT 'localhost:9200/_cluster/settings' -H 'Content-Type: application/json' -d '{
  "persistent": {
    "plugins.ml_commons.only_run_on_ml_node": false,
    "plugins.ml_commons.allow_registering_model_via_url": true,
    "plugins.ml_commons.native_memory_threshold": 99
  }
}'

Expected: {"acknowledged":true,"persistent":{"plugins":{"ml_commons":{...}}}}.

Warning: These are dev settings. In production you run models on dedicated ML nodes and do not blanket-allow URL registration. We relax them only so a laptop works.

Step 2 — Register a model group, then the model

A model group is a versioning/ACL container. Create one, capture its id:

curl -XPOST 'localhost:9200/_plugins/_ml/model_groups/_register' \
  -H 'Content-Type: application/json' -d '{
  "name": "semantic_search_models",
  "description": "dense encoders for VE labs"
}'

Expected response:

{ "model_group_id": "Z1xQk4cB...", "status": "CREATED" }

Now register the pretrained MiniLM encoder into that group (substitute your model_group_id). Registration is asynchronous — it returns a task_id:

curl -XPOST 'localhost:9200/_plugins/_ml/models/_register' \
  -H 'Content-Type: application/json' -d '{
  "name": "huggingface/sentence-transformers/all-MiniLM-L6-v2",
  "version": "1.0.1",
  "model_group_id": "Z1xQk4cB...",
  "model_format": "TORCH_SCRIPT"
}'

Expected:

{ "task_id": "aF2Rk4cB...", "status": "CREATED" }

Step 3 — Poll the register task, capture the model_id

curl -s 'localhost:9200/_plugins/_ml/tasks/aF2Rk4cB...?pretty'

While downloading you will see "state": "RUNNING". When done:

{
  "model_id": "bG3Sk4cB...",
  "task_type": "REGISTER_MODEL",
  "state": "COMPLETED",
  "worker_node": ["..."]
}

Capture model_id — call it <MODEL_ID> for the rest of the lab. If state is FAILED, read the error field (usually a settings or network issue → revisit Step 1).

Step 4 — Deploy the model onto the node

Registration stored the model; deploy loads it into memory so it can run inference:

curl -XPOST 'localhost:9200/_plugins/_ml/models/<MODEL_ID>/_deploy'
# -> { "task_id": "...", "status": "CREATED" }  (async again)

Poll the model's state until DEPLOYED:

curl -s 'localhost:9200/_plugins/_ml/models/<MODEL_ID>?pretty'

{
  "model_id": "bG3Sk4cB...",
  "model_state": "DEPLOYED",
  "model_config": { "embedding_dimension": 384, "framework_type": "SENTENCE_TRANSFORMERS" },
  "model_format": "TORCH_SCRIPT"
}

Note embedding_dimension: 384 — you will need it for the mapping in Step 6. A mismatch here is the #1 bug in this lab.

Step 5 — Smoke-test inference directly

Before wiring it into ingest, confirm the model embeds text. This also shows you what a raw embedding looks like:

curl -XPOST 'localhost:9200/_plugins/_ml/_predict/text_embedding/<MODEL_ID>' \
  -H 'Content-Type: application/json' -d '{ "text_docs": ["a fast red sports car"] }'

Expected (truncated):

{
  "inference_results": [{
    "output": [{
      "name": "sentence_embedding",
      "data_type": "FLOAT32",
      "shape": [384],
      "data": [0.0421, -0.0117, 0.0683, /* ... 384 floats ... */ -0.0094]
    }]
  }]
}

shape: [384] confirms the dimension. If this fails, nothing downstream will work — fix it here.

Step 6 — Create the ingest pipeline with a text_embedding processor

The processor embeds a source text field into a destination knn_vector field for every document that flows through it:

curl -XPUT 'localhost:9200/_ingest/pipeline/nlp-ingest-pipeline' \
  -H 'Content-Type: application/json' -d '{
  "description": "embed `text` into `text_embedding` knn_vector",
  "processors": [
    { "text_embedding": {
        "model_id": "<MODEL_ID>",
        "field_map": { "text": "text_embedding" }
    }}
  ]
}'

field_map reads source_field → writes destination_field. Expected: {"acknowledged":true}.

Step 7 — Create the index: knn_vector mapping + default pipeline

The index must (a) enable k-NN, (b) map text_embedding as a knn_vector of dimension 384 with an HNSW method, and (c) use the ingest pipeline by default so you never have to name it per-request:

curl -XPUT 'localhost:9200/semantic-demo' -H 'Content-Type: application/json' -d '{
  "settings": {
    "index.knn": true,
    "default_pipeline": "nlp-ingest-pipeline"
  },
  "mappings": {
    "properties": {
      "text":           { "type": "text" },
      "text_embedding": {
        "type": "knn_vector",
        "dimension": 384,
        "space_type": "l2",
        "method": {
          "name": "hnsw",
          "engine": "lucene",
          "parameters": { "ef_construction": 128, "m": 16 }
        }
      }
    }
  }
}'

We use engine: lucene so the dense distance runs through Lucene's VectorUtil → Panama SIMD (the SIMD chapter). faiss works too; see engines. Expected: {"acknowledged":true,...}.

Warning: dimension must equal the model's embedding_dimension (384 here). If you copy a 768-dim mapping by habit, every index request fails with a dimension mismatch. This is the most common failure in the lab — see Troubleshooting.

Step 8 — Index a small corpus (embeddings auto-generated)

Bulk-index documents with only text — the pipeline embeds them. The corpus is chosen so semantic vs keyword differences show up:

curl -XPOST 'localhost:9200/semantic-demo/_bulk' \
  -H 'Content-Type: application/json' -d '
{"index":{"_id":"1"}}
{"text":"A fast red sports car speeding down the highway"}
{"index":{"_id":"2"}}
{"text":"The automobile industry is shifting to electric vehicles"}
{"index":{"_id":"3"}}
{"text":"A delicious recipe for homemade margherita pizza"}
{"index":{"_id":"4"}}
{"text":"Quarterly earnings report shows strong revenue growth"}
{"index":{"_id":"5"}}
{"text":"Jaguars and cheetahs are the fastest land predators"}
{"index":{"_id":"6"}}
{"text":"Tips for tuning the engine of your motor vehicle"}
'
curl -XPOST 'localhost:9200/semantic-demo/_refresh'

Expected: a bulk response with "errors": false. If errors is true, inspect the per-item error.reason (almost always a dimension or model-deploy issue).

Step 9 — Verify the vectors were actually generated

Confirm the processor ran by reading one document's _source — it should now contain a 384-float text_embedding you never sent:

curl -s 'localhost:9200/semantic-demo/_doc/1?pretty&_source_includes=text,text_embedding' \
  | head -c 600

Expected (truncated): the original text plus a text_embedding array of 384 floats. To check the stored vector dimension explicitly without dumping 384 numbers:

# field caps confirm it's a knn_vector; doc count confirms ingestion
curl -s 'localhost:9200/semantic-demo/_field_caps?fields=text_embedding&pretty'
curl -s 'localhost:9200/semantic-demo/_count?pretty'   # -> { "count": 6, ... }

If text_embedding is missing from _source, the pipeline did not run — check that default_pipeline is set on the index (GET /semantic-demo/_settings).

Step 10 — Run a neural query

Now the payoff. Query by meaning. Ask for "automobile" — note the corpus says "car," "vehicle," "automobile," never the exact same word across all docs:

curl -XPOST 'localhost:9200/semantic-demo/_search?pretty' \
  -H 'Content-Type: application/json' -d '{
  "size": 4,
  "_source": ["text"],
  "query": {
    "neural": {
      "text_embedding": {
        "query_text": "automobile",
        "model_id": "<MODEL_ID>",
        "k": 4
      }
    }
  }
}'

Expected top hits (scores will vary slightly by model build, but the ordering is the point): the car/vehicle docs (1, 2, 6) rank highest, including doc 6 ("motor vehicle") and doc 2 ("automobile industry") even though "automobile" the exact token does not appear in doc 1 or 6. The pizza (3) and earnings (4) docs rank far lower.

{
  "hits": {
    "max_score": 0.73,
    "hits": [
      { "_id": "2", "_score": 0.71, "_source": { "text": "The automobile industry ..." } },
      { "_id": "1", "_score": 0.64, "_source": { "text": "A fast red sports car ..." } },
      { "_id": "6", "_score": 0.58, "_source": { "text": "Tips for tuning the engine ..." } },
      { "_id": "5", "_score": 0.41, "_source": { "text": "Jaguars and cheetahs ..." } }
    ]
  }
}

Step 11 — Compare against a plain BM25 match

Run the same query text through keyword search:

curl -XPOST 'localhost:9200/semantic-demo/_search?pretty' \
  -H 'Content-Type: application/json' -d '{
  "size": 4,
  "_source": ["text"],
  "query": { "match": { "text": "automobile" } }
}'

Expected: only doc 2 matches (it is the only one containing the literal token "automobile"); everything else scores 0 and is not returned. This is the whole point:

You have now demonstrated semantic recall that keyword search cannot achieve. The two are complementary — fusing them is Lab VE2.

Step 12 — Confirm the SIMD path underneath (optional but instructive)

The neural query became a Lucene HNSW search whose distances ran through VectorUtil. Confirm the node has the SIMD module loaded:

ps -ef | grep -i opensearch | grep -o "add-modules [^ ]*"
# expect: add-modules jdk.incubator.vector   (OpenSearch jvm.options enables it on JDK 21)

If it is present, your 384-dim distances were computed with PanamaVectorUtilSupport. To benchmark the difference yourself, do Lab VE4.

Deliverables

• A deployed ml-commons model (model_state: DEPLOYED, embedding_dimension: 384).
• An ingest pipeline nlp-ingest-pipeline with a text_embedding processor.
• An index semantic-demo with a knn_vector field + default_pipeline, holding 6 docs whose embeddings were auto-generated.
• A neural query result and a match query result on the same text, with a written explanation of why they differ.

Troubleshooting

Expected Output

A neural search for "automobile" returns the car/vehicle/engine documents ranked by semantic similarity (docs 2, 1, 6 at top), while the equivalent BM25 match returns only the single doc containing the literal token. The index holds 6 documents each carrying an auto-generated 384-float text_embedding.

Stretch Goals

• Query for "jungle predator" and watch doc 5 (jaguars/cheetahs) surface — a document that shares no query token. Pure semantic recall.
• Add "k": 6 and "size": 6; confirm all docs return ranked, including the unrelated pizza/earnings docs at the bottom (k-NN always returns something).
• Re-create the index with engine: faiss and re-run; results should match, and you can compare latency (took). Read engines on the difference.
• Register a second model at a different dimension (e.g. a 768-dim encoder), reuse the 384-dim mapping, and observe the exact dimension-mismatch failure — then fix it. This builds the muscle memory for the #1 bug.
• Inspect the generated HNSW vector files on disk: find the shard dir and list the .vec/.vex/.vem files (see Lucene HNSW files).

Coding Exercises

So far you have run the pipeline with curl. These exercises make you write code that drives, indexes, and grades it — the harnesses you would actually use to evaluate or debug a semantic-search change. Use Python 3 with requests (or opensearch-py); the cluster from the steps above is your fixture.

1. (warm-up) An encode-and-inspect script. Write encode.py that POSTs to _plugins/_ml/_predict/text_embedding/<MODEL_ID> for a list of texts, parses the JSON, and asserts every returned vector has length 384 and a finite L2 norm. Print the norm and the first 5 components. This is the single most useful debugging tool for this subsystem — it isolates the model hop from everything downstream. The norm/inner-product math behind it is in vector-math-foundations.md.

2. (warm-up) A bulk-indexer that verifies embeddings landed. Write index_corpus.py that bulk-indexes a list of texts (text only) into semantic-demo, refreshes, then reads back _doc/<id>?_source_includes=text_embedding for each and asserts the 384-float array is present and non-zero. Fail loudly with the doc id if any embedding is missing — exactly the "pipeline didn't run" bug from Step 9.

3. (core) A recall@k evaluation harness. Write eval_recall.py that takes a small labeled set (5–8 queries, each with a hand-marked "relevant" doc id), runs each as a neural query, and computes recall@k and mean reciprocal rank (MRR) over the set. Print a table. Run it once with k=3 and once with k=6 and show recall rising with k. This is the harness every relevance change is judged by.

4. (core) A cosine-vs-L2 contrast. Recreate the index twice, once with space_type: cosinesimil and once with l2, indexing the same corpus through the same model. Write space_compare.py that runs the same query set against both and reports where the rankings diverge. In your write-up, reduce the result to the math: for unit-normalized vectors, cosine and (negated) L2 induce the same ordering — confirm or refute on your data. See vector-math-foundations.md.

5. (advanced) Advanced challenge — reproduce, then assert, the #1 dimension-mismatch bug, in code. Write dim_guard.py that (a) reads the model's embedding_dimension from _plugins/_ml/models/<MODEL_ID>, (b) reads the index's mapped knn_vector dimension from _mapping, (c) asserts they are equal before any indexing, and (d) if you deliberately create a mismatched index (e.g. map 768 against a 384 model), captures the exact bulk-error reason string and asserts it contains dimension. Deliverable: a script that turns the most common production failure into a pre-flight check + a regression assertion — the kind of guard you would propose adding to a tool or test. Bonus: confirm the distance ran on SIMD by asserting jdk.incubator.vector is on the node's command line (parse ps output), tying to native-simd-and-faiss-kernels.md and Lab VE4.

Issues to Practice On

This pipeline spans three plugins; the bugs you will fix live mostly in opensearch-project/neural-search (the text_embedding processor and neural query) and opensearch-project/k-NN (the knn_vector field, HNSW). OpenSearch workflow: GitHub issues + PRs, a CHANGELOG.md entry, and a DCO Signed-off-by (git commit -s). Find work with:

gh label list --repo opensearch-project/neural-search   # confirm taxonomy (labels move; check the tracker)
gh issue list --repo opensearch-project/neural-search --label "good first issue" --state open
gh issue list --repo opensearch-project/neural-search --search "text_embedding OR neural query in:title,body" --state open
gh issue list --repo opensearch-project/k-NN --label "good first issue" --state open
gh issue list --repo opensearch-project/k-NN --label "bug" --state open

Representative issue patterns:

• Processor/field-map edge cases. "text_embedding skips a nested field," "empty text crashes the processor." Approach: reproduce with a minimal _bulk, locate the processor via rg "class TextEmbeddingProcessor\|field_map\|fieldMap" in neural-search, add an integration test (OpenSearchIntegTestCase-style), fix, PR with CHANGELOG + DCO.
• Dimension/validation messages (k-NN). "Dimension-mismatch error is unclear." Approach: reproduce, locate the validation via rg "dimension" src/main/java/.../mapper in k-NN, improve the message + add a test.

Planted-bug drill. Map semantic-demo with dimension: 768 against the 384-dim model and bulk-index. The bulk response has errors:true with a per-item dimension-mismatch reason. Capture it in dim_guard.py (exercise 5) as an assert "dimension" in reason, then fix the mapping to 384 and confirm the assertion flips to a clean index. You have now built the regression check that would have caught the most common failure in this lab.

Etiquette: claim the issue first, reproduce before coding; every neural-search / k-NN PR needs a test + a CHANGELOG.md entry + DCO Signed-off-by (git commit -s). See community-interaction.md.

Validation: Self-check

1. List every hop a document takes from _bulk body to a stored knn_vector. Which plugin owns each hop?
2. List every hop a neural query takes from REST to ranked hits. At which hop does it become a k-NN query, and what class is it then? (See query path.)
3. Why must the mapping dimension equal the model's embedding_dimension? What is the exact symptom when they disagree, and at which step does it surface?
4. Why must the same model_id be used at ingest and query time? What goes wrong silently if they differ?
5. Explain, using the "automobile" example, why neural returns documents that match cannot — in terms of embedding-space geometry.
6. Where does the serialized model live, and which node coordinates its deployment? (See the index.)
7. Confirm, two ways, that the dense distances underneath ran on SIMD rather than the scalar fallback.

When you can answer all seven and reproduce the two contrasting result sets, you have built and understood a dense semantic-search pipeline end to end. Next, fuse it with BM25 in Lab VE2: Hybrid Search.