Lab VE3: Neural Sparse Search

Dense embeddings (Lab VE1) capture semantics but cost a full HNSW graph in RAM and a SIMD distance per comparison. Neural sparse search (OpenSearch 2.13+) takes a different route: a SPLADE-style encoder turns text into a sparse vector — a handful of token:weight pairs, where the model expands the text into related vocabulary terms with learned importance weights. Those pairs are stored in a rank_features field, which rides the inverted index — so a neural-sparse query is scored by the same postings machinery as BM25 and is roughly as efficient as BM25, while still capturing learned semantics. This lab stands one up and compares it against the dense k-NN path.

This builds on Lab VE1 (model serving) and the neural-sparse description in the masterclass index.

Background

A dense encoder maps text to float[384]; a sparse encoder maps text to something like { "car": 2.1, "vehicle": 1.7, "automobile": 1.3, "fast": 0.9, "drive": 0.6 } — a sparse map over the model's vocabulary, with most entries zero (hence "sparse"). The key idea is term expansion: the input "a fast red car" produces weights not just for the literal tokens but for related terms the model learned are relevant ("vehicle", "automobile", "drive"). Stored in a rank_features field, these become postings; a query encoded the same way is scored by dot-producting the query's token weights against the documents' — which the inverted index does natively and fast. You get semantic term-matching with BM25-class cost and memory.

Why This Matters for Contributors

• Neural sparse is the answer to "semantic search but I can't afford the HNSW RAM." It is a distinct retrieval mode with its own ingest processor, query type, and field type — and a growing share of support questions. Knowing it rounds out the trade-off space.
• It demonstrates that "semantic" does not require dense vectors or k-NN at all: the same inverted index that serves BM25 (inverted index chapter) serves learned sparse vectors. That is a clarifying architectural insight.
• The doc-only vs bi-encoder distinction is a real production tuning lever (query-time inference cost vs recall) you will be asked about.

Prerequisites

• A local OpenSearch 2.13+ with opensearch-ml and neural-search (bundled).
• The dev ML settings from Lab VE1, Step 1 applied (URL registration on, single-node ML allowed).
• curl and (optionally) jq.
• Read the dense-vs-sparse-vs-hybrid table.

Note: Neural sparse search was added in OpenSearch 2.13. Earlier versions lack the neural_sparse query and the sparse encoding models. Check GET / for your version before starting.

Step-by-Step Tasks

Step 1 — Understand SPLADE-style term expansion

SPLADE ("Sparse Lexical and Expansion model") is a transformer fine-tuned to output a weight for every token in the BERT vocabulary, most of them zero. The non-zero entries include the input's own tokens and semantically-related tokens the model learned to "expand" into. Concretely, encoding "a fast red car" might yield:

input:  "a fast red car"
sparse: { car:2.4, vehicle:1.6, fast:1.5, automobile:1.1, speed:0.9, red:0.8, drive:0.5, ... }
                    ^^^^^^^         ^^^^^^^^^^         ^^^^^   ^^^^^   <- EXPANSION terms (not in the input)

Because the output is a token→weight map, it slots straight into an inverted index: each non-zero token becomes a posting with its weight. A query encoded the same way is scored by summing query_weight[t] × doc_weight[t] over shared tokens t — a sparse dot product the postings engine computes natively.

Step 2 — Register and deploy a sparse encoding model

OpenSearch ships pretrained sparse encoders, e.g. amazon/neural-sparse/opensearch-neural-sparse-encoding-v1 (a bi-encoder: it encodes both documents and queries). Register and deploy it exactly like the dense model in VE1:

curl -XPOST 'localhost:9200/_plugins/_ml/models/_register?deploy=true' \
  -H 'Content-Type: application/json' -d '{
  "name": "amazon/neural-sparse/opensearch-neural-sparse-encoding-v1",
  "version": "1.0.1",
  "model_format": "TORCH_SCRIPT"
}'
# -> { "task_id": "...", "status": "CREATED" }   (?deploy=true registers AND deploys)

Poll the task, capture the model_id, confirm model_state: DEPLOYED:

curl -s 'localhost:9200/_plugins/_ml/tasks/<TASK_ID>?pretty'    # -> model_id when COMPLETED
curl -s 'localhost:9200/_plugins/_ml/models/<MODEL_ID>?pretty'  # -> model_state: DEPLOYED

Call the result <SPARSE_MODEL_ID>.

Step 3 — Smoke-test the sparse encoding

Run inference directly to see the token:weight output — this is the most instructive moment in the lab:

curl -XPOST 'localhost:9200/_plugins/_ml/_predict/sparse_encoding/<SPARSE_MODEL_ID>' \
  -H 'Content-Type: application/json' -d '{ "text_docs": ["a fast red sports car"] }'

Expected (truncated) — note the expansion tokens that were not in the input:

{
  "inference_results": [{
    "output": [{
      "name": "output",
      "dataAsMap": {
        "response": [{
          "car": 2.41, "vehicle": 1.58, "fast": 1.49, "sports": 1.30,
          "automobile": 1.05, "speed": 0.92, "red": 0.81, "drive": 0.47
        }]
      }
    }]
  }]
}

Most of the BERT vocabulary is absent (weight 0) — that is the sparsity. The presence of "vehicle"/"automobile"/"drive", none of which appear literally in the input, is the learned expansion.

Step 4 — Create the ingest pipeline with a sparse_encoding processor

The neural-search sparse_encoding ingest processor is the sparse analogue of text_embedding: it encodes a text field into token:weight pairs in a destination field:

curl -XPUT 'localhost:9200/_ingest/pipeline/sparse-ingest-pipeline' \
  -H 'Content-Type: application/json' -d '{
  "description": "encode `text` into sparse `text_sparse` (rank_features)",
  "processors": [
    { "sparse_encoding": {
        "model_id": "<SPARSE_MODEL_ID>",
        "field_map": { "text": "text_sparse" }
    }}
  ]
}'

Expected: {"acknowledged":true}.

Step 5 — Create the index with a rank_features field

The destination field must be typed rank_features (not knn_vector, not text). rank_features is a Lucene field type that stores a map of feature→weight and scores them via the inverted index — no graph, no k-NN:

curl -XPUT 'localhost:9200/sparse-demo' -H 'Content-Type: application/json' -d '{
  "settings": { "default_pipeline": "sparse-ingest-pipeline" },
  "mappings": {
    "properties": {
      "text":        { "type": "text" },
      "text_sparse": { "type": "rank_features" }
    }
  }
}'

Note: no index.knn, no dimension, no HNSW method. Sparse rides the inverted index, so none of the dense machinery is involved. Expected: {"acknowledged":true,...}.

Step 6 — Index the corpus (sparse vectors auto-generated)

Reuse the VE1 corpus so you can compare across labs:

curl -XPOST 'localhost:9200/sparse-demo/_bulk' \
  -H 'Content-Type: application/json' -d '
{"index":{"_id":"1"}}
{"text":"A fast red sports car speeding down the highway"}
{"index":{"_id":"2"}}
{"text":"The automobile industry is shifting to electric vehicles"}
{"index":{"_id":"3"}}
{"text":"A delicious recipe for homemade margherita pizza"}
{"index":{"_id":"4"}}
{"text":"Quarterly earnings report shows strong revenue growth"}
{"index":{"_id":"5"}}
{"text":"Jaguars and cheetahs are the fastest land predators"}
{"index":{"_id":"6"}}
{"text":"Tips for tuning the engine of your motor vehicle"}
'
curl -XPOST 'localhost:9200/sparse-demo/_refresh'

Confirm the sparse field was generated:

curl -s 'localhost:9200/sparse-demo/_doc/1?pretty&_source_includes=text_sparse' | head -c 400
# -> "text_sparse": { "car": 2.41, "vehicle": 1.58, "fast": 1.49, ... }  token:weight pairs

Step 7 — Run a neural_sparse query

The neural_sparse query encodes the query text into the same sparse space and scores documents by the sparse dot product over shared tokens:

curl -XPOST 'localhost:9200/sparse-demo/_search?pretty' \
  -H 'Content-Type: application/json' -d '{
  "size": 4,
  "_source": ["text"],
  "query": {
    "neural_sparse": {
      "text_sparse": {
        "query_text": "automobile",
        "model_id": "<SPARSE_MODEL_ID>"
      }
    }
  }
}'

Expected: like the dense neural query in VE1, it returns the car/vehicle docs (2, 1, 6) high — because the query "automobile" expands to {automobile, car, vehicle, ...} and the documents expanded the same way, so they share high-weight tokens. Crucially, this scored through the inverted index, not an HNSW graph.

{
  "hits": {
    "hits": [
      { "_id": "2", "_score": 8.9, "_source": { "text": "The automobile industry ..." } },
      { "_id": "1", "_score": 6.2, "_source": { "text": "A fast red sports car ..." } },
      { "_id": "6", "_score": 5.1, "_source": { "text": "Tips for tuning the engine ..." } }
    ]
  }
}

Step 8 — Doc-only mode: skip query-time inference

A bi-encoder (Step 2's model) runs inference on both documents and queries. Doc-only mode is the efficiency trick: encode documents with the full sparse model at index time, but at query time skip the neural encoder and instead expand the query with a cheap tokenizer (analyzer). You configure this by giving the neural_sparse query an analyzer-backed or a lightweight query model. The trade: a small recall drop, but zero query-time model inference → query cost identical to BM25.

# Doc-only style: documents encoded by the sparse model (Step 4 pipeline unchanged),
# but the query side uses a tokenizer-based expansion (no heavy model call at query time).
curl -XPOST 'localhost:9200/sparse-demo/_search?pretty' \
  -H 'Content-Type: application/json' -d '{
  "size": 4, "_source": ["text"],
  "query": {
    "neural_sparse": {
      "text_sparse": {
        "query_text": "automobile",
        "analyzer": "bert-uncased"
      }
    }
  }
}'

Doc-only is the default recommendation for latency-sensitive production: the expensive expansion happens once, at index time, on the documents.

Step 9 — Compare against dense k-NN and BM25

Run the same "automobile" query three ways (you have all three indices if you did VE1):

echo "=== BM25 ===";   curl -s -XPOST 'localhost:9200/sparse-demo/_search' \
  -H 'Content-Type: application/json' -d '{"size":4,"_source":["text"],
  "query":{"match":{"text":"automobile"}}}' | jq -r '.hits.hits[]|"\(._id) \(._score)"'

echo "=== Neural sparse ==="; curl -s -XPOST 'localhost:9200/sparse-demo/_search' \
  -H 'Content-Type: application/json' -d '{"size":4,"_source":["text"],
  "query":{"neural_sparse":{"text_sparse":{"query_text":"automobile","model_id":"<SPARSE_MODEL_ID>"}}}}' \
  | jq -r '.hits.hits[]|"\(._id) \(._score)"'

echo "=== Dense neural (VE1 index) ==="; curl -s -XPOST 'localhost:9200/semantic-demo/_search' \
  -H 'Content-Type: application/json' -d '{"size":4,"_source":["text"],
  "query":{"neural":{"text_embedding":{"query_text":"automobile","model_id":"<DENSE_MODEL_ID>","k":4}}}}' \
  | jq -r '.hits.hits[]|"\(._id) \(._score)"'

Expected pattern: BM25 returns only doc 2 (literal token); both neural-sparse and dense neural surface docs 1, 2, 6 via expansion/similarity. Sparse and dense reach a similar result through different machinery — sparse via expanded postings, dense via graph proximity.

Dense k-NN vs neural sparse — the comparison that matters

The mental model: dense is "geometry in a continuous space, served by a graph"; sparse is "learned keyword expansion, served by the inverted index." Sparse wins on memory and rare-term recall and integrates with BM25 trivially (it is postings); dense often wins on smooth-paraphrase recall and cross-lingual cases. Hybrid (sparse or dense + BM25, Lab VE2) usually beats any single mode.

Deliverables

• A deployed sparse encoding model (<SPARSE_MODEL_ID>, DEPLOYED).
• An ingest pipeline with a sparse_encoding processor and a rank_features-mapped index holding the corpus with auto-generated token:weight vectors.
• A neural_sparse query result, plus a bi-encoder vs doc-only comparison.
• A written comparison of neural-sparse vs dense k-NN vs BM25 on recall, latency, and memory for this corpus.

Troubleshooting

Expected Output

A neural_sparse query for "automobile" returns the car/vehicle/engine documents — driven by learned term expansion stored as postings in a rank_features field — at BM25-class cost and memory, with no HNSW graph involved. Doc-only mode reaches nearly the same ranking with zero query-time model inference.

Stretch Goals

• Inspect a document's text_sparse map and manually compute the sparse dot product against the query's expanded map for the top hit; confirm it matches the _score ordering.
• Build a sparse hybrid: a hybrid query combining match (BM25) + neural_sparse with the normalization-processor from Lab VE2.
• Enable two-phase / pruning on the neural_sparse query (drop low-weight query tokens) and measure the latency/recall change.
• Compare index size on disk: GET /sparse-demo/_stats/store vs GET /semantic-demo/_stats/store — sparse should be far smaller than the dense graph.
• Read where this lives in the plugin: grep neural-search for SparseEncodingProcessor, NeuralSparseQueryBuilder, and rank_features handling.

Coding Exercises

You have run the sparse pipeline with curl. These exercises make you write code that encodes, scores, and grades it — and, crucially, reimplement the sparse dot product the inverted index computes for you. Use Python 3 with requests.

1. (warm-up) An encode-and-inspect script. Write sparse_encode.py that POSTs to _plugins/_ml/_predict/sparse_encoding/<SPARSE_MODEL_ID>, parses the dataAsMap.response token→weight map, sorts it by weight descending, and prints the top 10. Assert that at least one expansion token (one not present in the input text) appears with non-trivial weight — proving learned expansion in code.

2. (core) Reimplement the sparse dot product and match _score. Write sparse_dot.py that (a) reads a document's stored text_sparse map from _source, (b) encodes the query with the model, (c) computes the sparse dot product sum(query_w[t] * doc_w[t]) over shared tokens t, and (d) asserts your value reproduces the document's _score ordering from the neural_sparse query (absolute values may differ by query normalization; the ranking must match). This is the entire scoring story made explicit — the inverted index does exactly this dot product over postings.

3. (core) A bi-encoder vs doc-only A/B harness. Write mode_compare.py that runs the same query set in bi-encoder mode (model_id) and doc-only mode (analyzer), captures rankings and took latency for each, and reports the recall delta and the latency delta. Assert doc-only is faster (no query-time model inference) and quantify the recall it costs — the production trade-off the lab describes.

4. (core) A three-way recall harness: BM25 vs sparse vs dense. Extend the Step-9 comparison into tri_eval.py over a labeled query set, computing recall@k and MRR for match, neural_sparse, and dense neural (VE1 index). Print one table. Assert both neural modes beat BM25 on at least the paraphrase queries — the central claim, now measured rather than eyeballed.

5. (advanced) Advanced challenge — measure and exploit query-token pruning. The neural_sparse query supports two-phase / pruning that drops low-weight query tokens. Write prune_sweep.py that runs the query with progressively more aggressive pruning (drop tokens below a weight threshold you apply client-side before sending, or via the query's pruning config if your version exposes it), plotting recall@k and took vs the kept-token count. Then locate where this lives in the plugin: rg -ln "NeuralSparseQueryBuilder\|prune\|twoPhase\|maxTokenScore" src/main/java in a opensearch-project/neural-search checkout, read the pruning logic, and add (or extend) a JUnit test asserting that pruning a known low-weight token does not change the top-1 result. Deliverable: the recall/latency curve plus a passing upstream-style test — a real candidate for a docs/test PR. Cross-link the memory contrast to native-simd-and-faiss-kernels.md: sparse avoids the SIMD distance kernel entirely by riding postings.

Issues to Practice On

The sparse_encoding processor and neural_sparse query live in opensearch-project/neural-search; the rank_features field type is core Lucene/OpenSearch. OpenSearch workflow: GitHub issues + PRs, a CHANGELOG.md entry, and a DCO Signed-off-by (git commit -s). Find work with:

gh label list --repo opensearch-project/neural-search   # confirm taxonomy (labels move; check the tracker)
gh issue list --repo opensearch-project/neural-search --label "good first issue" --state open
gh issue list --repo opensearch-project/neural-search --search "neural_sparse OR sparse_encoding in:title,body" --state open
gh issue list --repo opensearch-project/neural-search --search "pruning OR two-phase OR rank_features in:title,body" --state open

Representative issue patterns:

• Sparse-query scoring/pruning bugs. "Pruning drops a high-weight token," "doc-only analyzer mismatch lowers recall." Approach: reproduce with a fixed corpus, locate via rg "NeuralSparseQueryBuilder\|SparseEncodingProcessor", add a test that asserts the top-1 invariant, fix, PR with CHANGELOG + DCO.
• Processor robustness (enhancement/good first issue). "Empty/blank text breaks sparse_encoding," "field_map nesting." Approach: minimal repro → integration test → fix → PR.

Planted-bug drill. In sparse_dot.py (exercise 2), compute the dot product over the union of tokens instead of the intersection (treating a missing token as its weight rather than 0). Re-run the "reproduces _score ordering" assertion: docs with many non-shared high-weight tokens now score wrongly and the ranking diverges. Watch the assertion fail, then restrict to shared tokens (multiply, so missing → 0 naturally) and confirm it passes — reproducing the exact semantics the inverted index enforces (only shared postings contribute).

Etiquette: claim the issue first, reproduce before coding; every neural-search PR needs a test + a CHANGELOG.md entry + DCO Signed-off-by (git commit -s). See community-interaction.md.

Validation: Self-check

1. What is a sparse vector here, and what does "term expansion" mean? Give an example of an expansion token from Step 3 that was not in the input.
2. Why can neural-sparse search be "as efficient as BM25"? What index structure does it ride, and why does that matter for memory?
3. Contrast bi-encoder and doc-only modes: what runs at query time in each, and what is the trade-off?
4. Walk the scoring: how is a neural_sparse query scored against a rank_features document? What arithmetic does the inverted index actually do?
5. Compare neural-sparse to dense k-NN on memory, query latency, and rare-term recall. When would you choose each?
6. Why is the destination field typed rank_features and not knn_vector? What breaks if you map it as knn_vector?
7. How would you combine neural-sparse with BM25, and which lab's machinery do you reuse?

When you can answer all seven and reproduce the BM25-vs-sparse-vs-dense comparison, you understand the third retrieval mode. Finally, drop to the bottom of the stack and prove the SIMD speedup that powers the dense path in Lab VE4: SIMD Vectorization Microbenchmark.