Lab VE2: Hybrid Search (BM25 + Neural)

Lab VE1 showed that a neural (dense) query finds semantically-related documents that a BM25 match misses — and, implicitly, the reverse: BM25 nails exact/rare terms that embeddings blur. Hybrid search runs both and fuses their results, so you get keyword precision and semantic recall. The catch is that BM25 scores and neural scores live on incomparable scales, so you cannot just add them. The fix is a search pipeline with the normalization-processor, which normalizes each clause's scores and then combines them. This lab builds that, then tunes the weights and compares hybrid ranking against pure-BM25 and pure-neural.

This builds directly on Lab VE1 (reuse its model, index, and pipeline) and the hybrid section of the masterclass index.

Background

A hybrid query holds multiple sub-queries (here: a match and a neural). Each sub-query runs independently and produces its own scored hit list. Those lists then meet in a normalization-processor that runs in the phase-results stage of a search pipeline: it rescales each list's scores into a comparable range, then merges per document with a combination technique. Without it, a bool should would simply add a BM25 score of, say, 8.4 to a cosine score of 0.7 — and BM25 would dominate by sheer magnitude, drowning the semantic signal.

Why This Matters for Contributors

Hybrid search is the most-recommended retrieval mode in OpenSearch's docs and the most common source of "my ranking looks wrong" reports — almost always a normalization or weighting issue, not a model issue. You need to see the fusion math to triage these.
The normalization-processor is a phase_results_processor — a part of the search pipeline machinery that runs between the query and fetch phases. Understanding where it sits in the pipeline connects this to the broader search-pipeline subsystem.
Weight tuning is an empirical skill; this lab gives you a repeatable A/B harness.

Prerequisites

Completed Lab VE1: a DEPLOYED MiniLM model (<MODEL_ID>), the nlp-ingest-pipeline, and the semantic-demo index with 6 docs. (If you tore them down, redo VE1 steps 1–9.)
curl and (optionally) jq.
Read the dense-vs-sparse-vs-hybrid table.

Note: The hybrid query and normalization-processor are part of the neural-search plugin, bundled by default. The hybrid query type was added in OpenSearch 2.10; weights in the combination block landed shortly after — use 2.11+.

Step-by-Step Tasks

Step 1 — Understand why you cannot just add the scores

Run the two clauses separately and read the raw _score magnitudes:

# Pure BM25
curl -s -XPOST 'localhost:9200/semantic-demo/_search?pretty' \
  -H 'Content-Type: application/json' -d '{
  "size": 3, "_source": ["text"],
  "query": { "match": { "text": "fast vehicle" } }
}' | grep -E '"_id"|"_score"'

# Pure neural
curl -s -XPOST 'localhost:9200/semantic-demo/_search?pretty' \
  -H 'Content-Type: application/json' -d '{
  "size": 3, "_source": ["text"],
  "query": { "neural": { "text_embedding": {
      "query_text": "fast vehicle", "model_id": "<MODEL_ID>", "k": 6 } } }
}' | grep -E '"_id"|"_score"'

You will see BM25 scores in the single digits (e.g. 3.1, 1.8) and neural scores in [0,1]-ish (e.g. 0.72, 0.55). If you summed these in a bool should, the document with the highest BM25 score wins almost regardless of semantic relevance — the scales are incommensurable. That is the problem normalization solves.

	BM25 (`match`)	Neural (cosine/L2)
Range	`[0, ∞)`, unbounded	bounded (≈ `[0,1]` for cosine)
Depends on	corpus stats (idf), term freq	only the two vectors
Typical magnitude here	~1–8	~0.3–0.8

Step 2 — Create a search pipeline with the normalization-processor

The processor normalizes each sub-query's scores, then combines them. Start with min-max normalization and an arithmetic mean, equal weights:

curl -XPUT 'localhost:9200/_search/pipeline/nlp-search-pipeline' \
  -H 'Content-Type: application/json' -d '{
  "description": "normalize + combine BM25 and neural",
  "phase_results_processors": [
    { "normalization-processor": {
        "normalization": { "technique": "min_max" },
        "combination": {
          "technique": "arithmetic_mean",
          "parameters": { "weights": [0.5, 0.5] }
        }
    }}
  ]
}'

Expected: {"acknowledged":true}. The two weights line up positionally with the two clauses in the hybrid query's queries array — weights[0] for the first clause, weights[1] for the second.

Step 3 — Run the `hybrid` query

The hybrid query holds a queries array; you attach the pipeline with ?search_pipeline=:

curl -XPOST 'localhost:9200/semantic-demo/_search?pretty&search_pipeline=nlp-search-pipeline' \
  -H 'Content-Type: application/json' -d '{
  "size": 6,
  "_source": ["text"],
  "query": {
    "hybrid": {
      "queries": [
        { "match": { "text": "fast vehicle" } },
        { "neural": { "text_embedding": {
            "query_text": "fast vehicle", "model_id": "<MODEL_ID>", "k": 6 } } }
      ]
    }
  }
}'

Expected: all _score values are now in a normalized, comparable range (min-max maps each clause's best hit to 1.0 and worst to 0.0 before combining), and the ranking blends both signals. Doc 1 ("fast red sports car"), doc 6 ("motor vehicle"), and doc 2 ("automobile/electric vehicles") cluster at the top — each for a different reason (doc 1 matches "fast" lexically and is semantically a vehicle; doc 6 is semantically a vehicle; doc 2 is semantically about vehicles).

Warning: Running a hybrid query without a normalization-processor search pipeline is an error (or, in some versions, silently mis-scores). If you get "hybrid query ... must be executed with a search pipeline", you forgot ?search_pipeline= or the pipeline lacks the processor.

Step 4 — Build the A/B comparison: three rankings, one corpus

Capture all three rankings for "fast vehicle" so you can compare them side by side:

echo "=== BM25 only ===";    curl -s -XPOST 'localhost:9200/semantic-demo/_search' \
  -H 'Content-Type: application/json' -d '{"size":6,"_source":["text"],
  "query":{"match":{"text":"fast vehicle"}}}' | jq -r '.hits.hits[]|"\(._id)  \(._score)"'

echo "=== Neural only ===";  curl -s -XPOST 'localhost:9200/semantic-demo/_search' \
  -H 'Content-Type: application/json' -d '{"size":6,"_source":["text"],
  "query":{"neural":{"text_embedding":{"query_text":"fast vehicle","model_id":"<MODEL_ID>","k":6}}}}' \
  | jq -r '.hits.hits[]|"\(._id)  \(._score)"'

echo "=== Hybrid ===";       curl -s -XPOST 'localhost:9200/semantic-demo/_search?search_pipeline=nlp-search-pipeline' \
  -H 'Content-Type: application/json' -d '{"size":6,"_source":["text"],
  "query":{"hybrid":{"queries":[
    {"match":{"text":"fast vehicle"}},
    {"neural":{"text_embedding":{"query_text":"fast vehicle","model_id":"<MODEL_ID>","k":6}}}]}}}' \
  | jq -r '.hits.hits[]|"\(._id)  \(._score)"'

A representative outcome (your exact scores differ; the pattern is the lesson):

Rank	BM25 only	Neural only	Hybrid (0.5/0.5)
1	doc 1 (has "fast")	doc 1	doc 1
2	doc 6 (has "vehicle")	doc 6	doc 6
3	doc 2 (has "vehicles")	doc 2	doc 2
4	— (no more lexical)	doc 5 (predators, "fast")	doc 5
5	—	doc 4	doc 4

Note where they diverge: BM25 stops returning anything after the docs with literal token overlap; neural ranks the whole corpus by similarity; hybrid keeps BM25's precision on the top lexical matches while also surfacing the semantic-only doc 5. On a larger corpus the divergence is dramatic — hybrid consistently beats either alone on recall@k.

Step 5 — Tune the weights

The combination weights bias toward one signal. Re-create the pipeline favoring neural (0.3 BM25 / 0.7 neural), then BM25 (0.7 / 0.3), and re-run the hybrid query each time:

# Neural-leaning
curl -XPUT 'localhost:9200/_search/pipeline/nlp-search-pipeline' \
  -H 'Content-Type: application/json' -d '{
  "phase_results_processors": [{ "normalization-processor": {
    "normalization": { "technique": "min_max" },
    "combination": { "technique": "arithmetic_mean", "parameters": { "weights": [0.3, 0.7] } }
  }}]}'
# ... re-run the hybrid query from Step 3 ...

# BM25-leaning
curl -XPUT 'localhost:9200/_search/pipeline/nlp-search-pipeline' \
  -H 'Content-Type: application/json' -d '{
  "phase_results_processors": [{ "normalization-processor": {
    "normalization": { "technique": "min_max" },
    "combination": { "technique": "arithmetic_mean", "parameters": { "weights": [0.7, 0.3] } }
  }}]}'
# ... re-run again ...

Observe: the neural-leaning pipeline pushes semantic-only docs (like doc 5) up; the BM25-leaning one pushes literal-token-match docs up. There is no universal best weight — it is a per-corpus, per-query-mix tuning decision you make against a labeled eval set.

Step 6 — Try L2 normalization and other combination techniques

Min-max is one of two normalizations; arithmetic mean is one of three combinations. Swap them:

curl -XPUT 'localhost:9200/_search/pipeline/nlp-search-pipeline' \
  -H 'Content-Type: application/json' -d '{
  "phase_results_processors": [{ "normalization-processor": {
    "normalization": { "technique": "l2" },
    "combination": { "technique": "harmonic_mean", "parameters": { "weights": [0.5, 0.5] } }
  }}]}'

Normalization	What it does
`min_max`	rescale each list to `[0,1]` (best→1, worst→0); most common
`l2`	divide each score by the L2 norm of its list; preserves relative gaps

Combination	Behaviour	When to prefer
`arithmetic_mean`	weighted average; lenient — a high score in one signal lifts the doc	default; balanced
`geometric_mean`	weighted geo-mean; rewards docs scoring well in both	"must be good in both"
`harmonic_mean`	weighted harmonic; punishes a near-zero in either signal hardest	strict AND-like fusion

Re-run the hybrid query after each change and watch the ordering shift. Geometric and harmonic means penalize docs that score near-zero on either clause — useful when you want a hit only if both lexical and semantic signals agree.

Step 7 — Confirm the pipeline placement

The normalization-processor is a phase-results processor: it runs after the query phase produced per-clause hit lists but before fetch. Confirm the pipeline shape:

curl -s 'localhost:9200/_search/pipeline/nlp-search-pipeline?pretty'

You should see your processor under phase_results_processors (not request_processors or response_processors). That placement is why it can see and rescale each sub-query's full scored list — it operates on the phase results, not on the request or the final response.

Deliverables

A search pipeline nlp-search-pipeline with a normalization-processor.
A hybrid query result fusing match + neural, with normalized comparable scores.
A three-way comparison table (BM25 / neural / hybrid) for one query, plus at least two weightings, with a written note on which ranking is best for this corpus and why.
One run each of min-max+arithmetic, l2+harmonic (and ideally geometric), with the observed ranking differences.

Troubleshooting

Symptom	Cause	Fix
"hybrid query must be executed with a search pipeline"	no `?search_pipeline=` or processor missing	attach the pipeline; ensure `normalization-processor` is present
One clause dominates entirely	scores not normalized (wrong pipeline), or extreme weights	verify the processor ran; rebalance `weights`
`weights` ignored	array length ≠ number of `queries`, or wrong version	weights must match clause count positionally; use 2.11+
Hybrid returns fewer docs than expected	`match` matched nothing for some docs; with harmonic/geometric a 0 in one clause sinks them	use arithmetic mean, or raise neural `k`
Scores identical to pure-neural	the `match` clause matched nothing (no overlap)	choose a query with lexical overlap, or it's expected for that query
400 on pipeline create	typo in `technique` name	valid: `min_max`/`l2`; `arithmetic_mean`/`geometric_mean`/`harmonic_mean`

Expected Output

A hybrid query whose hits carry normalized, comparable scores and whose ranking blends exact-keyword precision (from BM25) with semantic recall (from neural). Adjusting the combination weights visibly shifts the ranking toward whichever signal you favor, and swapping min-max↔l2 or arithmetic↔harmonic↔geometric changes how aggressively a weak signal in one clause is penalized.

Stretch Goals

Set the pipeline as the index's index.search.default_pipeline so you can omit ?search_pipeline= on every request.
Add a third clause (e.g. a match on a title field) and a third weight; observe how three-way fusion behaves.
Build a tiny labeled eval: pick 5 queries with a known "best" doc, score each weighting by mean reciprocal rank, and find the weight that wins on your corpus.
Replace the dense neural clause with a neural_sparse clause (Lab VE3) to build a sparse-hybrid and compare against the dense-hybrid.
Trace where in the code the hybrid query and normalization run: grep the neural-search plugin for HybridQueryBuilder, NormalizationProcessor, ScoreNormalizationTechnique, ScoreCombinationTechnique.

Coding Exercises

You have tuned fusion by hand with curl. These exercises make you implement and test the score math, so you can defend a ranking change with numbers. Use Python 3 for the harnesses; the hybrid-score test (exercise 3) is the centerpiece — it re-derives the normalize-then-combine math the plugin runs.

(warm-up) Capture the raw scale gap. Write scales.py that runs the pure-BM25 and pure-neural queries from Step 1, extracts the _score lists, and prints min/max/mean for each. Assert the two ranges barely overlap (BM25 max ≫ neural max) — the quantitative justification for normalization, in code rather than prose.
(core) Reimplement min-max + arithmetic-mean fusion client-side. Write fuse.py that runs the match and neural clauses separately, applies min-max normalization to each hit list (best→1, worst→0), combines per doc with a weighted arithmetic mean, and produces a fused ranking. Then run the real hybrid query through nlp-search-pipeline and assert your client-side ranking matches the server's top-k order (scores may differ in absolute value; ordering must match). This proves you understand exactly what the normalization-processor does.
(core) A hybrid-score unit test. Write test_hybrid_score.py (pytest) with a hand-built fixture: two clauses, doc scores A=[0.0, 10.0, 4.0] and B=[0.8, 0.2, 0.5]. Implement min_max, l2, arithmetic_mean, geometric_mean, harmonic_mean as pure functions and assert: (a) min-max maps the per-list max to 1.0 and min to 0.0; (b) for a doc scoring 0 in one clause, harmonic_mean → 0 but arithmetic_mean > 0 (the "punishes a near-zero hardest" property from Step 6); (c) weights [0.7,0.3] vs [0.3,0.7] flip the rank of two specific docs. This is the math the plugin's ScoreNormalizationTechnique / ScoreCombinationTechnique implement — grep them to compare your formulas: rg -n "min_max\|l2\|arithmetic\|geometric\|harmonic" src/main/java in a neural-search checkout.
(core) An MRR weight-sweep. Write sweep.py that, over a tiny labeled set (5 queries × known-best doc), sweeps the BM25/neural weight from [1.0,0.0] to [0.0,1.0] in steps of 0.1 (re-PUT the pipeline each time), runs the hybrid query, and computes mean reciprocal rank. Print the weight that maximizes MRR on your corpus and assert hybrid's best MRR ≥ both pure methods' MRR — the empirical claim the lab makes.
(advanced) Advanced challenge — port one fusion path into a real plugin test. In a opensearch-project/neural-search checkout, locate the existing combination tests: rg -ln "ScoreCombinationTechnique\|ArithmeticMeanScoreCombination\|HarmonicMean" src/test. Add a new JUnit test case (or extend one) that asserts a specific property your test_hybrid_score.py proved — e.g. that geometric mean with a zero-score clause yields zero. Build just that test (./gradlew test --tests '*ScoreCombination*'). Deliverable: a passing upstream-style unit test plus a note on which existing test class you extended and whether the property was already covered (if not, you have a real PR candidate). This is the bridge from your Python harness to a mergeable contribution.

Issues to Practice On

The hybrid query and the normalization-processor live in opensearch-project/neural-search. OpenSearch workflow: GitHub issues + PRs, a CHANGELOG.md entry, and a DCO Signed-off-by (git commit -s). Find work with:

gh label list --repo opensearch-project/neural-search   # confirm taxonomy (labels move; check the tracker)
gh issue list --repo opensearch-project/neural-search --label "good first issue" --state open
gh issue list --repo opensearch-project/neural-search --search "hybrid OR normalization in:title,body" --state open
gh issue list --repo opensearch-project/neural-search --search "combination OR weights in:title,body" --state open

Representative issue patterns:

Normalization/combination correctness. "Weights ignored when length ≠ clause count," "min-max divides by zero when all scores equal." Approach: reproduce with a degenerate score list, locate the technique via rg "ScoreNormalizationTechnique\|MinMaxScoreNormalization", add a unit test for the edge case, fix, PR with CHANGELOG + DCO.
New combination/normalization technique (enhancement). A common enhancement: add a technique (e.g. RRF-style). Approach: study the ScoreCombinationTechnique interface, implement, register, test against a fixture like exercise 3.

Planted-bug drill. In your fuse.py (exercise 2), swap the min-max formula to (x - min) without dividing by (max - min) — i.e. drop the normalization denominator. Re-run the "client matches server top-k" assertion from exercise 2: a clause with large raw scores now dominates and the ordering diverges from the server's. Watch the assertion fail, then restore the divisor and confirm it passes — you have reproduced exactly the "one clause dominates entirely" symptom from Troubleshooting, and the assertion is the guard that catches it.

Etiquette: claim the issue first, reproduce before coding; every neural-search PR needs a test + a CHANGELOG.md entry + DCO Signed-off-by (git commit -s). See community-interaction.md.

Validation: Self-check

Why can't you put a match and a neural clause in a bool should and rely on the summed _score? Give concrete magnitudes from Step 1.
What does the normalization step do, and what does the combination step do? At which stage of the search pipeline (request / phase-results / response) do they run, and why must it be that stage?
Explain min-max vs l2 normalization in one sentence each.
Contrast arithmetic, geometric, and harmonic mean combination: which most punishes a doc that scores near-zero on one clause, and when would you want that?
From your three-way comparison, name one document that hybrid ranks well but one of the pure methods misses entirely, and explain why.
How do the combination weights map to the hybrid query's queries array? What happens if their lengths disagree?
Describe how you would empirically choose a weighting for a real corpus.

When you can answer all seven and show hybrid beating both pure methods on at least one query, you understand score fusion. Next: the sparse alternative to dense embeddings in Lab VE3: Neural Sparse Search.

Open-Source Engineer & Contributor