Lab QE1: QueryBuilder to Lucene Query

Background

The intensive drew the pipeline: JSON DSL → QueryBuilder tree (via NamedXContentRegistry) → rewrite(QueryRewriteContext) → toQuery(QueryShardContext) → a Lucene Query. This lab makes that pipeline visible on a running cluster. You will issue a bool/match query and use two APIs to see exactly what it becomes: _validate/query?rewrite=true shows the rewritten Lucene query string, and _search?profile=true shows the per-component timing of the Weight/Scorer machinery. Then you will code-trace the OpenSearch classes that do the work — MatchQueryBuilder.doToQuery calling the analyzer, and BoolQueryBuilder mapping its clause lists onto BooleanClause.Occur — so the API output stops being magic.

The single most important thing you will internalize: match analyzes at query time using the field's search analyzer, term does not. That one fact explains a large fraction of "why doesn't my query match anything" tickets.

Why This Matters for Contributors

When a user files "my query returns the wrong docs," the first question is which representation is wrong — the parsed builder, the rewritten query, or the scoring. _validate/query?rewrite=true collapses the first two into one inspectable string, and profile=true opens up the third. A contributor who can read those two outputs triages a query bug in minutes instead of guessing. And when you add a new query type via SearchPlugin.getQueries(), the only way to verify your doToQuery produced the Lucene query you intended is to read it back out of _validate/query. This lab builds that reflex.

Prerequisites

• A running OpenSearch (3.x). Quickest: docker run -p 9200:9200 -e "discovery.type=single-node" -e "DISABLE_SECURITY_PLUGIN=true" opensearchproject/opensearch:latest — or ./gradlew run from a source checkout (attaches a debugger on 5005 with --debug-jvm).
• curl and jq.
• An OpenSearch source checkout for the code-trace steps: git clone https://github.com/opensearch-project/OpenSearch.
• You've read the intensive and the deep-dive Query DSL and QueryBuilders.

Note: DISABLE_SECURITY_PLUGIN=true is for the lab only. On a real cluster the requests are identical; you just add credentials/TLS. "Cluster manager" (formerly master) is the elected coordinating-config node; none of this lab needs it beyond a healthy single-node cluster.

Step-by-Step Tasks

Step 1 — Create an index with a known analyzer

We want a text field (analyzed) and a keyword field (not), so the match vs term contrast is unambiguous.

H='-H content-type:application/json'
curl -s -XDELETE localhost:9200/qe1 >/dev/null

curl -s -XPUT localhost:9200/qe1 $H -d '{
  "settings": { "number_of_shards": 1, "number_of_replicas": 0 },
  "mappings": { "properties": {
    "title":  { "type": "text",    "analyzer": "standard" },
    "status": { "type": "keyword" }
  }}
}' | jq .

curl -s -XPOST 'localhost:9200/qe1/_bulk?refresh=true' $H --data-binary '
{"index":{"_id":1}}
{"title":"Open Source Search Engine","status":"published"}
{"index":{"_id":2}}
{"title":"open data and open formats","status":"draft"}
{"index":{"_id":3}}
{"title":"closed proprietary system","status":"published"}
' | jq '.errors'

number_of_shards: 1 keeps per-shard idf stable and makes the profile output a single shard — easier to read.

Step 2 — See the parsed-and-rewritten Lucene query

_validate/query?rewrite=true&explain=true does parse + rewrite + toQuery and hands you the Lucene string form without running the search:

curl -s 'localhost:9200/qe1/_validate/query?rewrite=true&explain=true' $H -d '{
  "query": { "bool": {
    "must":   [ { "match": { "title": "Open Source" } } ],
    "filter": [ { "term":  { "status": "published" } } ]
  }}
}' | jq -r '.explanations[0].explanation'

You should see something close to:

+(title:open title:source) #status:published

Read it carefully:

• title:open title:source — the match analyzed "Open Source" with the standard analyzer: lowercased and split into two tokens, producing a BooleanQuery of two SHOULD TermQuerys.
• The leading +(...) marks the whole match clause as MUST (required, scored).
• #status:published — the # prefix is Lucene's notation for a FILTER clause (required, not scored). published was not analyzed because status is a keyword.

Step 3 — Prove term on a text field is the classic bug

curl -s 'localhost:9200/qe1/_validate/query?rewrite=true&explain=true' $H -d '{
  "query": { "term": { "title": "Open Source" } }
}' | jq -r '.explanations[0].explanation'

You get title:Open Source — a single term with a capital O and a space, because term does no analysis. No indexed token equals "Open Source" (the indexed tokens are open, source, lowercased and split), so this matches nothing. Confirm:

curl -s 'localhost:9200/qe1/_search' $H -d '{"query":{"term":{"title":"Open Source"}}}' \
  | jq '.hits.total.value'        # -> 0
curl -s 'localhost:9200/qe1/_search' $H -d '{"query":{"match":{"title":"Open Source"}}}' \
  | jq '.hits.total.value'        # -> 2  (docs 1 and 2)

This is the index-time vs query-time analysis lesson: the indexed tokens were produced by the analyzer at index time; match runs the same analyzer at query time so the tokens line up; term skips analysis entirely and compares your raw string to the stored tokens.

Step 4 — Profile the execution

Now run it for real with profile=true and inspect the Weight/Scorer breakdown:

curl -s 'localhost:9200/qe1/_search?pretty' $H -d '{
  "profile": true,
  "query": { "bool": {
    "must":   [ { "match": { "title": "open source" } } ],
    "filter": [ { "term":  { "status": "published" } } ]
  }}
}' > /tmp/qe1-profile.json

# The Lucene query type and the per-method timings:
jq '.profile.shards[0].searches[0].query[0]
    | {type, description, time_ns: .time_in_nanos, breakdown}' /tmp/qe1-profile.json

You'll see a BooleanQuery at the top with children, and a breakdown object with the exact phases from the intensive: create_weight, build_scorer, next_doc, advance, score, match, set_min_competitive_score. Map them:

# The collector tree (TopScoreDocCollector etc.) is reported separately:
jq '.profile.shards[0].searches[0].collector' /tmp/qe1-profile.json

Step 5 — Code-trace MatchQueryBuilder.doToQuery

Now connect the API output to the source. In your OpenSearch checkout:

cd ~/src/OpenSearch
# Where match becomes a Lucene query, and where it grabs the analyzer:
grep -n "doToQuery\|MatchQuery\|getSearchAnalyzer\|MappedFieldType\|analyzer" \
  server/src/main/java/org/opensearch/index/query/MatchQueryBuilder.java | head
# The heavy lifting is delegated to a MatchQuery / analyzer helper:
grep -rn "class MatchQuery\b\|createBooleanQuery\|createPhraseQuery\|Analyzer" \
  server/src/main/java/org/opensearch/index/search/MatchQuery.java | head

Answer in your notes, from the code:

• Which MappedFieldType/analyzer method supplies the analyzer? (Look for the field type's search analyzer — the same one the mapping declared.)
• For a two-token input, which Lucene factory is called, and does it return a BooleanQuery, a PhraseQuery, or a SynonymQuery? (It depends on positions and synonyms — trace the branch.)

Step 6 — Code-trace BoolQueryBuilder.doToQuery

grep -n "doToQuery\|Occur\|MUST\|SHOULD\|FILTER\|MUST_NOT\|minimumShouldMatch\|addBooleanClauses" \
  server/src/main/java/org/opensearch/index/query/BoolQueryBuilder.java

Confirm the mapping you saw in _validate/query: the four DSL clause lists (must, should, filter, must_not) each become BooleanClauses with the matching BooleanClause.Occur, and minimum_should_match sets the BooleanQuery.Builder.setMinimumNumberShouldMatch. The + and # in the Lucene string are exactly MUST and FILTER.

Step 7 — (Optional) Attach a debugger to watch it live

If you ran from source with ./gradlew run --debug-jvm, attach your IDE to port 5005, set a breakpoint in MatchQueryBuilder.doToQuery and BoolQueryBuilder.doToQuery, and re-issue the Step 4 query. Step through and watch the analyzer produce [open, source] and the BooleanQuery.Builder accumulate clauses. There is no substitute for seeing the tree assemble.

Deliverables

• The _validate/query?rewrite=true output for the bool query, with the +(title:open title:source) #status:published form annotated (which token came from analysis, which clause is the filter).
• The term-on-text output (title:Open Source) plus the hits.total of 0 vs the match hits.total of 2, with a one-sentence explanation.
• The profile=true breakdown for the BooleanQuery, with each phase mapped to a Query/Weight/Scorer concept.
• A reading-log artifact (a short markdown file) answering Step 5 and Step 6: which analyzer method MatchQueryBuilder uses, which Lucene factory a two-token match calls, and how BoolQueryBuilder maps clause lists to Occur.

Expected Output

# Step 2 _validate
+(title:open title:source) #status:published

# Step 3 term-on-text
title:Open Source
term total = 0
match total = 2

# Step 4 profile (abridged)
{
  "type": "BooleanQuery",
  "description": "+(title:open title:source) #status:published",
  "time_ns": 412345,
  "breakdown": {
    "create_weight": 51000, "build_scorer": 90000,
    "next_doc": 41000, "advance": 12000, "score": 33000,
    "match": 0, "set_min_competitive_score": 1200, ...
  }
}

Troubleshooting

Stretch Goals

• Phrase vs boolean. Run _validate/query?rewrite=true on {"match_phrase":{"title":"open source"}} and compare to match. You'll see a PhraseQuery (title:"open source") — it reads positions, where match is a position-free BooleanQuery. Tie back to IndexOptions in The Inverted Index and Postings.
• Watch a rewrite collapse a query. Validate {"bool":{"must":[{"match_all":{}}]}} and a single-clause bool; observe Lucene simplifying it.
• Custom analyzer surprise. Add an analyzer with a stop filter, re-index, and watch match on "the open source" drop the — visible in the validate string.
• Register-and-inspect. Skim SearchModule.registerQuery and find one query type registered by a plugin (e.g., grep the k-NN plugin for getQueries), then validate that query and read its Lucene form.

Coding Exercises

_validate/query and the profiler showed you what a builder becomes; these exercises make you assert it in code. The canonical pattern is a JUnit test extending AbstractQueryTestCase<YourBuilder> (or a hand-built QueryShardContext) that calls doToQuery and inspects the returned Lucene Query. Find the base with rg -l "class AbstractQueryTestCase" test/; find a sibling test with rg -l "class MatchQueryBuilderTests|class BoolQueryBuilderTests" server/.

1. (warm-up) Assert match analyzes and term does not. Write a JUnit test that builds new MatchQueryBuilder("title", "Open Source"), calls doToQuery(context) against a text field with the standard analyzer, and asserts the result is a BooleanQuery of two lowercased TermQuerys (title:open, title:source). Add a sibling assertion that new TermQueryBuilder("title", "Open Source") yields a single un-analyzed TermQuery. This is the Step 3 lesson as an executable contract.

2. (core) Map every bool clause to its Occur. Write a test that builds a BoolQueryBuilder with one clause in each of must/should/filter/must_not, calls doToQuery, and asserts each resulting BooleanClause carries the right BooleanClause.Occur (MUST/SHOULD/FILTER/MUST_NOT) — the code form of the +/# markers you read in Step 2. Then set minimum_should_match(1) and assert getMinimumNumberShouldMatch() on the built BooleanQuery.

3. (core) A rewrite test. Write a test that rewrites bool { must: [ match_all ] } (call Rewriteable.rewrite(...) or the builder's rewrite(QueryRewriteContext); find it with rg -n "rewrite\(QueryRewriteContext" server/.../index/query/) and asserts the tree collapses (e.g. a single-clause bool simplifies). Capture the before/after in the test as comments so the simplification is documented.

4. (core) match_phrase produces a PhraseQuery. Write a test asserting new MatchPhraseQueryBuilder("title", "open source").doToQuery(ctx) returns a PhraseQuery (positions), contrasting with the position-free BooleanQuery from exercise 1. Tie it to IndexOptions from The Inverted Index and Postings.

5. (advanced) Advanced challenge — ship a tiny query type via SearchPlugin.getQueries() and prove its Lucene form. Build a minimal SearchPlugin registering a QueryBuilder (the smallest real one: a thin wrapper whose doToQuery returns, say, a TermQuery or delegates to an existing builder — confirm the QuerySpec API with rg -n "getQueries|class QuerySpec" server/.../plugins/SearchPlugin.java). Register it, write an AbstractQueryTestCase-style serialization + doToQuery test, build the plugin, then verify end to end by issuing your query through _validate/query?rewrite=true and asserting the Lucene string matches what your test predicted. Deliverable: a registered query type whose unit test and live _validate/query output agree.

Issues to Practice On

The query-translation skill you just built triages "wrong docs returned" tickets. Practice on opensearch-project/OpenSearch.

Labels drift; confirm with gh label list --repo opensearch-project/OpenSearch (look for Search, Search:Query Capabilities, Search:Relevance).

Representative patterns. (1) "term/match on field X returns nothing / wrong docs." — almost always an analysis mismatch (index-time vs query-time), exactly the Step 3 bug; reproduce with _validate/query?rewrite=true, then locate the analyzer fetch in MatchQuery/MappedFieldType. (2) "bool with minimum_should_match behaves unexpectedly." — reproduce, trace BoolQueryBuilder.doToQuery to where it sets setMinimumNumberShouldMatch, and add a test. Approach: reproduce → locate via rg → fix → test → PR with CHANGELOG + DCO.

Planted-bug drill. In BoolQueryBuilder.doToQuery, swap the Occur for the filter list to MUST (so a filter clause starts scoring). Rebuild and run the bool query tests (rg -l "class BoolQueryBuilderTests" server/); watch the Occur-mapping assertion go red — and note the subtler consequence that a previously non-scored, cacheable clause now scores. Revert, then add an explicit assertion that the filter list maps to Occur.FILTER (not MUST). That assertion is the regression test for a class of "my filter suddenly affects scores" bugs.

Etiquette: claim the issue with a comment first, reproduce before fixing, and every PR needs a test + CHANGELOG entry + DCO Signed-off-by (git commit -s). See community interaction and the prepare-a-PR lab.

Validation / Self-check

• You can predict the _validate/query?rewrite=true string for any bool of match/term/range clauses, including the +/# markers and which tokens analysis produced.
• You can explain index-time vs query-time analysis and why match matches where term does not on a text field.
• You can read a profile=true breakdown and name what each phase is doing in Query/Weight/Scorer terms.
• You can point at the line in MatchQueryBuilder/MatchQuery where the analyzer is fetched, and the line in BoolQueryBuilder where a clause list becomes BooleanClause.Occur.

Next: Lab QE2 — BM25 and Scoring Internals.