Query DSL and QueryBuilders

The JSON you put in a _search request body is not executed directly. It is parsed into a tree of QueryBuilder objects, optionally rewritten into a simpler equivalent tree, and finally converted to a Lucene Query that actually runs against the index. Three distinct representations, three distinct classes of bug. This chapter walks that pipeline end to end and shows where a SearchPlugin injects a custom query type.

It sits under Search Execution (which calls into the query phase) and above the Lucene scoring machinery. It pairs with Mapping and Analysis (field types decide what a query can do) and DocValues and Fielddata (what sorting and some queries read).

After this chapter you can:

• Trace a match query from JSON to a Lucene BooleanQuery of TermQuerys.
• Explain the difference between parse-time, rewrite-time, and toQuery-time, and which context each runs in.
• Register a custom QueryBuilder through SearchPlugin.getQueries().
• Use _validate/query?explain and _explain to debug parsing and scoring.

The class hierarchy

ls server/src/main/java/org/opensearch/index/query/ | grep QueryBuilder | head -40
grep -n "interface QueryBuilder\|toQuery\|doRewrite\|getWriteableName" \
  server/src/main/java/org/opensearch/index/query/QueryBuilder.java

Stage 1 — Parse JSON into a QueryBuilder tree

The query JSON object has exactly one key naming the query type (match, bool, term, …). AbstractQueryBuilder.parseInnerQueryBuilder reads that key and dispatches through the NamedXContentRegistry to the registered parser for that name (each builder exposes a static fromXContent(XContentParser)).

flowchart LR
    J["{ bool: { must: [ { match: { title: hi } } ] } }"] --> P[parseInnerQueryBuilder reads first key 'bool']
    P --> R[NamedXContentRegistry lookup: bool -> BoolQueryBuilder.fromXContent]
    R --> B[BoolQueryBuilder]
    B --> P2[recurse into 'match' -> MatchQueryBuilder]
    P2 --> T[QueryBuilder tree built, NOT yet a Lucene Query]

grep -n "parseInnerQueryBuilder\|fromXContent\|NamedXContentRegistry" \
  server/src/main/java/org/opensearch/index/query/AbstractQueryBuilder.java
grep -n "public static.*fromXContent" \
  server/src/main/java/org/opensearch/index/query/BoolQueryBuilder.java

Parsing failures (unknown query name, malformed body) surface here as ParsingException / XContentParseException — before any index is touched. This is why a typo'd query type fails fast with a clear message.

The registry that maps "bool" → BoolQueryBuilder is built in SearchModule from every core query plus everything SearchPlugin.getQueries() contributes.

grep -n "registerQuery\|QuerySpec\|getQueries" \
  server/src/main/java/org/opensearch/search/SearchModule.java

Stage 2 — Rewrite

Before conversion, the tree is rewritten (Rewriteable.rewrite → QueryBuilder.rewrite(QueryRewriteContext) → each builder's doRewrite). Rewrite simplifies and resolves:

• MatchNoneQueryBuilder shortcuts (e.g., a range that can't match anything on this shard rewrites to match-none — feeds the can_match optimization in Search Execution).
• Asynchronous resolution: terms lookup, wrapper, percolator, and some queries fetch data during rewrite (QueryRewriteContext can do async fetches), so rewrite can be a multi-round process.
• Constant folding: bool with a single clause may collapse.

grep -n "doRewrite\|rewrite(\|MatchNoneQueryBuilder" \
  server/src/main/java/org/opensearch/index/query/AbstractQueryBuilder.java \
  server/src/main/java/org/opensearch/index/query/RangeQueryBuilder.java

Note: Rewrite runs on the coordinating node (with a QueryRewriteContext) and per shard (with a QueryShardContext, which is a QueryRewriteContext subclass that also has mapper/searcher access). A query may rewrite differently per shard — that is by design.

Stage 3 — toQuery: become a Lucene Query

The final step is AbstractQueryBuilder.toQuery(QueryShardContext) → doToQuery(...), which produces an actual Lucene org.apache.lucene.search.Query. This is where field types from the mapping (Mapping and Analysis) matter: MatchQueryBuilder consults the field's analyzer to tokenize the input, then emits one TermQuery per token wrapped in a BooleanQuery. A term query does not analyze — it matches the exact token, which is why term on an analyzed text field is a classic mistake.

grep -n "doToQuery\|MappedFieldType\|getMapperService\|analyzer" \
  server/src/main/java/org/opensearch/index/query/MatchQueryBuilder.java \
  server/src/main/java/org/opensearch/index/query/TermQueryBuilder.java

The full pipeline:

flowchart TD
    A[JSON DSL] -->|parseInnerQueryBuilder + NamedXContentRegistry| B[QueryBuilder tree]
    B -->|rewrite QueryRewriteContext| C[Simplified QueryBuilder tree]
    C -->|toQuery QueryShardContext| D[Lucene Query]
    D -->|IndexSearcher.search with Similarity| E[scored hits in QueryPhase]

Scoring: BM25 and Similarity

Once you have a Lucene Query, Lucene's IndexSearcher runs it against each segment and scores matches using a Similarity. The default is BM25 (BM25Similarity), driven by term frequency (tf), inverse document frequency (idf), and field-length normalization. filter/must_not clauses and constant_score produce no score contribution (they affect matching, not ranking).

Per-field similarity can be customized via index.similarity.* and the field's similarity mapping. Global term statistics for idf can be made cluster-consistent via the DFS phase (see Search Execution).

grep -rn "BM25Similarity\|SimilarityProvider\|index.similarity" \
  server/src/main/java/org/opensearch/index/similarity/

Note: filter context (in bool.filter, constant_score) is both faster and cacheable because it skips scoring entirely and Lucene can cache the DocIdSet. Move non-relevance predicates (timestamps, status flags) into filter, not must.

Registering a custom query in a SearchPlugin

A plugin adds a query type by implementing SearchPlugin.getQueries() and returning QuerySpecs, each binding: the query name, a Writeable.Reader (stream deserialization — see Serialization and BWC), and an fromXContent parser.

// In your Plugin implements SearchPlugin
@Override
public List<QuerySpec<?>> getQueries() {
    return List.of(new QuerySpec<>(
        MyQueryBuilder.NAME,                 // "my_query"
        MyQueryBuilder::new,                 // StreamInput ctor (wire)
        MyQueryBuilder::fromXContent));      // JSON parser
}

SearchModule folds these into both the NamedXContentRegistry (for JSON parsing) and the NamedWriteableRegistry (for cross-node transport). Your MyQueryBuilder extends AbstractQueryBuilder<MyQueryBuilder> must implement doWriteTo, doXContent, doRewrite, doToQuery, doEquals, doHashCode, and getWriteableName.

grep -n "interface SearchPlugin\|QuerySpec\|getQueries" \
  server/src/main/java/org/opensearch/plugins/SearchPlugin.java

See Plugin Architecture for how SearchPlugin is discovered and wired, and the Level-7 plugin lab for a full custom-query build.

Debugging tools: _validate/query and _explain

# Did my query parse, and what Lucene query does it become?
curl -s 'localhost:9200/idx/_validate/query?explain=true&pretty' \
  -H 'content-type: application/json' \
  -d '{"query":{"match":{"title":"open source"}}}'

# Why did (or didn't) THIS doc match, and how was it scored?
curl -s 'localhost:9200/idx/_explain/DOC_ID?pretty' \
  -H 'content-type: application/json' \
  -d '{"query":{"match":{"title":"open source"}}}'

_validate/query?explain returns the Lucene string form of the rewritten query per index — the fastest way to confirm that match analyzed your text the way you expected (e.g., you'll see title:open title:source, proving the analyzer split and lowercased it). _explain returns the BM25 score breakdown for one document: tf, idf, boost, field-length norm.

Warning: The Lucene string in _validate/query reflects the query after analysis and rewrite. If you see title:Open (capital O) you have a keyword field or a non-lowercasing analyzer — a mapping problem, not a query problem.

Reading exercise

# 1. The parse dispatch
grep -n "parseInnerQueryBuilder\|fromXContent" \
  server/src/main/java/org/opensearch/index/query/AbstractQueryBuilder.java

# 2. A leaf builder end to end
sed -n '1,60p' server/src/main/java/org/opensearch/index/query/TermQueryBuilder.java
grep -n "doToQuery\|doRewrite\|doWriteTo\|fromXContent\|NAME" \
  server/src/main/java/org/opensearch/index/query/MatchQueryBuilder.java

# 3. Compound query
grep -n "must\|should\|filter\|mustNot\|doToQuery" \
  server/src/main/java/org/opensearch/index/query/BoolQueryBuilder.java

# 4. Registry wiring
grep -n "registerQuery\|QuerySpec\|getQueries\|namedWriteables" \
  server/src/main/java/org/opensearch/search/SearchModule.java

Answer:

1. Where, exactly, does the registry decide that the JSON key "bool" maps to BoolQueryBuilder? Trace from parseInnerQueryBuilder to the registry lookup.
2. Name three things that can happen during rewrite that change the query tree, and give one query type for each.
3. Run _validate/query?explain for {"term":{"title":"Open Source"}} against a text field. Predict the Lucene query string and explain why it likely matches nothing.
4. In toQuery, where does MatchQueryBuilder obtain the analyzer, and what Lucene query does a two-token input produce?
5. List the six do* methods a custom AbstractQueryBuilder subclass must implement, and say which two are about wire/JSON serialization vs query semantics.
6. Why is a predicate in bool.filter cheaper than the same predicate in bool.must? What can Lucene cache for the filter case?

Common bugs and symptoms

Validation: prove you understand this

1. From memory, list the three pipeline stages (parse, rewrite, toQuery), the context object active in each, and the kind of exception each stage throws on failure.
2. Given {"match":{"title":"Open Source"}} against a standard-analyzed text field, write the Lucene query you expect _validate/query?explain to return and justify each token.
3. Explain why {"term":{"title":"Open Source"}} on that same field almost certainly returns no hits, and rewrite it correctly two different ways.
4. Sketch the QuerySpec you'd pass from SearchPlugin.getQueries() for a new my_query, naming the three arguments and what each is for.
5. Explain the role of NamedXContentRegistry vs NamedWriteableRegistry for a query builder — which is used for JSON, which for transport, and why a custom query needs both.
6. Take a slow relevance query of the form bool.must:[range, match] and rewrite it for performance. Justify the change in terms of scoring and query caching.