Lab QE3: Query Cache and Optimization

Background

The intensive drew the three caches and stressed that they key on different things: the node query cache (LRUQueryCache + UsageTrackingQueryCachingPolicy) caches a DocIdSet bitset per cacheable clause, the shard request cache (IndicesRequestCache) caches a whole size:0/aggregation response, and the fielddata cache holds in-heap field values. This lab makes those caches observable and measurable. You will watch the query cache fill as a filter recurs, prove that the same predicate in must does not cache (because it scores), see the shard request cache serve an aggregation response verbatim, and finish on two rewrite-time optimizations — pre_filter_shard_size / can_match shard skipping — that cut work before any cache is even consulted.

Why This Matters for Contributors

"Why is my dashboard slow on every refresh when nothing changed?" is a caching question, and the answer is almost always one of: the predicate is in must (so it scores and never caches), the request isn't cacheable (it has a now-relative range, so the key changes every second), or the filter is too rare for UsageTrackingQueryCachingPolicy to bother. A contributor who can read _nodes/stats/indices/query_cache and _stats?level=shards diagnoses all three from the metrics. And anyone optimizing the query path has to know what is cacheable before proposing a change — caching a scoring clause is a correctness bug, not a speedup.

Prerequisites

• A running OpenSearch 3.x (docker run -p 9200:9200 -e discovery.type=single-node -e DISABLE_SECURITY_PLUGIN=true opensearchproject/opensearch:latest).
• curl, jq.
• You've read the intensive caching section and done Lab QE1.
• For deeper context: Tiered Caching and Search Execution.

Note: the node query cache only caches segments above a minimum size and only caches a clause after it has been seen frequently enough (UsageTrackingQueryCachingPolicy). To make caching observable in a lab you need (a) enough docs to clear the tiny-segment floor and (b) to repeat the query several times. We do both. "Cluster manager" (formerly master) plays no role here beyond a healthy cluster.

Step-by-Step Tasks

Step 1 — Build a big-enough index

The query cache skips tiny segments, so we index enough docs to matter and force a single shard so the stats are unambiguous.

H='-H content-type:application/json'
curl -s -XDELETE localhost:9200/qe3 >/dev/null
curl -s -XPUT localhost:9200/qe3 $H -d '{
  "settings": { "number_of_shards": 1, "number_of_replicas": 0, "refresh_interval": "1s" },
  "mappings": { "properties": {
    "status": { "type": "keyword" },
    "level":  { "type": "integer" },
    "msg":    { "type": "text" }
  }}
}' | jq .

# Bulk-load ~20k docs in batches (status cycles, level cycles).
for batch in $(seq 0 19); do
  awk -v b=$batch 'BEGIN{
    for(i=0;i<1000;i++){
      id=b*1000+i; st=(id%3==0)?"published":((id%3==1)?"draft":"archived");
      lvl=id%5;
      printf("{\"index\":{\"_id\":%d}}\n",id);
      printf("{\"status\":\"%s\",\"level\":%d,\"msg\":\"log line number %d about search\"}\n",st,lvl,id);
    }
  }' | curl -s "localhost:9200/qe3/_bulk" $H --data-binary @- >/dev/null
done
curl -s -XPOST 'localhost:9200/qe3/_refresh' >/dev/null
curl -s 'localhost:9200/qe3/_count' | jq '.count'   # ~20000

Step 2 — Baseline the query cache stats

qc() {
  curl -s 'localhost:9200/qe3/_stats/query_cache?level=shards' \
    | jq '.indices.qe3.primaries.query_cache
          | {hit_count, miss_count, cache_count, cache_size, total_count, evictions}'
}
qc   # everything near zero at the start

Step 3 — A filter clause caches; watch hits climb

Run the same filter query several times. The first builds the bitset (miss), later runs serve it (hits).

for i in $(seq 1 8); do
  curl -s 'localhost:9200/qe3/_search' $H -d '{
    "size": 0,
    "query": { "bool": { "filter": [ { "term": { "status": "published" } } ] } }
  }' >/dev/null
done
qc

You should see cache_count >= 1, cache_size >= 1, and hit_count climbing run over run. The term on status ran in filter context (no scoring), so UsageTrackingQueryCachingPolicy deemed it cacheable and LRUQueryCache stored the matching-docs bitset. The second and later runs skipped postings iteration entirely.

Note: it may take a few repetitions before hit_count moves — UsageTrackingQueryCachingPolicy requires the query to be seen frequently before caching. That delay is the heuristic working, not a bug. The exact frequency threshold varies by Lucene version; grep it: grep -n "MIN_FREQUENCY\|frequency\|shouldCache" lucene/core/src/java/org/apache/lucene/search/UsageTrackingQueryCachingPolicy.java

Step 4 — The same predicate in must does NOT cache

Now move the identical term from filter to must. It now scores, so the cached bitset (which has no scores) can't represent it — the policy refuses.

before=$(curl -s 'localhost:9200/qe3/_stats/query_cache' \
  | jq '.indices.qe3.primaries.query_cache.cache_count')

for i in $(seq 1 8); do
  curl -s 'localhost:9200/qe3/_search' $H -d '{
    "size": 0,
    "query": { "bool": { "must": [ { "term": { "status": "draft" } } ] } }
  }' >/dev/null
done

after=$(curl -s 'localhost:9200/qe3/_stats/query_cache' \
  | jq '.indices.qe3.primaries.query_cache.cache_count')
echo "cache_count before=$before after=$after  (no new entry for the scoring must clause)"

cache_count does not grow for the must clause: scoring clauses are not cacheable. This is the concrete reason the intensive tells you to put non-relevance predicates in filter. (A constant_score wrapper around the must would make it cacheable again — it strips the scoring.)

Step 5 — The shard request cache for size:0 / aggregations

The query cache is per-clause; the request cache stores the whole shard response for size:0/agg requests. It must be requested (or be the default for size:0) and the index must allow it.

rc() {
  curl -s 'localhost:9200/qe3/_stats/request_cache' \
    | jq '.indices.qe3.primaries.request_cache | {hit_count, miss_count, memory_size_in_bytes, evictions}'
}
rc   # baseline

AGG='{"size":0,"aggs":{"by_status":{"terms":{"field":"status"}}}}'
for i in $(seq 1 5); do
  curl -s 'localhost:9200/qe3/_search?request_cache=true' $H -d "$AGG" >/dev/null
done
rc

miss_count = 1 (first run computed the aggregation), then hit_count climbs: runs 2-5 returned the serialized shard response without touching the query phase at all. Confirm the cache is invalidated by a write:

curl -s -XPOST 'localhost:9200/qe3/_doc/999999?refresh=true' $H \
  -d '{"status":"published","level":1,"msg":"new doc"}' >/dev/null
curl -s 'localhost:9200/qe3/_search?request_cache=true' $H -d "$AGG" >/dev/null
rc   # one more miss: the refresh invalidated the entry

Warning: a request with a now-relative range ("gte":"now-1h") is not cached — the key changes every millisecond. Round the range ("now-1h/m"/"now/d") to make the request cacheable. This single fix is the most common dashboard-latency win in production.

Step 6 — Node-level view

The per-node rollup confirms the same numbers and shows total memory:

curl -s 'localhost:9200/_nodes/stats/indices/query_cache,request_cache' \
  | jq '.nodes[] | {name, query_cache: .indices.query_cache, request_cache: .indices.request_cache}'

Step 7 — Rewrite-time optimization: can_match shard skipping

Before any cache, the coordinator can skip whole shards that cannot match. With many shards and a selective range, pre_filter_shard_size triggers a can_match pre-phase: each shard cheaply rewrites the query against its min/max and, if it rewrites to match-none, is skipped entirely.

# A range that excludes most data on most shards is the canonical can_match win.
curl -s 'localhost:9200/qe3/_search?pre_filter_shard_size=1&pretty' $H -d '{
  "size": 0,
  "query": { "range": { "level": { "gte": 99, "lte": 100 } } }
}' | jq '{total_shards: ._shards.total, skipped: ._shards.skipped, successful: ._shards.successful}'

On a single-shard lab index skipped may be 0 (there's nothing to skip to), but the _shards.skipped field is the metric to watch on a real multi-shard index — it is the count of shards the can_match phase eliminated before the query phase. This connects directly to the per-shard rewrite (range → MatchNoneQueryBuilder) from the intensive and the distributed flow in Search Execution.

# Profile a filter query to confirm a cached clause shows ~zero build_scorer on re-run:
curl -s 'localhost:9200/qe3/_search' $H -d '{
  "profile": true, "size": 0,
  "query": { "bool": { "filter": [ { "term": { "status": "published" } } ] } }
}' | jq '.profile.shards[0].searches[0].query[0]
         | {type, build_scorer: .breakdown.build_scorer, score: .breakdown.score}'

The filter's score breakdown is 0 (filter context never scores), and on a cache hit build_scorer is dramatically lower than the first run — the bitset is reused instead of rebuilt.

Deliverables

• _stats/query_cache before and after the repeated filter query, showing cache_count/cache_size/hit_count rising.
• Proof that the same predicate as must does not add a cache_count entry, with one sentence on why (scoring clauses aren't cacheable).
• _stats/request_cache showing one miss then hits for the size:0 aggregation, plus the extra miss after a write invalidated it.
• A short note on pre_filter_shard_size/can_match: what _shards.skipped means and why a now-relative range defeats the request cache.

Expected Output

# Step 3 (filter, after several runs)
{ "hit_count": 6, "miss_count": 2, "cache_count": 1, "cache_size": 1, "evictions": 0 }

# Step 4 (must)
cache_count before=1 after=1   (no new entry for the scoring must clause)

# Step 5 (request cache)
{ "hit_count": 0, "miss_count": 0, ... }          # baseline
{ "hit_count": 4, "miss_count": 1, ... }          # 1 compute, 4 served
{ "hit_count": 4, "miss_count": 2, ... }          # +1 miss after the write

Troubleshooting

Stretch Goals

• constant_score rescues caching. Wrap the Step 4 must term in constant_score and show cache_count now grows — scoring stripped, bitset cacheable again.
• now vs rounded range. Send {"range":{"@timestamp":{"gte":"now-1h"}}} and the rounded now-1h/m variant five times each; show only the rounded one gets request-cache hits.
• Eviction under pressure. Shrink indices.queries.cache.size (node setting), push many distinct filters, and watch evictions climb in query_cache stats.
• Tiered spill. Read Tiered Caching and enable the disk tier for the request cache; observe a hit served from disk after the heap tier evicts it.
• Fielddata trap. Sort/agg on a text field (not keyword) and watch the fielddata cache fill in _stats/fielddata — then fix it by using a keyword sub-field and doc-values.

Coding Exercises

You measured the caches with _stats; now write code that asserts their behavior, so a regression that breaks caching (or, worse, caches a scoring clause) fails CI. The pattern is an OpenSearchIntegTestCase that runs a query repeatedly and reads the same stats you read by hand (client().admin().indices().prepareStats(...) → getQueryCache() / getRequestCache()). Find a model with rg -l "query_cache|QueryCacheStats|IndicesRequestCacheIT" server/.

1. (warm-up) Assert a filter clause caches. Write an integration test that indexes enough docs to clear the tiny-segment floor, runs the same bool { filter: [ term ] } query several times, and asserts cacheCount/hitCount rise across runs (read via getQueryCache()). This is Step 3 as a test.

2. (core) Assert a must clause does NOT cache. Add a test that runs the identical predicate in must, captures cacheCount before/after, and asserts it is unchanged — encoding the Step 4 rule that scoring clauses aren't cacheable. Then add a constant_score-wrapped variant and assert it caches again. Two asserts, one law: scoring kills caching, stripping scores restores it.

3. (core) Assert request-cache hit + write invalidation. Write a test that runs a size:0 aggregation with request_cache=true N times (1 miss, N-1 hits via getRequestCache()), then indexes a doc + refresh and asserts exactly one additional miss — proving the refresh invalidated the entry (Step 5).

4. (core) A can_match shard-skipping test. On a multi-shard index where each shard holds a disjoint level range, run a selective range query with pre_filter_shard_size=1 and assert searchResponse.getSkippedShards() > 0 — the executable form of Step 7's _shards.skipped. Find the rewrite that makes it match-none with rg -n "MatchNoneQueryBuilder|canMatch|CanMatch" server/.../search/.

5. (advanced) Advanced challenge — a now-range caching experiment with a test. Build an OpenSearchIntegTestCase that issues, five times each, (a) a request with a raw now-relative range ("gte":"now-1h") and (b) the rounded variant ("now-1h/m"), and asserts via getRequestCache() that only the rounded one accrues hits (the raw key changes every ms, so it always misses). Add a second assertion using getQueryCache() for the eviction path: shrink indices.queries.cache.size (rg -n "indices.queries.cache.size|INDICES_CACHE_QUERY_SIZE" server/), push many distinct filters, and assert evictions climbs. Deliverable: one test proving (i) rounding fixes the most common dashboard-latency bug and (ii) the LRU evicts under pressure — the two cache facts most worth pinning, cross-linked to Tiered Caching.

Issues to Practice On

Caching questions ("slow on every refresh," "filter not caching") are common triage. Practice on opensearch-project/OpenSearch.

Labels drift; confirm with gh label list --repo opensearch-project/OpenSearch (look for Search, Performance, Caching if present).

Representative patterns. (1) "Dashboard slow on every refresh though data is unchanged." — nearly always a now-relative range defeating the request cache, or a predicate in must instead of filter. Reproduce by reading _stats/request_cache and _stats/query_cache exactly as in this lab. (2) "Filter never caches." — either the segment is below the floor or UsageTrackingQueryCachingPolicy hasn't seen it enough; reproduce, then locate the policy threshold with rg. Approach: reproduce → locate via rg → fix → test → PR with CHANGELOG + DCO.

Planted-bug drill. In the bool-query translation (or a local test harness), force a filter clause to be added in scoring context (e.g. wrap it as a scoring must before it reaches the cache, or flip the Occur as in QE1's drill). Run your exercise-1/2 tests: the "filter caches, must doesn't" assertions now disagree — watch which goes red. This demonstrates that caching a scoring clause is a correctness bug, not a speedup. Revert, then keep the asserted cacheCount-unchanged-for-must test as the guard.

Etiquette: claim the issue first, reproduce before fixing, and every PR needs a test + CHANGELOG entry + DCO Signed-off-by (git commit -s). See community interaction and the prepare-a-PR lab.

Validation / Self-check

• You can read _nodes/stats/indices/query_cache and _stats?level=shards and explain every field (hit/miss/cache_count/cache_size/evictions).
• You can state, with the metrics to prove it, why filter caches and the same predicate in must does not, and how constant_score changes that.
• You can distinguish the node query cache (per-clause bitset) from the shard request cache (whole size:0/agg response) by what each keys on.
• You can explain pre_filter_shard_size/can_match shard skipping and why a now-relative range silently defeats the request cache.

Back to The Query Engine — Intensive. Related: Tiered Caching, Search Execution.