Tiered Caching

OpenSearch has always had a request cache: a per-shard cache of the results of expensive, repeatable reads — most importantly aggregation results on indices that do not change between requests (think time-series dashboards hitting yesterday's data over and over). It lives on the JVM heap, and that is its ceiling. Heap is the scarcest resource on a data node; a request cache big enough to hold a busy dashboard's whole working set competes directly with everything else that wants heap, and the circuit breakers will not let it win.

Tiered caching breaks that ceiling. Instead of one on-heap cache that evicts to nowhere, it makes the cache a hierarchy: a small, fast on-heap tier backed by a much larger on-disk tier. An entry evicted from heap does not vanish — it spills over to disk, where a cache hit is still vastly cheaper than recomputing the aggregation. The data structure that does this is TieredSpilloverCache, and the whole thing is built on a deliberately pluggable cache SPI so the disk tier (today, ehcache) can be swapped.

This chapter explains the hierarchy, the SPI that makes it pluggable, the disk tier, the serialization that crossing the heap/disk boundary forces, and how it all relates to the request cache and the circuit breakers you already know. It does not re-derive the request cache or the search path — those are search execution and the existing cache code; this is the layer added beneath the request cache.

The design and benchmark issues to read alongside this chapter: [Proposal] Tiered caching — OpenSearch #10024 and [META] performance benchmark for tiered caching — #11464.

After this chapter you can:

• Describe the on-heap → disk hierarchy and what spillover means precisely.
• Name TieredSpilloverCache, the ICache/CacheService SPI, and the cache-ehcache disk tier, and find each in the tree.
• Explain why crossing to disk forces key/value serialization and what that costs.
• Reason about eviction, invalidation, the relevant settings, and the interaction with the request cache and circuit breakers.

Note: This is the infrastructure under the request cache, exposed as a general caching SPI. The same machinery is intended to back other on-heap caches over time, not just the request cache. Read it as "OpenSearch grew a pluggable, tiered cache framework, and the request cache was the first consumer."

The problem, precisely

The request cache is heap-bound, and heap is precious. That produces a sharp, unsatisfying trade-off:

• Make the request cache small → high miss rate on dashboards with a large working set → expensive aggregations recomputed constantly.
• Make it large → it steals heap from the rest of the node, raising GC pressure and tripping the parent circuit breaker.

There is no good on-heap-only answer for a working set that is bigger than you want to keep on heap but much smaller and cheaper to fetch from disk than to recompute. That is exactly the gap a second, on-disk tier fills. Disk is slower than heap but orders of magnitude faster than re-running the aggregation, and it does not consume the resource (heap) that you are trying to protect.

flowchart LR
    Q["repeatable read<br/>(e.g. dashboard agg)"] --> H{"on-heap tier hit?"}
    H -->|hit| FAST["return (fastest)"]
    H -->|miss| D{"on-disk tier hit?"}
    D -->|hit| MED["deserialize + return<br/>(slower than heap,<br/>far faster than recompute)"]
    D -->|miss| COMP["recompute the result<br/>(slowest), then cache"]

The bet tiered caching makes: most "misses" in a heap-only cache are recently evicted entries that are still useful. Catch them on disk and you convert recomputes into cheap disk reads.

The cache hierarchy and TieredSpilloverCache

The core class is TieredSpilloverCache. It implements the same cache interface as any single-tier cache, but internally it holds two caches — an on-heap tier and an on-disk tier — and routes between them.

The behavior, as a contract:

• get(key) — check the on-heap tier first. On a hit, return. On a miss, check the on-disk tier. A disk hit is returned (and, depending on policy, may be promoted back to heap). A miss in both is a real cache miss.
• put(key, value) — write to the on-heap tier.
• eviction from heap — when the on-heap tier evicts an entry (it is full), the entry spills over into the on-disk tier rather than being discarded. This is the word in the class name and the whole point of the design.
• eviction from disk — the disk tier has its own bound; when it is full, its evictions are the ones that truly leave the cache.

flowchart TD
    PUT["put(key, value)"] --> HEAP["on-heap tier<br/>(fast, small, bounded by heap budget)"]
    HEAP -->|evicted (heap full)| SPILL["spill over"]
    SPILL --> DISK["on-disk tier<br/>(ehcache, larger, serialized)"]
    DISK -->|evicted (disk full)| GONE["leaves the cache (true eviction)"]
    GET["get(key)"] --> HEAP
    HEAP -->|miss| DISK
    DISK -->|hit| BACK["deserialize → return<br/>(optionally promote to heap)"]

cd ~/src/OpenSearch
# Find the spillover cache and its tier wiring.
grep -rln "TieredSpilloverCache" server/src/main/java modules plugins | head
grep -rn "spillover\|onHeap\|getOnHeapCache\|getDiskCache\|evict" \
  $(grep -rl "class TieredSpilloverCache" server modules plugins) | head -20

Warning: "spill over to disk" is not free. Every entry that crosses the heap→disk boundary must be serialized to bytes (and deserialized on a disk hit). A cache whose values are huge or whose serializer is slow can spend more on serialization than it saves on recompute. The benchmark issue #11464 exists precisely to measure where that trade pays off and where it does not.

The pluggable cache SPI: ICache and CacheService

Tiered caching did not just add a class; it added a caching framework. The two tiers and the spillover logic sit behind a deliberately pluggable Service Provider Interface so that the disk implementation is not hard-wired into core.

The pieces, conceptually (grep to confirm exact names/packages in your checkout — they live under the cache common package):

# The SPI lives under the common cache package. Walk it.
ls server/src/main/java/org/opensearch/common/cache/
grep -rn "interface ICache\|interface Factory\|class CacheService\|interface CachePlugin\|enum CacheType\|enum CacheStoreType" \
  server/src/main/java/org/opensearch/common/cache/ | head -20

The shape to internalize: TieredSpilloverCache is itself an ICache. It is a cache built out of two caches. That recursion is what makes the framework clean — the request cache asks CacheService for "the cache for INDICES_REQUEST_CACHE," and what it gets back might be a plain on-heap cache or a tiered spillover cache, depending on settings, and it does not care which.

flowchart TD
    RC["IndicesRequestCache<br/>(consumer)"] --> CS["CacheService.createCache(CacheType.INDICES_REQUEST_CACHE)"]
    CS -->|tiered enabled?| TSC["TieredSpilloverCache : ICache"]
    CS -->|tiered disabled| OHC["OpenSearchOnHeapCache : ICache"]
    TSC --> T1["on-heap ICache"]
    TSC --> T2["disk ICache (ehcache)"]

The disk tier: cache-ehcache

The on-disk tier ships as a plugin, not core code: cache-ehcache. It wraps Ehcache's disk store and exposes it as an ICache implementation that CacheService can hand back as the lower tier.

Why a plugin and not core:

• It pulls in an external dependency (Ehcache + its licensing surface — relevant to licensing). Keeping it as a plugin keeps that dependency out of the core distribution for clusters that do not want a disk tier.
• It proves the SPI works. If the disk tier is just "another CachePlugin that registers an ICache.Factory," then the framework is genuinely pluggable and a future tier (a different disk store, a remote cache) is the same shape.

# The ehcache disk-tier plugin.
ls plugins/cache-ehcache/ 2>/dev/null || find . -type d -name "*ehcache*"
grep -rn "Ehcache\|EhcacheDiskCache\|CachePlugin\|getCacheFactoryMap\|ICache.Factory" \
  plugins/cache-ehcache/src/main/java 2>/dev/null | head -20

The disk tier is responsible for the things heap caches never had to think about: file management, on-disk size accounting, durability semantics (a cache is not a source of truth — see invalidation below), and the serialization boundary.

Key/value serialization across the boundary

An on-heap cache stores Java object references. A disk cache cannot — it stores bytes. So the SPI carries a serialization contract: to put an entry in the disk tier you must turn its key and value into bytes, and a disk hit must turn them back.

This is not a footnote; it shapes the whole design:

• Keys (for the request cache, a composite of shard + the request's cache key) must serialize deterministically — the same logical request must produce the same bytes, or you get false misses.
• Values (the cached result bytes) are serialized aggregation/query results. They are already close to a serialized form, which is part of why the request cache was a natural first consumer.
• The disk tier deals in BytesReference/byte arrays; a Serializer<T, byte[]>-style abstraction converts between the consumer's objects and bytes. Grep for the serializer interface to see the exact contract.

grep -rn "Serializer\|serialize\|BytesReference\|byte\[\]" \
  server/src/main/java/org/opensearch/common/cache/ | grep -i serial | head
grep -rn "Serializer\|serialize\|deserialize" \
  plugins/cache-ehcache/src/main/java 2>/dev/null | head

Note: serialization cost is the variable that decides whether tiered caching wins. A heap hit is a pointer follow; a disk hit is a disk read plus deserialization. The design pays off when (recompute cost) ≫ (disk read + deserialize). For cheap-to-recompute results, the disk tier can be net negative — which is the entire reason #11464 is a benchmark meta-issue, not a feature issue.

Eviction and invalidation

Two different mechanisms, often confused:

• Eviction is capacity management — the cache is full, something has to go. In the tiered cache, on-heap eviction spills to disk; disk eviction is true removal. Each tier has its own size policy (entry count and/or byte size).
• Invalidation is correctness management — the cached entry is no longer valid and must be dropped regardless of capacity. For the request cache, the classic trigger is a refresh that changes the shard's contents: a cached aggregation result is only valid as long as the underlying segments are unchanged. When a shard refreshes (see refresh/flush/merge), the entries keyed to the old reader are invalidated.

In a tiered cache, invalidation must reach both tiers — invalidating only the heap tier while a stale copy lingers on disk is a correctness bug. The cleanup that removes invalidated entries (and the keys that map to them) has to walk the hierarchy.

grep -rn "invalidate\|invalidateAll\|cleanCache\|CleanupKey\|keysToClean\|refreshKey" \
  server/src/main/java/org/opensearch/common/cache server/src/main/java/org/opensearch/indices | head -20

Warning: the most dangerous tiered-cache bug class is a stale entry surviving on disk after the heap tier was invalidated. Single-tier, invalidation was "drop it from the map." Tiered, you must invalidate through to disk. When you review or write invalidation code, prove the disk tier is reached — a unit test that only checks the heap tier is a test that passes while the bug ships.

Settings

The exact keys evolve; grep your checkout. The shape is: a master switch per cache type to enable tiering, a choice of disk store, and per-tier size bounds.

# Find the real setting keys and defaults in your checkout.
grep -rn "tiered_spillover\|TIERED\|store.name\|disk.store\|onheap.store" \
  server/src/main/java/org/opensearch/common/cache \
  plugins/cache-ehcache/src/main/java 2>/dev/null | head -20

# Enable the tiered store for the request cache (key names vary — confirm above first).
curl -s -XPUT 'localhost:9200/_cluster/settings' -H 'Content-Type: application/json' -d '{
  "persistent": {
    "indices.requests.cache.store.name": "tiered_spillover"
  }
}'
# Then exercise a cacheable agg twice and watch request_cache stats.
curl -s 'localhost:9200/myindex/_stats/request_cache?pretty' | grep -A6 request_cache

Relationship to the request cache and circuit breakers

Two existing subsystems you must hold in your head while working here.

The request cache is the first and primary consumer. Tiered caching does not replace it; it gives it a bigger, cheaper backing store. The request cache still decides what is cacheable (only size: 0 / non-scoring repeatable requests on unchanged shards, by default) and still keys entries the same way; tiering changes only where evicted entries go. If you are debugging "my aggregation isn't being cached," that is request-cache eligibility logic, not tiered caching.

The circuit breakers (circuit breakers and memory) guard the JVM heap. The on-heap tier is inside that guarded space — it counts against heap and the breakers can still trip on it. The on-disk tier is outside heap; the heap circuit breaker does not see it (which is the upside — you got cache capacity that does not pressure heap), but it is also not free: disk space and serialization CPU are real, just guarded by the cache's own size bounds rather than a breaker. This is structurally the same lesson as k-NN's off-heap memory: memory that lives outside the heap needs its own accounting, because the heap-oriented tools cannot see it.

flowchart TD
    subgraph heap["JVM heap (guarded by circuit breakers)"]
        OH["on-heap cache tier"]
    end
    subgraph disk["disk (guarded by the cache's own size bounds)"]
        DK["ehcache disk tier"]
    end
    OH -->|spill| DK
    CB["parent / request circuit breaker"] -. sees .-> OH
    CB -. blind to .-> DK

Common bugs and symptoms

Validation: prove you understand this

1. Draw the on-heap → disk hierarchy and define spillover precisely. What happens to an entry evicted from the heap tier vs evicted from the disk tier?
2. Explain why TieredSpilloverCache is itself an ICache, and what that buys the request cache (which does not know whether its cache is tiered).
3. Name the SPI pieces — ICache, ICache.Factory, CacheService, CachePlugin, CacheType — and say which one a plugin like cache-ehcache implements to contribute a disk tier.
4. Why does crossing to the disk tier force serialization of keys and values, and what is the failure mode of a non-deterministic key serializer?
5. Distinguish eviction from invalidation. Why is "invalidate the heap tier only" a correctness bug, and what triggers request-cache invalidation in the first place?
6. Explain the relationship to the circuit breakers: which tier they guard, which they are blind to, and why that is the same lesson as off-heap native memory in k-NN.
7. Read #10024 and #11464. State the one variable that decides whether tiered caching is a win, and explain why the benchmark issue is separate from the proposal.

When you can do all seven, revisit circuit breakers and memory with fresh eyes (the heap/off-heap accounting story is the same shape) and search execution to place the request cache in the read path it short-circuits.

Next: Star-tree aggregations — another "precompute to buy latency" feature — or back to real issues and RFCs.