Circuit Breakers and Memory

A search engine that lets a single greedy request OOM the JVM is a search engine that takes down a node for everyone. Circuit breakers are OpenSearch's defense: a hierarchy of accountants that track estimated heap usage per category (fielddata, aggregation request memory, in-flight network bytes, …) and throw a CircuitBreakingException before the allocation that would blow the heap, rather than letting the JVM die. Understanding the breakers — and the BigArrays machinery that feeds them — is essential both for diagnosing production "Data too large" errors and for writing engine code that participates in accounting instead of silently leaking heap.

This chapter dissects HierarchyCircuitBreakerService, the individual breakers, the real-memory parent breaker, and how BigArrays/PageCacheRecycler make heap allocations measurable. It connects to Aggregations (which trip the request breaker), DocValues and Fielddata (the fielddata breaker), and The Transport Layer (in-flight requests).

After this chapter you can:

• Name the breakers, what each tracks, and their default limits.
• Explain the difference between the summing parent breaker and the real-memory parent breaker.
• Read a CircuitBreakingException and _nodes/stats/breaker and find the culprit.
• Explain how aggregations and BigArrays reserve and release bytes against a breaker.

The breaker hierarchy

grep -n "class HierarchyCircuitBreakerService\|CircuitBreaker.FIELDDATA\|CircuitBreaker.REQUEST\|IN_FLIGHT\|ACCOUNTING\|PARENT" \
  server/src/main/java/org/opensearch/indices/breaker/HierarchyCircuitBreakerService.java
ls server/src/main/java/org/opensearch/core/common/breaker/ 2>/dev/null || \
  find . -path '*common/breaker/CircuitBreaker.java'

Note: The CircuitBreaker interface and ChildMemoryCircuitBreaker live in libs/core (org.opensearch.core.common.breaker), while the service that wires the hierarchy and reads settings lives in server (org.opensearch.indices.breaker.HierarchyCircuitBreakerService). Know both locations.

The two parent-breaker strategies

The parent breaker can enforce its limit two ways, controlled by indices.breaker.total.use_real_memory:

Real-memory mode is strictly better at catching the heap usage children don't account for (Lucene internals, object overhead, third-party allocations). It is why a request can be rejected even when the named breakers look fine — the parent saw real heap pressure. It pairs with G1 GC heuristics so it doesn't trip on transient post-allocation-pre-GC spikes.

grep -n "use_real_memory\|USE_REAL_MEMORY\|realMemoryUsage\|MemoryMXBean\|getHeapMemoryUsage\|G1" \
  server/src/main/java/org/opensearch/indices/breaker/HierarchyCircuitBreakerService.java

How a breaker actually gates an allocation

A breaker is not magic — code must call it. Before reserving heap, code calls breaker.addEstimateBytesAndMaybeBreak(bytes, label), which adds to the running total and, if the new total would exceed the limit, throws CircuitBreakingException instead of letting the allocation proceed. When the work finishes, code calls breaker.addWithoutBreaking(-bytes) to release the reservation.

sequenceDiagram
    participant A as Aggregator / BigArrays
    participant CB as request CircuitBreaker
    participant P as parent breaker
    A->>CB: addEstimateBytesAndMaybeBreak(n, "<agg>")
    CB->>P: check parent (sum or real heap) + this child
    alt over limit
        P-->>A: throw CircuitBreakingException (Data too large)
    else ok
        CB-->>A: reserved
        A->>A: allocate the bytes (e.g., grow a LongArray)
        Note over A: ... do work ...
        A->>CB: addWithoutBreaking(-n)  (release on close)
    end

grep -rn "addEstimateBytesAndMaybeBreak\|addWithoutBreaking\|CircuitBreakingException" \
  server/src/main/java/org/opensearch/ | head

Warning: If engine code reserves bytes but forgets to release them on the failure path (no try/finally), the breaker's accounting leaks — it keeps rejecting requests even though the heap is actually free. This is a real class of bug; always release reservations in finally/close.

BigArrays and PageCacheRecycler — measurable, recyclable heap

Aggregations and other hot paths don't new long[hugeSize]. They allocate through BigArrays, which (a) reserves the bytes against the request breaker before allocating, (b) hands out array-like abstractions (LongArray, DoubleArray, ByteArray, ObjectArray) backed by reusable pages, and (c) returns those pages to a PageCacheRecycler on release to avoid GC churn. This is the mechanism that makes aggregation memory both bounded (breaker-checked) and efficient (page-recycled).

grep -n "class BigArrays\|newLongArray\|newDoubleArray\|adjustBreaker\|class PageCacheRecycler" \
  server/src/main/java/org/opensearch/common/util/BigArrays.java \
  server/src/main/java/org/opensearch/common/util/PageCacheRecycler.java

When a terms aggregation over millions of buckets trips the request breaker, it is almost always a BigArrays allocation (a growing bucket-ordinal array) calling addEstimateBytesAndMaybeBreak and hitting the limit. The fix is rarely "raise the limit" — it's "ask for fewer buckets" (see Aggregations, size/shard_size/cardinality).

How aggregations trip the request breaker

flowchart TD
    Q[Big terms/cardinality agg] --> C[Aggregator grows BigArrays for buckets]
    C --> R[BigArrays reserves bytes on request breaker]
    R -->|under limit| OK[allocate, keep collecting]
    R -->|would exceed| X[CircuitBreakingException: Data too large for request]
    X --> RESP[search returns 503 / per-shard failure in _shards]

The exception is the intended outcome — a rejected query is infinitely better than a dead node. As a contributor, when you see a circuit-breaking test failure, the question is almost never "is the breaker wrong"; it's "is my code reserving an unbounded amount of memory."

grep -rn "REQUEST\|CircuitBreaker.REQUEST\|addEstimateBytesAndMaybeBreak" \
  server/src/main/java/org/opensearch/search/aggregations/ | head

Observability

# Per-node breaker state: limit, estimated usage, trip count, overhead
curl -s 'localhost:9200/_nodes/stats/breaker?pretty'

# Just the parent + fielddata estimates
curl -s 'localhost:9200/_nodes/stats/breaker?pretty' | grep -A6 -E 'parent|fielddata'

A climbing fielddata.estimated_size with tripped > 0 points at fielddata; a high request trip count points at aggregations; in_flight_requests trips point at huge bulk/ search payloads (transport).

Reading exercise

# 1. The service: breakers, limits, parent strategy
grep -n "registerBreaker\|childCircuitBreakers\|parentLimit\|checkParentLimit\|use_real_memory" \
  server/src/main/java/org/opensearch/indices/breaker/HierarchyCircuitBreakerService.java

# 2. The breaker interface + child impl (libs/core)
find . -path '*common/breaker/CircuitBreaker.java'
grep -rn "addEstimateBytesAndMaybeBreak\|addWithoutBreaking\|class ChildMemoryCircuitBreaker" \
  libs/core/src/main/java/org/opensearch/core/common/breaker/ 2>/dev/null

# 3. BigArrays reserving against the breaker
grep -n "adjustBreaker\|breaker\|newLongArray\|resize" \
  server/src/main/java/org/opensearch/common/util/BigArrays.java

# 4. Where aggs hold the request breaker
grep -rn "bigArrays\|CircuitBreaker.REQUEST\|breakerService" \
  server/src/main/java/org/opensearch/search/aggregations/AggregatorBase.java

Answer:

1. List the five breakers and one example of a workload that trips each.
2. Explain the difference between summing and real-memory parent strategies and why real-memory catches heap pressure the named breakers miss.
3. Trace one allocation through addEstimateBytesAndMaybeBreak: what is checked, what is thrown, and when is the reservation released?
4. Why does aggregation memory go through BigArrays instead of plain arrays? Name two things BigArrays provides (hint: one is the breaker, one is recycling).
5. A node keeps rejecting requests with "parent breaker tripped" but fielddata/request estimates look small. What is the likely explanation and which setting governs the behavior?
6. Describe the accounting-leak bug: how can a missing release cause persistent false CircuitBreakingExceptions, and what code pattern prevents it?

Common bugs and symptoms

Validation: prove you understand this

1. Draw the breaker hierarchy with the parent on top and the four children, and annotate each child with its default limit and one triggering workload.
2. Explain, with the MemoryMXBean in the picture, why the default real-memory parent breaker can reject a request that the summed child reservations say is fine.
3. Write the two breaker calls (reserve and release) an aggregator makes around a BigArrays allocation, and explain where the release must live to avoid an accounting leak.
4. Given _nodes/stats/breaker showing request.tripped: 42 and small fielddata, identify the subsystem at fault and the user-facing fix.
5. Explain why "increase indices.breaker.request.limit" is usually the wrong response to a tripped request breaker, and what the right response is.
6. Describe how BigArrays + PageCacheRecycler make aggregation memory both bounded and cheap, naming the breaker interaction and the recycling behavior.