Circuit Breakers and Memory

A search engine that lets a single greedy request OOM the JVM is a search engine that takes down a node for everyone. Circuit breakers are OpenSearch's defense: a hierarchy of accountants that track estimated heap usage per category (fielddata, aggregation request memory, in-flight network bytes, …) and throw a CircuitBreakingException before the allocation that would blow the heap, rather than letting the JVM die. Understanding the breakers — and the BigArrays machinery that feeds them — is essential both for diagnosing production "Data too large" errors and for writing engine code that participates in accounting instead of silently leaking heap.

This chapter dissects HierarchyCircuitBreakerService, the individual breakers, the real-memory parent breaker, and how BigArrays/PageCacheRecycler make heap allocations measurable. It connects to Aggregations (which trip the request breaker), DocValues and Fielddata (the fielddata breaker), and The Transport Layer (in-flight requests).

After this chapter you can:

• Name the breakers, what each tracks, and their default limits.
• Explain the difference between the summing parent breaker and the real-memory parent breaker.
• Read a CircuitBreakingException and _nodes/stats/breaker and find the culprit.
• Explain how aggregations and BigArrays reserve and release bytes against a breaker.

The breaker hierarchy

grep -n "class HierarchyCircuitBreakerService\|CircuitBreaker.FIELDDATA\|CircuitBreaker.REQUEST\|IN_FLIGHT\|ACCOUNTING\|PARENT" \
  server/src/main/java/org/opensearch/indices/breaker/HierarchyCircuitBreakerService.java
ls server/src/main/java/org/opensearch/core/common/breaker/ 2>/dev/null || \
  find . -path '*common/breaker/CircuitBreaker.java'

Note: The CircuitBreaker interface, CircuitBreakingException, and NoopCircuitBreaker live in libs/core (org.opensearch.core.common.breaker). The concrete ChildMemoryCircuitBreaker and the service that wires the hierarchy and reads settings both live in server — org.opensearch.common.breaker.ChildMemoryCircuitBreaker and org.opensearch.indices.breaker.HierarchyCircuitBreakerService. Know all three locations.

The two parent-breaker strategies

The parent breaker can enforce its limit two ways, controlled by indices.breaker.total.use_real_memory:

Real-memory mode is strictly better at catching the heap usage children don't account for (Lucene internals, object overhead, third-party allocations). It is why a request can be rejected even when the named breakers look fine — the parent saw real heap pressure. It pairs with G1 GC heuristics so it doesn't trip on transient post-allocation-pre-GC spikes.

grep -n "use_real_memory\|USE_REAL_MEMORY\|realMemoryUsage\|MemoryMXBean\|getHeapMemoryUsage\|G1" \
  server/src/main/java/org/opensearch/indices/breaker/HierarchyCircuitBreakerService.java

How a breaker actually gates an allocation

A breaker is not magic — code must call it. Before reserving heap, code calls breaker.addEstimateBytesAndMaybeBreak(bytes, label), which adds to the running total and, if the new total would exceed the limit, throws CircuitBreakingException instead of letting the allocation proceed. When the work finishes, code calls breaker.addWithoutBreaking(-bytes) to release the reservation.

sequenceDiagram
    participant A as Aggregator / BigArrays
    participant CB as request CircuitBreaker
    participant P as parent breaker
    A->>CB: addEstimateBytesAndMaybeBreak(n, "<agg>")
    CB->>P: check parent (sum or real heap) + this child
    alt over limit
        P-->>A: throw CircuitBreakingException (Data too large)
    else ok
        CB-->>A: reserved
        A->>A: allocate the bytes (e.g., grow a LongArray)
        Note over A: ... do work ...
        A->>CB: addWithoutBreaking(-n)  (release on close)
    end

grep -rn "addEstimateBytesAndMaybeBreak\|addWithoutBreaking\|CircuitBreakingException" \
  server/src/main/java/org/opensearch/ | head

Warning: If engine code reserves bytes but forgets to release them on the failure path (no try/finally), the breaker's accounting leaks — it keeps rejecting requests even though the heap is actually free. This is a real class of bug; always release reservations in finally/close.

BigArrays and PageCacheRecycler — measurable, recyclable heap

Aggregations and other hot paths don't new long[hugeSize]. They allocate through BigArrays, which (a) reserves the bytes against the request breaker before allocating, (b) hands out array-like abstractions (LongArray, DoubleArray, ByteArray, ObjectArray) backed by reusable pages, and (c) returns those pages to a PageCacheRecycler on release to avoid GC churn. This is the mechanism that makes aggregation memory both bounded (breaker-checked) and efficient (page-recycled).

grep -n "class BigArrays\|newLongArray\|newDoubleArray\|adjustBreaker\|class PageCacheRecycler" \
  server/src/main/java/org/opensearch/common/util/BigArrays.java \
  server/src/main/java/org/opensearch/common/util/PageCacheRecycler.java

When a terms aggregation over millions of buckets trips the request breaker, it is almost always a BigArrays allocation (a growing bucket-ordinal array) calling addEstimateBytesAndMaybeBreak and hitting the limit. The fix is rarely "raise the limit" — it's "ask for fewer buckets" (see Aggregations, size/shard_size/cardinality).

How aggregations trip the request breaker

flowchart TD
    Q[Big terms/cardinality agg] --> C[Aggregator grows BigArrays for buckets]
    C --> R[BigArrays reserves bytes on request breaker]
    R -->|under limit| OK[allocate, keep collecting]
    R -->|would exceed| X[CircuitBreakingException: Data too large for request]
    X --> RESP[search returns 503 / per-shard failure in _shards]

The exception is the intended outcome — a rejected query is infinitely better than a dead node. As a contributor, when you see a circuit-breaking test failure, the question is almost never "is the breaker wrong"; it's "is my code reserving an unbounded amount of memory."

grep -rn "REQUEST\|CircuitBreaker.REQUEST\|addEstimateBytesAndMaybeBreak" \
  server/src/main/java/org/opensearch/search/aggregations/ | head

Walked example — a parent-breaker trip under real-memory accounting

The most confusing breaker exception in the field is the one where the named breakers look fine but the request dies anyway. Here is exactly how that happens, traced from the allocation to the message.

The classes and method names are exact (run the greps); the scenario numbers are illustrative.

The scenario

A node has a 1 GiB heap. Real-memory accounting is on (indices.breaker.total.use_real_memory defaults to true), so the parent limit is 95 % of heap ≈ 973 MB:

grep -n "use_real_memory\|\"95%\"\|\"70%\"\|TOTAL_CIRCUIT_BREAKER_LIMIT_SETTING" \
  server/src/main/java/org/opensearch/indices/breaker/HierarchyCircuitBreakerService.java

A large terms aggregation is collecting. Live heap — Lucene buffers, the segment reader, other in-flight work — is already sitting near 940 MB, none of it charged to the request breaker. The aggregator now needs to grow its bucket-ordinal array by 48 MB.

The call path

The growth goes through BigArrays, which reserves before it allocates:

grep -n "adjustBreaker\|addEstimateBytesAndMaybeBreak\|<reused_arrays>" \
  server/src/main/java/org/opensearch/common/util/BigArrays.java

// BigArrays.adjustBreaker(delta, isDataAlreadyCreated=false), delta = 48 MB
breaker.addEstimateBytesAndMaybeBreak(delta, "<reused_arrays>");

That lands in the request ChildMemoryCircuitBreaker, which does two checks:

grep -n "addEstimateBytesAndMaybeBreak\|checkParentLimit\|addWithoutBreaking" \
  server/src/main/java/org/opensearch/common/breaker/ChildMemoryCircuitBreaker.java

1. Its own limit. The request breaker's cap is 60 % of heap ≈ 614 MB. Its current reservation is tiny (the agg has only just started), so 48 MB clears easily — the child breaker is not the one that trips.
2. The parent limit. The child then calls parent.checkParentLimit((long)(bytes * overhead), "<reused_arrays>").

Where the trip happens

grep -n "checkParentLimit\|memoryUsed\|currentMemoryUsage\|trackRealMemoryUsage\|MEMORY_MX_BEAN" \
  server/src/main/java/org/opensearch/indices/breaker/HierarchyCircuitBreakerService.java

With real-memory mode on, memoryUsed(newBytesReserved) does not sum the children — it reads live heap from the MemoryMXBean and adds the pending reservation:

final long current = currentMemoryUsage();                         // ~940 MB, measured from the JVM
return new MemoryUsage(current, current + newBytesReserved, ...);   // totalUsage ≈ 988 MB

totalUsage ≈ 988 MB is above the ≈ 973 MB parent limit. The service double-checks with its G1 over-limit strategy (which can trigger a GC and re-measure to avoid tripping on a transient post-allocation spike); if heap is genuinely that high, it increments parentTripCount and throws. This is the answer to "why did it die when request looked small": the parent saw real heap the children never accounted for.

The exception message shape

The [parent] message is assembled in checkParentLimit and is the exact shape you will see in the _shards failure or the node log (the extra real usage / new bytes reserved fields appear only in real-memory mode):

[parent] Data too large, data for [<reused_arrays>] would be [1035993088/988mb],
 which is larger than the limit of [1020054732/972.8mb],
 real usage: [985661440/940mb], new bytes reserved: [50331648/48mb],
 usages [request=..., fielddata=..., in_flight_requests=..., accounting=...]

Read the fields left to right: the offending label (<reused_arrays>, i.e. a BigArrays growth), the projected would be total, the limit, then — the tell for real-memory mode — real usage (measured heap) and new bytes reserved (this allocation), followed by the per-child usages. When real usage is present and high while the child usages are small, you are looking at a real-memory parent trip, not a runaway aggregation.

The rollback you must not forget

Because the parent tripped, the child breaker had already incremented its own total in step 1, so addEstimateBytesAndMaybeBreak rolls it back before rethrowing:

} catch (CircuitBreakingException e) {
    this.addWithoutBreaking(-bytes);   // undo the child reservation; the alloc never happened
    throw e;
}

BigArrays.adjustBreaker was called with isDataAlreadyCreated = false, so it does not re-add the delta — the array was never created. This is the correct, leak-free pattern, and it is exactly what your own code must replicate: reserve, and if you then fail, release before you rethrow. Skip that release and you get the persistent-false-trip bug from the table below.

Observability

# Per-node breaker state: limit, estimated usage, trip count, overhead
curl -s 'localhost:9200/_nodes/stats/breaker?pretty'

# Just the parent + fielddata estimates
curl -s 'localhost:9200/_nodes/stats/breaker?pretty' | grep -A6 -E 'parent|fielddata'

A climbing fielddata.estimated_size with tripped > 0 points at fielddata; a high request trip count points at aggregations; in_flight_requests trips point at huge bulk/ search payloads (transport).

Reading exercise

# 1. The service: breakers, limits, parent strategy
grep -n "registerBreaker\|childCircuitBreakers\|parentLimit\|checkParentLimit\|use_real_memory" \
  server/src/main/java/org/opensearch/indices/breaker/HierarchyCircuitBreakerService.java

# 2. The breaker interface (libs/core) and the concrete child impl (server)
find . -path '*core/common/breaker/CircuitBreaker.java'
grep -n "addEstimateBytesAndMaybeBreak\|addWithoutBreaking\|class ChildMemoryCircuitBreaker\|checkParentLimit" \
  server/src/main/java/org/opensearch/common/breaker/ChildMemoryCircuitBreaker.java

# 3. BigArrays reserving against the breaker
grep -n "adjustBreaker\|breaker\|newLongArray\|resize" \
  server/src/main/java/org/opensearch/common/util/BigArrays.java

# 4. Where aggs hold the request breaker
grep -rn "bigArrays\|CircuitBreaker.REQUEST\|breakerService" \
  server/src/main/java/org/opensearch/search/aggregations/AggregatorBase.java

Answer:

1. List the five breakers and one example of a workload that trips each.
2. Explain the difference between summing and real-memory parent strategies and why real-memory catches heap pressure the named breakers miss.
3. Trace one allocation through addEstimateBytesAndMaybeBreak: what is checked, what is thrown, and when is the reservation released?
4. Why does aggregation memory go through BigArrays instead of plain arrays? Name two things BigArrays provides (hint: one is the breaker, one is recycling).
5. A node keeps rejecting requests with "parent breaker tripped" but fielddata/request estimates look small. What is the likely explanation and which setting governs the behavior?
6. Describe the accounting-leak bug: how can a missing release cause persistent false CircuitBreakingExceptions, and what code pattern prevents it?

Common bugs and symptoms

Validation: prove you understand this

1. Draw the breaker hierarchy with the parent on top and the four children, and annotate each child with its default limit and one triggering workload.
2. Explain, with the MemoryMXBean in the picture, why the default real-memory parent breaker can reject a request that the summed child reservations say is fine.
3. Write the two breaker calls (reserve and release) an aggregator makes around a BigArrays allocation, and explain where the release must live to avoid an accounting leak.
4. Given _nodes/stats/breaker showing request.tripped: 42 and small fielddata, identify the subsystem at fault and the user-facing fix.
5. Explain why "increase indices.breaker.request.limit" is usually the wrong response to a tripped request breaker, and what the right response is.
6. Describe how BigArrays + PageCacheRecycler make aggregation memory both bounded and cheap, naming the breaker interaction and the recycling behavior.