Project 7: A Search Backpressure Signal

A search cluster under load fails in one of two ways. The good way: it rejects the marginal request early, with a clear 429, and keeps serving the rest. The bad way: it accepts everything, every search threadpool queue fills, heap climbs, GC stalls, the cluster-manager loses contact with nodes, and the whole cluster cascades into a coordinated outage that takes an hour to recover from. The difference between those two failure modes is backpressure: the machinery that watches resource consumption per task and cancels or rejects work before the node tips over.

OpenSearch has a search-backpressure framework that tracks per-task resource usage (heap, CPU, elapsed time), identifies the search tasks responsible for strain, and cancels them under a node-level duress signal. It is real, it is tunable, and — like all admission-control systems — it is full of heuristics that are almost right and signals that are almost sensitive enough. This project asks you to add or tune a backpressure signal: a new resource tracker, a better cancellation-eligibility heuristic, or a shard-indexing-pressure analogue — starting from an observability slice you can scope in a weekend and ending at a tuned, tested, RFC-anchored change.

Note: Read Backpressure and Admission Control and Threadpools and Concurrency before you start. This brief assumes you know what the search threadpool is, what a CancellableTask is, how the task framework tracks resource consumption, and the difference between node duress and a task being the cause of it. It will not re-derive them.

This is a "Maybe — RFC first" project. Admission control changes cluster failure behaviour, so the bar for landing a new signal upstream is high and starts with a design discussion. The mergeable slice here is observability and a well-justified, measured heuristic tweak — not a brand-new rejection policy dropped on the maintainers.

Problem & motivation

Backpressure is hard because it is a control system, and control systems fail by being wrong in either direction:

• Too insensitive and the cluster cascades. If the trackers underestimate the cost of an expensive search (a deep aggregation, an unbounded scroll, a fan-out over thousands of shards), the node accepts it, the search queue backs up, heap pressure mounts, and instead of one slow query you get a node — then a cluster — falling over. The famous tail-latency-to-outage path.
• Too sensitive and you reject healthy traffic. If the cancellation heuristic is trigger-happy, you cancel queries that would have completed fine, turning a transient blip into a wave of 429s and user-visible failures. A backpressure system that cancels too much is its own incident.
• The signal that decides is often coarse. "Node is under duress" is a blunt instrument. Which resource (heap vs CPU vs queue depth)? Which tasks are actually responsible? The cancellation-eligibility ranking (cancel the most expensive offenders, not random victims) is a heuristic that can be measurably improved.
• It is hard to observe. When a query gets cancelled by backpressure, why? Which tracker fired? What was the node-duress reason? Operators frequently cannot tell a backpressure cancellation from a client timeout, which makes the system impossible to tune in production.

This project makes one of those better. The phased plan starts at making cancellations explainable (genuinely mergeable, low blast radius) and builds toward a tuned or new signal — the kind of change that needs an RFC and a measured before/after.

Real-world grounding

Backpressure and admission control are RFC-driven in OpenSearch, on both the indexing and search sides. These are real:

• [Meta] Indexing backpressure — OpenSearch #1446: https://github.com/opensearch-project/OpenSearch/issues/1446
• [Meta] Shard-level indexing back-pressure — OpenSearch #478: https://github.com/opensearch-project/OpenSearch/issues/478
• Shard Indexing Pressure implementation PR — OpenSearch #1336: https://github.com/opensearch-project/OpenSearch/pull/1336

PR #1336 is the canonical example of how a pressure/backpressure system is built and merged in this codebase — read it end to end before you design anything. The indexing side is the mature template; the search side (task-cancellation under duress) is where there is more room to contribute.

Citation discipline: search backpressure evolves and its settings get renamed. Do not cite a setting or issue you have not verified. Before you scope, run: is:issue label:"distributed framework" backpressure OR admission and is:issue search backpressure cancel in opensearch-project/OpenSearch, and grep the live settings in the source (below). Link the current issue in your design note; #1446 / #478 / #1336 are the anchors.

Subsystems you'll touch

Shard indexing pressure (the alternative target) lives under org.opensearch.index.ShardIndexingPressure / ShardIndexingPressureSettings / the ShardIndexingPressureTracker. Grep ShardIndexingPressure — it is the indexing-side analogue and PR #1336 is its origin.

Deep dives that cover the surrounding ground: Backpressure and admission control · Threadpools and concurrency · Search execution (the tasks pressure protects) · Circuit breakers and memory (the related "stop before OOM" mechanism — backpressure is cancellation, circuit breakers are rejection; know the difference).

Phased plan

The discipline of this project is that Phase 1 makes backpressure cancellations explainable — a mergeable observability change — and that explanation is what lets you tune a signal honestly in Phases 3–4. You cannot tune a control loop you cannot see fire.

Phase 0 — Build it, trip it, and trace the cancellation (1 day)

Build OpenSearch and deliberately trigger search backpressure so you can watch the control loop run.

# In your OpenSearch clone:
./gradlew assemble
./gradlew run

# Put backpressure in enforced mode and lower thresholds so it trips easily (test cluster only):
curl -s -X PUT localhost:9200/_cluster/settings -H 'Content-Type: application/json' -d '{
  "persistent": { "search_backpressure.mode": "enforced" }
}'
# (grep SearchBackpressureSettings for the exact threshold setting names for your version)

# Now run an abusive query (deep agg / huge size / heavy script) repeatedly under load and watch:
curl -s localhost:9200/_nodes/stats/search_backpressure?pretty

Find the cancellation decision in code:

grep -rn "class SearchBackpressureService\|cancel\|NodeDuress\|TaskResourceUsageTracker" \
  server/src/main/java/org/opensearch/search/backpressure | head -40
grep -rn "search_backpressure\|SearchBackpressureSettings\|getCancellationThreshold" \
  server/src/main/java/org/opensearch/search/backpressure
grep -rn "SearchBackpressureStats\|cancellationCount\|currentTracker" \
  server/src/main/java/org/opensearch/search/backpressure/stats

Write a one-page capstone-work/cancellation-path.md: node-duress check → tracker evaluation → cancellation-eligibility ranking → CancellableTask.cancel(reason). With file:line citations. This is your execution-path-mastery artifact and you cannot skip it. You must be able to point at the exact branch where node duress is declared and the exact branch where a task is selected to die.

Phase 1 — Make the cancellation reason observable (the scoped, mergeable slice)

When backpressure cancels a search, the reason is often opaque to the operator and the client. Make it precise. Two complementary, small changes:

1. Enrich the cancellation reason string on the task so it names the tracker and the node-duress reason that fired:

grep -rn "cancellationReason\|getCancellationReason\|reasonString\|cancel(" \
  server/src/main/java/org/opensearch/search/backpressure

// When SearchBackpressureService decides to cancel, build a precise reason:
String reason = "search backpressure cancellation; node_duress=[" + duressReason + "]"
    + " offending_tracker=[" + topTracker.name() + "]"
    + " task_heap=[" + humanReadableBytes(task.getHeapUsage()) + "]"
    + " task_elapsed=[" + task.getElapsedTime() + "ms]";
task.cancel(reason);

2. Expose a per-tracker cancellation breakdown in the search-backpressure stats so an operator can see which tracker is doing the cancelling over time (heap-driven vs cpu-driven vs elapsed-driven):

grep -rn "class SearchBackpressureStats\|writeTo\|toXContent" \
  server/src/main/java/org/opensearch/search/backpressure/stats

Then tests: a unit test that the reason string contains the tracker and duress reason, and an integration test (OpenSearchIntegTestCase) that trips backpressure in enforced mode and asserts the stats report the expected tracker as the cancellation cause.

// SearchBackpressureIT-style test
public void testCancellationReasonNamesTracker() throws Exception {
    setBackpressureEnforcedWithLowHeapThreshold();
    Exception e = expectThrows(Exception.class, () -> runHeapHeavyAggUnderLoad());
    assertThat(rootCauseMessage(e), containsString("offending_tracker=[heap_usage_tracker]"));
}

Run the gates:

./gradlew spotlessApply
./gradlew :server:test --tests "*SearchBackpressure*"
./gradlew precommit

Why this is the right Phase 1: it is a minimum diff, it does not change when anything is cancelled (so it cannot make the control loop worse), and it is exactly the operability improvement maintainers merge. A "name the tracker in the cancellation reason + stats" PR is a credible first backpressure contribution — and you cannot tune the heuristic in Phase 3 without it.

Phase 2 — Reproduce a wrong decision and characterize it

Using your Phase-1 observability, construct a scenario where the current signal is measurably wrong in one direction:

• False negative: an expensive query the trackers underweight, so the node accepts it past the point it should have shed load (cascade risk).
• False positive: a query the heuristic cancels that would have completed fine (over-rejection).

Capture it as a reproducible test scenario in capstone-work/design.md: the workload, the settings, the observed wrong decision, and the metric that proves it is wrong (queue depth and heap over time, or completion-vs-cancellation outcome).

Phase 3 — Tune or add a signal (the change that needs an RFC)

Now the real work. Choose one, scoped tightly, and write a design note / RFC comment first:

The non-negotiable: any new or tuned signal ships monitor_only first. It logs/stats what it would have done without actually cancelling, so it can be validated in production before it is allowed to reject traffic. This is how admission-control changes are landed safely, and proposing it any other way will (correctly) get the PR blocked.

// Every new signal must respect the mode gate:
if (settings.getMode() == SearchBackpressureMode.MONITOR_ONLY) {
    stats.recordWouldHaveCancelled(task, reason);   // observe, do not cancel
} else if (settings.getMode() == SearchBackpressureMode.ENFORCED) {
    task.cancel(reason);
}

Phase 4 — Prove the control loop got better, not just different

A backpressure change is a control-system change; "it cancels differently" is not a result. You must show, on a reproducible load scenario, that your change improves the trade-off:

scenario: 200 concurrent heavy aggs + steady healthy traffic, heap-pressure regime

metric                        baseline    yours      better?
healthy queries cancelled        38          11       yes (fewer false positives)
node max heap %                  94          88       yes (still sheds enough load)
p99 of healthy queries (ms)     1900        1200       yes
cluster stayed up?               yes         yes       (negative control: never trade safety for it)

The headline is "fewer healthy queries cancelled while still preventing the cascade." A tweak that cancels less but lets the node cascade is not an improvement — it is a regression in the dimension that matters most. Your negative control is always "the cluster still survives the overload."

Deliverables

• capstone-work/cancellation-path.md — duress → trackers → eligibility → cancel, with citations
• capstone-work/design.md — the wrong-decision scenario, the signal you chose, the false-pos/false-neg framing
• Phase 1: precise cancellation reason + per-tracker stats breakdown, with unit + integration tests
• Phase 2: a reproducible scenario demonstrating a measurably wrong current decision
• (Phase 3) a tuned/new signal, shipped monitor_only-first behind a setting, with tests
• (Phase 4) a load-scenario before/after table showing a better trade-off, with the "cluster survives" negative control
• capstone-work/validation.md — ./gradlew spotlessApply precommit output, test commands, seeds, load harness
• A CHANGELOG.md entry under ## [Unreleased]
• An upstreaming decision: a DCO-signed PR for the Phase-1 observability slice, or an RFC/issue comment proposing the signal change
• A 500–1000 word write-up: the control-system framing, the trade-off you improved, the safety argument

Difficulty & time

The trap is treating this as a coding project. It is a control-systems project: most of the work is constructing a load scenario you can re-run and a measurement that distinguishes "better trade-off" from "different behaviour." Budget heavily for the load harness.

Stretch goals

• Add an _explain-style endpoint or a per-request response header that tells a client why its query was cancelled by backpressure (closing the "was it a timeout or a rejection?" gap).
• Implement hysteresis on the node-duress signal so it does not flap on and off at the threshold boundary — a classic control-loop improvement with a clean before/after (count the state flips).
• Cross-link the search-backpressure stats with the search threadpool queue stats so an operator sees pressure building before cancellations start.
• Reproduce the spirit of #1336 on the search side: a per-shard search-pressure tracker that attributes duress to the specific shards/queries causing it, not just the node.

Evaluation

Self-grade against the 100-point rubric. For this project:

A finished Phase 1 at 90+ is a real, merged operability contribution in OpenSearch's resiliency layer — and the instrument that makes the heuristic tuning in Phase 3 defensible.

How to turn this into a real contribution

1. Start from observability, not the policy. The Phase-1 "name the tracker + stats breakdown" PR is the upstream target. It is low-blast-radius (it changes no cancellation decisions) and is exactly the operability gap operators hit. No RFC needed.
2. RFC any new or tuned decision. Changing when work is cancelled changes cluster failure behaviour. Read PR #1336 to see how the indexing side did it, then propose your search-side change as an issue/RFC comment with the monitor_only-first plan before writing the decision code.
3. One slice per PR. Observability, then (separately, after RFC) the heuristic. Never bundle a new tracker with the reason-string change.
4. Ship monitor_only first, always. The maintainers will require it. A signal that observes before it enforces is the only mergeable shape for new admission control — arrive with that design.
5. Bring a load scenario, not an anecdote. "It felt better" loses. The before/after table with the "cluster survives" negative control is what makes a control-loop change trustworthy.
6. DCO applies. Core OpenSearch: git commit -s, CHANGELOG.md, the backport bot.

If only Phase 1 lands, you have made one of OpenSearch's hardest-to-operate subsystems explainable — which is the prerequisite for anyone, including future-you, to tune it safely. That is the point of scoping from the observability slice up.