Stage 10 — Performance Improvements

What this stage teaches

A performance PR without numbers is a guess. Stage 10 drills the discipline that separates a real improvement from a plausible-looking refactor: measure first, change, measure again, and prove the delta with a reproducible benchmark. The skill set:

• Write a JMH microbenchmark in the :benchmarks module to isolate a hot method and show a before/after allocation or throughput change.
• Use OpenSearch Benchmark (opensearch-benchmark, formerly Rally) for macro numbers — end-to-end indexing/search throughput and latency against a running cluster.
• Reason about allocation and GC pressure: object churn on a hot path, autoboxing, and using BigArrays (the engine's pooled, circuit-breaker-aware array allocator) instead of raw arrays.
• Avoid regressions: a change that helps one workload often hurts another; the bar is "no workload meaningfully worse, target workload meaningfully better."

Prerequisite: Stage 6 or Stage 7 (you must be in a real hot path), Stage 9 (you need deterministic measurement), plus the Level 9 performance lab and the circuit breakers & memory deep dive.

The measurement loop

flowchart LR
  P[profile / identify hot path] --> M1[baseline: JMH micro + OSB macro]
  M1 --> C[change]
  C --> M2[re-measure same JMH + OSB]
  M2 --> D{delta real &<br/>no regression?}
  D -->|yes| PR[PR with before/after numbers]
  D -->|no| C

Rules:

1. Never optimise un-profiled code. Find the hot path with a profiler (async-profiler / JFR) or an existing benchmark, not intuition.
2. Micro proves the mechanism; macro proves it matters. A JMH win that does not move an OSB workload is noise — say so honestly.
3. One change, one number. Bundle two optimisations and you cannot attribute the delta.

Finding Stage 10 issues

is:issue is:open label:enhancement no:assignee "performance" in:title,body
is:issue is:open label:bug no:assignee "regression" "performance" in:body
is:issue is:open label:"help wanted" no:assignee label:"Search:Performance"
is:issue is:open label:"help wanted" no:assignee label:"Storage:Performance"

Area labels: Search:Performance, Storage:Performance, Indexing:Performance, Performance. Check the current set.

Fallback grep — hot paths that allocate per-document or per-request:

ls benchmarks/src/main/java/org/opensearch/benchmark/
# Existing JMH benchmarks tell you which paths are already considered hot:
grep -rln "@Benchmark" benchmarks/src/main/java/org/opensearch/
# Raw arrays on hot paths that could use BigArrays:
grep -rn "new long\[\|new double\[\|new int\[" \
  server/src/main/java/org/opensearch/search/aggregations/ | head

Walked example — a hot-path allocation reduction with a JMH benchmark

Illustrative of the pattern. The grep finds a real hot method; treat the details as a stand-in for "a per-document path that allocates a throwaway object."

Symptom: an aggregation's per-bucket collection allocates a small temporary object (a boxed Long, or a fresh array) for every document, producing heavy young-gen GC pressure on high-cardinality aggregations. The fix reuses a pooled BigArrays-backed structure or hoists the allocation out of the per-doc loop.

Profile to confirm it is hot

# Run a node, drive a high-cardinality aggregation with OSB, attach async-profiler:
./gradlew run &
# (in another shell) drive load with opensearch-benchmark, then profile allocations:
# async-profiler: ./profiler.sh -e alloc -d 30 -f /tmp/alloc.html <opensearch-pid>

The allocation flame graph should show the per-doc allocation dominating. If it does not, stop — you have not found the hot path.

Locate the allocation

grep -n "collect(\|LongValuesSource\|new Long\|new long\[" \
  server/src/main/java/org/opensearch/search/aggregations/bucket/terms/<SomeAggregator>.java | head

The schematic of the churn:

@Override
public void collect(int doc, long owningBucketOrd) throws IOException {
    long[] tmp = new long[1];          // allocated per document — GC pressure
    tmp[0] = values.nextValue();
    addToBucket(owningBucketOrd, tmp[0]);
}

Diff — hoist / pool the allocation

--- a/server/src/main/java/org/opensearch/search/aggregations/bucket/terms/SomeAggregator.java
+++ b/server/src/main/java/org/opensearch/search/aggregations/bucket/terms/SomeAggregator.java
@@
-    public void collect(int doc, long owningBucketOrd) throws IOException {
-        long[] tmp = new long[1];
-        tmp[0] = values.nextValue();
-        addToBucket(owningBucketOrd, tmp[0]);
-    }
+    public void collect(int doc, long owningBucketOrd) throws IOException {
+        // No per-doc allocation: read the value directly.
+        addToBucket(owningBucketOrd, values.nextValue());
+    }

For a structure that genuinely needs growable, large backing storage on a hot path, use BigArrays (it is pooled and accounted against the request circuit breaker — see circuit breakers & memory):

// Allocated once, grown as needed, released in close(); accounted by the breaker.
private LongArray counts = bigArrays.newLongArray(1, true);
// ... counts = bigArrays.grow(counts, owningBucketOrd + 1);

Write a JMH microbenchmark that proves it

Add a benchmark under :benchmarks isolating the path:

@Fork(2)
@Warmup(iterations = 5)
@Measurement(iterations = 10)
@BenchmarkMode(Mode.Throughput)
@OutputTimeUnit(TimeUnit.MILLISECONDS)
@State(Scope.Benchmark)
public class SomeAggregatorBenchmark {

    private long[] values;

    @Setup
    public void setup() {
        values = new long[1_000_000];
        Random r = new Random(42);   // deterministic
        for (int i = 0; i < values.length; i++) values[i] = r.nextInt(100_000);
    }

    @Benchmark
    public long collectAll() {
        long acc = 0;
        for (long v : values) acc += collectOne(v);   // calls the path under test
        return acc;
    }
}

Run it before and after the change:

# Build the benchmarks jar and run JMH:
./gradlew :benchmarks:jmh -Pjmh.include="SomeAggregatorBenchmark" 2>&1 | tee /tmp/after.txt
# (run on the baseline commit too -> /tmp/before.txt)

Capture both throughput and the gc.alloc.rate JMH profiler if you enable -prof gc — the allocation-rate drop is the headline number for a GC-pressure fix.

Prove it matters with a macro benchmark

JMH shows the mechanism; OpenSearch Benchmark shows the user-visible effect. Run the same workload against a baseline node and a patched node:

# Against a running cluster on :9200, with a workload that exercises the aggregation:
opensearch-benchmark execute-test --target-hosts=localhost:9200 \
  --pipeline=benchmark-only --workload=<a high-cardinality agg workload>

Compare the Mean Throughput / p50/p99 Service Time and the young-GC time between runs. A real fix moves the macro number or at least the GC time; if it does not, the micro-improvement was not on a path the macro workload stresses — report that honestly.

Build, CHANGELOG, PR

./gradlew :server:test --tests "*SomeAggregatorTests" -q   # correctness must still hold
./gradlew :server:precommit

--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ ### Changed
+- Remove per-document allocation in `SomeAggregator.collect`, reducing young-gen GC on high-cardinality terms aggs ([#NNNNN](...))

In the PR description, paste: (1) the JMH before/after with Score ± error, (2) the OSB throughput/latency/GC deltas, and (3) the seeds/workload so a reviewer can reproduce. A perf PR with no reproducible numbers is unmergeable in this project.

Reading the numbers honestly

Warning: Overlapping JMH error bars mean no measured difference, no matter how good the change looks on paper. Two forks (@Fork(2)) and enough iterations are the minimum to trust a delta. A single-fork JMH run is noise.

Pitfalls

• Optimising without a profile. Intuition about hot paths is wrong more often than right. Profile, then optimise the thing the profiler points at.
• Micro-only evidence. A JMH win that no OSB workload reflects is not worth the complexity. State the macro result, even when it is "no measurable change."
• Trading correctness for speed. Re-run the subsystem's unit tests. A faster wrong answer is a Stage 6/7 bug.
• Ignoring the circuit breaker. Pooled BigArrays allocations are accounted against the request breaker for a reason. Replacing them with un-accounted raw arrays "to go faster" reintroduces the OOM the breaker exists to prevent.
• Helping one workload, hurting another. Always run a couple of other OSB workloads. The bar is target-better, others-not-worse.
• Non-reproducible numbers. No seed, no workload name, no hardware note = reviewers cannot trust it. Include all three.
• Bundling optimisations. Two changes, one number = unattributable. One change per PR.

Exit criteria — when you're ready for Stage 11

• One performance PR is merged with a JMH before/after (non-overlapping error bars) and an OpenSearch Benchmark macro delta, both reproducible from the PR.
• You can profile (async-profiler/JFR), find the hot path, and resist optimising anything else.
• You reach for BigArrays (breaker-accounted) on growable hot-path storage rather than raw arrays.
• You report honest negatives: when a micro win does not move the macro, you say so.

Measurement is the evidence base for the hardest judgment calls in the project: when changing the wire or storage format is worth it. Stage 11 applies that judgment to backward compatibility.