Lab 9.2: Analyze a Performance Regression

Lab type: Research & Benchmark Estimated time: 4–6 hours

Background

A performance PR without numbers is a guess. At OpenSearch's scale a 5% allocation regression on a path that runs once per document, once per request, or once per shard compounds into real latency and real cost across every cluster running the code. Maintainers therefore treat performance on hot paths as a correctness property: they do not accept "this should be faster," they accept "here is the JMH microbenchmark, here is the before/after, here is the OpenSearch Benchmark macro result that proves it matters and that nothing else regressed."

This lab teaches the measurement loop. You will profile to find a hot path, write a JMH microbenchmark in the :benchmarks module to isolate it, read the results (including allocation and GC pressure), and then connect the micro result to a macro result from OpenSearch Benchmark. You will walk a realistic regression — a per-request allocation or an accidental O(n²) on a search/aggregation/coordination path — and prove the fix the way a maintainer demands.

This lab is the hands-on counterpart to Stage 10: Performance Improvements; read that stage for the issue-finding workflow and the policy framing. The allocation and circuit-breaker mechanics live in the circuit breakers & memory deep dive.

Why This Lab Matters for Contributors

Performance patches are held to a higher review bar than correctness patches, for two reasons. First, "faster" is unfalsifiable without a benchmark — a maintainer cannot approve a claim they cannot verify. Second, an optimization that helps one workload often hurts another, and the cost of shipping a regression is paid by every user, not just the one whose case you optimized. Learning to produce micro + macro numbers, and to read them honestly (including saying "this JMH win does not move any real workload"), is what lets a maintainer trust your perf PRs — and what lets you, as a reviewer, gate others'.

Prerequisites

• You can build OpenSearch and run ./gradlew :server:test.
• You have read Stage 10 and the circuit breakers & memory deep dive.
• You understand at least one hot path well — the search execution, aggregations, or engine internals deep dive.
• JMH basics: @Benchmark, @BenchmarkMode, @State, Blackhole, warmup vs measurement iterations.

The Measurement Loop

flowchart LR
  P[profile: async-profiler / JFR<br/>find the hot path] --> M1[baseline: JMH micro + OSB macro]
  M1 --> C[make ONE change]
  C --> M2[re-measure same JMH + same OSB]
  M2 --> D{delta real AND<br/>no regression?}
  D -->|yes| PR[PR with before/after numbers]
  D -->|no| C

Three rules, drilled until they are reflex:

1. Never optimize un-profiled code. Find the hot path with a profiler or an existing benchmark, not intuition. Most "obvious" optimizations target cold code.
2. Micro proves the mechanism; macro proves it matters. A JMH win that does not move an OSB workload is noise — and you must say so.
3. One change, one number. Bundle two optimizations and you cannot attribute the delta, and one may quietly regress.

Step-by-Step Tasks

Step 1: Find a hot path (profile, don't guess)

Hot paths run per-document, per-request, per-shard, or inside the coordination layer. Start from one of these and confirm with a profiler.

Profile a representative workload (you can run one from OpenSearch Benchmark) under async-profiler or JFR, then grep for the allocation site you suspect:

# Per-document / per-request allocations are the usual culprits. Look for new objects,
# autoboxing, and collection growth inside loops on these paths.
grep -rn "new HashMap\|new ArrayList\|new Object\[\|\.add(" \
  server/src/main/java/org/opensearch/search/aggregations/ | head

# DocValues / BigArrays usage on hot paths (the right way to allocate):
grep -rn "bigArrays\|BigArrays\|PageCacheRecycler\|newLongArray\|newDoubleArray" \
  server/src/main/java/org/opensearch/search/aggregations/ | head

Note: A new HashMap inside a per-document collect() is a classic regression: it allocates an entry array, churns the young generation, and at high doc counts trips the request circuit breaker. The fix is usually to hoist the allocation out of the loop or use a pooled BigArrays structure. See the circuit breakers & memory deep dive.

Step 2: Locate the :benchmarks module

JMH microbenchmarks live in their own Gradle module.

# Where the benchmarks live and how they're organized.
ls benchmarks/src/main/java/org/opensearch/benchmark/
# expect subpackages like: search/ routing/ time/ store/ ...

# Read an existing benchmark end to end as a template.
grep -rln "@Benchmark" benchmarks/src/main/java/org/opensearch/benchmark/ | head
grep -n "@Benchmark\|@State\|@Setup\|@BenchmarkMode\|Blackhole" \
  $(grep -rln "@Benchmark" benchmarks/src/main/java/org/opensearch/benchmark/ | head -1)

Step 3: Write a JMH microbenchmark for the hot path

Isolate the suspect method. The example below contrasts a per-call HashMap allocation against a reused, pre-sized structure — the shape of a real regression and its fix. Put it under benchmarks/src/main/java/org/opensearch/benchmark/....

package org.opensearch.benchmark.search;

import org.openjdk.jmh.annotations.*;
import org.openjdk.jmh.infra.Blackhole;

import java.util.HashMap;
import java.util.Map;
import java.util.concurrent.TimeUnit;

@Fork(value = 2)
@Warmup(iterations = 5, time = 1)
@Measurement(iterations = 10, time = 1)
@BenchmarkMode(Mode.AverageTime)
@OutputTimeUnit(TimeUnit.NANOSECONDS)
@State(Scope.Thread)
public class BucketAggregationBenchmark {

    @Param({"16", "256", "4096"})
    public int buckets;

    private long[] keys;
    private long[] vals;

    @Setup
    public void setup() {
        keys = new long[buckets];
        vals = new long[buckets];
        for (int i = 0; i < buckets; i++) {
            keys[i] = i;
            vals[i] = i * 31L;
        }
    }

    /** Regressed path: allocates a HashMap (with boxing) on every call. */
    @Benchmark
    public void allocatingPerCall(Blackhole bh) {
        Map<Long, Long> m = new HashMap<>();          // allocation + autoboxing
        for (int i = 0; i < buckets; i++) {
            m.merge(keys[i], vals[i], Long::sum);
        }
        bh.consume(m.size());
    }

    /** Fixed path: primitive arrays, no boxing, sized once. */
    @Benchmark
    public void primitiveArrays(Blackhole bh) {
        long[] acc = new long[buckets];               // single primitive array
        for (int i = 0; i < buckets; i++) {
            acc[(int) keys[i]] += vals[i];
        }
        bh.consume(acc[buckets - 1]);
    }
}

Warning: A JMH benchmark that the JIT can prove has no side effects gets optimized away — you measure nothing. Always bh.consume(...) the result, take work in from @State/@Param (not constants), and trust the warmup iterations. If a benchmark reports 0 ns/op, the dead-code eliminator ate it.

Step 4: Run the benchmark and read the results

# Build the runnable JMH uber-jar.
./gradlew :benchmarks:jmhJar -q

# Run the whole suite, or scope to one benchmark with a regex and turn on the
# allocation profiler (-prof gc) — allocation is usually the real story.
java -jar benchmarks/build/distributions/opensearch-benchmarks-*.jar \
  "BucketAggregationBenchmark" -prof gc

# Some checkouts expose a convenience task; either works:
./gradlew :benchmarks:jmh -Pjmh.includes="BucketAggregationBenchmark"

Read the output as a maintainer reads it — two numbers matter, not one:

Benchmark                              (buckets)  Mode  Cnt    Score    Error  Units
BucketAggregationBenchmark.allocatingPerCall  4096  avgt   20  41210.3 ± 980.1  ns/op
BucketAggregationBenchmark.primitiveArrays    4096  avgt   20   3120.7 ±  88.4  ns/op

# With -prof gc, the line that decides perf-sensitive PRs:
BucketAggregationBenchmark.allocatingPerCall:·gc.alloc.rate.norm  4096   ~196600  B/op
BucketAggregationBenchmark.primitiveArrays:·gc.alloc.rate.norm    4096    ~32792  B/op

What you extract:

• Score ± Error — mean time per op and its confidence interval. The intervals must not overlap for the delta to be real.
• gc.alloc.rate.norm — bytes allocated per operation. This is the number that predicts GC pressure and circuit-breaker behavior at scale. A throughput win that raises allocation is suspect.
• @Param scaling — how the gap behaves as buckets grows. A regression that is flat at 16 buckets and explosive at 4096 is an algorithmic problem (e.g. O(n²)), not a constant-factor one.

Step 5: Macro benchmark with OpenSearch Benchmark

The microbenchmark proves the mechanism in isolation. To prove it matters you run a macro benchmark against a real cluster with OpenSearch Benchmark (opensearch-benchmark, formerly Rally). It drives standard workloads and reports end-to-end service time and throughput.

pip install opensearch-benchmark   # one-time

# List the standard workloads.
opensearch-benchmark list workloads
# nyc_taxis        — heavy aggregation + range queries over taxi-trip data
# http_logs        — log-style indexing + search, big volume
# geonames, pmc, noaa, eventdata, so (Stack Overflow), ...

# Run a search-heavy workload against a cluster you launched from your build
# (./gradlew run gives you a node on :9200). Baseline THEN your change.
opensearch-benchmark execute-test \
  --target-hosts localhost:9200 \
  --pipeline benchmark-only \
  --workload nyc_taxis \
  --kill-running-processes

Interpret the summary the way a maintainer does:

Note: Run baseline and candidate on the same machine, same workload, same warmup, ideally back-to-back, and compare the distributions, not single runs. OpenSearch Benchmark can store results and compare two test executions (opensearch-benchmark compare --baseline=<id> --contender=<id>). A 3% mean change inside the run-to-run noise is not a result.

Step 6: Prove the fix — before / after, micro and macro

Apply the one change (and only that change). Re-run the identical JMH benchmark and the identical OSB workload. Lay the numbers side by side:

                              JMH ns/op (4096)   JMH B/op (4096)   OSB nyc_taxis p99
allocatingPerCall (before)         41210               196600           188 ms
primitiveArrays  (after)            3120                 32792           171 ms
delta                              -92%                 -83%             -9%

This is the artifact a maintainer wants in the PR description: a micro result that explains the mechanism (allocation and time both fell), a macro result that shows it moves a real workload's p99, and confirmation that no other workload regressed.

How Maintainers Gate Perf-Sensitive PRs

When reviewing a PR that touches a hot path, a maintainer asks, in order:

1. Where are the numbers? No before/after benchmark → request changes. "Should be faster" is not reviewable.
2. Micro and macro? JMH alone can mislead (it measures one method out of context); ask for an OSB workload that exercises the path. JMH-only is acceptable only when the change is provably isolated.
3. What about allocation and the tail? Throughput up but gc.alloc.rate.norm up, or mean down but p99 up, is a likely net loss. The breaker behavior matters.
4. Any other workload worse? The bar is "target workload meaningfully better, no workload meaningfully worse." Ask which workloads were checked.
5. Is it one change? Two optimizations in one diff → ask to split so each delta is attributable.
6. Is it on a real hot path at all? Optimizing cold code adds complexity for no gain; a clean, simple version often wins.

Apply this same checklist to your own perf PRs before you open them. That is the maintainer reflex this lab builds.

Pitfalls

Expected Output

A JMH run with -prof gc showing both time and allocation, with non-overlapping confidence intervals between before and after; an OSB summary table for the same workload before and after, showing the target metric (e.g. p99 service time) improved and no other tracked metric regressed; and a side-by-side before/after table suitable for pasting into a PR description.

Stretch Goals

• Find a real merged OpenSearch PR labeled Search:Performance or Indexing:Performance and read the before/after numbers in its description. Reproduce its JMH benchmark.
• Convert a hot-path new long[]/new HashMap to BigArrays, measure the allocation and breaker-accounting difference, and read why BigArrays integrates with the request circuit breaker in the circuit breakers & memory deep dive.
• Construct a benchmark that exposes an O(n²) path: parameterize input size and show the time grows quadratically while the fixed version grows linearly.
• Profile ./gradlew run under an OSB nyc_taxis run with async-profiler and produce a flame graph; identify the single hottest frame and tie it to a class.

Validation / Self-check

Answer all of these before marking the lab complete:

1. Why is a JMH microbenchmark insufficient on its own to justify a perf PR, and what does an OpenSearch Benchmark macro run add that JMH cannot?
2. What is gc.alloc.rate.norm, why does a maintainer care about it as much as the time score, and how does it relate to circuit breakers?
3. Explain "micro proves the mechanism, macro proves it matters" with a concrete case where the micro number improves but the macro number does not move.
4. Your @Benchmark reports 0 ns/op. Name two causes and the fix for each.
5. Why is "one change, one number" a hard rule for performance PRs?
6. You see mean service time fall 8% but p99 rise 15% in OSB. Is this a win? Justify your answer in terms of what users experience.
7. Given a per-document new HashMap in an Aggregator.collect(...), describe the regression mechanism (allocation, GC, breaker) and two ways to fix it.