Lab CS3: Benchmark and Tune Slicing

Background

You can now turn concurrent segment search on, read its profile, and write a CollectorManager. This lab is where you earn the right to have an opinion about the slicing policy: you benchmark concurrent vs sequential across two opposite workloads — few large segments (the design case, should win) and many tiny segments (the overhead case, should lose) — sweep search.concurrent.max_slice_count to find the crossover where more slices stop helping (and start hurting), and tabulate latency and CPU. Then you reason about why the default is what it is, the way a maintainer does when someone proposes changing it.

This is the measurement discipline the Concurrent Search Slicing capstone demands. It builds directly on the perf-regression methodology from Lab 9.2: warm up, repeat, report distributions not single numbers, and never claim a win you can't reproduce.

Why This Matters for Contributors

The concurrent-search mechanism is settled. The policy — how many slices, which segments per slice, when to bother — is where the open performance work lives. A maintainer evaluating a proposed slicing change asks exactly three questions: (1) which workload regime does it help, (2) where's the crossover, (3) does it cost throughput elsewhere? If you can't produce that table, your patch doesn't land. This lab teaches you to produce it.

Prerequisites

Lab CS1 done — you can enable concurrency, profile slices, and read _cat/thread_pool.
A running node (./gradlew run) with idle cores. Note your core count: getconf _NPROCESSORS_ONLN (Linux) or sysctl -n hw.ncpu (macOS). The win is bounded by core count; benchmarking on a 2-core box is uninformative.
curl, jq. Optionally OpenSearch Benchmark (pip install opensearch-benchmark) for a rigorous version; a scripted curl loop is the fallback and is fully shown here.
You read the intensive trade-offs and force-merge sections.

Note: "took" (server-side, from the response) is the right metric here — it measures the shard work, not client/network noise. Always warm up first; the first few queries pay for cache fills and JIT, and including them poisons the mean.

Step-by-Step Tasks

Step 1 — A reusable benchmark function

Put this in your shell. It runs a query N times after a warm-up and prints the min / median / max took. This is your measurement primitive for the whole lab.

bench() {                       # bench <index> <runs> <json-body>
  local idx="$1" runs="$2" body="$3"
  # warm-up (3 ignored runs)
  for w in 1 2 3; do
    curl -s "localhost:9200/${idx}/_search" -H 'Content-Type: application/json' -d "$body" >/dev/null
  done
  local tooks=()
  for r in $(seq 1 "$runs"); do
    t=$(curl -s "localhost:9200/${idx}/_search" -H 'Content-Type: application/json' -d "$body" | jq '.took')
    tooks+=("$t")
  done
  printf '%s\n' "${tooks[@]}" | sort -n | awk '
    { a[NR]=$1; sum+=$1 }
    END { printf "  min=%d  median=%d  max=%d  mean=%.1f  (n=%d)\n",
          a[1], a[int((NR+1)/2)], a[NR], sum/NR, NR }'
}

set_mode() {                    # set_mode <none|all> <max_slice_count>
  curl -s -XPUT 'localhost:9200/_cluster/settings' -H 'Content-Type: application/json' \
    -d "{\"transient\":{\"search.concurrent_segment_search.mode\":\"$1\",
                        \"search.concurrent.max_slice_count\":$2}}" >/dev/null
}

Step 2 — Build two opposite workloads

The whole experiment hinges on contrasting two segment topologies on the same data volume.

# Common: a heavy, CPU-bound aggregation body.
read -r -d '' AGG <<'JSON'
{ "size": 0,
  "aggs": { "by_g": { "terms": { "field": "g", "size": 200 },
            "aggs": { "vs": { "stats": { "field": "v" } } } } } }
JSON

index_data() {                  # index_data <index> <batches> <docs_per_batch>
  curl -s -XDELETE "localhost:9200/$1" >/dev/null 2>&1
  curl -s -XPUT "localhost:9200/$1" -H 'Content-Type: application/json' -d '{
    "settings": { "number_of_shards": 1, "number_of_replicas": 0,
                  "refresh_interval": "-1" },
    "mappings": { "properties": {
        "g": { "type": "keyword" }, "v": { "type": "long" } } } }' >/dev/null
  for b in $(seq 1 "$2"); do
    for i in $(seq 1 "$3"); do
      printf '{"index":{}}\n{"v":%s,"g":"k%s"}\n' "$((RANDOM))" "$((RANDOM % 200))"
    done | curl -s -H 'Content-Type: application/x-ndjson' \
          -XPOST "localhost:9200/$1/_bulk" --data-binary @- >/dev/null
    curl -s -XPOST "localhost:9200/$1/_refresh" >/dev/null   # one segment per batch
  done
}

# Workload A: FEW LARGE segments. Index a lot, then force-merge to 8 fat segments.
index_data fewbig 4 250000
curl -s -XPOST 'localhost:9200/fewbig/_forcemerge?max_num_segments=8' >/dev/null
echo "fewbig segments:"; curl -s 'localhost:9200/_cat/segments/fewbig?v&h=segment,docs.count,size'

# Workload B: MANY TINY segments. Same total docs, but spread across 200 tiny commits.
index_data manytiny 200 5000
echo "manytiny segment count:"; curl -s 'localhost:9200/_cat/segments/manytiny?h=segment' | wc -l

Troubleshooting: Background merges will fight you on manytiny. To hold the segments tiny, keep refresh_interval: -1 and do not force-merge it. If Lucene still merges aggressively, lower the per-batch doc count and raise the batch count. You want a visibly large segment count (dozens+).

Step 3 — Benchmark sequential vs concurrent on BOTH workloads

echo "=== FEWBIG (few large segments) ==="
set_mode none 0;  echo -n "sequential:"; bench fewbig 11 "$AGG"
set_mode all  0;  echo -n "concurrent:"; bench fewbig 11 "$AGG"

echo "=== MANYTINY (many tiny segments) ==="
set_mode none 0;  echo -n "sequential:"; bench manytiny 11 "$AGG"
set_mode all  0;  echo -n "concurrent:"; bench manytiny 11 "$AGG"

Predict before you read the output. From the intensive: fewbig should win under concurrency (fat segments, idle cores); manytiny should be neutral or lose (slice + reduce overhead swamps the per-segment scan). Record both and see if your prediction holds.

Step 4 — Sweep max_slice_count and find the crossover

Now the core experiment. Hold the workload fixed (fewbig) and sweep the slice cap. More slices help — until they don't. Find the knee.

echo "=== max_slice_count sweep on FEWBIG ==="
for sc in 1 2 4 8 16; do
  set_mode all "$sc"
  echo -n "max_slice_count=$sc :"
  bench fewbig 11 "$AGG"
done

Then the same sweep on manytiny, where you expect no knee — just rising overhead:

echo "=== max_slice_count sweep on MANYTINY ==="
for sc in 1 2 4 8 16; do
  set_mode all "$sc"
  echo -n "max_slice_count=$sc :"
  bench manytiny 11 "$AGG"
done

The crossover is the slice count past which median took stops dropping (or starts rising). On fewbig you typically see it plateau around your core count; beyond that, more slices is pure scheduling overhead. On manytiny you often see no improvement at any slice count, because the per-segment work is too small to amortize the task cost.

Step 5 — Measure the CPU cost, not just latency

Latency-down is only half the trade. Concurrency spends CPU. Sample node CPU while looping each configuration so you can quantify the cost:

cpu_under_load() {              # cpu_under_load <index> <mode> <slices> <seconds>
  set_mode "$2" "$3"
  ( end=$((SECONDS+$4)); while [ $SECONDS -lt $end ]; do
      curl -s "localhost:9200/$1/_search" -H 'Content-Type: application/json' -d "$AGG" >/dev/null
    done ) &
  local loadpid=$!
  # OpenSearch process CPU over the window:
  curl -s 'localhost:9200/_nodes/stats/os,process?pretty' \
    | jq '.nodes[] | {cpu_percent: .os.cpu.percent, proc_cpu: .process.cpu.percent}'
  sleep "$4"
  curl -s 'localhost:9200/_nodes/stats/os,process?pretty' \
    | jq '.nodes[] | {cpu_percent: .os.cpu.percent, proc_cpu: .process.cpu.percent}'
  kill "$loadpid" 2>/dev/null
}

echo "sequential CPU:"; cpu_under_load fewbig none 0 6
echo "concurrent CPU:"; cpu_under_load fewbig all  8 6
# Also watch the pool: index_searcher should be busy only in concurrent mode.
curl -s '_cat/thread_pool?v&h=name,active,queue,rejected,completed' 'localhost:9200/_cat/thread_pool?v&h=name,active,queue,rejected,completed' \
  | grep -E 'name|index_searcher|search '

The story you're documenting: concurrent mode buys lower median took at the cost of higher CPU% per query. On a lightly loaded cluster that's a great trade; on a saturated one it's a throughput loss.

Step 6 — Tabulate

Assemble everything into one table — this is the deliverable a maintainer reads:

Workload   | mode | slices | median took (ms) | CPU% per query | verdict
-----------+------+--------+------------------+----------------+--------
fewbig     | none |   -    |       40         |      ~100       | baseline
fewbig     | all  |   2    |       24         |      ~190       | win
fewbig     | all  |   4    |       15         |      ~360       | win
fewbig     | all  |   8    |       13         |      ~680       | win (knee here)
fewbig     | all  |  16    |       14         |      ~690       | no gain, more sched
manytiny   | none |   -    |       33         |      ~100       | baseline
manytiny   | all  |   8    |       38         |      ~520       | LOSS (overhead)

(Your numbers will differ; the shape — fewbig wins and plateaus at ~core count, manytiny loses — is the result.)

Step 7 — Reason about the default like a maintainer

With the table in hand, answer the questions a reviewer would ask:

Why is the Lucene default maxDocsPerSlice ≈ 250k / maxSegmentsPerSlice = 5, not 1 segment per slice? (Your manytiny numbers are the evidence: tiny per-slice work loses. The default deliberately bundles small segments.)
Why is max_slice_count: 0 (heuristic) a safer default than a fixed large number? (A fixed large cap over-slices cheap queries and saturates the pool; the heuristic scales slices to actual doc volume.)
When would you override the default? (A read-only index force-merged to a few fat segments on a box with many idle cores — raise the cap toward core count. A high-QPS cluster — lower it or use mode: auto to avoid pool saturation.)

Confirm the constants you're reasoning about against source:

cd ~/src/lucene
grep -rn "maxDocsPerSlice\|maxSegmentsPerSlice\|250_000\|250000" \
  lucene/core/src/java/org/apache/lucene/search/IndexSearcher.java

Deliverables

The Step 6 table: both workloads × the slice sweep, with median took and CPU% per query.
The identified crossover slice count on fewbig, related to your core count.
Evidence (from _cat/thread_pool) that index_searcher is busy only in concurrent mode and that high slice counts grow its queue.
A short written answer to all three "reason about the default" questions in Step 7, citing your numbers.

Expected Output

=== FEWBIG (few large segments) ===
sequential:  min=37 median=40 max=46 mean=40.6 (n=11)
concurrent:  min=12 median=15 max=22 mean=15.8 (n=11)     # clear win
=== MANYTINY (many tiny segments) ===
sequential:  min=30 median=33 max=39 mean=33.4 (n=11)
concurrent:  min=34 median=38 max=51 mean=39.1 (n=11)     # LOSS, as predicted
=== max_slice_count sweep on FEWBIG ===
max_slice_count=1 : median=39   # ~sequential (1 slice)
max_slice_count=2 : median=24
max_slice_count=4 : median=15
max_slice_count=8 : median=13   # knee ~ core count
max_slice_count=16: median=14   # no further gain

Troubleshooting

Symptom	Cause	Fix
`manytiny` won't stay tiny	Background merges	`refresh_interval: -1`, don't force-merge, smaller batches
No win on `fewbig`	Few cores, or segments not actually large	check `nproc`; verify `_cat/segments` sizes; force-merge to 8
Numbers wildly noisy	No warm-up, other load on box	use the `bench` warm-up; quiesce the machine; raise `runs`
`max_slice_count=16` faster than 8	You have >8 cores	the knee is near your core count, not 8
CPU% reads 0	stats sampled outside the load window	sample during the loop; widen the window

Stretch Goals

OpenSearch Benchmark. Re-run fewbig vs manytiny as a proper OSB workload with a custom track; compare its latency percentiles to your bench medians. This is the rigorous version the capstone wants.
Concurrency under contention. Run N parallel bench loops at once (simulate QPS) and show that concurrent mode's per-query win inverts into a throughput loss once the index_searcher pool saturates — the single most important caveat to "concurrency is faster."
Force-merge sweep. Re-fewbig at max_num_segments ∈ {1, 2, 4, 8, 16} and plot median took vs segment count under concurrency — directly quantify the force-merge-to-1-kills-concurrency claim from the intensive.
mode: auto. If your version supports it, set mode: auto and show it picks sequential for a cheap query and concurrent for the heavy agg — the engine making the crossover decision for you.

Coding Exercises

Your shell bench function is a fine probe, but a maintainer's evidence is reproducible code. These exercises turn the experiment into harnesses and tests that compute the crossover and slicing constants programmatically.

(warm-up) A real benchmark harness in Python. Rewrite bench/set_mode/the Step 4 sweep as slice_sweep.py that takes an index name and a list of slice counts, warms up, runs N times, and prints min/median/p90/max per configuration plus the detected crossover (first slice count where median stops dropping by more than a threshold). It must exit non-zero if the expected fewbig-wins / manytiny-loses shape is violated. This makes Deliverable #1's table a script.
(core) Assert Lucene's slice heuristic in a unit test. Find the constants in a Lucene checkout: rg -n "maxDocsPerSlice|maxSegmentsPerSlice|DEFAULT_MAX" lucene/core/src/java/org/apache/lucene/search/IndexSearcher.java. Write a standalone JUnit test that builds an index with a known leaf/doc layout, constructs new IndexSearcher(reader, executor), calls getSlices(), and asserts the slice count matches what those constants predict for your layout. When the constant changes in a future Lucene, your test tells you. (Anti-staleness: read the value, don't hard-code a remembered one — "(verify on your branch)".)
(core) A JMH-style micro-benchmark of reduce. Promote the "measure the reduce cost" idea into code: write a small benchmark (JMH if available, else a warmed manual loop) that times CollectorManager.reduce for 2/4/8/16 partials versus the per-slice scan, for a cheap and a heavy query. Assert (in a test) that reduce cost grows with partial count — the quantitative basis for "tiny segments lose." Reuse your CountMax/TermsCount managers from Lab CS2.
(core) Drive the sweep from the Java high-level client. Write an OpenSearchIntegTestCase (or a standalone client program) that creates fewbig and manytiny, runs the agg under mode=none and mode=all with max_slice_count ∈ {1,2,4,8}, collects took from each SearchResponse, and asserts the directional invariants: concurrent ≤ sequential on fewbig (within noise), and concurrent ≥ sequential on manytiny. Build/run with ./gradlew :server:internalClusterTest --tests '*YourSweep*'.
(core) Quantify the throughput inversion. Code up the "concurrency under contention" Stretch Goal: a Python/Java load generator that fires the agg from M concurrent clients while sweeping slice count, and reports aggregate throughput (queries/sec), not per-query latency. Assert that beyond pool saturation, raising max_slice_count lowers throughput. Find the pool size with rg -n "INDEX_SEARCHER|index_searcher|searchOnlyPool|allocatedProcessors" server/src/main/java/org/opensearch/threadpool/ThreadPool.java and relate the inversion point to it.
(advanced challenge) An auto-tuner. Build a program that, given an index, probes a few slice counts, fits the latency-vs-slices and CPU-vs-slices curves, and recommends a max_slice_count for a target (minimize latency subject to a CPU-budget ceiling). Validate it: have it recommend a setting for fewbig and one for manytiny, apply each, and confirm with a held-out benchmark run that the recommendation is within X% of the best swept value. This is exactly the policy reasoning Capstone Project 3 asks you to defend — now backed by a tool, building on the methodology from Lab 9.2.

Issues to Practice On

Slicing policy is the open work here — core repo, Search plus performance and flaky-test labels. Performance claims demand reproducible numbers, so your benchmark harness is your reproduction.

gh issue list --repo opensearch-project/OpenSearch --label "Search" --state open
gh issue list --repo opensearch-project/OpenSearch --search "max_slice_count slicing performance" --state open
gh issue list --repo opensearch-project/OpenSearch --label "enhancement" --search "concurrent" --state open
gh issue list --repo opensearch-project/OpenSearch --label "flaky-test" --state open
# Performance/benchmark labels vary — confirm on the tracker:
gh label list --repo opensearch-project/OpenSearch | rg -i "perf|benchmark|search|flaky"

Two representative patterns:

A proposed slicing-policy change ("default max_slice_count should be N"). These RFC/enhancement issues live or die on a reproducible table: which workload regime it helps, where the crossover is, and the throughput cost. Bring your Step 6 table generated by Exercise 1, and an OpenSearch Benchmark run to back it.
A performance regression flagged after a concurrent-search change. Reproduce with fewbig/manytiny, bisect with git bisect against your harness as the test oracle, locate the responsible change, and propose a fix with before/after medians and percentiles.

Planted bug exercise. In an OpenSearch checkout, find where the slice cap is applied (rg -n "max_slice_count|maxSliceCount|getSlices|computeSlices|sliceCount" server/src/main/java/org/opensearch/search/). Introduce a bug that ignores the configured cap (e.g. always slice to leaf count regardless of max_slice_count). Re-run your Exercise 4 sweep test and watch the "plateau at the cap" assertion fail; confirm with the Lab CS1 profile that the slice count no longer responds to the setting. Revert, then add a test asserting that max_slice_count=K produces ≤ K slices for a multi-segment index — the guard that catches a cap-ignoring regression.

Etiquette: claim before you work, reproduce before you fix, and ship a test + CHANGELOG.md entry + DCO git commit -s. Performance PRs additionally need before/after numbers. See community interaction and the good-first-issue PR lab.

Validation / Self-check

You can show, with numbers, one workload where concurrency wins and one where it loses, and explain the difference via per-segment work.
You can state the crossover slice count on your box and relate it to core count and pool size.
You can articulate why the Lucene default bundles small segments rather than one-per-slice, using your manytiny data as evidence.
You can describe how the per-query latency win becomes a cluster throughput loss under high QPS — the trade the maintainer protects with the default.
You're ready for Capstone Project 3: you have a benchmark harness and the data discipline it requires.

This completes the Concurrent Segment Search intensive. Back to the masterclass index, or on to Capstone Project 3, which builds on Lab 9.2: Performance Regression.

Open-Source Engineer & Contributor