Project 3: Optimizing Concurrent Segment Search Slicing

Concurrent segment search splits a shard's segments into slices and searches them in parallel on the search threadpool, then reduces the per-slice results into one shard-local answer. It became a cluster-level default in 3.0. The mechanism is settled; the slicing policy — how many slices, and which segments go in each — is where the performance still lives. Slice too finely over many tiny segments and coordination overhead swamps the win. Slice too coarsely and you leave cores idle. This project asks you to analyze the current slicing decision, find a regime where it is suboptimal, and improve it — with a measurement gate that proves the change is real before any code lands.

This is the project for someone who wants search execution and parallelism, and who is willing to live or die by a benchmark. There is no "I think it's faster" here. You bring numbers or you bring nothing.

Note: Read the Concurrent Segment Search chapter first; this brief assumes you know what a slice is, the CollectorManager contract, and the settings that bound slice count. It builds directly on Lab 9.2: Performance Regression — the measurement discipline there is the spine of this whole project.

Problem & motivation

The slicing decision is a heuristic, and every heuristic has a regime where it is wrong:

• Small-segment overhead. A shard freshly written by many indexing threads has lots of tiny segments. Splitting them into slices creates more tasks than useful work — each slice's fork/join, collector creation, and reduce cost can exceed the time to scan a small segment sequentially. The default maxDocsPerSlice / maxSegmentsPerSlice heuristic (Lucene's IndexSearcher.slices(...)) does not always group these well for OpenSearch's workloads.
• Slice-count vs. threadpool. Slices contend for the search threadpool. Over-slicing one request steals threads from concurrent requests; the per-request latency win becomes a cluster-wide throughput loss. The cap (search.concurrent.max_slice_count) is a blunt global knob, not a per-request decision.
• Cheap queries don't benefit. For a query whose per-segment cost is tiny, the reduce and coordination dominate, and concurrency is pure overhead — but the policy slices anyway.

The opportunity: a smarter slicing policy that collapses tiny segments into fewer slices, or makes the slice count sensitive to estimated per-segment cost and threadpool pressure, can win latency where the win is real and stop spending where it is not. That is a measurable, reviewable change — if you can prove it.

The constraint that makes it hard: "faster" is workload-dependent. A change that helps the many-tiny-segments case must be shown not to regress the many-large-segments case it was designed for. The benchmark is the design.

Real-world grounding

The grounding is the concurrent-segment-search feature itself, which graduated to a 3.0 default behind real settings. Read the existing chapter and then find the live work:

• The slicing/maxSliceCount settings and the feature in the OpenSearch repo. Search:

# in opensearch-project/OpenSearch
is:issue concurrent segment search slice
is:issue "max_slice_count" OR "slice count" OR slicing
is:pr concurrent search slice latency OR throughput

• The Lucene side of the slicing primitive: IndexSearcher.slices(...) and LeafSlice in apache/lucene. The OpenSearch slice strategy wraps or replaces Lucene's default partitioning.

Citation discipline: the settings (search.concurrent_segment_search.enabled, search.concurrent.max_slice_count) and the 3.0-default fact are verified — cite them. The current slicing-tuning issue you choose to ground in must be one you actually find; link it in your design note. Do not invent an issue number.

Subsystems you'll touch

Cross-references: Concurrent Segment Search · search execution deep dive · threadpools and concurrency · refresh/flush/merge (why you get many small segments) · the performance-regression lab.

Phased plan

The measurement gate is Phase 1, not an afterthought. You do not write a line of slicing code until you can measure slicing latency reproducibly and show the regime where it is bad.

Phase 0 — Find the slicing decision (1 day)

# In an OpenSearch clone:
grep -rn "LeafSlice\|computeSlices\|slices(\|maxSliceCount\|SliceCount\|max_slice_count" \
  server/src/main/java/org/opensearch/search
grep -rn "class ContextIndexSearcher\|protected LeafSlice\|getSlices" \
  server/src/main/java/org/opensearch/search/internal/ContextIndexSearcher.java
grep -rn "CollectorManager\|reduce(" server/src/main/java/org/opensearch/search/query

Write capstone-work/slicing-trace.md: query phase → ContextIndexSearcher → slice computation → per-slice CollectorManager → reduce, with file:line citations and the exact heuristic (the maxDocs/maxSegments thresholds) that decides slice count today.

Phase 1 — Build the measurement gate (the real first deliverable)

You need two instruments:

(a) A JMH microbenchmark isolating the slicing+reduce cost as a function of segment count and size, independent of the rest of the query phase.

// benchmarks/src/main/java/org/opensearch/benchmark/search/SlicingBenchmark.java
@BenchmarkMode(Mode.AverageTime)
@OutputTimeUnit(TimeUnit.MILLISECONDS)
@State(Scope.Benchmark)
public class SlicingBenchmark {
    @Param({"4", "32", "256"})      int segmentCount;
    @Param({"100", "10000"})        int docsPerSegment;
    @Param({"1", "2", "4", "8"})    int maxSlices;

    // build a Directory with segmentCount segments, run the slice+collect+reduce path,
    // measure end-to-end query-phase time at each (segmentCount, docsPerSegment, maxSlices).
    @Benchmark public long sliceAndCollect() { /* ... */ }
}

./gradlew :benchmarks:jmh -Pjmh.include=SlicingBenchmark

(b) An OSB macro run on a realistic index, with concurrent search on, sweeping max_slice_count, so you see real cluster latency and throughput:

opensearch-benchmark execute-test --workload=nyc_taxis \
  --target-hosts=localhost:9200 \
  --workload-params='{"search_clients": 8}' \
  --kill-running-processes
# vary search.concurrent.max_slice_count between runs; record p50/p90/p99 and throughput

Produce capstone-work/baseline.md: a table showing the regime where current slicing is suboptimal (e.g. many tiny segments, or high concurrency where over-slicing hurts throughput). If you cannot show a bad regime, there is no project — pick a different one.

Phase 2 — A better slicing policy (scoped)

Pick one lever, justified by your baseline:

Implement behind a new, default-off setting so the existing default behaviour is untouched until proven. Sketch:

// in the slice computation:
int targetSlices = costAwareSliceCount(leaves, settings, threadPoolPressure());
List<LeafSlice> slices = groupByMinWork(leaves, minDocsPerSlice, targetSlices);
return slices.toArray(new LeafSlice[0]);

Phase 3 — Prove the win and the no-harm

Re-run the exact Phase-1 instruments with your policy on. The bar is two-sided:

workload / regime          metric    default     yours      verdict
many tiny segments (p90)   ms          42          2x        WIN
many large segments (p90)  ms          120        12x        NO REGRESSION
high-concurrency tput      qps         8x0         8x0+      WIN/EQUAL
cheap query (p50)          ms          3.1         3.x       NO REGRESSION

Every row must be explained. A win in one regime that regresses another is not done — either the policy auto-detects the regime, or it stays setting-gated and documented for the regime it helps. State the seed and the OSB workload + params so the result is reproducible.

Phase 4 — Correctness, not just speed

Concurrency bugs hide in reduce. Add/extend tests that prove your slicing produces identical results to the sequential path: same hits, same scores, same aggregation values, across segment counts. A faster wrong answer is the worst possible outcome here.

./gradlew :server:test --tests "*ConcurrentSegmentSearch*" --tests "*Slice*"
./gradlew :server:internalClusterTest --tests "*ConcurrentSearch*IT"

Deliverables

• capstone-work/slicing-trace.md — query phase → slice computation → reduce, with citations and the current heuristic
• capstone-work/baseline.md — JMH + OSB results showing the suboptimal regime (the gate)
• capstone-work/design.md — the lever chosen, alternatives rejected, why it's setting-gated
• A JMH microbenchmark committed under benchmarks/
• The new slicing policy behind a default-off setting
• capstone-work/results.md — two-sided before/after table (win and no-harm), with seeds + OSB params
• Correctness tests proving slice results == sequential results
• capstone-work/validation.md — ./gradlew :server:check output, test commands, seeds
• An upstreaming decision: a DCO-signed PR with the numbers in the description, or a written proposal + issue comment
• A 500–1000 word write-up: the regime, the heuristic, the measurement

Difficulty & time

Warning: Benchmark noise will lie to you. Pin CPU, disable turbo where you can, run enough JMH forks/iterations, and run OSB enough times to have a confidence interval — not one run. See the performance-regression lab for the discipline.

Stretch goals

• Make the policy adaptive: detect the regime at query time (segment-size histogram + threadpool queue depth) and choose slice count automatically, so it can be on by default.
• Add a per-request override (a search-request parameter) so an operator can tune slicing for a specific expensive query without changing the cluster default.
• Extend the win analysis to aggregations, where per-slice reduce is heavier than for plain top-K — the regime where concurrency pays best.
• Feed the slicing decision a profile signal so search profiling shows the slice count chosen and why.

Evaluation

Self-grade against the 100-point rubric. This project is weighted, in spirit, toward measurement:

A clean two-sided benchmark with a small, gated slicing improvement is exactly the shape of a mergeable search-performance PR.

How to turn this into a real contribution

1. Lead with the benchmark. A search-performance PR with no numbers will be asked for numbers; arrive with the two-sided table in the description and the JMH benchmark committed.
2. Default-off, then propose default-on with data. Land the policy behind a setting first. Changing the default slicing behaviour is a separate, data-backed conversation.
3. Prove no-harm explicitly. Reviewers will worry about the many-large-segments case the feature was built for. Show that row green.
4. Correctness first in the PR narrative. State up front that slice results are identical to sequential, with the test that proves it — concurrency reviewers look for that before speed.
5. DCO applies (GitHub-native core: git commit -s, CHANGELOG.md, backport bot).

The benchmark harness you build in Phase 1 is itself a reusable contribution — even if the policy change needs more iteration, a committed JMH slicing benchmark is mergeable on its own.