Lab CS1: Enable and Profile Concurrent Search

Background

You read the intensive: a shard's segments are searched in parallel as slices, each slice runs a fresh collector on the index_searcher threadpool, and the per-slice results are merged by a CollectorManager.reduce. This lab makes that concrete and measurable. You will start a node, index enough data to force several segments, run a heavy aggregation with concurrency off and on, read the per-slice timing breakdown from _search?profile=true, watch the search threadpool fill, and compare wall-clock latency.

The point isn't to "prove concurrency is fast." It's to build the muscle of observing what the slicing did — how many slices, where the time went, which pool ran them — so that in Lab CS3 and Capstone Project 3 you can reason about the slicing policy from evidence rather than vibes.

Why This Matters for Contributors

Every concurrent-search bug report you will ever triage starts with someone saying "it's slow" or "it's not faster." The first thing a maintainer does is ask: how many slices ran, and where did the time go? The answer is in profile=true and _cat/thread_pool. If you can't read those, you can't triage. This lab is the literacy test for the whole feature.

Prerequisites

• A local OpenSearch checkout you can run: ./gradlew run (see Level 2). OpenSearch 3.0+ has concurrent search default-on at the cluster level.
• curl and jq installed (jq optional but assumed below).
• You've read the intensive — you know what a slice is, the CollectorManager contract, and the settings.
• Familiarity with search execution (query-then-fetch) and aggregations.

Note: ./gradlew run starts a single-node cluster on localhost:9200. The node is your cluster manager (formerly "master") and data node both. A single-node cluster is fine here — concurrent segment search is a within-shard feature, so one shard on one node exercises it fully.

Step-by-Step Tasks

Step 1 — Start a node and confirm the feature exists

cd ~/src/OpenSearch
./gradlew run        # leave this running in one terminal

In a second terminal, confirm the node is up and check the concurrent-search settings exist and their defaults:

curl -s 'localhost:9200/_cluster/health?pretty' | grep '"status"'

# What are the effective concurrent-search settings right now?
curl -s 'localhost:9200/_cluster/settings?include_defaults=true&flat_settings=true&pretty' \
  | grep -i 'concurrent'

You should see keys like search.concurrent_segment_search.mode and search.concurrent.max_slice_count. Note their default values — on 3.0+ the mode is effectively on by default.

Troubleshooting: If grep -i concurrent returns nothing, the keys may have a slightly different name in your version. Find them in source:

grep -rn "concurrent_segment_search\|concurrent.max_slice_count" \
  server/src/main/java/org/opensearch/ | grep -i "Setting" | head

Step 2 — Create a single-shard index

One shard keeps the experiment clean: all the parallelism is within this shard, across its segments. No replica, so no replica copy to confuse took.

curl -s -XDELETE 'localhost:9200/csslab' >/dev/null 2>&1
curl -s -XPUT 'localhost:9200/csslab' -H 'Content-Type: application/json' -d '{
  "settings": { "number_of_shards": 1, "number_of_replicas": 0 },
  "mappings": {
    "properties": {
      "g":  { "type": "keyword" },
      "v":  { "type": "long" },
      "ts": { "type": "date" }
    }
  }
}' ; echo

Step 3 — Index in batches to force MANY segments

Each bulk with refresh=true flushes a new segment. Index five batches → at least five segments (Lucene may merge some in the background; that's fine, you'll check).

for b in 1 2 3 4 5; do
  for i in $(seq 1 5000); do
    printf '{"index":{}}\n{"v":%s,"g":"k%s","ts":"2026-06-1%sT00:00:00Z"}\n' \
      "$((RANDOM))" "$((RANDOM % 100))" "$b"
  done | curl -s -H 'Content-Type: application/x-ndjson' \
        -XPOST 'localhost:9200/csslab/_bulk?refresh=true' --data-binary @- >/dev/null
  echo "batch $b done"
done

# Confirm you have multiple segments and see their sizes/doc counts.
curl -s 'localhost:9200/_cat/segments/csslab?v&h=segment,docs.count,size'

You want several segments here. If you see only one, indexing was too small or a merge collapsed them — index more batches, or skip the next force-merge variations for now.

Step 4 — Run a heavy aggregation with concurrency OFF

Force sequential mode so you have a baseline. Use mode: none (or enabled: false on older lines):

curl -s -XPUT 'localhost:9200/_cluster/settings' -H 'Content-Type: application/json' \
  -d '{"transient":{"search.concurrent_segment_search.mode":"none"}}' ; echo

# A genuinely CPU-bound agg: a high-cardinality terms + a nested stats.
read -r -d '' AGG <<'JSON'
{
  "size": 0,
  "aggs": {
    "by_g": {
      "terms": { "field": "g", "size": 100 },
      "aggs": { "vstats": { "stats": { "field": "v" } } }
    }
  }
}
JSON

# Warm the caches, then take the took.
curl -s 'localhost:9200/csslab/_search' -H 'Content-Type: application/json' -d "$AGG" >/dev/null
echo "SEQUENTIAL took (ms):"
for r in 1 2 3; do
  curl -s 'localhost:9200/csslab/_search' -H 'Content-Type: application/json' -d "$AGG" \
    | jq '.took'
done

Record the took values. This is your sequential baseline.

Step 5 — Turn concurrency ON and compare

curl -s -XPUT 'localhost:9200/_cluster/settings' -H 'Content-Type: application/json' \
  -d '{"transient":{"search.concurrent_segment_search.mode":"all",
                    "search.concurrent.max_slice_count":0}}' ; echo

curl -s 'localhost:9200/csslab/_search' -H 'Content-Type: application/json' -d "$AGG" >/dev/null
echo "CONCURRENT took (ms):"
for r in 1 2 3; do
  curl -s 'localhost:9200/csslab/_search' -H 'Content-Type: application/json' -d "$AGG" \
    | jq '.took'
done

Note: On a tiny single-node dev box with small segments you may see no speedup or even a slowdown — that's the trade-off from the intensive, not a bug. The goal of this step is the comparison and the profile, not a guaranteed win. You force the win in the Stretch Goals by indexing far more data.

Step 6 — Read the per-slice profile

This is the core skill. profile=true returns a per-segment / per-slice timing breakdown. With concurrency on, the collector tree shows multiple slice collectors:

curl -s 'localhost:9200/csslab/_search?profile=true' \
  -H 'Content-Type: application/json' -d "$AGG" \
  | jq '.profile.shards[0].aggregations' | head -60

# Pull just the collector names + times to see the slice fan-out:
curl -s 'localhost:9200/csslab/_search?profile=true' \
  -H 'Content-Type: application/json' -d "$AGG" \
  | jq -r '.profile.shards[0].aggregations[]?
            | "agg: \(.type)  time_ns: \(.time_in_nanos)"'

# And the query-side collector tree (this is where "slice" structure shows up):
curl -s 'localhost:9200/csslab/_search?profile=true' \
  -H 'Content-Type: application/json' -d "$AGG" \
  | jq '.profile.shards[0].searches[0].collector'

Look for a collector named with slice or for multiple sibling collectors under the query — each corresponds to one slice. Count them. Compare that count to the slice count you'd derive by hand from _cat/segments doc counts (the worked example in the intensive).

Troubleshooting: If you don't see slice in the collector names, your version may label them differently. Grep the source for the profile collector names:

grep -rn "slice\|Slice\|MultiCollector\|collector_name\|REASON" \
  server/src/main/java/org/opensearch/search/profile/query/ | head

Step 7 — Watch the search threadpool

With concurrency on, a single search spreads across the index_searcher pool. Hammer the agg in a loop while sampling _cat/thread_pool:

# In terminal A: fire searches in a tight loop.
while true; do
  curl -s 'localhost:9200/csslab/_search' -H 'Content-Type: application/json' -d "$AGG" >/dev/null
done &
LOAD_PID=$!

# In terminal B: sample the pools (look for index_searcher active/queue/completed).
for s in 1 2 3 4 5; do
  curl -s 'localhost:9200/_cat/thread_pool?v&h=node_name,name,active,queue,rejected,completed' \
    | grep -E 'name|index_searcher|search '
  echo '---'
  sleep 1
done

kill "$LOAD_PID"

You should see the index_searcher pool's active and completed climb while concurrency is on. Flip to mode: none and repeat — index_searcher stays idle and search carries the load instead. That contrast is the whole feature made visible.

Step 8 — Force-merge variations

Prove the force-merge interaction from the intensive: 1 segment → no concurrency.

# (a) Force to ONE segment, then profile: expect a single slice/collector.
curl -s -XPOST 'localhost:9200/csslab/_forcemerge?max_num_segments=1' >/dev/null
curl -s 'localhost:9200/_cat/segments/csslab?v&h=segment,docs.count'
curl -s 'localhost:9200/csslab/_search?profile=true' \
  -H 'Content-Type: application/json' -d "$AGG" \
  | jq '.profile.shards[0].searches[0].collector'   # one slice now

# (b) Re-index a few batches to get segments back (step 3), then force to 4.
#     (force-merge can't *create* segments, only reduce; re-index first.)
curl -s -XPOST 'localhost:9200/csslab/_forcemerge?max_num_segments=4' >/dev/null
curl -s 'localhost:9200/_cat/segments/csslab?v&h=segment,docs.count'

Confirm: after max_num_segments=1, the profile shows a single slice and concurrency does nothing — exactly as predicted.

Deliverables

• A table of took (ms): sequential vs concurrent, 3 runs each.
• The slice/collector count from the profile, and the count you derived by hand from _cat/segments — do they match?
• A _cat/thread_pool sample showing index_searcher active under concurrent load and idle under mode: none.
• The force-merge=1 profile showing a single slice (concurrency disabled).
• One paragraph: did concurrency help on your box? If not, why not (segment sizes, core count, query cost)?

Expected Output

SEQUENTIAL took (ms): 38, 35, 36
CONCURRENT took (ms): 22, 19, 21      # win, on a box with idle cores + fat segments
# profile collector tree: 3-4 sibling slice collectors
# _cat/thread_pool: index_searcher active=3 ... completed climbing
# after forcemerge max_num_segments=1: ONE slice collector, took back to ~sequential

On a small dev box you may instead see concurrent ≈ sequential or slightly worse — record it honestly; that is the expected output for tiny segments.

Troubleshooting

Stretch Goals

• Make the win unambiguous. Index 1–2 million docs (raise the loop counts), force-merge to max_num_segments=8, ensure 8 fat segments, and re-run. With idle cores you should see a clear concurrent win. Tabulate took vs segment count.
• Sweep max_slice_count. Set it to 1, 2, 4, 8 and profile each — watch the slice count in the profile change, and the took with it. (This is the warm-up for Lab CS3.)
• Profile a non-agg query. Run a top-K match query with profile=true and find the slice structure in the query collector tree (not the agg tree).
• Correlate with CPU. Run top/htop while looping concurrent searches and confirm multiple cores light up; flip to mode: none and confirm one core does.

Coding Exercises

Reading the profile is half the literacy test; the other half is turning what you observed into code that asserts it. These exercises move you from curl + jq to tests and instrumentation that pin the slicing behavior in place.

1. (warm-up) A profile-parsing assertion in Python. Write count_slices.py that POSTs the AGG body to _search?profile=true, walks profile.shards[0].searches[0].collector recursively, and returns the number of leaf slice collectors. Assert (with assert/sys.exit(1)) that the count is > 1 when mode=all and == 1 after _forcemerge?max_num_segments=1. This is Step 6 + Step 8 turned into a pass/fail script you can re-run in CI.

2. (warm-up) Derive the expected slice count in code. Extend the script to also GET _cat/segments/csslab?h=segment,docs.count and compute the slice count you expect from Lucene's heuristic (maxDocsPerSlice / maxSegmentsPerSlice — find the constants with rg "maxDocsPerSlice|maxSegmentsPerSlice" lucene/core/src/.../IndexSearcher.java in a Lucene checkout). Print observed-vs-derived side by side and flag a mismatch. This grades Deliverable #2 automatically.

3. (core) An integration test that asserts the profile is concurrent. In an OpenSearch checkout, locate the existing concurrent-search profile tests with rg -l "concurrent" server/src/internalClusterTest/java/org/opensearch/search/profile/ and read one. Then write a new OpenSearchIntegTestCase method: index enough docs to force ≥4 segments, set search.concurrent_segment_search.mode=all, run a profiled aggregation via the Java client, and assert the response's collector tree contains more than one slice collector. Verify with ./gradlew :server:internalClusterTest --tests '*YourTest*'.

4. (core) Assert the threadpool contrast. Write an integration test (or extend the one above) that runs the heavy agg under mode=all and under mode=none, then reads index_searcher pool stats from NodesStatsResponse (find the pool name with rg "INDEX_SEARCHER|index_searcher" server/src/main/java/org/opensearch/threadpool/ThreadPool.java) and asserts completed advanced only in concurrent mode. This is Step 7's _cat/thread_pool observation, made into a regression guard.

5. (core) A settings round-trip unit test. Find the Setting<> definitions for search.concurrent_segment_search.mode and search.concurrent.max_slice_count (rg "concurrent_segment_search|concurrent.max_slice_count" server/src/main/java/ | rg "Setting"). Write an OpenSearchTestCase that constructs them, asserts the default value you recorded in Step 1, and asserts an invalid mode string is rejected. This locks the defaults you depend on across the lab.

6. (advanced challenge) A standalone slice-count probe. Build a small Java program (Lucene only, no OpenSearch — reuse the classpath setup from Lab CS2) that opens an index directory you point it at, constructs new IndexSearcher(reader, executor), calls getSlices(), and prints leaves, slices, and the docs-per-slice distribution. Then add a flag that overrides the slice computation with a fixed maxDocsPerSlice and re-prints — empirically reproducing the max_slice_count sweep from the Stretch Goals outside the OpenSearch process. Wrap it in a JUnit test asserting that a smaller maxDocsPerSlice yields more slices for the same reader. This is the bridge to Lab CS3 and Capstone Project 3.

Issues to Practice On

Concurrent segment search work lives in the core repo, mostly under the Search area. Reproduce first, then locate with rg, then fix with a test.

# Open, beginner-friendly issues:
gh issue list --repo opensearch-project/OpenSearch --label "good first issue" --state open
# Search-area issues (the home of concurrent-search work):
gh issue list --repo opensearch-project/OpenSearch --label "Search" --state open
gh issue list --repo opensearch-project/OpenSearch --search "concurrent segment search" --state open
# Flaky tests — concurrency surfaces these constantly:
gh issue list --repo opensearch-project/OpenSearch --label "flaky-test" --state open
# Confirm the live label taxonomy first (labels move; confirm on the tracker):
gh label list --repo opensearch-project/OpenSearch | rg -i "search|concurren|flaky|perf"

Two representative patterns for this subsystem:

• "Concurrent search is slower / not faster for query X." Reproduce with the Lab CS1 profile + _cat/thread_pool workflow, derive the slice count, and show whether the workload is in the overhead regime (tiny segments) or genuinely mis-sliced. Many such reports are working as designed and the right outcome is a doc/comment PR explaining the trade — itself a valuable first contribution.
• A flaky concurrent-search test. These almost always trace to an order-dependent reduce or a shared-state collector (the bug you build in Lab CS2). Locate the manager with rg "implements CollectorManager", reproduce under -Dtests.iters=50, and fix with a deterministic reduce plus an assertion.

Planted bug exercise. In an OpenSearch checkout, find where the searcher decides to slice: rg -n "getSlices|shouldReverseLeafReaderContexts|determineMaxSlice|slice" server/src/main/java/org/opensearch/search/internal/ContextIndexSearcher.java. Temporarily clamp the slice count to 1 (e.g. force the executor branch to fall back to sequential, or hard-cap the slice array length). Rebuild, run the concurrent-search integration tests (./gradlew :server:internalClusterTest --tests '*oncurrent*'), and note which test goes red and what it asserts. Then revert, and add a new assertion that the profile contains >1 slice for a multi-segment index — the test that would have caught your sabotage. This proves you understand exactly which invariant the slicing path guarantees.

Etiquette: claim an issue with a comment before working it, reproduce before proposing a fix, and every PR needs a test + a CHANGELOG.md entry + a DCO Signed-off-by line (git commit -s). See community interaction and the good-first-issue PR lab.

Validation / Self-check

• You can point at the line in the profile JSON that tells you how many slices ran.
• You can derive the expected slice count from _cat/segments and explain any mismatch with the profile.
• You can show, from _cat/thread_pool, which pool ran the slices and prove it goes idle when mode: none.
• You can state, with your own numbers, whether concurrency helped on your box and why — tying it back to segment sizes, core count, and query cost from the intensive trade-offs table.

Next: Lab CS2 — CollectorManager and Slice Reduction.