Lab 7.1: Trace a Search Through Query and Fetch Phases

Background

A _search request travels through more layers than any other request in OpenSearch. It enters as HTTP, is parsed into a SearchRequest, fans out from the coordinating node to every target shard, runs a query phase on each shard, is reduced on the coordinator, fans out again for a fetch phase, and is assembled into a SearchResponse. If you cannot point at the class and method for each of those hops, you cannot debug a search bug, a slow query, or a partial failure.

This is a timed code-reading trace. You will not modify code. You will follow a single search from REST to response, leaving a grep trail at every hop, and then correlate your trace with the Profile API so the abstract phases become concrete timings on a real cluster.

Why This Lab Matters for Contributors

• Search bugs are reported as "wrong results" or "slow" — both require you to know which phase.
• Partial failures (_shards.failed > 0) only make sense if you know the fan-out.
• Pagination cost (from/size/search_after) is invisible until you see the reduce.
• Almost every search PR touches one of these five classes. You must read them before you change them.

Prerequisites

• A built OpenSearch checkout (Level 1). Confirm ./gradlew assemble succeeds.
• The ability to run a local node: ./gradlew run (single-node, REST on :9200).
• Level 7 index read, especially the query-then-fetch diagram.
• Recommended reading alongside: Search Execution deep dive.

Set up a shell variable so the curls are copy-pasteable:

export OS=http://localhost:9200

Step-by-Step Tasks

This lab is timed: aim for 90 minutes. Each step has a target time. If you blow past it, write down where you got stuck — that confusion is the lab's real output.

Step 1 — Spin up a cluster and index sample data (10 min)

./gradlew run

In a second terminal, create an index with a few shards so the fan-out is real (a single-shard index hides the scatter-gather):

curl -s -XPUT "$OS/flights" -H 'Content-Type: application/json' -d '{
  "settings": { "number_of_shards": 3, "number_of_replicas": 0 },
  "mappings": { "properties": {
    "carrier":  { "type": "keyword" },
    "dest":     { "type": "keyword" },
    "delay_min":{ "type": "integer" },
    "ts":       { "type": "date" }
  }}
}'

for i in $(seq 1 50); do
  curl -s -XPOST "$OS/flights/_doc" -H 'Content-Type: application/json' -d "{
    \"carrier\":\"C$((RANDOM%4))\",\"dest\":\"D$((RANDOM%6))\",
    \"delay_min\":$((RANDOM%120)),\"ts\":\"2026-06-$((1+RANDOM%15))T00:00:00Z\"
  }" >/dev/null
done
curl -s -XPOST "$OS/flights/_refresh" >/dev/null
echo "indexed"

Run a baseline search and keep the response open in another buffer:

curl -s "$OS/flights/_search?size=5&sort=delay_min:desc" \
  -H 'Content-Type: application/json' -d '{ "query": { "range": { "delay_min": { "gte": 30 } } } }' | head -40

Step 2 — Hop 1: REST parsing in RestSearchAction (10 min)

The HTTP request is handled by a RestHandler. Find it:

grep -rn "class RestSearchAction" server/src/main/java/org/opensearch/rest/action/search/
grep -n "_search" server/src/main/java/org/opensearch/rest/action/search/RestSearchAction.java | head

Read prepareRequest(...) and parseSearchRequest(...). Answer in your reading log:

• Which method maps the URL params (from, size, search_type, scroll) onto the SearchRequest/SearchSourceBuilder?
• Where does the request body JSON get parsed into a SearchSourceBuilder? (look for SearchSourceBuilder.fromXContent or parseSearchSource).
• What ActionType is dispatched? (grep for client.execute( / SearchAction.INSTANCE.)

grep -n "SearchAction" server/src/main/java/org/opensearch/rest/action/search/RestSearchAction.java

Step 3 — Hop 2: coordinating node fan-out in TransportSearchAction (20 min)

This is the heart of the coordinating node. Locate it:

grep -rn "class TransportSearchAction" server/src/main/java/org/opensearch/action/search/

Read doExecute(...) and follow it into the search phase machinery. The fan-out is implemented as a sequence of SearchPhase objects. Grep for the phase classes:

ls server/src/main/java/org/opensearch/action/search/ | grep -i "Phase\|SearchAction\|Controller"

You are looking for the abstract async search phase driver (often AbstractSearchAsyncAction) and concrete phases. Trace these questions:

• How does the coordinator decide which shards to target? (index resolution → RoutingTable → GroupShardsIterator<SearchShardIterator>). Grep:

  grep -rn "SearchShardIterator\|GroupShardsIterator" server/src/main/java/org/opensearch/action/search/ | head
  

• Where is the can_match pre-filter phase? It can prune shards before the query phase:

  grep -rn "CanMatch\|canMatch" server/src/main/java/org/opensearch/action/search/ | head
  

• What is the difference between query_then_fetch and dfs_query_then_fetch in the dispatch?

  grep -rn "DFS_QUERY_THEN_FETCH\|QUERY_THEN_FETCH" server/src/main/java/org/opensearch/action/search/SearchType.java
  

Note: The coordinating node never holds the documents during the query phase — it holds SearchPhaseResult objects containing top doc IDs, scores, sort values, and aggregation partials. Confirm this by reading what the query-phase results class carries:

grep -rn "class QuerySearchResult" server/src/main/java/org/opensearch/search/query/

Step 4 — Hop 3: the per-shard query phase (15 min)

The fan-out lands on each target shard at SearchService. Find the entry point:

grep -n "executeQueryPhase" server/src/main/java/org/opensearch/search/SearchService.java | head

Follow executeQueryPhase into QueryPhase:

grep -rn "class QueryPhase" server/src/main/java/org/opensearch/search/query/
grep -n "execute\|executeInternal\|TopDocs\|collector" server/src/main/java/org/opensearch/search/query/QueryPhase.java | head -30

Answer:

• Where is the SearchContext/DefaultSearchContext created, and what does it hold? (the Lucene IndexSearcher, the parsed Query, the from/size, the aggregators).

  grep -rn "class DefaultSearchContext" server/src/main/java/org/opensearch/search/internal/
  

• Where does the QueryBuilder become a Lucene Query? (look for searchContext.query() set via QueryShardContext.toQuery).
• How many docs does each shard collect for a request of from=10000&size=10? (answer: top from + size = 10010 — this is the deep-pagination cost).

Step 5 — Hop 4: the coordinating-node reduce in SearchPhaseController (10 min)

After all shards return their query results, the coordinator merges them:

grep -rn "class SearchPhaseController" server/src/main/java/org/opensearch/action/search/
grep -n "reducedQueryPhase\|sortDocs\|merge" server/src/main/java/org/opensearch/action/search/SearchPhaseController.java | head -20

This is where:

• The global top size is computed from per-shard top-Ks (a merge of sorted lists).
• Aggregations are reduced (InternalAggregations.reduce / InternalAggregation.reduce).
• The coordinator decides which doc IDs on which shards must be fetched.

Find the agg reduce call:

grep -rn "InternalAggregation" server/src/main/java/org/opensearch/action/search/SearchPhaseController.java | head

Step 6 — Hop 5: the fetch phase (10 min)

Only winning docs are fetched. Entry point:

grep -n "executeFetchPhase" server/src/main/java/org/opensearch/search/SearchService.java | head
grep -rn "class FetchPhase" server/src/main/java/org/opensearch/search/fetch/

Answer:

• What does FetchPhase load that the query phase did not? (_source, stored fields, highlights, script fields — via FetchSubPhase implementations).

  ls server/src/main/java/org/opensearch/search/fetch/subphase/
  

• Why is the fetch fan-out usually smaller than the query fan-out? (only shards owning winners).

Step 7 — Correlate with the Profile API (15 min)

Now make the phases concrete. The Profile API instruments per-shard timings:

curl -s "$OS/flights/_search" -H 'Content-Type: application/json' -d '{
  "profile": true,
  "size": 5,
  "sort": [ { "delay_min": "desc" } ],
  "query": { "bool": { "must": [
    { "range": { "delay_min": { "gte": 30 } } },
    { "terms": { "carrier": ["C0","C1"] } }
  ]}},
  "aggs": { "by_carrier": { "terms": { "field": "carrier" } } }
}' > /tmp/profile.json

# How many shards reported? (this is your fan-out width)
grep -c '"shard_id"' /tmp/profile.json 2>/dev/null || python3 -c "import json;d=json.load(open('/tmp/profile.json'));print('shards:',len(d['profile']['shards']))"

Inspect one shard's profile. You will see the query rewritten into Lucene query nodes (BooleanQuery, IndexOrDocValuesQuery, TermInSetQuery), each with time_in_nanos broken into create_weight, build_scorer, next_doc, score, etc., plus a collector tree and an aggregations section.

python3 - <<'PY'
import json
d=json.load(open('/tmp/profile.json'))
sh=d['profile']['shards'][0]
print("shard:", sh['id'])
q=sh['searches'][0]['query'][0]
print("top query node:", q['type'], "time_ns:", q['time_in_nanos'])
print("collector:", sh['searches'][0]['collector'][0]['name'])
print("aggs:", [a['type'] for a in sh.get('aggregations',[])])
PY

Map what you see back onto your trace:

Note: The Profile API only shows per-shard work. The coordinator-side reduce and the fetch phase are not in the profile output. That gap is itself a lesson: profiling tells you about shard cost, not coordinator cost. Deep-pagination pain lives in the reduce, which the profile cannot see.

Pagination: from/size vs search_after (reading exercise)

Run all three and compare what the coordinator must do:

# Deep from/size — every shard must produce top 110 docs
curl -s "$OS/flights/_search?from=100&size=10&sort=delay_min:desc" -o /dev/null -w "from/size: %{http_code}\n"

# search_after — cursor by the last page's sort values (cheap, stateless)
LAST=$(curl -s "$OS/flights/_search?size=5&sort=delay_min:desc" | python3 -c "import json,sys;h=json.load(sys.stdin)['hits']['hits'];print(json.dumps(h[-1]['sort']))")
curl -s "$OS/flights/_search" -H 'Content-Type: application/json' -d "{
  \"size\":5, \"sort\":[{\"delay_min\":\"desc\"}], \"search_after\": $LAST }" -o /dev/null -w "search_after: %{http_code}\n"

In your log, explain: for from=10000, why does every shard pay, and why does search_after not? (Because from/size requires each shard to materialize the top from+size; search_after seeks each shard directly to the cursor and collects only size.)

Implementation Requirements

This is a reading/tracing lab. Your deliverable is a reading-log artifact — a Markdown file with one section per hop:

## Hop 1: RestSearchAction
- file: server/src/main/java/org/opensearch/rest/action/search/RestSearchAction.java
- grep used: grep -n "_search" .../RestSearchAction.java
- method: prepareRequest -> parseSearchRequest
- what crosses to next hop: a SearchRequest (with SearchSourceBuilder)

## Hop 2: TransportSearchAction  ... (etc for all five hops)

Each hop section must contain: the file path, the grep command you ran, the method, and what crosses the wire to the next hop.

Expected Output

• A flights index with 3 shards and ~50 docs.
• A Profile API JSON whose shard count equals your index's shard count.
• A reading log covering all five hops plus the pagination exercise.
• A clear statement of which two things the Profile API does not show (reduce, fetch).

Troubleshooting

Stretch Goals

1. Add "explain": true to a search and correlate the BM25 explanation with the score timing in the profile. Where does the term frequency come from on a single shard?
2. Force search_type=dfs_query_then_fetch and diff the scores against query_then_fetch on a multi-shard index. Explain the difference using the DfsPhase.
3. Find where terminate_after short-circuits the query-phase collector:

   grep -rn "terminate_after\|terminateAfter" server/src/main/java/org/opensearch/search/ | head
   

Validation / Self-check

You are done when you can answer, without re-reading:

1. Name the five classes, in order, that a _search passes through from REST to response.
2. In the query phase, does a shard return documents? If not, what does it return?
3. Why is from=10000&size=10 expensive, and which class pays the cost?
4. What does the fetch phase load that the query phase did not, and why is its fan-out smaller?
5. Where does a QueryBuilder become a Lucene Query, and what context object performs the conversion?
6. What does can_match do, and which class implements it?
7. Name two things the Profile API does not measure.

Cross-references: Search Execution deep dive, Query DSL and QueryBuilders, Lab 7.2: Aggregations, Capstone Step 2: Reproduction.