Lab AG1: Trace an Aggregation

Background

You are going to take one concrete request — "average price per category, top 3 categories" — and follow it the whole way down: from the JSON, through the aggregation profiler (which times the real collect/build/reduce phases), into the OpenSearch source with grep, until you can point at the exact line where a document's category ordinal becomes a bucket and the avg child gets driven with that bucket's ordinal. By the end you will have seen, not just read about, owningBucketOrd, the GlobalOrdinalsStringTermsAggregator, and InternalTerms.reduce.

This is the companion lab to the Aggregations intensive. Re-read its "collection lifecycle" and "bucket-ordinal model" sections before starting — this lab makes those concrete.

Why This Matters for Contributors

Every aggregation bug report — "counts are wrong," "it's slow," "it OOMs" — is debugged by answering the same three questions: which aggregator class ran, where did the time go (collect vs build vs reduce), and how did the bucket ordinals flow? The profiler answers the first two; reading the source answers the third. A contributor who can do this trace in fifteen minutes can triage an aggregation issue that an outsider stares at for a day. You are building that reflex.

Prerequisites

• A running OpenSearch (3.x) — ./gradlew run from an OpenSearch checkout, or a single-node tarball/Docker. curl localhost:9200 must return cluster info.
• An OpenSearch source checkout at ~/src/OpenSearch for the grep steps.
• jq installed (pretty-print the profile JSON). curl ... | jq throughout.
• Read Aggregations intensive and the docvalues/fielddata deep-dive.

Step-by-Step Tasks

Step 1 — Index data with a keyword field

Global-ordinals execution needs a keyword (or keyword sub-field). Force it explicitly so you know which engine runs.

curl -s -XDELETE 'localhost:9200/shop' >/dev/null

curl -s -XPUT 'localhost:9200/shop' -H 'Content-Type: application/json' -d '{
  "settings": { "number_of_shards": 1, "number_of_replicas": 0 },
  "mappings": { "properties": {
    "category": { "type": "keyword" },
    "price":    { "type": "double" }
  }}
}'

curl -s -XPOST 'localhost:9200/shop/_bulk' -H 'Content-Type: application/json' -d '
{"index":{}}
{"category":"books","price":10}
{"index":{}}
{"category":"books","price":20}
{"index":{}}
{"category":"toys","price":30}
{"index":{}}
{"category":"toys","price":50}
{"index":{}}
{"category":"games","price":40}
{"index":{}}
{"category":"books","price":12}
'
curl -s -XPOST 'localhost:9200/shop/_refresh' >/dev/null

Note: One shard, no replicas, on purpose. With one shard the coordinator reduce is a pass-through, so any timing you see in the profile is pure shard work. You add shards in the stretch goals to watch reduce light up.

Step 2 — Run the aggregation and read the result

curl -s 'localhost:9200/shop/_search' -H 'Content-Type: application/json' -d '{
  "size": 0,
  "aggs": {
    "by_category": {
      "terms": { "field": "category", "size": 3 },
      "aggs": { "avg_price": { "avg": { "field": "price" } } }
    }
  }
}' | jq '.aggregations'

Expected (counts exact, avg per bucket):

{
  "by_category": {
    "doc_count_error_upper_bound": 0,
    "sum_other_doc_count": 0,
    "buckets": [
      { "key": "books", "doc_count": 3, "avg_price": { "value": 14.0 } },
      { "key": "games", "doc_count": 1, "avg_price": { "value": 40.0 } },
      { "key": "toys",  "doc_count": 2, "avg_price": { "value": 40.0 } }
    ]
  }
}

• Record the three buckets and their avg_price. Note books=14.0 is (10+20+12)/3 — the avg child accumulated into bucket-ord 0.

Step 3 — Turn on the profiler and read the aggregation breakdown

curl -s 'localhost:9200/shop/_search' -H 'Content-Type: application/json' -d '{
  "profile": true,
  "size": 0,
  "aggs": {
    "by_category": {
      "terms": { "field": "category", "size": 3 },
      "aggs": { "avg_price": { "avg": { "field": "price" } } }
    }
  }
}' | jq '.profile.shards[0].aggregations'

You get a tree of timed aggregators. Pull out the names and the phase timers:

curl -s 'localhost:9200/shop/_search' -H 'Content-Type: application/json' -d '{
  "profile": true, "size": 0,
  "aggs": { "by_category": {
    "terms": { "field": "category", "size": 3 },
    "aggs": { "avg_price": { "avg": { "field": "price" } } } } }
}' | jq '.profile.shards[0].aggregations[] |
  { type, description,
    initialize: .breakdown.initialize,
    collect: .breakdown.collect,
    build_aggregation: .breakdown.build_aggregation,
    reduce: .breakdown.reduce,
    children: [.children[]? | { type, build_aggregation: .breakdown.build_aggregation }] }'

• The type field is the real Java class name. For category you should see something containing GlobalOrdinalsStringTermsAggregator (or MapStringTermsAggregator — record which). The child is an AvgAggregator.
• Note the four phase timers: initialize, collect, build_aggregation, reduce (nanoseconds). collect is the per-doc fold; build_aggregation is buildAggregations; reduce is the coordinator reduce (tiny here).

Note: description is the agg name from your request (by_category); type is the implementation class. The mapping between them is the whole point of the profiler — it tells you which engine the factory picked.

Step 4 — Confirm the terms execution engine

The two engines (global ordinals vs map) perform very differently at scale. Find where the factory chooses:

cd ~/src/OpenSearch
grep -n "GlobalOrdinalsStringTermsAggregator\|MapStringTermsAggregator\|ExecutionMode\|globalOrdinalsValues\|remapGlobalOrds" \
  server/src/main/java/org/opensearch/search/aggregations/bucket/terms/TermsAggregatorFactory.java

• Find the ExecutionMode enum (GLOBAL_ORDINALS, MAP) and the method that picks one. Read the comment on when MAP is chosen.

Now force the other engine and re-profile to see type change:

curl -s 'localhost:9200/shop/_search' -H 'Content-Type: application/json' -d '{
  "profile": true, "size": 0,
  "aggs": { "by_category": {
    "terms": { "field": "category", "size": 3, "execution_hint": "map" } } }
}' | jq '.profile.shards[0].aggregations[].type'

• Confirm the class changed to MapStringTermsAggregator. You just observed the factory's ExecutionMode decision from the outside.

Step 5 — Trace collect: where a doc becomes a bucket

Open the global-ordinals aggregator and find its getLeafCollector / collect(int doc, long owningBucketOrd):

grep -n "getLeafCollector\|void collect(int doc\|owningBucketOrd\|bucketOrds.add\|collectBucket\|collectExistingBucket" \
  server/src/main/java/org/opensearch/search/aggregations/bucket/terms/GlobalOrdinalsStringTermsAggregator.java

Read the collect body and answer in your log:

• What value does it read per doc? (a global ordinal from SortedSetDocValues.)
• What does bucketOrds.add(owningBucketOrd, globalOrd) return when the bucket is new vs already seen? (Hint: negative encodes "existing"; it does -1 - ord.)
• Which call drives the child avg? (collectBucket/collectExistingBucket → collectBucket(sub, doc, bucketOrd) → the sub's collect(doc, bucketOrd).) That bucketOrd is the avg's owningBucketOrd.

Now confirm the avg side reads from arrays indexed by that ordinal:

grep -n "class AvgAggregator\|DoubleArray\|LongArray\|sums\|counts\|void collect(int doc, long bucket" \
  server/src/main/java/org/opensearch/search/aggregations/metrics/AvgAggregator.java

• Confirm sums and counts are BigArrays-backed and indexed by bucket (the owning ord). Write the line: "books' sum lives in sums[0]."

Step 6 — Trace buildAggregations: ordinals become terms

grep -n "buildAggregations\|buildSubAggsForBuckets\|BucketPriorityQueue\|shardSize\|ordsToCollect\|lookupGlobalOrd" \
  server/src/main/java/org/opensearch/search/aggregations/bucket/terms/GlobalOrdinalsStringTermsAggregator.java

• Find where it picks the top-shard_size buckets (a priority queue) and where it resolves the surviving global ordinals back to BytesRef terms. Note: it only stringifies the survivors, not every ordinal — that's the global-ordinal win.
• Find buildSubAggsForBuckets — this is where each surviving bucket's avg child gets buildAggregations(bucketOrd) called to produce InternalAvg.

Step 7 — Trace reduce: shards become one answer

grep -n "reduce\|doReduce\|class InternalTerms\|sum_other_doc_count\|doc_count_error\|reduceBucket\|reduceContext" \
  server/src/main/java/org/opensearch/search/aggregations/bucket/terms/InternalTerms.java
grep -n "reduce\|class InternalAvg\|sum\|count" \
  server/src/main/java/org/opensearch/search/aggregations/metrics/InternalAvg.java

• In InternalTerms.reduce: find where it merges buckets by key, sums docCount, and recursively reduces sub-aggregations for each merged bucket (that's how the avg children combine across shards).
• In InternalAvg: confirm reduce sums the sums and sums the counts and divides once — associative and commutative, so the two-level reduce is safe. Write why "sum-then-divide" is order-independent but "average of averages" would not be.

Step 8 — Write the reading-log artifact

Create lab-ag1-trace.md capturing the full path. Template:

# Lab AG1 — Aggregation Trace

## Profile observations
- terms aggregator class (type): __________  (description: by_category)
- avg aggregator class (type):   AvgAggregator
- collect ns: ____  build_aggregation ns: ____  reduce ns: ____
- with execution_hint=map, type became: __________

## collect path (GlobalOrdinalsStringTermsAggregator)
- value read per doc: global ordinal from SortedSetDocValues
- bucketOrds.add(owningBucketOrd, globalOrd) returns:
    new bucket -> ____ ; existing bucket -> ____ (encoding: ____)
- child avg driven via: ____ -> collect(doc, bucketOrd)
- "books' running sum lives in sums[__]"

## buildAggregations path
- top-shard_size selection: ____ (data structure: ____)
- ordinals resolved to terms via: ____ (only for survivors? yes/no)
- sub-aggs built via: buildSubAggsForBuckets

## reduce path
- InternalTerms.reduce merges buckets by ____, sums ____, recursively reduces ____
- InternalAvg.reduce: sums ____, sums ____, divides ____ times
- Why associative+commutative: ____

## owningBucketOrd in one sentence
____

• Fill every blank from what you actually saw/read, not from memory.

Deliverables

• lab-ag1-trace.md completed with profile numbers and real class names.
• The two profile JSON captures (global ordinals and map engines) showing the type change.
• A one-paragraph explanation of owningBucketOrd and how the avg child is routed to the right bucket, in your own words.

Troubleshooting

Expected Output

• The profile shows a terms aggregator (global-ordinals class) with an AvgAggregator child, with nonzero collect and build_aggregation timers and a tiny reduce.
• execution_hint: map flips type to MapStringTermsAggregator.
• Your reading log names the exact methods on the collect → build → reduce path.

Stretch Goals

• Make reduce non-trivial. Recreate shop with number_of_shards: 3, re-index, and re-profile. Watch the reduce timer grow and doc_count_error_upper_bound potentially become nonzero. Explain the change using the deep-dive's shard_size discussion.
• Order by sub-metric. Add "order": { "avg_price": "desc" } to the terms agg and re-profile. Note the warning in the deep-dive about metric-order approximation; raise shard_size and observe.
• Concurrency. Set search.concurrent_segment_search.mode: all, force a few segments (index in batches with refreshes), and re-profile. Find the per-slice structure in the profile (a slices/max_slices field). Connect to the concurrent search masterclass slice-reduce level.
• Cardinality. Add a cardinality sub-agg on price and find HyperLogLogPlusPlus in the source; note how reduce merges sketches.

Coding Exercises

Tracing is only half-learned until you turn an observation into an assertion. Each exercise below makes you write code — a test, a patch, or a tiny program — that pins down something you saw in the trace. Locate every class with rg first (rg -l GlobalOrdinalsStringTermsAggregator server/) — never trust a remembered path or line number.

1. (warm-up) Assert the buckets you read off the profile. Write an OpenSearchIntegTestCase that indexes the six shop docs from Step 1, runs the terms+avg aggregation, and asserts the three buckets, their doc_counts, and books == 14.0. Model it on existing terms tests — find one with rg -l "class.*TermsIT|StringTermsIT" server/src/.../search/aggregations/bucket and copy its client().prepareSearch(...).addAggregation(...) shape. Green test = you encoded the Step 2 result as an executable fact.

2. (core) Unit-test the avg child's per-ord accumulation. Using AggregatorTestCase (find it with rg -l "class AggregatorTestCase"), write a test that indexes docs whose category ordinal you control, runs terms with an avg sub-agg over an in-memory Lucene index, and asserts each bucket's avg. This proves sums[owningBucketOrd] is routed correctly without a cluster — the same loop you traced in Step 5, now under JUnit.

3. (core) Instrument the execution-mode decision. Add a temporary logger.info in TermsAggregatorFactory at the branch that picks GLOBAL_ORDINALS vs MAP (the method you found in Step 4), printing the chosen ExecutionMode and the field. Rebuild, run both the default and execution_hint: map requests, and capture the two log lines. Then write a one-paragraph note tying each log line to the type you saw in the profile. (Revert the patch after — it's a probe, not a PR.)

4. (core) A reduce/partial-reduce test for InternalAvg. Write an OpenSearchTestCase that constructs several InternalAvg instances with known sum/count pairs and calls reduce(...) (find the signature with rg -n "InternalAggregation reduce" server/.../metrics/InternalAvg.java). Assert that reducing them all at once equals reducing them in two batches and then reducing the partials — i.e. partial reduce == full reduce. This is the executable form of the associativity argument from Step 7.

5. (advanced) Advanced challenge — a property-based reduce-order fuzzer. Write a parameterized OpenSearchTestCase that, for many random splits of N shard-level InternalAvg partials into random groups, asserts every grouping yields the same final average to within a float epsilon — and a contrasting test for a deliberately broken "average of averages" reducer (a tiny local subclass) that shows it does not hold. Then run the real terms+avg aggregation under search.concurrent_segment_search.mode: all (Step's stretch goal) and assert the concurrent result equals the sequential one. Deliverable: a single test file that proves the safe reducer is order-independent and the naive one is not, plus a short writeup connecting it to slice-reduce in concurrent search.

Issues to Practice On

Aggregation tracing is exactly the skill that triages the issues below. Work them on the core repo, opensearch-project/OpenSearch.

Label taxonomies drift, so confirm on the tracker first (gh label list --repo opensearch-project/OpenSearch | rg -i "search|agg").

Representative patterns. (1) "Counts/avg wrong with N shards but right with 1." — almost always a reduce or doc_count_error issue. Reproduce with a multi-shard index, then trace InternalTerms.reduce (Step 7); the planted-bug drill below rehearses exactly this. (2) "terms agg slower than expected on a high-cardinality keyword." — reproduce with the profiler, confirm which ExecutionMode ran (Step 4), and check whether MAP was chosen where global ordinals would win. Approach any of them the same way: reproduce → locate with rg → fix → add a test → PR with a CHANGELOG entry and DCO sign-off.

Planted-bug drill. In InternalAvg.reduce, change the divide so it averages the per-shard values instead of Σsum / Σcount (e.g. accumulate value() and divide by the number of partials). Rebuild and run the avg/terms test suite (rg -l "InternalAvgTests|AvgIT" server/). Watch which test goes red on a multi-shard or multi-partial case. Now revert, and add the assertion that would have caught it: a test where the partials have unequal counts (so "average of averages" visibly differs from the true weighted average). That single asserted case is the regression test a reviewer would demand.

Etiquette: claim an issue with a comment before working it, reproduce before proposing a fix, and remember every PR needs a test + CHANGELOG entry + DCO Signed-off-by (git commit -s). See community interaction and the prepare-a-PR lab.

Validation / Self-check

1. From the profile alone, how do you tell which terms engine ran, and why does it matter for performance?
2. Define owningBucketOrd and trace one document ({category: books, price:20}) from collect to the array slot sums[0].
3. What does bucketOrds.add return for a new vs existing bucket, and how is "existing" encoded?
4. In buildAggregations, why does the global-ordinals engine only stringify the surviving top-shard_size ordinals?
5. Argue InternalAvg.reduce is associative and commutative, and explain why "average of averages" would break the two-level reduce.

Next: Lab AG2 — Build a Custom Aggregation. You just traced an existing agg; next you write one end to end.