Lab P4: Bug Attribution — Core vs. Plugin vs. Dashboards vs. Lucene

Background

This is the keystone cross-repo skill, the direct analog of the Tez curriculum's "Hive-on-Tez attribution." A user reports a symptom — wrong results, an error, slowness. Before a single line of code is written, someone must decide which repository owns it: OpenSearch core, a named plugin (k-NN, security, sql, …), OpenSearch Dashboards, or Apache Lucene. File it on the wrong repo and the maintainer spends a week bouncing it before anyone touches the actual bug.

This lab gives you a decision flowchart, an attribution table, the "reproduce-without-the-suspect" bisection technique, and four worked examples — one per repo. It depends on the reading skill from P3, the request trace from P1, and the plugin↔SPI map from P2. It is the operational core of Stage 8.

Why This Lab Matters for Contributors

Attribution is where cross-repo contributors earn their reputation. A maintainer who consistently attributes correctly and attaches a minimal repro in the right repo is trusted; their issues get picked up fast. The bisection technique here — "reproduce the symptom with the suspect removed" — is the most powerful tool you have, because it converts an opinion ("I think it's core") into a proof ("here's the same bug with no plugin and no Dashboards, on a stock distro").

The decision flowchart

flowchart TD
  S[Symptom reported]
  S --> Q1{Reproducible with raw curl<br/>against core, no Dashboards?}
  Q1 -->|No, only via the UI| DASH[Dashboards owns it<br/>render/agg-config/time-zone]
  Q1 -->|Yes| Q2{Reproducible with the<br/>plugin uninstalled?}
  Q2 -->|Yes, no plugin needed| Q3{Does it involve a query type,<br/>field type, or scoring?}
  Q2 -->|No, needs the plugin| PLUG[A plugin owns it<br/>find which from the field/query]
  Q3 -->|Yes, and result is numerically wrong| Q4{Reproducible on a single shard<br/>with a trivial Lucene-level query?}
  Q3 -->|No — error, allocation, cluster state| CORE[Core owns it]
  Q4 -->|Yes, even a match_all/term is wrong| LUCENE[Lucene owns it<br/>codec/docvalues/scorer]
  Q4 -->|No, only the composite/agg is wrong| CORE
  PLUG --> Q5{Is the plugin's error/message<br/>itself the problem?}
  Q5 -->|Message unclear / wrong| PLUGB[Plugin bug — file on plugin]
  Q5 -->|Message clear, config is wrong| USER[User/config — no bug]

The flowchart encodes one idea: remove suspects one at a time and see if the symptom survives. Each "reproducible with X removed?" answer eliminates a repo.

The attribution table

The bisection technique: reproduce without the suspect

The single most valuable move. For each suspect repo, there's a way to take it out of the picture and see whether the bug survives.

Note: "Remove the suspect" sometimes means re-create the symptom with stock tools. You can't uninstall Lucene, but you can reproduce a scoring bug with a 5-line Lucene IndexSearcher test that uses no OpenSearch at all — if that reproduces it, you've proven Lucene owns it and can file apache/lucene with a Lucene-only repro.

Worked Example 1 — Wrong agg result (core vs Lucene)

Symptom: a sum aggregation over a double field returns a value that's off by a tiny amount on a multi-shard index.

Bisect:

# Reproduce via curl (removes Dashboards)
curl -s 'localhost:9200/sales/_search' -H 'Content-Type: application/json' -d '{
  "size":0, "aggs":{"t":{"sum":{"field":"amount"}}}}'

# Single-shard index (removes the multi-shard reduce)
curl -s -XPUT localhost:9200/sales1 -H 'Content-Type: application/json' \
  -d '{"settings":{"number_of_shards":1}}'
# reindex and re-run the sum

• If single-shard is correct but multi-shard is wrong, the fault is in core's cross-shard reduce (SearchPhaseController / InternalSum.reduce) — file on OpenSearch. Floating-point sum order across shards is a classic core concern; see Aggregations.
• If single-shard is also wrong, the per-shard aggregation read the wrong doc-values — drop to a Lucene SortedNumericDocValues read. If a Lucene-only test reproduces it, file apache/lucene. (Usually it's core's agg, not Lucene; doc-values reads are extremely well-tested.)

Attribution most often: core, the reduce. The decisive evidence is "single-shard correct, multi-shard wrong."

Worked Example 2 — A visualization renders wrong (Dashboards vs core)

Symptom: a date histogram bar chart shows counts in the wrong day buckets.

Bisect: capture the request (P1 Step 3) and replay it.

curl -s 'localhost:9200/sales/_search?typed_keys=true' -H 'Content-Type: application/json' -d '{
  "size":0,
  "aggs":{"h":{"date_histogram":{"field":"ts","calendar_interval":"day","time_zone":"+00:00"}}}}'

• If the buckets in the JSON are correct but the chart is wrong, the bug is in Dashboards' rendering — file OpenSearch-Dashboards.
• If the JSON buckets are wrong, look at the time_zone Dashboards put in the request. Dashboards sends the browser's time zone; if it sends the wrong one, that's still a Dashboards bug (wrong request), even though core honored it correctly. Core only owns it if core mis-bucketed for a correct time_zone — reproduce with an explicit time_zone via curl and check against a manual calculation.

Attribution most often: Dashboards — either rendering or the time_zone it chose. Core's date_histogram honoring an explicit time zone is rarely wrong.

Worked Example 3 — Vector search returns bad neighbors (k-NN vs core vs Lucene)

Symptom: a knn query returns neighbors that are obviously not the closest.

Bisect against a brute-force ground truth that uses no approximate engine:

# Approximate (k-NN ANN engine)
curl -s localhost:9200/vec/_search -H 'Content-Type: application/json' -d '{
  "query":{"knn":{"v":{"vector":[1,2,3],"k":3}}}}'

# Exact, scripted distance (removes the ANN engine; still uses the plugin's script)
curl -s localhost:9200/vec/_search -H 'Content-Type: application/json' -d '{
  "query":{"script_score":{"query":{"match_all":{}},
    "script":{"source":"knn_score","lang":"knn",
      "params":{"field":"v","query_value":[1,2,3],"space_type":"l2"}}}}}'

• If exact returns the right neighbors but approximate does not, the approximation is the issue: the k-NN engine (faiss/nmslib/Lucene HNSW) config — ef_search, m, index params — or a genuine engine bug. File on k-NN (it owns EnginePlugin.getEngineFactory from P2). Often it's a recall-vs-config tradeoff, not a bug.
• If both are wrong, the vectors were stored or read wrong — k-NN's codec, or Lucene's HNSW codec if the engine is lucene. Narrow by switching the mapping's engine and seeing which engine reproduces it.
• If the field type or query is rejected, that's the P3 Shape-B mismatch, owned by the mapping/config.

Attribution: k-NN (engine/config) is most common; Lucene only if the lucene HNSW engine is in use and even exact reads are wrong.

Worked Example 4 — A 403 (security)

Symptom: a request returns 403 security_exception.

Bisect: this needs the security plugin to even occur, so it's a security/policy matter unless core mis-declared the action.

# What privilege did the action require?  (the message names it)
# "no permissions for [indices:data/read/search] and User [name=guest]"
# Check the role mapping grants that action:
curl -sk -u admin:admin 'https://localhost:9200/_plugins/_security/api/roles/<role>' | python3 -m json.tool

• If the role should grant indices:data/read/search but doesn't, it's a config issue — fix the role, no bug.
• If the privilege string in the error doesn't match the action the user actually invoked, core may have registered the wrong ActionType name — a rare core bug; confirm with grep in core's ActionModule.
• If security denies an action it should allow given a correct role, that's a security plugin bug.

Attribution: usually config; occasionally security; rarely core.

Writing the bug report in the right repo

Once attributed, the report goes in the owning repo with a repro that excludes the other repos:

The discipline of excluding the other repos from the repro is what proves your attribution. A core bug report that requires installing k-NN and Dashboards to reproduce hasn't been attributed — it's been punted.

Expected Output

• The decision flowchart applied to one symptom of your choosing, written out.
• For two of the four worked examples, the actual curl bisection run on your local cluster with the result.
• One bug-report skeleton for the repo you attributed to, with a repro that does not drag in the other repos.

Stretch Goals

1. Take a real open issue from OpenSearch-Dashboards and decide, from its description alone, whether it's truly a Dashboards bug or a mis-filed core bug.
2. Build a Lucene-only IndexSearcher repro for a scoring question and confirm it reproduces (or doesn't) outside OpenSearch.
3. Write the "reproduce without the suspect" steps as a checklist you could hand to a triager.

Coding Exercises

Attribution is a judgment, but a proven attribution is code: the repro that excludes the other repos, run as a test. These exercises turn each branch of the flowchart into a runnable bisection and a repro that lives in the right repo. Locate real classes with rg/find; names drift by branch, so confirm them.

1. (warm-up) Encode the bisection as a verdict script. Write a bash script that runs the Worked Example 1 sum on a multi-shard index and on a single-shard index, compares the two values, and prints core/reduce vs lucene/per-shard per the rule. Make it deterministic (fixed data) and assert the verdict. You have converted "single-shard correct, multi-shard wrong ⇒ core reduce" into a test.

2. (core) Prove a Lucene-only repro for a scoring question. Write a standalone JUnit test using a Lucene IndexSearcher directly (no OpenSearch) — index two docs, run a TermQuery, and assert the score/order. Find a model in core's own Lucene-touching tests (rg -l "new IndexSearcher\(" ~/OpenSearch/server/src/test). This is the "you can't uninstall Lucene, but you can reproduce with 5 lines" move — the repro that would let you file apache/lucene cleanly.

3. (core) Reproduce the multi-shard reduce branch as an integ test. Write an OpenSearchIntegTestCase that indexes double values across multiple shards, runs a sum agg, and asserts against a hand-computed total — then a single-shard variant that passes. Find the reduce code you're attributing to (rg -n "class InternalSum|reduce\(" ~/OpenSearch/server/src/main/java/org/opensearch/search/aggregations/metrics/InternalSum.java). This makes Worked Example 1's core-vs-Lucene split executable.

4. (core) Build the k-NN ground-truth bisection (Worked Example 3) as a test. Write an integ test (with nodePlugins() loading the plugin) that indexes fixed vectors, runs both the approximate knn query and an exact script_score brute-force, and asserts they agree on the nearest neighbor. When they disagree, the test is the proof that the engine/config owns it — a k-NN-repo artifact, no Dashboards, no core changes.

5. (core) Reproduce a cross-plugin bug with an integ test, attributed. Pick one symptom (e.g. a mapping/engine mismatch from P3), write the integ test that reproduces it with the minimum repos involved, and in a comment annotate which flowchart branch each assertion eliminates. The test plus annotation is a bug report's Steps to reproduce in executable form.

6. (advanced challenge) Build a triage harness that emits the right repo. Write a program that takes a symptom descriptor (has-Dashboards? reproduces-via-curl? reproduces-without-plugin? single-shard? agg-only?) and walks the decision flowchart to output the owning repo and the decisive test to run, then actually runs the matching curl/Gradle command and confirms the verdict. Drive it through all four worked examples on your local cluster. You have built the Stage 8 attribution engine the whole lab describes — and each run produces a repro that excludes the other repos.

Issues to Practice On

Attribution is most valuable on the issues people get wrong; practice across all the candidate repos so the flowchart becomes reflex.

gh issue list --repo opensearch-project/OpenSearch --label "untriaged" --state open
gh issue list --repo opensearch-project/OpenSearch --label "bug" --state open
gh issue list --repo opensearch-project/OpenSearch-Dashboards --label "bug" --state open
gh issue list --repo opensearch-project/k-NN --label "bug" --state open
gh label list --repo opensearch-project/OpenSearch | grep -iE "triag|search|aggreg|lucene"

Labels move; confirm on each tracker before relying on a name. Lucene uses GitHub issues too (gh issue list --repo apache/lucene --label bug).

Two representative patterns:

• A mis-filed cross-repo issue. Take an open OpenSearch-Dashboards bug and decide, from its description, whether the request Dashboards built was wrong (Dashboards) or core mis-answered a correct request (core). Prove it by replaying the request with curl. Post the attribution with the decisive evidence — this is exactly Stretch Goal 1, now your default triage move.
• An untriaged "wrong number" issue. Apply the flowchart top to bottom, removing one suspect at a time, and comment with the repo and the repro that excludes the others. Correct attributions with minimal repros are how cross-repo contributors build reputation.

Planted-bug drill. Take Worked Example 1's reduce path (rg -n "doReduce|reduce\(" ~/OpenSearch/server/src/main/java/org/opensearch/search/aggregations/metrics/InternalSum.java) and perturb the cross-shard combine — e.g. drop one shard's partial from the accumulation. Run your multi-shard integ test (exercise 3) and watch it fail while the single-shard test stays green: the failure signature is "single-shard right, multi-shard wrong ⇒ core reduce," exactly as the flowchart predicts. Revert, then add an assertion comparing the multi-shard sum to the single-shard sum so the attribution signature is permanently guarded.

Etiquette: Claim an issue before working it; reproduce first; the repro must exclude the other repos to prove the attribution; every PR needs a test + a CHANGELOG entry + a DCO Signed-off-by (git commit -s). See community interaction and the PR-review lab.

Validation / Self-check

1. State the one-sentence rule for the entire flowchart.
2. For each of the four repos, give the single decisive test that removes it from suspicion.
3. A sum is wrong on a multi-shard index but right on one shard. Owner? Why?
4. A date histogram chart is wrong but the captured request's buckets are correct. Owner? Why?
5. Why does the requirement "the repro must exclude the other repos" actually prove the attribution rather than just being tidy?
6. When is a 403 a core bug, and how would you confirm it?