Lab P4: Bug Attribution — Core vs. Plugin vs. Dashboards vs. Lucene

Background

This is the keystone cross-repo skill, the direct analog of the Tez curriculum's "Hive-on-Tez attribution." A user reports a symptom — wrong results, an error, slowness. Before a single line of code is written, someone must decide which repository owns it: OpenSearch core, a named plugin (k-NN, security, sql, …), OpenSearch Dashboards, or Apache Lucene. File it on the wrong repo and the maintainer spends a week bouncing it before anyone touches the actual bug.

This lab gives you a decision flowchart, an attribution table, the "reproduce-without-the-suspect" bisection technique, and four worked examples — one per repo. It depends on the reading skill from P3, the request trace from P1, and the plugin↔SPI map from P2. It is the operational core of Stage 8.

Why This Lab Matters for Contributors

Attribution is where cross-repo contributors earn their reputation. A maintainer who consistently attributes correctly and attaches a minimal repro in the right repo is trusted; their issues get picked up fast. The bisection technique here — "reproduce the symptom with the suspect removed" — is the most powerful tool you have, because it converts an opinion ("I think it's core") into a proof ("here's the same bug with no plugin and no Dashboards, on a stock distro").

The decision flowchart

flowchart TD
  S[Symptom reported]
  S --> Q1{Reproducible with raw curl<br/>against core, no Dashboards?}
  Q1 -->|No, only via the UI| DASH[Dashboards owns it<br/>render/agg-config/time-zone]
  Q1 -->|Yes| Q2{Reproducible with the<br/>plugin uninstalled?}
  Q2 -->|Yes, no plugin needed| Q3{Does it involve a query type,<br/>field type, or scoring?}
  Q2 -->|No, needs the plugin| PLUG[A plugin owns it<br/>find which from the field/query]
  Q3 -->|Yes, and result is numerically wrong| Q4{Reproducible on a single shard<br/>with a trivial Lucene-level query?}
  Q3 -->|No — error, allocation, cluster state| CORE[Core owns it]
  Q4 -->|Yes, even a match_all/term is wrong| LUCENE[Lucene owns it<br/>codec/docvalues/scorer]
  Q4 -->|No, only the composite/agg is wrong| CORE
  PLUG --> Q5{Is the plugin's error/message<br/>itself the problem?}
  Q5 -->|Message unclear / wrong| PLUGB[Plugin bug — file on plugin]
  Q5 -->|Message clear, config is wrong| USER[User/config — no bug]

The flowchart encodes one idea: remove suspects one at a time and see if the symptom survives. Each "reproducible with X removed?" answer eliminates a repo.

The attribution table

The bisection technique: reproduce without the suspect

The single most valuable move. For each suspect repo, there's a way to take it out of the picture and see whether the bug survives.

Note: "Remove the suspect" sometimes means re-create the symptom with stock tools. You can't uninstall Lucene, but you can reproduce a scoring bug with a 5-line Lucene IndexSearcher test that uses no OpenSearch at all — if that reproduces it, you've proven Lucene owns it and can file apache/lucene with a Lucene-only repro.

Worked Example 1 — Wrong agg result (core vs Lucene)

Symptom: a sum aggregation over a double field returns a value that's off by a tiny amount on a multi-shard index.

Bisect:

# Reproduce via curl (removes Dashboards)
curl -s 'localhost:9200/sales/_search' -H 'Content-Type: application/json' -d '{
  "size":0, "aggs":{"t":{"sum":{"field":"amount"}}}}'

# Single-shard index (removes the multi-shard reduce)
curl -s -XPUT localhost:9200/sales1 -H 'Content-Type: application/json' \
  -d '{"settings":{"number_of_shards":1}}'
# reindex and re-run the sum

• If single-shard is correct but multi-shard is wrong, the fault is in core's cross-shard reduce (SearchPhaseController / InternalSum.reduce) — file on OpenSearch. Floating-point sum order across shards is a classic core concern; see Aggregations.
• If single-shard is also wrong, the per-shard aggregation read the wrong doc-values — drop to a Lucene SortedNumericDocValues read. If a Lucene-only test reproduces it, file apache/lucene. (Usually it's core's agg, not Lucene; doc-values reads are extremely well-tested.)

Attribution most often: core, the reduce. The decisive evidence is "single-shard correct, multi-shard wrong."

Worked Example 2 — A visualization renders wrong (Dashboards vs core)

Symptom: a date histogram bar chart shows counts in the wrong day buckets.

Bisect: capture the request (P1 Step 3) and replay it.

curl -s 'localhost:9200/sales/_search?typed_keys=true' -H 'Content-Type: application/json' -d '{
  "size":0,
  "aggs":{"h":{"date_histogram":{"field":"ts","calendar_interval":"day","time_zone":"+00:00"}}}}'

• If the buckets in the JSON are correct but the chart is wrong, the bug is in Dashboards' rendering — file OpenSearch-Dashboards.
• If the JSON buckets are wrong, look at the time_zone Dashboards put in the request. Dashboards sends the browser's time zone; if it sends the wrong one, that's still a Dashboards bug (wrong request), even though core honored it correctly. Core only owns it if core mis-bucketed for a correct time_zone — reproduce with an explicit time_zone via curl and check against a manual calculation.

Attribution most often: Dashboards — either rendering or the time_zone it chose. Core's date_histogram honoring an explicit time zone is rarely wrong.

Worked Example 3 — Vector search returns bad neighbors (k-NN vs core vs Lucene)

Symptom: a knn query returns neighbors that are obviously not the closest.

Bisect against a brute-force ground truth that uses no approximate engine:

# Approximate (k-NN ANN engine)
curl -s localhost:9200/vec/_search -H 'Content-Type: application/json' -d '{
  "query":{"knn":{"v":{"vector":[1,2,3],"k":3}}}}'

# Exact, scripted distance (removes the ANN engine; still uses the plugin's script)
curl -s localhost:9200/vec/_search -H 'Content-Type: application/json' -d '{
  "query":{"script_score":{"query":{"match_all":{}},
    "script":{"source":"knn_score","lang":"knn",
      "params":{"field":"v","query_value":[1,2,3],"space_type":"l2"}}}}}'

• If exact returns the right neighbors but approximate does not, the approximation is the issue: the k-NN engine (faiss/nmslib/Lucene HNSW) config — ef_search, m, index params — or a genuine engine bug. File on k-NN (it owns EnginePlugin.getEngineFactory from P2). Often it's a recall-vs-config tradeoff, not a bug.
• If both are wrong, the vectors were stored or read wrong — k-NN's codec, or Lucene's HNSW codec if the engine is lucene. Narrow by switching the mapping's engine and seeing which engine reproduces it.
• If the field type or query is rejected, that's the P3 Shape-B mismatch, owned by the mapping/config.

Attribution: k-NN (engine/config) is most common; Lucene only if the lucene HNSW engine is in use and even exact reads are wrong.

Worked Example 4 — A 403 (security)

Symptom: a request returns 403 security_exception.

Bisect: this needs the security plugin to even occur, so it's a security/policy matter unless core mis-declared the action.

# What privilege did the action require?  (the message names it)
# "no permissions for [indices:data/read/search] and User [name=guest]"
# Check the role mapping grants that action:
curl -sk -u admin:admin 'https://localhost:9200/_plugins/_security/api/roles/<role>' | python3 -m json.tool

• If the role should grant indices:data/read/search but doesn't, it's a config issue — fix the role, no bug.
• If the privilege string in the error doesn't match the action the user actually invoked, core may have registered the wrong ActionType name — a rare core bug; confirm with grep in core's ActionModule.
• If security denies an action it should allow given a correct role, that's a security plugin bug.

Attribution: usually config; occasionally security; rarely core.

Writing the bug report in the right repo

Once attributed, the report goes in the owning repo with a repro that excludes the other repos:

The discipline of excluding the other repos from the repro is what proves your attribution. A core bug report that requires installing k-NN and Dashboards to reproduce hasn't been attributed — it's been punted.

Expected Output

• The decision flowchart applied to one symptom of your choosing, written out.
• For two of the four worked examples, the actual curl bisection run on your local cluster with the result.
• One bug-report skeleton for the repo you attributed to, with a repro that does not drag in the other repos.

Stretch Goals

1. Take a real open issue from OpenSearch-Dashboards and decide, from its description alone, whether it's truly a Dashboards bug or a mis-filed core bug.
2. Build a Lucene-only IndexSearcher repro for a scoring question and confirm it reproduces (or doesn't) outside OpenSearch.
3. Write the "reproduce without the suspect" steps as a checklist you could hand to a triager.

Validation / Self-check

1. State the one-sentence rule for the entire flowchart.
2. For each of the four repos, give the single decisive test that removes it from suspicion.
3. A sum is wrong on a multi-shard index but right on one shard. Owner? Why?
4. A date histogram chart is wrong but the captured request's buckets are correct. Owner? Why?
5. Why does the requirement "the repro must exclude the other repos" actually prove the attribution rather than just being tidy?
6. When is a 403 a core bug, and how would you confirm it?