Lab P1: From a Dashboards Query to an OpenSearch Search

Background

This is the canonical cross-repo trace, the OpenSearch analog of "SQL to DAG." A user drags fields onto a Dashboards visualization. Dashboards (TypeScript) turns that into an aggregation _search (or _msearch) request, ships it over HTTP to the OpenSearch core engine (Java), where RestSearchAction parses it and TransportSearchAction fans it out to shards. The result comes back up the same path and Dashboards renders it.

You cannot debug a "the dashboard shows the wrong number" report without being able to walk this path in both directions and say, at each hop, who owns what. This lab makes you trace it concretely, capture the real request on the wire, and draw the boundary between Dashboards' job and core's job.

It builds on the Search Execution deep dive (what happens inside core once the request arrives), the REST Layer, and the Action Framework. It is the foundation for P4: Bug Attribution, where "is this Dashboards or core?" becomes a decision you make under time pressure.

Why This Lab Matters for Contributors

Most Dashboards bug reports arrive as screenshots ("this panel is wrong"). The maintainer's first job is to extract the actual request and response and decide whether the fault was in how Dashboards built the DSL or in how core answered it. If you can capture the request, replay it with curl, and confirm core returns the same "wrong" answer, you have proven the bug is in core (or Lucene) and can file a minimal repro there with zero Dashboards involvement. If curl returns the right answer for the request Dashboards sent, the bug is in Dashboards' rendering, not core. That bisection is the whole skill.

Prerequisites

Index a tiny dataset so a visualization has something to aggregate:

curl -s -XPUT localhost:9200/sales -H 'Content-Type: application/json' -d '{
  "mappings": { "properties": {
    "ts":     { "type": "date" },
    "region": { "type": "keyword" },
    "amount": { "type": "double" }
  }}}'

for r in us eu apac; do for i in 1 2 3 4 5; do
  curl -s -XPOST localhost:9200/sales/_doc -H 'Content-Type: application/json' \
    -d "{\"ts\":\"2026-06-1${i}\",\"region\":\"$r\",\"amount\":$((RANDOM % 100))}" >/dev/null
done; done
curl -s -XPOST localhost:9200/sales/_refresh >/dev/null

The two-sided picture

flowchart TD
  subgraph Browser["Browser (Dashboards client, TypeScript)"]
    V[Visualization editor] --> SS[SearchSource / agg config builder]
    SS --> DSL[Builds the _search DSL JSON]
    DSL --> HTTP1[POST /api/console or data plugin search route]
  end
  subgraph DashServer["Dashboards server (Node, TypeScript)"]
    HTTP1 --> PROXY[search strategy / OpenSearchClient]
    PROXY --> JS[opensearch-js client]
  end
  subgraph Core["OpenSearch core (Java)"]
    JS -->|HTTP POST /sales/_search| RC[RestController]
    RC --> RSA[RestSearchAction.parse]
    RSA --> NC[NodeClient.execute SearchAction]
    NC --> TSA[TransportSearchAction]
    TSA --> FAN[fan-out to shards: QueryPhase then FetchPhase]
    FAN --> RED[SearchPhaseController reduce]
  end
  RED -->|JSON response| JS
  JS --> PROXY
  PROXY --> HTTP1
  HTTP1 --> RENDER[Dashboards renders chart from agg buckets]

The dashed vertical line you must keep in your head: everything left of RestController is Dashboards' job; everything from RestController rightward is core's job. Lucene sits one level below FetchPhase/QueryPhase and owns the actual inverted-index and doc-values reads.

Step-by-Step Tasks

Step 1 — Build the request the way a visualization would

A Dashboards bar chart "sum of amount by region" is, on the wire, a terms aggregation with a sum sub-aggregation and size: 0 (Dashboards asks for no hits, only buckets). This is the JSON the TypeScript layer ultimately emits:

curl -s 'localhost:9200/sales/_search' -H 'Content-Type: application/json' -d '{
  "size": 0,
  "aggs": {
    "by_region": {
      "terms": { "field": "region", "size": 10 },
      "aggs": { "total": { "sum": { "field": "amount" } } }
    }
  }
}' | python3 -m json.tool

You should get one bucket per region with a total.value. That bucket array is exactly what Dashboards turns into bars. The request is the contract. If this curl is correct but the chart is wrong, the bug is in Dashboards' rendering.

Step 2 — Find where Dashboards constructs the DSL (TS side, high level)

You do not need to be a TypeScript expert; you need to know where the DSL is born so you can reason about Dashboards-side bugs. In the Dashboards repo, the search-source and aggregation-builder machinery lives under the data plugin:

# In an OpenSearch-Dashboards checkout
grep -rln "SearchSource\|buildOpenSearchQuery\|toDsl\|AggConfigs" \
  src/plugins/data/common src/plugins/data/public | head

# The agg builder that turns an AggConfig into request JSON
find src/plugins/data -name "agg_configs.ts" -o -name "*search_source*.ts" | head

Conceptually the chain is: the visualization's AggConfigs → agg_configs.toDsl() builds the aggs block → SearchSource assembles the full body (query filters from the search bar, time range, size: 0) → the request is handed to the data plugin's search strategy → the server-side OpenSearchClient (backed by opensearch-js) sends it to core. The Dashboards server never invents query semantics; it forwards. That matters for attribution: a malformed aggregation almost always traces to the client-side agg builder, not the server proxy.

Step 3 — Capture the actual request on the wire

Do not trust your reconstruction — capture the real bytes. Three ways, in order of preference:

(a) Browser devtools. Open the visualization, open devtools → Network, filter for _search or the data plugin's /internal/search route, and copy the request payload. This is the ground truth of what Dashboards built.

(b) Core slow log. Make core log every search by dropping the slow-log threshold to zero on the index, then read the request from the log:

curl -s -XPUT 'localhost:9200/sales/_settings' -H 'Content-Type: application/json' -d '{
  "index.search.slowlog.threshold.query.trace": "0ms",
  "index.search.slowlog.threshold.fetch.trace": "0ms"
}'
# now run the visualization; then tail the slow log
grep -A2 "took_millis" logs/*_index_search_slowlog.log 2>/dev/null | tail -40

The slow log records the shard-level source — the exact aggregation core received.

(c) profile: true. Add "profile": true to the body and core returns a profile section breaking down query/agg/fetch time per shard — invaluable when the question becomes "is it slow, and where?"

curl -s 'localhost:9200/sales/_search?typed_keys=true' -H 'Content-Type: application/json' -d '{
  "size": 0, "profile": true,
  "aggs": { "by_region": { "terms": { "field": "region" } } }
}' | python3 -c "import sys,json;d=json.load(sys.stdin);print(json.dumps(d['profile']['shards'][0]['aggregations'],indent=2)[:800])"

Note: typed_keys=true is what Dashboards sends so it can tell a terms agg from a date_histogram in the response by the key prefix (sterms#by_region). If you replay a captured request and the response shape differs from what Dashboards expects, check whether typed_keys was set — a classic Dashboards/core mismatch.

Step 4 — Watch the request land in core

Replay the captured request and confirm where it enters the Java side. The entry point is RestSearchAction:

grep -n "class RestSearchAction\|prepareRequest\|parseSearchRequest\|routes()" \
  server/src/main/java/org/opensearch/rest/action/search/RestSearchAction.java
grep -n "class TransportSearchAction\|protected void doExecute" \
  server/src/main/java/org/opensearch/action/search/TransportSearchAction.java

If you launched core with ./gradlew run, attach a debugger (the task prints the JDWP port) and set a breakpoint in RestSearchAction.prepareRequest. Run the visualization; the breakpoint fires with the parsed SearchRequest. You are now standing exactly on the Dashboards↔core boundary, looking at the request Dashboards built, parsed into core's object model. From here, the Search Execution deep dive takes over: TransportSearchAction fans out, QueryPhase/FetchPhase run per shard, SearchPhaseController reduces, and the JSON goes back.

Step 5 — Walk the response back up and find the rendering boundary

The reduced response carries the aggregations.by_region.buckets array. On the way back:

The rendering boundary is the last hop. If the buckets in the captured JSON are correct but the bars are wrong (wrong order, wrong label, off-by-one time-zone bucketing), the bug is in Dashboards' renderer or the agg-config that chose the wrong interval/time zone — not core.

The boundary: Dashboards' job vs core's job

The one-line test you will use constantly: replay the captured request with curl. Right answer from curl + wrong chart ⇒ Dashboards. Wrong answer from curl ⇒ core (or Lucene) — and now you have a Dashboards-free repro.

Expected Output

• A curl that reproduces the visualization's aggregation and returns one bucket per region with a total.value.
• The captured real request from devtools or the slow log, matching your hand-built one.
• A breakpoint hit in RestSearchAction.prepareRequest (or a slow-log line) proving the request entered core as you expected.
• A one-sentence statement, for your dataset, of what Dashboards owned vs what core owned in producing the chart.

Stretch Goals

1. Trigger an _msearch: a Dashboards dashboard with several panels batches requests. Capture it and find RestMultiSearchAction / TransportMultiSearchAction in core.
2. Introduce a deliberate Dashboards-vs-core mismatch: send the request without typed_keys and observe how a generic client must disambiguate agg types from the response.
3. Set track_total_hits: false and watch the hits.total.relation change to gte; explain why Dashboards usually wants an exact total for the hit count display.

Validation / Self-check

1. Draw the full path from a visualization to a shard and back, naming the class/route at each core-side hop.
2. Where exactly does Dashboards stop building the request and core start parsing it? Name the first core class.
3. Given "the bar chart shows 0 for eu but data exists," describe the two-curl bisection that decides Dashboards vs core in under five minutes.
4. What is typed_keys for, and what breaks if a client omits it?
5. Name three independent ways to capture the actual request Dashboards sent, and say which one you'd reach for first when you have no browser access.
6. Why does the Dashboards server (the Node proxy) almost never own a query-semantics bug, while the Dashboards client (the agg builder) often does?