Lab 3.1: Trace a REST Request from HTTP to TransportAction

This is a code-reading lab. You will not write a line of production code. You will follow two real requests — GET /_cluster/health and POST /<index>/_doc — from the HTTP socket all the way to the TransportAction that does the work, using only grep, find, and your IDE. The deliverable is a reading-log artifact you will reuse for the rest of the curriculum.

This is the OpenSearch analog of the Tez submitDAG → AsyncDispatcher trace. The skill is the same: navigate, don't memorize.

Background

Every user-facing OpenSearch operation follows the same skeleton (see Level 3 overview):

HTTP :9200 → HttpServerTransport → RestController.dispatchRequest
          → RestHandler (BaseRestHandler) → builds an ActionRequest
          → NodeClient.execute(ActionType, request, listener)
          → TransportAction (HandledTransportAction) → ActionFilters → doExecute
          → (over transport :9300) per-shard work → ActionListener → JSON

The two requests in this lab exercise the two most important variants of that skeleton:

• GET /_cluster/health — a cluster-manager-routed read. The work must run on (or be answered from the cluster state owned by) the elected cluster manager (formerly master).
• POST /<index>/_doc — a write. It enters as a single-doc index, is wrapped into a bulk, routed to the primary shard, and replicated.

Deep-dive companions: rest-layer.md, action-framework.md, transport-layer.md.

Why This Lab Matters for Contributors

When a bug report says "cluster health returns the wrong status" or "indexing a doc throws an NPE," the first thing a maintainer does is locate the exact class on the path. If you can't get from a URL to the responsible TransportAction in two minutes, you can't triage. Every issue lab in Level 8 and the capstone starts with this skill.

Prerequisites

• OpenSearch cloned and building (./gradlew assemble succeeds — see Level 1).
• A running node for the live half: ./gradlew run (REST on localhost:9200).
• An IDE with "Find Usages" / "Call Hierarchy" (IntelliJ Ctrl-Alt-H) — optional but faster.
• A scratch file for your reading log:

mkdir -p ~/opensearch-notes
: > ~/opensearch-notes/reading-log-3.1.md

Note: Class and method names are stable across recent branches, but line numbers are not. This lab gives you grep/find commands to locate each hop on your checkout rather than fabricated line numbers. If a name has drifted on your branch, the grep still points you at the right neighborhood.

Part A — GET /_cluster/health (budget: 45 min)

Step 1 (5 min) — Confirm the request live

curl -s 'localhost:9200/_cluster/health?pretty'

Expected (abridged):

{
  "cluster_name" : "opensearch",
  "status" : "green",
  "number_of_nodes" : 1,
  "active_primary_shards" : 0,
  "active_shards" : 0,
  "unassigned_shards" : 0
}

Now find the code that produced it.

Step 2 (8 min) — Find the RestHandler and its route

The handler name follows OpenSearch's convention: Rest<Thing>Action.

find server/src/main/java -name "RestClusterHealthAction.java"
grep -n "routes()\|new Route\|GET\|prepareRequest" \
  server/src/main/java/org/opensearch/rest/action/admin/cluster/RestClusterHealthAction.java

In routes() you will see the (GET, "/_cluster/health") and (GET, "/_cluster/health/{index}") registrations. Read prepareRequest(RestRequest, NodeClient). Note the two things every BaseRestHandler does:

1. Build a typed request — here a ClusterHealthRequest from path/query params (index, wait_for_status, timeout, …).
2. Return a RestChannelConsumer that calls client.admin().cluster().health(request, listener) (a thin wrapper over NodeClient.execute).

Log it:

cat >> ~/opensearch-notes/reading-log-3.1.md <<'EOF'
## GET /_cluster/health
1. RestController routes (GET,/_cluster/health) -> RestClusterHealthAction.prepareRequest
   - builds ClusterHealthRequest from RestRequest params
   - returns channel -> client.admin().cluster().health(req, listener)
EOF

Step 3 (7 min) — How RestController matched the route

The handler does not register itself; RestController does. Find the registration and the dispatch:

grep -n "registerHandler\|registerHandlerNoWrap\|dispatchRequest\|tryAllHandlers" \
  server/src/main/java/org/opensearch/rest/RestController.java | head

Read dispatchRequest(...) / tryAllHandlers(...): RestController walks its path trie, finds the handler whose routes() matched (method, path), applies any RestHandlerWrapper, and calls handler.handleRequest(request, channel, client). BaseRestHandler.handleRequest then calls your prepareRequest and consumes the returned channel consumer.

Who registered RestClusterHealthAction in the first place? ActionModule:

grep -n "RestClusterHealthAction" server/src/main/java/org/opensearch/action/ActionModule.java

You'll find it in initRestHandlers(...) via registerHandler.accept(new RestClusterHealthAction()).

Step 4 (10 min) — From ActionType to TransportAction

client.admin().cluster().health(...) ultimately calls NodeClient.execute(ClusterHealthAction.INSTANCE, request, listener). Find the ActionType:

find server/src/main/java -name "ClusterHealthAction.java"
grep -n "extends ActionType\|public static final\|INSTANCE\|NAME" \
  server/src/main/java/org/opensearch/action/admin/cluster/health/ClusterHealthAction.java

ClusterHealthAction is an ActionType<ClusterHealthResponse> with a NAME of "cluster:monitor/health" and a singleton INSTANCE. The NAME is the transport action name — the string registered on TransportService so other nodes can invoke it.

Now find where the ActionType is bound to its TransportAction:

grep -n "ClusterHealthAction.INSTANCE\|TransportClusterHealthAction" \
  server/src/main/java/org/opensearch/action/ActionModule.java

You'll see actions.register(ClusterHealthAction.INSTANCE, TransportClusterHealthAction.class) (the exact helper is an ActionRegistry/registerAction call). This is the pivot: the NodeClient holds a Map<ActionType, TransportAction> built from these registrations, and execute is just a map lookup followed by transportAction.execute(...).

grep -n "execute\|Map<.*ActionType\|actions.get" \
  server/src/main/java/org/opensearch/client/node/NodeClient.java

Log it:

cat >> ~/opensearch-notes/reading-log-3.1.md <<'EOF'
2. NodeClient.execute(ClusterHealthAction.INSTANCE, req, listener)
   - ActionType NAME = "cluster:monitor/health"
   - ActionModule binds INSTANCE -> TransportClusterHealthAction
   - NodeClient = map lookup ActionType -> TransportAction, then .execute(...)
EOF

Step 5 (10 min) — Inside the TransportAction

find server/src/main/java -name "TransportClusterHealthAction.java"
grep -n "extends Transport\|clusterManagerOperation\|masterOperation\|doExecute\|ClusterStateObserver" \
  server/src/main/java/org/opensearch/action/admin/cluster/health/TransportClusterHealthAction.java

Observe:

• It extends TransportClusterManagerNodeAction (formerly TransportMasterNodeAction) — the base for actions that must run on the elected cluster manager. If the local node isn't the manager, the base class transparently re-sends the request over transport to the manager node.
• The real work is in clusterManagerOperation(...) (older branches: masterOperation(...)), which reads/derives the answer from the current ClusterState and may register a ClusterStateObserver to wait for wait_for_status=green etc.
• The result is a ClusterHealthResponse that implements ToXContent; the REST channel serializes it back to the JSON you saw in Step 1.

This routing-to-the-manager behavior is your first contact with the coordination layer of Level 4. Note the pattern; you'll meet TransportClusterManagerNodeAction again.

Log it:

cat >> ~/opensearch-notes/reading-log-3.1.md <<'EOF'
3. TransportClusterHealthAction extends TransportClusterManagerNodeAction
   - re-routes to elected cluster manager if not local
   - clusterManagerOperation(): derives status from ClusterState (+ ClusterStateObserver waits)
   - ClusterHealthResponse (ToXContent) -> JSON on the REST channel
EOF

Step 6 (5 min) — Prove it with a breakpoint (optional but recommended)

In ./gradlew run, attach your debugger (default debug port 5005) and set breakpoints in RestClusterHealthAction.prepareRequest and TransportClusterHealthAction.clusterManagerOperation. Re-run the curl. Watch the call stack at the transport action — it shows you the NodeClient.execute → TransportAction.execute → ActionFilters frames you just read.

Part B — POST /<index>/_doc (budget: 45 min)

The write path is more interesting: a single-doc index is not its own action — it is funneled through bulk.

Step 1 (5 min) — Run it

curl -s -XPOST 'localhost:9200/blog/_doc?refresh=true' \
  -H 'Content-Type: application/json' \
  -d '{"title":"hello","views":1}' | python3 -m json.tool

Expected (abridged):

{
  "_index": "blog",
  "_id": "Xa3...",
  "_version": 1,
  "result": "created",
  "_shards": { "total": 2, "successful": 1, "failed": 0 }
}

Step 2 (8 min) — The RestHandler

find server/src/main/java -name "RestIndexAction.java"
grep -n "routes()\|new Route\|POST\|PUT\|prepareRequest\|IndexRequest\|bulk" \
  server/src/main/java/org/opensearch/rest/action/document/RestIndexAction.java

RestIndexAction.routes() registers (POST, "/{index}/_doc"), (PUT, "/{index}/_doc/{id}"), and the legacy _create variants. In prepareRequest it builds an IndexRequest (index name, optional id, routing, the source bytes, op type, refresh policy).

But note where it sends the request: it calls client.index(indexRequest, listener). Follow that.

Step 3 (7 min) — IndexRequest is dispatched via the bulk action

grep -n "IndexAction\|BulkAction\|bulk(" \
  server/src/main/java/org/opensearch/action/index/TransportIndexAction.java 2>/dev/null
grep -rn "class TransportIndexAction\|prepareBulkRequest\|singleItemBulkRequest" \
  server/src/main/java/org/opensearch/action/ | head

In current OpenSearch the single-document index path is implemented on top of bulk: a single IndexRequest is wrapped into a one-item BulkRequest and dispatched through BulkAction / TransportBulkAction. (Historically there was a standalone TransportIndexAction; the modern path goes through bulk. Verify which your branch uses with the grep above — if TransportIndexAction delegates to bulk, you're on the modern path.)

Find the bulk action registration and class:

grep -n "BulkAction.INSTANCE\|TransportBulkAction" \
  server/src/main/java/org/opensearch/action/ActionModule.java
find server/src/main/java -name "TransportBulkAction.java"

Step 4 (12 min) — TransportBulkAction → TransportShardBulkAction

grep -n "doExecute\|doRun\|executeBulk\|groupRequestsByShards\|createIndex\|shardBulkAction\|TransportShardBulkAction" \
  server/src/main/java/org/opensearch/action/bulk/TransportBulkAction.java | head -30

TransportBulkAction does the coordinating-node work:

1. Auto-creates the index if it doesn't exist (a cluster state update — note the link to Level 4).
2. Resolves each item's shard by routing (OperationRouting / _routing hash).
3. Groups items by target shard.
4. For each shard, sends a BulkShardRequest to the node holding that shard's primary, via TransportShardBulkAction.

Now the per-shard write:

find server/src/main/java -name "TransportShardBulkAction.java"
grep -n "extends TransportReplicationAction\|shardOperationOnPrimary\|shardOperationOnReplica\|applyIndexOperationOnPrimary\|executeBulkItemRequest" \
  server/src/main/java/org/opensearch/action/bulk/TransportShardBulkAction.java | head

TransportShardBulkAction extends TransportWriteAction extends TransportReplicationAction. This is the primary/replica write framework:

• shardOperationOnPrimary(...) runs on the node with the primary shard. It calls into IndexShard.applyIndexOperationOnPrimary(...) → InternalEngine.index(...) → Lucene IndexWriter + Translog.
• The framework then replicates the same operation to each replica via shardOperationOnReplica(...) over transport.

You have now reached the storage engine boundary. The engine internals (IndexShard, InternalEngine, translog, refresh) are Level 6 and the indexing-path deep dive. Stop here; you've traced REST → action → shard.

Log it:

cat >> ~/opensearch-notes/reading-log-3.1.md <<'EOF'
## POST /<index>/_doc
1. RestController routes (POST,/{index}/_doc) -> RestIndexAction.prepareRequest
   - builds IndexRequest -> client.index(req, listener)
2. single IndexRequest funneled through BulkAction/TransportBulkAction
   - coordinating node: auto-create index, resolve shard by routing, group by shard
3. per-shard: TransportShardBulkAction (extends TransportReplicationAction)
   - shardOperationOnPrimary -> IndexShard.applyIndexOperationOnPrimary -> InternalEngine.index
   - replicate via shardOperationOnReplica over transport :9300
   - (engine internals = Level 6)
EOF

How ActionType, ActionFilters, and the actions map fit together

Pull the three concepts together with one more pass:

Trace the filter chain yourself:

grep -n "ActionFilters\|filters\|apply\|proceed\|filterChain" \
  server/src/main/java/org/opensearch/action/support/TransportAction.java | head

Read TransportAction.execute(...): it constructs a RequestFilterChain and calls chain.proceed, which walks each ActionFilter and finally calls doExecute. This is where the security plugin enforces authz before any action runs — a fact you'll need in cross-repo debugging (plugin labs).

Expected Output / Deliverable

Your ~/opensearch-notes/reading-log-3.1.md must contain both traces with a file path for every hop. A complete artifact looks like:

GET /_cluster/health
  RestController.dispatchRequest -> RestClusterHealthAction (rest/action/admin/cluster/)
  -> NodeClient.execute(ClusterHealthAction.INSTANCE) [name=cluster:monitor/health]
  -> TransportClusterHealthAction (extends TransportClusterManagerNodeAction)
  -> clusterManagerOperation reads ClusterState -> ClusterHealthResponse (ToXContent)

POST /<index>/_doc
  RestController.dispatchRequest -> RestIndexAction (rest/action/document/)
  -> client.index(IndexRequest) funneled through BulkAction/TransportBulkAction
  -> group by shard -> TransportShardBulkAction (extends TransportReplicationAction)
  -> shardOperationOnPrimary -> IndexShard.applyIndexOperationOnPrimary -> InternalEngine.index

Stretch Goals

1. Trace GET /<index>/_search. Find RestSearchAction → SearchAction.INSTANCE → TransportSearchAction. Note it does not route to the cluster manager — it fans out to shards. How does its base class differ from TransportClusterManagerNodeAction? (Preview of Level 7.)
2. Find the transport action name for bulk-at-shard. Grep TransportShardBulkAction for its ACTION_NAME / transportPrimaryAction; you'll see indices:data/write/bulk[s]. Note the [s], [p], [r] suffixes the replication framework appends for shard/primary/replica sub-actions.
3. Find a _cat handler. Trace RestNodesAction (rest/action/cat/). _cat handlers are the easiest first PRs — see Level 2.

Validation / Self-check

Answer without looking back at this lab:

1. Which class matched (GET, /_cluster/health) to a handler, and which method did the matching?
2. What is the transport action name (NAME) of ClusterHealthAction, and where is it used besides the NodeClient map?
3. TransportClusterHealthAction extends TransportClusterManagerNodeAction. What does that base class do when the local node is not the elected cluster manager?
4. A single POST /<index>/_doc does not have its own dedicated transport action that talks to the shard. Through which action is it funneled, and why is that a sensible design?
5. Name the method on TransportShardBulkAction (via its base) that runs on the primary shard, and the one that runs on each replica.
6. Where in the path could a plugin (e.g. security) reject the request before doExecute runs? Name the class and the mechanism.
7. From IndexShard.applyIndexOperationOnPrimary, what is the next component that touches Lucene? (You don't need to read it yet — just name it.)

When you can answer all seven and your reading log has a file path for every hop, you've completed Lab 3.1. Continue to Lab 3.2: Node Roles, Discovery, and the Transport Layer.