Step 3: Execution Path Analysis

You have a reproducer. Now you have to understand why it fails, and that means tracing the exact path a request takes from the wire to the line where the wrong thing happens. Not a sketch. Not "it goes through the search service somewhere." The actual chain of methods, with file and (approximate) line citations at every layer, that you could read aloud and a maintainer would nod along to.

This step produces three artifacts:

1. An annotated execution path — a list of the methods the request passes through, top to bottom, each with a path/to/File.java citation.
2. A path diagram (mermaid) showing the same path visually, with the bug site annotated.
3. A confirmed observation point — the exact line where you can set a breakpoint or log statement and watch the wrong value appear.

The rule that governs this step: you must be able to point at one line and say "the value is correct above this line and wrong below it." Until you can, you are still guessing.

The Four Layers

Every OpenSearch request that reaches a bug crosses some subset of these. Know which ones your bug lives in before you start grepping.

Note: The "cluster manager" node (formerly called "master") owns the coordination layer. OpenSearch renamed master → cluster_manager for inclusive language; you will still see Master-prefixed class names in the code (TransportMasterNodeAction is aliased by TransportClusterManagerNodeAction) and old setting aliases like cluster.initial_master_nodes. When you read coordination code, both vocabularies are live.

A search-aggregation bug lives in REST + Action + Shard (search side). A listener-race lives in Coordination. A seqno/version edge lives in Shard/Engine. Pin your layer first; it tells you where to point grep.

Find the Entry Point with grep

Start at the top: which RestHandler services the request in your reproducer? The handler advertises its route in routes(). Grep for the path segment.

cd ~/OpenSearch    # your clone of opensearch-project/OpenSearch

# Which REST handler owns _search?
grep -rn "_search" server/src/main/java/org/opensearch/rest/action/search/
# -> RestSearchAction.java registers GET/POST {index}/_search

# Generic: find the handler class for any action
grep -rln "extends BaseRestHandler" server/src/main/java/org/opensearch/rest/action \
  | xargs grep -l "routes()"

RestSearchAction.prepareRequest(...) is the method that parses the request and hands off. Read it. It ends in a call like client.execute(SearchAction.INSTANCE, searchRequest, ...) — that string, SearchAction.INSTANCE, is the bridge to the transport layer.

Now find the transport action wired to that ActionType:

# The ActionType -> TransportAction wiring happens in ActionModule.
grep -rn "SearchAction.INSTANCE" server/src/main/java/org/opensearch/action/ActionModule.java
grep -rn "class TransportSearchAction" server/src/main/java/org/opensearch/action/search/

Repeat this descent for your specific bug. For a write-path bug:

grep -rn "class RestBulkAction"          server/src/main/java/org/opensearch/rest/action/document/
grep -rn "class TransportShardBulkAction" server/src/main/java/org/opensearch/action/bulk/
grep -rn "applyIndexOperationOnPrimary"   server/src/main/java/org/opensearch/index/shard/IndexShard.java
grep -rn "private IndexResult indexIntoLucene" server/src/main/java/org/opensearch/index/engine/InternalEngine.java

For a coordination bug:

grep -rn "class ClusterApplierService" server/src/main/java/org/opensearch/cluster/service/
grep -rn "callClusterStateListeners\|callClusterStateAppliers" \
  server/src/main/java/org/opensearch/cluster/service/ClusterApplierService.java

Tip: Class names are stable across branches but line numbers are not. Everywhere this curriculum cites a line, treat it as "run the grep on your checkout." Pin your work to a commit (git rev-parse HEAD) and record it, so your line citations are reproducible for the reviewer reading your Step 3 doc.

Use the IDE Call Hierarchy

grep gets you the players; the IDE call hierarchy gets you the order. Import the Gradle project into IntelliJ IDEA (the DEVELOPER_GUIDE.md documents this; ./gradlew idea is the legacy task but modern IntelliJ imports the Gradle build directly).

Once imported, two moves do most of the work:

• Call hierarchy (callers): put the cursor on the buggy method, Ctrl+Alt+H (Cmd+Alt+H on macOS). This shows who calls this — walk up until you reach a TransportAction or RestHandler. That is your path, bottom-up.
• Call hierarchy (callees) / Go to implementation: Ctrl+Alt+B on an interface method (e.g. ClusterStateListener.clusterChanged) shows every implementation. Polymorphism hides the real target; this reveals it. For serialization, StreamInput.readNamedWriteable resolves through NamedWriteableRegistry at runtime — the call hierarchy will not show the concrete Writeable; you find that by grepping the registry wiring instead.

Write the path down as you walk it. The deliverable is a numbered list:

1. RestSearchAction.prepareRequest          rest/action/search/RestSearchAction.java
2. NodeClient.execute                        client/node/NodeClient.java
3. TransportSearchAction.doExecute           action/search/TransportSearchAction.java
4. AbstractSearchAsyncAction.executePhase    action/search/AbstractSearchAsyncAction.java
5. SearchService.executeQueryPhase           search/SearchService.java
6. QueryPhase.execute                        search/query/QueryPhase.java
7. <bug fires here>                          ...

Confirm the Path with a Debugger

A path you reasoned about is a hypothesis. A path you stepped through is a fact. Two ways to attach a debugger to OpenSearch running from source.

Option A — ./gradlew run --debug-jvm

The single-node run task can launch suspended, waiting for a debugger:

./gradlew run --debug-jvm
# Gradle prints: "Listening for transport dt_socket at address: 5005"
# The JVM is SUSPENDED until you attach.

In IntelliJ: Run → Edit Configurations → + → Remote JVM Debug, host localhost, port 5005, then Debug. The node resumes. Set a breakpoint on the line your path analysis identified, then fire the reproducer curl from another shell. Execution stops on your breakpoint.

Option B — remote-debug a running test

If your reproducer is a JUnit test (it should be, from Step 2), debug that instead — it is faster and more isolated than a full node:

./gradlew :server:test \
  --tests "org.opensearch.search.aggregations.bucket.terms.TermsAggregatorTests.testIssueNNNNRepro" \
  --debug-jvm
# Gradle suspends the test JVM on port 5005; attach the same Remote JVM Debug config.

Either way, once stopped on the breakpoint, inspect the frame:

• Evaluate the value the issue is about. For the aggregation bug: result.getBuckets() — is it empty here? Step out one frame at a time (Shift+F8) and re-evaluate. The frame where the value flips from correct to wrong is your defect site.
• Watch the call stack. The "Frames" panel is your execution path, authoritatively. Screenshot it; paste it into your Step 3 doc.
• Conditional breakpoints keep you out of the hundreds of unrelated calls. Right-click the breakpoint → Condition, e.g. minDocCount == 0 or event.source().equals("..."). Only your trigger condition stops execution.

Warning: OpenSearch tests run under the RandomizedRunner with a security manager and randomized seeds. When debugging a test, pin the seed (-Dtests.seed=... from Step 2) so the run you debug is the run that failed.

TRACE Logging Without a Debugger

Sometimes a breakpoint is the wrong tool — the bug is timing-dependent (a race), or it only manifests in a multi-node integration test where stepping freezes the cluster and changes the behavior. Then you log.

OpenSearch logging is per-logger, configured by package, and changeable at runtime through cluster settings — no restart:

# Turn the relevant package up to TRACE on a running cluster.
curl -s -XPUT 'localhost:9200/_cluster/settings' -H 'Content-Type: application/json' -d '
{
  "persistent": {
    "logger.org.opensearch.cluster.service": "TRACE",
    "logger.org.opensearch.cluster.coordination": "DEBUG",
    "logger.org.opensearch.index.engine": "TRACE"
  }
}'

# ... fire the reproducer ...

# Reset when done (null clears the override).
curl -s -XPUT 'localhost:9200/_cluster/settings' -H 'Content-Type: application/json' -d '
{ "persistent": { "logger.org.opensearch.cluster.service": null,
                  "logger.org.opensearch.cluster.coordination": null,
                  "logger.org.opensearch.index.engine": null } }'

For an integration test, set the same loggers with an annotation on the test class so the output lands in the test logs:

@TestLogging(value = "org.opensearch.cluster.service:TRACE", reason = "trace listener ordering")
public class SomeBehaviorIT extends OpenSearchIntegTestCase { ... }

The pattern: turn a layer to TRACE, run the repro, read the log top to bottom, and find the line where the logged state stops matching what you expect. That line's logger tells you the class; grep the message string to find the exact source line:

grep -rn "applying cluster state version" server/src/main/java/org/opensearch/cluster/service/

Now you have an observation point without a debugger, which is the only thing that works for a race.

The Annotated Path Diagram

Pull it together into a diagram. This goes straight into your Step 4 root-cause doc and your eventual PR description. Annotate the node where the bug fires.

flowchart TD
    A["HTTP POST {index}/_search<br/>RestController"] --> B["RestSearchAction.prepareRequest<br/>rest/action/search/RestSearchAction.java"]
    B --> C["NodeClient.execute(SearchAction.INSTANCE, ...)"]
    C --> D["TransportSearchAction.doExecute<br/>action/search/TransportSearchAction.java"]
    D --> E["coordinating node fans out to shards<br/>AbstractSearchAsyncAction"]
    E --> F["per-shard: SearchService.executeQueryPhase<br/>search/SearchService.java"]
    F --> G["QueryPhase.execute<br/>+ AggregatorFactory -> Aggregator"]
    G --> H["coordinating node reduce<br/>SearchPhaseController + InternalAggregation.reduce"]
    H --> X["BUG: InternalAggregation.reduce drops the<br/>min_doc_count=0 'missing' bucket when<br/>no shard produced it"]

    style X fill:#ffd6d6,stroke:#c0392b,stroke-width:2px

For a write-path / engine bug, your diagram looks like:

flowchart TD
    A["POST {index}/_bulk<br/>RestBulkAction"] --> B["TransportBulkAction"]
    B --> C["TransportShardBulkAction<br/>action/bulk/TransportShardBulkAction.java"]
    C --> D["IndexShard.applyIndexOperationOnPrimary<br/>index/shard/IndexShard.java"]
    D --> E["InternalEngine.index<br/>index/engine/InternalEngine.java"]
    E --> F["versionMap / LocalCheckpointTracker<br/>seqno assignment"]
    F --> G["Lucene IndexWriter.addDocument/updateDocument<br/>+ Translog.add"]
    G --> R["TransportReplicationAction replicates to replica"]
    F --> X["BUG: seqno/version edge under<br/>concurrent update at the same _id"]

    style X fill:#ffd6d6,stroke:#c0392b,stroke-width:2px

Build the diagram that matches your bug. The non-negotiable parts: every box cites a real file, and exactly one box is styled as the bug site.

Build the Execution-Path Report

Save capstone-work/execution-path.md. A reviewer should be able to open every file at every line and follow the logic without asking you a question.

# Execution path: #NNNN

## Checkout
- Branch/commit traced on: `main` @ <sha from `git rev-parse HEAD`>

## Layers crossed
REST -> Action -> Shard (search/reduce side)

## Annotated path (top to bottom)
1. RestSearchAction.prepareRequest        rest/action/search/RestSearchAction.java:~110
2. NodeClient.execute                      client/node/NodeClient.java:~80
3. TransportSearchAction.doExecute         action/search/TransportSearchAction.java:~280
4. SearchService.executeQueryPhase         search/SearchService.java:~520
5. SearchPhaseController reduce            action/search/SearchPhaseController.java:~430
6. InternalAggregation.reduce  <- BUG      search/aggregations/InternalAggregation.java:~NN

## Observation point
- File/line: search/aggregations/.../InternalTerms.java reduce(...)
- Method to break on: doReduce(...)
- What is correct above this line: per-shard buckets include the 'missing' key
- What is wrong below this line: the merged bucket list omits it when min_doc_count=0

## How confirmed
- [x] Breakpoint hit on the cited line (call-stack screenshot attached)
- [x] TRACE on org.opensearch.search.aggregations showed the bucket present per-shard

Deliverable for Step 3

• A numbered, file-cited execution path from REST handler to bug site.
• A mermaid (or text-arrow) diagram with exactly one annotated bug node.
• A confirmed observation point: the one line where the value flips from correct to wrong, verified by a breakpoint or by TRACE logging.
• The traced commit SHA recorded so line citations are reproducible.
• capstone-work/execution-path.md complete.

Validation / Self-check

Before advancing to Step 4:

1. You can name every layer the request crosses (REST / Action / Shard / Coord) and justify which ones it does not cross.
2. Every method in your path has a real org.opensearch.* file citation; none are hand-waved as "then it goes through the engine somewhere."
3. You confirmed the path with a debugger (--debug-jvm) or TRACE logging, not by reading alone.
4. You can point at exactly one line and say "correct above, wrong below."
5. Your line numbers are pinned to a recorded commit SHA.
6. The diagram has exactly one styled bug node and would survive a maintainer opening each cited file.
7. You did not change any production code in this step — observation only.

Then go to Step 4: Root Cause Identification.