Lab 6.1: Trace a Document Index Request to Lucene

Background

You cannot fix the write path until you can see it. In this lab you follow a single POST /<index>/_doc request — and then a _bulk request — from the HTTP socket all the way down to Lucene's IndexWriter.addDocument(...) and the translog's Translog.add(...), naming every class on the way. This is a code-reading trace, the most important skill in the entire curriculum: a senior contributor can open an unfamiliar request path and narrate it hop by hop using nothing but grep and the IDE's "go to definition."

The output of this lab is a reading-log artifact — a written, hop-by-hop map of the path with the exact file, class, and method at each step, plus the two spots where versioning and the sequence number are assigned. You will reuse this map for the rest of Level 6 and whenever you touch indexing.

Note: This lab is timed (see Step 0). The point is not to read everything — it is to learn to follow a path efficiently and stop at the right boundaries. Pair it with the engine internals and translog deep dives, and the Level 6 overview diagram.

Why This Lab Matters for Contributors

Every indexing bug, performance issue, or feature request lands somewhere on this path. Knowing the path turns "I have no idea where to start" into "this is a TransportShardBulkAction problem."
Maintainers describe bugs by class (IndexShard.applyIndexOperationOnPrimary). You must be able to open that method and orient yourself in seconds.
The trace reveals where cross-cutting concerns live: routing, mapping, versioning, seqno assignment, replication, and the translog. Each is a future contribution area.
A reading log is reusable: you will paste hops from it into PR descriptions and issue comments to show you understand the code you're changing.

Prerequisites

A running node from source so you can correlate code with live behavior:
```
./gradlew run            # single-node cluster, REST on :9200
```
The repo open in an IDE with working "go to definition" on org.opensearch.*.
Familiarity with the REST layer deep dive and the action framework deep dive — those cover the upper hops in depth; here we move quickly through them to reach the engine.

Step-by-Step Tasks

Step 0: Set a timer and prepare the reading log

Give yourself 75 minutes. Create a scratch file and fill one row per hop as you go. The format:

HOP | FILE | CLASS#method | WHAT HAPPENS HERE | NOTE/QUESTION
----+------+--------------+-------------------+--------------

Resist the urge to read method bodies in full. For each hop, answer only: what does this hand off, and to whom? You are drawing a map, not auditing the code.

Step 1: Send the requests and watch them land

In one terminal, tail the node; in another, send the requests:

# Make the path observable: TRACE the bulk/shard action + engine for one request.
curl -s -XPUT 'localhost:9200/_cluster/settings' -H 'Content-Type: application/json' -d '{
  "transient": {
    "logger.org.opensearch.action.bulk": "TRACE",
    "logger.org.opensearch.index.shard": "TRACE",
    "logger.org.opensearch.index.engine": "TRACE",
    "logger.org.opensearch.index.translog": "TRACE"
  }
}'

# Single document
curl -s -XPOST 'localhost:9200/trace/_doc?pretty' -H 'Content-Type: application/json' \
  -d '{"title":"trace me","n":1}'

# Bulk
curl -s -XPOST 'localhost:9200/_bulk?pretty' -H 'Content-Type: application/x-ndjson' --data-binary $'
{"index":{"_index":"trace","_id":"b1"}}
{"title":"bulk one","n":2}
{"index":{"_index":"trace","_id":"b2"}}
{"title":"bulk two","n":3}
'

# Turn TRACE back off when done.
curl -s -XPUT 'localhost:9200/_cluster/settings' -H 'Content-Type: application/json' -d '{
  "transient": { "logger.org.opensearch.action.bulk": null, "logger.org.opensearch.index.shard": null,
                 "logger.org.opensearch.index.engine": null, "logger.org.opensearch.index.translog": null }
}'

The TRACE lines in the node's stdout name the classes you are about to read — use them to confirm your map matches reality.

Step 2: Hop 1 — REST handler

Both _doc and _bulk enter through a RestHandler registered in ActionModule. Find them:

grep -rn "class RestIndexAction"  server/src/main/java/org/opensearch/rest/action/document/
grep -rn "class RestBulkAction"   server/src/main/java/org/opensearch/rest/action/document/
# How each routes the parsed request into the transport layer:
grep -n "client.execute\|client.bulk\|prepareRequest\|BulkRequest\|IndexRequest" \
  server/src/main/java/org/opensearch/rest/action/document/RestBulkAction.java

Record: RestIndexAction/RestBulkAction parse the HTTP body into an IndexRequest/BulkRequest and call NodeClient.execute(BulkAction.INSTANCE, request, listener). A single _doc is internally wrapped into a one-item bulk — both paths converge on TransportBulkAction. Note that convergence; it's the key insight that collapses two paths into one.

Step 3: Hop 2 — coordinating action (`TransportBulkAction`)

find server/src/main/java -name "TransportBulkAction.java"
grep -n "doExecute\|doInternalExecute\|executeBulk\|createIndex\|resolveRouting\|BulkShardRequest\|groupRequestsByShards\|ingest\|pipeline" \
  server/src/main/java/org/opensearch/action/bulk/TransportBulkAction.java

Record what happens on the coordinating node:

auto-create the index if missing (a cluster-state update),
run ingest pipelines if any,
resolve routing → which shard each op targets,
group operations by shard into BulkShardRequests,
dispatch each BulkShardRequest to the shard's primary via TransportShardBulkAction.

This is the "fan-out by shard" hop. A single _doc produces exactly one BulkShardRequest.

Step 4: Hop 3 — the primary write (`TransportShardBulkAction`)

find server/src/main/java -name "TransportShardBulkAction.java"
grep -n "extends TransportWriteAction\|extends TransportReplicationAction\|shardOperationOnPrimary\|executeBulkItemRequest\|applyIndexOperationOnPrimary\|dispatchedShardOperationOnPrimary" \
  server/src/main/java/org/opensearch/action/bulk/TransportShardBulkAction.java

TransportShardBulkAction extends TransportWriteAction extends TransportReplicationAction — that inheritance is the primary→replica machinery. On the primary it calls (per item) IndexShard.applyIndexOperationOnPrimary(...). Record the method name that bridges into the shard.

Note: TransportReplicationAction is the reusable base for all write actions (index, delete, bulk). It handles routing to the primary, executing on the primary, then replicating to replicas and waiting for the required number of copies. See the replication deep dive.

Step 5: Hop 4 — the shard (`IndexShard.applyIndexOperationOnPrimary`)

grep -n "applyIndexOperationOnPrimary\|applyIndexOperationOnReplica\|applyIndexOperation\|prepareIndex\|docMapper\|MapperService\|markSeqNoAsNoop\|Engine.Index\|getEngine().index" \
  server/src/main/java/org/opensearch/index/shard/IndexShard.java

This is the hinge of the whole path. Record, in order:

The source is parsed and mapped here: prepareIndex(...) uses the DocumentMapper to build a ParsedDocument (Lucene fields). (Trace into MapperService/DocumentMapper only briefly — that's Lab 6.3's territory.)
An Engine.Index operation is constructed wrapping the parsed doc, the version, the version type, and (on the primary) the assignment of the next _seq_no/primary term.
It calls getEngine().index(engineIndex) → InternalEngine.index(...).

Note the two assignment sites you must pin down precisely:

# Where versioning + seqno are resolved on the primary vs replayed on the replica.
grep -n "primaryTerm\|seqNo\|SequenceNumbers\|UNASSIGNED_SEQ_NO\|versionType\|resolveDocVersion\|getOperationPrimaryTerm" \
  server/src/main/java/org/opensearch/index/shard/IndexShard.java | head

Step 6: Hop 5 — the engine (`InternalEngine.index`)

This is the bottom of the path that you own as a reader. Open it:

grep -n "public IndexResult index(Index index)\|private IndexResult indexIntoLucene\|planIndexingAsPrimary\|planIndexingAsNonPrimary\|generateSeqNoForOperationOnPrimary\|addDocs\|updateDocs\|addDocuments\|translog\.\|getTranslog().add\|markSeqNoAsCompleted\|versionMap" \
  server/src/main/java/org/opensearch/index/engine/InternalEngine.java | head -40

Record the engine's decision flow:

index(Index) builds an indexing plan (IndexingStrategy): is this a new doc (addDocument) or an update (updateDocument, which deletes-then-adds)? Does version conflict? Is this primary or replica?
On the primary, generateSeqNoForOperationOnPrimary(...) assigns the _seq_no (via the LocalCheckpointTracker). On a replica, the _seq_no arrives with the operation and is not regenerated — confirm this in the non-primary plan.
indexIntoLucene(...) calls Lucene IndexWriter.addDocument(...) (new) or updateDocument(...) (update — a delete + add keyed on _id).
Translog.add(new Translog.Index(...)) appends the op to the write-ahead log.
The local checkpoint is advanced (markSeqNoAsCompleted-style call) once the op is durable enough.

Step 7: Hop 6 — Lucene and the translog (the two destinations)

The path ends in two places, written for two different reasons:

# Lucene write (visibility-on-refresh, durability-on-commit):
grep -n "indexWriter.addDocument\|indexWriter.updateDocument\|indexWriter.softUpdateDocument\|addDocuments\|updateDocuments" \
  server/src/main/java/org/opensearch/index/engine/InternalEngine.java | head

# Translog write (durability-now, replay-on-recovery):
find server/src/main/java -name "Translog.java"
grep -n "public Location add(Operation operation)\|class Index\|void sync\|ensureSynced" \
  server/src/main/java/org/opensearch/index/translog/Translog.java | head

Record the why for each destination:

Destination	Why	Made durable by	Made visible by
`IndexWriter` (Lucene)	The searchable index	`flush` → `IndexWriter.commit()` (fsync)	`refresh` → new `DirectoryReader`
`Translog`	Crash recovery before the next commit	`Translog.sync()` (fsync, per request by default)	never — it is not a read path

This table is the answer to the most common confusion in the level: "if Lucene persists on commit, why a translog?" Because commits are expensive and infrequent; the translog makes every acknowledged write durable between commits, and is replayed on recovery. See the translog deep dive.

Step 8: Confirm the seqno/version assignment with a live request

Tie the code to behavior. Index, then read the metadata fields the engine assigned:

curl -s -XPOST 'localhost:9200/trace/_doc?pretty' -H 'Content-Type: application/json' \
  -d '{"title":"seqno demo"}'
# Response includes _version, _seq_no, _primary_term — the values assigned at Hop 5/6.

{
  "_index" : "trace",
  "_id" : "x1Ab...",
  "_version" : 1,
  "result" : "created",
  "_shards" : { "total" : 2, "successful" : 1, "failed" : 0 },
  "_seq_no" : 0,
  "_primary_term" : 1
}

Index the same _id again and watch _version/_seq_no advance — proof that updateDocument (Hop 6) ran, not addDocument:

curl -s -XPOST 'localhost:9200/trace/_doc/fixed?pretty' -H 'Content-Type: application/json' -d '{"v":1}'
curl -s -XPOST 'localhost:9200/trace/_doc/fixed?pretty' -H 'Content-Type: application/json' -d '{"v":2}'
# Second response: "_version": 2, "_seq_no" increments, "result": "updated".

Implementation Requirements / Deliverables

A completed reading-log artifact with one row per hop (Steps 2–7), each naming the exact file, class, and method.
The two assignment sites identified precisely: where the _seq_no/primary term is assigned on the primary vs replayed on the replica, with the method names.
The Lucene-vs-translog destination table, in your own words, explaining why both.
The live _doc response showing _version/_seq_no/_primary_term, and the update sequence showing them advance.
One open question per hop you couldn't fully answer — these are your future reading targets.

Troubleshooting

Symptom	Likely cause	Fix
`grep` finds nothing for a class	Path moved on your branch	Drop the path: `grep -rn "class TransportShardBulkAction" server/src/main`
TRACE produces no lines	Logger name typo or wrong package	Match package to the file's `package` declaration exactly
"go to definition" lands in a `.class` not source	IDE indexing incomplete	Re-import Gradle project; `./gradlew idea` if needed
Can't tell primary from replica path in the engine	Reading too fast	Find `planIndexingAsPrimary` vs `planIndexingAsNonPrimary` and compare
No `_seq_no` in the response	Old client/format	Use `?pretty` against a 2.x/3.x node; it's in the metadata block

Expected Output

Your reading log should resolve to roughly this chain (exact method names vary slightly by branch):

HTTP POST /trace/_doc
  → RestIndexAction / RestBulkAction          (rest/action/document/)
  → NodeClient.execute(BulkAction, ...)
  → TransportBulkAction.doExecute             (action/bulk/)        [coordinating: route, group by shard]
  → TransportShardBulkAction.shardOperationOnPrimary               [primary node]
  → IndexShard.applyIndexOperationOnPrimary   (index/shard/)        [parse+map, assign seqno/version]
  → InternalEngine.index(Engine.Index)        (index/engine/)       [plan add vs update]
  → IndexWriter.addDocument / updateDocument  (Lucene)              [the searchable index]
  → Translog.add(Translog.Index)              (index/translog/)     [write-ahead log]
  → (replication) TransportReplicationAction → replicas: applyIndexOperationOnReplica → InternalEngine.index

Stretch Goals

Trace a delete. Repeat for DELETE /trace/_doc/fixed: RestDeleteAction → TransportBulkAction (deletes are bulked too) → IndexShard.applyDeleteOperationOnPrimary → InternalEngine.delete → IndexWriter.deleteDocuments / soft-delete + Translog.add(Translog.Delete). Note how soft-deletes keep the doc for recovery.
Trace an update script. POST /trace/_update/fixed goes through TransportUpdateAction, which does a get-modify-reindex and then re-enters the bulk path. Find where it loops back.
Attach a debugger. ./gradlew run --debug-jvm, breakpoint in InternalEngine.indexIntoLucene and Translog.add, send one curl, and step through the plan.
Diagram it. Turn your reading log into a mermaid diagram and compare it to the one in the Level 6 overview. Where did your branch differ?

Coding Exercises

A trace lab still produces code: you turn each hop you mapped into an assertion. These run under :server:test (single-node) or :server:internalClusterTest. Locate every class/method with rg on your branch (e.g. rg "applyIndexOperationOnPrimary|generateSeqNoForOperationOnPrimary" server/src/main/java) — never cite a line number.

(warm-up) Assert refresh visibility, the behavior you traced. Write an OpenSearchSingleNodeTestCase (find the base with rg "class OpenSearchSingleNodeTestCase" test/framework/src/main/java) that indexes a doc with the default refresh, searches immediately and asserts 0 hits, then client().admin().indices().prepareRefresh(index).get() and asserts 1 hit. This is Hop 6's "visibility-on-refresh" row turned into a test. Prove it can fail by removing the refresh and watching the second assertion stay at 0.
(warm-up) Assert the seq_no/version assignment from Hop 5. In the same test, capture the IndexResponse of two writes to the same _id and assert: first is result == CREATED with _version == 1 and _seq_no == 0; second is UPDATED with _version == 2 and a larger _seq_no. This pins, in code, the "addDocument vs updateDocument" decision you read in InternalEngine.
(core) Instrument applyIndexOperationOnPrimary and capture it in a test. As a temporary patch, add a logger.trace(...) (or a test-only counter via a mock Engine plugin) at the point in IndexShard.applyIndexOperationOnPrimary where the Engine.Index is built. Write an integ test with @TestLogging("org.opensearch.index.shard:TRACE") that indexes one doc and asserts the captured log appears (use MockLogAppender/appender.assertAllExpectationsMatched() — find it with rg "MockLogAppender" test/framework/src/main/java). Deliver the patch as a small diff and revert the instrumentation after. This is how you prove a hop runs, not just claim it.
(core) Primary vs replica seq_no, asserted on a real replica. Write an OpenSearchIntegTestCase with 2 nodes, 1 primary + 1 replica. Index a doc, ensureGreen, then read the per-shard seqNo/localCheckpoint from both copies (via shard stats or IndexShard.seqNoStats() — rg "seqNoStats|getLocalCheckpoint" server/src/main/java) and assert the replica carries the same _seq_no the primary assigned (it was replayed, not regenerated). This turns the Hop 5 "two assignment sites" insight into a multi-node assertion.
(core) Translog grows on write, the second destination from Hop 6. Write a single-node test that indexes N docs into an index with refresh_interval: -1, then asserts client().admin().indices().prepareStats(index).get().getPrimaries().getTranslog().estimatedNumberOfOperations() (confirm the accessor with rg "getTranslog\(\)|estimatedNumberOfOperations|TranslogStats" server/src/main/java) equals N, then _flush and asserts uncommitted operations drop to ~0. Proves the Lucene-vs-translog table with code.
(advanced challenge) A "trace it, then guard it" regression test for the convergence path. Build one integ test that exercises the whole path you mapped and asserts an end-to-end invariant: with 1 primary + 2 replicas, index a doc with setRefreshPolicy(WriteRequest.RefreshPolicy.IMMEDIATE), then for every shard copy assert the doc is searchable and that _seq_no/_primary_term match across copies (read each copy via a node-pinned client and preference("_only_local") or routing). Then add a planted-bug-style negative arm: temporarily set number_of_replicas so one copy can't allocate, switch the gate to ensureYellow, and assert the as-yet-unassigned copy is reported in cluster health — proving your test distinguishes "converged" from "still converging." Survive -Dtests.iters=20. This is the reading log made executable: it would fail the day any hop on the path regresses. Cross-reference the replication deep dive.

Issues to Practice On

The indexing path touches routing, mapping, versioning, seqno, replication, and the translog — so "Indexing"/"Storage" issues land here constantly. Labels drift; list them with gh label list --repo opensearch-project/OpenSearch and confirm.

# Indexing / engine / write-path issues (labels move; confirm on the tracker).
gh issue list --repo opensearch-project/OpenSearch --label "good first issue" --label "Indexing" --state open
gh issue list --repo opensearch-project/OpenSearch --label "Indexing" --state open
gh issue list --repo opensearch-project/OpenSearch --label "Storage" --state open
gh issue list --repo opensearch-project/OpenSearch --search "seq_no OR primary_term in:title is:open"
gh issue list --repo opensearch-project/OpenSearch --search "version conflict in:title is:open label:bug"

Representative patterns:

"Indexing behavior X is undocumented/untested at the engine level." Reproduce the behavior with a curl, locate the responsible method on the path you mapped (rg "applyIndexOperationOnPrimary|planIndexingAsPrimary" server/src/main/java), and add the single-node or integ test that asserts it. Often a clean "add coverage" PR.
"_seq_no/_version reported wrong under condition Y." A versioning/seqno edge case. Reproduce, trace to IndexShard/InternalEngine, fix the assignment, and add the assertion (Exercises 2/4) that would have caught it.

Planted bug: In a local checkout, change the engine's new-vs-update decision — e.g. force the IndexingStrategy to treat an existing _id as an add (skip the updateDocument path), found via rg "planIndexingAsPrimary|IndexingStrategy|currentNotFoundOrDeleted" server/src/main/java/org/opensearch/index/engine/InternalEngine.java. Run your Exercise 2 test: the second write to the same _id no longer reports _version == 2/ UPDATED. That red is the bug. Revert, and keep the assertion that caught it.

Etiquette: Claim before you work, reproduce first, and ship a test + CHANGELOG.md + DCO Signed-off-by (git commit -s). See community interaction and the level-2 PR lab.

Validation / Self-check

Name all hops from POST /_doc to IndexWriter.addDocument, with the owning class at each.
Why do _doc and _bulk converge on the same code? At which class, and what does a single _doc become internally?
Exactly where is the _seq_no assigned on the primary, and why is it not re-assigned on a replica? Name the method on each side.
When does the engine call addDocument vs updateDocument, and what does updateDocument do under the hood?
The document is written to two destinations in InternalEngine.index. Name them and give the distinct reason each exists.
After indexing, your response showed _seq_no: 0, _primary_term: 1. Which classes produced those two numbers?
What did the live update sequence (fixed indexed twice) prove about which Lucene call ran the second time?

When your reading log narrates the full path without looking it up, proceed to Lab 6.2: IndexShard, InternalEngine, and the Translog.

Open-Source Engineer & Contributor