Lab 6.1: Trace a Document Index Request to Lucene

Background

You cannot fix the write path until you can see it. In this lab you follow a single POST /<index>/_doc request — and then a _bulk request — from the HTTP socket all the way down to Lucene's IndexWriter.addDocument(...) and the translog's Translog.add(...), naming every class on the way. This is a code-reading trace, the most important skill in the entire curriculum: a senior contributor can open an unfamiliar request path and narrate it hop by hop using nothing but grep and the IDE's "go to definition."

The output of this lab is a reading-log artifact — a written, hop-by-hop map of the path with the exact file, class, and method at each step, plus the two spots where versioning and the sequence number are assigned. You will reuse this map for the rest of Level 6 and whenever you touch indexing.

Note: This lab is timed (see Step 0). The point is not to read everything — it is to learn to follow a path efficiently and stop at the right boundaries. Pair it with the engine internals and translog deep dives, and the Level 6 overview diagram.

Why This Lab Matters for Contributors

• Every indexing bug, performance issue, or feature request lands somewhere on this path. Knowing the path turns "I have no idea where to start" into "this is a TransportShardBulkAction problem."
• Maintainers describe bugs by class (IndexShard.applyIndexOperationOnPrimary). You must be able to open that method and orient yourself in seconds.
• The trace reveals where cross-cutting concerns live: routing, mapping, versioning, seqno assignment, replication, and the translog. Each is a future contribution area.
• A reading log is reusable: you will paste hops from it into PR descriptions and issue comments to show you understand the code you're changing.

Prerequisites

• A running node from source so you can correlate code with live behavior:

  ./gradlew run            # single-node cluster, REST on :9200
  

• The repo open in an IDE with working "go to definition" on org.opensearch.*.
• Familiarity with the REST layer deep dive and the action framework deep dive — those cover the upper hops in depth; here we move quickly through them to reach the engine.

Step-by-Step Tasks

Step 0: Set a timer and prepare the reading log

Give yourself 75 minutes. Create a scratch file and fill one row per hop as you go. The format:

HOP | FILE | CLASS#method | WHAT HAPPENS HERE | NOTE/QUESTION
----+------+--------------+-------------------+--------------

Resist the urge to read method bodies in full. For each hop, answer only: what does this hand off, and to whom? You are drawing a map, not auditing the code.

Step 1: Send the requests and watch them land

In one terminal, tail the node; in another, send the requests:

# Make the path observable: TRACE the bulk/shard action + engine for one request.
curl -s -XPUT 'localhost:9200/_cluster/settings' -H 'Content-Type: application/json' -d '{
  "transient": {
    "logger.org.opensearch.action.bulk": "TRACE",
    "logger.org.opensearch.index.shard": "TRACE",
    "logger.org.opensearch.index.engine": "TRACE",
    "logger.org.opensearch.index.translog": "TRACE"
  }
}'

# Single document
curl -s -XPOST 'localhost:9200/trace/_doc?pretty' -H 'Content-Type: application/json' \
  -d '{"title":"trace me","n":1}'

# Bulk
curl -s -XPOST 'localhost:9200/_bulk?pretty' -H 'Content-Type: application/x-ndjson' --data-binary $'
{"index":{"_index":"trace","_id":"b1"}}
{"title":"bulk one","n":2}
{"index":{"_index":"trace","_id":"b2"}}
{"title":"bulk two","n":3}
'

# Turn TRACE back off when done.
curl -s -XPUT 'localhost:9200/_cluster/settings' -H 'Content-Type: application/json' -d '{
  "transient": { "logger.org.opensearch.action.bulk": null, "logger.org.opensearch.index.shard": null,
                 "logger.org.opensearch.index.engine": null, "logger.org.opensearch.index.translog": null }
}'

The TRACE lines in the node's stdout name the classes you are about to read — use them to confirm your map matches reality.

Step 2: Hop 1 — REST handler

Both _doc and _bulk enter through a RestHandler registered in ActionModule. Find them:

grep -rn "class RestIndexAction"  server/src/main/java/org/opensearch/rest/action/document/
grep -rn "class RestBulkAction"   server/src/main/java/org/opensearch/rest/action/document/
# How each routes the parsed request into the transport layer:
grep -n "client.execute\|client.bulk\|prepareRequest\|BulkRequest\|IndexRequest" \
  server/src/main/java/org/opensearch/rest/action/document/RestBulkAction.java

Record: RestIndexAction/RestBulkAction parse the HTTP body into an IndexRequest/BulkRequest and call NodeClient.execute(BulkAction.INSTANCE, request, listener). A single _doc is internally wrapped into a one-item bulk — both paths converge on TransportBulkAction. Note that convergence; it's the key insight that collapses two paths into one.

Step 3: Hop 2 — coordinating action (TransportBulkAction)

find server/src/main/java -name "TransportBulkAction.java"
grep -n "doExecute\|doInternalExecute\|executeBulk\|createIndex\|resolveRouting\|BulkShardRequest\|groupRequestsByShards\|ingest\|pipeline" \
  server/src/main/java/org/opensearch/action/bulk/TransportBulkAction.java

Record what happens on the coordinating node:

• auto-create the index if missing (a cluster-state update),
• run ingest pipelines if any,
• resolve routing → which shard each op targets,
• group operations by shard into BulkShardRequests,
• dispatch each BulkShardRequest to the shard's primary via TransportShardBulkAction.

This is the "fan-out by shard" hop. A single _doc produces exactly one BulkShardRequest.

Step 4: Hop 3 — the primary write (TransportShardBulkAction)

find server/src/main/java -name "TransportShardBulkAction.java"
grep -n "extends TransportWriteAction\|extends TransportReplicationAction\|shardOperationOnPrimary\|executeBulkItemRequest\|applyIndexOperationOnPrimary\|dispatchedShardOperationOnPrimary" \
  server/src/main/java/org/opensearch/action/bulk/TransportShardBulkAction.java

TransportShardBulkAction extends TransportWriteAction extends TransportReplicationAction — that inheritance is the primary→replica machinery. On the primary it calls (per item) IndexShard.applyIndexOperationOnPrimary(...). Record the method name that bridges into the shard.

Note: TransportReplicationAction is the reusable base for all write actions (index, delete, bulk). It handles routing to the primary, executing on the primary, then replicating to replicas and waiting for the required number of copies. See the replication deep dive.

Step 5: Hop 4 — the shard (IndexShard.applyIndexOperationOnPrimary)

grep -n "applyIndexOperationOnPrimary\|applyIndexOperationOnReplica\|applyIndexOperation\|prepareIndex\|docMapper\|MapperService\|markSeqNoAsNoop\|Engine.Index\|getEngine().index" \
  server/src/main/java/org/opensearch/index/shard/IndexShard.java

This is the hinge of the whole path. Record, in order:

1. The source is parsed and mapped here: prepareIndex(...) uses the DocumentMapper to build a ParsedDocument (Lucene fields). (Trace into MapperService/DocumentMapper only briefly — that's Lab 6.3's territory.)
2. An Engine.Index operation is constructed wrapping the parsed doc, the version, the version type, and (on the primary) the assignment of the next _seq_no/primary term.
3. It calls getEngine().index(engineIndex) → InternalEngine.index(...).

Note the two assignment sites you must pin down precisely:

# Where versioning + seqno are resolved on the primary vs replayed on the replica.
grep -n "primaryTerm\|seqNo\|SequenceNumbers\|UNASSIGNED_SEQ_NO\|versionType\|resolveDocVersion\|getOperationPrimaryTerm" \
  server/src/main/java/org/opensearch/index/shard/IndexShard.java | head

Step 6: Hop 5 — the engine (InternalEngine.index)

This is the bottom of the path that you own as a reader. Open it:

grep -n "public IndexResult index(Index index)\|private IndexResult indexIntoLucene\|planIndexingAsPrimary\|planIndexingAsNonPrimary\|generateSeqNoForOperationOnPrimary\|addDocs\|updateDocs\|addDocuments\|translog\.\|getTranslog().add\|markSeqNoAsCompleted\|versionMap" \
  server/src/main/java/org/opensearch/index/engine/InternalEngine.java | head -40

Record the engine's decision flow:

• index(Index) builds an indexing plan (IndexingStrategy): is this a new doc (addDocument) or an update (updateDocument, which deletes-then-adds)? Does version conflict? Is this primary or replica?
• On the primary, generateSeqNoForOperationOnPrimary(...) assigns the _seq_no (via the LocalCheckpointTracker). On a replica, the _seq_no arrives with the operation and is not regenerated — confirm this in the non-primary plan.
• indexIntoLucene(...) calls Lucene IndexWriter.addDocument(...) (new) or updateDocument(...) (update — a delete + add keyed on _id).
• Translog.add(new Translog.Index(...)) appends the op to the write-ahead log.
• The local checkpoint is advanced (markSeqNoAsCompleted-style call) once the op is durable enough.

Step 7: Hop 6 — Lucene and the translog (the two destinations)

The path ends in two places, written for two different reasons:

# Lucene write (visibility-on-refresh, durability-on-commit):
grep -n "indexWriter.addDocument\|indexWriter.updateDocument\|indexWriter.softUpdateDocument\|addDocuments\|updateDocuments" \
  server/src/main/java/org/opensearch/index/engine/InternalEngine.java | head

# Translog write (durability-now, replay-on-recovery):
find server/src/main/java -name "Translog.java"
grep -n "public Location add(Operation operation)\|class Index\|void sync\|ensureSynced" \
  server/src/main/java/org/opensearch/index/translog/Translog.java | head

Record the why for each destination:

This table is the answer to the most common confusion in the level: "if Lucene persists on commit, why a translog?" Because commits are expensive and infrequent; the translog makes every acknowledged write durable between commits, and is replayed on recovery. See the translog deep dive.

Step 8: Confirm the seqno/version assignment with a live request

Tie the code to behavior. Index, then read the metadata fields the engine assigned:

curl -s -XPOST 'localhost:9200/trace/_doc?pretty' -H 'Content-Type: application/json' \
  -d '{"title":"seqno demo"}'
# Response includes _version, _seq_no, _primary_term — the values assigned at Hop 5/6.

{
  "_index" : "trace",
  "_id" : "x1Ab...",
  "_version" : 1,
  "result" : "created",
  "_shards" : { "total" : 2, "successful" : 1, "failed" : 0 },
  "_seq_no" : 0,
  "_primary_term" : 1
}

Index the same _id again and watch _version/_seq_no advance — proof that updateDocument (Hop 6) ran, not addDocument:

curl -s -XPOST 'localhost:9200/trace/_doc/fixed?pretty' -H 'Content-Type: application/json' -d '{"v":1}'
curl -s -XPOST 'localhost:9200/trace/_doc/fixed?pretty' -H 'Content-Type: application/json' -d '{"v":2}'
# Second response: "_version": 2, "_seq_no" increments, "result": "updated".

Implementation Requirements / Deliverables

• A completed reading-log artifact with one row per hop (Steps 2–7), each naming the exact file, class, and method.
• The two assignment sites identified precisely: where the _seq_no/primary term is assigned on the primary vs replayed on the replica, with the method names.
• The Lucene-vs-translog destination table, in your own words, explaining why both.
• The live _doc response showing _version/_seq_no/_primary_term, and the update sequence showing them advance.
• One open question per hop you couldn't fully answer — these are your future reading targets.

Troubleshooting

Expected Output

Your reading log should resolve to roughly this chain (exact method names vary slightly by branch):

HTTP POST /trace/_doc
  → RestIndexAction / RestBulkAction          (rest/action/document/)
  → NodeClient.execute(BulkAction, ...)
  → TransportBulkAction.doExecute             (action/bulk/)        [coordinating: route, group by shard]
  → TransportShardBulkAction.shardOperationOnPrimary               [primary node]
  → IndexShard.applyIndexOperationOnPrimary   (index/shard/)        [parse+map, assign seqno/version]
  → InternalEngine.index(Engine.Index)        (index/engine/)       [plan add vs update]
  → IndexWriter.addDocument / updateDocument  (Lucene)              [the searchable index]
  → Translog.add(Translog.Index)              (index/translog/)     [write-ahead log]
  → (replication) TransportReplicationAction → replicas: applyIndexOperationOnReplica → InternalEngine.index

Stretch Goals

1. Trace a delete. Repeat for DELETE /trace/_doc/fixed: RestDeleteAction → TransportBulkAction (deletes are bulked too) → IndexShard.applyDeleteOperationOnPrimary → InternalEngine.delete → IndexWriter.deleteDocuments / soft-delete + Translog.add(Translog.Delete). Note how soft-deletes keep the doc for recovery.
2. Trace an update script. POST /trace/_update/fixed goes through TransportUpdateAction, which does a get-modify-reindex and then re-enters the bulk path. Find where it loops back.
3. Attach a debugger. ./gradlew run --debug-jvm, breakpoint in InternalEngine.indexIntoLucene and Translog.add, send one curl, and step through the plan.
4. Diagram it. Turn your reading log into a mermaid diagram and compare it to the one in the Level 6 overview. Where did your branch differ?

Validation / Self-check

1. Name all hops from POST /_doc to IndexWriter.addDocument, with the owning class at each.
2. Why do _doc and _bulk converge on the same code? At which class, and what does a single _doc become internally?
3. Exactly where is the _seq_no assigned on the primary, and why is it not re-assigned on a replica? Name the method on each side.
4. When does the engine call addDocument vs updateDocument, and what does updateDocument do under the hood?
5. The document is written to two destinations in InternalEngine.index. Name them and give the distinct reason each exists.
6. After indexing, your response showed _seq_no: 0, _primary_term: 1. Which classes produced those two numbers?
7. What did the live update sequence (fixed indexed twice) prove about which Lucene call ran the second time?

When your reading log narrates the full path without looking it up, proceed to Lab 6.2: IndexShard, InternalEngine, and the Translog.