Level 6: Indexing Path and Storage Engine

You have learned how OpenSearch defends itself with tests (Level 5). Now you learn what it is defending: the write path — the journey of a document from an HTTP request to durable, searchable bytes on disk. This is the heart of OpenSearch as a storage engine, and it is where the project meets Apache Lucene.

A document does not simply "get saved." It is routed to a shard, applied to a primary, versioned and stamped with a sequence number, written to Lucene's in-memory buffer and appended to a write-ahead log (the translog), replicated to replicas, then — separately and asynchronously — made visible by a refresh and made durable as a Lucene commit by a flush, while background merges keep the segment count sane. Every one of those words is a class you can open, a setting you can tune, and a source of real bugs you can fix.

By the end of this level you can stand at any point on the write path and say which class is running, what invariant it maintains, and what would break if it were wrong.

Learning Objectives

By the end of Level 6 you must be able to:

1. Trace a POST /<index>/_doc and a _bulk request from the REST layer all the way to Lucene's IndexWriter and the translog, naming every hop and the class that owns it.
2. Explain the division of labor between IndicesService, IndexService, IndexShard, the Engine abstraction and its InternalEngine implementation, the Translog, and the mapping stack (MapperService/DocumentMapper).
3. Explain how versioning and sequence numbers (_seq_no, primary term) are assigned on the primary and replayed on replicas, and how local/global checkpoints track durability.
4. Distinguish refresh (visibility — opens a new searcher), flush (durability — Lucene commit + translog trim), and merge (segment housekeeping), and reason about the settings that control each.
5. Reason about durability under failure: what the translog guarantees, what index.translog.durability controls, and what gets replayed on recovery.
6. Map a parsed JSON document through the mapping/analysis pipeline into Lucene fields.

The Write Path, End to End

A single-document index and a bulk request share almost the whole path; bulk just batches at the coordinating and shard level. The canonical flow:

flowchart TD
    A["HTTP POST /idx/_doc or _bulk"] --> B["RestController → RestIndexAction / RestBulkAction"]
    B --> C["NodeClient.execute(...) → TransportBulkAction"]
    C -->|resolve routing, group by shard| D["TransportShardBulkAction<br/>(extends TransportReplicationAction)"]
    D -->|on the PRIMARY node| E["IndexShard.applyIndexOperationOnPrimary(...)"]
    E -->|parse + map| F["MapperService / DocumentMapper → ParsedDocument (Lucene fields)"]
    E -->|assign _seq_no, version, primary term| G["InternalEngine.index(Engine.Index)"]
    G --> H["Lucene IndexWriter.addDocument / updateDocument"]
    G --> I["Translog.add(operation)  (write-ahead log)"]
    D -->|replicate the same op| J["replica shards: applyIndexOperationOnReplica → InternalEngine.index"]
    H -.->|later, async| K["refresh: open new DirectoryReader → docs become searchable"]
    H -.->|later, async| L["flush: IndexWriter.commit() + trim translog (durable Lucene commit)"]
    H -.->|background| M["merge: MergePolicy/MergeScheduler combine segments"]

Read it as three phases:

1. Coordination (any node): REST parses the request; TransportBulkAction resolves index/routing, may create the index, applies pipelines, and groups operations by target shard.
2. Primary write (the node holding the primary): TransportShardBulkAction runs each op through IndexShard.applyIndexOperationOnPrimary, which parses + maps the source, then calls InternalEngine.index(...), which writes to Lucene's IndexWriter and appends to the Translog. Versioning and the sequence number are assigned here, on the primary.
3. Replication (replica nodes): TransportReplicationAction ships the same operation (with the already-assigned _seq_no/version/primary term) to each replica, which applies it via applyIndexOperationOnReplica. (With segment replication, replicas instead copy finished segments from the primary — see replication.)

Then, decoupled from the request, three background processes govern visibility, durability, and storage shape:

Note: Refresh ≠ flush. Refresh makes docs visible; flush makes them durable as a Lucene commit. Between refreshes, a doc is buffered (visible after refresh) but already durable via the translog (fsync'd per request by default). This separation — visibility vs. durability — is the single most important mental model in this level.

Mappings and Analysis: From JSON to Lucene Fields

Before a document reaches the engine, its JSON source is turned into Lucene fields. The MapperService holds the index's mapping; the DocumentMapper parses the source into a ParsedDocument (a set of Lucene IndexableFields) according to each field's Mapper. Text fields run through an analyzer (a Tokenizer + TokenFilter chain) into indexed terms; keyword/numeric fields are indexed verbatim and as DocValues. Dynamic mapping infers types for unseen fields and triggers a cluster-state mapping update.

JSON source ──DocumentMapper──► fields:
  "title": "Fast Search"  ──(text, standard analyzer)──► terms: [fast, search]  + (optional) DocValues
  "views": 42             ──(long)──────────────────────► points + DocValues
  "tag":   "prod"         ──(keyword)────────────────────► term: [prod]         + DocValues

Analysis is the subject of Lab 6.3 (you build a custom analyzer plugin) and the mapping & analysis deep dive.

Sequence Numbers and Checkpoints

Every write on a primary is stamped with two coordinates that make replication and recovery correct:

The global checkpoint is what lets recovery be cheap: a recovering replica only needs operations after the global checkpoint (replayed from the translog), not the whole shard. This is the backbone of peer recovery. When you read _seq_no assignment in Lab 6.1, you are reading the code that makes all of this work.

# See the seqno/checkpoint plumbing for yourself.
grep -rn "LocalCheckpointTracker\|globalCheckpoint\|primaryTerm\|generateSeqNo\|SequenceNumbers" \
  server/src/main/java/org/opensearch/index/seqno \
  server/src/main/java/org/opensearch/index/engine/InternalEngine.java | head -30

Key Classes

# Confirm the locations on your branch (names are stable; paths may shift slightly).
for c in IndicesService IndexService index/shard/IndexShard index/engine/Engine \
         index/engine/InternalEngine index/translog/Translog index/mapper/MapperService \
         index/mapper/DocumentMapper; do
  find server/src/main/java -path "*org/opensearch/${c}.java" -o -name "$(basename $c).java" -path "*org/opensearch/*" 2>/dev/null
done | sort -u

The Labs

These build on each other: 6.1 maps the territory, 6.2 makes you operate the engine from the outside and reason about translog/refresh/flush, and 6.3 has you extend the analysis pipeline with a real plugin.

Deliverables

You must demonstrate all of the following before advancing to Level 7:

• A written reading-log artifact tracing _doc and _bulk from REST to IndexWriter.addDocument and Translog.add, naming each class and where _seq_no/version are assigned (Lab 6.1).
• Empirical observations, via curl, of refresh changing visibility, flush trimming the translog, and index.translog.durability changing fsync behavior — mapped back to the source (Lab 6.2).
• A working custom analyzer plugin: a Plugin implements AnalysisPlugin, a token filter factory, a Lucene TokenFilter, built with opensearch.opensearchplugin, installed, and verified with _analyze and an OpenSearchTokenStreamTestCase unit test (Lab 6.3).
• From memory: the difference between refresh, flush, and merge; what the global checkpoint enables; and why the translog exists when Lucene already persists segments on commit.

Common Mistakes

How to Verify Success

# 1) You can locate every hop on the write path from a single grep chain.
grep -rn "applyIndexOperationOnPrimary" server/src/main/java/org/opensearch/index/shard/IndexShard.java
grep -rn "public IndexResult index" server/src/main/java/org/opensearch/index/engine/InternalEngine.java
grep -rn "addDocument\|updateDocument\|translog.add\|getTranslog().add" \
  server/src/main/java/org/opensearch/index/engine/InternalEngine.java

# 2) You can drive visibility vs durability from the outside (Lab 6.2).
curl -s -XPUT  'localhost:9200/demo?pretty'
curl -s -XPOST 'localhost:9200/demo/_doc?pretty' -H 'Content-Type: application/json' -d '{"t":"hello"}'
curl -s 'localhost:9200/demo/_search?pretty'            # likely 0 hits before refresh
curl -s -XPOST 'localhost:9200/demo/_refresh?pretty'
curl -s 'localhost:9200/demo/_search?pretty'            # now 1 hit
curl -s 'localhost:9200/_cat/segments/demo?v'

When you can trace a document to Lucene, explain seqno/checkpoints, drive refresh/flush/merge from curl, and have built a working analysis plugin, you are ready for the search side in Level 7. For the underlying theory, lean on the engine internals, translog, refresh/flush/merge, mapping & analysis, and index-shard lifecycle deep dives.