Engine Internals

The Engine is the heart of a shard — the layer that wraps Lucene and turns "apply this index/delete operation" into concrete IndexWriter calls, a durable translog record, a versioning decision, and an eventually visible searcher. IndexShard (index-shard-lifecycle.md) delegates all real storage work to its Engine. This chapter covers the engine implementations (InternalEngine, ReadOnlyEngine, NRTReplicationEngine), the Lucene IndexWriter it drives, the Engine.Index/Engine.Delete operation model, versioning via the LiveVersionMap, sequence numbers (LocalCheckpointTracker, SeqNoStats), and how a refresh produces a new DirectoryReader through a SearcherManager. It is the connective chapter between the write path, the translog, and refresh/flush/merge.

After this chapter you should be able to: explain what InternalEngine.index() does step by step; describe how versioning and sequence numbers prevent lost or out-of-order updates; explain the LiveVersionMap's role; and find where a refresh swaps in a new reader.

Note: OpenSearch's engine is built directly on Apache Lucene. Everything here ultimately reduces to Lucene IndexWriter/DirectoryReader/IndexSearcher calls; the engine adds versioning, sequence numbers, the translog, and the OpenSearch operation model on top.

The engine implementations

find server -name "Engine.java" -path "*index/engine*"
find server -name "InternalEngine.java" -o -name "ReadOnlyEngine.java" -o -name "NRTReplicationEngine.java"
grep -n "abstract class Engine\|public abstract" \
  server/src/main/java/org/opensearch/index/engine/Engine.java | head

The default is InternalEngine. The engine is created via an EngineFactory, which a plugin can override (EnginePlugin) — see plugin-architecture.md.

grep -n "EngineFactory\|newReadWriteEngine\|getEngineFactory" \
  server/src/main/java/org/opensearch/index/engine/EngineFactory.java \
  server/src/main/java/org/opensearch/plugins/EnginePlugin.java 2>/dev/null

The operation model: Engine.Index and Engine.Delete

Every write is an Engine.Operation. The two you care about:

grep -n "class Index\|class Delete\|class NoOp\|abstract class Operation\|VersionType\|seqNo\|primaryTerm\|Origin" \
  server/src/main/java/org/opensearch/index/engine/Engine.java | head -40

The origin is crucial: the same index() method behaves differently for a fresh primary write (assign seq no, check version) versus a replayed translog op during recovery (trust the recorded seq no). This is the engine-level expression of the primary/replica asymmetry from index-shard-lifecycle.md.

InternalEngine.index(): step by step

grep -n "public IndexResult index\|planIndexingAsPrimary\|planIndexingAsNonPrimary\|\
indexIntoLucene\|addStaleDocs\|updateDocument\|addDocument\|Translog.add\|versionMap" \
  server/src/main/java/org/opensearch/index/engine/InternalEngine.java | head -40

The essential sequence for a primary index op:

flowchart TD
    Idx["InternalEngine.index(Engine.Index)"] --> Lock[acquire per-uid lock]
    Lock --> Plan{plan: new doc, update, or conflict?}
    Plan -->|version conflict| Conflict[return VersionConflictEngineException]
    Plan -->|ok| SeqNo[assign seqNo + primaryTerm if primary]
    SeqNo --> Lucene["IndexWriter.addDocument / updateDocument"]
    Lucene --> VMap["LiveVersionMap.put(uid, version+seqNo)"]
    VMap --> TL["Translog.add(operation)"]
    TL --> CP[LocalCheckpointTracker.markSeqNoAsProcessed]
    CP --> Result[IndexResult]

The order is deliberate: Lucene write, then version map update, then translog append, then checkpoint advance. The document is durable once it is in the translog (and fsynced per durability policy — see translog.md); it becomes visible only after a refresh opens a new reader.

Note: A new or updated document is durable before it is visible. This is the near-real-time (NRT) model: you can lose a node right after an acknowledged write and recover the op from the translog, even though no search would have returned it yet. Durability (translog) and visibility (refresh) are independent — see refresh-flush-merge.md.

Versioning and the LiveVersionMap

To decide whether an incoming write is newer than what is already stored — and to do it before a refresh makes the prior write searchable — the engine keeps an in-memory LiveVersionMap: _id → (version, seqNo, location) for recently written, not-yet-refreshed documents.

find server -name "LiveVersionMap.java" -o -name "VersionValue.java"
grep -n "class LiveVersionMap\|getUnderLock\|putIndexUnderLock\|putDeleteUnderLock\|maps\|archive" \
  server/src/main/java/org/opensearch/index/engine/LiveVersionMap.java

Why it exists: Lucene only knows what is in committed/refreshed segments. Between a write and the next refresh, the authoritative current version of a document is only in memory. The LiveVersionMap is that memory. On a version check the engine consults the map first, then falls back to reading the version from Lucene.

flowchart LR
    W["write _id=42, v=5"] --> Check[version check]
    Check -->|"in LiveVersionMap?"| Map["LiveVersionMap: 42 -> v5,seqNo"]
    Check -->|"not in map"| Lucene[read version from segments]
    Map --> Decide{incoming v > stored v?}
    Lucene --> Decide
    Decide -->|yes| Apply[apply]
    Decide -->|no| Conflict[VersionConflictEngineException]

After a refresh, the document is in a segment, so its entry can be dropped from the live map (it moves to an "archive" briefly to handle in-flight reads). A correctness bug in this map shows up as lost updates or spurious version conflicts under concurrency.

Sequence numbers and checkpoints

Every operation gets a sequence number (seq no), unique and increasing per shard within a primary term. Seq nos give a total order to a shard's history and are the foundation of fast recovery and replication.

find server -name "LocalCheckpointTracker.java" -o -name "SeqNoStats.java" -o -name "SequenceNumbers.java"
grep -n "generateSeqNo\|markSeqNoAsProcessed\|getProcessedCheckpoint\|getPersistedCheckpoint\|class LocalCheckpointTracker" \
  server/src/main/java/org/opensearch/index/seqno/LocalCheckpointTracker.java

The local checkpoint advances only when there are no gaps, which is why out-of-order arrival (common with concurrent replication) does not falsely advance it. The global checkpoint, derived across copies, is the boundary for trimming the translog and for sequence-number-based recovery (recovery.md, replication.md).

flowchart LR
    Ops["seq nos: 0 1 2 _ 4 5"] --> LCP["local checkpoint = 2 (gap at 3)"]
    LCP --> Fill["op 3 arrives"]
    Fill --> LCP2["local checkpoint = 5"]
    LCP2 --> GCP["global checkpoint = min(local cp across in-sync copies)"]

Searchers and refresh

Reads do not go through the IndexWriter; they go through an Engine.Searcher, a point-in-time view backed by a Lucene DirectoryReader. New writes are invisible until a refresh opens a fresh reader via the SearcherManager (Lucene's ReferenceManager for readers).

grep -n "acquireSearcher\|class Searcher\|ReferenceManager\|SearcherManager\|refresh(\|externalReaderManager\|getReferenceManager" \
  server/src/main/java/org/opensearch/index/engine/InternalEngine.java | head

flowchart TD
    Write["index op -> IndexWriter (in-memory buffer)"] --> Buffer[uncommitted, unsearchable]
    Refresh["refresh()"] --> NewReader["SearcherManager opens new DirectoryReader"]
    Buffer --> Refresh
    NewReader --> Visible[recent docs now searchable]
    Search["acquireSearcher()"] --> NewReader

acquireSearcher hands out a reference-counted searcher; the caller (the search layer) must release it. The relationship to refresh/flush/merge — what each does to readers, segments, and the commit point — is the subject of refresh-flush-merge.md.

The engine's three companions

The engine does not work alone. Its relationships:

Reading exercise

# 1. The op model.
grep -n "class Index\|class Delete\|class NoOp\|enum Origin\|VersionType" \
  server/src/main/java/org/opensearch/index/engine/Engine.java | head

# 2. The index() implementation.
grep -n "planIndexing\|indexIntoLucene\|addDocs\|updateDocs\|versionMap\|Translog.add\|markSeqNoAsProcessed" \
  server/src/main/java/org/opensearch/index/engine/InternalEngine.java

# 3. The version map.
grep -n "getUnderLock\|putIndexUnderLock\|beforeRefresh\|afterRefresh\|class LiveVersionMap" \
  server/src/main/java/org/opensearch/index/engine/LiveVersionMap.java

# 4. Checkpoints.
grep -n "generateSeqNo\|getProcessedCheckpoint\|markSeqNoAsProcessed" \
  server/src/main/java/org/opensearch/index/seqno/LocalCheckpointTracker.java

# 5. Tests + live stats.
./gradlew :server:test --tests "org.opensearch.index.engine.InternalEngineTests"
curl -s "localhost:9200/orders/_stats/docs,segments?pretty"
curl -s "localhost:9200/orders/_stats?filter_path=**.seq_no"

Answer:

1. Name the three engine implementations and exactly when each is used. Which one does not index documents locally, and why?
2. Walk InternalEngine.index() for a primary op: list the steps in order and say at which step the document becomes (a) durable and (b) visible.
3. What problem does LiveVersionMap solve that reading the version from Lucene cannot? When can an entry leave the map?
4. Explain the difference between the local checkpoint and the global checkpoint. Why does a gap stop the local checkpoint from advancing?
5. Why do reads use an Engine.Searcher rather than the IndexWriter, and what makes a just-written document appear in a searcher?
6. What does origin (PRIMARY vs PEER_RECOVERY) change about how index() handles versioning?

Common bugs and symptoms

Validation: prove you understand this

1. Draw the InternalEngine.index() pipeline and mark the durability point and the visibility point.
2. Explain LiveVersionMap in one paragraph: what it stores, why Lucene alone is insufficient, and what a bug in it causes.
3. Explain local vs global checkpoint with a worked seq-no example containing a gap.
4. Explain how origin makes the same index() method behave correctly for a primary write and for a translog replay during recovery.
5. Describe how a refresh turns an in-memory write into a searchable document, naming the Lucene constructs (IndexWriter, DirectoryReader, SearcherManager).
6. Name the three companions of the engine (translog, refresh/flush/merge, replication/recovery) and one way each depends on the engine's seq nos.