Stage 6 — Indexing and Engine Issues

What this stage teaches

Stage 6 goes inside a single shard, to the layer where documents are actually written, versioned, made durable, and made visible. This is the correctness-critical core: a bug here can lose data or return stale results. The skill is reasoning about the engine's concurrency and durability contracts and pinning them with InternalEngineTests-style unit tests.

• IndexShard — wraps the engine; entry points applyIndexOperationOnPrimary, applyIndexOperationOnReplica, applyDeleteOperationOnPrimary. Holds the versioning and sequence-number bookkeeping for the shard.
• InternalEngine — wraps Lucene's IndexWriter/DirectoryReader. Implements index(...), delete(...), refresh(...), flush(...), version/seqno conflict resolution, and the LiveVersionMap.
• Translog — the write-ahead log providing durability between Lucene commits; Translog.add, sync, generations, and the durability mode (REQUEST vs ASYNC).
• Sequence numbers — every operation gets a seqNo from LocalCheckpointTracker; the global checkpoint (ReplicationTracker) marks what is durably replicated. These drive replication, recovery, and dedup.

The two classic Stage 6 bugs: a versioning / seqno edge case (a concurrent update resolved wrong, a stale operation not skipped) and a translog durability bug (an operation acknowledged before it was synced, or a generation rolled at the wrong time).

Prerequisite: Stage 4 plus the engine deep dives: Engine internals, Translog, and Refresh/flush/merge. This stage assumes you know what a refresh, flush, and merge are.

The write path in one diagram

flowchart TD
  B[TransportShardBulkAction] --> P[IndexShard.applyIndexOperationOnPrimary]
  P --> V[resolve version + assign seqNo<br/>LocalCheckpointTracker]
  V --> E[InternalEngine.index]
  E --> LVM[LiveVersionMap update]
  E --> IW[Lucene IndexWriter.addDocument / updateDocument]
  E --> TL[Translog.add]
  P --> R[replicate to replicas<br/>applyIndexOperationOnReplica]
  TL --> SY{durability == REQUEST?}
  SY -->|yes| FS[Translog.sync before ack]
  SY -->|no| ASYNC[async sync on interval]

Two contracts you must not break:

1. Durability before acknowledgement. With index.translog.durability: request (the default), the operation's translog entry is fsync-ed before the client is told it succeeded. A bug that acks first and syncs later is silent data loss on crash.
2. Versioning monotonicity / seqno correctness. An operation with a stale version or an already-seen seqNo must be skipped or rejected deterministically, the same way on primary and replica, so the two never diverge. Get this wrong and a replica's documents differ from the primary's.

Finding Stage 6 issues

is:issue is:open label:bug no:assignee "translog" in:title,body
is:issue is:open label:bug no:assignee "version_conflict" in:title,body
is:issue is:open label:bug no:assignee "seq_no" in:title,body
is:issue is:open label:bug no:assignee label:"Storage:Durability"
is:issue is:open label:bug no:assignee label:"Indexing"

Area labels in this space include Storage:Durability, Storage:Performance, Indexing, and historically an Engine-flavoured label. Check the current list.

Fallback grep — the engine's edge-case logic clusters around version/seqno resolution:

grep -n "resolveDocVersion\|VersionConflictEngineException\|assertSameSeqNoOnReplica\|maxSeqNoOfUpdatesOrDeletes" \
  server/src/main/java/org/opensearch/index/engine/InternalEngine.java | head
grep -n "durability\|sync\|rollGeneration\|trimUnreferenced" \
  server/src/main/java/org/opensearch/index/translog/Translog.java | head

Walked example — a versioning / seqno edge case

Illustrative of the pattern. The grep finds the real plan-resolution method; do not trust the line numbers.

Symptom: an issue reports that under concurrent updates to the same doc id, a stale operation (lower version, already-superseded) is occasionally applied on a replica instead of being skipped, so the replica ends up one version behind the primary. The bug is in how the replica's index plan decides "is this operation stale?".

Locate the plan resolution

grep -n "planIndexingAsNonPrimary\|planIndexingAsPrimary\|IndexingStrategy\|isOptimizedFromDoc\|currentVersion" \
  server/src/main/java/org/opensearch/index/engine/InternalEngine.java | head
git log --oneline -n 5 -- server/src/main/java/org/opensearch/index/engine/InternalEngine.java

The schematic of a staleness check:

// In planIndexingAsNonPrimary (replica/recovery path):
if (index.versionType().isVersionConflictForWrites(currentVersion, index.version(), deleted)) {
    // treat as stale: skip but record seqNo as processed
    plan = IndexingStrategy.skipAsStale(...);
} else {
    plan = IndexingStrategy.processNormally(...);
}

The bug is typically a boundary in isVersionConflictForWrites or a missing check on the maxSeqNoOfUpdatesOrDeletes optimisation — an operation that looks fresh by version but is stale by seqNo, or vice versa.

Diff

--- a/server/src/main/java/org/opensearch/index/engine/InternalEngine.java
+++ b/server/src/main/java/org/opensearch/index/engine/InternalEngine.java
@@  IndexingStrategy planIndexingAsNonPrimary(Index index) {
-        if (index.seqNo() <= localCheckpointTracker.getProcessedCheckpoint()) {
-            // already processed
-            return IndexingStrategy.processButSkipLucene(false, index.version());
-        }
+        // An operation at or below the processed checkpoint has already been applied.
+        // It must be marked processed AND skipped in Lucene, otherwise a duplicate
+        // updateDocument can resurrect a stale version on the replica.
+        if (index.seqNo() <= localCheckpointTracker.getProcessedCheckpoint()) {
+            return IndexingStrategy.processButSkipLucene(false, index.version());
+        }
+        // Below maxSeqNoOfUpdatesOrDeletes we must read the live version map; above it,
+        // the optimization that skips the lookup is only safe because no update can target
+        // this id yet. Guard the optimization explicitly.
+        if (index.seqNo() > maxSeqNoOfUpdatesOrDeletes && canOptimizeAddDocument(index)) {
+            return IndexingStrategy.optimizedAppendOnly(index.version());
+        }

Warning: This is the single most dangerous region in the engine. Real fixes here are tiny but require understanding maxSeqNoOfUpdatesOrDeletes, the append-only optimisation, and the primary/replica symmetry. Always open a discuss comment on the issue with your reading before you write the diff — a maintainer in this area will catch a wrong mental model in one reply.

Test with InternalEngineTests

The engine has a rich unit-test base (EngineTestCase / InternalEngineTests) that builds a real InternalEngine over a temp Lucene directory, so you can replay exact operation sequences with chosen seqNos and versions:

public void testStaleReplicaOpIsSkipped() throws IOException {
    // Apply seqNo=2,v=2 first, then replay a stale seqNo=1,v=1 for the same id.
    ParsedDocument doc = testParsedDocument("1", null, testDocument(), B_1, null);

    Engine.IndexResult fresh = replicaEngine.index(replicaIndexForDoc(doc, /*version*/ 2, /*seqNo*/ 2, false));
    assertThat(fresh.getResultType(), equalTo(Engine.Result.Type.SUCCESS));

    Engine.IndexResult stale = replicaEngine.index(replicaIndexForDoc(doc, /*version*/ 1, /*seqNo*/ 1, false));
    // Stale op is accepted as "processed" but must NOT overwrite the fresher doc in Lucene.
    assertThat(stale.getResultType(), equalTo(Engine.Result.Type.SUCCESS));

    replicaEngine.refresh("test");
    try (Engine.Searcher searcher = replicaEngine.acquireSearcher("test")) {
        // The visible document must still be version 2.
        assertVisibleVersion(searcher, "1", 2L);
    }
}

replicaIndexForDoc, testParsedDocument, and the searcher helpers come from EngineTestCase. Extend it; never construct an InternalEngine by hand in a test.

grep -rln "extends EngineTestCase" server/src/test/java/org/opensearch/index/engine/
./gradlew :server:test --tests "org.opensearch.index.engine.InternalEngineTests" -q

Reproduce a specific randomized failure with the printed seed:

./gradlew :server:test --tests "org.opensearch.index.engine.InternalEngineTests" -Dtests.seed=<SEED> -q

A second pattern — translog durability

Illustrative of the pattern.

Symptom: with index.translog.durability: request, an operation is acknowledged before its translog location is synced, so a crash between ack and sync loses the write.

Locate the sync point

grep -n "ensureTranslogSynced\|Translog.Location\|sync(\|durability" \
  server/src/main/java/org/opensearch/index/shard/IndexShard.java | head
grep -n "Durability.REQUEST\|sync(" server/src/main/java/org/opensearch/index/translog/Translog.java | head

The fix shape

--- a/server/src/main/java/org/opensearch/index/shard/IndexShard.java
+++ b/server/src/main/java/org/opensearch/index/shard/IndexShard.java
@@
-        // ack the operation
-        listener.onResponse(result);
-        maybeSyncTranslog(result.getTranslogLocation());
+        // Durability contract: with REQUEST durability the translog must be fsync-ed
+        // for this location BEFORE we acknowledge to the client.
+        if (indexSettings.getTranslogDurability() == Translog.Durability.REQUEST) {
+            sync(result.getTranslogLocation());
+        }
+        listener.onResponse(result);

Test it

public void testRequestDurabilitySyncsBeforeAck() throws Exception {
    IndexShard shard = newStartedShard(true,
        Settings.builder().put(IndexSettings.INDEX_TRANSLOG_DURABILITY_SETTING.getKey(), "request").build());
    // index a doc; assert the translog is synced to at least this op's location at ack time.
    Engine.IndexResult r = indexDoc(shard, "_doc", "1");
    assertThat(shard.getTranslog().getLastSyncedGlobalCheckpoint() /* or syncedLocation */,
        greaterThanOrEqualTo(/* this op's location */));
    closeShards(shard);
}

newStartedShard, indexDoc, and closeShards come from IndexShardTestCase. This is the shard-level analog of EngineTestCase.

grep -rln "extends IndexShardTestCase" server/src/test/java/org/opensearch/index/shard/
./gradlew :server:test --tests "*IndexShardTests" -q

Pitfalls

• Asymmetry between primary and replica plans. planIndexingAsPrimary and planIndexingAsNonPrimary must agree on what is stale. A fix to one without the other silently diverges replicas — the worst class of bug because tests on a single node pass.
• Breaking the append-only optimisation. The maxSeqNoOfUpdatesOrDeletes optimisation skips the version-map lookup for ids that have never been updated. Touch it only with a maintainer's confirmation; getting it wrong loses or duplicates documents.
• Acking before sync. With REQUEST durability, sync precedes ack — always. With ASYNC, do not sync synchronously (that is a performance regression, Stage 10).
• Refresh ≠ flush ≠ commit. Refresh makes data visible; flush/commit makes it durable in Lucene and trims the translog. Confusing them produces both correctness and perf bugs. See refresh/flush/merge.
• Hand-building an InternalEngine. The setup (store, translog config, merge policy, seqno service) is intricate. Always extend EngineTestCase/IndexShardTestCase.
• Skipping the discuss step. Engine PRs that arrive without a stated mental model are the slowest to review. Three sentences on the issue first saves weeks.

Exit criteria — when you're ready for Stage 7

• One engine, translog, or shard fix is merged with an InternalEngineTests/IndexShardTestCase test that replays the exact operation sequence (chosen seqNos/versions) that triggers it.
• You can state the durability-before-ack contract and the primary/replica plan-symmetry contract from memory.
• You opened a discuss comment with your reading before writing the diff, and a maintainer confirmed (or corrected) the model.
• You can reproduce a randomized engine-test failure from its -Dtests.seed= line.

You have worked the write path. Stage 7 works the read path: queries, the query/fetch phases, and aggregation reduce.