The Translog

Lucene only makes data durable when it commits — an expensive fsync of all segment files — and commits are far too costly to perform per write. So how does OpenSearch acknowledge a write as durable without committing Lucene every time? The answer is the translog: a per-shard, append-only write-ahead log. Every indexing operation is appended (and, by default, fsynced) to the translog before the write is acknowledged. If the node crashes before the next Lucene commit, the operations are replayed from the translog on restart. This chapter covers the Translog class and its readers/writers, generations, durability modes, the relationship between the translog and the Lucene commit (flush), recovery from translog, and how the global checkpoint governs trimming.

After this chapter you should be able to: explain exactly what "durable" means for an acknowledged write; describe the difference between request and async durability; explain how a flush relates a translog generation to a Lucene commit; and trace how a crash recovers unflushed operations.

Note: "Durable" (survives a crash) and "visible" (searchable) are different. The translog gives durability immediately; a refresh gives visibility later. Do not conflate them.

The classes

find server -name "Translog.java" -o -name "TranslogWriter.java" -o -name "TranslogReader.java" \
  -o -name "TranslogConfig.java"
grep -n "class Translog\|class TranslogWriter\|class TranslogReader\|add(\|newSnapshot\|rollGeneration\|trimUnreferencedReaders" \
  server/src/main/java/org/opensearch/index/translog/Translog.java | head

flowchart TD
    Eng[InternalEngine.index] -->|"Translog.add(op)"| W["TranslogWriter (gen N, current)"]
    W --> File1["translog-N.tlog (being written)"]
    Trans[Translog] --> R1["TranslogReader gen N-1 (sealed)"]
    Trans --> R2["TranslogReader gen N-2 (sealed)"]
    R1 --> File2[translog-N-1.tlog]
    R2 --> File3[translog-N-2.tlog]
    Trans --> Ckp[translog.ckp - checkpoint file]

The files live under the shard's translog directory:

# On a running node, find the data path, then:
find . -path "*nodes*translog*" -name "*.tlog" 2>/dev/null | head
# Or read the config defaults:
grep -n "translog\|durability\|sync_interval\|flush_threshold_size" \
  server/src/main/java/org/opensearch/index/IndexSettings.java | head

The Operation model

grep -n "class Index\|class Delete\|class NoOp\|abstract class Operation\|enum Type\|writeTo\|readOperation" \
  server/src/main/java/org/opensearch/index/translog/Translog.java

Every op carries its sequence number and primary term (see engine-internals.md). That is what lets translog replay be deterministic and idempotent: a replayed op reuses its recorded seq no rather than getting a new one.

Durability: request vs async

The key tunable is index.translog.durability:

grep -n "durability\|REQUEST\|ASYNC\|sync_interval\|Durability" \
  server/src/main/java/org/opensearch/index/translog/Translog.java \
  server/src/main/java/org/opensearch/index/IndexSettings.java | head

sequenceDiagram
    participant C as Client
    participant E as Engine
    participant T as Translog
    C->>E: index doc
    E->>E: IndexWriter.addDocument (in-memory)
    E->>T: Translog.add(op) -> Location
    alt durability = request
        T->>T: fsync NOW
        T-->>E: synced
        E-->>C: 200 (durable)
    else durability = async
        T-->>E: appended (not yet fsynced)
        E-->>C: 200 (durable only after next interval sync)
        Note over T: fsync every sync_interval
    end

Warning: async durability is a deliberate durability-for-throughput trade. With async, a 200 OK does not guarantee the write survives an immediate power loss — up to sync_interval of operations are at risk. Default request durability is what makes OpenSearch's write acknowledgements meaningful. Change it only with eyes open.

Generations, flush, and the Lucene commit

The translog only needs to retain operations not yet captured in a Lucene commit. A flush is the operation that bridges them:

grep -n "flush\|rollGeneration\|commitIndexWriter\|getMinTranslogGenerationForRecovery\|trimUnreferencedReaders" \
  server/src/main/java/org/opensearch/index/engine/InternalEngine.java | head

A flush does three things atomically with respect to recovery:

1. Lucene commit — IndexWriter.commit() fsyncs all segments; the data is now durable in Lucene itself.
2. Roll the translog generation — start a new TranslogWriter (gen N+1); the commit records which translog generation it corresponds to.
3. Trim old generations — older translog files whose ops are now safely in the Lucene commit (and below the global checkpoint) are deleted.

flowchart LR
    subgraph "before flush"
      Seg1[committed segments @ gen 5] --> TL5["translog gens 5..7 retained"]
    end
    Flush["flush()"] --> Commit[IndexWriter.commit]
    Commit --> Roll[roll to translog gen 8]
    Roll --> Trim[trim gens now in commit + below global checkpoint]
    subgraph "after flush"
      Seg2[committed segments @ gen 8] --> TL8["translog gen 8 only"]
    end

Flushes happen automatically when the translog grows past index.translog.flush_threshold_size (default 512mb), or on an explicit POST /index/_flush. This is why a write-heavy index periodically flushes without you asking.

Note: Refresh, flush, and merge are three different operations and are easy to confuse. Refresh = visibility (new searcher). Flush = durability milestone (Lucene commit + translog roll). Merge = compaction (fewer segments). The whole comparison is in refresh-flush-merge.md.

Recovery from translog: how a crash is survived

On restart (or on local recovery), the engine opens the last Lucene commit, then replays the translog operations that came after that commit:

grep -n "recoverFromTranslog\|openEngineAndRecoverFromTranslog\|recoverFromTranslogInternal\|Snapshot\|applyTranslogOperation" \
  server/src/main/java/org/opensearch/index/engine/InternalEngine.java \
  server/src/main/java/org/opensearch/index/shard/IndexShard.java | head

sequenceDiagram
    participant N as Node restart
    participant E as Engine
    participant L as Lucene commit
    participant T as Translog
    N->>E: open shard
    E->>L: load last commit (durable segments)
    E->>T: open translog, snapshot ops AFTER commit's generation
    loop each op > commit
        T-->>E: op (seqNo, primaryTerm)
        E->>E: re-apply (origin=LOCAL_TRANSLOG_RECOVERY, reuse seqNo)
    end
    E->>E: shard now reflects all acked, durable writes

The ops replayed are exactly those acknowledged after the last commit. Because each op carries its original seq no and is applied idempotently, replaying is safe even if some ops were partially applied before the crash. This is the mechanism that makes request durability meaningful: the write was in the fsynced translog, so it is replayed.

Global checkpoint and trimming

The translog cannot be trimmed solely on the basis of "it's in a Lucene commit"; it must also respect the global checkpoint — the highest seq no known to be on all in-sync copies (engine-internals.md, replication.md). Operations at or below the global checkpoint are safe everywhere and recoverable from peers, so they can be trimmed locally; operations above it may still be needed for peer recovery and must be retained.

grep -n "trimUnreferencedReaders\|getMinGenerationForSeqNo\|globalCheckpoint\|minSeqNoToKeep\|getMinTranslogGeneration" \
  server/src/main/java/org/opensearch/index/translog/Translog.java | head

The interplay with recovery is the subject of recovery.md: a sequence-number-based peer recovery copies segments (phase 1) then replays the translog/operations above the global checkpoint (phase 2).

Reading exercise

# 1. Append, snapshot, roll, trim.
grep -n "public Location add\|newSnapshot\|rollGeneration\|trimUnreferencedReaders\|sync(" \
  server/src/main/java/org/opensearch/index/translog/Translog.java

# 2. Durability modes.
grep -n "Durability\|REQUEST\|ASYNC\|ensureSynced\|sync_interval" \
  server/src/main/java/org/opensearch/index/translog/Translog.java \
  server/src/main/java/org/opensearch/index/IndexSettings.java

# 3. Recovery replay.
grep -n "recoverFromTranslog\|applyTranslogOperation\|LOCAL_TRANSLOG_RECOVERY" \
  server/src/main/java/org/opensearch/index/engine/InternalEngine.java

# 4. Tests + live stats.
./gradlew :server:test --tests "org.opensearch.index.translog.TranslogTests"
curl -s "localhost:9200/orders/_stats/translog?pretty"

# 5. Inspect on disk.
find . -path "*translog*" -name "*.tlog" 2>/dev/null | head

Answer:

1. What exactly does it mean for an acknowledged write to be "durable" under the default durability setting? Which fsync provides that guarantee?
2. Contrast request and async durability. With async and a 5s interval, how much acknowledged data can a power loss destroy?
3. What three things does a flush do, and how does it let the translog be trimmed? Why does a busy index flush on its own?
4. Trace a crash recovery: which Lucene artifact is loaded first, and which translog operations are replayed?
5. Why must translog trimming respect the global checkpoint and not just the last Lucene commit?
6. Distinguish refresh, flush, and merge in one sentence each.

Common bugs and symptoms

Validation: prove you understand this

1. Explain, using the translog, why an acked write under default settings survives an immediate crash even though Lucene has not committed.
2. Draw the request vs async durability sequence and state the precise data-loss window of each.
3. Explain what a flush does to (a) Lucene, (b) the translog generation, and (c) old translog files, and how the global checkpoint constrains step (c).
4. Walk crash recovery end to end: last commit loaded, ops replayed, why replay is idempotent.
5. Explain why translog trimming respects the global checkpoint, with a scenario where ignoring it would break peer recovery.
6. In one sentence each, contrast refresh, flush, and merge so you never confuse them again.