Lab 6.2: IndexShard, InternalEngine, and the Translog

Background

Lab 6.1 mapped the write path. This lab makes you operate it. You will read the three classes that own a shard's storage — IndexShard, InternalEngine, and Translog — and then drive their behavior from the outside with curl, watching the three things that confuse every newcomer: visibility (refresh), durability (flush, translog), and the translog's role in recovery.

The thesis of the lab is one sentence: a document is durable long before it is searchable, and searchable long before its Lucene commit is durable. Refresh, flush, and the translog are the three mechanisms that make that sentence true, and you will see each one move under a curl.

Note: Keep the engine internals, translog, and refresh/flush/merge deep dives open. This lab is the hands-on counterpart; those chapters are the theory. The index-shard lifecycle deep dive explains the states a shard moves through around these operations.

Why This Lab Matters for Contributors

• "Why don't my docs show up immediately?" and "did I lose data on that crash?" are the two most common user questions. The answers are refresh and the translog — and you should be able to explain both with a curl demo.
• Most engine/storage bugs are about timing: a refresh that didn't happen, a translog that wasn't trimmed, a durability setting misread. Operating the engine builds the intuition to spot them.
• Tuning refresh_interval and translog.durability is a frequent production decision and a frequent source of issues; you must understand the trade-offs to review such changes.
• The translog-replay model is the foundation of recovery; you can't reason about recovery bugs without it.

Prerequisites

• A running node from source: ./gradlew run (REST on :9200).
• Lab 6.1 completed — you know the write path.
• jq is handy but optional (commands below also work with ?pretty).

curl -s 'localhost:9200/_cluster/health?pretty' | head    # node is up

Step-by-Step Tasks

Step 1: Read the Engine contract before the implementation

InternalEngine is large. Read the abstract Engine first — it is the contract, and it names every operation the storage engine supports:

find server/src/main/java -name "Engine.java" -path "*engine*"
grep -n "public abstract\|abstract IndexResult index\|abstract DeleteResult delete\|abstract void refresh\|abstract void flush\|abstract GetResult get\|acquireSearcher\|class Index\|class Delete\|class Get\|class Searcher" \
  server/src/main/java/org/opensearch/index/engine/Engine.java | head -40

Record the contract surface: index, delete, get, refresh, flush, forceMerge, acquireSearcher, plus the Engine.Index/Engine.Delete/Engine.Get operation value types. Every storage engine (including segment-replication variants) implements this.

Step 2: Read the visibility/durability machinery in InternalEngine

Now the implementation, focused on the three mechanisms:

# Visibility: the searcher manager and refresh.
grep -n "ExternalReaderManager\|OpenSearchReaderManager\|SearcherManager\|refresh(\|maybeRefresh\|reopen" \
  server/src/main/java/org/opensearch/index/engine/InternalEngine.java | head

# Durability: commit + translog interplay on flush.
grep -n "public void flush\|commitIndexWriter\|indexWriter.commit\|translog.trimUnreferenced\|rollGeneration\|ensureTranslogSynced\|translog.sync" \
  server/src/main/java/org/opensearch/index/engine/InternalEngine.java | head

Record:

• Refresh opens a new reader from the in-memory + committed segments → buffered docs become searchable. It does not fsync and does not trim the translog.
• Flush = IndexWriter.commit() (fsync segments, write a new commit point) then roll/trim the translog (the committed ops no longer need replaying). This is where durability and translog size meet.

Step 3: Read the translog itself

find server/src/main/java -name "Translog.java"
grep -n "public Location add\|public void sync\|class Durability\|REQUEST\|ASYNC\|rollGeneration\|trimUnreferencedReaders\|newSnapshot\|recoverFromTranslog\|getMinFileGeneration" \
  server/src/main/java/org/opensearch/index/translog/Translog.java | head -30

Record the two durability modes and what sync() does:

The translog add always appends in memory; sync is what fsyncs it. On recovery, recoverFromTranslog replays operations after the last commit's local checkpoint.

Step 4: Observe visibility — refresh changes what search sees

Create an index with refresh disabled so you control visibility manually:

curl -s -XPUT 'localhost:9200/lab62?pretty' -H 'Content-Type: application/json' -d '{
  "settings": { "index": { "number_of_shards": 1, "number_of_replicas": 0, "refresh_interval": "-1" } }
}'

# Index a doc WITHOUT refreshing.
curl -s -XPOST 'localhost:9200/lab62/_doc?pretty' -H 'Content-Type: application/json' -d '{"t":"invisible"}'

# Search immediately — expect 0 hits, because no refresh has opened a new searcher.
curl -s 'localhost:9200/lab62/_search?pretty' | grep '"value"'
#   "value" : 0,

# Force a refresh, then search again — now 1 hit.
curl -s -XPOST 'localhost:9200/lab62/_refresh?pretty' >/dev/null
curl -s 'localhost:9200/lab62/_search?pretty' | grep '"value"'
#   "value" : 1,

You just watched visibility turn on. The doc existed (and was durable in the translog) the whole time — refresh only made it searchable. Confirm the doc was retrievable by primary key the whole time (GET reads the live version map / translog, not the searcher):

# Index another with refresh still off, then GET it by id — works even without a refresh.
ID=$(curl -s -XPOST 'localhost:9200/lab62/_doc?pretty' -H 'Content-Type: application/json' \
  -d '{"t":"gettable"}' | sed -n 's/.*"_id" : "\(.*\)",/\1/p')
curl -s "localhost:9200/lab62/_doc/${ID}?pretty" | grep '"found"'
#   "found" : true,     <- visible to GET-by-id before any refresh

Note: This GET-vs-search asymmetry is real and important: GET-by-id can read the in-flight version map, so a just-indexed doc is retrievable by id immediately, but search (which uses the Lucene searcher) only sees it after a refresh. Many "why is my doc missing?" issues are this asymmetry.

Step 5: Observe the refresh interval as a knob

# Turn refresh back on at a slow 30s and watch search lag, then speed it up.
curl -s -XPUT 'localhost:9200/lab62/_settings?pretty' -H 'Content-Type: application/json' \
  -d '{"index":{"refresh_interval":"30s"}}' >/dev/null
curl -s -XPOST 'localhost:9200/lab62/_doc?pretty' -H 'Content-Type: application/json' -d '{"t":"slowvis"}' >/dev/null
curl -s 'localhost:9200/lab62/_search?pretty' | grep '"value"'   # may still be the old count for up to 30s

# A write with refresh=wait_for blocks until the next refresh makes it visible (no busy-wait).
curl -s -XPOST 'localhost:9200/lab62/_doc?refresh=wait_for&pretty' -H 'Content-Type: application/json' \
  -d '{"t":"waited"}' >/dev/null
curl -s 'localhost:9200/lab62/_search?pretty' | grep '"value"'   # now includes the waited doc

Record the trade-off: tiny refresh_interval = near-real-time search but many small segments and merge pressure; large = fewer/larger segments, less overhead, staler search. The default 1s is a balance.

Step 6: Observe durability — flush, the translog, and _stats

Watch the translog grow with writes and shrink on flush. _stats?translog exposes the live numbers:

# Index a batch, then look at translog size (operations not yet committed to a Lucene commit point).
for i in $(seq 1 50); do
  curl -s -XPOST 'localhost:9200/lab62/_doc?pretty' -H 'Content-Type: application/json' \
    -d "{\"t\":\"d$i\"}" >/dev/null
done
curl -s 'localhost:9200/lab62/_stats/translog?pretty' | grep -E '"operations"|"uncommitted_operations"|"size_in_bytes"'

# Flush: IndexWriter.commit() + trim the now-redundant translog. uncommitted_operations drops to ~0.
curl -s -XPOST 'localhost:9200/lab62/_flush?pretty' >/dev/null
curl -s 'localhost:9200/lab62/_stats/translog?pretty' | grep -E '"uncommitted_operations"'
#   "uncommitted_operations" : 0,

Record: flush moved durability from the translog into a Lucene commit and trimmed the translog. Before the flush, durability for those ops lived only in the translog; after, it lives in the committed segments and the translog generation can be discarded.

Step 7: See segments appear and merge

# Each refresh/flush can create a segment. List them.
curl -s 'localhost:9200/_cat/segments/lab62?v'
#   index  shard prirep ip        segment generation docs.count docs.deleted size  committed searchable ...

# Force-merge down to 1 segment (housekeeping; do NOT do this routinely in prod).
curl -s -XPOST 'localhost:9200/lab62/_forcemerge?max_num_segments=1&pretty' >/dev/null
curl -s 'localhost:9200/_cat/segments/lab62?v'   # fewer rows now

Record: refresh/flush create segments; merge (MergePolicy/MergeScheduler, forced here for the demo) combines them and reclaims deleted docs. In production merges run in the background — see the refresh/flush/merge deep dive.

Step 8: Reason about translog replay (the recovery model)

You cannot easily crash ./gradlew run mid-write safely, but you can reason precisely about what would be replayed. Inspect the durability setting and the uncommitted operation count — together they tell you the recovery contract:

curl -s 'localhost:9200/lab62/_settings?pretty' | grep -A1 'translog'
curl -s 'localhost:9200/lab62/_stats/translog?pretty' | grep -E '"uncommitted_operations"'

Answer in your reading log:

• With durability: request (default), every acknowledged write is fsync'd to the translog before the response returns. On a crash, all uncommitted_operations are replayed from the translog after the last Lucene commit's local checkpoint → no acknowledged write is lost.
• With durability: async, writes acknowledged within the last sync_interval may not be fsync'd → a crash can lose them. The translog still replays whatever was synced.
• A flush you just ran trims the translog, so a crash right after the flush replays almost nothing — recovery is fast because the committed segments already hold those ops.

Map this back to the source: recoverFromTranslog in InternalEngine/Translog and the local/global checkpoint plumbing (see the Level 6 overview on checkpoints).

grep -rn "recoverFromTranslog\|newSnapshotFromGen\|getMinSeqNoToKeep\|trimOperations\|recoverFromTranslogInternal" \
  server/src/main/java/org/opensearch/index/engine/InternalEngine.java \
  server/src/main/java/org/opensearch/index/translog/Translog.java | head

Step 9: Toggle translog.durability and observe (read-mostly)

# Switch to async with a long sync interval; index; note that ops are not fsync'd per request.
curl -s -XPUT 'localhost:9200/lab62/_settings?pretty' -H 'Content-Type: application/json' -d '{
  "index": { "translog": { "durability": "async", "sync_interval": "10s" } }
}'
curl -s 'localhost:9200/lab62/_settings?pretty' | grep -A3 translog
# Index a few docs; they are durable in *memory* but fsync only happens every 10s now.
# Switch back to the safe default before doing anything you care about:
curl -s -XPUT 'localhost:9200/lab62/_settings?pretty' -H 'Content-Type: application/json' \
  -d '{"index":{"translog":{"durability":"request"}}}' >/dev/null

Warning: durability: async trades data safety for write throughput. It is a deliberate, documented choice for specific ingest workloads — never a default you flip casually. Reviewing a PR that changes a default durability is a red flag worth a careful read.

Implementation Requirements / Deliverables

• A reading note distinguishing, in your own words, refresh (visibility), flush (Lucene commit + translog trim), and merge (segment housekeeping), each tied to its InternalEngine/ Translog method.
• The Step 4 demo output proving a doc is durable/GET-able before it is searchable.
• The Step 6 _stats/translog before/after a flush, showing uncommitted_operations drop to ~0.
• The Step 7 _cat/segments before/after a force-merge.
• A written translog-replay answer (Step 8) for both request and async durability.
• Source pointers (grep hits) for refresh, flush, Translog.add/sync, and recoverFromTranslog.

Troubleshooting

Expected Output

# Step 4
"value" : 0,        # before refresh
"value" : 1,        # after refresh
"found" : true,     # GET-by-id before any refresh

# Step 6
"uncommitted_operations" : 50,   # before flush
"uncommitted_operations" : 0,    # after flush

# Step 7
# _cat/segments shows several rows, then fewer after force-merge to 1 segment.

Stretch Goals

1. Measure refresh cost. Index 100k docs at refresh_interval: 1s vs 30s vs -1; compare _cat/segments counts and merge activity (_nodes/stats/indices/merges). Explain the trade-off with numbers.
2. Watch a merge happen for real. Index, delete a chunk of docs, refresh, and watch docs.deleted in _cat/segments shrink as background merges reclaim them — without force-merge.
3. Trace a recovery. Restart the node (./gradlew run again on the same data dir if configured) and TRACE org.opensearch.index.translog and org.opensearch.indices.recovery to watch translog replay on startup. Cross-reference the recovery deep dive.
4. Find the flush triggers. grep for flush_threshold_size / shouldPeriodicallyFlush in IndexShard/InternalEngine and explain what automatically triggers a flush (translog size/age, merges, shard close) without an explicit _flush.

Validation / Self-check

1. Explain the sentence "a document is durable before it is searchable, and searchable before its Lucene commit is durable" using refresh, flush, and the translog.
2. Why was your doc GET-able by id but not found by search before a refresh? Which component does each read from?
3. What exactly does flush do that refresh does not? Which two side effects make uncommitted_operations drop to 0?
4. Contrast translog.durability: request vs async. For each, what is lost in a crash, and why?
5. On recovery, what gets replayed from the translog, and from which point? Name the method.
6. You set refresh_interval: -1, then 30s, then used refresh=wait_for. Describe the visibility behavior in each case and the production trade-off of small vs large intervals.
7. Where in InternalEngine/Translog do refresh, flush, add, sync, and recoverFromTranslog live? Paste the grep lines.

When you can drive visibility and durability from curl and explain every transition from the source, build something on top of the engine in Lab 6.3: Build It — A Custom Analyzer.