IndexWriter and Merge Internals

IndexWriter is the write path. It takes documents, buffers them in RAM, flushes them into immutable segments, tracks deletes as side bitsets, merges small segments into bigger ones, and — when told to commit() — writes the segments_N that makes everything durable. OpenSearch's InternalEngine owns one IndexWriter per shard and drives every one of these operations. This chapter is the inside of those calls: the per-thread buffers (DWPT), the flush-to-segment pipeline, the merge policy and scheduler, how deletes and updates work without mutating segments, and the precise difference between flush() and commit().

This is the Lucene engine beneath OpenSearch's Refresh, Flush, and Merge: that chapter mapped OpenSearch refresh→openIfChanged, flush→commit(), merge→MergePolicy. Here you learn what each of those Lucene operations actually does to RAM, disk, and readers.

After this chapter you can: explain DWPT and why indexing scales with threads; trace the buffer→flush→segment pipeline; describe TieredMergePolicy + ConcurrentMergeScheduler; explain deletes/updates via liveDocs; state the commit() vs flush() distinction precisely; and explain near-real-time readers.

IndexWriter and DocumentsWriterPerThread (DWPT)

A naive writer would serialize all indexing through one buffer and one lock — disastrous under concurrent writes. Lucene instead gives each indexing thread its own in-RAM buffer: a DocumentsWriterPerThread (DWPT). A thread that calls addDocument grabs a DWPT (from a pool), analyzes and buffers the doc into its private structures, and never contends with other indexing threads on the hot path. The shared DocumentsWriter coordinates the pool; IndexWriter sits above it.

flowchart TD
    T1["thread 1 addDocument"] --> D1["DWPT #1 (private RAM buffer)"]
    T2["thread 2 addDocument"] --> D2["DWPT #2 (private RAM buffer)"]
    T3["thread 3 addDocument"] --> D3["DWPT #3 (private RAM buffer)"]
    D1 -->|flush| S1["segment _a"]
    D2 -->|flush| S2["segment _b"]
    D3 -->|flush| S3["segment _c"]
    S1 & S2 & S3 --> R["DirectoryReader sees all segments"]

grep -rn "class IndexWriter\b\|class DocumentsWriter\b\|class DocumentsWriterPerThread" \
  lucene/core/src/java/org/apache/lucene/index/
grep -rn "addDocument\|updateDocument\|getAndLock\|flushControl" \
  lucene/core/src/java/org/apache/lucene/index/DocumentsWriter.java | head

Each DWPT, when it fills up (RAM budget, doc count) or on a flush request, is flushed independently into its own segment. So one logical "flush" can produce several segments — one per active DWPT. This is the source of the parallelism in Lucene indexing: more threads → more DWPTs → more concurrent buffering and flushing. The OpenSearch indexing thread pool sizing (thread pools) interacts directly with how many DWPTs are in flight.

Note: "Indexing buffer" in OpenSearch (indices.memory.index_buffer_size) is the RAM budget shared across DWPTs of all shards on the node. When it fills, Lucene flushes the largest DWPTs to reclaim RAM — which is why a node under heavy indexing produces many small segments (and then leans on merges to clean up).

The buffer → flush → segment pipeline

What a DWPT does on flush, in order:

1. Invert the buffered docs: build the in-memory inverted index (terms → postings), doc values, points, stored fields, vectors — all the per-field structures.
2. Encode them through the active Codec's sub-formats into segment files (.tim/.doc, .dvd, .kdd, .fdt, etc. — see Segments and Codecs).
3. Write the .si and register the new SegmentCommitInfo with the writer's in-memory SegmentInfos (pending — not yet a commit point).
4. The segment is now flushed: it exists on disk and a refresh can open a reader over it. It is not yet committed (no new segments_N).

grep -rn "flush(\|doFlush\|class FlushByRamOrCountsPolicy\|getFlushingBytes" \
  lucene/core/src/java/org/apache/lucene/index/ | head

This is the crux of near-real-time: a flushed segment is visible (after a reader opens) but not durable across a crash via the index alone — Lucene itself only guarantees durability at commit(). OpenSearch closes that gap with the translog: the op is in the translog before it's acknowledged, so an unflushed-but-refreshed doc survives a crash via translog replay. Visibility (refresh) and durability (commit + translog) are independent — the recurring theme of this whole subsystem.

Deletes and updates: liveDocs, not mutation

Segments are immutable, so a delete cannot remove bytes. Instead Lucene keeps a per-segment liveDocs bitset: bit i is set if doc i is still alive. A delete clears the bit; queries skip dead docs by consulting liveDocs. The dead doc's bytes (postings, doc values, stored fields) stay on disk until a merge rewrites the segment and physically drops them.

grep -rn "liveDocs\|class .*LiveDocs\|applyDeletes\|deleteDocument" \
  lucene/core/src/java/org/apache/lucene/index/ | head

So an "update" in OpenSearch is, at the Lucene level, delete-by-_id-term + add. The old document is tombstoned in its old segment; the new one lives in a fresh segment. A high-churn index (constant updates) accumulates dead-doc debt — the percentage of deleted docs is a key health metric (_cat/segments shows deleted.docs), because that debt is only paid down by merges.

Merges: MergePolicy and MergeScheduler

Without merging, segment count and dead-doc debt grow forever: more segments means more per-query overhead (every query visits every segment), and tombstoned docs waste disk and slow scans. Merging combines several segments into one new segment, dropping deleted docs in the process. Two pluggable pieces decide which and when:

TieredMergePolicy (unlike the old log-based policies) does not require adjacent segments — it picks the best-value set anywhere, favoring segments with many deletes (to reclaim them) and avoiding repeatedly rewriting huge segments. Key knobs: maxMergeAtOnce, segmentsPerTier, maxMergedSegmentMB, deletesPctAllowed.

grep -rn "class TieredMergePolicy\|segmentsPerTier\|maxMergedSegment\|deletesPctAllowed\|class MergePolicy\b" \
  lucene/core/src/java/org/apache/lucene/index/TieredMergePolicy.java
grep -rn "class ConcurrentMergeScheduler\|maxThreadCount\|maxMergeCount\|doMerge" \
  lucene/core/src/java/org/apache/lucene/index/ConcurrentMergeScheduler.java

OpenSearch wires these with OpenSearchConcurrentMergeScheduler and MergePolicyConfig/MergePolicyProvider, exposing index.merge.* settings and merge stats. A _forcemerge to one segment is the manual override (great before a shard goes read-only; dangerous on a live write-heavy index — see the warning in refresh-flush-merge.md).

grep -rn "TieredMergePolicy\|ConcurrentMergeScheduler\|MergePolicyConfig\|index.merge" \
  server/src/main/java/org/opensearch/index/ | head

flowchart TD
    Buf["DWPT buffers"] -->|flush| Small["many small segments"]
    Small --> MP["TieredMergePolicy: pick best-value set"]
    MP --> CMS["ConcurrentMergeScheduler: run on bg thread, throttle I/O"]
    CMS --> Merged["one larger segment (dead docs dropped)"]
    Merged --> MP

commit() vs flush(): the precise distinction

This trips people up; nail it. The two words mean specific, different things in Lucene — and OpenSearch overloads "flush" to mean Lucene commit(), which adds to the confusion.

The mental model:

• Lucene flush = "get my RAM buffers onto disk as segments" (visibility-ish, not durability).
• Lucene commit = "make the current set of segments the official, crash-safe index by writing segments_N."
• OpenSearch refresh ≈ Lucene flush + open reader = visible.
• OpenSearch flush = Lucene commit + translog roll = durable, translog trimmed.

grep -rn "public long commit\|void flush(\|prepareCommit\|fsync\|SegmentInfos" \
  lucene/core/src/java/org/apache/lucene/index/IndexWriter.java | head
# OpenSearch side: refresh vs flush:
grep -rn "openIfChanged\|commitIndexWriter\|indexWriter.commit\|refresh(" \
  server/src/main/java/org/opensearch/index/engine/InternalEngine.java | head

Warning: A commit is expensive (fsync everything + a new commit point). This is exactly why OpenSearch does not commit on every write — it refreshes for visibility (cheap, no fsync) and relies on the translog for durability between commits, committing only on flush. Do not script periodic _flush; you fight the heuristics and create commit churn.

Near-real-time readers

Reads go through a DirectoryReader, a point-in-time view of a set of segments. To see new docs without a full commit, Lucene opens a near-real-time (NRT) reader directly from the IndexWriter:

• DirectoryReader.open(IndexWriter) — open an NRT reader that includes the writer's currently-flushed (uncommitted) segments.
• DirectoryReader.openIfChanged(oldReader, writer) — cheaply get a new reader reflecting changes since oldReader, reusing unchanged segment readers (only new/changed segments get new readers — this is what makes refresh cheap).

grep -rn "static DirectoryReader open\|openIfChanged\|class StandardDirectoryReader\|class DirectoryReader" \
  lucene/core/src/java/org/apache/lucene/index/DirectoryReader.java | head

OpenSearch wraps this in a SearcherManager (a Lucene ReferenceManager<DirectoryReader>): a refresh calls openIfChanged, swaps in the new reader, and reference-counts the old one until in-flight searches release it. That is the entire mechanism by which a just-indexed doc becomes searchable — covered from the OpenSearch side in engine internals.

The segment lifecycle, end to end

stateDiagram-v2
    [*] --> Buffered: addDocument -> DWPT RAM buffer
    Buffered --> Flushed: flush (RAM full / refresh) -> new immutable segment on disk
    Flushed --> Visible: DirectoryReader.openIfChanged (refresh) opens reader
    Flushed --> Committed: IndexWriter.commit() writes segments_N (durable)
    Visible --> Merging: TieredMergePolicy selects + ConcurrentMergeScheduler runs
    Committed --> Merging
    Merging --> Merged: dead docs dropped, segments combined
    Merged --> Committed: next commit records the merged set
    Committed --> [*]: segment dropped when superseded by a merge

A doc travels: buffered in a DWPT → flushed to a small segment → made visible by a refresh → made durable by a commit → eventually merged into a larger segment (its old segment dropped). Deletes ride along as liveDocs bits that merges finally reclaim. Everything in OpenSearch's write path is this lifecycle plus versioning, seq numbers, and the translog layered on top.

Reading exercise

# 1. The writer and DWPT.
grep -rn "class IndexWriter\b\|class DocumentsWriterPerThread\|class DocumentsWriter" \
  lucene/core/src/java/org/apache/lucene/index/

# 2. Flush pipeline.
grep -rn "doFlush\|flush(\|FlushByRamOrCountsPolicy" lucene/core/src/java/org/apache/lucene/index/ | head

# 3. Deletes / liveDocs.
grep -rn "liveDocs\|applyDeletes\|deleteDocuments" lucene/core/src/java/org/apache/lucene/index/ | head

# 4. Merge policy + scheduler.
grep -rn "class TieredMergePolicy\|class ConcurrentMergeScheduler\|segmentsPerTier\|maxMergedSegment" \
  lucene/core/src/java/org/apache/lucene/index/

# 5. commit vs flush vs NRT.
grep -rn "public long commit\|openIfChanged\|static DirectoryReader open" \
  lucene/core/src/java/org/apache/lucene/index/IndexWriter.java \
  lucene/core/src/java/org/apache/lucene/index/DirectoryReader.java

# 6. OpenSearch wiring + live merge stats.
grep -rn "indexWriter.commit\|openIfChanged\|ConcurrentMergeScheduler" \
  server/src/main/java/org/opensearch/index/engine/InternalEngine.java
curl -s 'localhost:9200/_cat/segments/my-index?v&h=segment,docs.count,docs.deleted,size'
curl -s 'localhost:9200/my-index/_stats/merge?pretty'

Answer:

1. What is a DWPT and why does giving each indexing thread its own buffer make indexing scale? What produces multiple segments from one flush?
2. List the steps a DWPT performs on flush, and say at which step the data becomes visible vs durable.
3. How do deletes and updates work given immutable segments? What reclaims the dead bytes, and what metric tracks the debt?
4. Distinguish MergePolicy from MergeScheduler, name the defaults, and give two TieredMergePolicy knobs and what they do.
5. State the exact difference between Lucene flush(), Lucene commit(), OpenSearch refresh, and OpenSearch flush — in one row each.
6. Explain how openIfChanged makes a refresh cheap and what SearcherManager adds on top.

Common bugs and symptoms

Validation: prove you understand this

1. Draw the multi-DWPT model and explain why it's the source of Lucene indexing parallelism and of "many small segments."
2. Walk the buffer→flush→segment pipeline and mark the visibility point and the durability point.
3. Explain deletes/updates via liveDocs and .liv, and what physically reclaims dead docs.
4. Compare TieredMergePolicy and ConcurrentMergeScheduler and name two tuning knobs on each.
5. Produce the four-row table distinguishing Lucene flush, Lucene commit, OpenSearch refresh, and OpenSearch flush, with durability/visibility/segments_N for each.
6. Draw the full segment lifecycle state machine and place a single document on it from addDocument to "dropped after merge."