Refresh, Flush, and Merge

Three background mechanisms turn a stream of indexing operations into a queryable, durable, and space-efficient Lucene index. Refresh makes recently-indexed documents visible to search. Flush makes them durable (a Lucene commit) and lets the translog be trimmed. Merge keeps the segment count bounded so search stays fast. None of them are the same thing, and conflating them is the single most common conceptual error new OpenSearch contributors make.

This chapter pins each mechanism to its classes, its REST surface, its settings, and the failure modes it produces. It builds directly on Engine Internals (which owns the IndexWriter) and The Translog (durability before commit), and it feeds Search Execution (which consumes the searcher refresh produces).

After this chapter you can:

• Explain, without hand-waving, why a document you just indexed isn't found by the next search, and what ?refresh=wait_for actually does.
• Trace _refresh, _flush, and _forcemerge from REST handler to Engine.
• Read _cat/segments and _stats and reason about whether a shard is over-segmented, refresh-bound, or merge-starved.
• Tune index.refresh_interval, merge throttling, and translog flush thresholds and predict the trade-off you just made.

The three operations at a glance

Note: Visibility and durability are orthogonal. A refreshed-but-unflushed document is searchable but lives only in the translog + an in-memory segment; a crash before flush recovers it by replaying the translog. A flushed document is on disk in a committed segment even if never refreshed (a flush implies a refresh of the committed point, but you should not rely on flush as your visibility mechanism).

Find the engine that owns all three:

find server/src/main/java/org/opensearch/index/engine -name "*.java"
grep -n "public void refresh\|public void flush\|void forceMerge\|maybeMerge" \
  server/src/main/java/org/opensearch/index/engine/InternalEngine.java

Refresh — the visibility boundary

Indexed documents land in Lucene's in-memory buffer via IndexWriter.addDocument/updateDocument (see Engine Internals). They are not searchable until a new near-real-time (NRT) DirectoryReader is opened over the writer's current state. That reader-open is a refresh.

The reader is managed by Lucene's ReferenceManager — specifically an OpenSearchReaderManager (OpenSearch's subclass of Lucene SearcherManager) that the engine wraps in an OpenSearchReferenceManager/SearcherManager. The searcher you acquire for a query (IndexShard.acquireSearcher) is a reference into the most recently refreshed reader.

grep -rn "ReaderManager\|SearcherManager\|acquireSearcher" \
  server/src/main/java/org/opensearch/index/engine/InternalEngine.java
grep -rn "class.*ReaderManager" server/src/main/java/org/opensearch/index/engine/

Who triggers a refresh

grep -rn "refresh_interval\|REFRESH_INTERVAL_SETTING" \
  server/src/main/java/org/opensearch/index/IndexSettings.java
grep -rn "search.idle\|searchIdle\|scheduledRefresh" \
  server/src/main/java/org/opensearch/index/shard/IndexShard.java

wait_for is not a forced refresh

?refresh=wait_for registers a listener (RefreshListeners) that completes the request when the operation's location becomes visible via the next refresh — it does not force one. This is the cheap way to get read-your-writes without hammering refresh.

grep -rn "RefreshListeners\|addOrNotify\|wait_for" \
  server/src/main/java/org/opensearch/index/shard/

Increasing throughput: the classic tuning move

For bulk-load workloads, set index.refresh_interval to 30s (or -1 during a backfill, restored afterward). Fewer refreshes mean fewer tiny segments, less merge pressure, and more CPU for indexing. The cost is higher search latency to freshness.

curl -s -XPUT localhost:9200/idx/_settings -H 'content-type: application/json' \
  -d '{"index":{"refresh_interval":"30s"}}'

Flush — durability and translog trim

A flush issues a Lucene IndexWriter.commit(): in-memory segments are written and fsync'd, a new commit point (segments_N) is created, and — crucially — the translog can be trimmed because every operation up to that commit is now recoverable from the Lucene index itself without replay.

sequenceDiagram
    participant W as Write op
    participant TL as Translog
    participant IW as Lucene IndexWriter
    participant CP as Commit point
    W->>TL: Translog.add(op)  (durable on fsync per index.translog.durability)
    W->>IW: addDocument / updateDocument (in-memory)
    Note over IW: refresh -> visible, still uncommitted
    Note over IW,CP: flush -> IndexWriter.commit() -> segments_N
    CP-->>TL: translog generation rolled & trimmed up to commit

When a flush happens

grep -rn "flush_threshold_size\|FLUSH_THRESHOLD\|shouldPeriodicallyFlush" \
  server/src/main/java/org/opensearch/index/ server/src/main/java/org/opensearch/index/engine/InternalEngine.java

Warning: Do not script periodic _flush calls as a "safety" measure. The engine decides flush timing far better than a cron job; manual flushing usually just creates extra small commits and fights the translog heuristics. Forced flush has legitimate uses (before a clean shutdown, for tests), but "flush every minute to be safe" is an anti-pattern.

The durability of writes before flush comes from the translog and its index.translog.durability setting (request vs async) — that is the Translog deep dive's territory; flush is where the translog gets to forget.

Merge — keeping segment count bounded

Every refresh can create a new segment. Every segment is an independent mini-index that search must visit. Left unchecked, segment count explodes and search degrades (more files, more terms dictionaries, more per-segment overhead, deleted docs never reclaimed). Merge combines smaller segments into larger ones and is how deletes/updates actually free space (Lucene deletes are tombstones until the segment is merged away).

The policy and the scheduler

grep -rn "TieredMergePolicy\|MergeScheduler\|ConcurrentMergeScheduler" \
  server/src/main/java/org/opensearch/index/
find server/src/main/java/org/opensearch/index -name "*Merge*"

Key merge settings

grep -rn "max_merged_segment\|segments_per_tier\|floor_segment\|max_thread_count" \
  server/src/main/java/org/opensearch/index/MergePolicyConfig.java \
  server/src/main/java/org/opensearch/index/MergeSchedulerConfig.java

force_merge — the loaded gun

POST /idx/_forcemerge?max_num_segments=1 forces merging down to N segments. It is the right tool for read-only / cold indices (e.g., time-based indices no longer being written) to collapse them to one big segment and purge deletes. It is the wrong tool for actively-written indices:

Warning: force_merge to 1 segment on a live index produces enormous segments that the normal TieredMergePolicy will never merge again (they exceed max_merged_segment), permanently distorting the tier structure and stranding deleted docs. Only force-merge indices you have stopped writing to. Forcing is also a blocking, I/O-heavy operation — never do it on a hot cluster during peak load.

grep -rn "forcemerge\|forceMerge\|max_num_segments\|only_expunge_deletes" \
  server/src/main/java/org/opensearch/action/admin/indices/forcemerge/

How they interact

flowchart TD
    I[Index op] --> B[Lucene in-memory buffer + Translog.add]
    B -->|refresh interval / explicit| R[Open new DirectoryReader]
    R --> V[Docs visible to search]
    B -->|buffer/segment created| S[New small segment]
    S -->|TieredMergePolicy selects| M[Background merge -> bigger segment]
    M -->|reclaims deletes| S
    B -->|translog > flush_threshold| F[IndexWriter.commit = flush]
    F --> TT[Translog trimmed]
    F --> CP[New commit point segments_N]
    M -.merged segments committed on.-> F

The feedback loops worth internalizing:

1. Refresh ↑ → segment count ↑ → merge pressure ↑. A 200ms refresh_interval on a write-heavy shard buries the merge scheduler.
2. Flush trims translog, so a stuck flush (e.g., I/O saturation from merges) makes the translog grow unbounded → flush_threshold_size exceeded → forced flush competes with the merges that caused it.
3. Segment replication (see Replication) changes who runs merges: replicas copy segments from the primary instead of indexing independently, so merge cost is paid once on the primary. Document replication pays it on every copy.

Observability: read the shard like an X-ray

# Per-segment view: size, doc count, deletes, committed/searchable flags
curl -s 'localhost:9200/_cat/segments/idx?v&h=index,shard,prirep,segment,docs.count,docs.deleted,size,committed,searchable'

# Aggregate refresh/flush/merge/segment stats for an index
curl -s 'localhost:9200/idx/_stats/refresh,flush,merge,segments?pretty'

Map columns to meaning:

Reading exercise

# 1. The searcher/reader plumbing the engine uses for refresh
grep -rn "ReferenceManager\|ReaderManager\|maybeRefresh\|maybeRefreshBlocking" \
  server/src/main/java/org/opensearch/index/engine/InternalEngine.java

# 2. Periodic refresh scheduling and search-idle
grep -rn "scheduledRefresh\|searchIdle\|markSearcherAccessed" \
  server/src/main/java/org/opensearch/index/shard/IndexShard.java

# 3. The decision to flush
grep -rn "shouldPeriodicallyFlush\|flush_threshold_size\|translog.*size" \
  server/src/main/java/org/opensearch/index/engine/InternalEngine.java

# 4. Merge wiring
grep -rn "mergePolicy\|mergeScheduler\|maybeMerge\|forceMerge" \
  server/src/main/java/org/opensearch/index/engine/InternalEngine.java

Answer:

1. When you call IndexShard.acquireSearcher, into which reader version do you get a reference — the latest refreshed one or the latest committed one? Where in the engine is that reference obtained?
2. What exactly does ?refresh=wait_for register, and what completes it? Why is it cheaper than ?refresh=true under high write load?
3. Trace POST /idx/_flush: name the transport action, the shard method, and the Lucene call it ultimately makes.
4. A shard has 80 segments and docs.deleted ≈ 30% of docs.count. The index is read-only now. What one command fixes it, and why is it safe here but dangerous on a live index?
5. With segment replication, which node runs the TieredMergePolicy merges, and how do the merged segments reach the replicas? (Cross-check Replication.)
6. Set refresh_interval to -1. Index 1000 docs. Search returns 0 hits. Name two distinct ways to make them visible and the durability difference between them.

Common bugs and symptoms

Validation: prove you understand this

1. Draw the timeline of a single document from index call to "durable on disk and visible to search," labeling exactly where refresh, translog fsync, and flush each occur. State which events are required for visibility vs durability.
2. Explain in two sentences why a flush implies the durability of a refresh's contents but a refresh does not imply durability.
3. Given _cat/segments showing 120 segments on a 2GB shard with 1% deletes on a write-heavy index, decide: is the problem refresh, merge, or neither? Justify from the numbers and name the setting you'd change.
4. Write the curl to disable periodic refresh during a backfill and the one to restore it to 1s afterward. Explain what wait_for would do during the backfill if a client needs read-your-writes on one specific doc.
5. From memory, name the Lucene class for merge policy, the Lucene class for merge scheduling, and the OpenSearch config class that maps index.merge.* onto them.
6. Explain why force_merge?max_num_segments=1 is correct for a closed time-series index but corrupts the tiering of a live index — cite max_merged_segment.