Remote-Backed Storage and Durability

In the classic OpenSearch durability model, "your data is safe" means "there is more than one copy on more than one node's local disk." Durability is replica count. That coupling — examined in Sharding, Scaling & Reader/Writer Separation — is exactly what makes independent read/write scaling impossible, and it has a second cost: a node failure can still lose unflushed data if you were not careful, and recovery means copying segments node-to-node.

Remote-backed storage moves durability off the cluster entirely and onto a remote object store (S3 or equivalent). The writer uploads its Lucene segments and its translog to that remote backend; the remote copy is now the source of truth for durability. A node can die, a shard can move, a whole cluster can be rebuilt — and the data is recovered from the object store, not from a surviving replica. This single change is what makes search replicas, scale-to-zero, and reader/writer separation possible.

After this chapter you should be able to: explain how remote store decouples durability from replica count; describe the segment-and-translog upload path and how it interacts with segment replication; trace recovery from the remote store; explain how remote store underpins reader/writer separation; and reason about the multi-writer safety problem.

Prerequisites. This builds directly on the translog, refresh/flush/merge, and replication (segment replication especially). It is the storage foundation under sharding & scaling.

The idea: durability is a remote object, not a replica

flowchart TD
    subgraph Node["writer node"]
      IW["IndexWriter writes segments locally"] --> Local[("local disk (cache)")]
      TL["Translog appends ops"] --> Local
    end
    Local -->|"upload on refresh/flush"| RS[("remote store: segments")]
    TL -->|"upload on write"| RST[("remote store: translog")]
    RS --> Recover["recovery / new copy: download from remote"]
    RST --> Recover

The local disk becomes, in effect, a cache of what is durably stored in the remote object store. Two upload streams keep the remote copy current:

1. Segments are uploaded after they are produced (on refresh/flush, when Lucene writes new immutable segment files).
2. The translog is uploaded as operations arrive, so the window between the last segment upload and "now" is also durable remotely.

With both streams, the remote store always holds enough to reconstruct the shard to its last acknowledged write — without any local replica. That is the whole point: durability no longer requires a second node holding the data.

cd ~/src/OpenSearch
# The remote-store directory + upload machinery (names vary by version — grep).
grep -rln "RemoteSegmentStoreDirectory\|RemoteStoreRefreshListener\|RemoteFsTranslog\|RemoteStoreEnums" \
  server/src/main/java/org/opensearch/index/ | head
ls server/src/main/java/org/opensearch/index/store/ 2>/dev/null | grep -i remote
ls server/src/main/java/org/opensearch/index/translog/ 2>/dev/null | grep -i remote

The upload path: segments and translog

Two listeners do the work. Understanding the cadence is the key to reasoning about the durability window and the staleness a reader might see.

flowchart LR
    Op["index op"] --> TLloc["local translog"]
    TLloc --> TLrem["upload to remote translog"]
    Op --> Buf["IndexWriter buffer"]
    Buf -->|refresh| Seg["new segment files"]
    Seg --> Segrem["upload to remote segment store"]
    TLrem --> Durable["acked write is durable remotely"]
    Segrem --> Searchable["state available to search replicas"]

Note: Just like the local NRT model (a doc is durable before it is visible, see engine internals), remote store has its own ordering: the translog upload provides durability of the tail, while the segment upload provides the searchable, replica-consumable state. A reader pulling from the remote store sees data as of the last segment upload, not the last write — which is the staleness knob for search replicas.

Interplay with segment replication

Remote store and segment replication are designed together, and confusing them is a common mistake. Segment replication (replication.md) means replicas copy segments from the primary instead of re-indexing each document. Remote store is where those segments can be copied from.

Pairing them is what lets a search replica exist at all: it is a segment-replication consumer whose source is the remote store rather than a live primary. Do not re-derive segment replication here — read replication.md; just hold that remote store is the storage substrate it can read from.

Recovery from the remote store

When a shard is (re)created — node restart, relocation, a new search replica, a rebuilt cluster — recovery becomes a download rather than a node-to-node copy:

flowchart TD
    Start["shard needs to recover"] --> Meta["read remote segment metadata (latest commit)"]
    Meta --> DL["download missing segment files from remote store"]
    DL --> TLrep["replay remote translog tail past the last commit"]
    TLrep --> Up["shard is up to last acked write"]

This is faster and more flexible than peer recovery because it does not need a healthy source node holding the data — the object store is the source. It is also what makes scale-to-zero safe: a search replica can be created from nothing, hydrate from remote, serve reads, and be torn down again, all without touching the writer.

# The remote-store recovery path (grep to find the real class names).
grep -rln "RemoteStoreRecovery\|recoverFromRemoteStore\|RemoteStoreReplicationSource" \
  server/src/main/java/org/opensearch/index/ \
  server/src/main/java/org/opensearch/indices/recovery/ 2>/dev/null | head

How this underpins reader/writer separation

Stack the pieces and the larger initiative falls out:

remote-backed storage   →  durability lives in the object store, not in replicas
        ↓
search replicas         →  read-only copies that hydrate from the object store
        ↓
reader/writer separation→  scale reads (search replicas) independently of writes
        ↓
scale-to-zero           →  zero search replicas when idle; durability never at risk

Each layer requires the one above it. You cannot have search replicas that pull from remote without remote store; you cannot separate read/write scaling without search replicas; you cannot scale reads to zero without durability being independent of replica count. This is precisely why the section reading order puts this chapter before sharding & scaling.

Multi-writer detection (#17763)

Once durability is a shared remote object that any node could upload to, a new and dangerous failure mode appears: two nodes both believing they are the writer for a shard (a network partition, a botched failover) and both uploading segments. Now the remote store has conflicting histories and the shard is corrupt.

The safety invariant — tracked in #17763 (multi-writer detection RFC) — is that at most one writer may upload for a shard at any time, enforced with fencing. The mechanism is the primary term: every upload is tagged with the primary term of the writer that produced it, and a stale writer (one whose primary term has been superseded) must be detected and prevented from corrupting the remote state. This is the storage-side complement to the routing-side safety in sharding & scaling.

# Primary-term / fencing on the upload path.
grep -rn "primaryTerm\|MultiWriter\|fenc\|stale.*upload\|metadata lock" \
  server/src/main/java/org/opensearch/index/store/ 2>/dev/null | head

Warning: Multi-writer corruption is silent until a read returns garbage or a recovery fails. Any change to the upload path must preserve the primary-term fencing invariant; a "small optimization" that drops the term check is a data-loss bug. Treat this as load-bearing.

Try it / read it

# 1. Read the remote-store directory + listeners.
grep -rln "RemoteSegmentStoreDirectory\|RemoteStoreRefreshListener" \
  server/src/main/java/org/opensearch/index/store/ | head
grep -rln "RemoteFsTranslog\|RemoteTranslog" \
  server/src/main/java/org/opensearch/index/translog/ | head

# 2. The settings that turn remote store on (names vary by version).
grep -rn "remote_store\|RemoteStoreSettings\|cluster.remote_store" \
  server/src/main/java/org/opensearch/ | head -20

# 3. Tests that exercise remote store + recovery.
./gradlew :server:internalClusterTest --tests "*RemoteStore*" 2>&1 | tail -15
./gradlew :server:test --tests "*RemoteSegmentStoreDirectory*" 2>&1 | tail -15

# 4. Inspect a remote-enabled shard's stats (on a configured cluster).
curl -s 'localhost:9200/_remotestore/stats/scaling-demo?pretty' 2>/dev/null | head -40

Common bugs and symptoms

Validation: prove you understand this

1. Explain how remote store decouples durability from replica count, and what that enables downstream.
2. Name the two upload streams (segments, translog), their triggers, and what each one makes durable.
3. Contrast peer segment replication with remote-store-backed replication on: where the replica pulls from, what provides durability, and what recovery looks like.
4. Walk remote-store recovery end to end and say why it does not need a live source node.
5. Draw the four-layer stack (remote store → search replicas → reader/writer separation → scale-to-zero) and explain why each layer requires the one above.
6. State the multi-writer invariant (#17763) and the mechanism (primary-term fencing) that enforces it.

Next: Backpressure and Admission Control, or back to Sharding, Scaling & Reader/Writer Separation to see what this foundation enables. Related capstone: segrep/RW-separation observability.