Replication: Document and Segment

A shard has a primary and zero or more replicas, and the replicas have to stay consistent with the primary. OpenSearch offers two fundamentally different ways to keep them in sync. The classic model is document replication: the primary indexes a write, then forwards the same operation to each replica, which indexes it independently. The newer model is segment replication: the primary indexes alone, and replicas copy the resulting Lucene segments from the primary instead of re-doing the indexing work. The choice trades CPU against a different durability/visibility story, and as a contributor you must understand both because half the engine's correctness arguments hinge on which one is active.

This chapter contrasts the two, dissects the document-replication write path (TransportReplicationAction, ReplicationOperation, ReplicationTracker, in-sync allocation IDs, the checkpoints), and the segment-replication services. It builds on Recovery (which uses the same checkpoints) and feeds Refresh, Flush, and Merge (segrep changes who merges).

After this chapter you can:

• Explain the document-replication primary→replica write path and how a write is acknowledged.
• Define in-sync allocation IDs, local/global checkpoints, and what they guarantee.
• Contrast document vs segment replication on CPU, durability, and freshness.
• Reason about when index.replication.type: SEGMENT is the right call.

Two models at a glance

grep -rn "replication.type\|ReplicationType\|SEGMENT\|DOCUMENT" \
  server/src/main/java/org/opensearch/indices/replication/common/ReplicationType.java \
  server/src/main/java/org/opensearch/index/IndexSettings.java

Document replication: the write path

A write enters via TransportShardBulkAction → IndexShard.applyIndexOperationOnPrimary → InternalEngine.index(...) (Lucene + translog). The replication itself is orchestrated by TransportReplicationAction / ReplicationOperation: the primary applies the op locally first, assigns it a sequence number, then forwards it to every replica in the in-sync set. The request is acknowledged to the client after enough copies confirm (per the write wait_for_active_shards/consistency rules).

sequenceDiagram
    participant C as Client
    participant P as Primary shard
    participant R1 as Replica 1
    participant R2 as Replica 2
    C->>P: index op
    Note over P: applyIndexOperationOnPrimary -> Engine.index -> assign seqNo + translog
    par forward to in-sync replicas
        P->>R1: replica request (same op + seqNo + primary term)
        P->>R2: replica request
    end
    R1-->>P: applied (local checkpoint advances)
    R2-->>P: applied
    Note over P: ReplicationTracker advances global checkpoint = min(in-sync local checkpoints)
    P-->>C: ack

grep -n "class ReplicationOperation\|performOnPrimary\|performOnReplica\|markUnavailableShardsAsStale\|getReplicationGroup" \
  server/src/main/java/org/opensearch/action/support/replication/ReplicationOperation.java
grep -n "class TransportReplicationAction\|shardOperationOnPrimary\|shardOperationOnReplica" \
  server/src/main/java/org/opensearch/action/support/replication/TransportReplicationAction.java

In-sync allocation IDs

The primary doesn't replicate to all assigned copies — only to those in the in-sync set: copies that are known to have every op up to the global checkpoint. The set is part of IndexMetadata (inSyncAllocationIds) and is maintained through cluster-state updates. A replica that falls behind or fails is removed from the in-sync set (so the primary stops blocking acks on it) and must re-enter via recovery (markAllocationIdAsInSync after it catches up). This is the mechanism that prevents acknowledging writes that a "replica" never actually received.

grep -rn "inSyncAllocationIds\|in_sync_allocation\|markAllocationIdAsInSync\|removeAllocationId" \
  server/src/main/java/org/opensearch/cluster/metadata/IndexMetadata.java \
  server/src/main/java/org/opensearch/index/seqno/ReplicationTracker.java

The checkpoints (shared with recovery)

The primary term guards against split-brain writes: a stale ex-primary's replica requests carry an old term and are rejected. Operations below the global checkpoint are durable across the in-sync set and are what recovery builds on.

grep -n "primaryTerm\|getGlobalCheckpoint\|getLocalCheckpoint\|computeGlobalCheckpoint" \
  server/src/main/java/org/opensearch/index/seqno/ReplicationTracker.java

Segment replication: copy segments, don't re-index

With index.replication.type: SEGMENT, only the primary indexes documents (runs Lucene IndexWriter, refreshes, and merges). After a refresh produces new segments, the primary publishes a new ReplicationCheckpoint; replicas fetch the new/changed segment files from the primary and open a reader over them. The replica never re-indexes and never runs its own merges — it just mirrors files.

flowchart TD
    subgraph Primary
      W[index ops] --> IW[Lucene IndexWriter]
      IW -->|refresh| SEG[new segments]
      SEG --> CP[publish ReplicationCheckpoint]
    end
    CP -->|notify| TS[SegmentReplicationTargetService on replica]
    TS -->|request diff| SRC[SegmentReplicationSourceService on primary]
    SRC -->|stream missing segment files| TS
    TS --> RR[replica opens reader over copied segments]
    RR --> V[replica search now sees primary's segments]

ls server/src/main/java/org/opensearch/indices/replication/
grep -n "class SegmentReplicationTargetService\|onNewCheckpoint\|startReplication\|class ReplicationCheckpoint" \
  server/src/main/java/org/opensearch/indices/replication/SegmentReplicationTargetService.java \
  server/src/main/java/org/opensearch/indices/replication/checkpoint/ReplicationCheckpoint.java

Why segrep changes the story

• CPU: indexing and merging happen once on the primary, not on every replica. For a 1-primary-5-replica read-heavy index, that's a large CPU/IO saving. (See Refresh, Flush, and Merge: replicas don't run TieredMergePolicy.)
• Freshness: a replica is only as fresh as the last checkpoint it copied. There is inherent replication lag — a search hitting a replica may return slightly staler results than one hitting the primary. Document replication forwards each op, so replicas can refresh to near-primary freshness.
• Durability/visibility: the replica's searchable state is the primary's segments, not ops it indexed itself. Writes are still made durable via the primary's translog; the replica doesn't independently re-derive segments. This is why the durability argument must be reasoned about per replication type.

Warning: Under segment replication, "read your write" is not guaranteed against a replica until the relevant checkpoint has been copied. If your test indexes a doc and immediately searches a replica, it can legitimately miss — this is a common source of "flaky" integration tests that are actually correct behavior. Search the primary, or wait for the checkpoint.

Comparison summary

flowchart LR
    subgraph DOC[Document replication]
      DP[Primary indexes op] --> DF[forward op to replicas]
      DF --> DR[each replica re-indexes + merges]
    end
    subgraph SEG[Segment replication]
      SP[Primary indexes + merges] --> SCP[publish checkpoint]
      SCP --> SR[replicas copy segments]
    end

Reading exercise

# 1. The replication operation: primary then in-sync replicas
grep -n "ReplicationOperation\|performOnReplicas\|getReplicationGroup\|onSuccessfulReplica" \
  server/src/main/java/org/opensearch/action/support/replication/ReplicationOperation.java

# 2. In-sync set + global checkpoint
grep -n "inSyncAllocationIds\|markAllocationIdAsInSync\|updateGlobalCheckpointOnPrimary\|computeGlobalCheckpoint" \
  server/src/main/java/org/opensearch/index/seqno/ReplicationTracker.java

# 3. Segment replication target flow
grep -n "onNewCheckpoint\|startReplication\|getCheckpoint\|forceReplication" \
  server/src/main/java/org/opensearch/indices/replication/SegmentReplicationTargetService.java

# 4. Where the replication type is chosen and what it gates
grep -rn "ReplicationType\|isSegRepEnabled\|replication.type" \
  server/src/main/java/org/opensearch/index/IndexSettings.java

Answer:

1. In document replication, after the primary applies an op locally, how does it decide which replicas to forward to? Name the set and the class that owns it.
2. Define the global checkpoint as a function of local checkpoints, and explain why an op below it is safe.
3. What role does the primary term play when a former primary's stale request arrives at a replica? Why is this essential for correctness?
4. In segment replication, what triggers a replica to start copying, and what object tells it what to copy?
5. Explain precisely why "index a doc, immediately search a replica" can return zero hits under segment replication but (after refresh) not under document replication.
6. For a read-heavy index with 1 primary and 5 replicas, argue from CPU and merge cost why segment replication is attractive, and name the cost you accept.

Common bugs and symptoms

Validation: prove you understand this

1. Diagram the document-replication write path from client to ack, marking where the seqno is assigned, where the global checkpoint advances, and what condition gates the ack.
2. Define in-sync allocation IDs and explain what would go wrong if the primary acked writes to copies not in the in-sync set.
3. Give the formula for the global checkpoint in terms of in-sync local checkpoints and explain why a lagging in-sync replica holds the global checkpoint back.
4. Draw both replication models side by side, labeling what crosses the wire and who runs Lucene merges in each.
5. Explain the read-your-write difference between the two models with a concrete test scenario, and how you'd write a correct integration test for a segment-replicated index.
6. Recommend document vs segment replication for (a) a write-heavy log index with 1 replica needing fresh reads, and (b) a read-heavy catalog index with 6 replicas. Justify each from CPU, freshness, and network.