Recovery

When a shard copy is assigned to a node (see Shard Allocation), it does not magically have the data. It has to recover it: a brand-new replica must copy the primary's Lucene segments and replay any operations it missed; a node that crashed and came back must replay its own translog; a relocating shard must hand off to its new home. Peer recovery — replica catching up from primary — is the workhorse, and modern OpenSearch makes it cheap with sequence numbers and retention leases so a briefly-disconnected replica copies only the operations it missed instead of the whole shard.

This chapter dissects the peer-recovery handshake (RecoverySourceHandler's two phases), the sequence-number machinery that makes it incremental, retention leases, and the _recovery API. It is the operational consequence of Replication (the checkpoints come from there) and The Translog (phase 2 replays translog ops), and it's what an allocation decision actually does.

After this chapter you can:

• Trace a peer recovery from StartRecoveryRequest through phase 1 and phase 2.
• Explain how sequence numbers + retention leases turn a full copy into an ops-only catch-up.
• Read RecoveryState.Stage and the _recovery API to see where a recovery is.
• Reason about recovery throttling and why a recovery storm hurts a cluster.

The cast

ls server/src/main/java/org/opensearch/indices/recovery/
grep -n "class RecoverySourceHandler\|recoverToTarget\|phase1\|phase2\|prepareTargetForTranslog" \
  server/src/main/java/org/opensearch/indices/recovery/RecoverySourceHandler.java

Recovery types

grep -rn "RecoverySource\|PeerRecoverySource\|ExistingStoreRecoverySource\|SnapshotRecoverySource" \
  server/src/main/java/org/opensearch/cluster/routing/RecoverySource.java

Peer recovery: the two-phase handshake

The replica (target) sends a StartRecoveryRequest to the primary (source) carrying the replica's current state: its SeqNoStats (local checkpoint, max seqno), its store metadata (which segment files it already has), and any RetentionLease info. The primary's RecoverySourceHandler.recoverToTarget decides between a full file copy and an ops-only catch-up, then runs:

sequenceDiagram
    participant T as Replica (RecoveryTarget)
    participant S as Primary (RecoverySourceHandler)
    T->>S: StartRecoveryRequest (seqNo stats, store metadata, retention lease)
    Note over S: decide: can we skip phase 1?<br/>(replica's seqno >= a retained safe commit)
    alt full copy needed
        S->>S: phase 1: snapshot a safe Lucene commit
        S->>T: send missing segment files (throttled)
        T->>S: filesReceived
    else ops-only catch-up
        Note over S: skip phase 1 entirely
    end
    S->>T: prepareForTranslogOperations
    Note over S: phase 2: replay translog ops from a startingSeqNo
    S->>T: send translog ops [startingSeqNo .. endingSeqNo] (batched, throttled)
    T->>S: ops applied
    S->>T: finalizeRecovery (mark replica in-sync, update global checkpoint)
    Note over T: RecoveryState.Stage = DONE; shard STARTED

Phase 1 — segments

The primary identifies a safe commit (one whose operations are all ≤ a checkpoint the replica can build on), diffs its segment files against the replica's store metadata, and ships only the missing/different files. Identical files (matched by name + checksum) are not resent. While this happens the primary holds a Retention so the underlying commit isn't merged away.

If the replica already has a recent-enough copy (sequence-number-based recovery, below), phase 1 is skipped entirely — no segments are copied.

Phase 2 — translog ops

The primary replays operations from a startingSeqNo (the first op the replica is missing) through the current endingSeqNo, streaming them as Translog.Operations the replica applies via its engine. New writes arriving during recovery are also captured (the primary tracks them) so the replica ends fully caught up before finalizeRecovery marks it in-sync.

grep -n "phase1\|recoverFilesFromSource\|sendFiles\|phase2\|sendSnapshotOfOperations\|finalizeRecovery\|prepareTargetForTranslog" \
  server/src/main/java/org/opensearch/indices/recovery/RecoverySourceHandler.java

Sequence numbers: how the replica catches up cheaply

Every write gets a monotonically increasing sequence number on the primary, assigned via the LocalCheckpointTracker. Two checkpoints matter:

The global checkpoint is the linchpin: operations at or below it are safely on every in-sync copy. A replica that disconnects and returns only needs operations after its last local checkpoint. If the primary still retains those ops (it hasn't trimmed/merged them away), phase 1 is skipped and recovery is just phase 2 over a tiny range. That retention is what retention leases guarantee.

grep -n "class LocalCheckpointTracker\|getProcessedCheckpoint\|markSeqNoAsProcessed" \
  server/src/main/java/org/opensearch/index/seqno/LocalCheckpointTracker.java
grep -n "globalCheckpoint\|getGlobalCheckpoint\|class ReplicationTracker\|markAllocationIdAsInSync" \
  server/src/main/java/org/opensearch/index/seqno/ReplicationTracker.java

Retention leases: keep the ops the replica will need

A retention lease is a promise the primary makes to retain operations above a given sequence number for a given retention period, so a lagging replica can do an ops-only recovery instead of a full copy. The primary creates a peer recovery retention lease per replica (ReplicationTracker manages RetentionLeases). Without leases, the primary's translog/soft-deletes could be trimmed before the replica reconnects, forcing an expensive full file copy.

grep -n "RetentionLease\|addPeerRecoveryRetentionLease\|retention_lease\|renewPeerRecoveryRetentionLeases" \
  server/src/main/java/org/opensearch/index/seqno/ReplicationTracker.java
grep -rn "soft_deletes.retention_lease.period\|SOFT_DELETES" \
  server/src/main/java/org/opensearch/index/IndexSettings.java

Note: "Why did a 5-minute node blip cause a full-shard recopy?" usually means the retention lease expired (node down longer than the lease period) or soft-deletes were disabled — so the ops the replica needed were already gone and only a full copy could rebuild it.

RecoveryState.Stage — the progress machine

RecoveryState tracks a recovery through a fixed sequence of stages, surfaced by the _recovery API:

stateDiagram-v2
    [*] --> INIT
    INIT --> INDEX: copying segment files (phase 1)
    INDEX --> VERIFY_INDEX: checksum/verify
    VERIFY_INDEX --> TRANSLOG: replay ops (phase 2)
    TRANSLOG --> FINALIZE: mark in-sync, bump global checkpoint
    FINALIZE --> DONE
    DONE --> [*]

grep -n "enum Stage\|INIT\|INDEX\|VERIFY_INDEX\|TRANSLOG\|FINALIZE\|DONE" \
  server/src/main/java/org/opensearch/indices/recovery/RecoveryState.java

Read live recoveries:

# Per-shard recovery progress: stage, % bytes, % translog ops, source/target node
curl -s 'localhost:9200/idx/_recovery?active_only=true&pretty'
curl -s 'localhost:9200/_cat/recovery?v&h=index,shard,type,stage,source_node,target_node,files_percent,translog_ops_percent,time'

A recovery stuck in INDEX with slow files_percent is bandwidth/throttle bound; stuck in TRANSLOG with a huge op count means a long range to replay (lease expired → big catch-up).

Throttling: don't let recovery DoS the cluster

Copying segments and replaying ops is heavy I/O and network. A node that loses several peers at once could try to recover dozens of shards simultaneously and saturate everything. Two layers prevent this:

grep -rn "max_bytes_per_sec\|RateLimiter\|recoverySettings" \
  server/src/main/java/org/opensearch/indices/recovery/RecoverySettings.java
grep -rn "node_concurrent_recoveries\|ThrottlingAllocationDecider" \
  server/src/main/java/org/opensearch/cluster/routing/allocation/decider/ThrottlingAllocationDecider.java

The throttling decider lives on the allocation side (see Shard Allocation) — it returns THROTTLE rather than NO, meaning "not now, retry next round," which is why recoveries trickle in rather than all firing at once.

Reading exercise

# 1. The source-side orchestration
grep -n "recoverToTarget\|phase1\|phase2\|isSequenceNumberBasedRecovery\|startingSeqNo" \
  server/src/main/java/org/opensearch/indices/recovery/RecoverySourceHandler.java

# 2. The target-side driver
grep -n "doRecovery\|StartRecoveryRequest\|getStartRecoveryRequest\|cleanFiles" \
  server/src/main/java/org/opensearch/indices/recovery/PeerRecoveryTargetService.java

# 3. Checkpoints
grep -n "getProcessedCheckpoint\|globalCheckpoint\|markAllocationIdAsInSync\|in-sync\|inSync" \
  server/src/main/java/org/opensearch/index/seqno/ReplicationTracker.java

# 4. Retention leases
grep -n "PeerRecoveryRetentionLease\|renewPeerRecovery\|retention_lease" \
  server/src/main/java/org/opensearch/index/seqno/ReplicationTracker.java

Answer:

1. What does the replica send in StartRecoveryRequest, and how does the primary use the replica's SeqNoStats and store metadata to decide whether to skip phase 1?
2. Distinguish the local checkpoint from the global checkpoint. Which one defines "all in-sync copies have this op," and which class tracks it?
3. Explain, step by step, how a replica that was disconnected for 2 minutes performs an ops-only recovery, and what guarantees the needed ops still exist on the primary.
4. What happens if the retention lease for an absent replica expires before it reconnects? Which stage of the next recovery will be expensive and why?
5. Map RecoveryState.Stage values to phase 1 vs phase 2 of RecoverySourceHandler.
6. ThrottlingAllocationDecider returns THROTTLE, not NO, for an over-the-limit recovery. Why does that distinction matter for cluster recovery behavior?

Common bugs and symptoms

Validation: prove you understand this

1. Draw the peer-recovery sequence (StartRecoveryRequest → optional phase 1 → phase 2 → finalize), labeling which side is source/target and what flows on each arrow.
2. Define local checkpoint and global checkpoint precisely, name the class that owns each, and explain why the global checkpoint is what makes ops-only recovery safe.
3. Walk through a 90-second replica disconnect that results in an ops-only recovery: what was retained, by what mechanism, and why phase 1 was skipped.
4. Explain the failure path where the same disconnect, but for 13 hours, forces a full file copy. Cite the exact setting involved.
5. Read a _recovery response stuck at stage: INDEX, files_percent: 12 and propose two distinct causes and the setting you'd check for each.
6. Explain why recovery throttling uses THROTTLE (retry-later) on the allocation side instead of NO, and what would go wrong if it used NO.