Project 8: Segment-Replication / RW-Separation Observability

In a document-replication cluster, a replica that has indexed up to operation N is, by definition, N operations behind nothing — it did the work itself. In a segment-replication cluster, a replica does not re-index; it copies finished segments from the primary after the primary refreshes. That is cheaper and faster, but it introduces a new, first-class concept that document replication never had: replication lag. The replica is serving searches against an older set of segments than the primary holds, and the gap between them — measured in bytes to copy, in checkpoints behind, in seconds stale — is something an operator now must be able to see. With reader/writer separation and search replicas layered on top (read nodes that scale independently and can scale to zero), this lag becomes the central health signal of the whole architecture: it is the thing that tells you whether your search tier is keeping up with your write tier.

This project asks you to improve the observability of segment replication / reader-writer separation: add or sharpen the metrics, _cat columns, stats-API fields, or logs that let an operator see replication lag and health — starting from a slice you can scope in a weekend and ending at a tested, upstreamable set of stats. It is the "Medium-Hard" project in the portfolio and one of the cleanest to land, because new, well-tested observability fields are exactly what maintainers merge.

Note: Read the Replication deep dive before you start — this brief does not re-derive how segment replication works. Also read Remote store and durability and Sharding and scaling for how remote-backed storage, segment replication, and reader/writer separation fit together. This brief assumes you know what a ReplicationCheckpoint is, what the primary→replica segment-copy round looks like, and what a "search replica" is.

Problem & motivation

Segment replication trades a familiar failure mode (replicas falling behind by re-indexing too slowly) for an unfamiliar one (replicas falling behind by copying segments too slowly), and the observability did not arrive fully formed alongside the feature:

• Lag is multi-dimensional and partially hidden. "How far behind is this replica?" has several answers: how many bytes of new segments remain to copy, how many refresh checkpoints behind the replica's last-applied checkpoint is, and how long (wall-clock) the replica has been stale. Some of these are exposed via _cat/segment_replication; some are awkward to get at, missing at the level an operator wants (per-shard vs aggregated), or absent from the programmatic stats API entirely.
• Reader/writer separation makes lag the headline metric, and it is newer. Search replicas are a recent capability. The whole value proposition — scale reads independently, even to zero — depends on operators trusting that a search replica is fresh enough. If you cannot cleanly see a search replica's lag and whether it is converging or diverging, you cannot safely run the architecture, and you certainly cannot autoscale it.
• Scale-to-zero adds a new state to observe. A search-replica tier that scaled to zero and is scaling back up has a cold period where it is catching up from far behind. The transition from "cold, catching up" to "warm, caught up" is exactly the moment an operator needs visibility into, and it is a state document replication never had.
• Failed/stalled replication is easy to miss. A replica whose segment copy is failing and retrying looks, from a distance, a lot like one that is merely slow. The stats should distinguish "behind and catching up" from "behind and stuck."

This project closes one of those gaps. The phased plan starts at exposing an existing-but-hard-to-get lag signal cleanly in the stats API and _cat (genuinely mergeable, low blast radius) and builds toward richer health fields for the search-replica path.

Real-world grounding

Reader/writer separation, search replicas, and scale-to-zero are RFC-driven and tracked by a meta-issue. These are real:

• [META] Reader and writer separation — OpenSearch #15306: https://github.com/opensearch-project/OpenSearch/issues/15306
• Scale to zero (search replicas) — OpenSearch #16720: https://github.com/opensearch-project/OpenSearch/issues/16720

The meta-issue #15306 is the durable anchor for the whole reader/writer-separation effort — including the search-replica observability work that this project lives inside. Scale-to-zero (#16720) is the feature that makes lag/freshness visibility non-optional, because a tier that scaled down and is scaling back up must be observably catching up.

Citation discipline: the search-replica stats surface is new and moving. Do not cite a field name or sub-issue you have not verified. Before you scope, search is:issue is:open label:"distributed framework" search replica OR segment replication stats and is:issue is:open "_cat/segment_replication" OR "search replica" lag in opensearch-project/OpenSearch, and grep the live _cat/stats code (below). Link the current sub-issue in your design note; #15306 and #16720 are the anchors.

Subsystems you'll touch

Deep dives that cover the surrounding ground: Replication deep dive (the mechanism — do not re-derive it) · Remote store and durability · Sharding and scaling (reader/writer separation, search replicas, scale-to-zero) · Recovery deep dive (the cold-catch-up case is recovery-adjacent) · REST layer (how _cat and stats endpoints are wired).

Phased plan

The discipline of this project is that Phase 1 exposes a real lag signal cleanly and is mergeable on its own. Observability is the contribution; you do not need to change replication behaviour at all.

Phase 0 — Build it, stand up segrep, and read the existing lag surface (1 day)

Build OpenSearch and stand up an index that uses segment replication, then make a replica fall behind and watch the existing observability report it.

# In your OpenSearch clone:
./gradlew assemble
./gradlew run        # for a multi-node segrep setup you may need a small InternalTestCluster
                     # test or a local 2-node config; segrep needs a replica to observe

# Create an index that uses segment replication:
curl -s -X PUT localhost:9200/segrep-test -H 'Content-Type: application/json' -d '{
  "settings": {
    "index.number_of_shards": 1,
    "index.number_of_replicas": 1,
    "index.replication.type": "SEGMENT"
  }
}' | python3 -m json.tool

# Index a burst so the replica has segments to copy, then read the existing lag surface:
curl -s 'localhost:9200/_cat/segment_replication?v&detailed'
curl -s 'localhost:9200/_cat/segment_replication/segrep-test?v'

Note: reader/writer separation / search replicas add index.number_of_search_replicas (or a similar setting) and a remote-store-backed cluster. The exact settings change across versions — grep for the current names before trusting any snippet: grep -rn "number_of_search_replicas\|SEARCH_ONLY\|searchReplica\|replication.type" server/src/main/java/org/opensearch/cluster/metadata.

Now find what _cat/segment_replication and the stats API actually report, and where the lag numbers come from:

grep -rn "class RestSegmentReplicationAction\|segment_replication\|Table.*addCell" \
  server/src/main/java/org/opensearch/rest/action/cat
grep -rn "class SegmentReplicationShardStats\|bytesBehind\|checkpointsBehind\|currentReplicationLag\|lastCompletedReplicationLag" \
  server/src/main/java/org/opensearch/indices/replication
grep -rn "SegmentReplicationStatsResponse\|SegmentReplicationPerGroupStats" \
  server/src/main/java/org/opensearch/action/admin/indices

Write a one-page capstone-work/lag-surface.md: what lag dimensions exist (bytes_behind / checkpoints_behind / current vs last-completed lag time), which are exposed in _cat vs the stats API vs neither, and the class that computes each. With file:line citations. This is your execution-path-mastery artifact and you cannot skip it. You must know exactly which lag number lives where before you add or move one.

Phase 1 — Expose a missing lag dimension cleanly (the scoped, mergeable slice)

Pick one lag/health dimension that exists in the internal SegmentReplicationShardStats (or is cheaply computable there) but is not cleanly available where an operator wants it — most commonly, something that is in _cat but missing from the programmatic stats API (so dashboards/automation cannot read it), or vice versa. Expose it.

# Find a field present in the shard stats object but absent from the stats-API response:
grep -rn "bytesBehind\|checkpointsBehind\|ReplicationLag" \
  server/src/main/java/org/opensearch/indices/replication/SegmentReplicationShardStats.java
grep -rn "toXContent\|writeTo\|field(" \
  server/src/main/java/org/opensearch/action/admin/indices/stats/SegmentReplicationPerGroupStats.java

// Add the field to the stats response's XContent + wire serialization (Version-guarded for BWC):
builder.field("checkpoints_behind", shardStats.getCheckpointsBehind());
builder.field("bytes_behind",       new ByteSizeValue(shardStats.getBytesBehind()).toString());
builder.field("current_replication_lag_millis", shardStats.getCurrentReplicationLagMillis());

Because this touches the wire, it needs a Version guard and a serialization round-trip test:

grep -rn "out.getVersion()\|in.getVersion()\|Version.onOrAfter\|writeOptional" \
  server/src/main/java/org/opensearch/action/admin/indices/stats/SegmentReplicationPerGroupStats.java

// AbstractWireSerializingTestCase<SegmentReplicationPerGroupStats> — round-trip the new field,
// AND assert it is dropped/defaulted when serializing to an older Version (BWC).
public void testSerializationRoundTripIncludesCheckpointsBehind() {
    SegmentReplicationPerGroupStats orig = createTestInstance();
    SegmentReplicationPerGroupStats copy = copyInstance(orig);
    assertEquals(orig.getCheckpointsBehind(), copy.getCheckpointsBehind());
}

Add the matching _cat/segment_replication column if it is the surface that was missing, and an integration test that makes a replica fall behind and asserts the field/column reports a non-zero lag, then converges to zero after the copy completes.

Run the gates:

./gradlew spotlessApply
./gradlew :server:test --tests "*SegmentReplication*Stats*"
./gradlew precommit

Why this is the right Phase 1: it is a minimum diff, it changes no replication behaviour (pure observability), and "expose this lag field in the stats API / _cat" is exactly what maintainers merge — especially under the active reader/writer-separation effort. The one subtlety is BWC on the wire, which is why the round-trip + old-Version test is mandatory. This is a credible first segrep contribution.

Phase 2 — Distinguish "catching up" from "stuck"

A single lag number cannot tell an operator whether a behind replica is converging or failing. Add the signal that distinguishes them: expose the replication state (e.g. last failure, retry count, or a derived "lag trend" — is bytes_behind decreasing?) so a stalled copy is visibly different from a slow one.

grep -rn "SegmentReplicationState\|failure\|retry\|getStage\|lastFailure" \
  server/src/main/java/org/opensearch/indices/replication

Test it by injecting a copy failure in an integration test and asserting the stats reflect "stuck" (non-zero retries / a failure reason), not merely "behind."

Phase 3 — Search-replica freshness, for reader/writer separation

Now the reader/writer-separation-specific work. A search replica's whole job is to serve fresh-enough reads. Expose its freshness as a first-class, queryable signal: per-search-replica lag, and ideally a roll-up at the index level ("worst search-replica lag in this index") that an autoscaler or operator can alert on.

grep -rn "searchReplica\|SEARCH_ONLY\|isSearchOnly\|SearchReplicaAllocation" \
  server/src/main/java/org/opensearch/cluster/routing

The deliverable is that an operator can answer, from the stats API alone: "is every search replica in this index within N checkpoints / N seconds of the primary?" — the exact question that makes scale-to-zero (#16720) safe to run, because the catch-up after scaling back up is now observable.

Phase 4 — Validate the freshness signal under the scale-to-zero transition

The freshness signal earns its keep at the moment it is hardest: a search-replica tier scaling back up from zero, catching up from far behind. Reproduce that transition in an integration test and assert the stats tell the right story end to end.

phase                       checkpoints_behind   bytes_behind   state          search-replica fresh?
search replicas at zero            n/a               n/a         (none)         n/a
just scaled up (cold)              412              1.8 GB       CATCHING_UP    NO
mid catch-up                        37              210 MB       CATCHING_UP    NO
caught up                            0                0 B        STEADY         YES

The headline is that the freshness signal transitions cleanly from "cold, not fresh" → "catching up" → "fresh," so an operator (or an autoscaler) can gate read traffic / scale decisions on it. That is the observability that makes reader/writer separation operable.

Deliverables

• capstone-work/lag-surface.md — which lag dimension lives in _cat vs stats API vs nowhere, with citations
• capstone-work/design.md — the gap (which field, which surface), the BWC/wire-format consideration, what you rejected
• Phase 1: a missing lag field exposed in the stats API and/or _cat, Version-guarded, with a serialization round-trip test
• Phase 2: a "catching up vs stuck" signal (retries / failure / lag trend), with a failure-injection test
• (Phase 3) per-search-replica freshness + an index-level roll-up for reader/writer separation
• (Phase 4) an integration test of the scale-to-zero catch-up transition asserting the freshness signal is correct
• capstone-work/validation.md — ./gradlew spotlessApply precommit output, test commands, seeds, segrep cluster setup
• A CHANGELOG.md entry under ## [Unreleased]
• An upstreaming decision: a DCO-signed PR for the Phase-1 stats field, or a written scoped proposal under #15306
• A 500–1000 word write-up: the lag-dimension framing, the BWC story, why search-replica freshness matters for scale-to-zero

Difficulty & time

The trap most people hit is the test environment: observing segment replication needs a real primary/replica (and for Phase 3+, a remote-store-backed search-replica) topology, which is fiddlier than a single node. Get the InternalTestCluster segrep setup working in Phase 0 before you scope.

Stretch goals

• Add a _cat/segment_replication column for the "catching up vs stuck" state from Phase 2 so it is visible at a glance, not just in the JSON stats.
• Emit a structured log line (at INFO/WARN) when a replica's lag crosses a threshold or a copy fails repeatedly, so the signal reaches log-based alerting, not only metrics scrapers.
• Add a cluster-health-style roll-up: a per-index "replication health" (green/yellow/red) derived from worst-replica lag, mirroring how shard allocation surfaces health.
• Wire the search-replica freshness signal into whatever autoscaling hook scale-to-zero (#16720) uses, so the catch-up state can actually gate scale decisions — the end-to-end payoff.
• Reproduce the spirit of a real #15306 sub-issue: take one concrete "operators cannot see X" complaint from that thread and close it with the field + test in the PR.

Evaluation

Self-grade against the 100-point rubric. For this project:

A finished Phase 1 at 90+ is a real, merged observability contribution in the reader/writer-separation effort — directly useful to every operator running segment replication.

How to turn this into a real contribution

1. Start from one missing field, not a dashboard. The Phase-1 "expose this lag dimension in the stats API / _cat" PR is the upstream target. It changes no replication behaviour, it is exactly the operability gap under #15306, and it needs no RFC — only a comment on the thread.
2. Comment before you code. Find the current search-replica observability sub-issue under #15306, confirm the field you want to add is wanted and not already in flight, and link #16720 (scale-to-zero) for why freshness visibility matters.
3. Respect the wire. Any stats field you add crosses the transport layer between mixed-version nodes during an upgrade. Version-guard it and ship the round-trip + old-Version serialization test in the same PR — see Serialization and BWC. Reviewers will block a stats change that is not BWC-safe.
4. One slice per PR. The lag field, then (separately) the "stuck vs catching up" state, then (after confirming scope) the search-replica freshness roll-up. Never bundle them.
5. Bring the convergence test. The reviewer's question is "does the number go to zero when the replica catches up, and stay non-zero when it is stuck?" Arrive with that integration test in the PR.
6. DCO applies. Core OpenSearch: git commit -s, CHANGELOG.md, the backport bot.

If only Phase 1 lands, you have given every operator running segment replication a lag number they could not cleanly get before — which is the difference between trusting and fearing the reader/writer separation architecture. That is the point of scoping from the single-field slice up.