Sharding, Scaling, and Reader/Writer Separation

The classic OpenSearch scaling model couples two things that workloads rarely want coupled: a replica is simultaneously your durability (a second copy of the data) and your read capacity (another shard to serve queries). If you have a read-heavy workload, you add replicas — and pay to keep every byte indexed twice, on local disk, by nodes that are also doing the write-side work. If your write and read traffic have wildly different shapes (steady ingest, bursty search), you cannot scale them independently. You scale the whole shard or nothing.

Reader/writer separation breaks that coupling. Built on top of remote-backed storage, it introduces search replicas that serve reads by pulling segments from the remote store — without participating in indexing or replication of writes — so you can scale read capacity up and down (even to zero) independently of write capacity. This is the largest re-architecture in modern OpenSearch, and it touches routing, allocation, recovery, and the shard role model at once.

After this chapter you should be able to: size shards from first principles; explain what a search replica is and how it differs from a regular replica; trace how reader/writer separation changes routing and allocation; cite the real RFC trail; and reason about scale-to-zero and multi-writer safety.

Prerequisites. This chapter builds on three deep dives and does not re-derive them: shard allocation, replication (segment replication especially), and remote-backed storage and durability. Read at least the first if shards-across-nodes is fuzzy.

Shard sizing fundamentals (the part everyone gets wrong)

Before any of the new machinery, you must be able to reason about shard count and size, because the most common "OpenSearch is slow / unstable" report is really an oversharding report (you saw this in the Warm-Up's Dataset 5).

# The settings that bound shard counts and the allocator that places them.
grep -rn "max_shards_per_node\|cluster.max_shards" \
  server/src/main/java/org/opensearch/indices/ServerlessProvider.java \
  server/src/main/java/org/opensearch/cluster/metadata/ 2>/dev/null | head
grep -rln "class BalancedShardsAllocator" \
  server/src/main/java/org/opensearch/cluster/routing/allocation/allocator/

The whole motivation for reader/writer separation is that replica count is a bad read-scaling dial because it also drives durability and write-side cost. The new model gives you a dial that scales only reads.

The coupling problem, made concrete

flowchart TD
    subgraph "Classic model"
      P1["primary (writes + reads + durability)"] --> R1["replica (reads + durability + applies writes)"]
      P1 --> R2["replica (reads + durability + applies writes)"]
    end
    Note["add a replica → +read capacity AND +durability AND +write work AND +local disk"]

Each classic replica must keep up with indexing (apply every write, via document or segment replication), stores a full copy on local disk, and counts toward durability. To get more read throughput you are forced to also buy durability and write-side work you may not need. Reader/writer separation severs read capacity from all three.

Search replicas: read capacity, decoupled

A search replica is a shard copy whose job is only to serve reads. It does not accept writes, does not participate in the in-sync replication group, and does not provide the primary's durability — instead it gets its data from the remote store, pulling the segments the writers uploaded.

flowchart TD
    W["writer (primary): indexes, uploads segments + translog"] --> RS[("remote store (object store)")]
    RS --> SR1["search replica 1 (reads only, pulls segments)"]
    RS --> SR2["search replica 2 (reads only)"]
    RS --> SR0["search replicas → can scale to 0"]
    Q["search request"] --> SR1
    Q --> SR2

Because a search replica's data comes from the remote store rather than from a live primary, you can add and remove them freely — and crucially, you can have zero of them when there is no read traffic, which is the scale-to-zero feature.

What changes in routing and allocation

This is the part a contributor must understand to work in this area. Search replicas are a new shard role, and that role threads through routing and allocation:

• ShardRouting gains a role/role-aware state. A shard copy is now "writer primary", "writer replica", or "search replica" — search routing must send read traffic to search replicas (and writers, depending on config) while write routing goes only to writers.
• Allocation deciders become role-aware. The allocator must place search replicas on nodes that can pull from the remote store, must not count them as in-sync copies, and must allow their count to drop to zero without turning the index red.
• Recovery for a search replica is a remote-store recovery, not a peer recovery — it hydrates from the object store (remote store & durability).

# The shard routing / role model — grep to find the real role enum in your branch.
grep -rn "SearchReplica\|search_replica\|ShardRouting.Role\|isSearchOnly\|searchOnly" \
  server/src/main/java/org/opensearch/cluster/routing/ | head -20
# Allocation deciders that must become role-aware.
ls server/src/main/java/org/opensearch/cluster/routing/allocation/decider/
# Where search routing chooses copies.
grep -rn "class OperationRouting\|activeShards\|searchShards\|preferredShard" \
  server/src/main/java/org/opensearch/cluster/routing/ | head

Note: Because this is an evolving initiative, exact class and enum names shift between releases. Trust the grep, not a memorized name. The META issue is the source of truth for the current decomposition.

The RFC trail (read these in order)

This is a textbook public design effort. Read it the way Engineering at Scale teaches: META first for the shape, RFC for the why, proposal for the what, PR for the how.

The multi-writer detection thread (#17763) is the distributed-correctness keystone: once durability lives in the remote store and writers are decoupled from readers, you must guarantee that two nodes never both believe they are the writer for a shard and both upload conflicting segments. That fencing problem is covered from the storage side in remote store & durability.

Scale-to-zero, concretely

The endgame: an index that is being written but rarely read keeps zero search replicas, spending nothing on read capacity, and spins search replicas up only when queries arrive (cold start), hydrating them from the remote store. The writer keeps indexing the whole time; durability is never at risk because it lives in the object store, not in any replica count.

no read traffic      →  search replicas = 0   (read cost ≈ 0; writer still indexing)
read traffic arrives →  allocate search replica(s); recover from remote store
read traffic falls   →  scale search replicas back toward 0

This only works because durability was first decoupled from replicas by remote-backed storage. That is why the reading order in the section index puts remote store before this chapter.

Try it / read it

# 1. Inspect the classic dials so you can feel the coupling.
curl -s -XPUT 'localhost:9200/scaling-demo' -H 'Content-Type: application/json' \
  -d '{"settings":{"number_of_shards":2,"number_of_replicas":1}}'
curl -s 'localhost:9200/_cat/shards/scaling-demo?v'        # primaries + replicas
curl -s -XPUT 'localhost:9200/scaling-demo/_settings' -H 'Content-Type: application/json' \
  -d '{"index":{"number_of_replicas":2}}'                  # the only classic read dial
curl -s 'localhost:9200/_cat/shards/scaling-demo?v'        # +read capacity AND +durability

# 2. Find search-replica settings/roles in your checkout (names vary by version).
grep -rn "number_of_search_replicas\|search_replicas\|SEARCH_REPLICA\|SearchReplica" \
  server/src/main/java/org/opensearch/ | head -20

# 3. Read the allocation + recovery surface that the initiative changes.
grep -rln "class AllocationService" server/src/main/java/org/opensearch/cluster/routing/allocation/
grep -rln "RemoteStoreRecovery\|remoteStore" server/src/main/java/org/opensearch/index/shard/ | head

# 4. Tests that exercise search replicas / separation.
./gradlew :server:internalClusterTest --tests "*SearchReplica*" 2>&1 | tail -15
./gradlew :server:test --tests "*SearchReplicaAllocation*" 2>&1 | tail -15

Common bugs and symptoms

Validation: prove you understand this

1. Explain the classic coupling: name the three things a regular replica provides at once, and why that makes it a bad read-scaling dial.
2. Define a search replica and contrast it with a regular replica on data source, durability, write participation, and independent scaling.
3. Name the three places the role model threads through (routing, allocation, recovery) and what changes in each.
4. Walk the RFC trail from #7258 to #17299 and say what each step contributes.
5. Explain why scale-to-zero is only possible because of remote-backed storage.
6. State the multi-writer safety invariant (#17763) and why it becomes critical once durability lives in the remote store.

Next: Remote-Backed Storage and Durability — the foundation this whole chapter stands on. Then the vector-aware-allocation capstone.