Lab SR3: Custom Routing and Partitioning

Background

By default a document's routing value is its _id, so documents scatter uniformly across all shards and a search must fan out to one copy of every shard. The concept chapter showed two levers that change this:

• Custom _routing — choose the value that gets hashed, so you can co-locate related documents (e.g. all of one tenant's data) on a single shard. A search filtered to that tenant, also given ?routing=, then hits one shard instead of N — a direct fan-out reduction.
• index.routing_partition_size — map a routing value to a contiguous window of k shards instead of one, so a heavy tenant spreads across k shards (less hotspotting) while a routed search still touches only k shards (still less than N).

This lab makes you co-locate by tenant, measure the fan-out reduction with _search_shards, then introduce routing_partition_size and watch the window appear. You will also feel the two costs: the _routing-required constraint on get/update/delete, and the hotspot when one routing value dominates.

Why this matters for contributors

Custom routing and partitioning are where users most often shoot themselves in the foot (silent 404s from a get without _routing, one giant shard from a skewed key), and they are the exact mechanism the in-place split routing RFC (#13925) must keep correct as partitions move. Understanding the fan-out math here is also the foundation for the vector-aware allocation capstone, where shard placement and per-shard cost meet.

Prerequisites

• A single-node cluster at localhost:9200.
• jq installed.
• You finished Lab SR1 (you have RouteOracle.java) and ideally Lab SR2.
• You read the "Custom _routing and routing_partition_size" and "Search-side routing" sections of the concept chapter.

Step 1 — Locate the custom-routing code

# Where _routing is read and where partition_size enters the shard math.
grep -rn "routing_partition_size\|partitionSize\|partitionOffset\|effectiveRouting\|REQUIRED" \
  server/src/main/java/org/opensearch/cluster/routing/IndexRouting.java \
  server/src/main/java/org/opensearch/cluster/routing/OperationRouting.java

# The setting + its validation (1 <= size < number_of_shards) and the "_routing required" rule.
grep -rn "INDEX_ROUTING_PARTITION_SIZE\|routing_required\|_routing.*required\|RoutingMissingException" \
  server/src/main/java/org/opensearch/cluster/metadata/IndexMetadata.java \
  server/src/main/java/org/opensearch/index/mapper/ 2>/dev/null | head

# Search-side: where ?routing= narrows the shard list.
grep -rn "searchShards\|effectiveRouting\|GroupShardsIterator" \
  server/src/main/java/org/opensearch/cluster/routing/OperationRouting.java | head

Step 2 — Baseline: a 6-shard index with default routing

curl -s -XPUT 'localhost:9200/orders-default' -H 'Content-Type: application/json' -d '{
  "settings": {"index.number_of_shards":6,"index.number_of_routing_shards":1024,"index.number_of_replicas":0}
}' | jq .

# 600 orders across 6 tenants, default routing (= _id).
for t in 1 2 3 4 5 6; do
  for i in $(seq 1 100); do
    printf '{"index":{"_id":"t%s-o%d"}}\n{"tenant":"tenant-%s","order":%d}\n' "$t" "$i" "$t" "$i"
  done
done | curl -s -H 'Content-Type: application/x-ndjson' \
        'localhost:9200/orders-default/_bulk?refresh=true' --data-binary @- | jq '.errors'

Now ask how many shards a tenant-filtered search must touch. With default routing, the answer is all 6, because tenant-3's orders are scattered by their _id:

# Fan-out of a tenant query WITHOUT routing: all shards.
curl -s 'localhost:9200/orders-default/_search_shards' \
  -H 'Content-Type: application/json' \
  -d '{"query":{"term":{"tenant":"tenant-3"}}}' \
  | jq '[.shards[][0].shard] | sort | "shards hit: \(length) -> \(.)"'

You should see shards hit: 6. There is no routing value, so OpenSearch cannot prune — it must query every shard and let each report its tenant-3 matches.

Step 3 — Co-locate by tenant with custom _routing

Recreate the data with ?routing=<tenant> so every order of a tenant hashes the tenant (not the _id) and lands on one shard.

curl -s -XPUT 'localhost:9200/orders-routed' -H 'Content-Type: application/json' -d '{
  "settings": {"index.number_of_shards":6,"index.number_of_routing_shards":1024,"index.number_of_replicas":0}
}' | jq .

# Same data, but routed by tenant. Bulk supports a per-action "routing" field.
for t in 1 2 3 4 5 6; do
  for i in $(seq 1 100); do
    printf '{"index":{"_id":"t%s-o%d","routing":"tenant-%s"}}\n{"tenant":"tenant-%s","order":%d}\n' \
      "$t" "$i" "$t" "$t" "$i"
  done
done | curl -s -H 'Content-Type: application/x-ndjson' \
        'localhost:9200/orders-routed/_bulk?refresh=true' --data-binary @- | jq '.errors'

Predict which shard each tenant lands on with the Lab SR1 oracle, then confirm:

java RouteOracle 6 1024 tenant-1 tenant-2 tenant-3 tenant-4 tenant-5 tenant-6

# Each tenant's docs are now on exactly ONE shard:
curl -s 'localhost:9200/_cat/shards/orders-routed?v&h=shard,docs'   # uneven by tenant count
for t in 1 2 3 4 5 6; do
  s=$(curl -s "localhost:9200/orders-routed/_search_shards?routing=tenant-$t" | jq -r '.shards[0][0].shard')
  echo "tenant-$t -> shard $s"
done

The _search_shards shard for tenant-$t must match RouteOracle 6 1024 tenant-$t.

Step 4 — Measure the fan-out reduction

This is the headline result. Run the same tenant search with ?routing= and count the shards:

# WITHOUT routing on the routed index — still all 6 (search can't know the value):
curl -s 'localhost:9200/orders-routed/_search_shards' \
  -H 'Content-Type: application/json' \
  -d '{"query":{"term":{"tenant":"tenant-3"}}}' \
  | jq '[.shards[][0].shard] | "no routing  -> shards hit: \(length)"'

# WITH routing — exactly 1 shard:
curl -s 'localhost:9200/orders-routed/_search_shards?routing=tenant-3' \
  | jq '[.shards[][0].shard] | "with routing -> shards hit: \(length) (\(.))"'

# And the real search returns the same 100 docs, touching 1 shard:
curl -s 'localhost:9200/orders-routed/_search?routing=tenant-3' \
  -H 'Content-Type: application/json' \
  -d '{"size":0,"query":{"term":{"tenant":"tenant-3"}}}' \
  | jq '{total:.hits.total.value, shards:._shards}'

The lesson is sharp: co-locating with _routing only helps if the search also passes ?routing=. The data layout alone doesn't prune the fan-out; the search has to tell OperationRouting#searchShards the value so it can compute the single shard. _shards.total in the real search response confirms it touched 1, not 6.

Step 5 — Feel the costs

Cost 1: _routing required on get/update/delete

A document indexed with custom routing can only be fetched with the same routing, because a bare get hashes the _id and looks on the wrong shard:

# Wrong: get without routing -> 404 (looked on the _id's shard, not tenant-3's).
curl -s -o /dev/null -w "no-routing get: %{http_code}\n" \
  'localhost:9200/orders-routed/_doc/t3-o5'

# Right: get WITH routing -> 200.
curl -s -o /dev/null -w "with-routing get: %{http_code}\n" \
  'localhost:9200/orders-routed/_doc/t3-o5?routing=tenant-3'

To make this failure loud instead of a silent 404, mark routing required in the mapping so an unrouted index/get is rejected outright:

curl -s -XPUT 'localhost:9200/orders-strict' -H 'Content-Type: application/json' -d '{
  "settings": {"index.number_of_shards":6,"index.number_of_replicas":0},
  "mappings": {"_routing": {"required": true}}
}' | jq .

# Indexing without routing now errors (RoutingMissingException) instead of going to the wrong shard:
curl -s -XPUT 'localhost:9200/orders-strict/_doc/x1' \
  -H 'Content-Type: application/json' -d '{"tenant":"t"}' | jq '.error.type'

You should see routing_missing_exception. That _routing.required: true mapping is the standard way multi-tenant indexes avoid the silent-404 footgun.

Cost 2: hotspot when a routing value dominates

Because each tenant is pinned to one shard, an unevenly sized tenant makes an unevenly sized shard, and the balancer cannot fix it (it moves whole shards, not partitions):

# Add a whale tenant: 5,000 docs all on one shard.
for i in $(seq 1 5000); do
  printf '{"index":{"_id":"whale-%d","routing":"tenant-whale"}}\n{"tenant":"tenant-whale","order":%d}\n' "$i" "$i"
done | curl -s -H 'Content-Type: application/x-ndjson' \
        'localhost:9200/orders-routed/_bulk?refresh=true' --data-binary @- | jq '.errors'

curl -s 'localhost:9200/_cat/shards/orders-routed?v&h=shard,docs,store'   # one shard is now huge
java RouteOracle 6 1024 tenant-whale                                       # which shard got the whale

One shard now dwarfs the others. That is the price of co-location.

Step 6 — routing_partition_size: spread the whale across a window

routing_partition_size = k maps a routing value to a window of k contiguous shards; the _id picks which one inside the window. So tenant-whale spreads over k shards (less hotspot) while a routed search still touches only k shards (still < N). It must satisfy 1 ≤ k < number_of_shards, and it forces _routing to be required (the _id alone can no longer find a doc).

curl -s -XPUT 'localhost:9200/orders-part' -H 'Content-Type: application/json' -d '{
  "settings": {
    "index.number_of_shards": 6,
    "index.number_of_routing_shards": 1024,
    "index.number_of_replicas": 0,
    "index.routing_partition_size": 3
  }
}' | jq .

# Index the whale here with routing required.
for i in $(seq 1 5000); do
  printf '{"index":{"_id":"whale-%d","routing":"tenant-whale"}}\n{"tenant":"tenant-whale","order":%d}\n' "$i" "$i"
done | curl -s -H 'Content-Type: application/x-ndjson' \
        'localhost:9200/orders-part/_bulk?refresh=true' --data-binary @- | jq '.errors'

# The whale now spans a WINDOW of 3 shards, not 1:
curl -s 'localhost:9200/_cat/shards/orders-part?v&h=shard,docs'
curl -s 'localhost:9200/orders-part/_search_shards?routing=tenant-whale' \
  | jq '[.shards[][0].shard] | sort | "routed search touches \(length) shards: \(.)"'

So routing_partition_size trades a little fan-out (1 → 3 shards) for a lot less hotspotting (one giant shard → three medium ones). Choosing k is the tuning knob: small k favors fan-out, large k favors balance.

Warning: routing_partition_size makes _routing required on every get/update/delete — a bare _id is no longer sufficient to locate a document because it now only selects the position within the window. Plan your access patterns accordingly.

Deliverables

• The fan-out table from Step 4: tenant-3 search touches 6 shards unrouted vs 1 when ?routing=tenant-3 is passed (with the _shards.total from a real search confirming 1).
• Confirmation that _search_shards?routing=tenant-$t matches RouteOracle 6 1024 tenant-$t.
• The two costs demonstrated: a 404 from an unrouted get (and a routing_missing_exception on a _routing.required index), and a _cat/shards showing the whale hotspot.
• The routing_partition_size=3 result: whale spread over 3 shards, routed search touching 3 shards.

Troubleshooting

Expected output

• Default index: tenant search = 6 shards. Routed index + ?routing=: 1 shard, same 100 hits, _shards.total = 1.
• Whale on orders-routed: a single oversized shard. Whale on orders-part (partition_size=3): three medium shards; routed search touches 3.
• Unrouted get on a routed doc: 404. Unrouted index on a _routing.required index: routing_missing_exception.

Stretch goals

1. Latency, not just fan-out. Time the tenant-3 search with and without ?routing= (time curl ..., many iterations). On a many-shard index the routed query is markedly cheaper — fewer shard requests to issue, collect, and merge.
2. Sweep routing_partition_size. Create indexes with k = 1, 2, 3 and the same whale; chart shard sizes vs k. Find the k that balances the whale without blowing up fan-out for the small tenants.
3. Connect to scaling. Read the Sharding, Scaling, and Reader/Writer Separation chapter's sizing section and argue: for a multi-tenant index, when is custom routing the right answer vs an index-per-tenant vs routing_partition_size?

Coding Exercises

You measured fan-out reduction and the partition window by hand; these exercises make each behavior a graded test, including the two footguns. Locate every class with rg/find first — never paste a line number from this page.

1. (warm-up) Extend the oracle to a partition window. Add a --partition-size k mode to RouteOracle.java that, given a routing value, prints the window of k contiguous shards it can land on (and which one the _id selects). Find the real math to match:

   rg -n "routing_partition_size|partitionSize|partitionOffset|effectiveRouting" \
     server/src/main/java/org/opensearch/cluster/routing/IndexRouting.java
   

   Verify your window for tenant-whale at k=3, N=6 equals the _search_shards?routing=tenant-whale set from Step 6.

2. (core) A fan-out reduction assertion. Write an OpenSearchIntegTestCase that builds orders-routed, indexes tenant data with custom _routing, and asserts that a tenant search without ?routing= reports _shards.total == 6 while the same search with ?routing=tenant-3 reports 1 — and returns the same hit count. Find the search-side narrowing and the response shard counts:

   rg -n "searchShards|effectiveRouting|GroupShardsIterator" \
     server/src/main/java/org/opensearch/cluster/routing/OperationRouting.java
   

   This encodes the headline result of the lab as a regression gate.

3. (core) A routing_missing_exception test. Write a test that creates an index with _routing.required: true, attempts an index/get without routing, and expectThrows/asserts the error is RoutingMissingException. Locate where it's thrown:

   rg -n "RoutingMissingException|routing_required|routing.*required" \
     server/src/main/java/org/opensearch/index/mapper/ \
     server/src/main/java/org/opensearch/action/
   

   You're converting the silent-404 footgun into a loud, tested contract.

4. (advanced) A routing_partition_size validation + spread test. Write a test that: (a) asserts routing_partition_size = N (= number_of_shards) and = 0 are both rejected (legal range is 1 ≤ k < N), finding the validator:

   rg -n "INDEX_ROUTING_PARTITION_SIZE|partition_size.*valid|must be.*less than" \
     server/src/main/java/org/opensearch/cluster/metadata/IndexMetadata.java
   

   and (b) creates a k=3 index, indexes a whale routing value, and asserts via _search_shards/OperationRouting that the value spans exactly 3 contiguous shards while a k=1 index spans 1. This proves both the constraint and the spread/fan-out trade-off in code.

5. (Advanced challenge) A skew-and-mitigation harness with assertions. Write an OpenSearchIntegTestCase that quantifies the hotspot and the fix: index a whale tenant (5,000 docs) plus six small tenants on a k=1 routed index and assert the max shard's doc count is > 4× the median (the hotspot the balancer can't fix — verify it can't by checking shard sizes after a reroute). Then repeat on a k=3 index and assert the whale's docs now span 3 shards with a lower max/median ratio, while a routed search still touches only 3 (< 6). Drive _cat/shards-equivalent stats via client().admin().indices().prepareStats(), and tie the partition window back to the contiguous ranges that make _split work. Find the integ-test and reroute plumbing:

   rg -ln "extends OpenSearchIntegTestCase" server/src/test/java/org/opensearch/routing/ server/src/test/java/org/opensearch/cluster/routing/
   rg -n "ClusterRerouteRequest|prepareStats|IndicesShardStoresRequest" server/src/main
   

   This is a complete decision harness: it measures skew, proves the balancer can't help, and proves routing_partition_size does — the exact analysis a multi-tenant sizing PR must show.

Issues to Practice On

Custom routing and partitioning are where users (and bugs) shoot themselves in the foot — silent 404s and skewed shards — and the mechanism the in-place split RFC must keep correct as partitions move.

List labels first (gh label list --repo opensearch-project/OpenSearch); the ShardManagement:* family covers routing, sizing, and placement. The in-place split routing RFC (#13925) is the live surface this lab feeds.

Representative patterns. (a) "Get returns 404 for a document I indexed" — almost always a custom-routing mismatch: reproduce, confirm the doc was routed, and either the fix is docs/_routing.required guidance or a real bug in how an action propagates routing (rg the get/update/delete actions for routing). (b) "One shard is huge" — a skewed routing key; reproduce the hotspot, and the contribution is often routing_partition_size guidance, a sizing diagnostic, or validation improvements (rg INDEX_ROUTING_PARTITION_SIZE).

Planted-bug drill. Find the partition-size validation bound:

rg -n "INDEX_ROUTING_PARTITION_SIZE|< .*numberOfShards|>= 1|must be" \
  server/src/main/java/org/opensearch/cluster/metadata/IndexMetadata.java

Loosen it (allow k == number_of_shards, defeating the point) or flip the lower bound, and run the routing/metadata tests (rg -l "PartitionSize|IndexMetadataTests|RoutingTests" server/src/test). Watch which validation assertion goes red — or, worse, watch a now-legal k==N make a routed search touch all shards. Revert, then add a test pinning the exact legal range 1 ≤ k < N with boundary cases — the guard the bug removed.

Etiquette: claim the issue before working it, reproduce first; every PR needs a test + a CHANGELOG.md entry + DCO Signed-off-by (git commit -s). See community-interaction.

Validation: self-check

1. You co-located tenant-3 with _routing but the search still hit 6 shards. What single change made it hit 1, and which method consumes that change?
2. Explain the silent-404 failure: why does a bare get of a custom-routed document look on the wrong shard, and how does _routing.required: true convert the silent failure into a loud one?
3. Define routing_partition_size precisely: what does the routing value select, what does the _id select, and what are the legal values of k?
4. State the trade-off routing_partition_size=3 makes for the whale tenant in terms of both hotspot and fan-out, with numbers from your run.
5. Why can the balancer not fix a custom-routing hotspot, and what are your two real options to mitigate it?
6. Tie it back: how does the partition window of routing_partition_size relate to the contiguous partition ranges that make _split work?

This closes the Sharding & Routing masterclass. For the node-placement half, study the Shard Allocation deep-dive; for a real contribution surface, take the Vector-Aware Allocation capstone or read the in-place split routing RFC #13925.