Lab SR2: Shrink and Split an Index

Background

The concept chapter argued — from the partition model — that:

• _split grows the shard count by a multiplicative factor by copying the parent's segments into children and deleting wrong-partition documents; no document is rehashed, and each old shard becomes a contiguous set of new shards.
• _shrink reduces the shard count by a factor by hard-linking the source shards' segment files into one target shard (shared inodes — no byte is rewritten).
• _clone is the 1:1 case (hard-link everything into a new index).

This lab makes you do both resizes on a running cluster and then prove the mechanism on disk: you will watch shard counts change in _cat/shards, segment files change in _cat/segments, and — the payoff — confirm with ls -i/find that a shrunk index's segment files share inodes with the source index's files. That inode identity is the hard link, made visible.

Why this matters for contributors

Resize is where routing, allocation, recovery, and the store all meet. Anyone working the in-place split design effort (#12918, #13923, #13925) must understand exactly what today's out-of-place resize does — what it copies, what it hard-links, what it recovers, and why the source must be read-only and (for shrink) co-located. You can't improve the operation you can't observe.

Prerequisites

• A single-node OpenSearch cluster at localhost:9200. (Single node is fine; for shrink you'd normally co-locate primaries on one node — with one node that's automatic.)
• Shell access to the node's path.data directory so you can find/ls -i the on-disk segment files. Note where it is: bash curl -s 'localhost:9200/_nodes/_local/settings?filter_path=**.path' | jq . # or, if you launched ./gradlew run, look under build/testclusters/.../data
• You finished Lab SR1 (you can predict a shard).
• jq and basic find/ls -i available.

Step 1 — Locate the resize code

# The create-index service implements all three resizes.
grep -rn "ResizeType\|SHRINK\|SPLIT\|CLONE\|resizeIndex\|prepareResizeIndexSettings\|validateSplitIndex\|validateShrinkIndex" \
  server/src/main/java/org/opensearch/cluster/metadata/MetadataCreateIndexService.java

# The request/transport action and the hard-link recovery from local shards.
grep -rn "class ResizeRequest\|class TransportResizeAction\|recoverFromLocalShards\|addIndices\|hardLink\|Files.createLink" \
  server/src/main/java/org/opensearch/action/admin/indices/shrink/ \
  server/src/main/java/org/opensearch/index/store/ \
  server/src/main/java/org/opensearch/indices/recovery/ 2>/dev/null | head -30

Note Files.createLink (or a hardLinkOrCopy helper) — that's the literal hard link a shrink/clone makes. Split, by contrast, goes through a local-shards recovery that copies and then prunes.

Step 2 — Create a source index with a known geometry and data

Pin number_of_routing_shards so the factor rules are crystal clear. Start at 4 shards with R = 1024 (so you can split up to 1024 and shrink down to 1).

curl -s -XPUT 'localhost:9200/src-idx' -H 'Content-Type: application/json' -d '{
  "settings": {
    "index.number_of_shards": 4,
    "index.number_of_routing_shards": 1024,
    "index.number_of_replicas": 0
  }
}' | jq .

# Index 2,000 docs so each shard has visible segments.
for i in $(seq 1 2000); do
  printf '{"index":{"_id":"%d"}}\n{"n":%d,"pad":"some text for doc %d"}\n' "$i" "$i" "$i"
done | curl -s -H 'Content-Type: application/x-ndjson' \
        'localhost:9200/src-idx/_bulk?refresh=true' --data-binary @- | jq '.errors'

curl -s 'localhost:9200/_cat/shards/src-idx?v&h=index,shard,prirep,docs,store'

Record the per-shard docs counts — they should sum to 2000 and be roughly even.

Step 3 — SPLIT 4 → 8 (a valid factor)

Split requires the source to be read-only first (no concurrent writes during the segment copy), and the target shard count must be a multiple of the source that also divides number_of_routing_shards. 4 → 8 qualifies (8 is 4×2, divides 1024).

# 1. Make the source read-only.
curl -s -XPUT 'localhost:9200/src-idx/_settings' -H 'Content-Type: application/json' \
  -d '{"settings":{"index.blocks.write": true}}' | jq .

# 2. Split into a new index with 8 shards.
curl -s -XPOST 'localhost:9200/src-idx/_split/split-8' -H 'Content-Type: application/json' -d '{
  "settings": { "index.number_of_shards": 8, "index.number_of_replicas": 0 }
}' | jq .

# 3. Wait for green, then compare.
curl -s 'localhost:9200/_cluster/health/split-8?wait_for_status=green&timeout=60s' | jq '.status'
curl -s 'localhost:9200/_cat/shards/split-8?v&h=index,shard,prirep,docs,store'

Observe:

• split-8 has 8 primary shards; src-idx still has 4.
• Total doc count is identical (_count both indexes — splitting moves no documents in or out, it subdivides).
• Each old shard's documents now live in a contiguous pair of new shards (per the partition map). Verify a specific id moved as predicted:

curl -s 'localhost:9200/src-idx/_count'  | jq '.count'   # 2000
curl -s 'localhost:9200/split-8/_count'  | jq '.count'   # 2000

# Predict id "1"'s shard at N=4 vs N=8 with the Lab SR1 oracle, then confirm:
java RouteOracle 4 1024 1     # shard at N=4
java RouteOracle 8 1024 1     # shard at N=8 (same partition, new shard)
curl -s 'localhost:9200/split-8/_search_shards?routing=1' | jq -r '.shards[0][0].shard'

The split-8 shard for id 1 must equal java RouteOracle 8 1024 1's shard.

Note: You cannot split 4 → 6: 6 is not a multiple of 4. Try it and read the rejection — it comes from the validation you grepped in Step 1 (validateSplitIndex in MetadataCreateIndexService).

Step 4 — SHRINK 4 → 2, and prove the hard links

Shrink reduces shards by a factor (the new count must divide the old). It requires the source read-only and all source primaries on a single node (so hard links can be made locally). On a single-node cluster the co-location is automatic; on multi-node you'd set index.routing.allocation.require._name to one node and wait. We use a fresh source so its segments are pristine.

# Fresh 4-shard source with data.
curl -s -XPUT 'localhost:9200/shrink-src' -H 'Content-Type: application/json' -d '{
  "settings": {"index.number_of_shards":4,"index.number_of_routing_shards":1024,"index.number_of_replicas":0}
}' | jq .
for i in $(seq 1 2000); do
  printf '{"index":{"_id":"%d"}}\n{"n":%d}\n' "$i" "$i"
done | curl -s -H 'Content-Type: application/x-ndjson' \
        'localhost:9200/shrink-src/_bulk?refresh=true' --data-binary @- | jq '.errors'

# Prerequisites: read-only + co-locate (single node => already co-located).
curl -s -XPUT 'localhost:9200/shrink-src/_settings' -H 'Content-Type: application/json' -d '{
  "settings": {
    "index.blocks.write": true,
    "index.routing.allocation.require._name": null
  }
}' | jq .

# Shrink 4 -> 2.
curl -s -XPOST 'localhost:9200/shrink-src/_shrink/shrunk-2' -H 'Content-Type: application/json' -d '{
  "settings": {"index.number_of_shards":2,"index.number_of_replicas":0}
}' | jq .
curl -s 'localhost:9200/_cluster/health/shrunk-2?wait_for_status=green&timeout=60s' | jq '.status'
curl -s 'localhost:9200/_cat/shards/shrunk-2?v&h=index,shard,prirep,docs,store'

Now the on-disk proof. Find both indexes' UUIDs and their shard index/ directories, then compare inodes of the segment files. Hard-linked files share an inode number.

DATA=$(curl -s 'localhost:9200/_nodes/_local/settings?filter_path=**.path.data' \
        | jq -r '..|.data? // empty' | head -1)
# Fallback for ./gradlew run:
[ -z "$DATA" ] && DATA=$(find . -type d -path '*testclusters*data' | head -1)

SRC_UUID=$(curl -s 'localhost:9200/shrink-src/_settings' | jq -r '.["shrink-src"].settings.index.uuid')
DST_UUID=$(curl -s 'localhost:9200/shrunk-2/_settings'   | jq -r '.["shrunk-2"].settings.index.uuid')
echo "data=$DATA  src=$SRC_UUID  dst=$DST_UUID"

# List segment files with inode numbers for the source and the shrunk target.
echo "=== source shards (.cfs/.si/.fdt etc) ==="
find "$DATA" -path "*$SRC_UUID*/index/*" \( -name '*.cfs' -o -name '*.fdt' -o -name '*.cfe' -o -name '*.si' \) -printf '%i  %p\n' | sort
echo "=== shrunk target shards ==="
find "$DATA" -path "*$DST_UUID*/index/*" \( -name '*.cfs' -o -name '*.fdt' -o -name '*.cfe' -o -name '*.si' \) -printf '%i  %p\n' | sort

macOS note: BSD find has no -printf. Use: find "$DATA" -path "*$DST_UUID*/index/*" -name '*.cfs' -exec ls -i {} \; and compare the leading inode numbers, or install GNU findutils (gfind).

The shrunk target's segment files should report the same inode numbers as the corresponding source files — that is the hard link. Confirm two paths share an inode directly:

# Pick one source .cfs and find a target file with the same inode.
SRC_FILE=$(find "$DATA" -path "*$SRC_UUID*/index/*" -name '*.cfs' | head -1)
SRC_INO=$(ls -i "$SRC_FILE" | awk '{print $1}')
echo "source file inode: $SRC_INO  ($SRC_FILE)"
find "$DATA" -path "*$DST_UUID*/index/*" -name '*.cfs' -exec ls -i {} \; | grep -w "$SRC_INO" \
  && echo "HARD LINK CONFIRMED: target shares inode $SRC_INO with source"

A matching inode line proves the shrink did not copy the bytes — the two indexes' files are the same blocks on disk, linked. (Deleting the source index decrements the link count but does not free the blocks until the target releases them too.)

Step 5 — Compare segments before/after

# Segment-level view: shrink concatenates source segments into the target's shards.
curl -s 'localhost:9200/_cat/segments/shrink-src?v&h=shard,segment,docs.count,size'
echo "---"
curl -s 'localhost:9200/_cat/segments/shrunk-2?v&h=shard,segment,docs.count,size'

You should see the shrunk index's shards holding the same segments (same names, same sizes) as the source — because they are literally the same files, just linked into a 2-shard layout. A later merge in the target may rewrite them into new segments (breaking the link for the merged ones); until then, the link is intact.

Step 6 — Clone (the 1:1 baseline)

curl -s -XPUT 'localhost:9200/clone-src/_settings' -H 'Content-Type: application/json' \
  -d '{"settings":{"index.blocks.write":true}}' 2>/dev/null
# (create clone-src like shrink-src first if you want a clean one)
curl -s -XPOST 'localhost:9200/shrink-src/_clone/clone-1' -H 'Content-Type: application/json' \
  -d '{"settings":{"index.number_of_replicas":0}}' | jq .
curl -s 'localhost:9200/_cat/shards/clone-1?v&h=index,shard,docs'

clone-1 has the same shard count as shrink-src and (like shrink) shares inodes — it's the cheapest resize, hard-linking everything with no shard-count change.

Deliverables

• _cat/shards output for src-idx (4) and split-8 (8) with equal total docs.
• A confirmed prediction: id 1's split-8 shard equals RouteOracle 8 1024 1.
• _cat/shards for shrunk-2 (2 shards) and the inode match output proving the hard link ("HARD LINK CONFIRMED").
• A one-line explanation of why 4 → 6 split and 4 → 3 shrink are both rejected.

Troubleshooting

Expected output

• split-8: 8 primaries, total docs unchanged, id-1 shard matches the oracle.
• shrunk-2: 2 primaries, total docs unchanged, segment files share inodes with shrink-src (the hard-link proof).
• Invalid factors are rejected by MetadataCreateIndexService validation with a clear message.

Stretch goals

1. Split all the way up. Split 1024-routing index 4 → 8 → 16 → 32, each from the previous (each must be read-only). Watch the partition map subdivide; confirm id 1's shard each time with RouteOracle.
2. Break the link with a merge. After shrink, POST /shrunk-2/_forcemerge?max_num_segments=1 and re-run the inode check: the merged segments are new files (new inodes) — the hard link is broken once Lucene rewrites them. Explain why that's expected.
3. Measure the cost. Time a _split vs a _shrink vs a _clone on the same data (time curl ...). Explain the ordering (clone ≈ shrink ≪ split) in terms of copy-vs-link-vs-copy+prune.

Coding Exercises

You drove resize by hand and saw the inodes; now write tests that assert the mechanism so a regression in the resize path is caught. Locate every class with rg/find first — never paste a line number from this page.

1. (warm-up) An inode-match assertion script. Promote the Step 4 manual check into assert_hardlink.sh that resolves $DATA, $SRC_UUID, $DST_UUID, finds a source .cfs, and exit 0 only if a target file shares its inode (else exit 1). Make it portable: detect BSD vs GNU find and use ls -i accordingly. This is the self-contained "HARD LINK CONFIRMED" gate, now scriptable for CI.

2. (core) A split doc-conservation + routing test. Write an OpenSearchIntegTestCase that creates a 4-shard source with R=1024, indexes 2,000 known docs, makes it read-only, _splits to 8, and asserts: (a) _count is identical before/after, and (b) a sampled id's post-split shard equals what RouteOracle 8 1024 <id> predicts. Find the existing resize tests to crib the harness:

   rg -ln "_split|ResizeType.SPLIT|prepareResizeIndex|class.*Split.*IT" server/src/test
   rg -n "ResizeType|SHRINK|SPLIT|CLONE" \
     server/src/main/java/org/opensearch/cluster/metadata/MetadataCreateIndexService.java
   

3. (core) Assert the invalid-factor rejections. Write a test that attempts 4 → 6 split and 4 → 3 shrink and asserts each fails with the validation message from the service you grepped. Find the exact validators:

   rg -n "validateSplitIndex|validateShrinkIndex|must be a (multiple|factor)|IllegalArgumentException" \
     server/src/main/java/org/opensearch/cluster/metadata/MetadataCreateIndexService.java
   

   expectThrows(IllegalArgumentException.class, ...) and assert the message text — so a refactor that loosens the factor rule can't slip through unnoticed.

4. (advanced) A hard-link proof inside an integration test. Beyond the shell check, write an OpenSearchIntegTestCase that shrinks 4 → 2, then via internalCluster().getInstance(...) or the node's path.data reads two segment files' inode numbers (Files.readAttributes(..., "unix:ino")) and asserts they match — the on-disk hard link, proven in Java. Find the recovery/hard-link code so your assertion targets the right files:

   rg -n "Files.createLink|hardLinkOrCopy|recoverFromLocalShards|addIndices" \
     server/src/main/java/org/opensearch/index/store/ \
     server/src/main/java/org/opensearch/indices/recovery/
   

   Skip the assert on filesystems that don't support hard links (assumeTrue).

5. (Advanced challenge) A full resize-matrix integration test. Write one OpenSearchIntegTestCase that parameterizes over {clone (4→4), shrink (4→2), split (4→8)} and for each asserts: doc count conserved, target reaches green, routing predictions hold for sampled ids (via RouteOracle/IndexRouting), and the resize type's expected storage behavior (clone/shrink share inodes; split does not). Then add a forcemerge step on the shrunk target and assert the previously linked segments are now new inodes — encoding Stretch Goal 2's "merge breaks the link." Find the forcemerge and segments plumbing:

   rg -n "ForceMergeRequest|max_num_segments|prepareForceMerge" server/src/main
   rg -ln "extends OpenSearchIntegTestCase" server/src/test/java/org/opensearch/action/admin/indices/
   

   This is the resize-correctness harness a contributor touching the in-place split design would extend — it proves all three operations conserve data and route correctly.

Issues to Practice On

Resize sits where routing, allocation, recovery, and the store meet — the busiest contribution surface in this masterclass thanks to the in-place split effort.

List labels first (gh label list --repo opensearch-project/OpenSearch). The live design effort is the in-place split RFC trio (#12918, #13923, #13925) — read them and their linked issues to find scoped, real work.

Representative patterns. (a) "Split/shrink fails or mis-routes under setting X" — reproduce on ./gradlew run, locate the validator/recovery path with rg in MetadataCreateIndexService and the shrink action, fix, and add an integ test asserting doc count + routing after resize. (b) "Resize leaves shards stuck initializing" — an allocation/recovery interaction: reproduce, read _cluster/allocation/explain, locate recoverFromLocalShards, and add a test (cross-link shard allocation).

Planted-bug drill. Find the split factor validation:

rg -n "validateSplitIndex|multiple|% .*numShards|factor" \
  server/src/main/java/org/opensearch/cluster/metadata/MetadataCreateIndexService.java

Weaken the check (e.g. drop the "target must be a multiple of source" clause, or flip a %==0 to !=0) and run the resize tests (rg -l "class.*Resize.*Tests|class.*Split.*IT|class.*Shrink.*IT" server/src/test). Watch a 4→6-style case slip through and a downstream assert blow up. Revert, then add a test asserting 4→6 is rejected with the precise message — the guard the bug removed.

Etiquette: claim the issue before working it, reproduce first; every PR needs a test + a CHANGELOG.md entry + DCO Signed-off-by (git commit -s). See community-interaction.

Validation: self-check

1. Why must the source index be read-only for both split and shrink? What would a concurrent write corrupt?
2. Why does shrink additionally require all source primaries on one node, but split does not?
3. You ran ls -i and the shrunk file's inode equals the source file's inode. State precisely what that proves about the bytes on disk.
4. After a _forcemerge on the shrunk index, the inodes no longer match. Why is that not a bug?
5. Explain why _split keeps the total document count identical even though every document's shard may change — connect it to the partition invariant from Lab SR1.
6. Which validation method rejects 4 → 6, and what rule is it enforcing?