Stage 5 — Shard Allocation Issues

What this stage teaches

Stage 5 drills the subsystem that decides where every shard lives: the allocation engine. The skill is reasoning about a chain of yes/no/throttle decisions over an immutable routing snapshot, and reproducing a placement bug in a fast unit test instead of a flaky cluster.

AllocationDeciders — the ordered chain of AllocationDecider implementations (SameShardAllocationDecider, DiskThresholdDecider, AwarenessAllocationDecider, FilterAllocationDecider, MaxRetryAllocationDecider, ThrottlingAllocationDecider, …). Each returns a Decision (YES / NO / THROTTLE) with a human-readable explanation.
AllocationService — orchestrates a reroute: applies deciders, moves shards, updates the RoutingTable, and produces the new ClusterState.
BalancedShardsAllocator — the default balancer that picks which eligible node a shard goes to, optimising a weight function across nodes.
RoutingAllocation / RoutingNodes — the mutable working copy the allocator operates on during a single reroute, derived from the immutable ClusterState.

The classic Stage 5 bug is a decider returning the wrong Decision for an edge case — a YES where it should THROTTLE, or a NO whose explanation misleads the operator. You reproduce it with a hand-built RoutingAllocation and confirm it through _cluster/allocation/explain.

Prerequisite: Stage 4 (you must be fluent in ClusterState and RoutingTable) plus the shard allocation deep dive.

The decision chain

flowchart LR
  R[reroute trigger] --> AS[AllocationService.reroute]
  AS --> RA[build RoutingAllocation<br/>RoutingNodes mutable copy]
  RA --> D{AllocationDeciders.canAllocate}
  D -->|each decider| d1[SameShard]
  D --> d2[DiskThreshold]
  D --> d3[Awareness]
  D --> d4[Filter]
  D --> d5[MaxRetry]
  D --> d6[Throttling]
  D -->|combined Decision| B[BalancedShardsAllocator picks node]
  B --> NRT[new RoutingTable -> new ClusterState]

Decision.Multi combines the chain: the most restrictive wins (NO beats THROTTLE beats YES), but every decider's explanation is retained so _cluster/allocation/explain can show the full reasoning. That retention is why allocation explanations are so valuable — and why a wrong explanation is itself a bug worth fixing.

Finding Stage 5 issues

is:issue is:open label:bug no:assignee "allocation" in:title,body
is:issue is:open label:bug no:assignee "unassigned" in:title,body
is:issue is:open label:bug no:assignee "rebalance" in:title,body
is:issue is:open label:"help wanted" no:assignee "AllocationDecider" in:body

Area labels may read Cluster Manager, distributed framework, or a Allocation-flavoured component label — check the current set.

Fallback grep — find the decider whose behaviour the issue describes:

ls server/src/main/java/org/opensearch/cluster/routing/allocation/decider/
grep -rln "extends AllocationDecider" server/src/main/java/org/opensearch/cluster/routing/allocation/decider/
# The disk decider, a frequent source of edge-case bugs:
grep -n "canAllocate\|canRemain\|Decision" \
  server/src/main/java/org/opensearch/cluster/routing/allocation/decider/DiskThresholdDecider.java | head

Walked example — a decider returning the wrong `Decision`

Illustrative of the pattern. The exact branch and class come from the grep; the disk decider stands in for "some decider whose threshold logic has an off-by-edge bug."

Symptom: an issue reports that when free disk is exactly at the low watermark, the DiskThresholdDecider returns YES (allowing a new shard) when it should return NO — an off-by-boundary in a > that should be >=. The operator only finds out when the node fills past the watermark and shards refuse to relocate.

Locate the decision

grep -n "freeBytesThresholdLow\|freeDiskThresholdLow\|canAllocate" \
  server/src/main/java/org/opensearch/cluster/routing/allocation/decider/DiskThresholdDecider.java
git log --oneline -n 5 -- server/src/main/java/org/opensearch/cluster/routing/allocation/decider/DiskThresholdDecider.java
git blame -L <start>,<end> .../DiskThresholdDecider.java

The suspect branch (schematic):

if (freeBytes > freeBytesThresholdLow.getBytes()) {
    return allocation.decision(Decision.YES, NAME, "enough disk on node [%s]", node.nodeId());
}
return allocation.decision(Decision.NO, NAME,
    "the node is above the low watermark ...");

At freeBytes == threshold the > is false, so it falls through to NO — but trace the whole method; the real bug is often a second comparison elsewhere (e.g. the canRemain path) that uses >= inconsistently. The lesson: read both canAllocate and canRemain, they must agree at the boundary.

Diff

--- a/server/src/main/java/org/opensearch/cluster/routing/allocation/decider/DiskThresholdDecider.java
+++ b/server/src/main/java/org/opensearch/cluster/routing/allocation/decider/DiskThresholdDecider.java
@@
-        if (freeBytes > freeBytesThresholdLow.getBytes()) {
+        if (freeBytes >= freeBytesThresholdLow.getBytes()) {
             return allocation.decision(Decision.YES, NAME,
                 "enough disk for the shard on node [%s], free: [%s], threshold: [%s]",
                 node.nodeId(), new ByteSizeValue(freeBytes), freeBytesThresholdLow);
         }
         return allocation.decision(Decision.NO, NAME,
             "the node [%s] is above the low watermark cluster setting [%s], free: [%s], threshold: [%s]",
             node.nodeId(), CLUSTER_ROUTING_ALLOCATION_LOW_DISK_WATERMARK_SETTING.getKey(),
             new ByteSizeValue(freeBytes), freeBytesThresholdLow);

Notice the diff also enriches the explanation (free bytes, threshold, setting key). A decider fix almost always improves its Decision text in the same PR — that is the Stage 3 diagnostic skill applied to allocation, and reviewers expect it.

Reproduce in a `RoutingAllocation` unit test

The whole point of this stage: you do not need a cluster. Build a RoutingAllocation with the deciders and a synthetic ClusterInfo (disk usage), then assert the decision.

public void testLowWatermarkBoundaryReturnsNo() {
    // Disk info: node has exactly the low-watermark amount free.
    ClusterInfo clusterInfo = new DevNullClusterInfo(
        /* leastUsages */ Map.of("node1", new DiskUsage("node1", "node1", "/data",
            /*total*/ 100, /*free*/ 15)),   // 15% free == low watermark of 85% used
        /* mostUsages */ Map.of("node1", new DiskUsage("node1", "node1", "/data", 100, 15)),
        /* shardSizes */ Map.of());

    Settings settings = Settings.builder()
        .put(CLUSTER_ROUTING_ALLOCATION_LOW_DISK_WATERMARK_SETTING.getKey(), "85%")
        .build();

    DiskThresholdDecider decider = new DiskThresholdDecider(settings,
        new ClusterSettings(settings, ClusterSettings.BUILT_IN_CLUSTER_SETTINGS));

    ClusterState state = /* one index, one unassigned shard, one node — built with helpers */;
    RoutingAllocation allocation = new RoutingAllocation(
        new AllocationDeciders(Collections.singleton(decider)),
        state.getRoutingNodes(), state, clusterInfo, null, System.nanoTime());
    allocation.debugDecision(true);   // capture explanations for the assertion

    ShardRouting shard = /* the unassigned primary */;
    RoutingNode node = allocation.routingNodes().node("node1");

    Decision decision = decider.canAllocate(shard, node, allocation);
    assertThat(decision.type(), equalTo(Decision.Type.NO));
    assertThat(((Decision.Single) decision).getExplanation(),
        containsString("above the low watermark"));
}

The OpenSearch test suite ships helpers (OpenSearchAllocationTestCase, DiskThresholdDeciderTests, ClusterInfo fakes like DevNullClusterInfo) that build these fixtures — extend the existing test class; do not hand-roll the routing table from scratch.

grep -rln "extends OpenSearchAllocationTestCase" server/src/test/java/org/opensearch/cluster/routing/
./gradlew :server:test --tests "*DiskThresholdDeciderTests" -q

Confirm through `_cluster/allocation/explain`

Prove the operator-facing behaviour on a running node:

./gradlew run &
# Fill a node to the watermark (or set a tiny watermark), create an index, then:
curl -s -XGET 'localhost:9200/_cluster/allocation/explain' -H 'content-type: application/json' -d '{
  "index": "idx", "shard": 0, "primary": true
}' | jq '.can_allocate, .node_allocation_decisions[].deciders[] | select(.decider=="disk_threshold")'

You should see the disk_threshold decider report NO at the boundary with your improved explanation text. Capture this in the PR.

Build and PR

./gradlew :server:test --tests "*DiskThreshold*" -q
./gradlew :server:precommit

CHANGELOG ### Fixed, git commit -s, push, PR. In the description, state the exact boundary condition (free == threshold) and confirm both canAllocate and canRemain agree — reviewers in this area always ask about the symmetric path.

Reading the balancer

If the issue is not "wrong decision" but "shards land on the wrong node" or "cluster never balances," the bug is in BalancedShardsAllocator, not a decider. The balancer minimises a weight function over (shard count, index spread, disk). Start here:

grep -n "weight\|balance\|WeightFunction\|allocateUnassigned\|balanceByWeights" \
  server/src/main/java/org/opensearch/cluster/routing/allocation/allocator/BalancedShardsAllocator.java | head

Balancer bugs are harder and frequently cross into Stage 10 (the balancer is hot during large rebalances). Confirm with a decider fix or an explanation fix first; the weight function is deep water.

Pitfalls

Fixing canAllocate but not canRemain. The two answer "can a shard start here?" and "can a shard stay here?". A boundary fix to one without the other produces a flapping shard: allowed to allocate, then immediately moved off. Always reconcile both.
Forgetting THROTTLE. Allocation has three answers, not two. A decider that should delay (recovery in progress, too many concurrent moves) must return THROTTLE, not NO — NO makes the shard look permanently unassignable in the explain output.
Building the routing table by hand. Use OpenSearchAllocationTestCase and its builders. A hand-built RoutingNodes that is subtly inconsistent will pass your test and hide the bug.
Not enabling debugDecision(true). Without it, RoutingAllocation does not retain explanations and your containsString assertion on the text will fail mysteriously.
Testing only the happy path. Allocation bugs live at boundaries: exactly-at-watermark, zero replicas, single node, all-nodes-excluded. Parameterise the test across them.
Changing a decider's order or default. The decider chain order and default watermarks are behaviour the whole ecosystem depends on; changing them is a BWC-sensitive decision (Stage 11), not a bug fix.

Exit criteria — when you're ready for Stage 6

One allocation fix is merged, reproduced with a RoutingAllocation/OpenSearchAllocationTestCase unit test, and confirmed via _cluster/allocation/explain.
Your fix reconciled both canAllocate and canRemain at the boundary.
You improved the Decision explanation as part of the same PR, naming the setting and the actual values.
You can explain how Decision.Multi combines the chain and why THROTTLE is distinct from NO.

Allocation places shards; the next stage works inside a single shard, where data is actually written and read. Stage 6 goes into IndexShard and the engine.

Open-Source Engineer & Contributor