Shard Allocation

Every shard — primary or replica — has to live on some node, and the choice is not arbitrary. Shard allocation is the cluster-manager-side subsystem that decides, on every relevant cluster-state change, where each shard should go: which unassigned shards to assign, which assigned shards to move for balance, and which moves to forbid because they'd violate a constraint (don't put a primary and its replica on the same node; don't fill a disk; respect rack awareness). Get this wrong as a contributor and you produce yellow/red clusters, hot nodes, or data loss across a rack failure.

This chapter dissects AllocationService, the BalancedShardsAllocator, the AllocationDeciders chain, and the operator-facing _cluster/allocation/explain and _cluster/reroute APIs. It runs on the cluster manager (formerly master) and mutates the RoutingTable inside the ClusterState; the new state is then published. It feeds directly into Recovery (an assignment triggers a recovery) and Replication.

After this chapter you can:

• Trace an allocation round from a triggering event to a new RoutingTable.
• Name the major deciders and the real-world failure each one prevents.
• Read _cluster/allocation/explain and translate a NO decision into a fix.
• Force or cancel an assignment with _cluster/reroute and know when it's safe.

The cast

ls server/src/main/java/org/opensearch/cluster/routing/allocation/decider/
grep -n "class AllocationService\|reroute\|applyStartedShards\|applyFailedShards" \
  server/src/main/java/org/opensearch/cluster/routing/allocation/AllocationService.java

What triggers an allocation round

Allocation runs as part of cluster-state updates. The common triggers:

grep -rn "reroute\|allocationService.reroute\|applyStartedShards\|applyFailedShards" \
  server/src/main/java/org/opensearch/cluster/routing/allocation/ | head
grep -rn "DiskThresholdMonitor" server/src/main/java/org/opensearch/cluster/routing/allocation/

The allocation round

flowchart TD
    T[Trigger: node change / shard started/failed / reroute / watermark] --> AS[AllocationService.reroute]
    AS --> RN[Build RoutingNodes + RoutingAllocation from current ClusterState]
    RN --> EXIST[GatewayAllocator / ExistingShardsAllocator: assign existing primaries from on-disk data]
    EXIST --> BSA[BalancedShardsAllocator.allocate]
    BSA --> U[allocateUnassigned: place unassigned shards]
    U --> M[moveShards: relocate shards that can no longer stay]
    M --> B[balance: even out weight across nodes]
    U -. each candidate move .-> D[AllocationDeciders.canAllocate / canRemain / canRebalance]
    M -. each candidate move .-> D
    B -. each candidate move .-> D
    D -->|YES / THROTTLE / NO| BSA
    B --> NRT[New RoutingTable]
    NRT --> CS[New ClusterState -> publish -> recovery starts]

The allocator proposes moves; the deciders veto or throttle them. A move happens only if every decider returns YES (or, for throttling, the round schedules it later). One NO blocks it. _cluster/allocation/explain literally replays this decider chain for a chosen shard and prints each decider's verdict — that's why it's the single most useful allocation-debugging tool.

grep -n "canAllocate\|canRemain\|canRebalance\|Decision" \
  server/src/main/java/org/opensearch/cluster/routing/allocation/decider/AllocationDeciders.java
grep -n "allocateUnassigned\|moveShards\|balance\|weight" \
  server/src/main/java/org/opensearch/cluster/routing/allocation/allocator/BalancedShardsAllocator.java

The deciders — each one prevents a specific disaster

ls server/src/main/java/org/opensearch/cluster/routing/allocation/decider/ | sed 's/AllocationDecider.java//'
grep -rn "Decision.NO\|Decision.THROTTLE\|Decision.single" \
  server/src/main/java/org/opensearch/cluster/routing/allocation/decider/DiskThresholdDecider.java

Warning: Before a rolling restart, operators set cluster.routing.allocation.enable: primaries (or none) via the EnableAllocationDecider so the cluster doesn't try to rebuild replicas of a node that's about to come right back. Forgetting to re-enable it afterward leaves replicas permanently unassigned — a top operational footgun.

RoutingAllocation and RoutingNodes — the scratch pad

AllocationService builds a mutable RoutingNodes (a node→shards view derived from the immutable RoutingTable) and a RoutingAllocation that carries the deciders, cluster info (disk usage via ClusterInfoService), and the unassignedInfo for each shard. The allocator mutates RoutingNodes; at the end, AllocationService reads the result back into a fresh, immutable RoutingTable for the new ClusterState. The immutable-snapshot discipline of ClusterState is preserved: the scratch pad is mutable, the published artifact is not.

grep -n "class RoutingNodes\|initializeShard\|relocateShard\|startShard\|class RoutingAllocation" \
  server/src/main/java/org/opensearch/cluster/routing/RoutingNodes.java \
  server/src/main/java/org/opensearch/cluster/routing/allocation/RoutingAllocation.java

UnassignedInfo records why a shard is unassigned (INDEX_CREATED, NODE_LEFT, ALLOCATION_FAILED, REPLICA_ADDED, …) and the failure count — that's what MaxRetryAllocationDecider reads and what allocation/explain reports.

grep -n "enum Reason\|allocationStatus\|failedAllocations" \
  server/src/main/java/org/opensearch/cluster/routing/UnassignedInfo.java

Where primaries of existing data come from

For a brand-new index, primaries can be allocated anywhere (empty). For an index with on-disk data (cluster restart, node rejoin), the primary must go where the best copy of the data already lives. The GatewayAllocator (an ExistingShardsAllocator) queries nodes for their on-disk shard copies and allocation IDs (see in-sync allocation IDs in Replication), then assigns the primary to the node with the most up-to-date copy. Replicas prefer nodes that can do a cheap sequence-number-based recovery from the primary (see Recovery).

grep -rn "ExistingShardsAllocator\|allocateUnassigned\|class GatewayAllocator\|TransportNodesListShardStoreMetadata" \
  server/src/main/java/org/opensearch/gateway/ | head

Operator surface: explain and reroute

# Why is THIS shard unassigned / why won't it move? Replays the decider chain.
curl -s 'localhost:9200/_cluster/allocation/explain?pretty' \
  -H 'content-type: application/json' \
  -d '{"index":"idx","shard":0,"primary":false}'

# Manually move / allocate / cancel (dry_run first!)
curl -s 'localhost:9200/_cluster/reroute?dry_run=true&pretty' \
  -H 'content-type: application/json' \
  -d '{"commands":[{"move":{"index":"idx","shard":0,"from_node":"nodeA","to_node":"nodeB"}}]}'

allocation/explain output for an unassigned shard lists, per node, every decider's decision and a human explanation. A red cluster's root cause is usually one decider saying NO (e.g., DiskThresholdDecider: "the node is above the high watermark") — read the explanation, fix the underlying condition, and the next reroute assigns it.

grep -rn "TransportClusterAllocationExplainAction\|ClusterAllocationExplanation" \
  server/src/main/java/org/opensearch/action/admin/cluster/allocation/
grep -rn "AllocationCommand\|MoveAllocationCommand\|AllocateEmptyPrimaryAllocationCommand" \
  server/src/main/java/org/opensearch/cluster/routing/allocation/command/

Warning: allocate_empty_primary and allocate_stale_primary discard data — they tell the cluster to accept an empty or stale copy as the new primary. They are last-resort recovery commands for "all good copies are permanently gone," never routine. dry_run everything first.

Reading exercise

# 1. The reroute entry and how started/failed shards feed back
grep -n "public ClusterState reroute\|applyStartedShards\|applyFailedShards\|deassociateDeadNodes" \
  server/src/main/java/org/opensearch/cluster/routing/allocation/AllocationService.java

# 2. The balancer's three jobs
grep -n "allocateUnassigned\|moveShards\|balanceByWeights\|weight(" \
  server/src/main/java/org/opensearch/cluster/routing/allocation/allocator/BalancedShardsAllocator.java

# 3. One decider in detail
sed -n '1,120p' server/src/main/java/org/opensearch/cluster/routing/allocation/decider/DiskThresholdDecider.java

# 4. Why-unassigned bookkeeping
grep -n "enum Reason\|AllocationStatus\|failedAllocations" \
  server/src/main/java/org/opensearch/cluster/routing/UnassignedInfo.java

Answer:

1. List the three phases inside BalancedShardsAllocator.allocate and what each accomplishes.
2. For a move to occur, how many deciders must say YES? What does THROTTLE mean vs NO, and which decider issues THROTTLE most often?
3. Name the decider that prevents a primary and replica from co-locating, and explain why co-location would defeat the purpose of replication.
4. Why does MaxRetryAllocationDecider exist? What would happen without it for a shard whose underlying disk is corrupt?
5. After a rolling restart, all replicas are unassigned and the cluster is yellow. Which setting/decider is the likely cause and what's the one-line fix?
6. Explain what allocate_stale_primary does and why it is a data-loss operation, citing in-sync allocation IDs from Replication.

Common bugs and symptoms

Validation: prove you understand this

1. Draw the allocation round (trigger → RoutingAllocation/RoutingNodes → existing-shards allocator → balancer's three phases → deciders → new RoutingTable), and state where the result re-enters the immutable ClusterState.
2. For each of SameShard, DiskThreshold, Awareness, Filter, and MaxRetry deciders, name the production incident it prevents in one sentence.
3. Given a red cluster, write the exact allocation/explain request for the primary of idx shard 0 and describe how you'd read the result to find the blocking decider.
4. Explain the rolling-restart enable: primaries pattern: what it prevents during the restart and the failure mode if you forget to revert it.
5. Distinguish move, allocate_replica, allocate_empty_primary, and allocate_stale_primary reroute commands by their data-safety, and say which two can lose data.
6. Describe how GatewayAllocator decides which node should host the primary of an index that already has on-disk data, and how that ties to in-sync allocation IDs.