IndexShard Lifecycle

A shard is the unit of storage and the unit of work in OpenSearch. Every document lives in exactly one shard; every search runs per-shard; every write applies to a shard's primary first. The Java object that owns a shard on a data node is IndexShard, and this chapter is about its full lifecycle: how the hierarchy IndicesService → IndexService → IndexShard is built, the shard state machine (IndexShardState), the Store/Directory that backs it, how a shard is created, recovered, started, and closed, and how index/delete operations are applied on the primary versus a replica. The shard is where the cluster layer (routing table) meets the storage layer (the engine, the translog, and refresh/flush/merge).

After this chapter you should be able to: draw the IndexShardState machine; explain who creates and starts shards in response to cluster-state changes; find applyIndexOperationOnPrimary/OnReplica and explain why they differ; and name the components an IndexShard owns.

Note: Distinguish IndexShardState (the data node's reality for the shard object) from ShardRouting.state() (the cluster manager's plan for the shard, in cluster state — see cluster-and-node-model.md). Both have states; they are not the same machine.

The hierarchy: IndicesService → IndexService → IndexShard

find server -name "IndicesService.java" -o -name "IndexService.java" -o -name "IndexShard.java"
grep -n "class IndicesService\|createIndex\|indexService(" \
  server/src/main/java/org/opensearch/indices/IndicesService.java
grep -n "class IndexService\|createShard\|removeShard\|getShard" \
  server/src/main/java/org/opensearch/index/IndexService.java

flowchart TD
    Node[Node] --> IsS[IndicesService]
    IsS --> IxS1["IndexService 'orders'"]
    IsS --> IxS2["IndexService 'logs'"]
    IxS1 --> MS[MapperService]
    IxS1 --> Sh0["IndexShard 0 (primary)"]
    IxS1 --> Sh1["IndexShard 1 (replica)"]
    Sh0 --> Eng[InternalEngine]
    Eng --> IW[Lucene IndexWriter]
    Eng --> TL[Translog]
    Sh0 --> St[Store -> Directory]

The bridge from cluster state to these objects is IndicesClusterStateService, an applier (see cluster-state-publishing.md). When the routing table assigns a shard to this node, the applier asks IndexService.createShard(...); when it un-assigns one, the applier removes it.

grep -n "createShard\|removeShard\|failAndRemoveShard\|createOrUpdateShards" \
  server/src/main/java/org/opensearch/indices/cluster/IndicesClusterStateService.java

The Store and Directory

The bytes of a shard live on disk under the node's data path, wrapped by Store, which holds a Lucene Directory.

find server -name "Store.java" -path "*index/store*"
grep -n "class Store\|Directory\|directory()\|verify\|MetadataSnapshot\|checkIntegrity" \
  server/src/main/java/org/opensearch/index/store/Store.java

Store is reference-counted: a shard can be in use by a search while it is being closed; the refcount keeps the directory alive until all users release it. This is why a shard close is not instantaneous.

The IndexShardState machine

find server -name "IndexShardState.java"
grep -n "CREATED\|RECOVERING\|POST_RECOVERY\|STARTED\|CLOSED\|enum IndexShardState" \
  server/src/main/java/org/opensearch/index/shard/IndexShardState.java
grep -n "changeState\|state =\|IndexShardState\." \
  server/src/main/java/org/opensearch/index/shard/IndexShard.java | head

stateDiagram-v2
    [*] --> CREATED: IndexService.createShard
    CREATED --> RECOVERING: recovery starts (markAsRecovering)
    RECOVERING --> POST_RECOVERY: engine opened, ops replayed
    POST_RECOVERY --> STARTED: shard marked started, serves traffic
    STARTED --> CLOSED: node leaves / shard removed / index closed
    RECOVERING --> CLOSED: recovery failed
    POST_RECOVERY --> CLOSED: failure
    CREATED --> CLOSED: aborted

The recovery source that fills a RECOVERING shard depends on context:

Creating, recovering, starting, closing

sequenceDiagram
    participant ICSS as IndicesClusterStateService (applier)
    participant IxS as IndexService
    participant Sh as IndexShard
    participant Eng as Engine
    participant CM as Cluster manager
    ICSS->>IxS: createShard(shardRouting) [routing assigned here]
    IxS->>Sh: new IndexShard (CREATED)
    ICSS->>Sh: startRecovery(...)
    Sh->>Sh: markAsRecovering (RECOVERING)
    Sh->>Eng: openEngineAndRecoverFromTranslog / fillFromSource
    Eng-->>Sh: ops replayed (POST_RECOVERY)
    Sh->>CM: shard-started action
    CM->>CM: cluster-state update: ShardRouting -> STARTED
    Note over Sh: applier observes STARTED -> IndexShard STARTED

Trace the key methods:

grep -n "markAsRecovering\|recoverFromStore\|openEngineAndRecover\|postRecovery\|markShardAsStarted\|close(" \
  server/src/main/java/org/opensearch/index/shard/IndexShard.java | head -30

Note the round-trip: the data node reports the shard started (ShardStateAction → a cluster-state update on the manager), and only then does the routing table mark the ShardRouting STARTED. The two state machines converge through this handshake.

grep -n "shardStarted\|class ShardStateAction\|sendShardStarted" \
  server/src/main/java/org/opensearch/cluster/action/shard/ShardStateAction.java

Applying operations: primary vs replica

A write reaches a shard via TransportReplicationAction (action-framework.md). The shard has two application paths, and the difference is fundamental:

grep -n "applyIndexOperationOnPrimary\|applyIndexOperationOnReplica\|applyDeleteOperationOnPrimary\|\
applyDeleteOperationOnReplica\|markSeqNoAsNoop" \
  server/src/main/java/org/opensearch/index/shard/IndexShard.java

flowchart TD
    Op[index request] --> Pr{primary or replica?}
    Pr -->|primary| AP[applyIndexOperationOnPrimary]
    AP --> Seq[assign seq no + primary term]
    Seq --> Ver[resolve version conflict]
    Ver --> Eng1["InternalEngine.index() -> IndexWriter + Translog.add"]
    Eng1 --> Repl[replicate to in-sync replicas]
    Repl --> AR[applyIndexOperationOnReplica on each]
    AR --> Eng2["InternalEngine.index() at given seq no (idempotent)"]

Why the split matters: the primary decides (seq no, version, conflict resolution); replicas obey. This makes replication deterministic and lets a recovering replica replay the primary's exact operation history. The sequence number machinery (LocalCheckpointTracker, global checkpoint via ReplicationTracker) is detailed in engine-internals.md and replication.md.

Warning: A replica must apply operations idempotently at the seq no the primary assigned. If a replica re-derived its own seq no or version, replicas would diverge from the primary — a data-correctness bug, not a performance one.

What IndexShard owns and exposes

The shard is the integration point. The hooks you will meet:

IndexEventListener is the extension seam: plugins (and core services) get notified on beforeIndexShardCreated, afterIndexShardStarted, beforeIndexShardClosed, etc.

find server -name "IndexEventListener.java"
grep -n "default void\|afterIndexShardStarted\|beforeIndexShardClosed\|onShardInactive" \
  server/src/main/java/org/opensearch/index/shard/IndexEventListener.java

Reading exercise

# 1. The state machine.
grep -n "IndexShardState\|changeState\|verifyActive\|verifyNotClosed" \
  server/src/main/java/org/opensearch/index/shard/IndexShard.java | head

# 2. Create/recover/start path.
grep -n "createShard\|startRecovery\|markAsRecovering\|recoverFromStore" \
  server/src/main/java/org/opensearch/index/IndexService.java \
  server/src/main/java/org/opensearch/index/shard/IndexShard.java | head

# 3. Primary vs replica application.
grep -n "applyIndexOperationOnPrimary\|applyIndexOperationOnReplica" \
  server/src/main/java/org/opensearch/index/shard/IndexShard.java

# 4. The started handshake.
grep -n "sendShardStarted\|shardStarted" \
  server/src/main/java/org/opensearch/cluster/action/shard/ShardStateAction.java

# 5. Tests + live view.
./gradlew :server:test --tests "org.opensearch.index.shard.IndexShardTests"
curl -s "localhost:9200/_cat/shards?v&h=index,shard,prirep,state,docs,node"

Answer:

1. Name the three levels of the IndicesService → IndexService → IndexShard hierarchy and one thing each level owns.
2. Draw IndexShardState from CREATED to STARTED and CLOSED. In which states can the shard serve traffic?
3. Who calls IndexService.createShard, and what cluster-state change triggers it?
4. Explain the started handshake: how does a data node's STARTED shard cause the routing table's ShardRouting to become STARTED?
5. Contrast applyIndexOperationOnPrimary and applyIndexOperationOnReplica. Which assigns the sequence number, and why must the other be idempotent?
6. What is IndexEventListener for, and name two lifecycle callbacks a plugin could hook?

Common bugs and symptoms

Validation: prove you understand this

1. Draw the full object graph from IndicesService down to the Lucene IndexWriter and Translog, naming each owning class.
2. Reproduce the IndexShardState diagram from memory and annotate which states serve reads/writes and which recovery source fills RECOVERING.
3. Explain the started handshake between a data node and the cluster manager, and why two state machines (IndexShardState, ShardRouting.state) exist.
4. Explain precisely why the primary assigns sequence numbers and replicas do not, and the data-correctness bug that the asymmetry prevents.
5. Explain why Store is reference-counted and what a missing release causes.
6. List two IndexEventListener callbacks and a realistic use for each (e.g. a plugin warming caches on afterIndexShardStarted).