Snapshots and Repositories

A snapshot is OpenSearch's backup-and-restore mechanism, but it is far more interesting than "copy the files somewhere." It is incremental (a new snapshot only stores segments that changed since the last one), repository-backed (a pluggable blob store: a shared filesystem, S3, GCS, Azure), and the foundation for forward-looking features like searchable snapshots and remote-backed storage. A contributor touching this code is one mistake away from a corrupt backup, so understanding the data model — shard-level blobs, RepositoryData, generations — is non-negotiable.

This chapter maps SnapshotsService, RepositoriesService, the Repository / BlobStoreRepository / BlobContainer abstraction, the incremental shard format, and the restore flow. It connects to Refresh, Flush, and Merge (snapshots capture committed segments), Recovery (restore is a snapshot recovery), and Plugin Architecture (repository-s3 is a RepositoryPlugin).

After this chapter you can:

• Explain how a snapshot is incremental at the segment level, not the index level.
• Trace the snapshot and restore flows from REST to BlobStoreRepository.
• Describe RepositoryData and why repository generations matter for consistency.
• Name the repository types and how a custom one is contributed.

The cast

ls server/src/main/java/org/opensearch/repositories/
ls server/src/main/java/org/opensearch/snapshots/
grep -n "class BlobStoreRepository\|snapshotShard\|restoreShard\|writeIndexGen\|getRepositoryData" \
  server/src/main/java/org/opensearch/repositories/blobstore/BlobStoreRepository.java

Repository types

Each non-fs type is a RepositoryPlugin that provides a Repository factory; the core engine never depends on the S3 SDK directly. See Plugin Architecture.

grep -rn "RepositoryPlugin\|getRepositories\|class S3Repository" \
  plugins/repository-s3/src/main/java/org/opensearch/repositories/s3/ | head
grep -rn "interface RepositoryPlugin\|getRepositories" \
  server/src/main/java/org/opensearch/plugins/RepositoryPlugin.java

Register a filesystem repo:

curl -s -XPUT 'localhost:9200/_snapshot/my_repo' \
  -H 'content-type: application/json' \
  -d '{"type":"fs","settings":{"location":"/mnt/backups","compress":true}}'

The data model: blobs, generations, RepositoryData

A repository is a flat blob namespace organized by convention:

<repo root>/
  index-N                 # RepositoryData for generation N (the source of truth)
  index.latest            # pointer to the current generation N
  snap-<uuid>.dat         # SnapshotInfo per snapshot
  meta-<uuid>.dat         # global metadata per snapshot
  indices/
    <indexUUID>/
      meta-<uuid>.dat      # index metadata
      0/                   # shard 0
        __<uuid>           # segment data blobs (the actual Lucene files, renamed)
        snap-<uuid>.dat    # this snapshot's shard-level file list
        index-<gen>        # BlobStoreIndexShardSnapshots: the shard's snapshot index
      1/ ...

RepositoryData (serialized as index-N) is the repository's master index: which snapshots exist, which indices each contains, and the mapping of index → shard generations. It is read at the start of every snapshot/restore and rewritten (bumping N) whenever the repository changes. index.latest names the current N.

grep -n "class RepositoryData\|EMPTY_REPO_GEN\|getGenId\|shardGenerations\|indexMetaDataGenerations" \
  server/src/main/java/org/opensearch/repositories/RepositoryData.java

Warning: Two cluster managers writing the same repository (e.g., the same S3 bucket registered in two clusters) corrupts RepositoryData — the generation bookkeeping assumes a single writer. This is why pointing two clusters at one repository is unsupported and a known foot-gun. Repository generation/UUID checks (SnapshotsService uses repository_uuid and pending_generations) exist precisely to detect this.

Why snapshots are incremental — at the segment level

Lucene segments are immutable once written (see Refresh, Flush, and Merge). A snapshot of a shard records the list of committed segment files and uploads only the segment blobs the repository doesn't already have (matched by file name + length + checksum from the previous snapshot's shard index). A second snapshot of an unchanged shard uploads nothing new — it just writes a new shard-level snap file referencing the existing blobs.

flowchart LR
    S1[Snapshot 1: shard has segments A,B,C] -->|upload A,B,C| R[(Repository)]
    M[merge: A,B -> D; new write -> E] --> S2[Snapshot 2: shard has C,D,E]
    S2 -->|C already in repo, upload D,E only| R

This is why the first snapshot of a large index is slow (everything uploads) and subsequent ones are fast (only deltas). It also means deleting a snapshot must reference-count blobs: a segment blob is only deleted when no remaining snapshot references it (BlobStoreRepository computes this during deleteSnapshots).

grep -n "snapshotShard\|incremental\|existingFiles\|FileInfo\|filesToSnapshot\|deleteSnapshots" \
  server/src/main/java/org/opensearch/repositories/blobstore/BlobStoreRepository.java

The snapshot flow

sequenceDiagram
    participant U as User
    participant CM as Cluster manager (SnapshotsService)
    participant DN as Data nodes (per shard)
    participant Repo as Repository (BlobStoreRepository)
    U->>CM: PUT _snapshot/repo/snap1 {indices}
    CM->>Repo: read RepositoryData (index-N)
    CM->>CM: cluster-state: SnapshotsInProgress entry
    Note over CM,DN: each shard snapshotted where its primary lives
    par per shard
        DN->>DN: acquire a committed Lucene index (engine snapshot)
        DN->>Repo: upload new segment blobs (incremental) + shard snap file
        DN-->>CM: shard snapshot SUCCESS/FAILED
    end
    CM->>Repo: write SnapshotInfo (snap-uuid.dat) + new RepositoryData (index-N+1)
    CM->>CM: remove SnapshotsInProgress entry
    CM-->>U: snapshot complete

Note the snapshot runs against a committed view (a Lucene commit / engine snapshot), on the SNAPSHOT thread pool, per shard on the node holding its primary. It does not block indexing — new writes land in new segments not part of this snapshot.

grep -n "class SnapshotsService\|SnapshotsInProgress\|beginSnapshot\|snapshotShard\|endSnapshot" \
  server/src/main/java/org/opensearch/snapshots/SnapshotsService.java

The restore flow

Restore is the inverse, and it routes through recovery:

flowchart TD
    U[POST _snapshot/repo/snap1/_restore] --> RS[RestoreService]
    RS --> CS[cluster-state: create indices with SnapshotRecoverySource]
    CS --> ALLOC[shards allocated unassigned, RecoverySource = SNAPSHOT]
    ALLOC --> REC[per shard: restore segments from repo via recovery]
    REC --> ST[shard STARTED]

Each restored shard becomes an unassigned shard with a SnapshotRecoverySource; shard allocation places it, and recovery performs a snapshot recovery (download segment blobs into the shard's store, then open it). You can rename indices on restore, restore a subset of indices, and restore with adjusted settings.

grep -n "class RestoreService\|SnapshotRecoverySource\|restoreSnapshot\|renamePattern" \
  server/src/main/java/org/opensearch/snapshots/RestoreService.java
curl -s -XPOST 'localhost:9200/_snapshot/my_repo/snap1/_restore' \
  -H 'content-type: application/json' \
  -d '{"indices":"idx","rename_pattern":"idx","rename_replacement":"idx_restored"}'

Forward-looking: searchable snapshots and remote-backed storage

The same blob model underpins newer capabilities you should know exist:

• Searchable snapshots — mount a snapshot as a read-only searchable index whose data lives in the repository, fetched and cached on demand rather than fully restored to local disk. Great for cold/warm tiers: keep old indices searchable without paying for hot storage.
• Remote-backed storage / remote store — index data (segments + translog) is continuously durably written to a remote repository, decoupling durability from local disk and enabling cheaper replicas and faster recovery (a remote-store shard recovers from the remote store rather than a peer).

Both reuse Repository/BlobContainer. Treat them as the strategic direction; classic snapshot/restore remains the baseline you must master first.

grep -rn "searchable_snapshot\|SearchableSnapshot\|remote_store\|RemoteStore\|remoteStoreEnabled" \
  server/src/main/java/org/opensearch/index/ server/src/main/java/org/opensearch/snapshots/ | head

Reading exercise

# 1. The repository abstraction surface
grep -n "void snapshotShard\|void restoreShard\|getRepositoryData\|writeIndexGen\|RepositoryMetadata" \
  server/src/main/java/org/opensearch/repositories/blobstore/BlobStoreRepository.java

# 2. Incremental shard upload
grep -n "snapshotShard\|existingFiles\|filesToSnapshot\|BlobStoreIndexShardSnapshot" \
  server/src/main/java/org/opensearch/repositories/blobstore/BlobStoreRepository.java

# 3. RepositoryData / generations
grep -n "getGenId\|shardGenerations\|index.latest\|INDEX_FILE_PREFIX\|resolveIndexName" \
  server/src/main/java/org/opensearch/repositories/RepositoryData.java

# 4. Restore -> recovery
grep -n "SnapshotRecoverySource\|restoreShard\|recoverFromRepository" \
  server/src/main/java/org/opensearch/snapshots/RestoreService.java

Answer:

1. At what granularity is a snapshot incremental — index, shard, or segment — and what property of Lucene segments makes that possible?
2. What does RepositoryData (the index-N blob) contain, and why does its generation number N increase on every snapshot/delete?
3. Why is registering one repository (e.g., one S3 bucket) in two clusters unsupported? What corruption does it cause and what check guards against it?
4. Trace a restore: name the recovery source type a restored shard gets and which subsystem actually downloads the segments.
5. When you delete a snapshot, why can't OpenSearch just delete that snapshot's blobs? What does it have to compute first?
6. How does repository-s3 avoid making the core engine depend on the AWS SDK? Name the plugin interface.

Common bugs and symptoms

Validation: prove you understand this

1. Draw the repository blob layout (index-N, index.latest, snap-*.dat, indices/<uuid>/<shard>/__*) and say which blob is the source of truth.
2. Explain segment-level incrementality with a two-snapshot example where a merge happened between them — say exactly which blobs the second snapshot uploads.
3. Define RepositoryData and the role of the generation N, and explain why a single-writer invariant is required.
4. Diagram the snapshot flow (cluster-manager orchestration + per-shard upload + final RepositoryData write) and the restore flow (down through recovery).
5. Explain what must happen before a snapshot's segment blobs can be safely deleted, and why naive deletion would corrupt other snapshots.
6. Describe, at a high level, how searchable snapshots and remote-backed storage reuse the same Repository/BlobContainer model and what they change about where shard data lives.