Searchable Snapshots

A snapshot is a durable copy of your shards sitting in cheap blob storage. Normally, to query one you must restore it: download every segment back onto local disk and open a full shard. That is fine for disaster recovery and hopeless for economics — if you have a year of daily indices you rarely touch, you cannot afford hot storage for all of them, and you cannot afford to restore an index just to answer one query against it.

Searchable snapshots break that trade-off. They mount a snapshot as a read-only searchable index whose bytes stay in the repository, pulling and caching only the file ranges a query actually reads. You keep a warm/cold tier searchable at a fraction of the storage cost, paying a latency premium on the first touch instead of a full restore up front.

Naming warning — read this before you grep. In this codebase (v3.8.0) the feature is implemented under the name remote_snapshot, not "searchable_snapshot". The classic tutorials name SearchableSnapshotDirectory and SearchableSnapshotIndexInput; those symbols do not exist here. They were renamed. The real classes are RemoteSnapshotDirectory and OnDemandBlockSnapshotIndexInput. Grep for remote_snapshot, not searchable_snapshot, or you will find nothing.

After this chapter you can:

• Explain why a searchable snapshot is not a restore, and where its bytes live.
• Create one via _restore with storage_type: remote_snapshot, and say why index.store.type: remote_snapshot is system-managed.
• Trace the read path: openInput → OnDemandBlockSnapshotIndexInput.fetchBlock → BlobFetchRequest → TransferManager.fetchBlob → FileCache → repository.
• Describe FileCache eviction, the node.search.cache.size budget, and why these shards only allocate to warm nodes.
• Contrast it crisply with a normal restore and with remote-backed storage.

The cast

find server/src/main/java/org/opensearch/index/store/remote -name "*.java" | sort
grep -rn "remote_snapshot\|REMOTE_SNAPSHOT" server/src/main/java --include="*.java" | head

The problem: a restore you cannot afford

Trace how a normal restore lands data on local disk — a searchable snapshot deliberately does not do this:

grep -n "class RestoreService\|SnapshotRecoverySource\|isRemoteSnapshot" \
  server/src/main/java/org/opensearch/snapshots/RestoreService.java

A classic restore creates shards with a SnapshotRecoverySource; recovery then downloads every segment blob into the shard's Store before the shard can start. Restore time and local disk both scale with the total index size, whether or not you ever read those bytes.

A searchable snapshot flips the default. The shard opens immediately — its Directory is a view over the repository, and the disk footprint is only whatever byte ranges queries have touched so far, bounded by a fixed local cache. You trade storage cost and restore time for a per-block fetch latency on cache misses.

Creating one: restore with storage_type: remote_snapshot

There is no separate "mount" API. You reuse the snapshot restore endpoint and set the storage type. Find the selector:

grep -n "enum StorageType\|REMOTE_SNAPSHOT\|storage_type\|storageType" \
  server/src/main/java/org/opensearch/action/admin/cluster/snapshots/restore/RestoreSnapshotRequest.java

RestoreSnapshotRequest.StorageType has exactly two values, defaulting to LOCAL:

// RestoreSnapshotRequest.StorageType
public enum StorageType {
    LOCAL("local"),
    REMOTE_SNAPSHOT("remote_snapshot");
    // ...
}
private StorageType storageType = StorageType.LOCAL;

The REST body carries it as storage_type (parsed in RestoreSnapshotRequest; the endpoint is the same one described in Snapshots and Repositories):

curl -s -XPOST 'localhost:9200/_snapshot/my_repo/snap1/_restore' \
  -H 'content-type: application/json' \
  -d '{"indices":"logs-2024-01","storage_type":"remote_snapshot"}'

Inside RestoreService, that flag drives everything:

// RestoreService (restore cluster-state update)
final boolean isRemoteSnapshot =
    IndexModule.Type.REMOTE_SNAPSHOT.match(request.storageType().toString());
// ...
if (isRemoteSnapshot) {
    snapshotIndexMetadata = addSnapshotToIndexSettings(snapshotIndexMetadata, snapshot, snapshotIndexId);

When isRemoteSnapshot, the service stamps the new index with index.store.type: remote_snapshot and the snapshot/repository coordinates it will read from — that is what addSnapshotToIndexSettings records. You must not set the store type yourself. Both RestoreService and MetadataCreateIndexService reject a hand-set index.store.type: remote_snapshot:

grep -n "remote_snapshot\|Store type can be set" \
  server/src/main/java/org/opensearch/cluster/metadata/MetadataCreateIndexService.java

MetadataCreateIndexService: "Store type can be set to remote_snapshot only when restoring a remote snapshot by using storage_type: remote_snapshot." The store type is system-managed; the only supported door is the restore request.

The architecture: a Directory over the repository

The store type resolves to a Directory factory. Find the wiring:

grep -n "REMOTE_SNAPSHOT\|RemoteSnapshotDirectoryFactory\|newDirectory" \
  server/src/main/java/org/opensearch/index/IndexModule.java

IndexModule.Type.REMOTE_SNAPSHOT ("remote_snapshot") maps to RemoteSnapshotDirectoryFactory, which implements IndexStorePlugin.DirectoryFactory and is handed the cluster's RepositoriesService and the node's FileCache:

// RemoteSnapshotDirectoryFactory implements IndexStorePlugin.DirectoryFactory
public Directory newDirectory(IndexSettings indexSettings, ShardPath localShardPath) throws IOException {
    // ...
    final Repository repository = repositoriesService.get().repository(repositoryName);
    // ...
    final BlobContainer blobContainer = blobStoreRepository.shardContainer(indexId, shardId);
    TransferManager transferManager =
        new TransferManager(blobContainer::readBlob, remoteStoreFileCache, threadPool);
    // ... -> new RemoteSnapshotDirectory(snapshot, localStoreDir, transferManager)
}

Read that carefully: the factory resolves the repository, reads the shard's BlobStoreIndexShardSnapshot (the per-shard file list you met in Snapshots and Repositories), and builds a TransferManager whose downloader is blobContainer::readBlob. The RemoteSnapshotDirectory then serves Lucene from that snapshot's file map:

grep -n "class RemoteSnapshotDirectory\|openInput\|fileInfoMap\|OnDemandBlockSnapshotIndexInput" \
  server/src/main/java/org/opensearch/index/store/remote/directory/RemoteSnapshotDirectory.java

// RemoteSnapshotDirectory (a Lucene Directory)
private final Map<String, BlobStoreIndexShardSnapshot.FileInfo> fileInfoMap;
private final TransferManager transferManager;

public IndexInput openInput(String name, IOContext context) throws IOException {
    final BlobStoreIndexShardSnapshot.FileInfo fileInfo = fileInfoMap.get(name);
    // ...
    return new OnDemandBlockSnapshotIndexInput(fileInfo, localStoreDir, transferManager);
}

There is no download in openInput. It hands back a lazy IndexInput. Nothing is fetched until Lucene actually reads a byte. If the Directory / IndexInput abstraction feels unfamiliar, work the storage-engine masterclass first — a searchable snapshot is just a Directory implementation with an unusual backing store.

flowchart TD
    IS["IndexShard (STARTED, read-only)"] --> Dir["RemoteSnapshotDirectory (Lucene Directory)"]
    Dir -->|openInput name| II["OnDemandBlockSnapshotIndexInput (lazy)"]
    II -->|per 8 MiB block| TM["TransferManager.fetchBlob"]
    TM --> FC["FileCache (bounded local scratch)"]
    FC -->|miss| BC["BlobContainer.readBlob"]
    BC --> Repo[(Repository: S3 / fs / GCS / Azure)]
    FC -->|hit| II

The read path: block-level lazy fetch

This is the heart of the feature. A searchable-snapshot IndexInput never loads a whole segment file; it slices each file into fixed blocks and fetches a block only when a read crosses into it.

grep -n "abstract IndexInput fetchBlock\|DEFAULT_BLOCK_SIZE_SHIFT\|currentBlockId" \
  server/src/main/java/org/opensearch/index/store/remote/file/AbstractBlockIndexInput.java
grep -n "protected IndexInput fetchBlock\|BlobFetchRequest\|transferManager" \
  server/src/main/java/org/opensearch/index/store/remote/file/OnDemandBlockSnapshotIndexInput.java

AbstractBlockIndexInput owns the block bookkeeping and declares the abstract fetchBlock(int blockId). The default block is 8 MiB (DEFAULT_BLOCK_SIZE_SHIFT = 23, 1 << 23). OnDemandBlockSnapshotIndexInput implements fetchBlock by building a BlobFetchRequest for that block's byte range and handing it to the TransferManager:

// OnDemandBlockSnapshotIndexInput.fetchBlock
protected IndexInput fetchBlock(int blockId) throws IOException {
    final String blockFileName = getBlockFileName(fileName, blockId);
    final long blockStart = getBlockStart(blockId);
    final long blockEnd = blockStart + getActualBlockSize(blockId, blockSizeShift, originalFileSize);
    BlobFetchRequest blobFetchRequest = BlobFetchRequest.builder()
        .blobParts(getBlobParts(blockStart, blockEnd))  // may span multiple snapshot part-blobs
        .directory(directory)
        .fileName(blockFileName)
        .build();
    return transferManager.fetchBlob(blobFetchRequest);
}

Note getBlobParts: a snapshot may have stored a large file as multiple part blobs (fileInfo.partName(...)), so a single logical block can require reads from more than one blob. The BlobFetchRequest carries the list of BlobParts (blobName, position, length).

TransferManager.fetchBlob is where cache-or-fetch is decided:

grep -n "class TransferManager\|fetchBlob\|fileCache.compute\|DelayedCreationCachedIndexInput\|decRef" \
  server/src/main/java/org/opensearch/index/store/remote/utils/TransferManager.java

// TransferManager.fetchBlob
final Path key = blobFetchRequest.getFilePath();          // the local block-file path
CachedIndexInput cacheEntry = fileCache.compute(key, (path, cachedIndexInput) -> {
    if (cachedIndexInput == null || cachedIndexInput.isClosed()) {
        // Miss: create a lazy entry that will download this block from the repo on first use
        return new DelayedCreationCachedIndexInput(fileCache, streamReader, blobFetchRequest);
    } else {
        return cachedIndexInput;                          // Hit: reuse the cached block
    }
});
try {
    return cacheEntry.getIndexInput().clone();            // triggers the download on a miss
} finally {
    fileCache.decRef(key);                               // release our ref; keeps the block pinned only while in use
}

The streamReader is the blobContainer::readBlob the factory wired in — a StreamReader functional interface (name, position, length) -> InputStream. On a miss, DelayedCreationCachedIndexInput downloads the requested range, writes it to a local block file under the shard path, and returns a FileCachedIndexInput over it. On a hit, no repository call happens at all.

sequenceDiagram
    participant L as Lucene read (query/agg)
    participant II as OnDemandBlockSnapshotIndexInput
    participant TM as TransferManager
    participant FC as FileCache
    participant BC as BlobContainer.readBlob
    participant R as Repository
    L->>II: readByte / readBytes (crosses block boundary)
    II->>II: fetchBlock(blockId) -> BlobFetchRequest(range)
    II->>TM: fetchBlob(request)
    TM->>FC: compute(blockPath)
    alt cache miss
        FC-->>TM: DelayedCreationCachedIndexInput
        TM->>BC: streamReader(part, position, length)
        BC->>R: GET byte range
        R-->>BC: bytes
        BC-->>FC: write block file, cache it
    else cache hit
        FC-->>TM: existing FileCachedIndexInput
    end
    TM-->>II: IndexInput.clone()
    II-->>L: bytes

The FileCache: bounded local scratch, ref-counted eviction

Downloaded blocks land in a bounded local cache. If it were unbounded a searchable snapshot would just be a slow restore. Read its contract:

grep -n "class FileCache\|RefCountedCache\|SegmentedCache\|compute\|incRef\|decRef\|prune\|capacity" \
  server/src/main/java/org/opensearch/index/store/remote/filecache/FileCache.java
grep -n "interface RefCountedCache\|incRef\|decRef\|evict\|prune\|pin" \
  server/src/main/java/org/opensearch/index/store/remote/utils/cache/RefCountedCache.java

FileCache implements RefCountedCache<Path, CachedIndexInput> and is backed by a SegmentedCache (segment-striped LRU). The key facts:

• Reference counting. Every block handed to a live query is incRef'd and decRef'd when released (TransferManager does exactly this around the clone()). A block with a non-zero ref count is in use and cannot be evicted mid-query.
• Eviction on capacity. When the cache reaches capacity it evicts entries with a zero ref count in LRU order to make room; prune() proactively drops all zero-ref entries. The cache is the working set, not a copy of the index.
• get promotes, does not insert. Lookups bump LRU priority; only put / compute add entries.

The cache size is the node's node.search.cache.size budget:

grep -n "NODE_SEARCH_CACHE_SIZE_SETTING\|node.search.cache.size\|isDedicatedWarmNode" \
  server/src/main/java/org/opensearch/node/Node.java

// Node
public static final Setting<String> NODE_SEARCH_CACHE_SIZE_SETTING = new Setting<>(
    "node.search.cache.size",
    s -> (DiscoveryNode.isDedicatedWarmNode(s)) ? "80%" : "0",   // default
    Node::validateFileCacheSize,
    Property.NodeScope
);

The FileCache and its NodeCacheService are only created on warm nodes (if (DiscoveryNode.isWarmNode(settings)) in Node). A node without a warm role has a zero-size (effectively absent) file cache, so it cannot open a remote_snapshot shard at all.

Two more cluster-level guards live nearby:

grep -n "remote_data_ratio\|DATA_TO_FILE_CACHE_SIZE_RATIO_SETTING" \
  server/src/main/java/org/opensearch/index/store/remote/filecache/FileCacheSettings.java
grep -n "activeusage.threshold\|CLUSTER_FILECACHE_ACTIVEUSAGE" \
  server/src/main/java/org/opensearch/cluster/routing/allocation/FileCacheThresholdSettings.java

cluster.filecache.remote_data_ratio (FileCacheSettings, default 5.0, min 1.0) caps how much total remote data a node may reference as a multiple of its file-cache size — with a 100 GB cache and ratio 5, up to 500 GB of searchable snapshot may be mounted. It is a safeguard against oversubscription, not a performance knob.

Why these shards only land on warm nodes

A remote_snapshot index needs a file cache, so it must be pinned to warm nodes. That is RoutingPool + TargetPoolAllocationDecider:

grep -n "REMOTE_CAPABLE\|LOCAL_ONLY\|getIndexPool\|isRemoteSnapshot\|isWarmNode" \
  server/src/main/java/org/opensearch/cluster/routing/RoutingPool.java
grep -n "REMOTE_CAPABLE\|LOCAL_ONLY\|canAllocate" \
  server/src/main/java/org/opensearch/cluster/routing/allocation/decider/TargetPoolAllocationDecider.java

// RoutingPool
public static RoutingPool getIndexPool(IndexMetadata indexMetadata) {
    return indexMetadata.isRemoteSnapshot() || (/* writable-warm flag */ isWarmIndex(indexMetadata))
        ? REMOTE_CAPABLE
        : LOCAL_ONLY;
}
public static RoutingPool getNodePool(DiscoveryNode node) {
    return node.isWarmNode() ? REMOTE_CAPABLE : LOCAL_ONLY;
}

A remote_snapshot index is REMOTE_CAPABLE; only warm nodes are REMOTE_CAPABLE; TargetPoolAllocationDecider forbids a REMOTE_CAPABLE shard on a LOCAL_ONLY node. That is the whole enforcement.

Do not confuse this with the search role. DiscoveryNodeRole.SEARCH_ROLE ("search") is a different feature — it hosts search replicas (index.number_of_search_replicas, SearchReplicaAllocationDecider) for read scaling, and it must not be combined with other roles. Searchable snapshots live on the warm role (WARM_ROLE). The old tutorials that put searchable snapshots on a "search" node predate this split — verify with the greps above.

How it differs — three modes, side by side

Contrast crisply; contributors muddle these constantly:

The one-liner: remote-backed storage keeps a full, writable, hot copy local and mirrors it remotely for durability; a searchable snapshot keeps almost nothing local and reads a read-only index straight out of the repository on demand. See Snapshots and Repositories for the shared Repository/BlobContainer layer both build on.

There is also a newer index.store.data_locality setting (IndexModule, DataLocalityType.FULL / PARTIAL) and a warm tiering path (org.opensearch.storage.tiering) that generalizes this partial-locality idea to writable warm indices — grep server/src/main/java/org/opensearch/storage/ if you are working the newer tier.

Performance characteristics

• First-query latency is real. A cold query pays a repository round-trip per 8 MiB block it touches — including the ones Lucene reads just to open a searcher (segment infos, field infos, terms metadata). Expect the first query against a freshly mounted snapshot to be markedly slower than a hot index; steady-state is close once the working set is cached.
• Block granularity is a knob with two edges. Bigger blocks amortize round-trips but waste bandwidth and cache on random point reads; smaller blocks do the opposite. 8 MiB is the default (DEFAULT_BLOCK_SIZE_SHIFT).
• Cache thrash kills you. If the live working set of your queries exceeds node.search.cache.size, blocks get evicted and re-fetched repeatedly. Size the cache to the hot fraction of your cold tier, and watch remote_data_ratio.
• Warming. There is no full pre-warm; the cache fills as queries run. A representative query sweep after mount is the practical warm-up.

Inspect the cache live via node stats — it is exposed as a file_cache object (NodeStats / AggregateFileCacheStats):

grep -n "file_cache\|AggregateFileCacheStats" \
  server/src/main/java/org/opensearch/action/admin/cluster/node/stats/NodeStats.java
curl -s "localhost:9200/_nodes/stats/file_cache?pretty"

Where a contributor would touch it

Settings and REST

grep -rn "storage_type" rest-api-spec/src/main/resources/rest-api-spec/api/snapshot.restore.json

Common bugs and symptoms

Reading exercise

# 1. The store type -> directory factory wiring.
grep -n "REMOTE_SNAPSHOT\|RemoteSnapshotDirectoryFactory" \
  server/src/main/java/org/opensearch/index/IndexModule.java

# 2. The lazy directory and its IndexInput.
grep -n "openInput\|OnDemandBlockSnapshotIndexInput\|fileInfoMap" \
  server/src/main/java/org/opensearch/index/store/remote/directory/RemoteSnapshotDirectory.java

# 3. The block fetch + cache decision.
grep -n "fetchBlock\|BlobFetchRequest" \
  server/src/main/java/org/opensearch/index/store/remote/file/OnDemandBlockSnapshotIndexInput.java
grep -n "fetchBlob\|fileCache.compute\|DelayedCreationCachedIndexInput\|decRef" \
  server/src/main/java/org/opensearch/index/store/remote/utils/TransferManager.java

# 4. Cache bounds + allocation pinning.
grep -n "NODE_SEARCH_CACHE_SIZE_SETTING\|isWarmNode" server/src/main/java/org/opensearch/node/Node.java
grep -n "getIndexPool\|REMOTE_CAPABLE" server/src/main/java/org/opensearch/cluster/routing/RoutingPool.java

Answer:

1. Why is a searchable snapshot not a restore? Name the class that serves Lucene, and say where the shard's bytes actually live.
2. Which request field creates one, and why does OpenSearch reject a hand-set index.store.type: remote_snapshot?
3. Walk a single readByte from OnDemandBlockSnapshotIndexInput through to a repository GET, naming each class and where the cache hit/miss is decided. What is the default block size?
4. Explain FileCache eviction: what does reference counting protect, and what happens when the cache is full and every entry is in use?
5. Why can a remote_snapshot shard only allocate to a warm node? Name the pool and the decider — and explain how the warm role differs from the search role.
6. Contrast searchable snapshots with remote-backed storage in one sentence each: what is local, what is writable, what is the source of truth?

Validation: prove you understand this

1. Draw the object stack from IndexShard down to the repository: RemoteSnapshotDirectory → OnDemandBlockSnapshotIndexInput → TransferManager → FileCache / BlobContainer. Say which arrow is lazy.
2. Reproduce the read-path sequence diagram from memory, marking exactly where a repository round-trip happens and where it is skipped.
3. Explain the FileCache as a bounded working set: reference counting, LRU eviction of zero-ref entries, and the node.search.cache.size / remote_data_ratio budgets.
4. Explain the allocation story end to end: isRemoteSnapshot() → RoutingPool.REMOTE_CAPABLE → warm nodes → TargetPoolAllocationDecider.
5. Given a cold-tier workload, size node.search.cache.size and defend it against cache thrash; predict first-query vs steady-state latency.
6. State the crisp three-way contrast (normal restore / searchable snapshot / remote-backed store) on: data of record, writability, local footprint, and time-to-searchable. Then note the one symbol you would grep for that the old docs get wrong.