Lab ST3: Store Types and Memory Mapping

Background

The Storage Engine — Intensive chapter explained that index.store.type chooses how Lucene reads files — hybridfs (default) mmaps the hot random-access structures (.tim/.tip/.dvd/.cfs/.kdd/.vec/...) and reads the rest with positional I/O, while mmapfs mmaps everything and niofs mmaps nothing. In this lab you stop trusting the docs and see the memory mappings in the live process: you create indices with different store types and use pmap//proc/<pid>/smaps (Linux) or vmmap (macOS) to show the exact .cfs/.tim/.dvd/.vec regions mapped — and the .fdt (stored fields) not mapped under hybridfs. Then you experiment with index.store.preload to warm pages, and you reason about vm.max_map_count and the page-cache-vs-heap trade.

Why This Matters for Contributors

mmap is where OpenSearch's storage performance lives, and it is where a lot of production pain comes from: nodes that Map failed because vm.max_map_count is too low, nodes that read from disk on every query because the heap is too big and the page cache too small, and confusion about whether mmap "uses memory." A contributor who can pmap a node and explain every mapped region — and who knows the one sysctl that prevents the most common shard-open failure — debugs these in minutes instead of days.

Prerequisites

• A running OpenSearch node you can inspect (./gradlew run or a tarball install).
• Linux: pmap, cat /proc/<pid>/smaps. macOS: vmmap. Either way: curl, python3, jps/ps.
• Finished Lab ST1 (you can find a shard's index/).
• Read index.md, especially MMapDirectory vs NIOFSDirectory and vm.max_map_count.

Note: Memory-mapping is per-OS. The commands differ between Linux (pmap, smaps) and macOS (vmmap); the behavior (which extensions get mapped) is the same because it is decided in FsDirectoryFactory/HybridDirectory, not by the OS.

Step-by-Step Tasks

Step 1 — Confirm the hybridfs extension set in the source

Before observing it, see where the decision lives. In an OpenSearch checkout:

grep -n "index.store.type\|hybridfs\|HybridDirectory\|MMapDirectory\|NIOFSDirectory\|openInput\|getExtension\|preLoad\|Set\.of\|Set\.copyOf" \
  server/src/main/java/org/opensearch/index/store/FsDirectoryFactory.java

You are looking for the static final Set<String> (or similar) of extensions that HybridDirectory routes to mmap, and the openInput method that pulls the extension off the filename and chooses the mmap or NIO delegate. The set typically contains tim, tip, doc, dvd, kdd, kdi, kdm, cfs, vec, vex, vem, nvd, ... — grep yours; it changes across versions (vectors/points were added as those features matured).

Step 2 — Create three indices, one per store type

for T in hybridfs mmapfs niofs; do
  curl -s -XPUT "localhost:9200/store-$T?pretty" -H 'Content-Type: application/json' -d"
  { \"settings\": {
      \"index.number_of_shards\": 1,
      \"index.number_of_replicas\": 0,
      \"index.store.type\": \"$T\"
  }}" > /dev/null
  echo "created store-$T"
done

# Verify the setting took:
curl -s 'localhost:9200/store-*/_settings?filter_path=**.store.type&pretty'

Note: index.store.type is set at index creation and cannot be changed on a live index (it governs how the directory is opened). To change it you reindex into a new index with the desired type.

Step 3 — Index enough data to produce mappable files

for T in hybridfs mmapfs niofs; do
  for i in $(seq 1 5000); do
    printf '{"index":{}}\n{"customer":"c%d","amount":%d.5,"note":"order %d alpha bravo charlie"}\n' \
      "$((i % 100))" "$i" "$i"
  done | curl -s -H 'Content-Type: application/x-ndjson' \
    "localhost:9200/store-$T/_bulk?refresh=true" --data-binary @- > /dev/null
  # Force one non-compound segment so individual extensions exist and get mapped:
  curl -s -XPOST "localhost:9200/store-$T/_forcemerge?max_num_segments=1" >/dev/null
  curl -s -XPOST "localhost:9200/store-$T/_flush" >/dev/null
  echo "loaded store-$T"
done

# Run a query against each so the reader actually opens/maps the files:
for T in hybridfs mmapfs niofs; do
  curl -s "localhost:9200/store-$T/_search?q=note:alpha&size=0" >/dev/null
  curl -s "localhost:9200/store-$T/_search" -H 'Content-Type: application/json' \
    -d '{"size":0,"aggs":{"by_c":{"terms":{"field":"customer.keyword"}}}}' >/dev/null
done

Step 4 — Find the PID and the three shard dirs

PID=$(jps -l | grep -i 'org.opensearch.bootstrap.OpenSearch' | awk '{print $1}')
echo "node PID = $PID"

DATA=/your/checkout/server/build/run/data
for T in hybridfs mmapfs niofs; do
  U=$(curl -s "localhost:9200/_cat/indices/store-$T?h=uuid" | tr -d ' ')
  echo "store-$T -> $DATA/nodes/0/indices/$U/0/index"
done

Step 5 (Linux) — pmap the mapped segment files

# All file-backed mappings in the index dirs, with sizes:
pmap -x "$PID" | grep -E 'indices/.*/index/' | sort -k3 -n | tail -40

Expected shape — for the hybridfs index you see the hot extensions mapped and .fdt/.fdx (stored fields) absent:

Address           Kbytes     RSS   Dirty Mode  Mapping
00007f...           4096    2100       0 r--s- _3.cfs        # (small/compound) mapped
00007f...           8192    5000       0 r--s- _5.tim        # terms dict  -> MMAP
00007f...           2048    1200       0 r--s- _5.tip        # FST index   -> MMAP
00007f...           6144    3000       0 r--s- _5.doc        # postings    -> MMAP
00007f...           4096    2500       0 r--s- _5.dvd        # docvalues   -> MMAP
00007f...           1024     600       0 r--s- _5.kdd        # BKD points  -> MMAP
# NOTE: no _5.fdt / _5.fdx mapping here -> stored fields use NIO under hybridfs

Narrow it to specific extensions to make the contrast undeniable:

echo "=== mmapped extensions under hybridfs ==="
HU=$(curl -s 'localhost:9200/_cat/indices/store-hybridfs?h=uuid' | tr -d ' ')
pmap "$PID" | grep "indices/$HU/" | grep -oE '_[0-9]+\.[a-z]+' | sed 's/.*\.//' | sort | uniq -c
echo "=== is .fdt mapped? (expect NOTHING for hybridfs) ==="
pmap "$PID" | grep "indices/$HU/" | grep -E '\.fdt|\.fdx' || echo "  .fdt/.fdx NOT mmapped (NIO) — correct"

Now compare to mmapfs (everything, including .fdt, is mapped) and niofs (nothing file-backed is mapped):

MU=$(curl -s 'localhost:9200/_cat/indices/store-mmapfs?h=uuid' | tr -d ' ')
NU=$(curl -s 'localhost:9200/_cat/indices/store-niofs?h=uuid'  | tr -d ' ')

echo "=== mmapfs: .fdt SHOULD be mapped ==="
pmap "$PID" | grep "indices/$MU/" | grep -E '\.fdt' && echo "  .fdt mmapped — correct for mmapfs"

echo "=== niofs: NO segment files mapped ==="
pmap "$PID" | grep "indices/$NU/" | grep -E '_[0-9]+\.' || echo "  no segment mappings — correct for niofs"

For richer detail (resident set, dirty pages) read smaps:

grep -A3 "indices/$HU/.*\.tim" /proc/$PID/smaps | head -20
#   Size:   8192 kB   <- virtual size of the mapping
#   Rss:    5000 kB   <- how much is resident in the page cache right now
#   Shared_Clean: ... <- shared with the page cache, clean (read-only mmap)

Step 5 (macOS) — vmmap instead

# macOS uses vmmap; filter for the index dir:
sudo vmmap "$PID" 2>/dev/null | grep -E 'indices/.*/index/' | head -40

# Which extensions are mapped under hybridfs:
HU=$(curl -s 'localhost:9200/_cat/indices/store-hybridfs?h=uuid' | tr -d ' ')
sudo vmmap "$PID" 2>/dev/null | grep "indices/$HU/" | grep -oE '_[0-9]+\.[a-z]+' | sed 's/.*\.//' | sort | uniq -c
sudo vmmap "$PID" 2>/dev/null | grep "indices/$HU/" | grep -E '\.fdt' \
  || echo "  .fdt NOT mapped under hybridfs (NIO) — correct"

vmmap labels file-backed regions mapped file with the path; the same hybridfs-maps-the-hot-files / niofs-maps-nothing contrast holds.

Step 6 — Experiment with index.store.preload

By default mmap pages fault in lazily. Preload warms chosen extensions on shard open. Set it and reopen the shard (close/open the index reopens the directory):

curl -s -XPUT 'localhost:9200/store-hybridfs/_settings' -H 'Content-Type: application/json' -d'
{ "index.store.preload": ["tim","tip","dvd","kdd"] }' | python3 -m json.tool

# Reopen so the directory is recreated with preload active:
curl -s -XPOST 'localhost:9200/store-hybridfs/_close' >/dev/null
curl -s -XPOST 'localhost:9200/store-hybridfs/_open?wait_for_active_shards=1' >/dev/null

Now compare the resident (Rss) bytes of the preloaded extensions right after open — preloaded mappings should be (near) fully resident immediately, before any query touches them:

HU=$(curl -s 'localhost:9200/_cat/indices/store-hybridfs?h=uuid' | tr -d ' ')
for ext in tim dvd; do
  echo "=== .$ext resident after preload+open ==="
  grep -A1 "indices/$HU/.*\.$ext" /proc/$PID/smaps | grep -E 'Size|Rss'
done

Expected: for a preloaded extension, Rss ≈ Size immediately on open (the pages were MemorySegment.load()-ed). Without preload, Rss starts near 0 and grows as queries fault pages in.

Warning: index.store.preload: ["*"] preloads every file on every shard open — it reads your entire index off disk at startup. That is rarely what you want; it lengthens shard-open and wastes page cache on files the workload never touches. Preload only the structures your hot queries need (tim/tip/dvd/kdd/vec).

Step 7 — Inspect vm.max_map_count and the mapping budget

# Linux: the per-process VMA limit and current count:
sysctl vm.max_map_count
wc -l < /proc/$PID/maps          # current number of mappings this node holds

# How many of those are OpenSearch index files:
grep -c 'indices/.*/index/' /proc/$PID/maps

Each mmapped file (and each large file is split into multiple mappings) consumes VMAs. With many shards × many segments × many mmapped extensions, a node can approach the limit. OpenSearch's bootstrap check requires ≥ 262144:

# Raise it (Linux) — required for production mmapfs/hybridfs nodes:
sudo sysctl -w vm.max_map_count=262144
echo 'vm.max_map_count=262144' | sudo tee -a /etc/sysctl.conf

Step 8 — Reason about the page-cache vs heap trade

The mapped pages live in OS page cache (off-heap), reclaimable by the kernel. Confirm that mmap is not on the JVM heap:

# JVM heap usage (should NOT grow with the size of mmapped index files):
curl -s 'localhost:9200/_nodes/stats/jvm?filter_path=**.mem.heap_used_in_bytes&pretty'

# Free RAM = your page cache budget; mmap needs it:
free -h    # (Linux)   look at "available"

The lesson, stated as a rule:

This is the single most important operational consequence of the store engine: mmap wants free RAM, not heap.

Deliverables

• pmap/vmmap output showing, for hybridfs, the mapped .cfs/.tim/.tip/.doc/.dvd/.kdd and the absent .fdt/.fdx.
• The contrast: mmapfs maps .fdt too; niofs maps no segment files.
• smaps Size/Rss for a preloaded extension showing near-full residency on open, vs a non-preloaded one starting cold.
• sysctl vm.max_map_count and the count of index-file mappings the node holds.
• A short written rule for heap-vs-page-cache sizing, justified from your measurements.

Troubleshooting

Expected Output (the headline)

# hybridfs index, mmapped extensions:
   1 cfs   1 dvd   1 doc   1 kdd   1 tim   1 tip   1 nvd
# hybridfs .fdt/.fdx:  NOT mmapped (NIO) — correct
# mmapfs  .fdt:        mmapped — correct
# niofs   segment files: none mapped — correct

Stretch Goals

• Drop the page cache (echo 1 | sudo tee /proc/sys/vm/drop_caches), run a cold query against the hybridfs index, and time it; then run it again (warm) and compare. Repeat with index.store.preload set and observe the cold query already warm.
• Add a knn_vector field and confirm .vec/.vex get mmapped under hybridfs (k-NN graph traversal is pure random access) — link HNSW vector search and native JNI and memory.
• Benchmark niofs vs hybridfs on a sort/agg-heavy query with a cold cache; quantify the syscall+copy overhead of NIO on the random-access .dvd/.tim.
• Count VMAs as you add shards (10, 50, 100 small indices) and extrapolate how many shards a node with the default vm.max_map_count could open under mmapfs.

Coding Exercises

You observed the mappings; now write code that asserts which extensions hybridfs routes to mmap, benchmarks the store types, and reproduces the Map failed failure mode — the literacy a contributor needs to fix store-layer issues.

1. (warm-up) A mapping classifier in Python. Write classify_mmaps.py that reads /proc/<pid>/smaps (or parses vmmap on macOS), filters file-backed mappings under indices/.../index/, and prints, per index, the set of mmapped extensions plus Rss/Size per mapping. Assert the hybridfs invariant in code: .tim/.tip/.doc/.dvd/.kdd present, .fdt/.fdx absent; mmapfs includes .fdt; niofs maps no segment files. This turns the Step 5 headline into a pass/fail script.

2. (warm-up) Extract the hybrid extension set from source. Write a script that greps FsDirectoryFactory for the Set<String> of mmap extensions (rg -n "Set\\.of|HYBRID|nioExtensions|mmapExtensions|getExtension" server/src/main/java/org/opensearch/index/store/FsDirectoryFactory.java) and cross-checks it against what you observed mapped in Exercise 1, flagging any drift. "(verify on your branch — the set changes as vectors/points mature.)"

3. (core) A Directory/Store-type unit test. In an OpenSearch checkout, read the directory-factory tests: rg -l "FsDirectoryFactoryTests|HybridDirectory|class.*DirectoryTest" server/src/test/java/org/opensearch/index/store/. Write an OpenSearchTestCase that constructs the directory for each index.store.type and asserts the delegate type chosen per extension — e.g. that a .tim input under hybridfs is an MMapDirectory-backed input while a .fdt input is the NIO delegate. Find the routing in rg -n "openInput|class HybridDirectory|MMapDirectory|NIOFSDirectory" server/src/main/java/org/opensearch/index/store/FsDirectoryFactory.java. Run with ./gradlew :server:test --tests '*FsDirectoryFactory*'.

4. (core) A store-type micro-benchmark with a test gate. Promote the niofs-vs- hybridfs Stretch Goal to code: write a harness (Python or the Java client) that, with a cold page cache (drop_caches), times a sort/agg query against store-hybridfs and store-niofs, repeats warm, and asserts hybridfs cold latency ≤ niofs cold latency (within tolerance) on the random-access .dvd/.tim path. Document the syscall+copy overhead NIO pays. Keep it reproducible (warm-up, N runs, medians) like Lab CS3.

5. (core) Assert preload residency. Code up Step 6: write a script that sets index.store.preload: ["tim","dvd"], _close/_opens the index, and asserts via smaps that Rss ≈ Size for the preloaded extensions immediately on open while a non-preloaded extension starts cold. Find the preload plumbing with rg -n "preload|PRELOAD|MemorySegment|preLoad|load\\(\\)" server/src/main/java/org/opensearch/index/store/ and the Lucene MMapDirectory.setPreload/load path. Assert it fails loudly if residency didn't change (directory not reopened).

6. (advanced challenge) A VMA budget simulator + bootstrap-check test. Build a tool that counts index-file VMAs (grep -c 'indices/.*/index/' /proc/<pid>/maps) as you create N small mmapfs indices, fits VMAs-per-shard, and predicts how many shards a node could open before hitting vm.max_map_count — then validate the prediction by creating shards until you near the limit (on a disposable cluster). Pair it with a JUnit test of the bootstrap check itself: find it with rg -n "max_map_count|MaxMapCountCheck|262144" server/src/main/java/org/opensearch/bootstrap/ and assert it requires ≥ 262144 and produces the right failure message below it. This connects the operational failure (Map failed) to the exact guard that prevents it — the diagnosis a contributor is expected to make in minutes.

Issues to Practice On

Store-type, mmap, and the vm.max_map_count failure mode are core-repo Storage issues, with a steady stream of production-pain reports. Reproduce the mapping behavior first, then locate the routing with rg.

gh issue list --repo opensearch-project/OpenSearch --label "good first issue" --state open
gh issue list --repo opensearch-project/OpenSearch --label "Storage" --search "mmap store type hybridfs" --state open
gh issue list --repo opensearch-project/OpenSearch --search "vm.max_map_count Map failed" --state open
gh issue list --repo opensearch-project/OpenSearch --label "enhancement" --search "store preload" --state open
# Labels drift — confirm on the tracker:
gh label list --repo opensearch-project/OpenSearch | rg -i "storage|store|memory|bootstrap"

Two representative patterns:

• "Node fails to open shards: Map failed / vm.max_map_count too low." The single most common mmap production failure. Reproduce by counting VMAs (Exercise 6), point the reporter at the sysctl, and if the issue is a bootstrap-check or docs gap, the PR is a clearer error/message — locate it with rg "MaxMapCountCheck|max_map_count" server/src/main/java/org/opensearch/bootstrap/.
• "Should extension X be mmapped under hybridfs?" As a feature (vectors, new points formats) matures, its extension is added to the hybrid set. Reproduce the current routing (Exercise 1–2), and an enhancement PR adds the extension to the set in FsDirectoryFactory with a test asserting it is now mmapped.

Planted bug exercise. In an OpenSearch checkout, find the hybrid extension set / openInput routing (rg -n "Set\\.of|openInput|getExtension|nioExtensions" server/src/main/java/org/opensearch/index/store/FsDirectoryFactory.java). Remove .tim (or .dvd) from the mmap set — now a hot random-access structure is read with NIO. Rebuild, re-run your Exercise 3 routing test and Exercise 1 classifier, and watch the assertion that .tim is mmapped under hybridfs go red. Revert, then add a test that pins the complete expected hybrid extension set for your branch — the guard that catches an accidental dropout.

Etiquette: claim the issue first, reproduce before fixing, and every PR needs a test + CHANGELOG.md entry + DCO git commit -s. See community interaction and the good-first-issue PR lab.

Validation / Self-check

1. From your pmap, list the extensions hybridfs mmapped and explain why each one is in the mmap set (random-access, re-read) while .fdt is not.
2. Why does niofs map no segment files, and what does it cost on every read that mmap avoids?
3. What does index.store.preload do at the syscall/MemorySegment level, and why is ["*"] usually a mistake?
4. Explain vm.max_map_count: what consumes a VMA, the failure symptom of running out, and why niofs sidesteps it. What value does the bootstrap check require?
5. Justify "mmap wants free RAM, not heap" from your heap-usage and free -h measurements — what happens to mmap if the heap is 90% of RAM?
6. Given a sort/agg-heavy workload that is latency-sensitive, which store type would you keep and why? When is switching to niofs the right call?