Lab ST1: Dissect a Shard's Data Directory

Background

You have read the Storage Engine — Intensive chapter and the Lucene Segments and Codecs chapter. Now you make it concrete. In this lab you start a single node, index a handful of documents, find the exact directory where shard 0 of your index lives, and then map every file on disk to the data structure it holds. You will read the segments_N commit point and a .si segment-info file with Lucene's own tools, peek at the translog checkpoint, and finally run org.apache.lucene.index.CheckIndex against a copy of the index to validate every CRC32 footer.

By the end you will never again be surprised by a file in a shard directory — you will know what _7.tim is, why a small segment is just .cfs/.cfe/.si, and where the translog and shard state live.

Why This Matters for Contributors

When you debug a corrupt shard, a recovery that copies too much, or a disk-usage mystery, you do it in this directory. A contributor who can find a shard and classify every extension can reason about merge debt, codec choices, mmap behavior, and corruption from the bytes up — instead of guessing from API responses. Half of the storage-layer issues in the OpenSearch tracker are ultimately "what does this file mean and why is it that size."

Prerequisites

• An OpenSearch source checkout you can run with ./gradlew run (a single-node dev cluster on localhost:9200), or a tarball install.
• curl, python3, find, du, xxd, and a JDK (java) on PATH.
• Read index.md (the on-disk layout section) and segments-and-codecs.md (the file zoo).

Note: The cluster manager (formerly master) node in a one-node dev cluster is the same node that holds your shards — convenient for inspection. Everything here works the same on a real multi-node cluster; you just inspect the node that holds the shard.

Step-by-Step Tasks

Step 1 — Start a node and find the data path

# From an OpenSearch checkout:
./gradlew run
# ... wait for "started" ...

# In another shell, ask the node where it stores data:
curl -s 'localhost:9200/_nodes/settings?filter_path=**.path.data' | python3 -m json.tool

Expected (a ./gradlew run cluster):

{
    "nodes": {
        "abc123...": {
            "settings": {
                "path": {
                    "data": [ "/your/checkout/server/build/run/data/nodes/0" ]
                }
            }
        }
    }
}

Save that path:

DATA=/your/checkout/server/build/run/data    # the dir that contains nodes/
ls "$DATA/nodes/0"
#   _state   indices   node.lock

Step 2 — Index data so there are real segments

curl -s -XPUT 'localhost:9200/orders?pretty' -H 'Content-Type: application/json' -d'
{ "settings": { "index.number_of_shards": 1, "index.number_of_replicas": 0 },
  "mappings": { "properties": {
    "customer": { "type": "keyword" },
    "amount":   { "type": "double" },
    "ts":       { "type": "date" },
    "note":     { "type": "text" }
  }}}'

# Bulk a few docs:
for i in $(seq 1 2000); do
  printf '{"index":{}}\n{"customer":"c%d","amount":%d.50,"ts":"2026-06-01","note":"order number %d"}\n' \
    "$((i % 50))" "$i" "$i"
done | curl -s -H 'Content-Type: application/x-ndjson' \
  'localhost:9200/orders/_bulk?refresh=true' --data-binary @- > /dev/null

# Force a commit so there is a segments_N on disk (refresh alone does not commit):
curl -s -XPOST 'localhost:9200/orders/_flush?pretty' | python3 -m json.tool

Step 3 — Locate shard 0's directory

The index is stored under its UUID, not its name. Map name→uuid:

curl -s 'localhost:9200/_cat/indices/orders?v&h=index,uuid,docs.count,store.size'

Expected:

index   uuid                     docs.count store.size
orders  9bQ2f0v1Q3yScq8mYtXJrA        2000      1.2mb

Now find the shard directory by UUID:

UUID=9bQ2f0v1Q3yScq8mYtXJrA           # paste yours
SHARD="$DATA/nodes/0/indices/$UUID/0"  # shard 0
ls -la "$SHARD"

Expected:

drwxr-xr-x  _state
drwxr-xr-x  index
drwxr-xr-x  translog

Step 4 — Map every file in index/

ls -la "$SHARD/index"
echo "--- by extension ---"
find "$SHARD/index" -maxdepth 1 -type f | sed 's/.*\.//' | sort | uniq -c

Expected (yours will differ in counts/generations):

_state
  write.lock
  segments_3
  _0.cfs  _0.cfe  _0.si
  _1.tim  _1.tip  _1.tmd  _1.doc  _1.pos  _1.fnm  _1.fdt  _1.fdx  _1.fdm  _1.dvd  _1.dvm  _1.nvd  _1.nvm  _1.si  _1.kdd  _1.kdi  _1.kdm
--- by extension ---
   1 cfe
   1 cfs
   2 dvd
   ...

Map each extension. Cross-reference the full table in segments-and-codecs.md:

Note: A segment that shows up as only _0.cfs _0.cfe _0.si is a compound segment — all its sub-files are packed inside the .cfs. Small fresh segments are compound; large merged ones are usually not. To see inside a .cfs, use Luke (Step 7) or CheckIndex (Step 8).

Force a merge to collapse to one non-compound segment and re-inspect (the extensions become individually visible):

curl -s -XPOST 'localhost:9200/orders/_forcemerge?max_num_segments=1&pretty' >/dev/null
curl -s -XPOST 'localhost:9200/orders/_flush?pretty' >/dev/null
find "$SHARD/index" -maxdepth 1 -type f | sed 's/.*\.//' | sort | uniq -c

Step 5 — Read the segments_N commit point

segments_N is binary, but its tail is human-readable diagnostics. Confirm the generation matches what the API reports:

ls "$SHARD/index"/segments_*          # the commit point, generation N
strings "$SHARD/index"/segments_* | head -40

You will see the codec name (Lucene<NNN>), the Lucene version, segment names, and diagnostics like os, java.version, source: flush|merge. Compare to the API:

curl -s 'localhost:9200/_cat/segments/orders?v'
curl -s 'localhost:9200/orders/_segments?pretty' | head -60

_cat/segments columns: segment, generation, docs.count, docs.deleted, size, committed (true/false), searchable, version, compound.

Step 6 — Read a .si segment-info file

SI=$(ls "$SHARD/index"/*.si | head -1)
echo "Segment info: $SI"
strings "$SI" | head -30          # codec name, Lucene version, file list, diagnostics
xxd "$SI" | head -8               # the CodecUtil header: magic + codec name + version
xxd "$SI" | tail -3               # the CodecUtil footer: footer magic + CRC32

The first bytes are the CodecUtil index header (a magic number 0x3fd76c17 followed by the codec name as a string); the last 16 bytes are the footer (footer magic + algorithm id + the 8-byte CRC32). You are looking at the exact integrity structure described in index.md.

Step 7 — (Optional) Open it in Luke

Luke is Lucene's GUI index browser. From an apache/lucene checkout:

# In an apache/lucene checkout (separate from OpenSearch):
./gradlew :lucene:luke:run
# File > Open Index Directory > point at:  $SHARD/index

Luke shows every field, the terms dictionary, postings, doc-values, points, and per-segment file sizes — the visual version of this whole lab. (Luke must be a Lucene version compatible with the one OpenSearch wrote the index with.)

Step 8 — Run CheckIndex against a copy

CheckIndex opens the index and verifies every segment's structures and CRC32 footers. Never run it against the live shard — copy first:

# Stop writes (optional) and copy the index dir somewhere safe:
mkdir -p /tmp/shard-copy
cp -R "$SHARD/index" /tmp/shard-copy/

# Find a lucene-core jar (Gradle cache, or your dist's lib/):
LUCENE_JAR=$(find ~/.gradle /your/checkout -name 'lucene-core-*.jar' 2>/dev/null | head -1)
echo "Using: $LUCENE_JAR"

java -cp "$LUCENE_JAR" -ea org.apache.lucene.index.CheckIndex /tmp/shard-copy/index

Expected (healthy index):

Opening index @ /tmp/shard-copy/index

Segments file=segments_3 numSegments=1 version=10.x.x id=...
  1 of 1: name=_2 maxDoc=2000
    version=10.x.x
    ...
    test: open reader.........OK
    test: check integrity.....OK
    test: check live docs.....OK
    test: field infos.........OK [5 fields]
    test: field norms.........OK [1 fields]
    test: terms, freq, prox...OK [... terms; ... postings; ... positions]
    test: stored fields.......OK [... total field count]
    test: docvalues...........OK [... docvalues fields]
    test: points..............OK [2 fields]

No problems were detected with this index.

Every OK line corresponds to a structure you mapped in Step 4. "check integrity" is the CRC32 footer pass.

Now break it on purpose and watch it caught (still on the copy):

# Corrupt one byte deep inside a data file (NOT the footer/header):
TARGET=$(ls -S /tmp/shard-copy/index/*.cfs /tmp/shard-copy/index/*.tim 2>/dev/null | head -1)
echo "Corrupting: $TARGET"
printf '\xFF' | dd of="$TARGET" bs=1 seek=500 count=1 conv=notrunc 2>/dev/null

java -cp "$LUCENE_JAR" -ea org.apache.lucene.index.CheckIndex /tmp/shard-copy/index

Expected (a flipped byte trips the checksum):

  1 of 1: name=_2 maxDoc=2000
    test: check integrity.....FAILED
    WARNING: exorciseIndex() would remove reference to this segment ...
    org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) :
      expected=... actual=... (resource=...)

You just saw a CorruptIndexException raised by the CodecUtil footer mismatch — the exact mechanism OpenSearch uses to fail a bad shard.

Step 9 — Inspect the translog and the _state dirs

ls -la "$SHARD/translog"
#   translog-12.tlog  translog-13.tlog  translog.ckp

# The checkpoint is small and partly readable:
ls -la "$SHARD/translog/translog.ckp"
strings "$SHARD/translog"/*.tlog | head -20   # you can see indexed doc sources

# Shard-level state (allocation id, primary term):
ls -la "$SHARD/_state"
#   state-N.st

# Index-level state (mappings, settings) one level up:
ls -la "$DATA/nodes/0/indices/$UUID/_state"

# Node-level state:
ls -la "$DATA/nodes/0/_state"

Cross-check the translog op count and generation against the API:

curl -s 'localhost:9200/orders/_stats/translog?pretty' \
  | python3 -c 'import sys,json; d=json.load(sys.stdin); print(json.dumps(d["indices"]["orders"]["primaries"]["translog"], indent=2))'

Right after a flush, operations should be 0 or small (the flush trimmed the translog) — you will exercise this in Lab ST2.

Deliverables

• The absolute path of orders shard 0's index/ directory on your node.
• A table mapping every extension present to its structure (your own, produced from find ... | sed | uniq -c).
• strings output of the segments_N showing the codec + Lucene version, and a xxd of a .si footer.
• CheckIndex output: one healthy run (all OK) and one corrupted run showing the CorruptIndexException.
• The contents of translog/, the shard _state/, the index _state/, and the node _state/.

Troubleshooting

Expected Output (summary)

A single non-compound segment after force-merge looks like:

segments_4
_3.si _3.fnm
_3.tim _3.tip _3.tmd _3.doc _3.pos
_3.fdt _3.fdx _3.fdm
_3.dvd _3.dvm
_3.nvd _3.nvm
_3.kdd _3.kdi _3.kdm

— one file per Lucene sub-format, each ending in a CRC32 footer, all listed in segments_4, all describable by you.

Stretch Goals

• du -h --max-depth=1 "$SHARD/index"-style per-extension sizing: sum bytes per extension (find ... -printf '%s %f\n' + awk) and find which structure dominates your index. Predict it from your mapping (text-heavy ⇒ .tim/.doc; verbose _source ⇒ .fdt).
• Add a knn_vector field, index vectors, and find the new .vec/.vex/.vem files. Cross-link HNSW vector search.
• Delete 100 docs (_delete_by_query), flush, and find the new .liv live-docs file; confirm docs.deleted in _cat/segments.
• Run CheckIndex with -verbose and read the per-field term/postings counts.

Coding Exercises

Dissecting a directory by hand is the reading skill; these exercises make you produce code that opens, classifies, and validates the same bytes — the muscle a contributor uses when debugging a corrupt shard or a disk-usage mystery.

1. (warm-up) An extension classifier in Python. Write classify_shard.py that takes a shard index/ path, lists the files, and prints a table mapping each extension to its structure name (use the Step 4 table as your lookup), plus total bytes per extension (the first Stretch Goal, in code). Assert it flags any extension it does not recognize, so a future codec file can't slip past you.

2. (warm-up) Parse the .si/segments_N CodecUtil header in code. Write a tiny Java program (Lucene core on the classpath — find it as in Lab ST1 Step 8) that opens a .si with Directory.openInput and reads the header via CodecUtil.checkIndexHeader(...), printing the codec name and version. Confirm the magic/footer story you saw with xxd. Find the helper with rg -n "checkIndexHeader|checkFooter|writeFooter|CODEC_MAGIC" lucene/core/src/java/org/apache/lucene/codecs/CodecUtil.java in a Lucene checkout.

3. (core) A Directory/Store unit test in OpenSearch. In an OpenSearch checkout, read the existing store tests: rg -l "class.*StoreTests|class.*StoreTest" server/src/test/java/org/opensearch/index/store/ and rg -n "verify|checkIntegrity|checksum" server/src/main/java/org/opensearch/index/store/Store.java. Write an OpenSearchTestCase that builds a small index in a Directory, then uses Store (or Store.MetadataSnapshot) to enumerate files and assert the per-file checksums/lengths match — programmatically reproducing the integrity pass you ran by hand with CheckIndex. Run with ./gradlew :server:test --tests '*YourStoreTest*'.

4. (core) Reproduce the corruption-detection path in a test. Promote Step 8 to code: in a JUnit test, write an index to a ByteBuffersDirectory (or temp FSDirectory), flip a byte in a data file (not header/footer), and assert that opening/verifying it throws CorruptIndexException (or that Store.checkIntegrity/CodecUtil.checksumEntireFile reports the mismatch). Find the exception and check path with rg -n "CorruptIndexException|checksumEntireFile|checkIntegrity" server/src/main/java/org/opensearch/index/store/. This is the exact mechanism OpenSearch uses to fail a bad shard — now under test.

5. (core) Programmatic per-extension sizing + a size invariant. Write a Java or Python tool that, given a shard, sums bytes per Lucene sub-format and asserts a structural invariant you predict from your mapping (e.g. for a text-heavy index, .tim+.doc dominate; for verbose _source, .fdt dominates). Run it before and after _forcemerge?max_num_segments=1 and assert the segment count dropped to 1 while total bytes stayed within a tolerance. This grades the Stretch-Goal sizing with a real pass/fail.

6. (advanced challenge) A standalone "shard dissector" CLI. Build a small Java program (Lucene only) that takes a shard index/ path and emits a JSON report: segments_N generation, per-segment SegmentInfo (doc count, codec, compound flag, file list) read via SegmentInfos.readLatestCommit, per-extension byte totals, and a CheckIndex-style integrity verdict (call new CheckIndex(dir).checkIndex() against a copy). Find the API with rg -n "readLatestCommit|class SegmentInfos|class CheckIndex|checkIndex\\(" lucene/core/src/java/org/apache/lucene/index/. Validate it against a healthy index (all OK) and a corrupted copy (integrity FAILED) — your own scriptable, CI-able replacement for Steps 4–8. This is the tool you'd actually reach for when triaging a storage issue.

Issues to Practice On

Shard-directory and store work is core-repo, under Storage (and sometimes Indexing). Half of storage issues are "what does this file mean / why is it that size" — exactly what you just learned to answer.

gh issue list --repo opensearch-project/OpenSearch --label "good first issue" --state open
gh issue list --repo opensearch-project/OpenSearch --label "Storage" --state open
gh issue list --repo opensearch-project/OpenSearch --search "shard directory store corruption checksum" --state open
gh issue list --repo opensearch-project/OpenSearch --label "bug" --search "CorruptIndexException" --state open
# Confirm labels on the tracker (taxonomies drift):
gh label list --repo opensearch-project/OpenSearch | rg -i "storage|index|store|corrupt"

Two representative patterns:

• "Shard fails to allocate: CorruptIndexException / checksum failed." Reproduce by corrupting a copy (Step 8), locate the verification path with rg "checkIntegrity|CorruptIndexException" server/src/main/java/org/opensearch/index/store/, and understand whether the fix is detection, reporting, or recovery — usually the PR is a better error message or a CheckIndex/opensearch-shard tooling tweak.
• Disk-usage / store-size reporting discrepancies. _cat/segments size vs du vs _stats/store can disagree (compound files, merge debt, .liv). Reproduce, locate the size computation with rg "store.size|StoreStats|sizeInBytes" server/src/main/java/org/opensearch/index/store/, and add a test pinning the expected accounting.

Planted bug exercise. In an OpenSearch checkout, find a checksum/length verification site (rg -n "verify\\(|checkIntegrity|Store.MetadataSnapshot|recoveryDiff" server/src/main/java/org/opensearch/index/store/Store.java). Weaken one check — e.g. make a length comparison always return equal, or skip the checksum compare. Run the store tests (./gradlew :server:test --tests '*Store*') and note which goes red. Revert, then add an assertion that a length/checksum mismatch is always reported — the guard that catches a weakened integrity check.

Etiquette: claim the issue before working it, reproduce first, and every PR needs a test + CHANGELOG.md entry + DCO git commit -s. See community interaction and the good-first-issue PR lab.

Validation / Self-check

1. Given a shard's index/ listing, classify every file and name the chapter that owns each structure.
2. Why is segments_N the file that "is" the index, and what does a .si add per segment?
3. What does "check integrity" verify in CheckIndex, and which CodecUtil structure makes it possible without rescanning whole files?
4. Why must you copy the index dir before running CheckIndex?
5. Locate, from memory, the node _state/, the index _state/, the shard _state/, and explain what each holds.
6. After a flush, why does the translog operations count drop, and where did those ops go? (Answer with Lab ST2 in mind.)