Lucene File Formats Cheat-Sheet

This is a consolidated quick-reference for every per-segment file a Lucene index writes — extension, what it holds, the format class that defines it, and the chapter that takes it apart. It is the companion lookup table to the Lucene data-structures masterclass and the storage-engine masterclass: those teach you why each structure exists; this page is what you keep open while you find/xxd a real shard.

A Lucene index is a set of segments plus one segments_N commit file. Each segment _N is a bundle of files sharing the prefix _N, one per data structure, each written by a sub-format of the active Codec. Start from Segments and Codecs for the immutability and commit-point model; this page assumes it.

Warning: The default codec is versioned (LuceneNNNCodec, e.g. Lucene101Codec); NNN bumps whenever an on-disk format changes, which is exactly what a Lucene upgrade inside OpenSearch is. Do not hard-code the number — grep for it. The extensions below are stable across many versions; the format class names carry the version and move.

Master table: every segment file extension

Note: The k-NN plugin's faiss / nmslib engines do not use .vec/.vex/.vem. Those native engines write their own files (e.g. .faiss, historically .hnsw) through the plugin's own KnnVectorsFormat, and the graph lives in off-heap native memory, not the JVM heap — see native-jni-and-memory.md and knn/engines.md. The .vec/.vex/.vem triple is specifically Lucene's built-in HNSW engine.

Grouped by codec sub-format

The Codec is a bundle of sub-format factories. Map each one to the files it owns — this is the table from segments-and-codecs.md, reorganized as a lookup:

Each sub-format is independently SPI-pluggable and versioned, which is why per-field formats exist (PerFieldPostingsFormat, PerFieldDocValuesFormat, PerFieldKnnVectorsFormat): the codec can pick a different format per field, and the chosen format name is recorded in .fnm so the reader re-instantiates the right one. This is exactly how the k-NN plugin attaches its vector format to knn_vector fields only.

# See the default codec and its sub-formats in your Lucene checkout:
ls lucene/core/src/java/org/apache/lucene/codecs/lucene*/
grep -rln "extends Codec\b\|extends FilterCodec" lucene/core/src/java/org/apache/lucene/codecs/

# The SPI service file that registers the default codec:
find lucene -path "*META-INF/services/org.apache.lucene.codecs.Codec"

Compound vs non-compound segments

A small segment is usually written compound: every file above (except .si, .liv, and .fnm-gen-bumped files) is packed into a single .cfs blob with a .cfe table of contents, so the segment costs ~3 file handles instead of ~12. Large merged segments are typically non-compound (per-file overhead amortizes away).

# Per-segment file count tells you which is which:
find "$SHARD/index" -maxdepth 1 -type f | sed -E 's/.*\/_[0-9a-z]+//' | sort | uniq -c

To look inside a .cfs you need Luke or CheckIndex (next section) — .cfs is opaque to xxd beyond its header/footer.

The CodecUtil header and footer (magic + CRC32)

Every Lucene file written through the codec framework is wrapped by CodecUtil with a fixed header and footer. This is what makes a file self-describing and corruption-detectable.

On open, Lucene calls CodecUtil.checkHeader (name + version range) and, when verifying, CodecUtil.checksumEntireFile / checkFooter to compare the stored CRC32 against a recomputed one. A mismatch is a CorruptIndexException.

grep -rn "CODEC_MAGIC\|FOOTER_MAGIC\|writeHeader\|checkHeader\|writeFooter\|checkFooter\|checksumEntireFile" \
  lucene/core/src/java/org/apache/lucene/codecs/CodecUtil.java

# First 4 bytes of any codec file are the magic 0x3fd76c17:
xxd -l 16 "$SHARD/index/_5.si"
# 00000000: 3fd7 6c17 0014 4c75 6365 6e65 ...   .l..Lucene...
#           ^^^^^^^^^ CODEC_MAGIC   ^^ length+"Lucene..." codec name

Note: This header/footer is why a truncated or bit-rotted segment file is caught at open time rather than returning wrong results: the CRC32 footer won't match. It is also why you can identify a stray file's format by xxd-ing the first ~32 bytes — the codec name is right there in plaintext after the magic.

Inspecting an index: the toolbox

# 1. Files by extension and size in a real shard:
SHARD=/path/to/<data>/nodes/0/indices/<uuid>/0
find "$SHARD/index" -maxdepth 1 -type f | sed 's/.*\.//' | sort | uniq -c
du -ah "$SHARD/index" | sort -rh | head

# 2. CheckIndex (needs the lucene-core + codecs jars on the classpath; OpenSearch
#    ships them under lib/ and modules/lang-painless etc. — point -cp at them):
java -cp 'lib/*' org.apache.lucene.index.CheckIndex "$SHARD/index"
#   -> "No problems were detected with this index." or a per-segment failure

# 3. Read a file's codec name straight from its header:
for f in "$SHARD"/index/_*.si "$SHARD"/index/_*.fnm; do
  printf '%s -> ' "$(basename "$f")"; xxd -s 4 -l 24 "$f" | head -1
done

# 4. The cluster's own view:
curl -s 'localhost:9200/my-index/_segments?pretty' | head -60
curl -s 'localhost:9200/_cat/segments/my-index?v'

CheckIndex is the ground truth: it re-reads every sub-format and re-verifies every CRC32. If you write a custom codec (Lab L2), running CheckIndex over its output is your acceptance test.

hybridfs: how those files get mapped into memory

OpenSearch's default store type is hybridfs (index.store.type), which chooses, per file extension, whether to mmap the file (MMapDirectory → page cache, off-heap) or read it through NIO (NIOFSDirectory → heap buffers). The rule of thumb: random-access structures that benefit from the OS page cache are mmapped; large sequential-scan files are read with NIO so they don't blow out the virtual address space and mmap count.

Warning: The exact per-extension split is decided in code and shifts between versions — don't memorize the rows above as gospel, grep them. mmap requires a high vm.max_map_count (OpenSearch's docs require 262144); too low and the node fails to open shards with "max virtual memory areas vm.max_map_count too low." Store types are taken apart in storage-engine/lab-03-store-types-mmap.md.

# Where the hybridfs per-extension routing lives (names vary by version):
grep -rn "hybridfs\|HybridDirectory\|HybridFSDirectory\|FsDirectoryFactory\|nioExtensions\|mmapExtensions" \
  server/src/main/java/org/opensearch/index/store/

# Confirm the store type on a running index:
curl -s 'localhost:9200/my-index/_settings?filter_path=**.store' | python3 -m json.tool

The k-NN native engines (faiss/nmslib) sidestep this entirely: their graphs are loaded into native memory by JNI, not through the Directory at all — which is why they need explicit warmup.

Quick decoding drills

Given a find listing, name every file. Try it before reading the answer.

_0.cfs  _0.cfe  _0.si
_1.fnm  _1.tim  _1.tip  _1.tmd  _1.doc  _1.pos  _1.dvd  _1.dvm  _1.kdd  _1.kdi  _1.kdm  _1.vec  _1.vex  _1.vem  _1.si  _1.liv
segments_3   write.lock

If you can do that cold, you can read any shard directory in the wild.

Related references

• Segments and Codecs — the immutability / commit-point / Codec SPI model this page is a lookup for.
• Lucene data-structures masterclass — the deep dive on each structure (FST, postings, BKD, HNSW, doc-values).
• Storage-engine masterclass — the shard directory, translog, and store types (hybridfs/mmap) end to end.
• REST Endpoint Map — _segments, _forcemerge, _split and the snapshot APIs that operate on these files.
• k-NN native JNI & memory — why faiss/nmslib vectors live off-heap and outside the Directory.