Segments and Codecs

A Lucene index is not one file — it is a pile of segments, each segment a small, self-contained, immutable mini-index, plus one tiny commit file that says "this set of segments, right now, is the index." Every byte Lucene reads at query time comes from a segment; every byte it writes goes into a new segment. Nothing is ever modified in place. Understanding this immutability, the commit point that ties segments together, the per-segment file zoo, and the Codec SPI that defines those files' formats is the foundation for every other chapter in this section.

This is the Lucene-level companion to OpenSearch's Refresh, Flush, and Merge: that chapter told you a flush is an IndexWriter.commit() that writes segments_N; here you learn what that file is and what the segments it points to actually contain.

After this chapter you can: list a segment's files and say what each holds; explain why immutability shapes the entire engine (deletes, merges, NRT); describe the Codec SPI and its five sub-formats; explain how a Codec is discovered via META-INF/services; and set the codec on an OpenSearch index.

Immutable segments and the commit point

When IndexWriter flushes its in-RAM buffer, it writes a brand-new segment and never touches it again. This single design choice explains an enormous amount:

The index's current state is a commit point: a file named segments_N (generation N, monotonically increasing) listing exactly which segments are live and which codec wrote each. In Lucene this is the SegmentInfos object.

# In any index directory, the commit point is the segments_N file:
ls -la /path/to/index/ | grep segments
#   segments_7      <- the current commit (generation 7)
#   write.lock      <- the IndexWriter lock

grep -rn "class SegmentInfos\|class SegmentCommitInfo\|class SegmentInfo " \
  lucene/core/src/java/org/apache/lucene/index/

flowchart TD
    SN["segments_7 (SegmentInfos / commit point)"] --> S0["_0 (segment)"]
    SN --> S1["_1 (segment)"]
    SN --> S2["_5 (segment, merged)"]
    S0 --> F0["_0.si _0.cfs _0.cfe ..."]
    S1 --> F1["_1.fnm _1.tim _1.doc ..."]
    S2 --> F2["_5.dvd _5.kdd _5.vec ..."]

Note: A commit (segments_N) is what survives a crash. Refreshed-but-not- committed segments are visible to search but recovered from the translog after a crash, not from a commit. Visibility (refresh) and durability (commit/flush) are independent — the OpenSearch engine chapter calls this out, and it is exactly the Lucene segment/commit distinction.

The per-segment file zoo

Each segment _N is a set of files sharing the prefix _N, one per data structure. (When compound mode is on, most of them are packed into a single .cfs with a .cfe table of contents — fewer file handles.) Memorize this table; you'll read these extensions for the rest of your career.

Go look at a real one. Index a few docs into OpenSearch, find the shard, and find its files:

# 1. Locate the data path and a shard directory:
curl -s 'localhost:9200/_nodes/settings?filter_path=**.path.data' | python3 -m json.tool
#   <data>/nodes/0/indices/<index-uuid>/<shard>/index

# 2. List the segment files by extension:
SHARD=/path/to/<data>/nodes/0/indices/<uuid>/0/index
find "$SHARD" -maxdepth 1 -type f | sed 's/.*\.//' | sort | uniq -c
ls -la "$SHARD"

# 3. Ask OpenSearch to describe the same segments:
curl -s 'localhost:9200/my-index/_segments?pretty' | head -60
curl -s 'localhost:9200/_cat/segments/my-index?v'

If you only see .cfs/.cfe/.si per segment, the segment is compound. Small segments are compound by default; large merged ones are usually non-compound (the per-file overhead amortizes away). Luke (Lab L1) shows the same picture visually.

The Codec SPI

A Codec defines the on-disk format of all those files. It is the central extension point of Lucene's storage layer, wired through Java's ServiceLoader (the SPI mechanism). A Codec is really a bundle of sub-format factories:

The default codec is versioned — LuceneNNNCodec, where NNN bumps when the on-disk format changes. Do not hard-code the number; grep for it, because a Lucene upgrade in OpenSearch is precisely "the default codec version moved."

# Find the current default codec class in your Lucene checkout:
ls lucene/core/src/java/org/apache/lucene/codecs/lucene*/
grep -rln "extends Codec\b" lucene/core/src/java/org/apache/lucene/codecs/

# The default is also declared as a service — see next section.
grep -rn "Codec.getDefault\|setDefault" lucene/core/src/java/org/apache/lucene/codecs/Codec.java

A codec's sub-formats are themselves SPI-pluggable and versioned independently. This is why a single codec can read older segments written by an older codec (each segment records the codec that wrote it, in its .si) — Lucene keeps the old format readers around for one major version of back-compat. That back-compat window is the deep reason an OpenSearch major can read indices from the previous major but not from two majors ago.

SPI registration via META-INF/services

A Codec (or any sub-format) is discovered at runtime by listing its fully-qualified class name in a META-INF/services file named for the SPI interface. No central registry, no config — drop the jar on the classpath with the right service file and Lucene finds it. This is exactly how OpenSearch's k-NN plugin ships its own vector codec.

# The default codec registers itself here, in Lucene's core jar:
find lucene -path "*META-INF/services/org.apache.lucene.codecs.Codec"
cat lucene/core/src/resources/META-INF/services/org.apache.lucene.codecs.Codec

# And the sub-format services:
find lucene -path "*META-INF/services/org.apache.lucene.codecs.*Format"

# META-INF/services/org.apache.lucene.codecs.Codec
org.apache.lucene.codecs.lucene101.Lucene101Codec
# (number illustrative — grep your checkout for the real one)

To write your own, you extend Codec (usually FilterCodec, delegating most sub-formats and overriding one), then add a line to the service file. That is the whole of Lab L2: Write a custom codec.

Per-field codecs

A codec doesn't have to use one format for the entire index — it can choose a different format per field. Lucene exposes this through PerFieldPostingsFormat, PerFieldDocValuesFormat, and PerFieldKnnVectorsFormat: the codec is asked, per field name, which format to use. This is how a single index can have, say, a high-compression postings format on a rarely-queried field and the fast default on a hot one — and how the k-NN plugin attaches its HNSW format only to knn_vector fields while leaving the rest on the standard codec.

grep -rn "PerFieldPostingsFormat\|PerFieldDocValuesFormat\|PerFieldKnnVectorsFormat\|getPostingsFormatForField" \
  lucene/core/src/java/org/apache/lucene/codecs/perfield/

The mechanism: the codec returns a PerField*Format, whose getXForField(field) returns the concrete format; Lucene records the chosen format name per field in the .fnm so the reader can re-instantiate the right one. Override one method, target one field — clean.

How OpenSearch chooses the codec

OpenSearch exposes the codec choice as the index setting index.codec. It does not let you pick an arbitrary Lucene codec by class; it maps a few friendly names to codec configurations (chiefly different stored-fields compression).

# Set it at index creation:
curl -s -XPUT 'localhost:9200/logs' -H 'Content-Type: application/json' -d'
{ "settings": { "index.codec": "best_compression" } }'

# Where OpenSearch maps the name to a Lucene codec:
grep -rn "index.codec\|CodecService\|best_compression\|BEST_COMPRESSION\|Lucene.*Codec" \
  server/src/main/java/org/opensearch/index/codec/CodecService.java \
  server/src/main/java/org/opensearch/index/engine/EngineConfig.java 2>/dev/null

best_compression is the standard answer when disk is the constraint and _source fetches are rare (log/metrics indices). It only affects stored fields compression, not postings/points/docvalues — a common misconception. The k-NN plugin layers its own codec on top of whichever index.codec you pick, adding the vector format per knn_vector field.

Reading exercise

# 1. The commit point and its segments.
grep -rn "class SegmentInfos\|class SegmentInfo \|class SegmentCommitInfo" \
  lucene/core/src/java/org/apache/lucene/index/

# 2. The default codec and its sub-formats.
ls lucene/core/src/java/org/apache/lucene/codecs/lucene*/
grep -rln "extends Codec\b\|extends FilterCodec" lucene/core/src/java/org/apache/lucene/codecs/

# 3. SPI registration files.
find lucene -path "*META-INF/services/org.apache.lucene.codecs.*"

# 4. Per-field formats.
grep -rn "getPostingsFormatForField\|getKnnVectorsFormatForField" \
  lucene/core/src/java/org/apache/lucene/codecs/perfield/

# 5. Real files in a real shard.
find /path/to/shard/index -maxdepth 1 -type f | sed 's/.*\.//' | sort | uniq -c
curl -s 'localhost:9200/my-index/_segments?pretty' | head -40

# 6. OpenSearch's codec setting.
grep -rn "best_compression\|index.codec\|CodecService" \
  server/src/main/java/org/opensearch/index/codec/

Answer:

1. Why are segments immutable, and name three consequences (one of them about deletes, one about reads, one about merges).
2. What does segments_N contain, and what is the Lucene class for it? Why does each segment record the codec that wrote it?
3. Given the files _3.si _3.tim _3.doc _3.kdd _3.dvd _3.vec, say what each one holds and which chapter covers it.
4. List the five most-used sub-formats of a Codec and the file extensions each produces.
5. Explain SPI registration: how does Lucene find a custom codec at runtime without any central config?
6. What exactly does index.codec: best_compression change, and what does it not change? Why would you choose it for a logs index?

Common bugs and symptoms

Validation: prove you understand this

1. Draw an index as a segments_N commit pointing at three segments, and one segment as its set of extension files; label what each file holds.
2. Explain immutability and derive from it: how deletes work, why reads need no lock against writes, and why merges are mandatory.
3. Define the Codec SPI and list its sub-formats and their file extensions. Show the META-INF/services line that registers a codec.
4. Explain per-field formats and give the k-NN-plugin example of why you'd want one format on one field only.
5. State precisely what index.codec: best_compression does and does not affect, and the workload it suits.
6. From a find over a real shard, classify every file extension you see and say which chapter of this section owns it.