Lab L1: Crack Open a Lucene Index

Background

You have read about segments and codecs, the inverted index, BKD trees, DocValues, and HNSW vectors. Each of those chapters named files on disk — .tim, .kdd, .dvd, .vec, and friends. This lab makes them real. You will create an actual Lucene index, find the files on disk, decode them with three different tools — CheckIndex, Luke, and a small Java program using the reader API — and map every file extension back to the chapter that explains it.

This is the foundational "I can see inside the box" lab. Once you can open an index and enumerate its segments, fields, terms, and per-field codecs by hand, every later lab (custom codecs, HNSW from scratch, k-NN debugging) becomes legible — you can always drop down and inspect what Lucene actually wrote.

Note: OpenSearch indices are Lucene indices. The same files, the same tools, work on an OpenSearch shard's data directory. We will create a standalone Lucene index for full control, then point the same techniques at an OpenSearch shard so you see they are identical.

Why This Lab Matters for Contributors

• Bugs in storage, merges, and corruption show up as files — knowing how to read them is a core debugging skill. CheckIndex is the first thing a Lucene/OpenSearch committer runs on a suspected-corrupt index.
• Mapping file extensions to codec components demystifies the codec SPI: you will see that a .tim is the terms dictionary and a .vec is vector data, not just read it.
• Luke is the canonical index inspector; being fluent in it (and in the reader API it is built on) is table stakes for working on the storage layer.
• The reader API (DirectoryReader → LeafReader → terms/fields/codec) is the same API the search layer uses. Walking it by hand teaches you how a query reaches the data.

Prerequisites

• JDK 21 (the OpenSearch/Lucene bundled JDK is fine).
• An apache/lucene checkout (for Luke and for the Lucene jars). Clone it:

  git clone https://github.com/apache/lucene.git
  cd lucene && ./gradlew assemble    # first build is slow; grab coffee
  

• Optionally, a running OpenSearch from ./gradlew run (for the "do it on a real shard" step).
• The Lucene lucene-core jar on your classpath for the standalone program (we resolve it below).

Step-by-Step Tasks

Step 1: Hold the file-extension map in your head

This is the decoder ring. Each per-segment file belongs to one codec component and one chapter:

Step 2: Write a tiny indexer that uses every component

We index documents with a text field (postings + norms), a stored field, a numeric point + docvalue, and a float vector — so the segment contains all the file types above. Resolve the Lucene jars from your checkout first:

# Find the built lucene-core jar in your apache/lucene checkout:
LUCENE=$(pwd)                                   # your apache/lucene dir
CORE=$(find "$LUCENE" -name "lucene-core-*.jar" | grep -v sources | head -1)
echo "lucene-core: $CORE"
mkdir -p ~/lucene-lab && cd ~/lucene-lab

CrackOpenIndexer.java:

import org.apache.lucene.document.Document;
import org.apache.lucene.document.Field;
import org.apache.lucene.document.IntField;            // indexes a point + docvalue
import org.apache.lucene.document.KnnFloatVectorField;
import org.apache.lucene.document.StoredField;
import org.apache.lucene.document.TextField;
import org.apache.lucene.index.IndexWriter;
import org.apache.lucene.index.IndexWriterConfig;
import org.apache.lucene.index.VectorSimilarityFunction;
import org.apache.lucene.store.Directory;
import org.apache.lucene.store.FSDirectory;

import java.nio.file.Paths;
import java.util.Random;

/** Builds an index that exercises postings, stored fields, points, docvalues, and vectors. */
public class CrackOpenIndexer {
    public static void main(String[] args) throws Exception {
        Directory dir = FSDirectory.open(Paths.get("index"));
        IndexWriterConfig cfg = new IndexWriterConfig();
        // Keep files NON-compound so each extension is a separate file we can find/inspect.
        cfg.setUseCompoundFile(false);
        Random rnd = new Random(42);

        try (IndexWriter w = new IndexWriter(dir, cfg)) {
            String[] bodies = {
                "the quick brown fox jumps over the lazy dog",
                "a fast dark fox leaps across a sleepy hound",
                "lucene segments are immutable and merged over time",
                "vector search finds nearest neighbours in high dimensions"
            };
            for (int i = 0; i < bodies.length; i++) {
                Document doc = new Document();
                doc.add(new StoredField("id", i));                              // .fdt/.fdx
                doc.add(new TextField("body", bodies[i], Field.Store.NO));      // .tim/.doc/.nvd
                doc.add(new IntField("year", 2020 + i, Field.Store.NO));        // .kdd (point) + .dvd
                float[] vec = new float[8];                                     // small vector for clarity
                for (int d = 0; d < vec.length; d++) vec[d] = rnd.nextFloat();
                doc.add(new KnnFloatVectorField("embedding", vec, VectorSimilarityFunction.COSINE)); // .vec/.vex/.vem
                w.addDocument(doc);
            }
            w.commit();   // writes segments_N
        }
        System.out.println("Indexed. Files written to ./index");
    }
}

Compile and run:

javac -cp "$CORE" CrackOpenIndexer.java
java  -cp "$CORE:." --add-modules jdk.incubator.vector CrackOpenIndexer

Note: --add-modules jdk.incubator.vector is needed so the vector format can use SIMD (SIMD chapter). It works without it (scalar fallback), but include it for realism.

Step 3: find the files and map each one

cd ~/lucene-lab
ls -la index/
find index -type f | sort

You should see the segments_N commit and a set of _0.* files (segment _0). Match every one to the table in Step 1:

# Group by extension and eyeball the map:
find index -type f | sed 's/.*\.//' | sort | uniq -c
#   expect: si fnm fdt fdx fdm tim tip tmd doc pos nvd nvm kdd kdi kdm dvd dvm vec vex vem ...

• Find the .vec, .vex, .vem trio — those are your vectors. Note their relative sizes (.vec is raw values, .vex is the graph).
• Find .tim/.tip — the terms dictionary for body.
• Find .kdd — the BKD point for year. Find .dvd — its docvalue.

Step 4: Decode with CheckIndex

CheckIndex walks the whole index and prints, per segment, every component it verifies — a perfect human-readable inventory and the standard corruption check:

java -cp "$CORE" --add-modules jdk.incubator.vector \
  org.apache.lucene.index.CheckIndex index

Read the output: it lists the segment, the number of docs, and a line for each part — field infos, field norms, terms index, stored fields, term vectors, docvalues, points, and vectors (it reports the vector fields and their dimension/similarity). Confirm:

• No problems were detected with this index.
• The vectors check reports field embedding, dimension 8, similarity COSINE.
• The points check reports field year.

Step 5: Decode with Luke (the GUI inspector)

Luke is Lucene's index browser. Launch it from your apache/lucene checkout:

cd /path/to/apache/lucene
./gradlew :lucene:luke:run

In the GUI:

• Open ~/lucene-lab/index.
• Overview tab: see the fields (body, year, embedding, id) and per-field term counts.
• Click the body field → browse the terms (brown, fox, quick, …) and their doc frequencies — this is the inverted index from the chapter, visualized.
• Documents tab: step through docs; see the stored id.
• Commits / Segments: see segment _0, its codec, and the files it owns.

Luke is built on the same reader API you use next — it is a GUI over DirectoryReader.

Step 6: Decode with the reader API (the real skill)

A GUI is nice; the skill is doing it in code, because that is how you debug and how the search layer works. This program opens the index and enumerates segments, fields, per-field codec formats, and terms:

CrackOpenReader.java:

import org.apache.lucene.codecs.Codec;
import org.apache.lucene.index.DirectoryReader;
import org.apache.lucene.index.FieldInfo;
import org.apache.lucene.index.LeafReader;
import org.apache.lucene.index.LeafReaderContext;
import org.apache.lucene.index.SegmentReader;
import org.apache.lucene.index.Terms;
import org.apache.lucene.index.TermsEnum;
import org.apache.lucene.store.FSDirectory;
import org.apache.lucene.util.BytesRef;

import java.nio.file.Paths;

public class CrackOpenReader {
    public static void main(String[] args) throws Exception {
        try (DirectoryReader reader = DirectoryReader.open(FSDirectory.open(Paths.get("index")))) {
            System.out.println("maxDoc=" + reader.maxDoc() + " numDocs=" + reader.numDocs()
                + " leaves(segments)=" + reader.leaves().size());

            // One LeafReader per segment.
            for (LeafReaderContext ctx : reader.leaves()) {
                LeafReader leaf = ctx.reader();
                System.out.println("\n=== segment ord " + ctx.ord + " (docBase=" + ctx.docBase + ") ===");

                // The codec that wrote this segment.
                if (leaf instanceof SegmentReader sr) {
                    Codec codec = sr.getSegmentInfo().info.getCodec();
                    System.out.println("codec: " + codec.getName() + " (" + codec.getClass().getName() + ")");
                }

                // Field infos: every field and the features it carries.
                System.out.println("fields:");
                for (FieldInfo fi : leaf.getFieldInfos()) {
                    StringBuilder feats = new StringBuilder();
                    if (fi.getIndexOptions() != org.apache.lucene.index.IndexOptions.NONE) feats.append("postings ");
                    if (fi.getPointDimensionCount() > 0) feats.append("points(" + fi.getPointDimensionCount() + "d) ");
                    if (fi.getDocValuesType() != org.apache.lucene.index.DocValuesType.NONE)
                        feats.append("docvalues(" + fi.getDocValuesType() + ") ");
                    if (fi.getVectorDimension() > 0)
                        feats.append("vectors(dim=" + fi.getVectorDimension()
                            + ", sim=" + fi.getVectorSimilarityFunction() + ") ");
                    System.out.println("  - " + fi.name + ": " + feats.toString().trim());
                }

                // Enumerate the terms of the analyzed text field.
                Terms terms = leaf.terms("body");
                if (terms != null) {
                    System.out.println("terms of 'body' (term -> docFreq):");
                    TermsEnum te = terms.iterator();
                    BytesRef t;
                    while ((t = te.next()) != null) {
                        System.out.println("  " + t.utf8ToString() + " -> " + te.docFreq());
                    }
                }
            }
        }
    }
}

javac -cp "$CORE" CrackOpenReader.java
java  -cp "$CORE:." --add-modules jdk.incubator.vector CrackOpenReader

Expected (abridged):

maxDoc=4 numDocs=4 leaves(segments)=1

=== segment ord 0 (docBase=0) ===
codec: Lucene101  (org.apache.lucene.codecs.lucene101.Lucene101Codec)   # version varies — grep to confirm
fields:
  - id: docvalues(...)            # or just stored, depending
  - body: postings
  - year: points(1d) docvalues(SORTED_NUMERIC)
  - embedding: vectors(dim=8, sim=COSINE)
terms of 'body' (term -> docFreq):
  a -> 2
  brown -> 1
  fox -> 2
  ...

Note: The codec name (Lucene101, Lucene103, …) depends on your Lucene version. Do not hard-code it — read it from info.getCodec() as above. This is the "grep to confirm the real name" discipline from the deep dives.

Step 7: Do it on a real OpenSearch shard

The whole point — these are the same files. Start OpenSearch, index a doc, find the shard, and run CheckIndex on it:

# In the OpenSearch repo:
./gradlew run &
curl -s -XPOST 'localhost:9200/lab/_doc?refresh' -H 'Content-Type: application/json' \
  -d '{"body":"opensearch indices are lucene indices"}'

# Find the shard's Lucene index directory on disk:
find . -path "*indices*0/index/segments_*" 2>/dev/null | head
SHARD=$(dirname "$(find . -path '*indices*0/index/segments_*' 2>/dev/null | head -1)")
find "$SHARD" -type f | sed 's/.*\.//' | sort | uniq -c    # same extensions you mapped in Step 3

CheckIndex works on it identically (stop the node first or use a copy, since the files are open):

cp -r "$SHARD" /tmp/shard-copy
java -cp "$CORE" org.apache.lucene.index.CheckIndex /tmp/shard-copy

• Confirm the OpenSearch shard directory has the same .tim/.kdd/.dvd (and .vec if you added a knn_vector field) extensions. They are Lucene segments.

Implementation Requirements / Deliverables

• CrackOpenIndexer builds an index exercising postings, stored fields, points, docvalues, and vectors (non-compound so files are separate).
• A find listing of index/ with every extension mapped to its chapter in Step 1's table.
• CheckIndex output showing no problems and reporting the vector field (dim, similarity) and the point field.
• Luke opened on the index; you browsed the body terms and saw segment _0's codec/files.
• CrackOpenReader prints, per segment: the codec name, every field with its features, and the body terms with doc frequencies.
• You ran find/CheckIndex against a real OpenSearch shard and confirmed identical file types.

Troubleshooting

Validation / Self-check

1. List every file extension your index produced and name the codec component and chapter for each.
2. Which three files make up the vector data, and what does each hold? Why is .vec usually the biggest of the three for a high-dimensional field?
3. What does CheckIndex verify, and why is it the first tool a committer runs on a suspected-corrupt index? Name three components it checks.
4. In CrackOpenReader, what is a LeafReader and how does it relate to a segment? How did you read the codec that wrote a segment?
5. How did you enumerate the terms of body, and what does docFreq() mean? Tie it back to the inverted index chapter.
6. Show that an OpenSearch shard is a Lucene index: which extensions did it share with your standalone index, and which tool proved it?
7. Why did we set setUseCompoundFile(false), and what would change in the find output if we left it on?

When you can crack open any Lucene (or OpenSearch shard) index, map its files, and enumerate its contents three ways, you are ready to modify the codec: continue to Lab L2: Write a Custom Codec.