Lab L2: Write a Custom Codec

Background

In Lab L1 you cracked open an index and saw that every file — .tim, .kdd, .dvd, .vec — is written by a codec component. The segments and codecs chapter explained the codec SPI: Codec is the top-level format plugin, and it delegates to a PostingsFormat, DocValuesFormat, KnnVectorsFormat, StoredFieldsFormat, and PointsFormat, each pluggable via META-INF/services. This lab makes you write one.

You will build a custom Codec that delegates to the current default but wraps one component to observe and (optionally) influence what Lucene writes — a FilterCodec that logs every postings format and vector format decision, plus a thin PostingsFormat wrapper that counts the terms written. You will register it via SPI, drive it from a Lucene unit test, and see exactly how OpenSearch's k-NN plugin uses this same mechanism to inject its faiss/lucene vector formats and how OpenSearch exposes codecs through the index.codec setting.

This is the canonical "extend the storage layer" exercise. The k-NN plugin's KNN990Codec (and its versioned siblings) is precisely a FilterCodec that overrides knnVectorsFormat(). After this lab, that class will read like something you could have written.

Note: A custom codec is one of the most powerful extension points in Lucene — and one of the most dangerous. A codec change affects how bytes are written to immutable segments; get it wrong and you corrupt or can't reopen the index. This lab uses the safe pattern: delegate to the default, wrap, observe. Never hand-roll a binary format for a real index without a strong reason and extensive testing.

Why This Lab Matters for Contributors

• The k-NN plugin's entire native-vector storage (faiss/nmslib graphs as segment files) is delivered through a custom FilterCodec overriding knnVectorsFormat(). Understanding FilterCodec is a prerequisite to reading or contributing to k-NN's codec layer.
• OpenSearch's index.codec setting (default, best_compression, zstd, …) is the codec SPI surfaced to users. Knowing how codecs are selected explains that setting.
• The META-INF/services SPI is how every Lucene format is discovered. Writing one teaches you Java's ServiceLoader mechanism, which recurs throughout OpenSearch (analyzers, score functions).
• Wrapping a format to log/transform is a real debugging and experimentation technique — e.g. to see which vector format a field actually gets, or to A/B a quantization setting.

Prerequisites

• Lab L1 done — you can build and inspect an index.
• An apache/lucene checkout (./gradlew assemble done) — for running the unit test and resolving jars.
• JDK 21.
• Know the current default codec name in your Lucene version (it moves: Lucene101Codec, Lucene103Codec, …). Find it:

  grep -rn "DEFAULT_CODEC\|public static final Codec" \
    lucene/core/src/java/org/apache/lucene/codecs/Codec.java
  grep -rln "extends FilterCodec\|new Lucene.*Codec()" lucene/core/src/java | head
  # Or read it at runtime: System.out.println(Codec.getDefault().getName());
  

Step-by-Step Tasks

Step 1: Understand the codec hierarchy you are extending

flowchart TD
    SPI["META-INF/services/org.apache.lucene.codecs.Codec"] --> MyCodec["LoggingCodec extends FilterCodec"]
    MyCodec -->|delegate| Default["Lucene10xCodec (the current default)"]
    MyCodec -->|override postingsFormat| PF["LoggingPostingsFormat extends PostingsFormat"]
    MyCodec -->|override knnVectorsFormat| VF["(log which vector format the field gets)"]
    PF -->|delegate fieldsConsumer/Producer| DefaultPF["Lucene...PostingsFormat"]

Step 2: Write the FilterCodec

FilterCodec is the safe base: its constructor takes a name and a delegate, and every xxxFormat() method returns the delegate's by default. You override only what you want to change:

import org.apache.lucene.codecs.Codec;
import org.apache.lucene.codecs.FilterCodec;
import org.apache.lucene.codecs.KnnVectorsFormat;
import org.apache.lucene.codecs.PostingsFormat;

/**
 * A codec that behaves exactly like the default codec, but:
 *  - wraps the postings format to count/log terms written, and
 *  - logs which KNN vectors format each field is assigned.
 */
public class LoggingCodec extends FilterCodec {

    private final PostingsFormat postingsFormat;

    public LoggingCodec() {
        // SPI name "Logging", delegate = whatever the current default codec is.
        super("Logging", Codec.getDefault());
        this.postingsFormat = new LoggingPostingsFormat(super.postingsFormat());
    }

    @Override
    public PostingsFormat postingsFormat() {
        return postingsFormat;        // our wrapper instead of the delegate's
    }

    @Override
    public KnnVectorsFormat knnVectorsFormat() {
        KnnVectorsFormat delegate = super.knnVectorsFormat();
        // This is the exact hook k-NN uses to inject faiss/lucene vector formats.
        System.out.println("[LoggingCodec] knnVectorsFormat -> " + delegate.getClass().getSimpleName());
        return delegate;              // pass through; a real plugin would return its own format here
    }
}

Note: super("Logging", Codec.getDefault()) captures the default at construction time. That is fine for a lab. A production FilterCodec (like k-NN's) typically delegates to a specific versioned codec (e.g. new Lucene101Codec()) so its on-disk format is pinned and reproducible, not dependent on whatever default is active.

Step 3: Write the wrapping PostingsFormat

A PostingsFormat produces a FieldsConsumer (write side) and FieldsProducer (read side). We delegate both, but wrap the consumer to count terms as they are written:

import org.apache.lucene.codecs.FieldsConsumer;
import org.apache.lucene.codecs.FieldsProducer;
import org.apache.lucene.codecs.PostingsFormat;
import org.apache.lucene.index.Fields;
import org.apache.lucene.index.SegmentReadState;
import org.apache.lucene.index.SegmentWriteState;
import org.apache.lucene.index.Terms;

import java.io.IOException;

/** Delegates to a real PostingsFormat but logs how many fields/terms get written. */
public class LoggingPostingsFormat extends PostingsFormat {

    private final PostingsFormat delegate;

    /** Public no-arg ctor required for SPI: wraps the default. */
    public LoggingPostingsFormat() {
        this(PostingsFormat.forName(defaultPostingsName()));
    }

    public LoggingPostingsFormat(PostingsFormat delegate) {
        super("Logging");                 // SPI name for this PostingsFormat
        this.delegate = delegate;
    }

    private static String defaultPostingsName() {
        // The default postings format's SPI name (e.g. "Lucene101"). Read it at runtime.
        return org.apache.lucene.codecs.Codec.getDefault().postingsFormat().getName();
    }

    @Override
    public FieldsConsumer fieldsConsumer(SegmentWriteState state) throws IOException {
        FieldsConsumer inner = delegate.fieldsConsumer(state);
        return new FieldsConsumer() {
            @Override
            public void write(Fields fields, org.apache.lucene.index.NormsProducer norms) throws IOException {
                long fieldCount = 0, termCount = 0;
                for (String field : fields) {
                    fieldCount++;
                    Terms terms = fields.terms(field);
                    if (terms != null) termCount += terms.size();   // -1 if unknown; fine for a log
                }
                System.out.println("[LoggingPostingsFormat] segment=" + state.segmentInfo.name
                    + " fields=" + fieldCount + " terms=" + termCount);
                inner.write(fields, norms);     // do the real write
            }

            @Override
            public void close() throws IOException { inner.close(); }
        };
    }

    @Override
    public FieldsProducer fieldsProducer(SegmentReadState state) throws IOException {
        return delegate.fieldsProducer(state);   // read path: pure delegation
    }
}

Warning: The wrapped write path must call inner.write(...) exactly once and propagate close(). Forgetting either produces an empty or unclosed postings file — and a corrupt segment. The "delegate, don't reimplement" rule is what keeps this safe.

Step 4: Register both via META-INF/services

ServiceLoader discovers formats by reading text files listing implementation class names. Create two files (use your real package; this example uses the default package):

mkdir -p src/main/resources/META-INF/services
# The Codec SPI file:
printf 'LoggingCodec\n' > src/main/resources/META-INF/services/org.apache.lucene.codecs.Codec
# The PostingsFormat SPI file:
printf 'LoggingPostingsFormat\n' > src/main/resources/META-INF/services/org.apache.lucene.codecs.PostingsFormat

(If your classes are in a package, list the fully qualified names, e.g. com.example.LoggingCodec.) On the classpath, both files must be visible. Confirm the format:

src/main/resources/META-INF/services/
├── org.apache.lucene.codecs.Codec            # contains: LoggingCodec
└── org.apache.lucene.codecs.PostingsFormat   # contains: LoggingPostingsFormat

Note: Lucene's own codecs ship the same way — look at lucene-core.jar's META-INF/services/org.apache.lucene.codecs.Codec to see the built-in list. The k-NN plugin's jar has a META-INF/services/org.apache.lucene.codecs.Codec listing its KNN*Codec.

Step 5: Drive it from a test (standalone)

Wire the codec into an IndexWriterConfig and index a doc — you'll see the log lines fire:

import org.apache.lucene.codecs.Codec;
import org.apache.lucene.document.Document;
import org.apache.lucene.document.Field;
import org.apache.lucene.document.TextField;
import org.apache.lucene.index.IndexWriter;
import org.apache.lucene.index.IndexWriterConfig;
import org.apache.lucene.store.ByteBuffersDirectory;
import org.apache.lucene.store.Directory;

public class LoggingCodecDemo {
    public static void main(String[] args) throws Exception {
        Directory dir = new ByteBuffersDirectory();
        IndexWriterConfig cfg = new IndexWriterConfig();
        cfg.setCodec(new LoggingCodec());          // <-- inject our codec
        try (IndexWriter w = new IndexWriter(dir, cfg)) {
            Document d = new Document();
            d.add(new TextField("body", "the quick brown fox", Field.Store.NO));
            w.addDocument(d);
            w.commit();                            // triggers the postings write -> our log line
        }
        System.out.println("default codec name: " + Codec.getDefault().getName());
    }
}

CORE=$(find /path/to/apache/lucene -name "lucene-core-*.jar" | grep -v sources | head -1)
javac -cp "$CORE" LoggingCodec.java LoggingPostingsFormat.java LoggingCodecDemo.java
java  -cp "$CORE:.:src/main/resources" --add-modules jdk.incubator.vector LoggingCodecDemo

Expected:

[LoggingPostingsFormat] segment=_0 fields=1 terms=4
[LoggingCodec] knnVectorsFormat -> Lucene...HnswVectorsFormat
default codec name: Lucene101    # version varies

Step 6: Run it as a Lucene unit test

Lucene has a strong testing framework. To run a real Lucene unit test that exercises a codec, use the Gradle test task (the canonical contributor command):

cd /path/to/apache/lucene
# Run an existing codec/SPI test to see the harness:
./gradlew :lucene:core:test --tests "org.apache.lucene.codecs.TestCodecLoadingDeadlock"
# Run the postings-format SPI test:
./gradlew :lucene:core:test --tests "*TestNamedSPILoader*"

To make LoggingCodec itself testable, drop it into a Lucene test module (or your own Gradle module that depends on lucene-core + lucene-test-framework) and extend LuceneTestCase:

import org.apache.lucene.codecs.Codec;
import org.apache.lucene.tests.util.LuceneTestCase;

public class LoggingCodecTest extends LuceneTestCase {
    public void testSpiLoads() {
        // Proves the META-INF/services file is on the classpath and the codec is discoverable by name.
        Codec c = Codec.forName("Logging");
        assertNotNull(c);
        assertEquals("Logging", c.getName());
    }

    public void testRoundTrip() throws Exception {
        // A real round-trip: index with LoggingCodec, reopen, read back.
        // Use newDirectory()/newIndexWriterConfig() from LuceneTestCase, set cfg.setCodec(new LoggingCodec()).
        // Index a doc, commit, DirectoryReader.open, assert numDocs()==1.
    }
}

./gradlew :lucene:core:test --tests "*LoggingCodecTest*"

Note: Codec.forName("Logging") succeeding is the proof your SPI registration works — it is ServiceLoader finding your META-INF/services entry. If it throws IllegalArgumentException: An SPI class of type ... with name 'Logging' does not exist, your SPI file is missing, misnamed, or not on the classpath.

Step 7: How this plugs into OpenSearch

Two connection points:

1. The k-NN plugin's codec. k-NN registers a FilterCodec (historically named per Lucene version, e.g. KNN990Codec / a KNNCodecVersion enum) whose knnVectorsFormat() returns either Lucene's HNSW format (lucene engine) or a native format that writes the faiss/nmslib graph as segment files. It is the Step 2 pattern, with the override actually returning a custom format instead of passing through:

# In a k-NN checkout:
grep -rln "extends FilterCodec\|knnVectorsFormat\|KNNCodecVersion\|KNN.*Codec" src/main/java
find . -path "*META-INF/services/org.apache.lucene.codecs.Codec"

2. The index.codec setting. OpenSearch lets users pick a codec per index (index.codec: default | best_compression | zstd | zstd_no_dict | ...). Each name maps to a Codec in core. To find how OpenSearch resolves it:

# In the OpenSearch repo:
grep -rln "index.codec\|CodecService\|class CodecService" server/src/main/java | head
grep -rn "best_compression\|Lucene.*Codec\|zstd" \
  server/src/main/java/org/opensearch/index/codec/CodecService.java 2>/dev/null | head

A custom plugin codec is registered through EnginePlugin/a codec service provider so an index can select it by name — the same SPI idea, surfaced as an OpenSearch setting. (Building a full OpenSearch-installable codec plugin is a stretch goal; the mechanism is the FilterCodec you wrote.)

Implementation Requirements / Deliverables

• LoggingCodec extends FilterCodec delegating to the default, overriding postingsFormat() and knnVectorsFormat() (the latter logging the assigned format).
• LoggingPostingsFormat extends PostingsFormat that delegates read/write and logs field/term counts on the write path, calling inner.write(...) exactly once.
• Two META-INF/services files registering the codec and the postings format by name.
• Codec.forName("Logging") succeeds (SPI proof), demonstrated in a test.
• An index built with LoggingCodec round-trips: reopen with DirectoryReader, numDocs() matches; the log lines fired during the write.
• You located the k-NN plugin's FilterCodec (and its knnVectorsFormat override) and the OpenSearch CodecService/index.codec mapping with grep.

Troubleshooting

Coding Exercises

Each exercise extends the LoggingCodec / LoggingPostingsFormat you wrote and is proven by a passing round-trip — index, reopen with DirectoryReader, read back. The iron rule of codecs holds throughout: delegate, never reimplement the binary format; you only wrap to observe or to route to a different delegate. Run them either standalone (javac -cp "$CORE") or inside a Gradle module that depends on lucene-core + lucene-test-framework.

1. (warm-up) Make the term count an assertion, not a log line. Refactor LoggingPostingsFormat so the wrapper accumulates fields/terms totals into accessible counters (e.g. a static AtomicLong or a field on the format). Then write a LuceneTestCase test that indexes a document with a known body ("the quick brown fox" → 4 terms) using LoggingCodec, commits, and asserts the counter equals 4. This turns Step 5's "you'll see the log line" into a graded, repeatable check.

2. (core) A real round-trip test. Fill in the testRoundTrip() stub from Step 6: use newDirectory() and newIndexWriterConfig(...) from LuceneTestCase, call cfg.setCodec(new LoggingCodec()), index three documents (text + a KnnFloatVectorField so the knnVectorsFormat() override fires), w.commit(), then DirectoryReader.open(dir) and assert reader.numDocs() == 3. Additionally read back one term via leaf.terms("body").iterator() and assert a known term is present — proving the wrapped write and the delegated read are byte-compatible.

3. (core) Wrap a DocValuesFormat too. The lab wraps postings; now do the same for docvalues. Add LoggingDocValuesFormat extends DocValuesFormat that delegates fieldsConsumer/fieldsProducer to Codec.getDefault().docValuesFormat() and logs the field names written, and override docValuesFormat() in LoggingCodec to return it. Register it with a third META-INF/services file (org.apache.lucene.codecs.DocValuesFormat). Index a SortedNumericDocValuesField, run, and confirm both the postings and docvalues wrappers fire. find rg "extends DocValuesFormat" in your Lucene checkout for the exact method signatures on your version.

4. (core) A per-field codec. Instead of wrapping globally, route formats per field. Subclass org.apache.lucene.codecs.perfield.PerFieldPostingsFormat (or override LoggingCodec.postingsFormat() to return a PerFieldPostingsFormat whose getPostingsFormatForField(field) returns your LoggingPostingsFormat only for field "body" and the default for everything else). Index two text fields, and assert via your counter from Exercise 1 that the wrapper saw "body" but not the other field. This is the exact mechanism the segments-and-codecs chapter describes for mixing formats by field.

5. (advanced) Tweak the format, not just the wrapper. Build BlockSizePostingsFormat that does not wrap the default but instead constructs new Lucene101PostingsFormat(minBlockSize, maxBlockSize) (grep your version for the real class name and constructor: rg "class Lucene.*PostingsFormat" lucene/core/src/java). Index the same corpus with two different block-size settings, write each to its own directory, and use the Lab L1 find/fileLength technique to compare .tim/.tip byte sizes. Assert in a test that both indexes round-trip (numDocs and a sample term match) despite the different on-disk layout — proving you changed the format parameters safely while preserving read compatibility.

6. (advanced) Advanced challenge — a KNN990Codec look-alike. Write LabKnnCodec extends FilterCodec whose knnVectorsFormat() returns an actual different vector format chosen by a constructor flag: a plain Lucene99HnswVectorsFormat(m, efConstruction) vs a Lucene99HnswScalarQuantizedVectorsFormat (grep for the exact quantized class on your version). Index the same 8-dim vectors with each, round-trip both, and use Lab L1's reader API to confirm the field still reports the right dimension/similarity. Then rg "extends FilterCodec" -l in a k-NN plugin checkout, open the KNN*Codec you find, and write one paragraph mapping its knnVectorsFormat() override onto yours — they are the same shape. This is the lab's payoff: you have written, in miniature, the class k-NN ships.

Issues to Practice On

The codec SPI lives in apache/lucene (Apache, on GitHub) — and contributing there is a different workflow from OpenSearch: no DCO Signed-off-by, no changelog bot. You build and test with ./gradlew :lucene:core:test, format with ./gradlew tidy, and hand-edit CHANGES.txt. The k-NN-plugin connection point (the index.codec setting, the KNN*Codec) lives in opensearch-project/k-NN and opensearch-project/OpenSearch, which do use DCO — know which repo you are in. (The full Lucene workflow is Lab L4.)

Note: Labels move; confirm on the tracker with gh label list. Lucene uses GitHub issues but historically tracked work as LUCENE-NNNN JIRA keys you'll still see referenced in CHANGES.txt.

Representative issue patterns for this subsystem (codecs / formats / SPI):

• A format that doesn't surface a useful knob, or a wrapper/per-field edge case. Approach: reproduce with a tiny index (you can now build one), locate the format in lucene/core/src/java/org/apache/lucene/codecs/ via rg "class .*Format", add the option plus a round-trip test in lucene/core/src/test/.../codecs/, then ./gradlew :lucene:core:test --tests "*<YourTest>*". SPI/ServiceLoader issues are similar: reproduce a name-resolution failure, fix the META-INF/services handling or the NamedSPILoader message, add a test.
• (k-NN side, DCO required) a KNN*Codec not handling a new field type or quantization mode. Locate via rg "extends FilterCodec|knnVectorsFormat" src/main/java, fix, add a test, PR with a CHANGELOG entry and git commit -s.

Planted-bug drill. In LoggingPostingsFormat.fieldsConsumer, comment out the inner.write(fields, norms) call (keep the log line and the close()). Run your round-trip test from Exercise 2 — watch it fail to reopen (CorruptIndexException/empty postings), exactly the Step 3 warning made real. Now fix it by restoring the single inner.write(...), and add an assertion to the test that catches this class of bug cheaply: after commit, assert leaf.terms("body").size() > 0. You have written the regression test for "the wrapper forgot to delegate the write." For the etiquette of taking either repo's fix upstream — claim first, reproduce first; Lucene PRs need a test + CHANGES.txt (no DCO), OpenSearch/k-NN PRs need a test + CHANGELOG + git commit -s — see community interaction.

Validation / Self-check

1. What does FilterCodec give you, and why is "delegate + override only what you need" the safe pattern for a codec?
2. Which two META-INF/services files did you create, what are their exact filenames, and what does each contents line mean to ServiceLoader?
3. In LoggingPostingsFormat, why must the wrapped FieldsConsumer.write call inner.write(...) exactly once and propagate close()? What breaks otherwise?
4. How does Codec.forName("Logging") prove your registration worked? What exception do you get if it didn't, and what are the three usual causes?
5. Which Codec method does the k-NN plugin override to inject faiss/lucene vector storage, and how is that the same shape as your LoggingCodec?
6. What is the OpenSearch index.codec setting, and which core class resolves a codec name to a Codec?
7. Why should you almost never hand-roll a new binary format (vs wrapping) for a real index?

When Codec.forName("Logging") resolves, your wrapper logs during a write, and the index round-trips, you understand the codec SPI well enough to read k-NN's codec. Next, build a real vector index and benchmark it in Lab L3: Build an HNSW Graph from Scratch.