Lab L2: Write a Custom Codec

Background

In Lab L1 you cracked open an index and saw that every file — .tim, .kdd, .dvd, .vec — is written by a codec component. The segments and codecs chapter explained the codec SPI: Codec is the top-level format plugin, and it delegates to a PostingsFormat, DocValuesFormat, KnnVectorsFormat, StoredFieldsFormat, and PointsFormat, each pluggable via META-INF/services. This lab makes you write one.

You will build a custom Codec that delegates to the current default but wraps one component to observe and (optionally) influence what Lucene writes — a FilterCodec that logs every postings format and vector format decision, plus a thin PostingsFormat wrapper that counts the terms written. You will register it via SPI, drive it from a Lucene unit test, and see exactly how OpenSearch's k-NN plugin uses this same mechanism to inject its faiss/lucene vector formats and how OpenSearch exposes codecs through the index.codec setting.

This is the canonical "extend the storage layer" exercise. The k-NN plugin's KNN990Codec (and its versioned siblings) is precisely a FilterCodec that overrides knnVectorsFormat(). After this lab, that class will read like something you could have written.

Note: A custom codec is one of the most powerful extension points in Lucene — and one of the most dangerous. A codec change affects how bytes are written to immutable segments; get it wrong and you corrupt or can't reopen the index. This lab uses the safe pattern: delegate to the default, wrap, observe. Never hand-roll a binary format for a real index without a strong reason and extensive testing.

Why This Lab Matters for Contributors

• The k-NN plugin's entire native-vector storage (faiss/nmslib graphs as segment files) is delivered through a custom FilterCodec overriding knnVectorsFormat(). Understanding FilterCodec is a prerequisite to reading or contributing to k-NN's codec layer.
• OpenSearch's index.codec setting (default, best_compression, zstd, …) is the codec SPI surfaced to users. Knowing how codecs are selected explains that setting.
• The META-INF/services SPI is how every Lucene format is discovered. Writing one teaches you Java's ServiceLoader mechanism, which recurs throughout OpenSearch (analyzers, score functions).
• Wrapping a format to log/transform is a real debugging and experimentation technique — e.g. to see which vector format a field actually gets, or to A/B a quantization setting.

Prerequisites

• Lab L1 done — you can build and inspect an index.
• An apache/lucene checkout (./gradlew assemble done) — for running the unit test and resolving jars.
• JDK 21.
• Know the current default codec name in your Lucene version (it moves: Lucene101Codec, Lucene103Codec, …). Find it:

  grep -rn "DEFAULT_CODEC\|public static final Codec" \
    lucene/core/src/java/org/apache/lucene/codecs/Codec.java
  grep -rln "extends FilterCodec\|new Lucene.*Codec()" lucene/core/src/java | head
  # Or read it at runtime: System.out.println(Codec.getDefault().getName());
  

Step-by-Step Tasks

Step 1: Understand the codec hierarchy you are extending

flowchart TD
    SPI["META-INF/services/org.apache.lucene.codecs.Codec"] --> MyCodec["LoggingCodec extends FilterCodec"]
    MyCodec -->|delegate| Default["Lucene10xCodec (the current default)"]
    MyCodec -->|override postingsFormat| PF["LoggingPostingsFormat extends PostingsFormat"]
    MyCodec -->|override knnVectorsFormat| VF["(log which vector format the field gets)"]
    PF -->|delegate fieldsConsumer/Producer| DefaultPF["Lucene...PostingsFormat"]

Step 2: Write the FilterCodec

FilterCodec is the safe base: its constructor takes a name and a delegate, and every xxxFormat() method returns the delegate's by default. You override only what you want to change:

import org.apache.lucene.codecs.Codec;
import org.apache.lucene.codecs.FilterCodec;
import org.apache.lucene.codecs.KnnVectorsFormat;
import org.apache.lucene.codecs.PostingsFormat;

/**
 * A codec that behaves exactly like the default codec, but:
 *  - wraps the postings format to count/log terms written, and
 *  - logs which KNN vectors format each field is assigned.
 */
public class LoggingCodec extends FilterCodec {

    private final PostingsFormat postingsFormat;

    public LoggingCodec() {
        // SPI name "Logging", delegate = whatever the current default codec is.
        super("Logging", Codec.getDefault());
        this.postingsFormat = new LoggingPostingsFormat(super.postingsFormat());
    }

    @Override
    public PostingsFormat postingsFormat() {
        return postingsFormat;        // our wrapper instead of the delegate's
    }

    @Override
    public KnnVectorsFormat knnVectorsFormat() {
        KnnVectorsFormat delegate = super.knnVectorsFormat();
        // This is the exact hook k-NN uses to inject faiss/lucene vector formats.
        System.out.println("[LoggingCodec] knnVectorsFormat -> " + delegate.getClass().getSimpleName());
        return delegate;              // pass through; a real plugin would return its own format here
    }
}

Note: super("Logging", Codec.getDefault()) captures the default at construction time. That is fine for a lab. A production FilterCodec (like k-NN's) typically delegates to a specific versioned codec (e.g. new Lucene101Codec()) so its on-disk format is pinned and reproducible, not dependent on whatever default is active.

Step 3: Write the wrapping PostingsFormat

A PostingsFormat produces a FieldsConsumer (write side) and FieldsProducer (read side). We delegate both, but wrap the consumer to count terms as they are written:

import org.apache.lucene.codecs.FieldsConsumer;
import org.apache.lucene.codecs.FieldsProducer;
import org.apache.lucene.codecs.PostingsFormat;
import org.apache.lucene.index.Fields;
import org.apache.lucene.index.SegmentReadState;
import org.apache.lucene.index.SegmentWriteState;
import org.apache.lucene.index.Terms;

import java.io.IOException;

/** Delegates to a real PostingsFormat but logs how many fields/terms get written. */
public class LoggingPostingsFormat extends PostingsFormat {

    private final PostingsFormat delegate;

    /** Public no-arg ctor required for SPI: wraps the default. */
    public LoggingPostingsFormat() {
        this(PostingsFormat.forName(defaultPostingsName()));
    }

    public LoggingPostingsFormat(PostingsFormat delegate) {
        super("Logging");                 // SPI name for this PostingsFormat
        this.delegate = delegate;
    }

    private static String defaultPostingsName() {
        // The default postings format's SPI name (e.g. "Lucene101"). Read it at runtime.
        return org.apache.lucene.codecs.Codec.getDefault().postingsFormat().getName();
    }

    @Override
    public FieldsConsumer fieldsConsumer(SegmentWriteState state) throws IOException {
        FieldsConsumer inner = delegate.fieldsConsumer(state);
        return new FieldsConsumer() {
            @Override
            public void write(Fields fields, org.apache.lucene.index.NormsProducer norms) throws IOException {
                long fieldCount = 0, termCount = 0;
                for (String field : fields) {
                    fieldCount++;
                    Terms terms = fields.terms(field);
                    if (terms != null) termCount += terms.size();   // -1 if unknown; fine for a log
                }
                System.out.println("[LoggingPostingsFormat] segment=" + state.segmentInfo.name
                    + " fields=" + fieldCount + " terms=" + termCount);
                inner.write(fields, norms);     // do the real write
            }

            @Override
            public void close() throws IOException { inner.close(); }
        };
    }

    @Override
    public FieldsProducer fieldsProducer(SegmentReadState state) throws IOException {
        return delegate.fieldsProducer(state);   // read path: pure delegation
    }
}

Warning: The wrapped write path must call inner.write(...) exactly once and propagate close(). Forgetting either produces an empty or unclosed postings file — and a corrupt segment. The "delegate, don't reimplement" rule is what keeps this safe.

Step 4: Register both via META-INF/services

ServiceLoader discovers formats by reading text files listing implementation class names. Create two files (use your real package; this example uses the default package):

mkdir -p src/main/resources/META-INF/services
# The Codec SPI file:
printf 'LoggingCodec\n' > src/main/resources/META-INF/services/org.apache.lucene.codecs.Codec
# The PostingsFormat SPI file:
printf 'LoggingPostingsFormat\n' > src/main/resources/META-INF/services/org.apache.lucene.codecs.PostingsFormat

(If your classes are in a package, list the fully qualified names, e.g. com.example.LoggingCodec.) On the classpath, both files must be visible. Confirm the format:

src/main/resources/META-INF/services/
├── org.apache.lucene.codecs.Codec            # contains: LoggingCodec
└── org.apache.lucene.codecs.PostingsFormat   # contains: LoggingPostingsFormat

Note: Lucene's own codecs ship the same way — look at lucene-core.jar's META-INF/services/org.apache.lucene.codecs.Codec to see the built-in list. The k-NN plugin's jar has a META-INF/services/org.apache.lucene.codecs.Codec listing its KNN*Codec.

Step 5: Drive it from a test (standalone)

Wire the codec into an IndexWriterConfig and index a doc — you'll see the log lines fire:

import org.apache.lucene.codecs.Codec;
import org.apache.lucene.document.Document;
import org.apache.lucene.document.Field;
import org.apache.lucene.document.TextField;
import org.apache.lucene.index.IndexWriter;
import org.apache.lucene.index.IndexWriterConfig;
import org.apache.lucene.store.ByteBuffersDirectory;
import org.apache.lucene.store.Directory;

public class LoggingCodecDemo {
    public static void main(String[] args) throws Exception {
        Directory dir = new ByteBuffersDirectory();
        IndexWriterConfig cfg = new IndexWriterConfig();
        cfg.setCodec(new LoggingCodec());          // <-- inject our codec
        try (IndexWriter w = new IndexWriter(dir, cfg)) {
            Document d = new Document();
            d.add(new TextField("body", "the quick brown fox", Field.Store.NO));
            w.addDocument(d);
            w.commit();                            // triggers the postings write -> our log line
        }
        System.out.println("default codec name: " + Codec.getDefault().getName());
    }
}

CORE=$(find /path/to/apache/lucene -name "lucene-core-*.jar" | grep -v sources | head -1)
javac -cp "$CORE" LoggingCodec.java LoggingPostingsFormat.java LoggingCodecDemo.java
java  -cp "$CORE:.:src/main/resources" --add-modules jdk.incubator.vector LoggingCodecDemo

Expected:

[LoggingPostingsFormat] segment=_0 fields=1 terms=4
[LoggingCodec] knnVectorsFormat -> Lucene...HnswVectorsFormat
default codec name: Lucene101    # version varies

Step 6: Run it as a Lucene unit test

Lucene has a strong testing framework. To run a real Lucene unit test that exercises a codec, use the Gradle test task (the canonical contributor command):

cd /path/to/apache/lucene
# Run an existing codec/SPI test to see the harness:
./gradlew :lucene:core:test --tests "org.apache.lucene.codecs.TestCodecLoadingDeadlock"
# Run the postings-format SPI test:
./gradlew :lucene:core:test --tests "*TestNamedSPILoader*"

To make LoggingCodec itself testable, drop it into a Lucene test module (or your own Gradle module that depends on lucene-core + lucene-test-framework) and extend LuceneTestCase:

import org.apache.lucene.codecs.Codec;
import org.apache.lucene.tests.util.LuceneTestCase;

public class LoggingCodecTest extends LuceneTestCase {
    public void testSpiLoads() {
        // Proves the META-INF/services file is on the classpath and the codec is discoverable by name.
        Codec c = Codec.forName("Logging");
        assertNotNull(c);
        assertEquals("Logging", c.getName());
    }

    public void testRoundTrip() throws Exception {
        // A real round-trip: index with LoggingCodec, reopen, read back.
        // Use newDirectory()/newIndexWriterConfig() from LuceneTestCase, set cfg.setCodec(new LoggingCodec()).
        // Index a doc, commit, DirectoryReader.open, assert numDocs()==1.
    }
}

./gradlew :lucene:core:test --tests "*LoggingCodecTest*"

Note: Codec.forName("Logging") succeeding is the proof your SPI registration works — it is ServiceLoader finding your META-INF/services entry. If it throws IllegalArgumentException: An SPI class of type ... with name 'Logging' does not exist, your SPI file is missing, misnamed, or not on the classpath.

Step 7: How this plugs into OpenSearch

Two connection points:

1. The k-NN plugin's codec. k-NN registers a FilterCodec (historically named per Lucene version, e.g. KNN990Codec / a KNNCodecVersion enum) whose knnVectorsFormat() returns either Lucene's HNSW format (lucene engine) or a native format that writes the faiss/nmslib graph as segment files. It is the Step 2 pattern, with the override actually returning a custom format instead of passing through:

# In a k-NN checkout:
grep -rln "extends FilterCodec\|knnVectorsFormat\|KNNCodecVersion\|KNN.*Codec" src/main/java
find . -path "*META-INF/services/org.apache.lucene.codecs.Codec"

2. The index.codec setting. OpenSearch lets users pick a codec per index (index.codec: default | best_compression | zstd | zstd_no_dict | ...). Each name maps to a Codec in core. To find how OpenSearch resolves it:

# In the OpenSearch repo:
grep -rln "index.codec\|CodecService\|class CodecService" server/src/main/java | head
grep -rn "best_compression\|Lucene.*Codec\|zstd" \
  server/src/main/java/org/opensearch/index/codec/CodecService.java 2>/dev/null | head

A custom plugin codec is registered through EnginePlugin/a codec service provider so an index can select it by name — the same SPI idea, surfaced as an OpenSearch setting. (Building a full OpenSearch-installable codec plugin is a stretch goal; the mechanism is the FilterCodec you wrote.)

Implementation Requirements / Deliverables

• LoggingCodec extends FilterCodec delegating to the default, overriding postingsFormat() and knnVectorsFormat() (the latter logging the assigned format).
• LoggingPostingsFormat extends PostingsFormat that delegates read/write and logs field/term counts on the write path, calling inner.write(...) exactly once.
• Two META-INF/services files registering the codec and the postings format by name.
• Codec.forName("Logging") succeeds (SPI proof), demonstrated in a test.
• An index built with LoggingCodec round-trips: reopen with DirectoryReader, numDocs() matches; the log lines fired during the write.
• You located the k-NN plugin's FilterCodec (and its knnVectorsFormat override) and the OpenSearch CodecService/index.codec mapping with grep.

Troubleshooting

Validation / Self-check

1. What does FilterCodec give you, and why is "delegate + override only what you need" the safe pattern for a codec?
2. Which two META-INF/services files did you create, what are their exact filenames, and what does each contents line mean to ServiceLoader?
3. In LoggingPostingsFormat, why must the wrapped FieldsConsumer.write call inner.write(...) exactly once and propagate close()? What breaks otherwise?
4. How does Codec.forName("Logging") prove your registration worked? What exception do you get if it didn't, and what are the three usual causes?
5. Which Codec method does the k-NN plugin override to inject faiss/lucene vector storage, and how is that the same shape as your LoggingCodec?
6. What is the OpenSearch index.codec setting, and which core class resolves a codec name to a Codec?
7. Why should you almost never hand-roll a new binary format (vs wrapping) for a real index?

When Codec.forName("Logging") resolves, your wrapper logs during a write, and the index round-trips, you understand the codec SPI well enough to read k-NN's codec. Next, build a real vector index and benchmark it in Lab L3: Build an HNSW Graph from Scratch.