Lab 1.4: Project — Build a Minimal Lucene Index

Background

Every OpenSearch shard is a Lucene index. That sentence is repeated everywhere in this book, but it does not become real until you build a Lucene index with your own hands — no cluster, no REST, no org.opensearch.*, just the library underneath everything. In this build-it project you write a ~120-line standalone Java program that:

1. opens a Directory and an IndexWriter,
2. adds a few Documents,
3. commits, opens a DirectoryReader and an IndexSearcher,
4. runs a TermQuery and a BooleanQuery, and prints the hits,
5. observes segments and a merge.

Then you map each Lucene primitive to the OpenSearch class that wraps it — IndexWriter → InternalEngine, DirectoryReader/IndexSearcher → Engine.Searcher, the whole bundle → IndexShard. After this lab, "a shard is a Lucene index" is something you have seen, not something you have read.

Why This Lab Matters for Contributors

• The single most useful mental model for OpenSearch internals is "OpenSearch is distribution + durability + an API on top of Lucene." This lab installs that model permanently.
• When you trace the index path in Level 6 and the search path in Level 7, every Lucene call you see (IndexWriter.addDocument, IndexSearcher.search) will already be familiar.
• Understanding segments and merges first-hand makes the refresh/flush/merge deep dive and engine internals read like review.

Prerequisites

• Lab 1.1 complete, and you found the exact Lucene version your branch bundles (Stretch Goal 2 of that lab). You will use it here so your standalone program uses the same Lucene OpenSearch runs.
• A JDK on your PATH (17+ is fine to compile/run this tiny program).

# Re-confirm the bundled Lucene version from inside your OpenSearch checkout:
grep -rn "lucene" gradle/libs.versions.toml 2>/dev/null \
  || grep -rn "lucene" buildSrc/version.properties 2>/dev/null \
  || ./gradlew :server:dependencies --configuration compileClasspath | grep -i lucene-core
# Note the version, e.g. 9.11.1 — call it LUCENE_VERSION below.

Step-by-Step Tasks

Step 1: Create the Project Skeleton

Work outside the OpenSearch repo so you do not pollute it:

mkdir -p ~/lucene-shard/src ~/lucene-shard/lib
cd ~/lucene-shard

You need three Lucene JARs at the version OpenSearch bundles: lucene-core, lucene-queryparser, and lucene-analysis-common (the package names changed at Lucene 9; this lab targets the 9.x line OpenSearch 3.x uses).

Step 2: Obtain the Lucene JARs

You already have them — they are in your Gradle cache from building OpenSearch. Copy them so your classpath is self-contained:

# Replace 9.11.1 with YOUR LUCENE_VERSION from Step 0.
LUCENE_VERSION=9.11.1
find ~/.gradle/caches -name "lucene-core-${LUCENE_VERSION}.jar"            -exec cp {} lib/ \;
find ~/.gradle/caches -name "lucene-queryparser-${LUCENE_VERSION}.jar"     -exec cp {} lib/ \;
find ~/.gradle/caches -name "lucene-analysis-common-${LUCENE_VERSION}.jar" -exec cp {} lib/ \;
ls -1 lib/
# lucene-analysis-common-9.11.1.jar
# lucene-core-9.11.1.jar
# lucene-queryparser-9.11.1.jar

Note: If a JAR is not in the cache, run a build that needs it (./gradlew :server:assemble), or download those three artifacts from Maven Central at the matching version. Using the same Lucene OpenSearch ships is the whole point — APIs differ across major Lucene versions.

Step 3: Write the Program

Create src/MiniShard.java. This is the full, runnable source — read it as you type it.

import org.apache.lucene.analysis.standard.StandardAnalyzer;
import org.apache.lucene.document.Document;
import org.apache.lucene.document.Field;
import org.apache.lucene.document.IntPoint;
import org.apache.lucene.document.StringField;
import org.apache.lucene.document.TextField;
import org.apache.lucene.index.DirectoryReader;
import org.apache.lucene.index.IndexWriter;
import org.apache.lucene.index.IndexWriterConfig;
import org.apache.lucene.index.Term;
import org.apache.lucene.search.BooleanClause;
import org.apache.lucene.search.BooleanQuery;
import org.apache.lucene.search.IndexSearcher;
import org.apache.lucene.search.Query;
import org.apache.lucene.search.ScoreDoc;
import org.apache.lucene.search.TermQuery;
import org.apache.lucene.search.TopDocs;
import org.apache.lucene.store.ByteBuffersDirectory;
import org.apache.lucene.store.Directory;

import java.util.List;

/**
 * A minimal Lucene index — the same primitives OpenSearch wraps inside a single shard.
 *
 *   Directory        -> where segments live (we use an in-memory one)
 *   IndexWriter      -> wrapped by OpenSearch InternalEngine
 *   DirectoryReader  -> the point-in-time view a "refresh" opens
 *   IndexSearcher    -> wrapped by OpenSearch Engine.Searcher
 */
public class MiniShard {

    public static void main(String[] args) throws Exception {
        // 1. A "shard" needs a place to put its segments. In OpenSearch this is an
        //    FSDirectory on disk; here we use RAM so the program leaves no files.
        Directory dir = new ByteBuffersDirectory();

        // 2. The analyzer turns text into terms. OpenSearch's "standard" analyzer is this one.
        StandardAnalyzer analyzer = new StandardAnalyzer();
        IndexWriterConfig cfg = new IndexWriterConfig(analyzer);

        // 3. The IndexWriter is what InternalEngine.index(...) ultimately drives.
        try (IndexWriter writer = new IndexWriter(dir, cfg)) {
            writer.addDocument(doc("1", "Lucene in Action",      "search", 2010));
            writer.addDocument(doc("2", "Mastering OpenSearch",  "search", 2023));
            writer.addDocument(doc("3", "Data-Intensive Apps",   "systems", 2017));
            writer.addDocument(doc("4", "The Definitive Guide",  "search", 2015));
            // commit() == a Lucene commit == what OpenSearch "flush" does (durability point).
            writer.commit();
            System.out.println("Indexed 4 documents; segment count after first commit: "
                    + writer.getSegmentCount());

            // Add more in a second batch so a second segment forms, then force a merge.
            writer.addDocument(doc("5", "OpenSearch Internals",  "search", 2024));
            writer.commit();
            System.out.println("Segment count after second commit: " + writer.getSegmentCount());
            writer.forceMerge(1); // what POST /_forcemerge?max_num_segments=1 triggers
            writer.commit();
            System.out.println("Segment count after forceMerge(1): " + writer.getSegmentCount());
        }

        // 4. Open a point-in-time view. This is exactly what an OpenSearch "refresh" does:
        //    it opens a new DirectoryReader so newly-committed docs become searchable.
        try (DirectoryReader reader = DirectoryReader.open(dir)) {
            IndexSearcher searcher = new IndexSearcher(reader);
            System.out.println("Reader sees " + reader.numDocs() + " live docs across "
                    + reader.leaves().size() + " segment(s).");

            // 5a. A TermQuery on the analyzed 'title' field. Note: StandardAnalyzer lowercases,
            //     so we query the lowercased term.
            runQuery(searcher, "TermQuery title:opensearch",
                    new TermQuery(new Term("title", "opensearch")));

            // 5b. A TermQuery on the un-analyzed 'tag' StringField (exact match, like a keyword).
            runQuery(searcher, "TermQuery tag:search",
                    new TermQuery(new Term("tag", "search")));

            // 5c. A BooleanQuery: title contains 'opensearch' AND tag == 'search'.
            BooleanQuery bool = new BooleanQuery.Builder()
                    .add(new TermQuery(new Term("title", "opensearch")), BooleanClause.Occur.MUST)
                    .add(new TermQuery(new Term("tag", "search")),       BooleanClause.Occur.MUST)
                    .build();
            runQuery(searcher, "BooleanQuery (title:opensearch AND tag:search)", bool);
        }
    }

    /** Build a document. 'title' is analyzed (TextField); 'tag' is exact (StringField). */
    private static Document doc(String id, String title, String tag, int year) {
        Document d = new Document();
        d.add(new StringField("id", id, Field.Store.YES));      // exact, stored
        d.add(new TextField("title", title, Field.Store.YES));  // analyzed, stored
        d.add(new StringField("tag", tag, Field.Store.YES));    // exact, stored
        d.add(new IntPoint("year", year));                      // numeric, range-searchable
        return d;
    }

    private static void runQuery(IndexSearcher searcher, String label, Query q) throws Exception {
        TopDocs top = searcher.search(q, 10);
        System.out.printf("%n[%s] -> %d hit(s)%n", label, top.totalHits.value);
        List<ScoreDoc> hits = List.of(top.scoreDocs);
        for (ScoreDoc sd : hits) {
            Document d = searcher.storedFields().document(sd.doc);
            System.out.printf("    id=%s  title=%-24s  score=%.3f%n",
                    d.get("id"), d.get("title"), sd.score);
        }
    }
}

Note: Two Lucene-9 API details that bite people coming from older examples: stored-field access is searcher.storedFields().document(docId) (the old searcher.doc(id) / reader.document(id) are deprecated/removed in 9.x), and analyzer packages moved to org.apache.lucene.analysis.standard. If your bundled Lucene is a different major version, adjust these two call sites — the concepts are identical.

Step 4: Compile and Run

cd ~/lucene-shard
# Compile against the Lucene JARs:
javac -cp "lib/*" -d out src/MiniShard.java

# Run with the same classpath plus your compiled class:
java -cp "out:lib/*" MiniShard

Expected output (scores vary slightly by Lucene version):

Indexed 4 documents; segment count after first commit: 1
Segment count after second commit: 2
Segment count after forceMerge(1): 1
Reader sees 5 live docs across 1 segment(s).

[TermQuery title:opensearch] -> 2 hit(s)
    id=2  title=Mastering OpenSearch     score=...
    id=5  title=OpenSearch Internals     score=...

[TermQuery tag:search] -> 4 hit(s)
    id=1  title=Lucene in Action         score=1.000
    id=2  title=Mastering OpenSearch     score=1.000
    id=4  title=The Definitive Guide     score=1.000
    id=5  title=OpenSearch Internals     score=1.000

[BooleanQuery (title:opensearch AND tag:search)] -> 2 hit(s)
    id=2  title=Mastering OpenSearch     score=...
    id=5  title=OpenSearch Internals     score=...

You just built, committed, merged, and queried a Lucene index — a shard, by hand.

Step 5: Map Lucene Primitives to OpenSearch

Now connect what you wrote to where OpenSearch wraps it. Verify each mapping with a grep in your OpenSearch checkout — do not take the table on faith.

Read the class-level Javadoc of InternalEngine with this table in hand — it will now read as "a managed, recoverable, refreshable wrapper around exactly the IndexWriter/DirectoryReader I just used." That is the engine. Detail lives in the engine internals deep dive and the IndexShard lifecycle deep dive.

Step 6: Reflect on Segments and Merges

Your program printed the segment count rising from 1 → 2 → 1. Re-read what happened and connect it to OpenSearch:

• Each commit() flushed buffered documents into a new immutable segment. Segments are never edited in place — updates/deletes are recorded as new segments + tombstones.
• forceMerge(1) rewrote all live documents into one segment, reclaiming space from deletes. OpenSearch's background MergePolicy/MergeScheduler does this continuously, on a schedule, so the number of segments stays bounded without you asking.
• A refresh in OpenSearch == opening a new DirectoryReader == making newly committed docs visible to search. A flush == a Lucene commit + trimming the translog == the durability point. This is precisely the lifecycle in the refresh/flush/merge deep dive.

This is the same behavior you watched over REST in Lab 1.3, Step 7 via _cat/segments and _forcemerge — except now you have seen it from inside, with nothing distributed in the way.

Implementation Requirements

Deliverables:

• MiniShard.java compiles against the same Lucene version OpenSearch bundles and runs.
• Program output shows the segment count going 1 → 2 → 1 across the two commits and the force-merge.
• All three queries (two TermQuery, one BooleanQuery) return the expected hit counts (2, 4, 2).
• The Lucene→OpenSearch mapping table above, with each row's grep run and the file/line you found pasted in.
• A written reflection (4–6 sentences) answering: why is "a shard is a Lucene index" the right mental model, and what does OpenSearch add on top of Lucene that this program does not?

Troubleshooting

error: cannot find symbol on storedFields() or StandardAnalyzer

Your Lucene version differs from what these calls assume. storedFields() is Lucene 9; older lines use searcher.doc(id). The analyzer is in org.apache.lucene.analysis.standard in Lucene 9; older lines used org.apache.lucene.analysis.standard differently or core. Match your imports to your JARs — run jar tf lib/lucene-core-*.jar | grep -i StandardAnalyzer if unsure where a class lives.

NoClassDefFoundError at runtime

Your runtime classpath is missing a JAR. The analyzer needs lucene-analysis-common, the query parsing helpers need lucene-queryparser. Confirm all three JARs are on -cp: java -cp "out:lib/*" MiniShard (the lib/* glob must be quoted so the shell does not expand it).

TermQuery title:opensearch returns 0 hits

StandardAnalyzer lowercases. The indexed term is opensearch, not OpenSearch. A TermQuery does not analyze its input — it matches the raw term. Query the lowercased form (as the code does). This is exactly why OpenSearch distinguishes text (analyzed) from keyword (not), which you saw in Lab 1.3.

Different scores than shown

BM25 scores depend on document/term statistics and Lucene version. Hit counts are what you validate; exact scores will differ and that is fine.

Stretch Goals

1. Add a numeric range query. You stored year as an IntPoint. Add an IntPoint.newRangeQuery("year", 2015, 2025) and print the hits. This is the Lucene primitive behind OpenSearch's range query on numeric fields.

2. Delete and observe. Open a fresh IndexWriter on the same directory, call writer.deleteDocuments(new Term("id", "3")), commit, and re-check numDocs() vs reader.maxDoc(). The gap is "deleted but not yet merged away" — exactly docs.deleted in _cat/segments.

3. Switch to an on-disk Directory. Replace ByteBuffersDirectory with FSDirectory.open(Path.of("./idx")), run the program, and inspect the files it writes (ls -la idx/). You will see .cfs/.si/.cfe segment files — the literal bytes an OpenSearch shard stores on disk.

4. Use the QueryParser. Pull in lucene-queryparser and parse "title:opensearch AND tag:search" with QueryParser instead of building the BooleanQuery by hand. The OpenSearch query_string query is conceptually this parser, wired into the DSL.

Validation / Self-check

You are done when you can answer these without notes:

1. In one sentence: what is an OpenSearch shard, in Lucene terms?
2. Which OpenSearch class holds the Lucene IndexWriter, and which method drives addDocument?
3. What does an OpenSearch refresh correspond to in your program, and what does a flush correspond to?
4. Why does a TermQuery for OpenSearch (capital O) miss, but match over a text field for "OpenSearch" hits? What component is responsible for the difference?
5. What is a segment, why is it immutable, and what does a merge accomplish?
6. Name three things OpenSearch adds on top of the bare Lucene primitives in this program (hint: durability across crashes, distribution across nodes, an API).

You have now seen OpenSearch from the REST surface (Lab 1.3) down to the Lucene floor (this lab). Proceed to Level 2 — OpenSearch Contributor Onboarding to learn how to turn understanding into merged pull requests.