Lab 1.4: Project — Build a Minimal Lucene Index

Background

Every OpenSearch shard is a Lucene index. That sentence is repeated everywhere in this book, but it does not become real until you build a Lucene index with your own hands — no cluster, no REST, no org.opensearch.*, just the library underneath everything. In this build-it project you write a ~120-line standalone Java program that:

1. opens a Directory and an IndexWriter,
2. adds a few Documents,
3. commits, opens a DirectoryReader and an IndexSearcher,
4. runs a TermQuery and a BooleanQuery, and prints the hits,
5. observes segments and a merge.

Then you map each Lucene primitive to the OpenSearch class that wraps it — IndexWriter → InternalEngine, DirectoryReader/IndexSearcher → Engine.Searcher, the whole bundle → IndexShard. After this lab, "a shard is a Lucene index" is something you have seen, not something you have read.

Why This Lab Matters for Contributors

• The single most useful mental model for OpenSearch internals is "OpenSearch is distribution + durability + an API on top of Lucene." This lab installs that model permanently.
• When you trace the index path in Level 6 and the search path in Level 7, every Lucene call you see (IndexWriter.addDocument, IndexSearcher.search) will already be familiar.
• Understanding segments and merges first-hand makes the refresh/flush/merge deep dive and engine internals read like review.

Prerequisites

• Lab 1.1 complete, and you found the exact Lucene version your branch bundles (Stretch Goal 2 of that lab). You will use it here so your standalone program uses the same Lucene OpenSearch runs.
• A JDK on your PATH (17+ is fine to compile/run this tiny program).

# Re-confirm the bundled Lucene version from inside your OpenSearch checkout:
grep -rn "lucene" gradle/libs.versions.toml 2>/dev/null \
  || grep -rn "lucene" buildSrc/version.properties 2>/dev/null \
  || ./gradlew :server:dependencies --configuration compileClasspath | grep -i lucene-core
# Note the version, e.g. 9.11.1 — call it LUCENE_VERSION below.

Step-by-Step Tasks

Step 1: Create the Project Skeleton

Work outside the OpenSearch repo so you do not pollute it:

mkdir -p ~/lucene-shard/src ~/lucene-shard/lib
cd ~/lucene-shard

You need three Lucene JARs at the version OpenSearch bundles: lucene-core, lucene-queryparser, and lucene-analysis-common (the package names changed at Lucene 9; this lab targets the 9.x line OpenSearch 3.x uses).

Step 2: Obtain the Lucene JARs

You already have them — they are in your Gradle cache from building OpenSearch. Copy them so your classpath is self-contained:

# Replace 9.11.1 with YOUR LUCENE_VERSION from Step 0.
LUCENE_VERSION=9.11.1
find ~/.gradle/caches -name "lucene-core-${LUCENE_VERSION}.jar"            -exec cp {} lib/ \;
find ~/.gradle/caches -name "lucene-queryparser-${LUCENE_VERSION}.jar"     -exec cp {} lib/ \;
find ~/.gradle/caches -name "lucene-analysis-common-${LUCENE_VERSION}.jar" -exec cp {} lib/ \;
ls -1 lib/
# lucene-analysis-common-9.11.1.jar
# lucene-core-9.11.1.jar
# lucene-queryparser-9.11.1.jar

Note: If a JAR is not in the cache, run a build that needs it (./gradlew :server:assemble), or download those three artifacts from Maven Central at the matching version. Using the same Lucene OpenSearch ships is the whole point — APIs differ across major Lucene versions.

Step 3: Write the Program

Create src/MiniShard.java. This is the full, runnable source — read it as you type it.

import org.apache.lucene.analysis.standard.StandardAnalyzer;
import org.apache.lucene.document.Document;
import org.apache.lucene.document.Field;
import org.apache.lucene.document.IntPoint;
import org.apache.lucene.document.StringField;
import org.apache.lucene.document.TextField;
import org.apache.lucene.index.DirectoryReader;
import org.apache.lucene.index.IndexWriter;
import org.apache.lucene.index.IndexWriterConfig;
import org.apache.lucene.index.Term;
import org.apache.lucene.search.BooleanClause;
import org.apache.lucene.search.BooleanQuery;
import org.apache.lucene.search.IndexSearcher;
import org.apache.lucene.search.Query;
import org.apache.lucene.search.ScoreDoc;
import org.apache.lucene.search.TermQuery;
import org.apache.lucene.search.TopDocs;
import org.apache.lucene.store.ByteBuffersDirectory;
import org.apache.lucene.store.Directory;

import java.util.List;

/**
 * A minimal Lucene index — the same primitives OpenSearch wraps inside a single shard.
 *
 *   Directory        -> where segments live (we use an in-memory one)
 *   IndexWriter      -> wrapped by OpenSearch InternalEngine
 *   DirectoryReader  -> the point-in-time view a "refresh" opens
 *   IndexSearcher    -> wrapped by OpenSearch Engine.Searcher
 */
public class MiniShard {

    public static void main(String[] args) throws Exception {
        // 1. A "shard" needs a place to put its segments. In OpenSearch this is an
        //    FSDirectory on disk; here we use RAM so the program leaves no files.
        Directory dir = new ByteBuffersDirectory();

        // 2. The analyzer turns text into terms. OpenSearch's "standard" analyzer is this one.
        StandardAnalyzer analyzer = new StandardAnalyzer();
        IndexWriterConfig cfg = new IndexWriterConfig(analyzer);

        // 3. The IndexWriter is what InternalEngine.index(...) ultimately drives.
        try (IndexWriter writer = new IndexWriter(dir, cfg)) {
            writer.addDocument(doc("1", "Lucene in Action",      "search", 2010));
            writer.addDocument(doc("2", "Mastering OpenSearch",  "search", 2023));
            writer.addDocument(doc("3", "Data-Intensive Apps",   "systems", 2017));
            writer.addDocument(doc("4", "The Definitive Guide",  "search", 2015));
            // commit() == a Lucene commit == what OpenSearch "flush" does (durability point).
            writer.commit();
            System.out.println("Indexed 4 documents; segment count after first commit: "
                    + writer.getSegmentCount());

            // Add more in a second batch so a second segment forms, then force a merge.
            writer.addDocument(doc("5", "OpenSearch Internals",  "search", 2024));
            writer.commit();
            System.out.println("Segment count after second commit: " + writer.getSegmentCount());
            writer.forceMerge(1); // what POST /_forcemerge?max_num_segments=1 triggers
            writer.commit();
            System.out.println("Segment count after forceMerge(1): " + writer.getSegmentCount());
        }

        // 4. Open a point-in-time view. This is exactly what an OpenSearch "refresh" does:
        //    it opens a new DirectoryReader so newly-committed docs become searchable.
        try (DirectoryReader reader = DirectoryReader.open(dir)) {
            IndexSearcher searcher = new IndexSearcher(reader);
            System.out.println("Reader sees " + reader.numDocs() + " live docs across "
                    + reader.leaves().size() + " segment(s).");

            // 5a. A TermQuery on the analyzed 'title' field. Note: StandardAnalyzer lowercases,
            //     so we query the lowercased term.
            runQuery(searcher, "TermQuery title:opensearch",
                    new TermQuery(new Term("title", "opensearch")));

            // 5b. A TermQuery on the un-analyzed 'tag' StringField (exact match, like a keyword).
            runQuery(searcher, "TermQuery tag:search",
                    new TermQuery(new Term("tag", "search")));

            // 5c. A BooleanQuery: title contains 'opensearch' AND tag == 'search'.
            BooleanQuery bool = new BooleanQuery.Builder()
                    .add(new TermQuery(new Term("title", "opensearch")), BooleanClause.Occur.MUST)
                    .add(new TermQuery(new Term("tag", "search")),       BooleanClause.Occur.MUST)
                    .build();
            runQuery(searcher, "BooleanQuery (title:opensearch AND tag:search)", bool);
        }
    }

    /** Build a document. 'title' is analyzed (TextField); 'tag' is exact (StringField). */
    private static Document doc(String id, String title, String tag, int year) {
        Document d = new Document();
        d.add(new StringField("id", id, Field.Store.YES));      // exact, stored
        d.add(new TextField("title", title, Field.Store.YES));  // analyzed, stored
        d.add(new StringField("tag", tag, Field.Store.YES));    // exact, stored
        d.add(new IntPoint("year", year));                      // numeric, range-searchable
        return d;
    }

    private static void runQuery(IndexSearcher searcher, String label, Query q) throws Exception {
        TopDocs top = searcher.search(q, 10);
        System.out.printf("%n[%s] -> %d hit(s)%n", label, top.totalHits.value);
        List<ScoreDoc> hits = List.of(top.scoreDocs);
        for (ScoreDoc sd : hits) {
            Document d = searcher.storedFields().document(sd.doc);
            System.out.printf("    id=%s  title=%-24s  score=%.3f%n",
                    d.get("id"), d.get("title"), sd.score);
        }
    }
}

Note: Two Lucene-9 API details that bite people coming from older examples: stored-field access is searcher.storedFields().document(docId) (the old searcher.doc(id) / reader.document(id) are deprecated/removed in 9.x), and analyzer packages moved to org.apache.lucene.analysis.standard. If your bundled Lucene is a different major version, adjust these two call sites — the concepts are identical.

Step 4: Compile and Run

cd ~/lucene-shard
# Compile against the Lucene JARs:
javac -cp "lib/*" -d out src/MiniShard.java

# Run with the same classpath plus your compiled class:
java -cp "out:lib/*" MiniShard

Expected output (scores vary slightly by Lucene version):

Indexed 4 documents; segment count after first commit: 1
Segment count after second commit: 2
Segment count after forceMerge(1): 1
Reader sees 5 live docs across 1 segment(s).

[TermQuery title:opensearch] -> 2 hit(s)
    id=2  title=Mastering OpenSearch     score=...
    id=5  title=OpenSearch Internals     score=...

[TermQuery tag:search] -> 4 hit(s)
    id=1  title=Lucene in Action         score=1.000
    id=2  title=Mastering OpenSearch     score=1.000
    id=4  title=The Definitive Guide     score=1.000
    id=5  title=OpenSearch Internals     score=1.000

[BooleanQuery (title:opensearch AND tag:search)] -> 2 hit(s)
    id=2  title=Mastering OpenSearch     score=...
    id=5  title=OpenSearch Internals     score=...

You just built, committed, merged, and queried a Lucene index — a shard, by hand.

Step 5: Map Lucene Primitives to OpenSearch

Now connect what you wrote to where OpenSearch wraps it. Verify each mapping with a grep in your OpenSearch checkout — do not take the table on faith.

Read the class-level Javadoc of InternalEngine with this table in hand — it will now read as "a managed, recoverable, refreshable wrapper around exactly the IndexWriter/DirectoryReader I just used." That is the engine. Detail lives in the engine internals deep dive and the IndexShard lifecycle deep dive.

Step 6: Reflect on Segments and Merges

Your program printed the segment count rising from 1 → 2 → 1. Re-read what happened and connect it to OpenSearch:

• Each commit() flushed buffered documents into a new immutable segment. Segments are never edited in place — updates/deletes are recorded as new segments + tombstones.
• forceMerge(1) rewrote all live documents into one segment, reclaiming space from deletes. OpenSearch's background MergePolicy/MergeScheduler does this continuously, on a schedule, so the number of segments stays bounded without you asking.
• A refresh in OpenSearch == opening a new DirectoryReader == making newly committed docs visible to search. A flush == a Lucene commit + trimming the translog == the durability point. This is precisely the lifecycle in the refresh/flush/merge deep dive.

This is the same behavior you watched over REST in Lab 1.3, Step 7 via _cat/segments and _forcemerge — except now you have seen it from inside, with nothing distributed in the way.

Implementation Requirements

Deliverables:

• MiniShard.java compiles against the same Lucene version OpenSearch bundles and runs.
• Program output shows the segment count going 1 → 2 → 1 across the two commits and the force-merge.
• All three queries (two TermQuery, one BooleanQuery) return the expected hit counts (2, 4, 2).
• The Lucene→OpenSearch mapping table above, with each row's grep run and the file/line you found pasted in.
• A written reflection (4–6 sentences) answering: why is "a shard is a Lucene index" the right mental model, and what does OpenSearch add on top of Lucene that this program does not?

Troubleshooting

error: cannot find symbol on storedFields() or StandardAnalyzer

Your Lucene version differs from what these calls assume. storedFields() is Lucene 9; older lines use searcher.doc(id). The analyzer is in org.apache.lucene.analysis.standard in Lucene 9; older lines used org.apache.lucene.analysis.standard differently or core. Match your imports to your JARs — run jar tf lib/lucene-core-*.jar | grep -i StandardAnalyzer if unsure where a class lives.

NoClassDefFoundError at runtime

Your runtime classpath is missing a JAR. The analyzer needs lucene-analysis-common, the query parsing helpers need lucene-queryparser. Confirm all three JARs are on -cp: java -cp "out:lib/*" MiniShard (the lib/* glob must be quoted so the shell does not expand it).

TermQuery title:opensearch returns 0 hits

StandardAnalyzer lowercases. The indexed term is opensearch, not OpenSearch. A TermQuery does not analyze its input — it matches the raw term. Query the lowercased form (as the code does). This is exactly why OpenSearch distinguishes text (analyzed) from keyword (not), which you saw in Lab 1.3.

Different scores than shown

BM25 scores depend on document/term statistics and Lucene version. Hit counts are what you validate; exact scores will differ and that is fine.

Stretch Goals

1. Add a numeric range query. You stored year as an IntPoint. Add an IntPoint.newRangeQuery("year", 2015, 2025) and print the hits. This is the Lucene primitive behind OpenSearch's range query on numeric fields.

2. Delete and observe. Open a fresh IndexWriter on the same directory, call writer.deleteDocuments(new Term("id", "3")), commit, and re-check numDocs() vs reader.maxDoc(). The gap is "deleted but not yet merged away" — exactly docs.deleted in _cat/segments.

3. Switch to an on-disk Directory. Replace ByteBuffersDirectory with FSDirectory.open(Path.of("./idx")), run the program, and inspect the files it writes (ls -la idx/). You will see .cfs/.si/.cfe segment files — the literal bytes an OpenSearch shard stores on disk.

4. Use the QueryParser. Pull in lucene-queryparser and parse "title:opensearch AND tag:search" with QueryParser instead of building the BooleanQuery by hand. The OpenSearch query_string query is conceptually this parser, wired into the DSL.

Coding Exercises

This is already a build-it lab; these exercises extend MiniShard into a small Lucene playground and then a JUnit-tested mini-project. Each one is real Lucene code against the same version OpenSearch bundles. Verify Lucene API shapes against your JARs (jar tf lib/lucene-core-*.jar | grep -i <Class>) rather than assuming — the package moves happen across major versions.

1. (warm-up) Add a numeric range query. You stored year as an IntPoint. Add a method that runs IntPoint.newRangeQuery("year", 2015, 2025) through runQuery(...) and prints the hits. Verify: the printed count matches the documents with year in [2015, 2025] (it should be 4 of the 5 — all but the 2010 book). This is the Lucene primitive behind OpenSearch's numeric range query (Stretch Goal 1, now with an asserted expected count).

2. (warm-up) Delete and observe the live/deleted gap. Open a fresh IndexWriter on the same Directory, call writer.deleteDocuments(new Term("id", "3")), commit(), reopen the DirectoryReader, and print both reader.numDocs() (live) and reader.maxDoc() (live + deleted). Verify: maxDoc - numDocs == 1 before a merge, and == 0 after forceMerge(1). That gap is docs.deleted from Lab 1.3's _cat/segments.

3. (core) Switch to an on-disk FSDirectory and inspect the bytes. Replace ByteBuffersDirectory with FSDirectory.open(Path.of("./idx")), run, and ls -la idx/. Write a short note mapping each file extension you see (.cfs, .cfe, .si, segments_N) to its role. Verify: the directory contains real segment files after the run, and the file set shrinks after a forceMerge(1). These are the literal bytes an OpenSearch shard writes via its Store (grep -rn "FSDirectory" server/src/main/java/org/opensearch/index/store/).

4. (core) Use the QueryParser. Pull in lucene-queryparser (already in lib/) and replace the hand-built BooleanQuery with new QueryParser("title", analyzer).parse("title:opensearch AND tag:search"). Verify: the parsed query returns the same 2 hits as the hand-built one. The OpenSearch query_string query is conceptually this parser wired into the DSL — confirm by finding where OpenSearch invokes a query parser (grep -rln "QueryParser\|query_string\|QueryStringQueryBuilder" server/src/main/java | head).

5. (core) Add a stored-fields/highlighting style read. After a search, for each hit, fetch the stored title via searcher.storedFields().document(sd.doc) (Lucene 9 API — older lines use searcher.doc(id)), and print the title alongside the score, sorted by score descending. Verify: the highest-scoring doc prints first and titles are correct. This is the fetch phase in miniature — what FetchPhase does in OpenSearch (Level 7).

6. (advanced) Advanced challenge — a JUnit harness for MiniShard. Turn the playground into a tested mini-project. Add JUnit (download junit-jupiter or use a build tool) and write MiniShardTest with methods that assert: (a) segment count is 1 → 2 → 1 across the two commits and the force-merge; (b) the TermQuery title:opensearch returns exactly the two expected ids; (c) the range query from Exercise 1 returns the expected ids; (d) after the delete from Exercise 2, numDocs() and maxDoc() differ by exactly 1. Refactor MiniShard so the index-building logic is callable from a test (extract a method that returns the Directory or a reader) rather than living only in main. Verify: all four test methods pass (mvn test / gradle test, or javac+java -cp with the JUnit console launcher). You now have a tested Lucene shard — the same discipline OpenSearch applies to InternalEngine (whose tests live at find server -name "InternalEngineTests.java").

Issues to Practice On

This lab is pure Apache Lucene — no org.opensearch.* — so the right tracker is apache/lucene (Lucene uses GitHub issues). When the same concept shows up inside OpenSearch's engine wrappers, the work moves to opensearch-project/OpenSearch; know which layer you are in.

# Lucene's own good-first-issues and bugs (labels move; confirm on the tracker):
gh issue list --repo apache/lucene --label "good first issue" --state open
gh issue list --repo apache/lucene --label "bug" --state open

# List Lucene's labels and pick a relevant area (e.g. core/index, core/search):
gh label list --repo apache/lucene | head -40

# For the OpenSearch *engine wrapper* side (InternalEngine, Store, merges):
gh issue list --repo opensearch-project/OpenSearch --label "good first issue" --state open
gh label list --repo opensearch-project/OpenSearch | grep -iE "storage|engine|index"

Representative issue patterns for this subsystem:

• A Lucene docs/Javadoc or example fix. The friendliest first Lucene contribution — a wrong code sample, an outdated API reference (the searcher.doc() → storedFields() migration is a classic source of stale examples). Reproduce the staleness by compiling the snippet against current Lucene, fix, and verify it compiles/runs. Apache projects use a similar PR + sign-off discipline.
• An OpenSearch engine/merge issue. On the OpenSearch side, issues about merge behavior, segment counts, or Store interaction live in opensearch-project/OpenSearch. Reproduce with ./gradlew run + _cat/segments (Lab 1.3), locate the wrapper with grep -n "forceMerge\|MergePolicy" server/src/main/java/org/opensearch/index/engine/InternalEngine.java, fix, and add a test under InternalEngineTests.

Planted-bug drill. Break a Lucene invariant in MiniShard and let your Exercise-6 tests catch it:

1. Change the tag field from StringField (exact) to TextField (analyzed). Re-run your MiniShardTest. Watch the TermQuery tag:search assertion behavior change (the analyzed value may tokenize differently). Restore it and write down why — this is exactly the keyword vs text distinction OpenSearch exposes, seen at the Lucene floor.
2. Or break ordering: index a document with the analyzer set to new KeywordAnalyzer() for one field and watch the title:opensearch TermQuery miss because the term was never lowercased/tokenized. Restore StandardAnalyzer and add an assertion pinning the expected analyzed term — the same reasoning behind the lab's "TermQuery does not analyze its input" troubleshooting note.

Etiquette: for Apache repos and OpenSearch alike — claim/comment before working an issue, reproduce first, and every PR carries a test plus the project's sign-off requirement (OpenSearch: git commit -s for DCO). OpenSearch workflow: Lab 2.2; norms: community interaction.

Validation / Self-check

You are done when you can answer these without notes:

1. In one sentence: what is an OpenSearch shard, in Lucene terms?
2. Which OpenSearch class holds the Lucene IndexWriter, and which method drives addDocument?
3. What does an OpenSearch refresh correspond to in your program, and what does a flush correspond to?
4. Why does a TermQuery for OpenSearch (capital O) miss, but match over a text field for "OpenSearch" hits? What component is responsible for the difference?
5. What is a segment, why is it immutable, and what does a merge accomplish?
6. Name three things OpenSearch adds on top of the bare Lucene primitives in this program (hint: durability across crashes, distribution across nodes, an API).

You have now seen OpenSearch from the REST surface (Lab 1.3) down to the Lucene floor (this lab). Proceed to Level 2 — OpenSearch Contributor Onboarding to learn how to turn understanding into merged pull requests.