Lab L4: Contribute to Apache Lucene

Background

The previous three labs cracked open a Lucene index (L1), wrote a custom codec (L2), and built an HNSW graph from scratch (L3). You now know Lucene from the inside. This lab takes the obvious next step: contributing a fix upstream, to apache/lucene itself — the Apache Software Foundation project that OpenSearch and the k-NN lucene engine are built on.

This is a different game from contributing to OpenSearch. Different repo, different governance (Apache, not a single-company-led project), a CHANGES.txt instead of a changelog bot, a developer mailing list that still matters, and a release cadence you do not control. But the engineering muscle is the same one PR quality trains: a small, well-scoped, well-tested change, explained well, that a maintainer can say yes to quickly.

Why an OpenSearch contributor should care. OpenSearch bundles a specific Lucene version. A fix you land in apache/lucene does not help OpenSearch the day it merges — it flows in on the next Lucene upgrade inside OpenSearch (a recurring, high-skill core task; you can watch it happen by searching the OpenSearch repo for Upgrade to Lucene). More striking: a number of the vector and quantization features that OpenSearch's k-NN lucene engine relies on landed in Lucene first — int8 scalar quantization, the scalar-quantized HNSW formats, faster graph merging. If you want to shape vector search at its root, the root is Lucene. This lab is how you reach it.

Note: This is a real contribution lab. You will clone Lucene, build it, run its tests with a reproducible seed, and walk the exact shape of a PR — CHANGES.txt entry, the JIRA/GitHub history, the mailing list. The walked example (a small VectorUtil/vector-format improvement) is illustrative — a realistic shape, not a claim that one specific line exists. You will grep to find the real site before you touch anything.

What you will do

• Clone apache/lucene and build it with Gradle (JDK 21).
• Run the full check, a single module's tests, and reproduce a test with a fixed seed.
• Run tidy so your change is format-clean before you ever push.
• Find a real, small improvement site in VectorUtil / a vector format by grep.
• Stage the change as an illustrative diff with a test.
• Write the CHANGES.txt entry and the PR the Apache way.
• Trace how that fix would reach OpenSearch on the next Lucene upgrade.

Step 1 — Clone and build

Lucene moved from Ant to Gradle; it targets JDK 21 (the same JDK as OpenSearch and k-NN, which is not a coincidence — they move together). Build it once cold so the toolchain is warm before you care about a diff.

mkdir -p ~/src/oss-repos && cd ~/src/oss-repos
git clone https://github.com/apache/lucene.git
cd lucene
git log --oneline -1            # note the HEAD you started from
java -version                   # must be 21.x; Lucene's Gradle will complain otherwise

# Cold build + the full validation suite (slow the first time — go get coffee).
./gradlew check

./gradlew check is the gate: it compiles, runs tests, and runs the static/format checks. It is the "is my tree even sane" command, and you run it before and after your change.

Warning: the first ./gradlew check downloads a toolchain and a lot of dependencies and runs the entire test suite — it can take a long while and a lot of RAM. For day-to-day iteration you do not run the whole thing; you run one module (next step). Run the full check before you push.

# Sanity-inspect an index visually with Luke (Lucene's index inspector) — the GUI cousin of Lab L1.
./gradlew :lucene:luke:run

Step 2 — The iteration loop: module tests, seeds, tidy

You will spend almost all your time in three commands. Learn them cold.

Run one module's tests. Vector code lives in lucene-core, so:

./gradlew :lucene:core:test

This is your inner loop — seconds-to-minutes, not the whole suite.

Run one test class or method. Far tighter:

./gradlew :lucene:core:test --tests "org.apache.lucene.util.TestVectorUtil"
./gradlew :lucene:core:test --tests "org.apache.lucene.util.TestVectorUtil.testDotProduct*"

Reproduce a failure with a fixed seed. Lucene's tests are randomized (the RandomizedTesting framework). A failure prints a seed; you reproduce it exactly by passing it back. This is the single most important Lucene-testing skill, because "it passed on my machine" means nothing when the seed differs.

# A failing run prints something like:  -Ptests.seed=DEADBEEFCAFE
./gradlew :lucene:core:test --tests "org.apache.lucene.util.TestVectorUtil" -Ptests.seed=DEADBEEFCAFE

# Hammer a flaky test across many random seeds to flush out rare failures:
./gradlew :lucene:core:test --tests "org.apache.lucene.util.TestVectorUtil" -Ptests.iters=50

Make it format-clean. Lucene enforces formatting; an unformatted diff fails check. Run tidy before you push so the formatter, not the reviewer, fixes your whitespace:

./gradlew tidy
git diff            # tidy may have reformatted — review what it touched

Note: -Ptests.seed=... is also how a reviewer reproduces the failure your test catches. When you report a bug, include the seed. When you fix one, your new test should fail on the old code and pass on the new — independent of seed, ideally, or with the triggering seed pinned.

Step 3 — Find a real improvement site (grep, do not guess)

The walked example below is a small numeric/vector-format improvement — the kind of contained change that is realistic for a first upstream PR and squarely in the vector territory this curriculum cares about. Find the real site yourself; do not trust a line number printed in a lab.

# The SIMD-backed vector kernels — dot product, squared distance, cosine.
# This is the same VectorUtil / VectorizationProvider machinery from the SIMD chapter
# (../../lucene/simd-and-the-vector-api.md).
grep -rn "class VectorUtil\|dotProduct\|squareDistance\|cosine\|VectorUtilSupport" \
  lucene/core/src/java/org/apache/lucene/util/ | head -20

# The vector formats — HNSW + scalar quantization (the features that landed in Lucene first).
grep -rln "HnswVectorsFormat\|ScalarQuantizedVectorsFormat\|ScalarQuantizer" \
  lucene/core/src/java/org/apache/lucene/codecs/ | head

# The scalar quantizer itself — bounds, error handling, edge cases are fertile ground.
grep -rn "class ScalarQuantizer\|confidenceInterval\|quantize\|MINIMUM_MAGNITUDE" \
  lucene/core/src/java/org/apache/lucene/util/quantization/ 2>/dev/null | head

Good first contributions in this area tend to be one of:

• A missing bounds/precondition check (a method that NaNs or silently misbehaves on a zero-length or mismatched-dimension vector).
• A test gap (an edge case the existing tests do not cover — the easiest, most welcome first PR).
• A small clarity/correctness fix in an error message or an off-by-one in a fallback path.

Warning: do not open your first Lucene PR as a sweeping refactor of VectorUtil or a new SIMD kernel. Those touch performance-critical, heavily-benchmarked code and invite long review. Start with a test gap or a precondition. Earn the context first — exactly the PR quality discipline, applied to a project where you have less standing than you do in OpenSearch.

Step 4 — The illustrative change

Suppose your grep found that a VectorUtil distance helper does not reject mismatched-dimension inputs early — it would read past one array or produce a meaningless result instead of failing loudly. (Confirm against your checkout; the real method name and whether the check already exists vary by version.) A defensive precondition plus a test is a clean, contained contribution.

// ILLUSTRATIVE shape — grep to find the real method/signature in YOUR checkout first.
// In org.apache.lucene.util.VectorUtil (or its support class):

public static float dotProduct(float[] a, float[] b) {
  if (a.length != b.length) {
    throw new IllegalArgumentException(
        "vector dimensions differ: " + a.length + " != " + b.length);
  }
  return IMPL.dotProduct(a, b);
}

The test is the part that actually carries the PR. It must fail on the old code and pass on the new, and it should respect Lucene's randomized-testing conventions (extend LuceneTestCase, use the framework's RNG via random()).

// ILLUSTRATIVE — in TestVectorUtil (org.apache.lucene.util):
public void testDotProductRejectsMismatchedDimensions() {
  float[] a = new float[atLeast(1)];
  float[] b = new float[a.length + 1];          // deliberately different dimension
  expectThrows(IllegalArgumentException.class, () -> VectorUtil.dotProduct(a, b));
}

# Run just your new test, then the whole VectorUtil suite, then tidy.
./gradlew :lucene:core:test --tests "*TestVectorUtil.testDotProductRejectsMismatchedDimensions"
./gradlew :lucene:core:test --tests "*TestVectorUtil"
./gradlew tidy

Note: if the precondition turns out to already exist, that is a successful investigation, not a failure — you learned the code is more careful than you guessed. Pivot to a genuine gap your grep surfaced (a format edge case, an untested space type, a quantization bound). The skill being trained is finding the real, small, defensible change, not forcing a predetermined one.

Step 5 — CHANGES.txt and the contribution conventions

This is where Lucene differs visibly from OpenSearch. Three artifacts matter.

1. CHANGES.txt. Lucene tracks user-facing changes in a hand-edited CHANGES.txt at the repo root, grouped by version and by category (New Features, Improvements, Bug Fixes, Optimizations). You add a one-line entry under the unreleased version, in the right category, crediting yourself.

head -60 CHANGES.txt          # see the current unreleased section + the format
grep -n "Bug Fixes\|Improvements\|Optimizations\|API Changes" CHANGES.txt | head

* GITHUB#NNNNN: VectorUtil.dotProduct now rejects vectors of mismatched
  dimension with a clear IllegalArgumentException.  (Your Name)

2. The issue/PR history — JIRA then GitHub. Lucene's history lives partly in Apache JIRA as LUCENE-NNNN issues — that is why you will see LUCENE-10577 (the origin of int8 scalar quantization) and LUCENE-NNNN references throughout the code and CHANGES.txt. The ASF then moved active development to GitHub Pull Requests (you will see apache/lucene #11613, #12497 and similar). Today you open a GitHub PR; older entries reference the JIRA key. Cross-reference both when you cite prior art.

# How the codebase itself cites its history — JIRA keys and GitHub PR numbers side by side.
grep -rn "LUCENE-[0-9]\+\|GITHUB#[0-9]\+" CHANGES.txt | head -20

3. The mailing list. Unlike a single-vendor project, an Apache project's center of gravity includes the dev@lucene.apache.org developer list. For anything non-trivial — a design question, "would a PR for X be welcome?", a release timing question — the list is the right venue, and a short, polite message there before a big PR is the Apache-native version of design via GitHub. Subscribe before you need it.

Step 6 — Open the PR the Apache way

# Fork apache/lucene on GitHub, then:
git checkout -b vectorutil-dim-check
git add -p                                  # stage the precise hunks, nothing stray
git commit -m "Reject mismatched-dimension vectors in VectorUtil.dotProduct"
git push -u origin vectorutil-dim-check
gh pr create --repo apache/lucene --fill    # or open the PR in the browser

A reviewable Lucene PR has:

• A focused diff — one concern. (A dimension check; not a check plus a refactor plus a rename.)
• A test that fails before and passes after.
• A clean ./gradlew check (run the full one before you push) and a ./gradlew tidy-clean tree.
• A CHANGES.txt entry in the right category.
• A PR description that says what and why, links any LUCENE-NNNN/GitHub prior art, and notes the seed if a randomized test was involved.

This is precisely the PR quality checklist — the change is the easy part; the reviewability is the contribution. In a project where you have no standing yet, it matters even more.

Step 7 — How this reaches OpenSearch

Close the loop. Your Lucene fix is not in OpenSearch the moment it merges. The path is:

flowchart LR
    PR["your PR merges in apache/lucene"] --> REL["it ships in a Lucene release<br/>(e.g. 10.x.y)"]
    REL --> UP["OpenSearch bumps its bundled Lucene<br/>(the 'Upgrade to Lucene N' PR)"]
    UP --> OS["the fix is now in OpenSearch<br/>(and the k-NN lucene engine inherits it)"]

# See the upgrade task happen, repeatedly, in the OpenSearch repo.
gh search prs --repo opensearch-project/OpenSearch "Upgrade to Lucene" --limit 20
# Find which Lucene version a given OpenSearch checkout bundles.
grep -rn "lucene" ~/src/OpenSearch/buildSrc/version.properties 2>/dev/null \
  || grep -rn "lucene =" ~/src/OpenSearch/gradle/libs.versions.toml 2>/dev/null

This is why the work compounds. A vector or quantization improvement you land in Lucene — exactly the kind of feature that has historically landed in Lucene first (int8 SQ via LUCENE-10577 / apache/lucene #11613; the scalar-quantized vectors format via apache/lucene #12497) — eventually becomes a capability of OpenSearch's k-NN lucene engine, with zero additional OpenSearch code beyond the routine version bump. You contributed to one project and improved two.

Note: the inverse is the upgrade contributor's job: when OpenSearch bumps Lucene, something usually breaks (an API moved, a default changed, a format version bumped). Fixing those breakages is one of the highest-leverage recurring tasks in the OpenSearch repo, and it requires exactly the Lucene fluency these four labs built. The "Upgrade to Lucene" PRs are a standing source of real work.

Validation / Self-check

You are done when you can answer these without notes. Several are runnable; run them.

1. Build Lucene from a clean clone and run ./gradlew check. What does check actually gate (compile, tests, format)? Why do you run a single module's tests during iteration instead?
2. A test fails and prints a seed. Reproduce the failure exactly with one command. Why is the seed essential, and how do you hammer a flaky test across many seeds?
3. Run ./gradlew tidy. What did it change, and why must you run it before you push rather than letting the reviewer find the formatting?
4. By grep, name one real site in VectorUtil or a vector/quantization format where a small precondition, test, or clarity fix would be a defensible first PR — and say how you'd write a test that fails on the old code and passes on the new.
5. Where does a user-facing Lucene change get recorded, in which categories, and what is the difference between a LUCENE-NNNN reference and an apache/lucene #NNNNN reference?
6. What is dev@lucene.apache.org for, and when would you post there before opening a PR? Tie it to the OpenSearch design via GitHub habit.
7. Draw the path from "my Lucene PR merges" to "the fix is in OpenSearch." Name the intermediate release and the recurring OpenSearch PR, and run the gh search that proves the upgrade task is real. Why do some vector features reach you through Lucene first?

When you can do all seven, you have closed the loop these four Lucene labs opened: you can crack an index, write a codec, build a graph — and now push a fix into the library all of it rests on. Carry that fluency back to the HNSW chapter, the SIMD chapter, and every "Upgrade to Lucene" PR you will ever review.