Lab L4: Contribute to Apache Lucene

Background

The previous three labs cracked open a Lucene index (L1), wrote a custom codec (L2), and built an HNSW graph from scratch (L3). You now know Lucene from the inside. This lab takes the obvious next step: contributing a fix upstream, to apache/lucene itself — the Apache Software Foundation project that OpenSearch and the k-NN lucene engine are built on.

This is a different game from contributing to OpenSearch. Different repo, different governance (Apache, not a single-company-led project), a CHANGES.txt instead of a changelog bot, a developer mailing list that still matters, and a release cadence you do not control. But the engineering muscle is the same one PR quality trains: a small, well-scoped, well-tested change, explained well, that a maintainer can say yes to quickly.

Why an OpenSearch contributor should care. OpenSearch bundles a specific Lucene version. A fix you land in apache/lucene does not help OpenSearch the day it merges — it flows in on the next Lucene upgrade inside OpenSearch (a recurring, high-skill core task; you can watch it happen by searching the OpenSearch repo for Upgrade to Lucene). More striking: a number of the vector and quantization features that OpenSearch's k-NN lucene engine relies on landed in Lucene first — int8 scalar quantization, the scalar-quantized HNSW formats, faster graph merging. If you want to shape vector search at its root, the root is Lucene. This lab is how you reach it.

Note: This is a real contribution lab. You will clone Lucene, build it, run its tests with a reproducible seed, and walk the exact shape of a PR — CHANGES.txt entry, the JIRA/GitHub history, the mailing list. The walked example (a small VectorUtil/vector-format improvement) is illustrative — a realistic shape, not a claim that one specific line exists. You will grep to find the real site before you touch anything.

What you will do

• Clone apache/lucene and build it with Gradle (JDK 21).
• Run the full check, a single module's tests, and reproduce a test with a fixed seed.
• Run tidy so your change is format-clean before you ever push.
• Find a real, small improvement site in VectorUtil / a vector format by grep.
• Stage the change as an illustrative diff with a test.
• Write the CHANGES.txt entry and the PR the Apache way.
• Trace how that fix would reach OpenSearch on the next Lucene upgrade.

Step 1 — Clone and build

Lucene moved from Ant to Gradle; it targets JDK 21 (the same JDK as OpenSearch and k-NN, which is not a coincidence — they move together). Build it once cold so the toolchain is warm before you care about a diff.

mkdir -p ~/src/oss-repos && cd ~/src/oss-repos
git clone https://github.com/apache/lucene.git
cd lucene
git log --oneline -1            # note the HEAD you started from
java -version                   # must be 21.x; Lucene's Gradle will complain otherwise

# Cold build + the full validation suite (slow the first time — go get coffee).
./gradlew check

./gradlew check is the gate: it compiles, runs tests, and runs the static/format checks. It is the "is my tree even sane" command, and you run it before and after your change.

Warning: the first ./gradlew check downloads a toolchain and a lot of dependencies and runs the entire test suite — it can take a long while and a lot of RAM. For day-to-day iteration you do not run the whole thing; you run one module (next step). Run the full check before you push.

# Sanity-inspect an index visually with Luke (Lucene's index inspector) — the GUI cousin of Lab L1.
./gradlew :lucene:luke:run

Step 2 — The iteration loop: module tests, seeds, tidy

You will spend almost all your time in three commands. Learn them cold.

Run one module's tests. Vector code lives in lucene-core, so:

./gradlew :lucene:core:test

This is your inner loop — seconds-to-minutes, not the whole suite.

Run one test class or method. Far tighter:

./gradlew :lucene:core:test --tests "org.apache.lucene.util.TestVectorUtil"
./gradlew :lucene:core:test --tests "org.apache.lucene.util.TestVectorUtil.testDotProduct*"

Reproduce a failure with a fixed seed. Lucene's tests are randomized (the RandomizedTesting framework). A failure prints a seed; you reproduce it exactly by passing it back. This is the single most important Lucene-testing skill, because "it passed on my machine" means nothing when the seed differs.

# A failing run prints something like:  -Ptests.seed=DEADBEEFCAFE
./gradlew :lucene:core:test --tests "org.apache.lucene.util.TestVectorUtil" -Ptests.seed=DEADBEEFCAFE

# Hammer a flaky test across many random seeds to flush out rare failures:
./gradlew :lucene:core:test --tests "org.apache.lucene.util.TestVectorUtil" -Ptests.iters=50

Make it format-clean. Lucene enforces formatting; an unformatted diff fails check. Run tidy before you push so the formatter, not the reviewer, fixes your whitespace:

./gradlew tidy
git diff            # tidy may have reformatted — review what it touched

Note: -Ptests.seed=... is also how a reviewer reproduces the failure your test catches. When you report a bug, include the seed. When you fix one, your new test should fail on the old code and pass on the new — independent of seed, ideally, or with the triggering seed pinned.

Step 3 — Find a real improvement site (grep, do not guess)

The walked example below is a small numeric/vector-format improvement — the kind of contained change that is realistic for a first upstream PR and squarely in the vector territory this curriculum cares about. Find the real site yourself; do not trust a line number printed in a lab.

# The SIMD-backed vector kernels — dot product, squared distance, cosine.
# This is the same VectorUtil / VectorizationProvider machinery from the SIMD chapter
# (../../lucene/simd-and-the-vector-api.md).
grep -rn "class VectorUtil\|dotProduct\|squareDistance\|cosine\|VectorUtilSupport" \
  lucene/core/src/java/org/apache/lucene/util/ | head -20

# The vector formats — HNSW + scalar quantization (the features that landed in Lucene first).
grep -rln "HnswVectorsFormat\|ScalarQuantizedVectorsFormat\|ScalarQuantizer" \
  lucene/core/src/java/org/apache/lucene/codecs/ | head

# The scalar quantizer itself — bounds, error handling, edge cases are fertile ground.
grep -rn "class ScalarQuantizer\|confidenceInterval\|quantize\|MINIMUM_MAGNITUDE" \
  lucene/core/src/java/org/apache/lucene/util/quantization/ 2>/dev/null | head

Good first contributions in this area tend to be one of:

• A missing bounds/precondition check (a method that NaNs or silently misbehaves on a zero-length or mismatched-dimension vector).
• A test gap (an edge case the existing tests do not cover — the easiest, most welcome first PR).
• A small clarity/correctness fix in an error message or an off-by-one in a fallback path.

Warning: do not open your first Lucene PR as a sweeping refactor of VectorUtil or a new SIMD kernel. Those touch performance-critical, heavily-benchmarked code and invite long review. Start with a test gap or a precondition. Earn the context first — exactly the PR quality discipline, applied to a project where you have less standing than you do in OpenSearch.

Step 4 — The illustrative change

Suppose your grep found that a VectorUtil distance helper does not reject mismatched-dimension inputs early — it would read past one array or produce a meaningless result instead of failing loudly. (Confirm against your checkout; the real method name and whether the check already exists vary by version.) A defensive precondition plus a test is a clean, contained contribution.

// ILLUSTRATIVE shape — grep to find the real method/signature in YOUR checkout first.
// In org.apache.lucene.util.VectorUtil (or its support class):

public static float dotProduct(float[] a, float[] b) {
  if (a.length != b.length) {
    throw new IllegalArgumentException(
        "vector dimensions differ: " + a.length + " != " + b.length);
  }
  return IMPL.dotProduct(a, b);
}

The test is the part that actually carries the PR. It must fail on the old code and pass on the new, and it should respect Lucene's randomized-testing conventions (extend LuceneTestCase, use the framework's RNG via random()).

// ILLUSTRATIVE — in TestVectorUtil (org.apache.lucene.util):
public void testDotProductRejectsMismatchedDimensions() {
  float[] a = new float[atLeast(1)];
  float[] b = new float[a.length + 1];          // deliberately different dimension
  expectThrows(IllegalArgumentException.class, () -> VectorUtil.dotProduct(a, b));
}

# Run just your new test, then the whole VectorUtil suite, then tidy.
./gradlew :lucene:core:test --tests "*TestVectorUtil.testDotProductRejectsMismatchedDimensions"
./gradlew :lucene:core:test --tests "*TestVectorUtil"
./gradlew tidy

Note: if the precondition turns out to already exist, that is a successful investigation, not a failure — you learned the code is more careful than you guessed. Pivot to a genuine gap your grep surfaced (a format edge case, an untested space type, a quantization bound). The skill being trained is finding the real, small, defensible change, not forcing a predetermined one.

Step 5 — CHANGES.txt and the contribution conventions

This is where Lucene differs visibly from OpenSearch. Three artifacts matter.

1. CHANGES.txt. Lucene tracks user-facing changes in a hand-edited CHANGES.txt at the repo root, grouped by version and by category (New Features, Improvements, Bug Fixes, Optimizations). You add a one-line entry under the unreleased version, in the right category, crediting yourself.

head -60 CHANGES.txt          # see the current unreleased section + the format
grep -n "Bug Fixes\|Improvements\|Optimizations\|API Changes" CHANGES.txt | head

* GITHUB#NNNNN: VectorUtil.dotProduct now rejects vectors of mismatched
  dimension with a clear IllegalArgumentException.  (Your Name)

2. The issue/PR history — JIRA then GitHub. Lucene's history lives partly in Apache JIRA as LUCENE-NNNN issues — that is why you will see LUCENE-10577 (the origin of int8 scalar quantization) and LUCENE-NNNN references throughout the code and CHANGES.txt. The ASF then moved active development to GitHub Pull Requests (you will see apache/lucene #11613, #12497 and similar). Today you open a GitHub PR; older entries reference the JIRA key. Cross-reference both when you cite prior art.

# How the codebase itself cites its history — JIRA keys and GitHub PR numbers side by side.
grep -rn "LUCENE-[0-9]\+\|GITHUB#[0-9]\+" CHANGES.txt | head -20

3. The mailing list. Unlike a single-vendor project, an Apache project's center of gravity includes the dev@lucene.apache.org developer list. For anything non-trivial — a design question, "would a PR for X be welcome?", a release timing question — the list is the right venue, and a short, polite message there before a big PR is the Apache-native version of design via GitHub. Subscribe before you need it.

Step 6 — Open the PR the Apache way

# Fork apache/lucene on GitHub, then:
git checkout -b vectorutil-dim-check
git add -p                                  # stage the precise hunks, nothing stray
git commit -m "Reject mismatched-dimension vectors in VectorUtil.dotProduct"
git push -u origin vectorutil-dim-check
gh pr create --repo apache/lucene --fill    # or open the PR in the browser

A reviewable Lucene PR has:

• A focused diff — one concern. (A dimension check; not a check plus a refactor plus a rename.)
• A test that fails before and passes after.
• A clean ./gradlew check (run the full one before you push) and a ./gradlew tidy-clean tree.
• A CHANGES.txt entry in the right category.
• A PR description that says what and why, links any LUCENE-NNNN/GitHub prior art, and notes the seed if a randomized test was involved.

This is precisely the PR quality checklist — the change is the easy part; the reviewability is the contribution. In a project where you have no standing yet, it matters even more.

Step 7 — How this reaches OpenSearch

Close the loop. Your Lucene fix is not in OpenSearch the moment it merges. The path is:

flowchart LR
    PR["your PR merges in apache/lucene"] --> REL["it ships in a Lucene release<br/>(e.g. 10.x.y)"]
    REL --> UP["OpenSearch bumps its bundled Lucene<br/>(the 'Upgrade to Lucene N' PR)"]
    UP --> OS["the fix is now in OpenSearch<br/>(and the k-NN lucene engine inherits it)"]

# See the upgrade task happen, repeatedly, in the OpenSearch repo.
gh search prs --repo opensearch-project/OpenSearch "Upgrade to Lucene" --limit 20
# Find which Lucene version a given OpenSearch checkout bundles.
grep -rn "lucene" ~/src/OpenSearch/buildSrc/version.properties 2>/dev/null \
  || grep -rn "lucene =" ~/src/OpenSearch/gradle/libs.versions.toml 2>/dev/null

This is why the work compounds. A vector or quantization improvement you land in Lucene — exactly the kind of feature that has historically landed in Lucene first (int8 SQ via LUCENE-10577 / apache/lucene #11613; the scalar-quantized vectors format via apache/lucene #12497) — eventually becomes a capability of OpenSearch's k-NN lucene engine, with zero additional OpenSearch code beyond the routine version bump. You contributed to one project and improved two.

Note: the inverse is the upgrade contributor's job: when OpenSearch bumps Lucene, something usually breaks (an API moved, a default changed, a format version bumped). Fixing those breakages is one of the highest-leverage recurring tasks in the OpenSearch repo, and it requires exactly the Lucene fluency these four labs built. The "Upgrade to Lucene" PRs are a standing source of real work.

Coding Exercises

This lab's deliverable is code — a real, reviewable change against apache/lucene. Each exercise produces something you run: a Lucene-style randomized test, a reproduced failure, a clean ./gradlew check. Everything uses the ./gradlew build (no DCO, no changelog bot — CHANGES.txt by hand), exactly as Steps 1–6 describe. Do them in your apache/lucene clone.

1. (warm-up) Write a Lucene-style randomized test against existing code. Pick a real VectorUtil method you found by grep in Step 3 (e.g. dotProduct or squareDistance). Add a test to lucene/core/src/test/org/apache/lucene/util/TestVectorUtil.java that uses the framework RNG — random(), atLeast(n), expectThrows(...) — rather than hand-rolled randomness, asserting a property that already holds (e.g. dotProduct(a, a) >= 0 for the magnitudes, or symmetry dotProduct(a,b) == dotProduct(b,a) within an epsilon). Run it ./gradlew :lucene:core:test --tests "*TestVectorUtil.testYourProperty*", then re-run with a pinned -Ptests.seed=... to prove you can reproduce a randomized run exactly.

2. (warm-up) Reproduce a real good-first-issue locally. Run gh issue list --repo apache/lucene --label "good first issue" --state open and pick one whose area you can build a failing case for (a test gap, an edge case, an unclear message). Reproduce it: write the smallest test or main that demonstrates the reported behavior on current main, and capture the output. You are not fixing yet — the deliverable is a reproduction (the thing every Lucene reviewer asks for first). Note the issue number; you'll reference it in CHANGES.txt.

3. (core) Make a test fail on old code, pass on new — the precondition pattern. Take Step 4's illustrative dotProduct dimension-check (or the genuine gap your grep surfaced). First write only the test (testDotProductRejectsMismatchedDimensions style, using expectThrows) and run it — confirm it fails on unmodified code (no precondition yet). Then add the precondition to the real method and confirm the same test passes. This fail-then-pass evidence is what carries the PR; record both runs. If the precondition already exists on your version, pivot per the Step 4 note and pick a genuine gap — the skill is the fail→pass discipline, not the specific line.

4. (core) Run the full gate and read what it checks. Run ./gradlew check on a clean tree and again with your Exercise 3 change. Note the wall-clock difference and identify the three things it gates (compile, tests, static/format checks). Then deliberately break formatting (e.g. mangle the indentation of your new test), run ./gradlew tidy, and git diff to see tidy repair it — proving you can hand the reviewer a format-clean diff instead of making them flag whitespace.

5. (core) Hammer for flakiness across seeds. Run your new test with -Ptests.iters=50 (and try a handful of explicit -Ptests.seed= values). Confirm it passes under every seed — a property test that depends on a lucky seed is not done. If you wrote a randomized assertion in Exercise 1, this is where you discover (and fix) any seed-dependence. Document the command in your PR description, since it is how a reviewer re-runs your evidence.

6. (advanced) Advanced challenge — assemble the full PR and trace it to OpenSearch. Take your Exercise 3 change to completion the Apache way: a focused diff (one concern), the fail→pass test, a CHANGES.txt entry in the correct category (Bug Fixes / Improvements / Optimizations) crediting yourself and citing the issue/PR number, a clean ./gradlew check, and a ./gradlew tidy-clean tree. Stage precise hunks with git add -p, commit (no -s — Lucene uses no DCO), push to your fork, and draft the PR body with gh pr create --repo apache/lucene --fill (you may leave it as a draft if it is illustrative — the deliverable is the assembled, reviewable change, not necessarily a merged one). Then close the loop in code: run gh search prs --repo opensearch-project/OpenSearch "Upgrade to Lucene" --limit 20 and find which Lucene version OpenSearch currently bundles (grep -rn "lucene" .../gradle/libs.versions.toml). Write the two-step path your fix would travel — Lucene release → OpenSearch "Upgrade to Lucene N" PR — and name the release that would first carry it.

Issues to Practice On

This lab targets apache/lucene itself, so the issues you practice on are Lucene's — and the workflow is the Apache one this whole lab teaches: no DCO Signed-off-by (git commit without -s), no changelog bot, build/test/format with ./gradlew, record user-facing changes by hand in CHANGES.txt. Contrast with OpenSearch's DCO-plus-CHANGELOG flow you will use when the fix later reaches OpenSearch on a version bump.

Note: Labels move; confirm with gh label list --repo apache/lucene before relying on one. Lucene uses GitHub issues now but its history is partly in Apache JIRA — you'll see LUCENE-NNNN keys (e.g. LUCENE-10577, int8 scalar quantization) cited alongside apache/lucene #NNNNN PR numbers in code and CHANGES.txt. Cite both when you reference prior art.

Representative issue patterns for a first Lucene contribution:

• A test gap (the most welcome first PR). An edge case — a zero-length vector, a mismatched dimension, an untested space type or quantization bound — that the existing tests don't cover. Approach: reproduce (Exercise 2) → locate the code via rg under lucene/core/src/java/org/apache/lucene/util/ or .../codecs/ → add a randomized test that pins the case → ./gradlew :lucene:core:test --tests "*" → tidy → CHANGES.txt → PR. No DCO sign-off.
• A missing precondition / clarity fix. A method that NaNs or misbehaves on a bad input, or an unclear error message. Small, contained, defensible — Step 4's pattern. Keep the diff to one concern; do not make your first PR a VectorUtil/SIMD refactor (Step 3's warning).

Planted-bug drill. This lab's analogue is a self-inflicted regression in your clone. In a real VectorUtil SIMD method, introduce an off-by-one in a tail/remainder loop or flip a < to <= (grep for the scalar fallback in lucene/core/src/java/.../util/), then run ./gradlew :lucene:core:test --tests "*TestVectorUtil*" and watch which assertion goes red and what seed it prints. Reproduce it with -Ptests.seed=.... Then revert, and add a test asserting the exact boundary you broke (the last element of an odd-length vector contributes correctly) — the test that would have caught it. git checkout the file when done so your tree is clean before the real PR. For etiquette: claim/comment on the issue before working it, reproduce before you fix, and remember that the Apache PR needs a test + CHANGES.txt (no -s), while the same fix arriving in OpenSearch later needs CHANGELOG + DCO. See PR quality and community interaction.

Validation / Self-check

You are done when you can answer these without notes. Several are runnable; run them.

1. Build Lucene from a clean clone and run ./gradlew check. What does check actually gate (compile, tests, format)? Why do you run a single module's tests during iteration instead?
2. A test fails and prints a seed. Reproduce the failure exactly with one command. Why is the seed essential, and how do you hammer a flaky test across many seeds?
3. Run ./gradlew tidy. What did it change, and why must you run it before you push rather than letting the reviewer find the formatting?
4. By grep, name one real site in VectorUtil or a vector/quantization format where a small precondition, test, or clarity fix would be a defensible first PR — and say how you'd write a test that fails on the old code and passes on the new.
5. Where does a user-facing Lucene change get recorded, in which categories, and what is the difference between a LUCENE-NNNN reference and an apache/lucene #NNNNN reference?
6. What is dev@lucene.apache.org for, and when would you post there before opening a PR? Tie it to the OpenSearch design via GitHub habit.
7. Draw the path from "my Lucene PR merges" to "the fix is in OpenSearch." Name the intermediate release and the recurring OpenSearch PR, and run the gh search that proves the upgrade task is real. Why do some vector features reach you through Lucene first?

When you can do all seven, you have closed the loop these four Lucene labs opened: you can crack an index, write a codec, build a graph — and now push a fix into the library all of it rests on. Carry that fluency back to the HNSW chapter, the SIMD chapter, and every "Upgrade to Lucene" PR you will ever review.