Lab 1.1: Build OpenSearch from Source

Background

OpenSearch is a large, multi-project Gradle build (not Maven). Building from source is the mandatory first step for any contributor: you need the ability to compile, rebuild a single project, run tests against your local changes, and produce a runnable distribution. Unlike many Java projects, OpenSearch bundles its own JDK and provisions Gradle through the wrapper — so "works on my machine" failures from JDK drift are largely designed away, if you let the build manage the toolchain.

This lab walks the full build: clone, the ./gradlew wrapper and bundled JDK, assemble, localDistro, building a single project, IntelliJ import, and the Spotless/precommit machinery you will rely on in Level 2.

Why This Lab Matters for Contributors

• You cannot submit a credible PR without first proving it builds cleanly.
• Knowing which Gradle task touches which project saves hours of needless full builds.
• A clean build baseline is what lets you tell a real regression from a local mistake.
• The same assemble/precommit/spotless tasks are exactly what CI (gradle-check) runs on your PR — running them locally first is the difference between a one-round review and five.

Prerequisites

Verify before starting:

git --version          # 2.x
java -version          # informational only; the build provisions its own JDK
./gradlew --version    # run from the repo root after cloning; should print Gradle 8.x

Note: You do not need a perfectly matching system JDK. OpenSearch uses Gradle toolchains and downloads/uses the JDK it needs (JDK 21 is the baseline for the 3.x/main line; older lines used 11/17). A modern system JDK (17+) on your PATH is enough to launch Gradle itself. If java -version shows something ancient (8), install a current JDK first.

Resource floor:

Step-by-Step Tasks

Step 1: Clone the Repository

git clone https://github.com/opensearch-project/OpenSearch.git
cd OpenSearch

This is the canonical repository — GitHub is the source of truth (there is no JIRA). When you start contributing in Level 2 you will fork this and add your fork as a remote; for now, the upstream clone is what you build.

Confirm the remote and the line you are on:

git remote -v
# origin  https://github.com/opensearch-project/OpenSearch.git (fetch)
# origin  https://github.com/opensearch-project/OpenSearch.git (push)

git branch -r | grep -vi HEAD | sort | head

You will see branches like:

• origin/main — the development trunk (current major line, 3.x).
• origin/2.x — the maintenance line for the 2.x series.
• origin/1.x — legacy.

For contributor work, use main unless you are reproducing an issue specific to a release branch. Fixes generally land on main and are backported to 2.x via the backport 2.x label (you will see this in Lab 2.2).

Step 2: Meet the Wrapper and the Bundled Toolchain

The gradlew script is the Gradle wrapper. It pins the exact Gradle version the build expects, downloading it on first use. Never install Gradle globally and run gradle — always run ./gradlew (or gradlew.bat on Windows).

cat gradle/wrapper/gradle-wrapper.properties | grep distributionUrl
# e.g. distributionUrl=https\://services.gradle.org/distributions/gradle-8.x-all.zip

./gradlew --version

OpenSearch's build configures a Java toolchain: Gradle resolves (and, if necessary, downloads) the JDK the build targets, independent of your JAVA_HOME. You can see the requirement and the build-tool plumbing:

# The baseline build JDK is declared in the build logic / CI; grep for it:
grep -rn "JavaLanguageVersion\|languageVersion" build.gradle buildSrc/ build-tools*/ 2>/dev/null | head
# Where the build looks for / provisions JDKs:
./gradlew javaToolchains | head -40

Note: If your network blocks the automatic toolchain download, point Gradle at a locally installed JDK of the right major version with -Porg.gradle.java.installations.paths=/path/to/jdk21 (or set org.gradle.java.installations.paths in ~/.gradle/gradle.properties). The DEVELOPER_GUIDE.md section on "JDK" documents the current accepted versions for your branch.

Step 3: List the Projects

OpenSearch is a multi-project build. See the project tree before building anything:

./gradlew projects | head -60

You will see the top-level projects that mirror the source layout — :server, :libs:*, :modules:*, :plugins:*, :client:*, :distribution:*, :test:framework, :qa:*, :rest-api-spec. You will explore this map in detail in Lab 2.1. For now, note that :server is the core engine and where you will spend most of your time.

Step 4: Assemble the Build (No Tests)

assemble compiles and packages all build artifacts without running tests. This is your "does it build?" command.

./gradlew assemble

Expected duration: 10–30 minutes on the first run (downloading the toolchain JDK and all dependencies dominates), then minutes on warm, incremental builds.

What you should not do at this stage is run ./gradlew build or ./gradlew check — those run the full test suite and the precommit gate across every project and can take well over an hour. Save those for pre-PR (Lab 1.2).

On success you will see:

BUILD SUCCESSFUL in 14m 32s
1234 actionable tasks: 1234 executed

If you see BUILD FAILED, go to Troubleshooting below and re-run with --stacktrace.

Step 5: Build a Single Project

In day-to-day development you almost never build everything. Scope to the project you are editing:

# Compile only the server main sources (the most common inner loop):
./gradlew :server:compileJava

# Compile the server test sources too:
./gradlew :server:compileTestJava

# Assemble a single library or module:
./gradlew :libs:core:assemble
./gradlew :modules:analysis-common:assemble

The :project:task syntax (Gradle "path") is how you address any project. Gradle automatically builds upstream dependencies first (e.g. :server depends on :libs:core, so compiling :server recompiles :libs:core if it changed). This incremental, scoped build is the command you will run hundreds of times.

Tip: --offline reuses the dependency cache and skips network checks — much faster once your first build has populated ~/.gradle. --console=plain gives clean, scrollback-friendly output.

Step 6: Produce a Runnable Local Distribution

assemble builds JARs; localDistro assembles a full, runnable OpenSearch distribution under distribution/archives/ — the same shape an end user downloads.

./gradlew localDistro

# Find what was produced (the exact path/architecture suffix varies by platform):
find distribution/archives -maxdepth 3 -type d -name "*local*" 2>/dev/null
ls distribution/archives/*/build/install/ 2>/dev/null

You generally will not run OpenSearch from this distribution during development — that is what the dedicated ./gradlew run task is for (it launches a debuggable single node from source; see Lab 1.3). localDistro matters because it is what packaging and QA build, and because being able to produce it proves your build is end-to-end healthy.

Step 7: Import into IntelliJ IDEA

IntelliJ understands Gradle multi-project builds natively. Import the build, not the files.

1. File → Open → select the OpenSearch/ root directory (the one containing settings.gradle).
2. When prompted, choose "Open as Project" and let IntelliJ import it as a Gradle project. It reads settings.gradle / build.gradle and materializes every project as a module.
3. Set the Gradle JVM to a JDK of the build's target major version (Settings → Build, Execution, Deployment → Build Tools → Gradle → Gradle JVM). Matching the build toolchain avoids resolution surprises.
4. Wait for the initial sync and index build (several minutes on first import).

Verify the import worked:

• Open server/src/main/java/org/opensearch/index/shard/IndexShard.java.
• Cmd/Ctrl+Click a class reference (e.g. Engine) — it should navigate.
• Find Class (Cmd+O / Ctrl+N) → InternalEngine should resolve to server/src/main/java/org/opensearch/index/engine/InternalEngine.java.

Note: OpenSearch enforces formatting with Spotless and a host of checks under precommit. Rather than fighting IntelliJ's default formatter, rely on ./gradlew spotlessApply (Step 8) to normalize formatting before you commit. The repo includes IDE config the DEVELOPER_GUIDE.md points to; import it if offered.

Step 8: Meet Spotless and Precommit (Awareness)

You will run these constantly from Level 2 onward; meet them now so they are not a surprise.

# Auto-format your changes to satisfy the formatter:
./gradlew spotlessApply

# Verify formatting without changing files (what CI checks):
./gradlew spotlessJavaCheck

# The full static-analysis gate: checkstyle, forbidden APIs, license/SPDX headers,
# dependency checks, logger-usage, etc. Run this before every PR.
./gradlew precommit

precommit is the local mirror of much of what CI's gradle-check enforces. A clean precommit locally is the single biggest predictor of a smooth review. Every new .java file must carry the SPDX header (you will see it flagged here if you forget; details in Lab 2.1).

Implementation Requirements

This lab has no code to implement. Deliverables:

1. A successful ./gradlew assemble run (terminal output showing BUILD SUCCESSFUL).
2. The Gradle version printed by ./gradlew --version and the build's target JDK major version.
3. A successful ./gradlew :server:compileJava (proving the scoped inner loop works).
4. A produced local distribution from ./gradlew localDistro (path identified).
5. A working IntelliJ Gradle import that resolves InternalEngine via Find Class.
6. A clean ./gradlew precommit (or a clear note of any pre-existing failures on your branch).

Troubleshooting

BUILD FAILED with no obvious cause

Re-run with diagnostics:

./gradlew assemble --stacktrace --info 2>&1 | tail -80

Read the first failing task, not the last line. Gradle prints > Task :some:project:task FAILED near the actual error; everything after is fallout.

JDK / toolchain mismatch

> Could not determine the dependencies of task ...
  No compatible toolchains found for request specification: {languageVersion=21 ...}

Cause: Gradle cannot find or download the JDK the build targets. Fix: Either allow the automatic toolchain download (ensure network access), or point Gradle at a locally installed JDK of the right major version:

# One-off:
./gradlew assemble -Porg.gradle.java.installations.paths=/Library/Java/JavaVirtualMachines/jdk-21.jdk/Contents/Home

# Or persistently in ~/.gradle/gradle.properties:
#   org.gradle.java.installations.paths=/path/to/jdk21
./gradlew javaToolchains   # confirm the JDK is now detected

Gradle daemon out of memory / Killed

> Java heap space
# or the JVM is OOM-killed mid-build

Cause: The Gradle daemon's heap is too small for a full build on your machine. Fix: Raise org.gradle.jvmargs. Create or edit gradle.properties in the repo root (or ~/.gradle/gradle.properties):

org.gradle.jvmargs=-Xmx4g -XX:+HeapDumpOnOutOfMemoryError
org.gradle.daemon=true

Then restart the daemon so it picks up the new args:

./gradlew --stop
./gradlew assemble

First build is extremely slow or stalls on downloads

Cause: Cold caches; the first run downloads Gradle, the toolchain JDK, and all dependencies. Fix: This is expected once. After the first successful build, use --offline to skip network checks. If you are behind a proxy, configure systemProp.http(s).proxyHost/Port in ~/.gradle/gradle.properties.

"Spotless found violations" when you only changed one file

Cause: Formatting drift. Fix: ./gradlew spotlessApply, then re-stage. Do not hand-format to match — let the tool do it.

Configuration cache or stale-daemon weirdness

If a build behaves inconsistently after you switched branches or changed gradle.properties:

./gradlew --stop                 # kill stale daemons
./gradlew assemble --rerun-tasks # ignore the up-to-date cache for this run

Expected Output

A clean assemble ends like this (numbers vary):

> Task :distribution:archives:integ-test-zip:buildExpanded
> Task :distribution:archives:darwin-tar:assemble

BUILD SUCCESSFUL in 16m 04s
2871 actionable tasks: 2871 executed

A clean precommit ends like this:

> Task :server:precommit
> Task :precommit

BUILD SUCCESSFUL in 6m 41s

Stretch Goals

1. Inspect the dependency graph of :server. See what :server actually depends on:

   ./gradlew :server:dependencies --configuration compileClasspath | head -60
   

   Confirm :libs:core (and the Lucene artifacts) appear.

2. Find the bundled Lucene version. You will need this exact version in Lab 1.4:

   grep -rn "lucene" buildSrc/version.properties 2>/dev/null \
     || grep -rn "lucene" gradle/libs.versions.toml 2>/dev/null \
     || ./gradlew :server:dependencies --configuration compileClasspath | grep -i lucene-core
   

3. Time a no-op incremental build. Run ./gradlew :server:compileJava twice in a row; the second run should report UP-TO-DATE for the compile task — proof that Gradle's incremental build is working.

4. Build with the build scan (optional): add --scan to any task to get a shareable web report of exactly what ran and how long it took. Useful when asking for help on a slow build.

Validation / Self-check

You are done when you can answer these without notes:

1. Why do you run ./gradlew rather than a globally installed gradle?
2. What does the Gradle toolchain mechanism do for you, and how do you point it at a local JDK?
3. What is the difference between ./gradlew assemble, ./gradlew localDistro, and ./gradlew run (the last one you will use in Lab 1.3)?
4. Which Gradle task addresses only the server's main Java compilation, and how do you express it?
5. Which property controls the Gradle daemon's heap, and in which files can you set it?
6. Which two tasks (spotless*, precommit) must be green before you open a PR, and what does each one check?
7. What exact Apache Lucene version does your branch bundle? (You will reuse this in Lab 1.4.)

Next: Lab 1.2 — Run Unit and Integration Tests.