Lab 2.1: Navigate the OpenSearch Repository Structure

Background

The OpenSearch repository is large — hundreds of Gradle projects, tens of thousands of Java files. A contributor who cannot navigate it quickly wastes hours and, worse, edits the wrong layer. This lab is a guided tour: you will use ./gradlew projects, find, and grep to build a durable map of the codebase, learn to locate any class in seconds, read a build.gradle to understand inter-project dependencies, and recognize the SPDX header convention that every source file carries (and that precommit enforces).

This is a reading and orientation lab. You will not change any code — but the muscle memory you build here is what makes every later lab fast.

Why This Lab Matters for Contributors

• "Where does this live?" is the question you answer dozens of times per PR. Answering it in seconds instead of minutes compounds.
• Editing the right layer matters: a fix in :server vs :libs:core vs a :modules:* has different review owners, BWC implications, and test scopes.
• Reading build.gradle dependency blocks tells you what you are allowed to call from where — and why some "obvious" imports are forbidden.
• The SPDX header is a hard precommit gate; knowing it up front saves a failed CI run.

Prerequisites

• Lab 1.1 complete; you can run ./gradlew.
• A clone of opensearch-project/OpenSearch (your fork is fine; see Lab 2.2 for fork setup).

Step-by-Step Tasks

Step 1: The Top-Level Map

From the repo root, list directories and the Gradle project tree side by side:

ls -d */ | sort
./gradlew projects 2>/dev/null | sed -n '1,60p'

The directories you must know:

Note: :server is allowed to depend on libs/*, but libs/* must not depend on :server — the dependency arrow only points one way. That is why core serialization primitives live in libs/core: so everything (including :server) can use them. You will see this enforced by the build.gradle files in Step 5.

Step 2: Walk server/

:server is where you will spend most of your time. Get its internal shape:

ls server/src/main/java/org/opensearch | sort

The packages map directly to subsystems you will study:

# How big is the engine, roughly?
find server/src/main/java -name "*.java" | wc -l
# Where is each subsystem rooted?
find server/src/main/java/org/opensearch/index/engine -name "*.java" | head

Step 3: Source Sets — main vs test vs internalClusterTest

Each project has multiple source sets. Know which one a file is in before you edit it:

ls -d server/src/*/
# server/src/main/                -> production code
# server/src/test/                -> unit tests (OpenSearchTestCase, ...)
# server/src/internalClusterTest/ -> in-JVM multi-node integration tests (*IT)

This matters because the Gradle task differs (:server:test vs :server:internalClusterTest, from Lab 1.2) and because production code must never depend on test code.

Step 4: Locate Any Class

The core skill. Three reliable methods, fastest first:

# 1. By file name — when you know the class name:
find server -name "IndexShard.java" -path "*/main/*"
#   server/src/main/java/org/opensearch/index/shard/IndexShard.java

# 2. By declaration — when you are not sure of the file name or want the exact site:
grep -rn "class TransportSearchAction" server/src/main/java | head

# 3. By usage — when you want callers of something:
grep -rn "applyIndexOperationOnPrimary" server/src/main/java | head

Practice until it is reflex. Find each of these and note the path:

for c in Node ClusterService RestController IndicesService IndexShard \
         InternalEngine SearchService StreamInput Writeable; do
  echo "== $c =="
  find server libs -name "$c.java" -path "*/main/*"
done

Note that StreamInput and Writeable resolve under libs/core, not server — the wire primitives live in the shared library, exactly as Step 1 predicted.

Step 5: Read a build.gradle for Dependencies

A project's build.gradle declares what it may depend on. This is how you learn the allowed direction of imports. Read the server's:

sed -n '1,80p' server/build.gradle
grep -nE "api project|implementation project|testImplementation project" server/build.gradle

You will see :server declaring dependencies on :libs:core, :libs:common, :libs:x-content, and the Lucene artifacts — and you will not see any libs/* project depending on :server. Now read a library's to confirm the one-way arrow:

grep -nE "project\(" libs/core/build.gradle
# libs:core depends only on other libs / external jars — never on :server

This dependency structure is why certain code lives where it does. A Writeable in libs/core can be used by :server, every module, and every plugin; if it lived in :server, the libraries below it could not use it. Keep this in mind when a reviewer says "this belongs in libs, not server."

Tip: To see the resolved dependency graph (including transitive Lucene/Netty/etc.): ./gradlew :server:dependencies --configuration compileClasspath | head -60.

Step 6: The SPDX Header Convention

Every OpenSearch source file carries an SPDX license header. New files must include it or precommit fails. Inspect a real one:

head -20 server/src/main/java/org/opensearch/index/shard/IndexShard.java

The canonical header for a new file is:

/*
 * SPDX-License-Identifier: Apache-2.0
 *
 * The OpenSearch Contributors require contributions made to
 * this file be licensed under the Apache-2.0 license or a
 * compatible open source license.
 */

Older files that predate the fork also carry an Apache-2.0 attribution block referencing the original Elasticsearch copyright (the fork preserved upstream attribution). You do not remove those. For any new file you create (e.g. a new test in Lab 2.3), copy the SPDX header from a sibling file in the same directory.

Confirm precommit cares:

grep -rn "licenseHeaders\|forbiddenApis\|spotless" server/build.gradle build-tools*/ buildSrc/ 2>/dev/null | head

Step 7: rest-api-spec/ and qa/ — Know They Exist

Two directories you will not edit yet but must recognize when reading PRs:

ls rest-api-spec/src/main/resources/rest-api-spec/api | head
#   the JSON specs describing every REST endpoint (params, paths, bodies)
ls rest-api-spec/src/yamlRestTest 2>/dev/null
ls qa/ | head
#   bwc-test, rolling-upgrade, mixed-cluster, full-cluster-restart, ...

rest-api-spec is the contract for the REST API and the home of shared YAML REST tests; qa/ holds the cross-version backward-compatibility tests you will write in Level 9. A PR that changes a REST endpoint usually touches rest-api-spec; a PR that changes the wire format usually adds a qa/ BWC test.

Implementation Requirements

This lab produces a map, not code. Deliverables:

1. A filled-in copy of the top-level dir → Gradle project → purpose table, verified against your own ./gradlew projects output.
2. The file path (from memory, then verified) for: Node, IndexShard, InternalEngine, SearchService, StreamInput, Writeable.
3. Two sentences explaining why StreamInput/Writeable live in libs/core and not :server.
4. A note of the :server project dependencies you found in server/build.gradle.
5. The SPDX header pasted from a real file, plus the path you copied it from.

Troubleshooting

./gradlew projects is overwhelming

Pipe it. ./gradlew projects | grep -E "':(server|libs|modules):" narrows to the projects you care about. You rarely need the full list at once.

find returns test and main copies of the same class name

Constrain the path: add -path "*/main/*" for production code or -path "*/test/*" for tests. Many classes have a FooTests.java next to Foo.java.

grep -rn "class X" returns nothing

The class may be nested, generic, or named differently than you assume. Try grep -rn "X" --include=*.java -l to find files mentioning it, or search the declaration loosely: grep -rn "class X\b\|interface X\b\|enum X\b".

A new file fails precommit with a license-header error

You omitted the SPDX header. Copy it verbatim from a sibling file in the same package. Then re-run ./gradlew precommit.

Expected Output

Your locate-the-class drill should produce paths like:

== Node ==           server/src/main/java/org/opensearch/node/Node.java
== IndexShard ==     server/src/main/java/org/opensearch/index/shard/IndexShard.java
== InternalEngine == server/src/main/java/org/opensearch/index/engine/InternalEngine.java
== SearchService ==  server/src/main/java/org/opensearch/search/SearchService.java
== StreamInput ==    libs/core/src/main/java/org/opensearch/core/common/io/stream/StreamInput.java
== Writeable ==      libs/core/src/main/java/org/opensearch/core/common/io/stream/Writeable.java

(Exact libs/core sub-paths vary by branch — the point is they are under libs/core, not server.)

Stretch Goals

1. Map a module's extension points. Open modules/analysis-common/build.gradle and its *Plugin.java; identify which Plugin interfaces it implements (you will study these in Level 3 and the plugin-architecture deep dive).

2. Trace a REST route to its handler. Pick an endpoint from rest-api-spec (say _count), then grep -rn "_count" server/src/main/java/org/opensearch/rest to find the handler that registers it. This is the bridge to Level 3, Lab 3.1.

3. Find every place a setting is defined. Settings are Setting<T> constants. Run grep -rn "Setting.intSetting\|Setting.boolSetting" server/src/main/java/org/opensearch/cluster | head to see how cluster settings are declared.

4. Diff the dependency graph of two projects. Compare ./gradlew :libs:core:dependencies with ./gradlew :server:dependencies and articulate why the library's graph is a strict subset of the server's.

Validation / Self-check

You are done when you can answer these without notes:

1. Which top-level dir holds the core engine, and which holds the shared wire-serialization primitives? Why are they separate?
2. In which direction may dependencies point between :server and libs/*? How do you verify it from a build.gradle?
3. Give the three ways to locate a class, and which you reach for when you know only the behavior, not the name.
4. What are the three source sets of :server, and which Gradle test task runs each?
5. What must every new .java file contain to pass precommit, and where do you copy it from?
6. What is rest-api-spec/ for, and what does qa/ hold?

Next: Lab 2.2 — Prepare a PR Using OpenSearch Practices.