Lab 2.1: Navigate the OpenSearch Repository Structure

Background

The OpenSearch repository is large — hundreds of Gradle projects, tens of thousands of Java files. A contributor who cannot navigate it quickly wastes hours and, worse, edits the wrong layer. This lab is a guided tour: you will use ./gradlew projects, find, and grep to build a durable map of the codebase, learn to locate any class in seconds, read a build.gradle to understand inter-project dependencies, and recognize the SPDX header convention that every source file carries (and that precommit enforces).

This is a reading and orientation lab. You will not change any code — but the muscle memory you build here is what makes every later lab fast.

Why This Lab Matters for Contributors

• "Where does this live?" is the question you answer dozens of times per PR. Answering it in seconds instead of minutes compounds.
• Editing the right layer matters: a fix in :server vs :libs:core vs a :modules:* has different review owners, BWC implications, and test scopes.
• Reading build.gradle dependency blocks tells you what you are allowed to call from where — and why some "obvious" imports are forbidden.
• The SPDX header is a hard precommit gate; knowing it up front saves a failed CI run.

Prerequisites

• Lab 1.1 complete; you can run ./gradlew.
• A clone of opensearch-project/OpenSearch (your fork is fine; see Lab 2.2 for fork setup).

Step-by-Step Tasks

Step 1: The Top-Level Map

From the repo root, list directories and the Gradle project tree side by side:

ls -d */ | sort
./gradlew projects 2>/dev/null | sed -n '1,60p'

The directories you must know:

Note: :server is allowed to depend on libs/*, but libs/* must not depend on :server — the dependency arrow only points one way. That is why core serialization primitives live in libs/core: so everything (including :server) can use them. You will see this enforced by the build.gradle files in Step 5.

Step 2: Walk server/

:server is where you will spend most of your time. Get its internal shape:

ls server/src/main/java/org/opensearch | sort

The packages map directly to subsystems you will study:

# How big is the engine, roughly?
find server/src/main/java -name "*.java" | wc -l
# Where is each subsystem rooted?
find server/src/main/java/org/opensearch/index/engine -name "*.java" | head

Step 3: Source Sets — main vs test vs internalClusterTest

Each project has multiple source sets. Know which one a file is in before you edit it:

ls -d server/src/*/
# server/src/main/                -> production code
# server/src/test/                -> unit tests (OpenSearchTestCase, ...)
# server/src/internalClusterTest/ -> in-JVM multi-node integration tests (*IT)

This matters because the Gradle task differs (:server:test vs :server:internalClusterTest, from Lab 1.2) and because production code must never depend on test code.

Step 4: Locate Any Class

The core skill. Three reliable methods, fastest first:

# 1. By file name — when you know the class name:
find server -name "IndexShard.java" -path "*/main/*"
#   server/src/main/java/org/opensearch/index/shard/IndexShard.java

# 2. By declaration — when you are not sure of the file name or want the exact site:
grep -rn "class TransportSearchAction" server/src/main/java | head

# 3. By usage — when you want callers of something:
grep -rn "applyIndexOperationOnPrimary" server/src/main/java | head

Practice until it is reflex. Find each of these and note the path:

for c in Node ClusterService RestController IndicesService IndexShard \
         InternalEngine SearchService StreamInput Writeable; do
  echo "== $c =="
  find server libs -name "$c.java" -path "*/main/*"
done

Note that StreamInput and Writeable resolve under libs/core, not server — the wire primitives live in the shared library, exactly as Step 1 predicted.

Step 5: Read a build.gradle for Dependencies

A project's build.gradle declares what it may depend on. This is how you learn the allowed direction of imports. Read the server's:

sed -n '1,80p' server/build.gradle
grep -nE "api project|implementation project|testImplementation project" server/build.gradle

You will see :server declaring dependencies on :libs:core, :libs:common, :libs:x-content, and the Lucene artifacts — and you will not see any libs/* project depending on :server. Now read a library's to confirm the one-way arrow:

grep -nE "project\(" libs/core/build.gradle
# libs:core depends only on other libs / external jars — never on :server

This dependency structure is why certain code lives where it does. A Writeable in libs/core can be used by :server, every module, and every plugin; if it lived in :server, the libraries below it could not use it. Keep this in mind when a reviewer says "this belongs in libs, not server."

Tip: To see the resolved dependency graph (including transitive Lucene/Netty/etc.): ./gradlew :server:dependencies --configuration compileClasspath | head -60.

Step 6: The SPDX Header Convention

Every OpenSearch source file carries an SPDX license header. New files must include it or precommit fails. Inspect a real one:

head -20 server/src/main/java/org/opensearch/index/shard/IndexShard.java

The canonical header for a new file is:

/*
 * SPDX-License-Identifier: Apache-2.0
 *
 * The OpenSearch Contributors require contributions made to
 * this file be licensed under the Apache-2.0 license or a
 * compatible open source license.
 */

Older files that predate the fork also carry an Apache-2.0 attribution block referencing the original Elasticsearch copyright (the fork preserved upstream attribution). You do not remove those. For any new file you create (e.g. a new test in Lab 2.3), copy the SPDX header from a sibling file in the same directory.

Confirm precommit cares:

grep -rn "licenseHeaders\|forbiddenApis\|spotless" server/build.gradle build-tools*/ buildSrc/ 2>/dev/null | head

Step 7: rest-api-spec/ and qa/ — Know They Exist

Two directories you will not edit yet but must recognize when reading PRs:

ls rest-api-spec/src/main/resources/rest-api-spec/api | head
#   the JSON specs describing every REST endpoint (params, paths, bodies)
ls rest-api-spec/src/yamlRestTest 2>/dev/null
ls qa/ | head
#   bwc-test, rolling-upgrade, mixed-cluster, full-cluster-restart, ...

rest-api-spec is the contract for the REST API and the home of shared YAML REST tests; qa/ holds the cross-version backward-compatibility tests you will write in Level 9. A PR that changes a REST endpoint usually touches rest-api-spec; a PR that changes the wire format usually adds a qa/ BWC test.

Implementation Requirements

This lab produces a map, not code. Deliverables:

1. A filled-in copy of the top-level dir → Gradle project → purpose table, verified against your own ./gradlew projects output.
2. The file path (from memory, then verified) for: Node, IndexShard, InternalEngine, SearchService, StreamInput, Writeable.
3. Two sentences explaining why StreamInput/Writeable live in libs/core and not :server.
4. A note of the :server project dependencies you found in server/build.gradle.
5. The SPDX header pasted from a real file, plus the path you copied it from.

Troubleshooting

./gradlew projects is overwhelming

Pipe it. ./gradlew projects | grep -E "':(server|libs|modules):" narrows to the projects you care about. You rarely need the full list at once.

find returns test and main copies of the same class name

Constrain the path: add -path "*/main/*" for production code or -path "*/test/*" for tests. Many classes have a FooTests.java next to Foo.java.

grep -rn "class X" returns nothing

The class may be nested, generic, or named differently than you assume. Try grep -rn "X" --include=*.java -l to find files mentioning it, or search the declaration loosely: grep -rn "class X\b\|interface X\b\|enum X\b".

A new file fails precommit with a license-header error

You omitted the SPDX header. Copy it verbatim from a sibling file in the same package. Then re-run ./gradlew precommit.

Expected Output

Your locate-the-class drill should produce paths like:

== Node ==           server/src/main/java/org/opensearch/node/Node.java
== IndexShard ==     server/src/main/java/org/opensearch/index/shard/IndexShard.java
== InternalEngine == server/src/main/java/org/opensearch/index/engine/InternalEngine.java
== SearchService ==  server/src/main/java/org/opensearch/search/SearchService.java
== StreamInput ==    libs/core/src/main/java/org/opensearch/core/common/io/stream/StreamInput.java
== Writeable ==      libs/core/src/main/java/org/opensearch/core/common/io/stream/Writeable.java

(Exact libs/core sub-paths vary by branch — the point is they are under libs/core, not server.)

Stretch Goals

1. Map a module's extension points. Open modules/analysis-common/build.gradle and its *Plugin.java; identify which Plugin interfaces it implements (you will study these in Level 3 and the plugin-architecture deep dive).

2. Trace a REST route to its handler. Pick an endpoint from rest-api-spec (say _count), then grep -rn "_count" server/src/main/java/org/opensearch/rest to find the handler that registers it. This is the bridge to Level 3, Lab 3.1.

3. Find every place a setting is defined. Settings are Setting<T> constants. Run grep -rn "Setting.intSetting\|Setting.boolSetting" server/src/main/java/org/opensearch/cluster | head to see how cluster settings are declared.

4. Diff the dependency graph of two projects. Compare ./gradlew :libs:core:dependencies with ./gradlew :server:dependencies and articulate why the library's graph is a strict subset of the server's.

Coding Exercises

A "reading" lab still produces code — here you build the navigation tooling a fluent contributor keeps in their pocket. Each script must run against a real OpenSearch checkout. Use rg/find exclusively to locate things; never bake in a line number.

1. (warm-up) A module-map generator. Write module-map.sh that prints, for each top-level source dir (server libs modules plugins client), the directory name, its Gradle project path, and the count of *.java files under it (find <dir> -name '*.java' | wc -l). Verify: the output reproduces the dir → project → purpose table from Step 1 with real file counts, and the script handles a missing dir gracefully (some branches lack sandbox/).

2. (warm-up) A "find any class" wrapper. Write findclass.sh <ClassName> that tries, in order: (1) find ... -name "<ClassName>.java" -path "*/main/*", then (2) grep -rn "class <ClassName>\b\|interface <ClassName>\b\|enum <ClassName>\b" --include=*.java, printing whichever hits first and labeling which method found it. Verify: findclass.sh IndexShard, findclass.sh StreamInput, and findclass.sh Writeable each return the path the lab's Expected Output shows — and StreamInput/Writeable resolve under libs/core, not server.

3. (core) A dependency-direction checker. Write check-dep-direction.sh that greps every libs/*/build.gradle for a project(":server") dependency and fails (exit 1) if any library declares one — encoding the one-way arrow from Step 5 as an executable rule. Verify: on a clean checkout it exits 0; if you add a fake implementation project(':server') line to libs/core/build.gradle in a scratch branch, it exits 1 and names the offending file.

4. (core) A settings-inventory tool. Settings are Setting<T> constants. Write a script that rgs Setting\.(int|bool|long|byteSize|time|simpleString)Setting across server/src/main/java/org/opensearch/cluster and prints a sorted, de-duplicated list of the setting keys (the first string argument). Verify: the list is non-empty and includes a setting you can also see in _cluster/settings from a running node (Lab 1.3). This generalizes Stretch Goal 3 into a real inventory.

5. (core) A SPDX-header auditor. Write spdx-audit.sh that finds every *.java under a given directory whose first 20 lines do not contain SPDX-License-Identifier, printing the offenders. Verify: on server/src/main/java it prints nothing (all files carry the header); create a new .java file without the header in a scratch dir and confirm the auditor flags it — the same gate ./gradlew precommit enforces (Step 6).

6. (advanced) Advanced challenge — a code-spelunking tool that traces a route to its action. Write trace-endpoint.sh <path-fragment> (e.g. _count) that: (1) finds the Rest*Action whose routes() registers a matching new Route(, via rg; (2) reads that handler to find the *Request/*Action it builds; (3) rgs for the matching Transport*Action class declaration; and (4) prints the chain REST handler → request → transport action with file paths. Verify: for _count it prints RestCountAction → ... → TransportSearchAction-family (confirm the real chain on your branch — names drift), and for _search it prints RestSearchAction → TransportSearchAction. This is the bridge to Level 3, Lab 3.1: you have automated the trace you will do by hand there.

Issues to Practice On

The skills here — locating code, reading build.gradle, knowing the right layer — are exactly what let you triage issues, so this lab pairs naturally with issue triage on opensearch-project/OpenSearch.

# Untriaged issues are where navigation skill pays off — you find where the bug lives:
gh issue list --repo opensearch-project/OpenSearch --label "untriaged" --state open
gh issue list --repo opensearch-project/OpenSearch --label "good first issue" --state open

# Area labels point you at a layer (labels move; confirm on the tracker):
gh issue list --repo opensearch-project/OpenSearch --label "Storage" --state open
gh issue list --repo opensearch-project/OpenSearch --label "Cluster Manager" --state open
gh label list --repo opensearch-project/OpenSearch | grep -iE "search|cluster|storage|index|plugin"

Representative issue patterns for this subsystem:

• An "untriaged" issue that needs locating. The triage skill is navigation: read the report, reproduce, then use your findclass.sh / trace-endpoint.sh tooling to pinpoint the responsible class and propose the right label/area. A high-quality triage comment ("this lives in server/.../X.java, the relevant method is Y, likely area Search") is itself a valued contribution and often the first step a maintainer asks for.
• A "wrong layer" question. Issues sometimes propose a fix in :server that actually belongs in libs/* (or a module). Use the dependency-direction check from Exercise 3 to argue where it should live, citing the build.gradle evidence.

Planted-bug drill. Break the layering rule and watch the build catch it:

1. In a scratch branch, add implementation project(':server') to libs/core/build.gradle and run ./gradlew :libs:core:compileJava. Watch Gradle either reject the circular dependency or surface it. Revert, and confirm your Exercise-3 checker would have flagged it before you ran Gradle.
2. Then a header plant: create a new .java file under server/src/main/java/... without the SPDX header, run ./gradlew :server:precommit (or the licenseHeaders task), and watch it fail. Note the exact task name in the error. Fix by copying the header from a sibling. This is the gate every new file in Lab 2.3 must pass.

Etiquette: triage and PRs alike — claim/comment before working, reproduce first, and any code PR needs a test + CHANGELOG + DCO Signed-off-by (git commit -s; Lab 2.2). Triage etiquette and tone: community interaction.

Validation / Self-check

You are done when you can answer these without notes:

1. Which top-level dir holds the core engine, and which holds the shared wire-serialization primitives? Why are they separate?
2. In which direction may dependencies point between :server and libs/*? How do you verify it from a build.gradle?
3. Give the three ways to locate a class, and which you reach for when you know only the behavior, not the name.
4. What are the three source sets of :server, and which Gradle test task runs each?
5. What must every new .java file contain to pass precommit, and where do you copy it from?
6. What is rest-api-spec/ for, and what does qa/ hold?

Next: Lab 2.2 — Prepare a PR Using OpenSearch Practices.