Level 1: Lucene and OpenSearch Foundation

This level establishes the technical baseline every subsequent level depends on. You will build OpenSearch from source with Gradle, run its randomized test suite, launch a real single-node cluster from your own checkout, index documents and query them over REST, and — critically — write a tiny standalone Lucene program so you understand what a "shard" actually is before you ever read IndexShard.

OpenSearch is a distributed search and analytics engine built on Apache Lucene. It was forked from Elasticsearch 7.10.2 in 2021 (after Elastic relicensed under SSPL) and is licensed Apache 2.0. Governance now sits with the OpenSearch Software Foundation under the Linux Foundation, overseen by a Technical Steering Committee (TSC). The source of truth is GitHub — github.com/opensearch-project/OpenSearch — not JIRA, and there is no CLA: contributions are made by GitHub Pull Request with a DCO sign-off.

This curriculum will not hold your hand. It will point you at the right parts of the codebase, give you the right questions to ask, and make you run everything you read.

Learning Objectives

By the end of Level 1 you must be able to:

1. Explain where OpenSearch sits in the search stack — Dashboards above it, Lucene beneath it — and which problems each layer owns.
2. Build OpenSearch from source with the Gradle wrapper and its bundled JDK, with and without running tests.
3. Run unit and integration tests scoped to a single project, reproduce a randomized failure from its -Dtests.seed, and read the HTML test reports.
4. Launch a single-node cluster with ./gradlew run, index documents over REST, and run a match query and a terms aggregation against localhost:9200.
5. Map a REST call (PUT /index, POST /_bulk, GET /_search) to the RestHandler and TransportAction that service it.
6. Write a standalone Lucene program that builds an index, opens a reader, and runs a query — and explain how IndexShard/InternalEngine wrap exactly those primitives.
7. Locate any class named in Levels 2–9 (Node, ClusterService, IndexShard, SearchService, …) without a search engine.

The Search-Stack Context

Before touching a line of OpenSearch code, build an accurate mental model of the stack. OpenSearch is one layer in a three-layer system:

┌───────────────────────────────────────────────────────────┐
│              OpenSearch Dashboards (TypeScript)             │  ← Visualization / UI
│         visualizations, dev tools, index management        │     (separate repo)
└───────────────────────────────────────────────────────────┘
                     │ HTTP: _search / _msearch / _bulk (JSON)
                     ▼
┌───────────────────────────────────────────────────────────┐
│                  OpenSearch (org.opensearch.*)             │  ← Distributed engine
│   REST layer  →  Transport/Action layer  →  cluster mgmt  │     (this repo)
│   RestController → TransportAction → IndicesService        │
│                    → IndexShard → Engine                   │
└───────────────────────────────────────────────────────────┘
                     │ Engine wraps a Lucene IndexWriter / DirectoryReader
                     ▼
┌───────────────────────────────────────────────────────────┐
│                     Apache Lucene (library)                │  ← Inverted index / search
│       IndexWriter, DirectoryReader, Query, Document        │
└───────────────────────────────────────────────────────────┘
                     │ Directory abstraction
                     ▼
┌───────────────────────────────────────────────────────────┐
│              Filesystem: segments (.cfs/.si/.dvd/…)         │  ← Immutable on-disk files
└───────────────────────────────────────────────────────────┘

The boundaries matter for everything that follows, especially debugging:

• Dashboards never touches your data directly. It issues _search/_bulk HTTP calls to OpenSearch through opensearch-js. A "slow visualization" is almost always a slow _search underneath. Dashboards is a client.
• OpenSearch owns the distributed problem: clusters, nodes, shards, replication, coordination, the REST API, the query DSL, aggregations. It does not implement the inverted index — it delegates to Lucene.
• Lucene owns the single-machine search problem: building inverted indexes, merging immutable segments, executing Query objects, scoring with BM25. A single OpenSearch shard is one Lucene index. You will prove this to yourself in Lab 1.4.

Note: When something is wrong, the first question a maintainer asks is which layer. Is it a Dashboards rendering bug, a coordinating-node reduce bug, a per-shard query bug, or a Lucene bug? Levels 7–9 and the plugin labs drill this attribution skill.

OpenSearch vs. Elasticsearch vs. Solr

You will be reading code whose history predates OpenSearch. Know the lineage:

The three share a Lucene heart. OpenSearch and Elasticsearch share a codebase up to 7.10.2 — which is why many class names (IndexShard, InternalEngine, SearchService) are identical and why Elasticsearch documentation from the 7.x era is often still accurate for OpenSearch internals. The most visible OpenSearch-specific change is the rename of "master" → "cluster manager" for inclusive language (the master role and cluster.initial_master_nodes are deprecated aliases of cluster.initial_cluster_manager_nodes).

Required Reading

Read these in order, in your own checkout, before starting the labs. In a mature codebase the best documentation is often in-repo Markdown and class-level Javadoc — read it seriously.

Run this to confirm the files exist on your branch (names occasionally move between major lines):

ls README.md DEVELOPER_GUIDE.md TESTING.md CONTRIBUTING.md CHANGELOG.md MAINTAINERS.md
find server/src/main/java/org/opensearch/node/Node.java \
     server/src/main/java/org/opensearch/index/shard/IndexShard.java

Source Code Areas to Inspect

You are not modifying anything yet — you are building a map. Skim these before and after the labs.

server/ — the core engine

This is the bulk of what you will read across the whole curriculum.

libs/ — shared low-level libraries

modules/ — bundled-by-default modules

test/framework/ — the test harness

Key Classes Quick Reference

Memorize the role of each. By the end of Level 1 you should be able to name the file path of any of these from memory.

Build the muscle memory now:

# You should be able to predict each path before running this.
for c in Node ClusterService ClusterState RestController IndicesService \
         IndexShard InternalEngine SearchService; do
  echo "== $c =="
  find server -name "$c.java" -path "*/main/*"
done

GitHub Issue Categories for Level 1 Contributors

OpenSearch tracks everything on GitHub. At this stage, restrict yourself to issues that exercise the workflow, not the engine:

• good first issue — curated, scoped, beginner-appropriate. Always start here.
• Documentation — incorrect Javadoc, stale REST examples, broken links, outdated version references in in-repo docs.
• flaky-test — tests that fail intermittently. At Level 1 you observe these and learn to read the -Dtests.seed reproduction line; you fix them in Level 5.

How to find them:

# The label-filtered issue lists (open in a browser):
#   https://github.com/opensearch-project/OpenSearch/issues?q=is:open+label:%22good+first+issue%22
#   https://github.com/opensearch-project/OpenSearch/issues?q=is:open+label:flaky-test

# Or with the gh CLI:
gh issue list --repo opensearch-project/OpenSearch \
  --label "good first issue" --state open --limit 30

Warning: Do not start coding on an issue without reading the full comment thread. If it has an assignee or an open linked PR, move on. Leave a comment saying you intend to work on it before you do. Etiquette is covered in depth in Community Interaction.

Deliverables

Demonstrate all of the following before advancing to Level 2:

• A successful ./gradlew assemble run — no build failures (Lab 1.1).
• At least one unit-test class run green via ./gradlew :server:test --tests ... (Lab 1.2).
• A running single-node cluster: GET _cat/health returns green and you have indexed and queried documents (Lab 1.3).
• A standalone Lucene program that builds an index and runs a TermQuery, plus a written explanation of how InternalEngine wraps the same primitives (Lab 1.4).
• Ability to name the file path of Node, IndexShard, InternalEngine, and SearchService from memory.
• A written explanation (2–3 sentences) of why an OpenSearch shard is a Lucene index.

Common Mistakes

How to Verify Success

# 1. Build artifacts without tests.
./gradlew assemble -q && echo "ASSEMBLE OK"

# 2. One unit-test class, green.
./gradlew :server:test --tests "org.opensearch.common.UUIDTests"

# 3. Launch a cluster in one terminal...
./gradlew run
#    ...and in another, confirm it is alive:
curl -s "localhost:9200/_cat/health?v"
#    expected: a row whose 'status' column is 'green'

# 4. The Lucene project (Lab 1.4) prints its hits, e.g.:
#    Found 2 hit(s) for term body:lucene

PR Profile: Level 1 Graduate

A Level 1 graduate can credibly open these kinds of PRs. Scope is everything.

You are not yet ready to submit: validation-logic changes, new REST actions, engine or allocation changes, or anything touching the wire protocol. Those start in Level 2 (workflow) and ramp through Levels 3–9.

Next: Lab 1.1 — Build OpenSearch from Source.