OpenSearch Warm-Up: From User to Contributor

Before you read a single line of IndexShard.java, you need to have sat in the seat of the person whose workload OpenSearch is serving. The engineers who built OpenSearch's coordination layer, search fan-out, and indexing engine were solving specific, painful problems that show up in production log analytics, full-text search, and observability workloads every day. If you skip that context and go straight to the source, you will memorize class names without understanding why the design exists.

This chapter is the missing first mile. You will run OpenSearch from the outside — as a user would — across a series of practical scenarios covering different data shapes, query patterns, and cluster behaviors. After each scenario, the chapter maps what you observed back to the org.opensearch.* source structures that own it, with real grep/find commands. By the end, every internal class will feel like an old acquaintance rather than an alien term.

Everything here assumes a local cluster from the prerequisites: ./gradlew run live, REST on localhost:9200.


What OpenSearch Actually Is (Two Sentences)

OpenSearch is a distributed search and analytics engine built on Apache Lucene: it stores JSON documents in sharded, replicated indices and serves full-text search, structured queries, and aggregations over a REST API. It is the engineOpenSearch Dashboards is a completely separate TypeScript application (its own repo, opensearch-project/OpenSearch-Dashboards) that talks to the engine over HTTP to draw charts and dashboards.

Hold that boundary the entire curriculum: when a chart is wrong, the bug is in Dashboards (the query it built), in core search (how the engine ran it), in a plugin, or in Lucene itself — and your job as a contributor is to attribute it correctly. This warm-up keeps that line sharp.


Where OpenSearch Sits in the Search/Analytics Spectrum

┌──────────────────────────────────────────────────────────────────────────────┐
│                   Search & Analytics Tool Spectrum                            │
│                                                                              │
│  Full-text / relevance ◄──────────────────────────────► Analytics / OLAP     │
│                                                                              │
│  Solr      OpenSearch / Elasticsearch        ClickHouse        Splunk         │
│  (Lucene)  (Lucene, distributed)             (columnar OLAP)   (logs, SPL)     │
│                                                                              │
│  Vector / semantic:  Milvus · pgvector · OpenSearch k-NN plugin               │
│                                                                              │
│  ──────────────────────────────────────────────────────────────────────       │
│  Ingest:   Logstash · Data Prepper · Fluent Bit · Beats → OpenSearch          │
│  Store:    Lucene segments on local disk / remote-backed store (S3)           │
│  Query UI: OpenSearch Dashboards (the analog of Kibana / Grafana)             │
└──────────────────────────────────────────────────────────────────────────────┘

OpenSearch lives at the intersection: a distributed, near-real-time engine that is strong at both relevance-ranked full-text search and log/observability analytics, while not being a true columnar OLAP database. Knowing the neighbors tells you when OpenSearch is the right tool and where a reported "OpenSearch is slow" problem is really a "wrong tool" problem.

OpenSearch vs. Elasticsearch (the fork)

This is the most important comparison, because OpenSearch is a fork of Elasticsearch 7.10.2.

DimensionOpenSearchElasticsearch
OriginForked from Elasticsearch 7.10.2 (2021)The original Lucene-based engine
LicenseApache 2.0SSPL / Elastic License (8.x); Elastic re-added AGPL option later
GovernanceOpenSearch Software Foundation under the Linux Foundation, a TSCElastic N.V. (a company)
Source of truthGitHub issues + PRs, DCO sign-off, no CLAGitHub + Elastic's CLA
Package namespaceorg.opensearch.*org.elasticsearch.*
Terminology"cluster manager" (renamed from master)still "master"
Notable divergencesegment replication, remote-backed storage, Extensions SDK, the cluster-manager renameits own 8.x feature line

Because of the shared lineage, much of the architecture is identical in shape — but class names are org.opensearch.*, and many features have diverged. When you read old Elasticsearch blog posts, mentally translate mastercluster_manager and org.elasticsearchorg.opensearch.

OpenSearch vs. Apache Solr

DimensionOpenSearchApache Solr
Lucene-basedYesYes
Distribution modelNative: shards, replicas, cluster manager, RoutingTableSolrCloud + ZooKeeper
CoordinationBuilt-in Coordinator (Zen2/Raft-like)External ZooKeeper ensemble
API styleJSON over REST, rich Query DSLXML/JSON, query params + JSON DSL
Sweet spotLogs, observability, app search, dashboardsEnterprise/site search, faceting
EcosystemDashboards, plugins (k-NN, SQL, alerting)Solr admin UI, plugins

Both wrap Lucene. The structural difference is that OpenSearch ships its own coordination layer (no external ZooKeeper); Solr leans on ZooKeeper for cluster state. If you ever debug split-brain or election issues, that is org.opensearch.cluster.coordination, not a separate service.

OpenSearch vs. ClickHouse

DimensionOpenSearchClickHouse
Engine typeInverted index + DocValues (Lucene)True columnar MPP OLAP
Best atRelevance search, flexible filtering, near-real-time logsMassive analytical scans/aggregations
SchemaDynamic mappings, JSON documentsStrongly typed columnar tables
Aggregationsdate_histogram, terms, etc. over DocValuesSQL GROUP BY over columns, vectorized
UpdatesDocument-level, near-real-timeAppend-optimized, slow point updates

If a user runs trillion-row GROUP BY reports with no text search and no per-document relevance, ClickHouse will often win. OpenSearch wins when you need full-text relevance, flexible ad-hoc filtering, and a search-shaped data model alongside aggregations.

OpenSearch vs. Vector Databases (Milvus / pgvector)

DimensionOpenSearch (+ k-NN plugin)Milvus / pgvector
Vector searchk-NN plugin (HNSW/IVF via Lucene/FAISS/nmslib)Purpose-built ANN engines
Hybrid (text + vector)Native: combine match + k-NN in one queryUsually needs a separate text store
Core engineVectors are an added field type via pluginVectors are the entire point

OpenSearch does vector/semantic search through the k-NN plugin (a separate repo), layered on the same shards and search path. For pure, billion-scale ANN with nothing else, a dedicated vector DB may be leaner; for hybrid lexical+semantic search in one system, OpenSearch is compelling.

OpenSearch vs. Splunk

DimensionOpenSearchSplunk
License/costOpen source, Apache 2.0Commercial, volume-priced
Query languageQuery DSL (JSON), SQL/PPL via pluginSPL (Search Processing Language)
UIOpenSearch DashboardsSplunk Web
Sweet spotLogs, security analytics, observabilityLogs, SIEM, ops analytics

OpenSearch is frequently adopted as an open, self-hosted alternative to Splunk for log analytics and security. The compute model differs (Lucene index vs. Splunk's index-on-ingest), but the workload (ingest logs, search/filter, aggregate, dashboard) is the same — which is exactly Scenarios 1 and 3 below.


The Data Model: Documents, Indices, Shards, Segments

You must be able to say precisely what is OpenSearch and what is Lucene — this distinction decides where a bug lives.

Cluster
 └── Index  "logs-2026.06.16"  (OpenSearch concept: mappings, settings, aliases)
      ├── Primary shard 0  ─────────┐
      │    └── Lucene index         │  (LUCENE: this is a real Lucene index)
      │         ├── segment _0      │      immutable, append-only files
      │         ├── segment _1      │      built by IndexWriter, read by DirectoryReader
      │         └── segment _2      │
      ├── Replica shard 0  (copy of primary 0, on another node)
      ├── Primary shard 1
      └── Replica shard 1
ConceptOwnerWhat it is
DocumentOpenSearchA JSON object you index. Has an _id, lives in one index, routed to one shard.
IndexOpenSearchA named collection of documents with mappings (field types) and settings (shard count, etc.).
Shard (primary/replica)OpenSearchA horizontal slice of an index. The unit of distribution and replication. Each shard is a Lucene index.
SegmentLuceneAn immutable mini-index inside a shard. New docs go to new segments; merges combine them.
MappingOpenSearchField → type declaration (text, keyword, long, date, nested, …). Drives how Lucene fields are built.
Analyzer / Tokenizer / TokenFiltermostly Lucene (configured by OpenSearch)Turns text field values into terms ("Hello World"[hello, world]).
Inverted indexLuceneterm → posting list (which docs contain the term). The core of full-text search.
DocValuesLuceneColumnar per-field storage used for sorting and aggregations.
TranslogOpenSearchA write-ahead log for durability between Lucene commits. Not a Lucene concept.
Cluster state / routing / allocationOpenSearchWhere shards live, who is cluster manager, index metadata. Pure OpenSearch.

Rule of thumb: anything about scoring, tokenization, segments, postings, DocValues is Lucene (OpenSearch configures it). Anything about shards across nodes, replication, cluster state, REST, transport, translog, aggregation reduce is OpenSearch. Keep this table near your desk.


What the user does — bulk-index some structured logs, then filter by status and time range.

# Bulk-index four log lines into an index "weblogs".
curl -s -H 'Content-Type: application/x-ndjson' \
  -XPOST 'localhost:9200/weblogs/_bulk?refresh=true' --data-binary $'
{"index":{}}
{"@timestamp":"2026-06-16T10:00:00Z","status":200,"bytes":512,"path":"/home","client":"10.0.0.1"}
{"index":{}}
{"@timestamp":"2026-06-16T10:00:05Z","status":404,"bytes":128,"path":"/missing","client":"10.0.0.2"}
{"index":{}}
{"@timestamp":"2026-06-16T10:01:00Z","status":500,"bytes":256,"path":"/api","client":"10.0.0.1"}
{"index":{}}
{"@timestamp":"2026-06-16T10:02:30Z","status":200,"bytes":900,"path":"/home","client":"10.0.0.3"}
'

# Filter: 5xx errors in a time window. A "filter" context = no scoring, just yes/no.
curl -s -H 'Content-Type: application/json' 'localhost:9200/weblogs/_search?pretty' -d '
{
  "query": {
    "bool": {
      "filter": [
        { "range": { "status": { "gte": 500 } } },
        { "range": { "@timestamp": { "gte": "2026-06-16T10:00:00Z", "lte": "2026-06-16T10:05:00Z" } } }
      ]
    }
  }
}'

You get back the one status:500 document. Notice "max_score": null — filter context does no relevance scoring.

What OpenSearch does under the hood:

  1. RestBulkAction parses the NDJSON body, builds a BulkRequest, and calls the NodeClient.
  2. TransportBulkAction auto-creates the weblogs index (a cluster state update that adds index metadata and a RoutingTable entry), dynamically maps statuslong, @timestampdate, pathtext+keyword, then groups documents per target shard.
  3. Each shard's writes go through TransportShardBulkActionIndexShard.applyIndexOperationOnPrimary(...)InternalEngine.index(...) → Lucene IndexWriter.addDocument(...) + Translog.add(...). ?refresh=true forces an immediate refresh so the docs are searchable.
  4. The search request hits RestSearchActionTransportSearchAction. The coordinating node fans out to each shard; per shard, SearchService.executeQueryPhase runs the range filters as Lucene PointRangeQuery over the numeric/date _doc_values/points, collecting matching doc IDs.
  5. FetchPhase loads the stored _source for the hits; SearchPhaseController merges shard results on the coordinating node.

Bridge to source code:

cd ~/src/OpenSearch

# The REST handler that accepts /_bulk
find server -name "RestBulkAction.java"
#   server/src/main/java/org/opensearch/rest/action/document/RestBulkAction.java

# The transport action that splits a bulk by shard and creates indices/mappings
grep -rn "class TransportBulkAction" server/src/main/java/org/opensearch/action/bulk/

# Where a range filter becomes a Lucene query
grep -rn "PointRangeQuery\|newRangeQuery" server/src/main/java/org/opensearch/index/mapper/ | head

Scenario 2: Full-Text Relevance Query (match, BM25, _explain)

What the user does — index a few articles, then run a relevance-ranked match query and ask the engine to explain the score.

curl -s -H 'Content-Type: application/x-ndjson' \
  -XPOST 'localhost:9200/articles/_bulk?refresh=true' --data-binary $'
{"index":{"_id":"1"}}
{"title":"OpenSearch query performance tuning","body":"Tuning search relevance and query latency in OpenSearch."}
{"index":{"_id":"2"}}
{"title":"A gentle introduction to Lucene","body":"Lucene powers search; OpenSearch builds on it."}
{"index":{"_id":"3"}}
{"title":"Cooking with cast iron","body":"Nothing to do with search at all."}
'

# Relevance query in "query" context — scored by BM25 by default.
curl -s -H 'Content-Type: application/json' 'localhost:9200/articles/_search?pretty' -d '
{ "query": { "match": { "body": "search relevance" } } }'

# Ask WHY document 1 scored what it did.
curl -s -H 'Content-Type: application/json' 'localhost:9200/articles/_explain/1?pretty' -d '
{ "query": { "match": { "body": "search relevance" } } }'

The _explain response is a tree of BM25 components (termFreq, idf, tf, field length norm). The top scoring doc is the one whose body best matches the analyzed terms [search, relevance].

What OpenSearch does under the hood:

  1. The match query is a QueryBuilder (MatchQueryBuilder). At query time it runs the field's analyzer over "search relevance" to produce terms, then builds a Lucene BooleanQuery of TermQuery clauses via QueryShardContext.
  2. Per shard, QueryPhase executes the Lucene query; Lucene scores each matching doc with BM25 (the default Similarity), using term frequency, inverse document frequency, and field-length normalization — all computed inside Lucene, not OpenSearch.
  3. The coordinating node merges the top-K from each shard in SearchPhaseController; FetchPhase loads _source for the survivors.
  4. _explain runs the same query against one doc and returns Lucene's Explanation tree verbatim.

Bridge to source code:

# The match query builder
find server -name "MatchQueryBuilder.java"
#   server/src/main/java/org/opensearch/index/query/MatchQueryBuilder.java

# Where QueryBuilders turn into Lucene Query objects
grep -rn "QueryShardContext\|toQuery" server/src/main/java/org/opensearch/index/query/AbstractQueryBuilder.java | head

# The query phase itself
find server -name "QueryPhase.java"
#   server/src/main/java/org/opensearch/search/query/QueryPhase.java

# BM25 / Similarity wiring (OpenSearch configures Lucene's Similarity)
grep -rn "BM25\|Similarity" server/src/main/java/org/opensearch/index/similarity/ | head

Scenario 3: Aggregations (date_histogram + terms sub-agg) — The Dashboards Workload

What the user does — the canonical "log volume over time, broken down by status" dashboard panel.

curl -s -H 'Content-Type: application/json' 'localhost:9200/weblogs/_search?pretty' -d '
{
  "size": 0,
  "aggs": {
    "per_minute": {
      "date_histogram": { "field": "@timestamp", "fixed_interval": "1m" },
      "aggs": {
        "by_status": { "terms": { "field": "status" } }
      }
    }
  }
}'

"size": 0 means "no hits, just the aggregation". The result is buckets per minute, each with a nested breakdown by status. This single request shape is what powers a huge fraction of OpenSearch Dashboards visualizations.

What OpenSearch does under the hood:

  1. RestSearchAction parses the aggs block into an AggregatorFactories tree: DateHistogramAggregationBuilder with a child TermsAggregationBuilder.
  2. Per shard, each AggregatorFactory produces an Aggregator. The aggregators iterate matching docs reading the @timestamp and status values from DocValues (columnar, not the inverted index), bucketing as they go. This is why aggregations need doc_values (on by default for numerics/dates and keyword).
  3. Each shard returns a partial InternalAggregation. The coordinating node calls InternalAggregation.reduce(...) to merge partial buckets into the final result — this reduce step is where shard-local partial results become a globally correct answer.

Bridge to source code:

# Date histogram aggregation
grep -rln "class DateHistogramAggregator" server/src/main/java/org/opensearch/search/aggregations/

# Terms aggregation
grep -rln "class TermsAggregator" server/src/main/java/org/opensearch/search/aggregations/bucket/terms/

# The reduce contract — how partial shard aggs merge on the coordinating node
grep -rn "public InternalAggregation reduce" server/src/main/java/org/opensearch/search/aggregations/ | head

# Aggregations read DocValues, not the inverted index
grep -rn "getLeafCollector\|LeafBucketCollector" server/src/main/java/org/opensearch/search/aggregations/AggregatorBase.java | head

See the Aggregations deep dive and the DocValues and Fielddata deep dive for the full mechanics.


Scenario 4: Multi-Node Cluster — Replicas, Health, Killing a Node

What the user does — bring up a 3-node cluster, create an index with replicas, check health, then kill a node and watch shards reallocate. This is the heart of OpenSearch's distributed nature.

# Run a 3-node local cluster from source instead of the single-node default.
./gradlew run -PnumNodes=3
# Create an index with 2 primaries and 1 replica each (4 shards total).
curl -s -H 'Content-Type: application/json' -XPUT 'localhost:9200/inventory' -d '
{ "settings": { "number_of_shards": 2, "number_of_replicas": 1 } }'

# Cluster health: should be green (every replica has a home).
curl -s 'localhost:9200/_cluster/health?pretty'

# See exactly where each shard lives and its state (STARTED / RELOCATING / UNASSIGNED).
curl -s 'localhost:9200/_cat/shards/inventory?v'
curl -s 'localhost:9200/_cat/nodes?v'

Now kill one node process (find it from _cat/nodes or your process list) and immediately re-check:

# In the ./gradlew run output, identify a data node's PID; kill it. Then:
curl -s 'localhost:9200/_cluster/health?pretty'        # status flips to yellow, then back
curl -s 'localhost:9200/_cat/shards/inventory?v'       # watch UNASSIGNED → INITIALIZING → STARTED

You will see health go yellow (a replica lost its home), the cluster manager re-elect if you killed it, and the AllocationService promote a surviving replica to primary and rebuild the missing copy elsewhere — eventually back to green.

What OpenSearch does under the hood:

  1. The elected cluster manager (formerly master) owns the authoritative ClusterState. When a node dies, FollowersChecker/LeaderChecker detect it and the cluster manager publishes a new ClusterState removing that node from DiscoveryNodes.
  2. AllocationService + BalancedShardsAllocator, gated by AllocationDeciders (SameShardAllocationDecider, DiskThresholdDecider, MaxRetryAllocationDecider, …), recompute the RoutingTable: promote a replica to primary, allocate a new replica.
  3. The new state is published two-phase (publish → commit) from the cluster manager to all nodes via PublicationTransportHandler; each node's ClusterApplierService applies it locally.
  4. Recovery of the rebuilt replica runs through PeerRecoverySourceService/PeerRecoveryTargetService, tracked by sequence numbers (LocalCheckpointTracker, global checkpoint via ReplicationTracker).

Bridge to source code:

# The coordination layer: election, joins, publishing
ls server/src/main/java/org/opensearch/cluster/coordination/
grep -rln "class Coordinator" server/src/main/java/org/opensearch/cluster/coordination/

# The allocation engine and the deciders that gate it
grep -rln "class AllocationService" server/src/main/java/org/opensearch/cluster/routing/allocation/
ls server/src/main/java/org/opensearch/cluster/routing/allocation/decider/

# Where a node-left event triggers reroute
grep -rn "disassociateDeadNodes\|reroute" server/src/main/java/org/opensearch/cluster/routing/allocation/AllocationService.java | head

The Discovery and Coordination, Shard Allocation, and Recovery deep dives cover this in full. Level 4 is built entirely around this scenario.


Scenario 5: Snapshot to a Filesystem Repository and Restore

What the user does — register a filesystem snapshot repository, snapshot an index, delete it, and restore from the snapshot. This is the backup/restore and disaster-recovery story.

# ./gradlew run permits the build's temp dir as a snapshot path by default; if not,
# add path.repo to the run config. Register an "fs" repository:
curl -s -H 'Content-Type: application/json' -XPUT 'localhost:9200/_snapshot/my_fs_repo' -d '
{ "type": "fs", "settings": { "location": "my_backup_dir" } }'

# Take a snapshot of the "weblogs" index, waiting for it to finish.
curl -s -XPUT 'localhost:9200/_snapshot/my_fs_repo/snap-1?wait_for_completion=true&pretty' \
  -H 'Content-Type: application/json' -d '{ "indices": "weblogs" }'

# Destroy the index, then restore it from the snapshot.
curl -s -XDELETE 'localhost:9200/weblogs'
curl -s -XPOST 'localhost:9200/_snapshot/my_fs_repo/snap-1/_restore?wait_for_completion=true&pretty'

# Confirm the data is back.
curl -s 'localhost:9200/weblogs/_count?pretty'

What OpenSearch does under the hood:

  1. RepositoriesService registers the fs repository (FsRepository, a BlobStoreRepository).
  2. SnapshotsService runs a snapshot as a series of cluster-state transitions: it copies Lucene segment files (incrementally — only segments not already in the repo) plus index metadata into the blob store. Snapshots are incremental at the segment level.
  3. Restore is the inverse: RestoreService builds new shards from the snapshotted segments and recovers them (a special recovery source), then the AllocationService allocates them.

Bridge to source code:

# Snapshot and repository services
grep -rln "class SnapshotsService"   server/src/main/java/org/opensearch/snapshots/
grep -rln "class RepositoriesService" server/src/main/java/org/opensearch/repositories/
grep -rln "class BlobStoreRepository" server/src/main/java/org/opensearch/repositories/blobstore/

# The fs repository implementation
find server modules plugins -name "FsRepository.java"

# Restore is recovery from a snapshot source
grep -rln "class RestoreService" server/src/main/java/org/opensearch/snapshots/

See the Snapshots and Repositories deep dive.


Dataset Scenarios for Testing Edge Cases

When you write a repro or validate a fix, the dataset you choose determines which code path you exercise. Use these as starting templates; each names the subsystem it stresses.

Dataset 1: The Empty Index

curl -s -XPUT 'localhost:9200/empty_idx'
curl -s 'localhost:9200/empty_idx/_search?pretty'        # 0 hits, must NOT error
curl -s 'localhost:9200/empty_idx/_search?pretty' -H 'Content-Type: application/json' \
  -d '{ "aggs": { "s": { "sum": { "field": "missing" } } } }'

What this tests: the coordinating-node reduce path with zero shard results, and aggregations over a field that does not exist. SearchPhaseController.reducedQueryPhase(...) and InternalAggregation.reduce(...) must produce an empty-but-valid response, not an NPE. Historically a rich source of "empty result" bugs. Source: server/src/main/java/org/opensearch/action/search/SearchPhaseController.java.

Dataset 2: A Single Huge Document

# One ~5 MB document — stresses source storage, field length, and the http.max_content_length limit.
python3 - <<'PY' > /tmp/huge.json
import json
print(json.dumps({"blob": "x" * 5_000_000, "n": 1}))
PY
curl -s -XPOST 'localhost:9200/huge/_doc?refresh=true' -H 'Content-Type: application/json' --data-binary @/tmp/huge.json

What this tests: _source storage, the http.max_content_length guard in the REST/HTTP layer, and Lucene stored-field limits. Exercises RestController request-size handling and IndexShard/InternalEngine for a single oversized doc. Source: server/src/main/java/org/opensearch/rest/RestController.java and the HTTP transport in modules/transport-netty4.

Dataset 3: Deeply Nested Objects

curl -s -XPUT 'localhost:9200/nested_idx' -H 'Content-Type: application/json' -d '
{ "mappings": { "properties": { "user": { "type": "nested",
    "properties": { "comments": { "type": "nested" } } } } } }'

What this tests: the nested field type and the index.mapping.nested_objects.limit / index.mapping.depth.limit guards. Nested docs are stored as hidden child Lucene documents, so this exercises ObjectMapper/NestedObjectMapper and the join-based nested query. Source: server/src/main/java/org/opensearch/index/mapper/ObjectMapper.java and NestedObjectMapper.

Dataset 4: High-Cardinality Terms Field

# Index many unique keyword values, then a terms agg over them.
for i in $(seq 1 5000); do
  printf '{"index":{}}\n{"uid":"user-%s"}\n' "$i"
done | curl -s -H 'Content-Type: application/x-ndjson' \
  -XPOST 'localhost:9200/hicard/_bulk?refresh=true' --data-binary @-

curl -s 'localhost:9200/hicard/_search?pretty' -H 'Content-Type: application/json' \
  -d '{ "size": 0, "aggs": { "u": { "terms": { "field": "uid", "size": 10 } } } }'

What this tests: the terms aggregator's bucket collection over a high-cardinality field — memory pressure, the request circuit breaker, and shard-level vs. coordinating-node accuracy (doc_count_error_upper_bound). This is a classic OOM/breaker-trip path. Source: server/src/main/java/org/opensearch/search/aggregations/bucket/terms/ and HierarchyCircuitBreakerService.

Dataset 5: Many Tiny Shards

# 50 shards for a trivially small index — a real anti-pattern users hit.
curl -s -XPUT 'localhost:9200/oversharded' -H 'Content-Type: application/json' \
  -d '{ "settings": { "number_of_shards": 50, "number_of_replicas": 0 } }'
curl -s 'localhost:9200/_cat/shards/oversharded?v'
curl -s 'localhost:9200/_cluster/health/oversharded?pretty'

What this tests: AllocationService/BalancedShardsAllocator placing many shards, cluster-state size growth (50 routing entries), and the per-shard search fan-out overhead in TransportSearchAction (50 shard requests for almost no data). Source: server/src/main/java/org/opensearch/cluster/routing/allocation/allocator/BalancedShardsAllocator.java and org/opensearch/action/search/TransportSearchAction.java.

Dataset 6: Time-Series With Rollover

# Create a write alias backed by a dated index, then roll it over on a size/age condition.
curl -s -XPUT 'localhost:9200/ts-000001' -H 'Content-Type: application/json' \
  -d '{ "aliases": { "ts-write": { "is_write_index": true } } }'
curl -s -XPOST 'localhost:9200/ts-write/_rollover?pretty' -H 'Content-Type: application/json' \
  -d '{ "conditions": { "max_docs": 1, "max_age": "7d" } }'
curl -s 'localhost:9200/_cat/indices/ts-*?v'

What this tests: the rollover path (alias → new backing index), index lifecycle thinking, and the metadata mutations that create a new index as a cluster-state update task. Source: server/src/main/java/org/opensearch/action/admin/indices/rollover/ and MetadataRolloverService.


The Bridge: User Scenario → Source Code (Master Table)

Use this whenever you observe a runtime behavior and want to find the code that owns it. Exact class names can vary by branch — when in doubt, grep for the class name under server/src/main/java.

Observed behaviorOwning subsystem / class
HTTP request hits :9200 and is routedRestController → a RestHandler (RestSearchAction, RestBulkAction, RestIndexAction)
A REST handler dispatches to server logicNodeClient.execute(ActionType, request, listener) → a TransportAction (wired in ActionModule)
A search fans out to shards and mergesTransportSearchAction → per-shard SearchService.executeQueryPhase/executeFetchPhase; merge in SearchPhaseController
The query phase runs the Lucene queryQueryPhase, with SearchContext/DefaultSearchContext per shard
The fetch phase loads _source for hitsFetchPhase
Global term stats across shards (optional)DfsPhase
An aggregation is computed and reducedAggregatorFactoryAggregatorInternalAggregation.reduce(...)
A document is indexed on the primaryTransportBulkAction/TransportShardBulkActionIndexShard.applyIndexOperationOnPrimaryInternalEngine.index
A write is replicated to replicasTransportReplicationAction (document replication) or SegmentReplicationTargetService (segment replication)
A doc becomes searchable / durablerefresh (new searcher) / Translog + flush (Lucene commit)
A request must run on the cluster managerTransportClusterManagerNodeAction (formerly TransportMasterNodeAction)
Cluster metadata changesClusterStateUpdateTask/ClusterStateTaskExecutor on the cluster-manager service
A new cluster state is distributedMasterService computes it; PublicationTransportHandler publishes; ClusterApplierService applies
Leader election / node failure detectionCoordinator, PreVoteCollector, ElectionSchedulerFactory, FollowersChecker, LeaderChecker
Shards placed / rebalancedAllocationService + BalancedShardsAllocator, gated by AllocationDeciders
A replica rebuilt after a node diesPeerRecoverySourceService/PeerRecoveryTargetService, ReplicationTracker
Snapshot/restoreSnapshotsService, RepositoriesService, BlobStoreRepository, RestoreService
Cross-node wire serializationWriteable + StreamInput/StreamOutput, NamedWriteableRegistry, over TransportService/Netty4Transport
Memory protection tripsHierarchyCircuitBreakerService (parent, fielddata, request, in_flight_requests)

Each row has a deep dive: REST layer, Action framework, Transport layer, Search execution, Engine internals, Cluster state, Cluster state publishing, Shard allocation, Recovery, Circuit breakers and memory.


Running OpenSearch End-to-End: The Local Developer Loop

Every contributor should be able to do this loop in under ten minutes, from cold:

cd ~/src/OpenSearch

# 1. Launch a local cluster from source (REST on :9200). Leave running.
./gradlew run

# 2. The canonical health check (separate terminal).
curl -s 'localhost:9200/_cluster/health?pretty'

# 3. Smoke test: index a doc and read it back.
curl -s -XPOST 'localhost:9200/sanity/_doc/1?refresh=true' \
  -H 'Content-Type: application/json' -d '{"hello":"world"}'
curl -s 'localhost:9200/sanity/_doc/1?pretty'
curl -s 'localhost:9200/sanity/_search?pretty' -d '{"query":{"match_all":{}}}' \
  -H 'Content-Type: application/json'

# 4. The fast inner loop you will live in: run one test class / one method.
./gradlew :server:test --tests "org.opensearch.search.query.QueryPhaseTests"
./gradlew :server:test --tests "org.opensearch.cluster.ClusterStateTests.testToXContent"

# 5. Multi-node integration tests (in-JVM, backed by InternalTestCluster).
./gradlew :server:internalClusterTest --tests "org.opensearch.cluster.*"

Your baseline health check is: ./gradlew run serves :9200, a known-good :server:test class passes, and a trivial index+search round-trips. If any of those fail before you have changed a single line, stop and fix your environment — do not start Level 1 on a broken baseline, because every later ./gradlew test will produce false failures that hide the real work.


What to Verify Before Starting Level 1

Run through this checklist once. It takes 30–45 minutes and proves your environment and your understanding.

# Environment
java -version                       # JDK 21
cd ~/src/OpenSearch
./gradlew --version | head -3        # wrapper healthy

# Build + run
./gradlew localDistro                # archive under distribution/archives/
./gradlew run                        # serves :9200 (leave running, new terminal below)

# Functional round-trip
curl -s 'localhost:9200/_cluster/health?pretty'                 # green/yellow
curl -s -XPOST 'localhost:9200/check/_doc/1?refresh=true' \
  -H 'Content-Type: application/json' -d '{"k":"v"}'
curl -s 'localhost:9200/check/_search?pretty'                   # 1 hit

# Tests
./gradlew :server:test --tests "org.opensearch.cluster.ClusterStateTests" 2>&1 | tail -5

You are ready when, without notes, you can:

  • State the OpenSearch-vs-Lucene boundary for: scoring, segments, DocValues, translog, cluster state, REST/transport.
  • Name the four steps of the search path: REST handler → transport action → query phase → fetch phase, plus the coordinating-node reduce.
  • Name the indexing path: bulk/index action → IndexShardInternalEngine → Lucene IndexWriter + translog.
  • Explain why killing a node turns the cluster yellow and how it returns to green (cluster manager → new ClusterStateAllocationService → recovery).
  • Run a single ./gradlew :server:test --tests ... and a single curl search from memory.
  • Open RestSearchAction, TransportSearchAction, QueryPhase, and IndexShard in your IDE via Go to Class.

If any box is unchecked, re-run the scenario it maps to before moving on.


Continue to the 16-Week Plan and Milestones, or jump to Level 1: Lucene and OpenSearch Foundation. The internals you previewed here are covered in full in the Deep Dives.