OpenSearch Warm-Up: From User to Contributor
Before you read a single line of IndexShard.java, you need to have sat in the seat of the person
whose workload OpenSearch is serving. The engineers who built OpenSearch's coordination layer, search
fan-out, and indexing engine were solving specific, painful problems that show up in production log
analytics, full-text search, and observability workloads every day. If you skip that context and go
straight to the source, you will memorize class names without understanding why the design exists.
This chapter is the missing first mile. You will run OpenSearch from the outside — as a user would —
across a series of practical scenarios covering different data shapes, query patterns, and cluster
behaviors. After each scenario, the chapter maps what you observed back to the org.opensearch.*
source structures that own it, with real grep/find commands. By the end, every internal class
will feel like an old acquaintance rather than an alien term.
Everything here assumes a local cluster from the prerequisites: ./gradlew run live,
REST on localhost:9200.
What OpenSearch Actually Is (Two Sentences)
OpenSearch is a distributed search and analytics engine built on Apache Lucene: it stores JSON
documents in sharded, replicated indices and serves full-text search, structured queries, and
aggregations over a REST API. It is the engine — OpenSearch Dashboards is a completely separate
TypeScript application (its own repo, opensearch-project/OpenSearch-Dashboards) that talks to the
engine over HTTP to draw charts and dashboards.
Hold that boundary the entire curriculum: when a chart is wrong, the bug is in Dashboards (the query it built), in core search (how the engine ran it), in a plugin, or in Lucene itself — and your job as a contributor is to attribute it correctly. This warm-up keeps that line sharp.
Where OpenSearch Sits in the Search/Analytics Spectrum
┌──────────────────────────────────────────────────────────────────────────────┐
│ Search & Analytics Tool Spectrum │
│ │
│ Full-text / relevance ◄──────────────────────────────► Analytics / OLAP │
│ │
│ Solr OpenSearch / Elasticsearch ClickHouse Splunk │
│ (Lucene) (Lucene, distributed) (columnar OLAP) (logs, SPL) │
│ │
│ Vector / semantic: Milvus · pgvector · OpenSearch k-NN plugin │
│ │
│ ────────────────────────────────────────────────────────────────────── │
│ Ingest: Logstash · Data Prepper · Fluent Bit · Beats → OpenSearch │
│ Store: Lucene segments on local disk / remote-backed store (S3) │
│ Query UI: OpenSearch Dashboards (the analog of Kibana / Grafana) │
└──────────────────────────────────────────────────────────────────────────────┘
OpenSearch lives at the intersection: a distributed, near-real-time engine that is strong at both relevance-ranked full-text search and log/observability analytics, while not being a true columnar OLAP database. Knowing the neighbors tells you when OpenSearch is the right tool and where a reported "OpenSearch is slow" problem is really a "wrong tool" problem.
OpenSearch vs. Elasticsearch (the fork)
This is the most important comparison, because OpenSearch is a fork of Elasticsearch 7.10.2.
| Dimension | OpenSearch | Elasticsearch |
|---|---|---|
| Origin | Forked from Elasticsearch 7.10.2 (2021) | The original Lucene-based engine |
| License | Apache 2.0 | SSPL / Elastic License (8.x); Elastic re-added AGPL option later |
| Governance | OpenSearch Software Foundation under the Linux Foundation, a TSC | Elastic N.V. (a company) |
| Source of truth | GitHub issues + PRs, DCO sign-off, no CLA | GitHub + Elastic's CLA |
| Package namespace | org.opensearch.* | org.elasticsearch.* |
| Terminology | "cluster manager" (renamed from master) | still "master" |
| Notable divergence | segment replication, remote-backed storage, Extensions SDK, the cluster-manager rename | its own 8.x feature line |
Because of the shared lineage, much of the architecture is identical in shape — but class names are
org.opensearch.*, and many features have diverged. When you read old Elasticsearch blog posts,
mentally translate master → cluster_manager and org.elasticsearch → org.opensearch.
OpenSearch vs. Apache Solr
| Dimension | OpenSearch | Apache Solr |
|---|---|---|
| Lucene-based | Yes | Yes |
| Distribution model | Native: shards, replicas, cluster manager, RoutingTable | SolrCloud + ZooKeeper |
| Coordination | Built-in Coordinator (Zen2/Raft-like) | External ZooKeeper ensemble |
| API style | JSON over REST, rich Query DSL | XML/JSON, query params + JSON DSL |
| Sweet spot | Logs, observability, app search, dashboards | Enterprise/site search, faceting |
| Ecosystem | Dashboards, plugins (k-NN, SQL, alerting) | Solr admin UI, plugins |
Both wrap Lucene. The structural difference is that OpenSearch ships its own coordination layer
(no external ZooKeeper); Solr leans on ZooKeeper for cluster state. If you ever debug split-brain or
election issues, that is org.opensearch.cluster.coordination, not a separate service.
OpenSearch vs. ClickHouse
| Dimension | OpenSearch | ClickHouse |
|---|---|---|
| Engine type | Inverted index + DocValues (Lucene) | True columnar MPP OLAP |
| Best at | Relevance search, flexible filtering, near-real-time logs | Massive analytical scans/aggregations |
| Schema | Dynamic mappings, JSON documents | Strongly typed columnar tables |
| Aggregations | date_histogram, terms, etc. over DocValues | SQL GROUP BY over columns, vectorized |
| Updates | Document-level, near-real-time | Append-optimized, slow point updates |
If a user runs trillion-row GROUP BY reports with no text search and no per-document relevance,
ClickHouse will often win. OpenSearch wins when you need full-text relevance, flexible ad-hoc
filtering, and a search-shaped data model alongside aggregations.
OpenSearch vs. Vector Databases (Milvus / pgvector)
| Dimension | OpenSearch (+ k-NN plugin) | Milvus / pgvector |
|---|---|---|
| Vector search | k-NN plugin (HNSW/IVF via Lucene/FAISS/nmslib) | Purpose-built ANN engines |
| Hybrid (text + vector) | Native: combine match + k-NN in one query | Usually needs a separate text store |
| Core engine | Vectors are an added field type via plugin | Vectors are the entire point |
OpenSearch does vector/semantic search through the k-NN plugin (a separate repo), layered on the same shards and search path. For pure, billion-scale ANN with nothing else, a dedicated vector DB may be leaner; for hybrid lexical+semantic search in one system, OpenSearch is compelling.
OpenSearch vs. Splunk
| Dimension | OpenSearch | Splunk |
|---|---|---|
| License/cost | Open source, Apache 2.0 | Commercial, volume-priced |
| Query language | Query DSL (JSON), SQL/PPL via plugin | SPL (Search Processing Language) |
| UI | OpenSearch Dashboards | Splunk Web |
| Sweet spot | Logs, security analytics, observability | Logs, SIEM, ops analytics |
OpenSearch is frequently adopted as an open, self-hosted alternative to Splunk for log analytics and security. The compute model differs (Lucene index vs. Splunk's index-on-ingest), but the workload (ingest logs, search/filter, aggregate, dashboard) is the same — which is exactly Scenarios 1 and 3 below.
The Data Model: Documents, Indices, Shards, Segments
You must be able to say precisely what is OpenSearch and what is Lucene — this distinction decides where a bug lives.
Cluster
└── Index "logs-2026.06.16" (OpenSearch concept: mappings, settings, aliases)
├── Primary shard 0 ─────────┐
│ └── Lucene index │ (LUCENE: this is a real Lucene index)
│ ├── segment _0 │ immutable, append-only files
│ ├── segment _1 │ built by IndexWriter, read by DirectoryReader
│ └── segment _2 │
├── Replica shard 0 (copy of primary 0, on another node)
├── Primary shard 1
└── Replica shard 1
| Concept | Owner | What it is |
|---|---|---|
| Document | OpenSearch | A JSON object you index. Has an _id, lives in one index, routed to one shard. |
| Index | OpenSearch | A named collection of documents with mappings (field types) and settings (shard count, etc.). |
| Shard (primary/replica) | OpenSearch | A horizontal slice of an index. The unit of distribution and replication. Each shard is a Lucene index. |
| Segment | Lucene | An immutable mini-index inside a shard. New docs go to new segments; merges combine them. |
| Mapping | OpenSearch | Field → type declaration (text, keyword, long, date, nested, …). Drives how Lucene fields are built. |
| Analyzer / Tokenizer / TokenFilter | mostly Lucene (configured by OpenSearch) | Turns text field values into terms ("Hello World" → [hello, world]). |
| Inverted index | Lucene | term → posting list (which docs contain the term). The core of full-text search. |
| DocValues | Lucene | Columnar per-field storage used for sorting and aggregations. |
| Translog | OpenSearch | A write-ahead log for durability between Lucene commits. Not a Lucene concept. |
| Cluster state / routing / allocation | OpenSearch | Where shards live, who is cluster manager, index metadata. Pure OpenSearch. |
Rule of thumb: anything about scoring, tokenization, segments, postings, DocValues is Lucene (OpenSearch configures it). Anything about shards across nodes, replication, cluster state, REST, transport, translog, aggregation reduce is OpenSearch. Keep this table near your desk.
Scenario 1: Log Analytics Ingest + Range/Term Search
What the user does — bulk-index some structured logs, then filter by status and time range.
# Bulk-index four log lines into an index "weblogs".
curl -s -H 'Content-Type: application/x-ndjson' \
-XPOST 'localhost:9200/weblogs/_bulk?refresh=true' --data-binary $'
{"index":{}}
{"@timestamp":"2026-06-16T10:00:00Z","status":200,"bytes":512,"path":"/home","client":"10.0.0.1"}
{"index":{}}
{"@timestamp":"2026-06-16T10:00:05Z","status":404,"bytes":128,"path":"/missing","client":"10.0.0.2"}
{"index":{}}
{"@timestamp":"2026-06-16T10:01:00Z","status":500,"bytes":256,"path":"/api","client":"10.0.0.1"}
{"index":{}}
{"@timestamp":"2026-06-16T10:02:30Z","status":200,"bytes":900,"path":"/home","client":"10.0.0.3"}
'
# Filter: 5xx errors in a time window. A "filter" context = no scoring, just yes/no.
curl -s -H 'Content-Type: application/json' 'localhost:9200/weblogs/_search?pretty' -d '
{
"query": {
"bool": {
"filter": [
{ "range": { "status": { "gte": 500 } } },
{ "range": { "@timestamp": { "gte": "2026-06-16T10:00:00Z", "lte": "2026-06-16T10:05:00Z" } } }
]
}
}
}'
You get back the one status:500 document. Notice "max_score": null — filter context does no
relevance scoring.
What OpenSearch does under the hood:
RestBulkActionparses the NDJSON body, builds aBulkRequest, and calls theNodeClient.TransportBulkActionauto-creates theweblogsindex (a cluster state update that adds index metadata and aRoutingTableentry), dynamically mapsstatus→long,@timestamp→date,path→text+keyword, then groups documents per target shard.- Each shard's writes go through
TransportShardBulkAction→IndexShard.applyIndexOperationOnPrimary(...)→InternalEngine.index(...)→ LuceneIndexWriter.addDocument(...)+Translog.add(...).?refresh=trueforces an immediate refresh so the docs are searchable. - The search request hits
RestSearchAction→TransportSearchAction. The coordinating node fans out to each shard; per shard,SearchService.executeQueryPhaseruns therangefilters as LucenePointRangeQueryover the numeric/date_doc_values/points, collecting matching doc IDs. FetchPhaseloads the stored_sourcefor the hits;SearchPhaseControllermerges shard results on the coordinating node.
Bridge to source code:
cd ~/src/OpenSearch
# The REST handler that accepts /_bulk
find server -name "RestBulkAction.java"
# server/src/main/java/org/opensearch/rest/action/document/RestBulkAction.java
# The transport action that splits a bulk by shard and creates indices/mappings
grep -rn "class TransportBulkAction" server/src/main/java/org/opensearch/action/bulk/
# Where a range filter becomes a Lucene query
grep -rn "PointRangeQuery\|newRangeQuery" server/src/main/java/org/opensearch/index/mapper/ | head
Scenario 2: Full-Text Relevance Query (match, BM25, _explain)
What the user does — index a few articles, then run a relevance-ranked match query and ask the
engine to explain the score.
curl -s -H 'Content-Type: application/x-ndjson' \
-XPOST 'localhost:9200/articles/_bulk?refresh=true' --data-binary $'
{"index":{"_id":"1"}}
{"title":"OpenSearch query performance tuning","body":"Tuning search relevance and query latency in OpenSearch."}
{"index":{"_id":"2"}}
{"title":"A gentle introduction to Lucene","body":"Lucene powers search; OpenSearch builds on it."}
{"index":{"_id":"3"}}
{"title":"Cooking with cast iron","body":"Nothing to do with search at all."}
'
# Relevance query in "query" context — scored by BM25 by default.
curl -s -H 'Content-Type: application/json' 'localhost:9200/articles/_search?pretty' -d '
{ "query": { "match": { "body": "search relevance" } } }'
# Ask WHY document 1 scored what it did.
curl -s -H 'Content-Type: application/json' 'localhost:9200/articles/_explain/1?pretty' -d '
{ "query": { "match": { "body": "search relevance" } } }'
The _explain response is a tree of BM25 components (termFreq, idf, tf, field length norm). The
top scoring doc is the one whose body best matches the analyzed terms [search, relevance].
What OpenSearch does under the hood:
- The
matchquery is aQueryBuilder(MatchQueryBuilder). At query time it runs the field's analyzer over"search relevance"to produce terms, then builds a LuceneBooleanQueryofTermQueryclauses viaQueryShardContext. - Per shard,
QueryPhaseexecutes the Lucene query; Lucene scores each matching doc with BM25 (the defaultSimilarity), using term frequency, inverse document frequency, and field-length normalization — all computed inside Lucene, not OpenSearch. - The coordinating node merges the top-K from each shard in
SearchPhaseController;FetchPhaseloads_sourcefor the survivors. _explainruns the same query against one doc and returns Lucene'sExplanationtree verbatim.
Bridge to source code:
# The match query builder
find server -name "MatchQueryBuilder.java"
# server/src/main/java/org/opensearch/index/query/MatchQueryBuilder.java
# Where QueryBuilders turn into Lucene Query objects
grep -rn "QueryShardContext\|toQuery" server/src/main/java/org/opensearch/index/query/AbstractQueryBuilder.java | head
# The query phase itself
find server -name "QueryPhase.java"
# server/src/main/java/org/opensearch/search/query/QueryPhase.java
# BM25 / Similarity wiring (OpenSearch configures Lucene's Similarity)
grep -rn "BM25\|Similarity" server/src/main/java/org/opensearch/index/similarity/ | head
Scenario 3: Aggregations (date_histogram + terms sub-agg) — The Dashboards Workload
What the user does — the canonical "log volume over time, broken down by status" dashboard panel.
curl -s -H 'Content-Type: application/json' 'localhost:9200/weblogs/_search?pretty' -d '
{
"size": 0,
"aggs": {
"per_minute": {
"date_histogram": { "field": "@timestamp", "fixed_interval": "1m" },
"aggs": {
"by_status": { "terms": { "field": "status" } }
}
}
}
}'
"size": 0 means "no hits, just the aggregation". The result is buckets per minute, each with a
nested breakdown by status. This single request shape is what powers a huge fraction of OpenSearch
Dashboards visualizations.
What OpenSearch does under the hood:
RestSearchActionparses theaggsblock into anAggregatorFactoriestree:DateHistogramAggregationBuilderwith a childTermsAggregationBuilder.- Per shard, each
AggregatorFactoryproduces anAggregator. The aggregators iterate matching docs reading the@timestampandstatusvalues from DocValues (columnar, not the inverted index), bucketing as they go. This is why aggregations needdoc_values(on by default for numerics/dates andkeyword). - Each shard returns a partial
InternalAggregation. The coordinating node callsInternalAggregation.reduce(...)to merge partial buckets into the final result — this reduce step is where shard-local partial results become a globally correct answer.
Bridge to source code:
# Date histogram aggregation
grep -rln "class DateHistogramAggregator" server/src/main/java/org/opensearch/search/aggregations/
# Terms aggregation
grep -rln "class TermsAggregator" server/src/main/java/org/opensearch/search/aggregations/bucket/terms/
# The reduce contract — how partial shard aggs merge on the coordinating node
grep -rn "public InternalAggregation reduce" server/src/main/java/org/opensearch/search/aggregations/ | head
# Aggregations read DocValues, not the inverted index
grep -rn "getLeafCollector\|LeafBucketCollector" server/src/main/java/org/opensearch/search/aggregations/AggregatorBase.java | head
See the Aggregations deep dive and the DocValues and Fielddata deep dive for the full mechanics.
Scenario 4: Multi-Node Cluster — Replicas, Health, Killing a Node
What the user does — bring up a 3-node cluster, create an index with replicas, check health, then kill a node and watch shards reallocate. This is the heart of OpenSearch's distributed nature.
# Run a 3-node local cluster from source instead of the single-node default.
./gradlew run -PnumNodes=3
# Create an index with 2 primaries and 1 replica each (4 shards total).
curl -s -H 'Content-Type: application/json' -XPUT 'localhost:9200/inventory' -d '
{ "settings": { "number_of_shards": 2, "number_of_replicas": 1 } }'
# Cluster health: should be green (every replica has a home).
curl -s 'localhost:9200/_cluster/health?pretty'
# See exactly where each shard lives and its state (STARTED / RELOCATING / UNASSIGNED).
curl -s 'localhost:9200/_cat/shards/inventory?v'
curl -s 'localhost:9200/_cat/nodes?v'
Now kill one node process (find it from _cat/nodes or your process list) and immediately re-check:
# In the ./gradlew run output, identify a data node's PID; kill it. Then:
curl -s 'localhost:9200/_cluster/health?pretty' # status flips to yellow, then back
curl -s 'localhost:9200/_cat/shards/inventory?v' # watch UNASSIGNED → INITIALIZING → STARTED
You will see health go yellow (a replica lost its home), the cluster manager re-elect if you
killed it, and the AllocationService promote a surviving replica to primary and rebuild the
missing copy elsewhere — eventually back to green.
What OpenSearch does under the hood:
- The elected cluster manager (formerly master) owns the authoritative
ClusterState. When a node dies,FollowersChecker/LeaderCheckerdetect it and the cluster manager publishes a newClusterStateremoving that node fromDiscoveryNodes. AllocationService+BalancedShardsAllocator, gated byAllocationDeciders(SameShardAllocationDecider,DiskThresholdDecider,MaxRetryAllocationDecider, …), recompute theRoutingTable: promote a replica to primary, allocate a new replica.- The new state is published two-phase (publish → commit) from the cluster manager to all nodes
via
PublicationTransportHandler; each node'sClusterApplierServiceapplies it locally. - Recovery of the rebuilt replica runs through
PeerRecoverySourceService/PeerRecoveryTargetService, tracked by sequence numbers (LocalCheckpointTracker, global checkpoint viaReplicationTracker).
Bridge to source code:
# The coordination layer: election, joins, publishing
ls server/src/main/java/org/opensearch/cluster/coordination/
grep -rln "class Coordinator" server/src/main/java/org/opensearch/cluster/coordination/
# The allocation engine and the deciders that gate it
grep -rln "class AllocationService" server/src/main/java/org/opensearch/cluster/routing/allocation/
ls server/src/main/java/org/opensearch/cluster/routing/allocation/decider/
# Where a node-left event triggers reroute
grep -rn "disassociateDeadNodes\|reroute" server/src/main/java/org/opensearch/cluster/routing/allocation/AllocationService.java | head
The Discovery and Coordination, Shard Allocation, and Recovery deep dives cover this in full. Level 4 is built entirely around this scenario.
Scenario 5: Snapshot to a Filesystem Repository and Restore
What the user does — register a filesystem snapshot repository, snapshot an index, delete it, and restore from the snapshot. This is the backup/restore and disaster-recovery story.
# ./gradlew run permits the build's temp dir as a snapshot path by default; if not,
# add path.repo to the run config. Register an "fs" repository:
curl -s -H 'Content-Type: application/json' -XPUT 'localhost:9200/_snapshot/my_fs_repo' -d '
{ "type": "fs", "settings": { "location": "my_backup_dir" } }'
# Take a snapshot of the "weblogs" index, waiting for it to finish.
curl -s -XPUT 'localhost:9200/_snapshot/my_fs_repo/snap-1?wait_for_completion=true&pretty' \
-H 'Content-Type: application/json' -d '{ "indices": "weblogs" }'
# Destroy the index, then restore it from the snapshot.
curl -s -XDELETE 'localhost:9200/weblogs'
curl -s -XPOST 'localhost:9200/_snapshot/my_fs_repo/snap-1/_restore?wait_for_completion=true&pretty'
# Confirm the data is back.
curl -s 'localhost:9200/weblogs/_count?pretty'
What OpenSearch does under the hood:
RepositoriesServiceregisters thefsrepository (FsRepository, aBlobStoreRepository).SnapshotsServiceruns a snapshot as a series of cluster-state transitions: it copies Lucene segment files (incrementally — only segments not already in the repo) plus index metadata into the blob store. Snapshots are incremental at the segment level.- Restore is the inverse:
RestoreServicebuilds new shards from the snapshotted segments and recovers them (a special recovery source), then theAllocationServiceallocates them.
Bridge to source code:
# Snapshot and repository services
grep -rln "class SnapshotsService" server/src/main/java/org/opensearch/snapshots/
grep -rln "class RepositoriesService" server/src/main/java/org/opensearch/repositories/
grep -rln "class BlobStoreRepository" server/src/main/java/org/opensearch/repositories/blobstore/
# The fs repository implementation
find server modules plugins -name "FsRepository.java"
# Restore is recovery from a snapshot source
grep -rln "class RestoreService" server/src/main/java/org/opensearch/snapshots/
See the Snapshots and Repositories deep dive.
Dataset Scenarios for Testing Edge Cases
When you write a repro or validate a fix, the dataset you choose determines which code path you exercise. Use these as starting templates; each names the subsystem it stresses.
Dataset 1: The Empty Index
curl -s -XPUT 'localhost:9200/empty_idx'
curl -s 'localhost:9200/empty_idx/_search?pretty' # 0 hits, must NOT error
curl -s 'localhost:9200/empty_idx/_search?pretty' -H 'Content-Type: application/json' \
-d '{ "aggs": { "s": { "sum": { "field": "missing" } } } }'
What this tests: the coordinating-node reduce path with zero shard results, and aggregations
over a field that does not exist. SearchPhaseController.reducedQueryPhase(...) and
InternalAggregation.reduce(...) must produce an empty-but-valid response, not an NPE. Historically a
rich source of "empty result" bugs.
Source: server/src/main/java/org/opensearch/action/search/SearchPhaseController.java.
Dataset 2: A Single Huge Document
# One ~5 MB document — stresses source storage, field length, and the http.max_content_length limit.
python3 - <<'PY' > /tmp/huge.json
import json
print(json.dumps({"blob": "x" * 5_000_000, "n": 1}))
PY
curl -s -XPOST 'localhost:9200/huge/_doc?refresh=true' -H 'Content-Type: application/json' --data-binary @/tmp/huge.json
What this tests: _source storage, the http.max_content_length guard in the REST/HTTP layer,
and Lucene stored-field limits. Exercises RestController request-size handling and
IndexShard/InternalEngine for a single oversized doc.
Source: server/src/main/java/org/opensearch/rest/RestController.java and the HTTP transport in
modules/transport-netty4.
Dataset 3: Deeply Nested Objects
curl -s -XPUT 'localhost:9200/nested_idx' -H 'Content-Type: application/json' -d '
{ "mappings": { "properties": { "user": { "type": "nested",
"properties": { "comments": { "type": "nested" } } } } } }'
What this tests: the nested field type and the index.mapping.nested_objects.limit /
index.mapping.depth.limit guards. Nested docs are stored as hidden child Lucene documents, so
this exercises ObjectMapper/NestedObjectMapper and the join-based nested query.
Source: server/src/main/java/org/opensearch/index/mapper/ObjectMapper.java and
NestedObjectMapper.
Dataset 4: High-Cardinality Terms Field
# Index many unique keyword values, then a terms agg over them.
for i in $(seq 1 5000); do
printf '{"index":{}}\n{"uid":"user-%s"}\n' "$i"
done | curl -s -H 'Content-Type: application/x-ndjson' \
-XPOST 'localhost:9200/hicard/_bulk?refresh=true' --data-binary @-
curl -s 'localhost:9200/hicard/_search?pretty' -H 'Content-Type: application/json' \
-d '{ "size": 0, "aggs": { "u": { "terms": { "field": "uid", "size": 10 } } } }'
What this tests: the terms aggregator's bucket collection over a high-cardinality field — memory
pressure, the request circuit breaker, and shard-level vs. coordinating-node accuracy
(doc_count_error_upper_bound). This is a classic OOM/breaker-trip path.
Source: server/src/main/java/org/opensearch/search/aggregations/bucket/terms/ and
HierarchyCircuitBreakerService.
Dataset 5: Many Tiny Shards
# 50 shards for a trivially small index — a real anti-pattern users hit.
curl -s -XPUT 'localhost:9200/oversharded' -H 'Content-Type: application/json' \
-d '{ "settings": { "number_of_shards": 50, "number_of_replicas": 0 } }'
curl -s 'localhost:9200/_cat/shards/oversharded?v'
curl -s 'localhost:9200/_cluster/health/oversharded?pretty'
What this tests: AllocationService/BalancedShardsAllocator placing many shards, cluster-state
size growth (50 routing entries), and the per-shard search fan-out overhead in TransportSearchAction
(50 shard requests for almost no data).
Source: server/src/main/java/org/opensearch/cluster/routing/allocation/allocator/BalancedShardsAllocator.java
and org/opensearch/action/search/TransportSearchAction.java.
Dataset 6: Time-Series With Rollover
# Create a write alias backed by a dated index, then roll it over on a size/age condition.
curl -s -XPUT 'localhost:9200/ts-000001' -H 'Content-Type: application/json' \
-d '{ "aliases": { "ts-write": { "is_write_index": true } } }'
curl -s -XPOST 'localhost:9200/ts-write/_rollover?pretty' -H 'Content-Type: application/json' \
-d '{ "conditions": { "max_docs": 1, "max_age": "7d" } }'
curl -s 'localhost:9200/_cat/indices/ts-*?v'
What this tests: the rollover path (alias → new backing index), index lifecycle thinking, and the
metadata mutations that create a new index as a cluster-state update task.
Source: server/src/main/java/org/opensearch/action/admin/indices/rollover/ and
MetadataRolloverService.
The Bridge: User Scenario → Source Code (Master Table)
Use this whenever you observe a runtime behavior and want to find the code that owns it. Exact class
names can vary by branch — when in doubt, grep for the class name under server/src/main/java.
| Observed behavior | Owning subsystem / class |
|---|---|
HTTP request hits :9200 and is routed | RestController → a RestHandler (RestSearchAction, RestBulkAction, RestIndexAction) |
| A REST handler dispatches to server logic | NodeClient.execute(ActionType, request, listener) → a TransportAction (wired in ActionModule) |
| A search fans out to shards and merges | TransportSearchAction → per-shard SearchService.executeQueryPhase/executeFetchPhase; merge in SearchPhaseController |
| The query phase runs the Lucene query | QueryPhase, with SearchContext/DefaultSearchContext per shard |
The fetch phase loads _source for hits | FetchPhase |
| Global term stats across shards (optional) | DfsPhase |
| An aggregation is computed and reduced | AggregatorFactory → Aggregator → InternalAggregation.reduce(...) |
| A document is indexed on the primary | TransportBulkAction/TransportShardBulkAction → IndexShard.applyIndexOperationOnPrimary → InternalEngine.index |
| A write is replicated to replicas | TransportReplicationAction (document replication) or SegmentReplicationTargetService (segment replication) |
| A doc becomes searchable / durable | refresh (new searcher) / Translog + flush (Lucene commit) |
| A request must run on the cluster manager | TransportClusterManagerNodeAction (formerly TransportMasterNodeAction) |
| Cluster metadata changes | ClusterStateUpdateTask/ClusterStateTaskExecutor on the cluster-manager service |
| A new cluster state is distributed | MasterService computes it; PublicationTransportHandler publishes; ClusterApplierService applies |
| Leader election / node failure detection | Coordinator, PreVoteCollector, ElectionSchedulerFactory, FollowersChecker, LeaderChecker |
| Shards placed / rebalanced | AllocationService + BalancedShardsAllocator, gated by AllocationDeciders |
| A replica rebuilt after a node dies | PeerRecoverySourceService/PeerRecoveryTargetService, ReplicationTracker |
| Snapshot/restore | SnapshotsService, RepositoriesService, BlobStoreRepository, RestoreService |
| Cross-node wire serialization | Writeable + StreamInput/StreamOutput, NamedWriteableRegistry, over TransportService/Netty4Transport |
| Memory protection trips | HierarchyCircuitBreakerService (parent, fielddata, request, in_flight_requests) |
Each row has a deep dive: REST layer, Action framework, Transport layer, Search execution, Engine internals, Cluster state, Cluster state publishing, Shard allocation, Recovery, Circuit breakers and memory.
Running OpenSearch End-to-End: The Local Developer Loop
Every contributor should be able to do this loop in under ten minutes, from cold:
cd ~/src/OpenSearch
# 1. Launch a local cluster from source (REST on :9200). Leave running.
./gradlew run
# 2. The canonical health check (separate terminal).
curl -s 'localhost:9200/_cluster/health?pretty'
# 3. Smoke test: index a doc and read it back.
curl -s -XPOST 'localhost:9200/sanity/_doc/1?refresh=true' \
-H 'Content-Type: application/json' -d '{"hello":"world"}'
curl -s 'localhost:9200/sanity/_doc/1?pretty'
curl -s 'localhost:9200/sanity/_search?pretty' -d '{"query":{"match_all":{}}}' \
-H 'Content-Type: application/json'
# 4. The fast inner loop you will live in: run one test class / one method.
./gradlew :server:test --tests "org.opensearch.search.query.QueryPhaseTests"
./gradlew :server:test --tests "org.opensearch.cluster.ClusterStateTests.testToXContent"
# 5. Multi-node integration tests (in-JVM, backed by InternalTestCluster).
./gradlew :server:internalClusterTest --tests "org.opensearch.cluster.*"
Your baseline health check is: ./gradlew run serves :9200, a known-good :server:test class
passes, and a trivial index+search round-trips. If any of those fail before you have changed a single
line, stop and fix your environment — do not start Level 1 on a broken baseline, because every
later ./gradlew test will produce false failures that hide the real work.
What to Verify Before Starting Level 1
Run through this checklist once. It takes 30–45 minutes and proves your environment and your understanding.
# Environment
java -version # JDK 21
cd ~/src/OpenSearch
./gradlew --version | head -3 # wrapper healthy
# Build + run
./gradlew localDistro # archive under distribution/archives/
./gradlew run # serves :9200 (leave running, new terminal below)
# Functional round-trip
curl -s 'localhost:9200/_cluster/health?pretty' # green/yellow
curl -s -XPOST 'localhost:9200/check/_doc/1?refresh=true' \
-H 'Content-Type: application/json' -d '{"k":"v"}'
curl -s 'localhost:9200/check/_search?pretty' # 1 hit
# Tests
./gradlew :server:test --tests "org.opensearch.cluster.ClusterStateTests" 2>&1 | tail -5
You are ready when, without notes, you can:
- State the OpenSearch-vs-Lucene boundary for: scoring, segments, DocValues, translog, cluster state, REST/transport.
- Name the four steps of the search path: REST handler → transport action → query phase → fetch phase, plus the coordinating-node reduce.
-
Name the indexing path: bulk/index action →
IndexShard→InternalEngine→ LuceneIndexWriter+ translog. -
Explain why killing a node turns the cluster yellow and how it returns to green
(cluster manager → new
ClusterState→AllocationService→ recovery). -
Run a single
./gradlew :server:test --tests ...and a singlecurlsearch from memory. -
Open
RestSearchAction,TransportSearchAction,QueryPhase, andIndexShardin your IDE via Go to Class.
If any box is unchecked, re-run the scenario it maps to before moving on.
Continue to the 16-Week Plan and Milestones, or jump to Level 1: Lucene and OpenSearch Foundation. The internals you previewed here are covered in full in the Deep Dives.