Deep Dives: Reading Order

This directory contains 24 deep-dive chapters. They are the reference material behind the Level curriculum. Each chapter is self-contained, but most chapters depend on a handful of earlier ones. Read in the order below the first time through; thereafter use the index as a lookup.

OpenSearch is a distributed search and analytics engine built on Apache Lucene, forked from Elasticsearch 7.10.2. Almost everything you read lives under server/src/main/java/org/opensearch/..., with low-level primitives in libs/ and bundled extensions in modules/. The chapters below mirror the request path and the cluster lifecycle: you build up "what a cluster is" before "how requests flow" before "how a shard stores and searches data."

The chapters are grouped by subsystem. For each chapter we list:

• Title — the file.
• One-line summary — what you should walk away knowing.
• Consumed by — which Levels/Labs depend on it.

Note: Throughout the book, "cluster manager" is the current term for what Elasticsearch (and older OpenSearch) called "master." OpenSearch renamed the role and most settings for inclusive language; the old master names survive as deprecated aliases. Each chapter notes both terms on first use.

Group 1 — Cluster and Node Model

These four chapters define "what is an OpenSearch cluster" — the nodes, the shared state object they agree on, and how that agreement is reached and distributed — before any request-handling machinery exists.

Start here. Without the cluster/node/state model in your head, every later chapter feels like trivia.

Group 2 — Transport and Actions

How a request becomes work. These chapters explain the layered request path: HTTP at the edge, an internal RPC fabric between nodes, an action registry that ties request types to handlers, and the thread pools that run all of it.

These chapters explain how a request is dispatched and executed. They must precede the storage and search chapters, which assume you know the action and thread-pool model.

Group 3 — Indexing and Storage

The write path, end to end, inside a single shard: the shard lifecycle wrapper, the Lucene-backed engine, the write-ahead log, the schema/analysis layer, and the three operations (refresh, flush, merge) that govern visibility, durability, and segment count.

If you skip chapter 9, every later storage chapter will reference a shard state machine you have not seen. Read it first.

Group 4 — Search and Aggregations

The read path: how a query fans out across shards, how QueryBuilders become Lucene Querys, how aggregations build and reduce, and the columnar data structures that sorting and aggregations actually read.

Read 14 before 16 — an aggregation only makes sense once you understand the query/fetch phase split it runs inside.

Group 5 — Allocation, Recovery, Replication

How shards get placed on nodes, how a placed-but-empty shard is filled with data, and how primary and replica stay in sync after they are both running.

Group 6 — Cross-cutting

The subsystems that touch every other chapter: backups, memory safety, the extension model, and wire compatibility across versions.

How the groups depend on each other

The six groups are not a flat list; they stack. Group 1 is bedrock, Groups 2–5 each assume the groups beneath them, and Group 6 reaches into all of them. Read this as "what must already be in your head before this group makes sense":

flowchart TD
    G1["Group 1<br/>Cluster & Node Model<br/>(ch 1–4)"]
    G2["Group 2<br/>Transport & Actions<br/>(ch 5–8)"]
    G3["Group 3<br/>Indexing & Storage<br/>(ch 9–13)"]
    G4["Group 4<br/>Search & Aggregations<br/>(ch 14–17)"]
    G5["Group 5<br/>Allocation, Recovery, Replication<br/>(ch 18–20)"]
    G6["Group 6<br/>Cross-cutting<br/>(ch 21–24)"]
    G1 --> G2
    G1 --> G3
    G1 --> G5
    G2 --> G3
    G2 --> G4
    G3 --> G4
    G3 --> G5
    G4 --> G6
    G5 --> G6
    G3 --> G6
    G1 --> G6

The two edges worth internalizing: Group 3 (storage) depends on Group 2 because a write only reaches a shard after the action framework and thread pools have dispatched it; and Group 4 (search) depends on Group 3 because a query runs against the exact segments the write path produced. If either edge feels backwards, you have read the groups out of order.

Tracing a request through the chapters

The fastest way to see why the order above is the order is to follow one request across the chapters, in the order it visits them. Do this once for a write and once for a read; every chapter earns its place.

An indexing request: PUT /orders/_doc/1

1. The HTTP request lands in the REST layer. RestController matches the route to RestIndexAction, which parses the body and hands off through the NodeClient — rest-layer.md.
2. The request becomes an action. A single-document index is a one-item bulk, so it enters TransportBulkAction — an ActionType with a TransportAction handler registered in ActionModule — action-framework.md. The work runs on the write thread pool, not the caller's thread — threadpools-concurrency.md.
3. To route the doc to a shard, the coordinating node reads the RoutingTable out of the current ClusterState — cluster-state.md — to learn which node holds the primary. That state exists only because the cluster agreed on it (discovery-coordination.md) and published it (cluster-state-publishing.md), over the node model in cluster-and-node-model.md.
4. The shard-level request travels to the primary's node over the internal transport (port 9300) as TransportShardBulkAction — transport-layer.md.
5. On the primary, IndexShard.applyIndexOperationOnPrimary receives the op — index-shard-lifecycle.md.
6. The MapperService maps the JSON to fields and the analysis chain tokenizes the text — mapping-and-analysis.md.
7. InternalEngine.index writes to the Lucene IndexWriter and the LiveVersionMap, assigning a sequence number — engine-internals.md.
8. Before the response returns, the op is appended to the Translog for durability — translog.md.
9. The primary fans the op out to replicas via TransportReplicationAction, advancing the global checkpoint in the ReplicationTracker — replication.md.
10. The document is not searchable yet. It becomes visible at the next refresh, is made durable to a Lucene commit at the next flush, and its segment is later consolidated by a merge — refresh-flush-merge.md.

That single write touched Groups 1, 2, 3, and 5 — which is exactly why you read them in that order.

A search request: GET /orders/_search

1. RestController routes to RestSearchAction, which parses the query body — rest-layer.md.
2. It becomes TransportSearchAction, dispatched on the search thread pool — action-framework.md, threadpools-concurrency.md.
3. The coordinating node reads the RoutingTable (cluster-state.md) to pick one copy of each shard, then scatters a per-shard request and gathers the results — search-execution.md.
4. On each shard, the QueryBuilder is rewritten to a Lucene Query through the QueryShardContext — query-dsl-querybuilders.md — and run against the very segments the write path produced (engine-internals.md, refresh-flush-merge.md).
5. If the request aggregates, the Aggregator builds buckets in the query phase, reading columnar DocValues for its keys and metrics — aggregations.md, docvalues-fielddata.md. Heap spent here is charged to the request circuit breaker — circuit-breakers-memory.md.
6. Back on the coordinator, SearchPhaseController reduces the per-shard results, and the fetch phase pulls the actual _source/fields for the top hits — search-execution.md.

The read touched Groups 1, 2, 4 (and leaned on Group 3's output) — the mirror image of the write, and the reason Group 4 comes after Group 3.

Note: Both traces cross a node boundary at exactly one point — step 4 of the write (coordinator→primary) and step 3 of the read (coordinator→data shards). That hop is always the transport layer. When you cannot tell whether a bug is "coordination" or "shard-local," find that hop first; it is the seam between Group 2 and Groups 3–4.

A note on order vs index

The deep-dives are an index — they exist to be looked up later. The first read should follow the table above, top to bottom: cluster model, then request path, then storage, then search, then allocation/recovery/replication, then the cross-cutting subsystems. Each group assumes the previous ones.

But when you return to fix a bug, do not re-read linearly. Jump directly to the chapter most relevant to the failing component and use the cross-references inside it. A shard-stuck-UNASSIGNED bug starts in shard-allocation.md; a "documents not visible after index" bug starts in refresh-flush-merge.md; a "node won't join the cluster" bug starts in discovery-coordination.md.

Every chapter ends with a Validation: prove you understand this section. Treat that as the gate before declaring the chapter "read." If you cannot answer the validation questions from memory plus a grep, you have skimmed, not read.