Feature Masterclasses: Deep Intensives

The deep-dives teach you the system — how a search fans out, how a shard recovers, how the engine flushes. The lucene and knn sections teach you the formats and algorithms underneath. The masterclasses do something narrower and harder: they take one feature, nail it down to the algorithm, the data structures, and the exact classes that implement it, and then make you build and measure it in three or four full labs.

A masterclass is a deep intensive. Where a deep-dive chapter already covers a topic well, the masterclass extends it — it links to the chapter, assumes you read it, and then goes a level deeper: byte-level layout, line-level code reading (via grep, because line numbers drift between versions), the heuristic constants, the failure modes, and a worked numeric example you can reproduce. The bar is: a contributor who finishes a masterclass can open a real issue in that feature's area and have an informed opinion about the fix — not just "it's slow," but "the slice reduce isn't associative, here's the test that proves it."

Note: "Masterclass" is a teaching format, not a code module. Everything you learn maps to real org.opensearch.* / org.apache.lucene.* code you can check out, grep, build with ./gradlew, and run.

What makes a masterclass different from a deep-dive

Each masterclass has the same shape:

1. A concept file (index.md, 450–750 lines). The feature from first principles: the problem, the data structures, the algorithm, the OpenSearch and Lucene classes, real Java excerpts, two or three mermaid diagrams, a worked numeric example, a trade-offs table, a "Common bugs and symptoms" table, and a "Validation" self-check.
2. Three or four labs (250–520 lines each). Full runnable labs with the standard skeleton — Background, Why This Matters for Contributors, Prerequisites, numbered Step-by-Step Tasks, Deliverables, Troubleshooting, Expected Output, Stretch Goals, Validation. Real curl sessions, real Java you compile and run, real grep/find/pmap/xxd against a checkout.

The labs in a masterclass follow a deliberate arc: an observe lab (turn the feature on and watch it work — profile, stats, hex dumps), a build lab (write a minimal standalone version of the mechanism so you own the contract), and a measure lab (benchmark it, find the regime where the default is right or wrong). For Concurrent Segment Search that's CS1 (profile slices), CS2 (write a CollectorManager), CS3 (benchmark the crossover). Once you've done one masterclass you know the rhythm of all five.

Note: Don't skip the build lab because it "looks like toy code." Writing the mechanism standalone — a CollectorManager, an FST, a murmur3 router — is what converts "I read about it" into "I could fix it." The real OpenSearch class is the same contract with production concerns bolted on.

The five masterclasses

Reading order

The masterclasses are independent — you can take them in any order that matches what you're working on. But they form a natural progression, and if you're going to do all five, this order builds each on the last:

flowchart TD
    A["Lucene On-Disk Data Structures<br/>(the bytes everything reads)"] --> B["The Storage Engine<br/>(how those bytes live on disk + durability)"]
    B --> C["Sharding & Routing<br/>(how docs map to shards/segments)"]
    C --> D["Concurrent Segment Search<br/>(parallelizing the read over segments)"]
    A --> E["Vectorization & Embeddings<br/>(the vector files + SIMD + the ML path)"]
    D --> E

• Start with Lucene On-Disk Data Structures if you want to understand the thing every other feature manipulates. A slice scans segments; a segment is these files. A split moves docs between shards; a shard is these files.
• Then The Storage Engine to see how those files live on disk, how mmap exposes them to the page cache, and what makes a write durable.
• Sharding & Routing sits on top: which shard (and therefore which pile of segments) a document lands in, and how you split/shrink the layout.
• Concurrent Segment Search is the read-path intensive: now that you know what a segment is and how a shard is laid out, parallelize the scan over it.
• Vectorization & Embeddings is the capstone of the set — it touches the vector files (data structures), the SIMD hot loop (CPU), and the whole ML ingest/query pipeline (the system).

Note: If you're here for a specific contribution, skip the order and go straight to the masterclass that owns your feature. Each index.md lists its prerequisites; backfill only what you're missing.

Prerequisites for the whole set

You should be comfortable with the material in the early levels and the relevant deep-dives before starting a masterclass. Concretely:

• You can check out OpenSearch, build it, and run a node with ./gradlew run (see Level 2: Building from Source).
• You can read Java fluently and write a small standalone program with a classpath (the labs hand you the exact javac/java invocations).
• You understand the search execution query-then-fetch model and what a segment is — these two are load-bearing across most of the set.
• You can drive a cluster with curl and read JSON responses, including _cat APIs and _search?profile=true.

Each masterclass narrows this to a feature-specific prerequisite list. None of them assume you've done the others first (the reading order is a recommendation, not a hard dependency) except where a lab explicitly links back.

How masterclasses feed the capstone projects

The capstone projects are open-ended, real contributions — the kind of thing that becomes an actual PR. A masterclass is the training that earns you the right to attempt one. The mapping is direct:

The relationship is concrete, not aspirational. Project 3, for example, asks you to find a regime where the slicing heuristic is suboptimal and improve it with a measurement gate. The Concurrent Segment Search masterclass is exactly where you learn (a) what the heuristic is and where it lives in code, (b) how to read per-slice profile timings, and (c) how to benchmark concurrent vs sequential and find the crossover. By the time you finish Lab CS3, you have the benchmark harness and the data the capstone demands you bring.

Warning: Do not start a capstone project without the corresponding masterclass (or equivalent experience). The capstones assume you can already read the feature's code, run its tests, and measure it. The masterclass is where you build those reflexes on a known-good path before you go off-road.

A note on grep over line numbers

Throughout the masterclasses you'll see grep/find commands instead of "open file X line 412." That's deliberate. OpenSearch and Lucene move fast; line numbers are stale the day they're written. A grep for a class name or a constant finds the real site in your checkout, whatever version you're on. When a masterclass says "the default is about 250,000 docs per slice — grep to confirm," it means: the number is real, but you verify it against your code, not against this page. That habit — trust the source tree, not the doc — is the single most important thing the masterclasses teach.

What you walk away with

A finished masterclass leaves you with four concrete things, not just a warm feeling:

• A mental model precise enough to predict. You can say, before running it, how many slices a given segment topology produces, or which shard a given _id routes to — and then confirm it.
• A reproducible experiment. Each masterclass ends in a measurement you can re-run: a benchmark table, a hex dump you can diff, a recovery you can trigger. That artifact is what you attach to an issue or PR.
• A grep map of the feature. You know the handful of class names that own the behavior and can find them in any version of the tree. That's the difference between "I'd have to go read the code" and "it's in ContextIndexSearcher, here's the method."
• An informed opinion. When someone proposes a change to the feature, you can evaluate it against the trade-offs you measured yourself, not against documentation you half-remember.

If you finish a masterclass and can't do those four things for its feature, you skipped a lab. Go back and do the build lab and the measure lab — they're where the durable understanding lives.

Begin with the Concurrent Segment Search Intensive, or jump to the masterclass that owns the feature you're working on.