Engineering at Scale: Real RFCs & Hard Problems
Everything before this section was about becoming a contributor: build from source, trace a request, reproduce an issue, write a fix and a test, open a quality PR. That is depth 2 from the Hitchhiker's Guide — and it is a genuine, hireable skill. This section is about the jump to depth 3: driving a hard, cross-cutting change the way maintainers do.
The difference is not "harder bugs". It is kind. A good-first-issue is bounded: one file, one behavior, one test. The work in this section is unbounded by default — concurrent segment search touches the search threadpool, the collector contract, settings, defaults, and BWC simultaneously; reader/writer separation rewrites how routing, allocation, replication, and remote storage interact. You cannot land changes like these by reading one class. You land them the way the maintainers did: with a public design document, an explicit problem statement, measured trade-offs, and a sequence of PRs that each leave the system shippable.
So this section is built around the real RFCs that produced these features. You will read the actual GitHub issues, follow the design from proposal to merged code, and learn the subsystem well enough to extend it. The facts here are verified and the issue numbers are real — cite them, read them, and treat them as the canonical record of how serious OpenSearch engineering actually happens.
How to Read an OpenSearch RFC
OpenSearch design happens in the open, on GitHub, and it has a recognizable grammar. Learn the grammar and a 200-comment issue becomes navigable.
The label vocabulary
| Label | Means | What to expect inside |
|---|---|---|
RFC | "Request for Comments" — a proposal opened for community feedback before the design is locked. | A problem statement, one or more proposed approaches, open questions, and a discussion thread. The most valuable thing to read. |
META | An umbrella issue tracking a multi-PR initiative. | A checklist of sub-issues and PRs, a status table, links out to the RFC and design docs. Your map of the whole effort. |
proposal / [Proposal] | A concrete design, often following an accepted RFC. | The chosen approach in detail: interfaces, settings, phasing, BWC plan. |
enhancement | A scoped feature or improvement. | Smaller than an RFC; often the unit a single PR delivers. |
Roadmap | Tracked on the project roadmap board. | High-level intent; links down to RFCs/METAs. |
good first issue / help wanted | Explicitly open for new contributors. | A bounded task, sometimes with a starting pointer. Your on-ramp into a theme. |
The lifecycle
flowchart LR
Issue["issue: a real problem / pain point"] --> RFC["[RFC]: proposed approaches, community feedback"]
RFC --> Review["community-meeting review + maintainer buy-in"]
Review --> Design["[Proposal]/design: chosen interfaces, settings, phasing"]
Design --> META["[META]: checklist of sub-issues + PRs"]
META --> PRs["PRs: each leaves the system shippable"]
PRs --> Flag["feature flag / experimental setting"]
Flag --> Default["promoted to default in a release (e.g. 3.0)"]
Read an initiative in that order, not chronologically by comment. Start at the META to see the shape, drop into the RFC for the why and the rejected alternatives, skim the Proposal for the what, then read the PRs for the how. Big features land behind a feature flag or experimental setting first, soak, and only later flip to default — concurrent segment search becoming default-on in 3.0 is the canonical example.
Note: Many of these designs were reviewed live in an OpenSearch community meeting. When an RFC says "discussed in the community meeting", that is where the real decision was socialized. You will not see those minutes in the issue, but the outcome always lands back as a comment. Read for the resolution, not just the debate.
The Themes Covered
Each chapter takes one theme from problem statement to code, anchored on a real meta/RFC issue. Read them as a progression from "search a shard faster" up to "re-architect how reads and writes scale".
| Theme | Chapter | The hard problem | Anchor issue |
|---|---|---|---|
| Vector search & vectorization | the Lucene + k-NN sections; cataloged here | ANN recall vs latency vs memory; SIMD; GPU index build. | k-NN #2605 (META: new vector engine), k-NN #2293 (RFC: GPU) |
| Concurrent search | Concurrent Segment Search | Search a shard's segments in parallel without breaking the collector contract. | search threadpool + CollectorManager; default-on in 3.0 |
| Sharding / scaling | Sharding, Scaling & Reader/Writer Separation | Scale reads independently of writes; scale to zero. | #7258 (RFC), #15306 (META) |
| Durability / remote store | Remote-Backed Storage & Durability | Decouple durability from replica count using an object store. | underpins RW-separation; pairs with segment replication |
| Distributed load / backpressure | Backpressure & Admission Control | Reject smartly per-shard to prevent cascading failure. | #1446 (META), #478 (META) |
| Aggregation acceleration | Star-Tree Indexes | Precompute multi-field aggregations with bounded latency. | #12498 (RFC), #13875 (META) |
| Caching | Tiered Caching | Spill the request cache to disk when heap is exhausted. | #10024 (Proposal), #11464 (META: benchmark) |
The full, link-dense catalog of every real issue, RFC, and PR — grouped by theme, with a "how to start" for each — is the next chapter: Real Issues, RFCs, and Where to Contribute. That is the single highest-value page in this section if your goal is to contribute.
Reading Order
- Real Issues, RFCs, and Where to Contribute — the catalog. Read it first to see the whole landscape and pick a theme that excites you. Bookmark it; you will return constantly.
- Concurrent Segment Search — the most self-contained theme, and a clean example of an RFC that became a default. Best first deep read.
- Backpressure & Admission Control — a distributed-systems classic (smart rejection) with a tight blast radius.
- Star-Tree Indexes and Tiered Caching — two acceleration/memory features that build on subsystems you already know (aggregations, circuit breakers).
- Remote-Backed Storage & Durability then Sharding, Scaling & Reader/Writer Separation — the two most cross-cutting initiatives; read remote store first because RW separation is built on top of it.
Note: None of these require you to have memorized the others, but they share foundations covered earlier. If a chapter references segment replication, shard allocation, or circuit breakers and you feel shaky, the linked deep dive is the prerequisite — go read it, then come back.
How These Connect to the Capstone
This section is the bridge to the Capstone Projects. Each project is a portfolio-grade contribution rooted in a theme here:
The path is deliberate: read a theme here, find a real sub-task in the catalog, and let the matching capstone project turn it into something you can put your name on.
Start with the catalog of real issues and RFCs, then read Concurrent Segment Search.