Appendices & Reference

The chapters in this book are meant to be read — first principles, then data structures, then the real org.opensearch.* and org.apache.lucene.* classes, then a lab where you prove you understood it. This appendix is the opposite kind of document. It is meant to be looked up. When you are mid-task — staring at a stack trace, drafting a PR, tuning a setting, or trying to remember which class owns a behavior — you do not want to re-read a 600-line deep dive. You want a table, an exact name, and a one-click jump to the chapter that explains it.

That is what these six reference documents give you. They are deliberately tables-first, comprehensive, and heavily cross-linked back into the rest of the book. Nothing here is new material; everything here points at a chapter that teaches the concept in full. Think of the appendix as the index card layer that sits on top of the curriculum.

Note: Throughout the book, cluster manager (formerly master) is the current term for the node role and the settings that coordinate the cluster. OpenSearch renamed the role for inclusive language after the 7.10.2 fork from Elasticsearch; the old master names survive as deprecated aliases. The appendix uses the current term and notes the alias where it matters.

The six reference documents

Note: All six documents ship with this book. Each also has a fuller home in the deep dive it points at — the REST map expands in ../deep-dives/rest-layer.md, the file formats in the Lucene section, and the divergence notes throughout the deep dives. The appendix is the index card; the deep dive is the lecture.

How the appendix relates to the rest of the book

The curriculum has five layers, and the appendix indexes all of them:

flowchart LR
    A["Appendix<br/>(look up)"] --> DD["deep-dives/<br/>(subsystem reference)"]
    A --> LU["lucene/<br/>(storage primitives)"]
    A --> KN["knn/<br/>(vector search)"]
    A --> EN["engineering/<br/>(OpenSearch-specific systems)"]
    A --> MC["masterclass/<br/>(feature deep dives + labs)"]
    A --> LV["level-1..9/<br/>(graded curriculum)"]

• The deep dives (../deep-dives/index.md) are the subsystem reference: 24 chapters mirroring the request path and cluster lifecycle. The glossary and key-classes map point here most often.
• The Lucene chapters (../lucene/index.md) cover the storage primitives below OpenSearch: segments, postings, BKD trees, HNSW, DocValues, the codec SPI.
• The k-NN chapters (../knn/index.md) cover vector search: HNSW/IVF/PQ, the faiss/nmslib/lucene engines, the JNI boundary, quantization, and the query path.
• The engineering chapters (../engineering/index.md) cover the systems that make OpenSearch not Elasticsearch: concurrent segment search, star-tree, tiered caching, remote store, backpressure.
• The masterclasses (../masterclass/index.md) are feature-by-feature deep dives with runnable labs — security, debugging & profiling, aggregations, distributed consensus, the query engine, vector internals, and more.
• The levels (../level-1/index.md … ../level-9/index.md) are the graded, hands-on curriculum that consumes all of the above.

How to use these while contributing

The appendix earns its keep in the middle of real work. A few concrete patterns:

You hit an unfamiliar term in an issue or PR. Start in the Glossary. Get the one-sentence version, then follow the chapter link only if you need to go deep. Most terms resolve in ten seconds.

You have a stack trace and need to orient. Take the most specific class name in the trace and look it up in Key Classes by Subsystem. That tells you which subsystem owns the failure and which chapter explains it — so you read the right 600 lines, not all of them.

You are tuning or reproducing a bug. Open Configuration Keys Reference. Find the setting, check whether it is static (requires a restart / index recreation) or dynamic (settable live via _cluster/settings or _settings), and read the chapter the row points at before you change anything in a PR.

You are tracing a request from the edge. Open the REST Endpoint Map, find the endpoint, and follow it to its RestHandler → TransportAction → the chapter that traces the full path.

You are debugging on-disk state. Run find over a shard, then classify every extension with Lucene File Formats.

You are reading older Elasticsearch material. Sanity-check it against OpenSearch vs Elasticsearch so you don't apply a fact that diverged after the fork.

Worked example: from a stack trace to the right chapter

The "take the most specific class name and look it up" pattern deserves a real walkthrough, because the skill is not looking up one class — it is reading the package prefixes down the trace to see the request cross subsystem boundaries. Here is a trace you will actually see, from a search that built too many aggregation buckets:

[WARN][o.o.s.SearchService] [data-node-3] [orders][2]: query phase failed
org.opensearch.search.aggregations.MultiBucketConsumerService$TooManyBucketsException:
  Trying to create too many buckets. Must be less than or equal to: [65535] but was
  [65536]. This limit can be set by changing the [search.max_buckets] cluster level setting.
    at org.opensearch.search.aggregations.MultiBucketConsumerService$MultiBucketConsumer.accept(MultiBucketConsumerService.java)
    at org.opensearch.search.aggregations.bucket.BucketsAggregator.grow(BucketsAggregator.java)
    at org.opensearch.search.query.QueryPhase.execute(QueryPhase.java)
    at org.opensearch.search.SearchService.executeQueryPhase(SearchService.java)
    ... 14 more (org.opensearch.action.search.*, org.opensearch.transport.*, java.util.concurrent.*) ...
  [search][T#4]

Read it top-down, one package prefix at a time, and let each prefix name a subsystem and a chapter:

The reasoning is entirely in the prefixes. org.opensearch.search.aggregations.* is the deepest, most specific package, so it owns the failure — this is a search.max_buckets limit (default 65535), not heap exhaustion. Confirm the class is real and the setting name is current with a grep before you touch anything:

grep -rn "class MultiBucketConsumerService" \
  server/src/main/java/org/opensearch/search/aggregations/
grep -rn "search.max_buckets\|MAX_BUCKET_SETTING" \
  server/src/main/java/org/opensearch/search/aggregations/MultiBucketConsumerService.java

Contrast the near-identical memory failure, which starts with a different prefix and therefore routes to a different chapter:

org.opensearch.core.common.breaker.CircuitBreakingException: [parent] Data too large,
  data for [<agg [by_customer]>] would be [...], which is larger than the limit of [...]
    at org.opensearch.indices.breaker.HierarchyCircuitBreakerService.checkParentLimit(...)
    at org.opensearch.common.breaker.ChildMemoryCircuitBreaker.addEstimateBytesAndMaybeBreak(...)

Same aggregation query, but the top frames are …breaker.* — so the owning subsystem is the circuit breakers, not aggregations, and the fix lives in breaker settings and heap, not search.max_buckets. Two traces that look alike land in two different chapters purely on their package prefixes. That is the whole technique: the most specific org.opensearch.* prefix names the subsystem; Key Classes by Subsystem turns that class into the chapter; you read those 600 lines and none of the others.

Note: The prefix that owns the failure is the deepest one at the top of the trace, not the framework frames beneath it. Every trace bottoms out in org.opensearch.transport.* and java.util.concurrent.* — those are the plumbing every request rides, not the culprit. Read down until the prefix stops being …search.aggregations (or …index.engine, or …cluster.routing.allocation) and becomes generic transport/executor frames; the last specific prefix is your subsystem.

Warning: Treat every name, default, and line number in this appendix as a starting point to verify, not gospel. OpenSearch and Lucene move fast: class names get refactored, settings change defaults across minor versions, and a Lucene upgrade renames the default codec. Every table here tells you the grep or curl that confirms the current truth in your checkout and your version. When the appendix and your grep disagree, your grep wins — and that is a documentation fix worth a PR.

The verification reflex

Two commands settle almost every "is this still true?" question. From a server source checkout:

# Confirm a class still exists and see its package / role:
grep -rn "class BalancedShardsAllocator" \
  server/src/main/java/org/opensearch/cluster/routing/allocation/

# Confirm a setting key, its default, and its scope (look at the Setting.* call):
grep -rn "number_of_replicas\|SETTING_NUMBER_OF_REPLICAS" \
  server/src/main/java/org/opensearch/cluster/metadata/IndexMetadata.java

From a running cluster:

# What settings are actually in effect (defaults included):
curl -s 'localhost:9200/_cluster/settings?include_defaults=true&flat_settings=true' \
  | python3 -m json.tool | less

# What an index's effective settings are:
curl -s 'localhost:9200/my-index/_settings?include_defaults=true&flat_settings=true&pretty'

Keep this appendix open in one pane and your checkout in another. The reference is fast; the grep is authoritative. Use both.