The Hitchhiker's Guide to OpenSearch, Lucene & Vectors

Don't Panic.

You are about to read a tour of a system with roughly two decades of accumulated engineering inside it: a distributed coordination layer, a near-real-time indexing engine, a 20-year-old search library called Apache Lucene buried at the bottom, and — bolted on the side and rapidly becoming the main event — a vector search engine that talks to native C++ and, on a good day, a GPU.

Most people meet this system in a state of mild terror. They open IndexShard.java, see 4,000 lines, scroll back up, and quietly close the tab. This guide exists so that does not happen to you. The trick is the same one the actual Hitchhiker's Guide recommends: you need a friend who has been here before, a rough map, and the confidence that the scary words are just names for things you already half-understand.

So here is the deal. We are going to follow one single document from the moment it leaves a curl command until the moment it is an immutable file on disk — and, if it happens to be a vector, until it is a node in a graph being scored by SIMD instructions. Along the way you will meet the cast of characters: the cluster manager, the shard, the engine, the codec, the inverted index, the HNSW graph, and DocValues. Each one gets a memorable introduction and then, immediately, its real org.opensearch.* or org.apache.lucene.* name and the file it lives in, so that the friendliness never costs you accuracy.

This is the flagship on-ramp. After this, the rest of the curriculum — the nine Levels, the Lucene section, the k-NN section, the Engineering section, and the Deep Dives — stops being a wall of links and becomes a set of places you have already glimpsed from the road.

Everything below assumes a local cluster from the prerequisites: ./gradlew run is live, REST is on localhost:9200. If it is not, go fix that first; the guide is much funnier when you can run the commands.

The Three Depths of Mastery

Before the journey, fix your destination. There are exactly three depths at which a person can "know OpenSearch", and conflating them is the single biggest source of wasted effort. Know which one you are aiming for today.

Nobody starts at depth 3. Everybody thinks they need depth 3 on day one. You do not. You need depth 1 in your hands (run the scenarios in the Warm-Up) and depth 2 in your sights. Depth 3 is what the Engineering section is for, and you will get there by doing the journey below enough times that the class names stop being scary.

Note: A wonderful property of this codebase is that depth 2 and depth 3 use the same map. The difference is not "more classes" — it is that the core engineer also holds the on-disk format, the failure modes, and the public design history in their head simultaneously. Same territory, more dimensions.

The One-Sentence Mental Model

If you remember nothing else, remember this boundary, because it decides where every bug lives:

OpenSearch is a distributed system that wraps thousands of single-machine Apache Lucene indices and makes them look like one searchable thing.

Everything about shards, nodes, replication, cluster state, REST, transport, the translog, and the aggregation reduce is OpenSearch. Everything about scoring, tokenization, segments, postings, points, DocValues, and the HNSW graph is Lucene, which OpenSearch merely configures and drives. The Warm-Up has the full ownership table; tape it to your monitor. When someone says "search is slow", your first reflex should be to ask which layer — and by the end of this guide you will have the vocabulary to answer.

The Journey: One curl Command, End to End

Here is our protagonist. A single document, an order, being written to an index.

curl -s -XPUT 'localhost:9200/orders/_doc/42?refresh=true' \
  -H 'Content-Type: application/json' -d '{
    "customer": "Ford Prefect",
    "total": 42.00,
    "note": "mostly harmless",
    "embedding": [0.12, 0.07, 0.99, 0.03]
  }'

That HTTP request is going to pass through nine layers, touch the authoritative state of the entire cluster, land on exactly one machine, and ultimately become bytes in an immutable file. Let us watch it.

flowchart TD
    Curl["curl PUT /orders/_doc/42"] --> REST["REST layer: RestController → RestIndexAction"]
    REST --> Action["Action framework: NodeClient → TransportIndexAction"]
    Action --> CM["Cluster manager: does 'orders' exist? map 'embedding' as knn_vector?"]
    CM --> Route["Routing: hash(_id=42) → primary shard 3"]
    Route --> Transport["Transport layer: send to the node holding shard 3"]
    Transport --> Shard["IndexShard.applyIndexOperationOnPrimary"]
    Shard --> Engine["InternalEngine.index → Lucene IndexWriter + Translog"]
    Engine --> Refresh["refresh() → new DirectoryReader → searchable"]
    Engine --> Segment["flush/merge → immutable segment files on disk"]
    Segment --> Vec[".vec / HNSW graph for the embedding field"]

We will take these in order. Each stop introduces a character, gives you the real name, and tells you where it lives.

Stop 1 — The Doorman: the REST layer

The character. Every request to OpenSearch comes in over HTTP and is met by a doorman whose entire job is to recognize the shape of what you asked (PUT /orders/_doc/42), check it is well-formed, and hand it to the right specialist inside. The doorman does no real work. He just routes.

The real names. The HTTP request lands in RestController, which matches the method+path against a registry of RestHandlers and dispatches to RestIndexAction. The handler parses your JSON into an IndexRequest and calls the client. It does not know what a shard is.

cd ~/src/OpenSearch
find server -name "RestController.java"
#   server/src/main/java/org/opensearch/rest/RestController.java
find server -name "RestIndexAction.java"
#   server/src/main/java/org/opensearch/rest/action/document/RestIndexAction.java

Full treatment: the REST layer deep dive.

Stop 2 — The Dispatcher: the action framework

The character. Inside the building, nobody talks HTTP. Work is expressed as actions — typed request/response pairs — and a dispatcher matches an action type to the one transport action that knows how to perform it. This indirection is what lets the same logic run whether the request arrived over HTTP or from another node.

The real names. RestIndexAction calls NodeClient.execute(...), which looks up the registered TransportAction for the index action — TransportIndexAction (itself routed through the bulk machinery). The wiring lives in ActionModule, where every action type is bound to its handler at startup.

grep -rn "class ActionModule" server/src/main/java/org/opensearch/action/
grep -rln "class TransportIndexAction\|class TransportBulkAction" server/src/main/java/org/opensearch/action/

Full treatment: the action framework deep dive.

Stop 3 — The Keeper of Truth: the cluster manager and cluster state

The character. Somewhere in your cluster, one node has been elected to a lonely and important job: it is the single source of truth about what exists. Which indices are there, what their mappings are, where every shard lives, which nodes are alive. This node is the cluster manager. When our document arrives for an index that does not exist yet, this node is who decides to create it, who picks a field type for embedding, and who writes that decision down.

Note: "Cluster manager" is the term you will see in modern OpenSearch. It was renamed from master during the fork from Elasticsearch. Old blog posts, some metric names, and a few legacy class names still say master / Master — mentally translate. The deep dives say cluster manager; the code is mid-migration.

The real names. The authoritative object is an immutable ClusterState: a big value containing Metadata (index mappings, settings), RoutingTable (where shards are), and DiscoveryNodes (who is alive). Creating the orders index is a ClusterStateUpdateTask executed on the cluster manager's service (ClusterManagerService, formerly MasterService). Dynamically deciding that embedding is a vector field is mapping resolution (MapperService / a knn_vector KNNVectorFieldMapper if the k-NN plugin is installed). The new state is then published two-phase to every node and applied locally by each node's ClusterApplierService.

flowchart LR
    Need["new index 'orders' needed"] --> Task["ClusterStateUpdateTask on cluster manager"]
    Task --> Compute["compute new ClusterState (Metadata + RoutingTable)"]
    Compute --> Publish["PublicationTransportHandler: publish → commit"]
    Publish --> Apply["every node: ClusterApplierService applies new state"]
    Apply --> Ready["index 'orders' now exists cluster-wide"]

grep -rln "class ClusterState" server/src/main/java/org/opensearch/cluster/
ls server/src/main/java/org/opensearch/cluster/coordination/   # election, publishing
grep -rln "class AllocationService" server/src/main/java/org/opensearch/cluster/routing/allocation/

Full treatment: cluster state, cluster state publishing, discovery and coordination. The Warm-Up's Scenario 4 lets you watch the cluster manager react to a dying node.

Stop 4 — The Sorting Hat: routing to a shard

The character. An index is not one thing; it is sliced into shards, and every shard lives on some node with zero or more replica copies. Our document has an _id of 42. Something has to decide, deterministically, which shard owns 42 — and it must give the same answer every time, or reads and writes would disagree.

The real names. Routing is a hash: by default murmur3(_routing ?? _id) % number_of_primary_shards. OperationRouting consults the RoutingTable from cluster state to find the primary shard for that slot and the node hosting it. Each shard is, at the storage level, a complete and independent Lucene index. A primary plus its replicas is one logical shard; the primary takes writes, the replicas take reads and serve as failover.

grep -rln "class OperationRouting" server/src/main/java/org/opensearch/cluster/routing/
grep -rn "murmur3\|generateShardId\|partition" server/src/main/java/org/opensearch/cluster/routing/OperationRouting.java | head

Full treatment: shard allocation.

Stop 5 — The Foreman: IndexShard

The character. Our request has now been transported to the one node holding primary shard 3 of orders. On that node, a foreman takes over. The foreman owns the lifecycle of the shard — is it recovering? started? relocating? — and is the gatekeeper for every operation that touches its data. He does not do the storage himself; he delegates that to a specialist. But nothing happens to this shard without going through him.

The real names. That foreman is IndexShard. The write enters at IndexShard.applyIndexOperationOnPrimary(...), which checks the shard is in a state that can accept writes, manages permits and sequence numbers, and then delegates the real storage work to its Engine.

find server -name "IndexShard.java"
#   server/src/main/java/org/opensearch/index/shard/IndexShard.java
grep -n "applyIndexOperationOnPrimary\|public Engine.IndexResult" \
  server/src/main/java/org/opensearch/index/shard/IndexShard.java | head

Full treatment: index shard lifecycle.

Stop 6 — The Engine Room: InternalEngine and Lucene's IndexWriter

The character. Here, finally, is where a document becomes data. The engine is the layer that turns "apply this index operation" into a concrete Lucene write, a durable log record, and — eventually — a searchable view. It is the single most important class to understand for depth 2, because it sits exactly on the OpenSearch/Lucene boundary: above it is OpenSearch's operation model (versions, sequence numbers, the translog); below it is raw Lucene.

The real names. InternalEngine.index(Engine.Index) does roughly this: acquire a per-_id lock, decide whether this is a new doc / update / conflict, assign a sequence number and primary term, call Lucene IndexWriter.addDocument (or updateDocument), record the result in the in-memory LiveVersionMap, append the operation to the Translog for durability, and advance the LocalCheckpointTracker.

flowchart TD
    Idx["InternalEngine.index(Engine.Index)"] --> Lock["per-_id lock"]
    Lock --> Plan{new / update / conflict?}
    Plan -->|conflict| Conf["VersionConflictEngineException"]
    Plan -->|ok| Seq["assign seqNo + primaryTerm"]
    Seq --> IW["Lucene IndexWriter.addDocument"]
    IW --> VM["LiveVersionMap.put"]
    VM --> TL["Translog.add  (durable here)"]
    TL --> CP["LocalCheckpointTracker advance"]

The crucial insight, worth more than any class name: a document becomes durable the moment it is in the translog, but it becomes visible only after a refresh opens a new Lucene DirectoryReader. Durability and visibility are independent. This is the near-real-time (NRT) model.

find server -name "InternalEngine.java"
#   server/src/main/java/org/opensearch/index/engine/InternalEngine.java
grep -n "public IndexResult index\|IndexWriter\|Translog.add\|versionMap" \
  server/src/main/java/org/opensearch/index/engine/InternalEngine.java | head

Full treatment: the Engine Internals deep dive — the next thing you should read after this guide, because it is the connective tissue of the whole system. Also: the translog and refresh/flush/merge.

Stop 7 — The Stenographer: the Lucene segment and the codec

The character. Lucene never edits anything in place. When the in-memory buffer fills (or a refresh/flush happens), Lucene writes a segment: a small, immutable, self-contained mini-index. Your document is now frozen into a set of files that will never be rewritten — only, eventually, merged with others into a larger segment and then deleted. Immutability is not a limitation; it is the secret to lock-free reads, cheap snapshots, and crash safety.

But who decides the byte layout of those files? The codec — a pluggable format specification. It is the stenographer that knows exactly how to transcribe terms, doc values, points, and vectors into bytes, and how to read them back.

The real names. A Lucene index is a set of segments plus a segments_N commit point (SegmentInfos). Each segment is a bundle of files written by the current default Codec — call it LuceneNNNCodec (e.g. Lucene101Codec / Lucene103Codec; grep to find the exact one your bundled Lucene ships). The codec is an SPI: PostingsFormat, DocValuesFormat, KnnVectorsFormat, StoredFieldsFormat, PointsFormat, each pluggable via META-INF/services.

# Find the immutable segment files in a real shard directory.
DATA=$(find . -type d -name index -path "*nodes*shard*" | head -1)
find "$DATA" -maxdepth 1 -type f | sed 's:.*/::' | sort
# Expect: segments_N, plus per-segment .si .fnm .fdt/.fdx .tim/.tip .doc/.pos
#         .dvd/.dvm .nvd/.nvm .kdd/.kdi .vec/.vex/.vem ...

Full treatment: Segments and Codecs and IndexWriter and Merges. To physically open one of these directories with a GUI, the Crack Open a Lucene Index lab points Lucene's Luke tool at it.

Stop 8 — The Library Card Catalog: the inverted index and DocValues

We are now inside the segment, looking at two of its most important structures. They are opposites, and knowing which one a query uses is half of understanding search performance.

The inverted index is the card catalog. For the note field ("mostly harmless"), the analyzer produced terms [mostly, harmless], and the index stores, for each term, the posting list: the set of document IDs containing it. To answer "which docs contain harmless?" Lucene jumps straight to that term and reads the list. This is the structure that makes full-text search fast, and it is term → documents.

DocValues is the opposite: document → value, stored column-wise. To sort by total or compute sum(total), you do not want the inverted index — you want to stream the total value for every matching doc, in order, like a column in a database. That is DocValues, and it is the backbone of sorting and aggregations.

The real names. The terms dictionary is a BlockTree backed by an FST (finite-state transducer); postings are read via PostingsEnum. DocValues come in NUMERIC / SORTED / SORTED_SET / SORTED_NUMERIC / BINARY flavors and are read through DocValuesProducer and types like SortedNumericDocValues.

# OpenSearch's own DocValues/fielddata deep dive ties this to aggregations.
grep -rn "SortedNumericDocValues\|getLeafCollector" \
  server/src/main/java/org/opensearch/search/aggregations/ | head

Full treatment: The Inverted Index and Postings, Points and BKD Trees, DocValues: The Columnar Store, and the existing DocValues and Fielddata deep dive.

Stop 9 — The Constellation: the HNSW graph and SIMD scoring

Our document had an embedding. Vectors do not fit the card-catalog model at all — there is no "term" to look up. Finding the nearest vectors to a query vector, exactly, would mean comparing against every vector in the segment, which at scale is hopeless. So vector search cheats, beautifully.

The character. Picture every vector as a star. HNSW (Hierarchical Navigable Small World) connects each star to a handful of nearby stars, building a navigable constellation with a few sparse "highway" layers on top and dense local streets at the bottom. To find the nearest stars to a query, you parachute into the top layer, greedily walk toward the query along the highways, drop a layer, walk again, and so on — visiting a tiny fraction of all vectors and still landing on the true neighbors almost always. "Almost" is the whole game: this is approximate nearest neighbor (ANN), and you trade a sliver of recall for orders of magnitude of speed.

The scoring. Walking the graph means computing distances — dot products, squared Euclidean distances — over and over. This inner loop is where vector search lives or dies, so Lucene vectorizes it with SIMD: the Panama Vector API (jdk.incubator.vector) computes many lane-wise multiply-adds per instruction, picked at runtime by a VectorizationProvider (PanamaVectorUtilSupport when the CPU supports it, a scalar fallback otherwise). This is why HNSW scoring is fast, and it is the "vectorization" theme you will meet again in the Engineering section.

The real names (Lucene engine). KnnFloatVectorField / KnnByteVectorField declare the field; FloatVectorValues reads vectors; VectorSimilarityFunction (EUCLIDEAN, DOT_PRODUCT, COSINE, MAXIMUM_INNER_PRODUCT) defines distance; HnswGraph / HnswGraphBuilder are the graph; KnnFloatVectorQuery runs the search; the on-disk format is Lucene99HnswVectorsFormat (and scalar-quantized variants, up to the Lucene104* formats), written into the .vec / .vex / .vem files; VectorUtil is the SIMD-accelerated math.

The real names (k-NN plugin). OpenSearch's k-NN plugin adds the knn_vector field type and three engines: faiss (the default — native C++ via JNI), lucene (the Lucene HNSW above), and nmslib (deprecated). For faiss, a custom codec writes the native graph as segment files, and the graph is loaded into native memory outside the JVM heap, capped by a circuit breaker.

flowchart TD
    Q["query vector q"] --> Top["enter top HNSW layer"]
    Top --> Walk["greedy walk toward q (SIMD distance via VectorUtil/Panama)"]
    Walk --> Down["descend a layer, repeat"]
    Down --> Bottom["dense bottom layer: collect candidates"]
    Bottom --> K["return top-k approximate neighbors"]

Full treatment: HNSW Vector Search in Lucene, SIMD and the Panama Vector API, and the entire k-NN section. The HNSW-from-scratch lab has you build a toy graph by hand.

The Return Trip: a search, in one paragraph

We followed a write all the way down. A read runs the same map in reverse and fanned out. RestSearchAction → TransportSearchAction on a coordinating node, which uses the RoutingTable to fan the query out to one copy of every shard. On each shard, the query phase (QueryPhase) runs the Lucene query against that shard's segments and returns the top-K doc IDs and scores; a separate fetch phase (FetchPhase) loads _source for the survivors; the coordinating node merges the per-shard results in SearchPhaseController. Aggregations add a reduce step where partial per-shard results are combined into the global answer. A k-NN query is the same shape, but the per-shard work walks the HNSW graph instead of the inverted index. See the search execution deep dive and the Warm-Up's Scenarios 2 and 3.

The Whole Cast, On One Page

Where To Go Next (the map)

You have now seen the whole road once. Here is where each branch leads, and the order most people should take them.

flowchart TD
    Guide["You are here: the Hitchhiker's Guide"] --> Warmup["Warm-Up: run it as a user (depth 1)"]
    Warmup --> Levels["Levels 1–9: become a contributor (depth 2)"]
    Levels --> Lucene["Lucene section: segments, codecs, HNSW, SIMD"]
    Levels --> KNN["k-NN section: vectors, engines, JNI, quantization"]
    Lucene --> Eng["Engineering at Scale: real RFCs (depth 3)"]
    KNN --> Eng
    Eng --> Projects["Capstone Projects: ship a real contribution"]
    Levels --> Deep["Deep Dives: the reference for every subsystem"]

Before You Drive On

You are ready to leave this guide when, without notes, you can:

• Name the three depths of mastery and say which one you are aiming for now.
• State the one-sentence model and split a list of concepts into "OpenSearch owns it" vs "Lucene owns it".
• Trace our document's nine stops in order, naming the real class at each: RestIndexAction → TransportIndexAction → cluster manager / ClusterState → OperationRouting → IndexShard → InternalEngine → Lucene IndexWriter → segment/Codec → (for a vector) HnswGraph + VectorUtil.
• Explain why a document is durable before it is visible.
• Explain in one breath what HNSW does and why SIMD makes it fast.
• Point at the right next section for whatever you want to learn.

If a box is empty, the offending stop above is one re-read away.

The Guide's final, load-bearing advice has not changed in this edition: Don't Panic. The system is large, but it is legible — every scary name is just a character with a job, a class, and a file. You have met them all now.

Continue to the Warm-Up to run this journey as a user, or straight to Level 1 to start building from source. When you are ready for the deep end, Engineering at Scale is waiting.