Level 3: OpenSearch Architecture

This level builds the mental model you will use for the rest of the curriculum: how a running OpenSearch process is organized, how a request travels from an HTTP socket to a shard, and which thread does the work at each hop. By the end you will be able to open the codebase, point at the class that handles any given request, and explain why that class exists.

Levels 1 and 2 got you building, testing, and submitting a trivial PR. From here on, every lab assumes you can find a class without a search engine and run a local cluster. Level 3 is where you stop reading about OpenSearch and start reading OpenSearch.

Learning Objectives

By the end of Level 3 you must be able to:

1. Explain the node / cluster / index / shard model and how it maps to running processes and Lucene.
2. Enumerate the node roles (cluster_manager, data, ingest, coordinating, search, …) and say what each one is allowed to do.
3. Trace GET /_cluster/health and POST /<index>/_doc from the HTTP layer through RestController → RestHandler → NodeClient → TransportAction without a guide.
4. Distinguish the REST layer (port 9200, HTTP/JSON) from the transport layer (port 9300, binary, node-to-node) and say which classes own each.
5. Explain how actions are registered in ActionModule and how an ActionType resolves to a concrete TransportAction.
6. Name the major thread pools (ThreadPool.Names) and predict which one runs a given operation.
7. Build a minimal plugin that registers a new REST endpoint and transport action (Lab 3.3).

The Node / Cluster / Index / Shard Model

Start with the four nouns. Everything in OpenSearch is built out of them.

An index is split into primary shards at creation time (immutable count, modulo _split/_shrink); each primary has zero or more replicas. A shard — primary or replica — is a self-contained Apache Lucene index made of immutable segments. IndexShard wraps a Lucene IndexWriter/DirectoryReader through an Engine (the default is InternalEngine). Indexing, refresh, flush, and merge all happen at the shard level. The cluster's job is to decide which node holds which shard — that is the routing table, and computing it is the subject of Level 4.

Cluster "logs-prod"
├── Node A (cluster_manager, data)
│   ├── shard [orders][0] primary      ← a Lucene index (segments on disk)
│   └── shard [orders][1] replica
├── Node B (data, ingest)
│   ├── shard [orders][0] replica
│   └── shard [orders][1] primary
└── Node C (coordinating only)         ← holds no data; routes requests

Note: "Index" is overloaded. An OpenSearch index is a collection of shards; a Lucene index is one shard. When reading the engine code, the word almost always means the Lucene one.

For the full storage story — IndexShard, InternalEngine, translog, segments, refresh/flush — see Level 6 and the indexing-path deep dive.

The Request Path: REST → RestController → Action → Shards

This is the spine of the system. Memorize it. Every user-facing operation — search, index, cluster health, snapshot — follows the same skeleton.

flowchart TD
    HTTP[HTTP request on :9200] --> HC[HttpServerTransport<br/>Netty4HttpServerTransport]
    HC --> RC[RestController.dispatchRequest]
    RC -->|matches path+method| RH[RestHandler<br/>e.g. RestClusterHealthAction / RestIndexAction]
    RH -->|parse JSON/params| REQ[builds an ActionRequest]
    REQ --> NC[NodeClient.execute ActionType, request, listener]
    NC -->|looks up in actions map| TA[TransportAction<br/>HandledTransportAction]
    TA --> AF[ActionFilters chain]
    AF --> DO[doExecute on the right node]
    DO -->|fan out over transport :9300| SH[per-shard work<br/>IndexShard / SearchService]
    SH -->|results| LIS[ActionListener.onResponse]
    LIS --> XC[ToXContent -> JSON]
    XC --> HTTP

Read it as five questions:

1. Who accepts the socket? HttpServerTransport (default Netty4HttpServerTransport from modules/transport-netty4). It turns bytes into a RestRequest.
2. Who routes it? RestController.dispatchRequest matches (method, path) against the registered handlers and calls the matching RestHandler.
3. Who parses and validates it? A RestHandler — almost always a subclass of BaseRestHandler. It reads path params, query params, and the JSON body, builds a typed ActionRequest, and returns a RestChannelConsumer that invokes the NodeClient.
4. Who dispatches to logic? NodeClient.execute(ActionType, request, listener) looks up the TransportAction registered for that ActionType and runs it locally.
5. Who does the work? A TransportAction (typically HandledTransportAction). For cluster-wide reads it may route to the cluster manager; for writes it routes to the primary shard and replicates; for searches it fans out to many shards and reduces.

The crucial decoupling: a RestHandler never calls business logic directly. It only knows how to turn HTTP into an ActionRequest and hand it to the client. The same TransportAction can be invoked from REST, from the Java client, or from another action — REST is just one front door.

For the layered detail, read the REST layer deep dive, the transport layer deep dive, and the action framework deep dive.

Node Roles

A node advertises a set of roles via the node.roles setting in opensearch.yml (or -Enode.roles=... on the command line). Roles decide what work a node is allowed to do; the cluster manager decides what it actually does.

Terminology — cluster manager vs master: OpenSearch renamed the "master" role and APIs to cluster manager for inclusive language. The old master role and settings like cluster.initial_master_nodes and ?master_timeout still work as deprecated aliases of cluster_manager / cluster.initial_cluster_manager_nodes / ?cluster_manager_timeout. New code uses cluster_manager. You will see both spellings throughout the codebase; first reference in this curriculum always notes both.

Every node is also implicitly a coordinating node: any node can accept a client request, fan it out to the nodes that hold the relevant shards, and reduce the results. A node with node.roles: [] is only a coordinator — useful as a load-balancing tier.

Find the role definitions yourself:

grep -rn "class DiscoveryNodeRole" server/src/main/java/org/opensearch/cluster/node/
grep -rn "CLUSTER_MANAGER_ROLE\|DATA_ROLE\|INGEST_ROLE\|roleMap" \
  server/src/main/java/org/opensearch/cluster/node/DiscoveryNodeRole.java

REST vs Transport: Two Ports, Two Protocols

A single user request usually uses both: HTTP on 9200 to reach the coordinating node, then binary transport on 9300 to reach the data nodes that hold the shards. A search hitting 10 shards on 3 nodes is one HTTP request and many transport round-trips.

The transport payload contract is Writeable: a type implements writeTo(StreamOutput) and a StreamInput-taking constructor. Polymorphic types (e.g., which QueryBuilder is on the wire) resolve through NamedWriteableRegistry. Getting this wire format right is the heart of backward compatibility — see Level 9 and the transport layer deep dive.

Thread Pools

OpenSearch never blocks a transport or HTTP I/O thread on real work. Work is dispatched onto named thread pools owned by ThreadPool. Picking the wrong pool (or blocking on GENERIC) is a classic contributor bug.

Inspect them on a live node:

curl -s 'localhost:9200/_cat/thread_pool?v&h=node_name,name,active,queue,rejected'

Find the constants and defaults in source:

grep -n "public static final String" server/src/main/java/org/opensearch/threadpool/ThreadPool.java | head -40
grep -n "ThreadPool.Names" server/src/main/java/org/opensearch/action/search/TransportSearchAction.java | head

The single-threaded CLUSTER_MANAGER_SERVICE pool is why cluster state updates are serialized — a detail you will revisit in Level 4.

Source Code Areas to Inspect

Read these before and after the labs. You are not modifying anything yet.

REST layer (server/src/main/java/org/opensearch/rest/)

Action framework (server/src/main/java/org/opensearch/action/)

Transport layer (server/ + modules/transport-netty4/)

Node bootstrap & thread pools (server/)

Key Classes Quick Reference

The Labs

Deliverables

You must demonstrate all of the following before advancing to Level 4:

• A reading-log artifact tracing GET /_cluster/health from RestController to TransportClusterHealthAction (Lab 3.1).
• A reading-log artifact tracing POST /<index>/_doc from RestIndexAction to TransportShardBulkAction (Lab 3.1).
• A 3-node local cluster you started, with _cat/nodes showing distinct roles (Lab 3.2).
• A one-paragraph explanation of REST (9200) vs transport (9300), naming the entry class for each.
• A working plugin that adds a new REST endpoint and transport action, installed into a local distro and curled successfully (Lab 3.3).
• From memory: name the thread pool that runs a search, a bulk index, and a cluster state update.

Common Mistakes

How to Verify Success

# 1. Cluster is up and you can read its roles
curl -s 'localhost:9200/_cat/nodes?v&h=name,node.role,cluster_manager'

# 2. You can find any hop in the request path by grep, not by memory of line numbers
grep -rn "registerHandler" server/src/main/java/org/opensearch/rest/RestController.java | head
grep -rn "register(ClusterHealthAction" server/src/main/java/org/opensearch/action/ActionModule.java

# 3. Thread pools are visible
curl -s 'localhost:9200/_cat/thread_pool/search,write?v'

When you can open the codebase cold and point at the handler, the action, and the thread pool for any request, you are ready for Level 4: Cluster Coordination and State.