Thread Pools and Concurrency

OpenSearch is a heavily concurrent server, and almost all of that concurrency is mediated by a small set of named thread pools. Understanding which pool runs which work — and which threads you must never block — is the difference between code that scales and code that wedges a node. This chapter covers the ThreadPool abstraction and its named pools (SEARCH, WRITE, GET, MANAGEMENT, GENERIC, etc.), the pool types (fixed, scaling, fixed-auto-queue-size), the single-writer-per-shard invariant that simplifies the engine, how rejections and queues behave, ThreadContext propagation, and the cardinal rule that you must never block the cluster-applier thread.

After this chapter you should be able to: name the pool that runs any given operation; read _cat/thread_pool to diagnose a saturated node; explain why the write path can avoid locks inside a shard; and explain the cluster-wide cascade that follows from blocking a coordination thread.

Note: Blocking the wrong thread is the most damaging concurrency mistake in the codebase. It rarely fails a test; it manifests in production as a node that stops accepting work or a cluster that stops applying state. The rules in this chapter are not style — they are correctness.

The ThreadPool abstraction

find server -name "ThreadPool.java" -path "*threadpool*"
grep -n "public static class Names\|public static final String SEARCH\|public static final String WRITE\|\
public static final String GET\|public static final String GENERIC\|public static final String MANAGEMENT" \
  server/src/main/java/org/opensearch/threadpool/ThreadPool.java

ThreadPool owns every executor and exposes them by name via ThreadPool.Names. You obtain one with threadPool.executor(Names.SEARCH) or schedule with threadPool.schedule(...). The pools that matter most:

# See the full configured set and sizes on a running node:
curl -s "localhost:9200/_cat/thread_pool?v&h=node_name,name,type,size,queue,active,rejected,completed"

Pool types

OpenSearch sizes executors with three strategies. Find them:

find server libs -name "OpenSearchExecutors.java" -o -name "EsExecutors.java"
grep -n "newFixed\|newScaling\|newAutoQueueFixed\|class OpenSearchExecutors" \
  server/src/main/java/org/opensearch/common/util/concurrent/OpenSearchExecutors.java 2>/dev/null \
  || grep -rn "newFixed\|newScaling\|newAutoQueueFixed" server/src/main/java/org/opensearch/common/util/concurrent/

The choice encodes a policy: bounded pools (fixed) protect the node by rejecting rather than queueing unboundedly; scaling pools tolerate bursts of work that is expected to be occasional or slow.

Warning: Never submit blocking or long-running work to a fixed pool sized for short operations (SEARCH, WRITE, GET). A blocked thread there is a thread permanently removed from a tiny pool; a few of them stall all search or all indexing on the node. Long/blocking work goes on GENERIC.

Rejections and queues

A fixed pool with a full queue rejects new work, surfacing as OpenSearchRejectedExecutionException and, to clients, HTTP 429 Too Many Requests with a rejected_execution cause. This is a feature — back-pressure — not a crash. The relevant counters:

curl -s "localhost:9200/_nodes/stats/thread_pool?filter_path=nodes.*.thread_pool.search.rejected,nodes.*.thread_pool.write.rejected"
curl -s "localhost:9200/_cat/thread_pool/search,write?v&h=node_name,name,active,queue,rejected,completed"

Rising rejected on write means indexing back-pressure (clients should slow down / retry); rising rejected on search means the query load exceeds capacity. The fix is usually upstream (fewer/cheaper requests, more nodes) — not "make the queue bigger," which just trades rejection for latency and heap pressure.

The single-writer-per-shard model

A crucial simplification: only one thread writes to a given shard's Lucene IndexWriter at a time. Write operations for a shard are serialized so the engine (engine-internals.md) does not need fine-grained locking around the writer for indexing.

grep -n "indexShardOperationPermits\|acquirePrimaryOperationPermit\|acquireReplicaOperationPermit\|\
class IndexShardOperationPermits" \
  server/src/main/java/org/opensearch/index/shard/IndexShard.java \
  server/src/main/java/org/opensearch/index/shard/IndexShardOperationPermits.java 2>/dev/null

The mechanism is IndexShardOperationPermits: operations acquire a permit before touching the shard. Normal ops share permits (many can read/index concurrently at the Lucene level, which is itself thread-safe for add/update), but operations that must run exclusively (relocation, primary-term bump, resync) acquire all permits, briefly blocking new ops. This permit system — not a giant lock — is how the shard coordinates concurrent access while keeping the hot path lock-light.

flowchart TD
    Op1[index op] -->|acquire shared permit| Permits[IndexShardOperationPermits]
    Op2[index op] -->|acquire shared permit| Permits
    Relocate["relocation / primary-term bump"] -->|acquire ALL permits| Permits
    Permits -->|"exclusive: wait for in-flight to drain"| Block[new ops queue briefly]

ThreadContext propagation

A single logical request hops across thread pools and across nodes. Headers, the response cap, the request's security identity (used by the security plugin), and transient values must travel with it. That is ThreadContext.

find server libs -name "ThreadContext.java"
grep -n "stashContext\|newStoredContext\|putHeader\|putTransient\|class ThreadContext" \
  libs/common/src/main/java/org/opensearch/common/util/concurrent/ThreadContext.java 2>/dev/null \
  || grep -rn "class ThreadContext" server libs

Warning: When you hand work to another thread pool (or a listener), the framework normally restores the originating ThreadContext so headers and identity follow the request. If you bypass the standard executor wrappers (raw Thread, raw Executor), you lose the context — security headers vanish, and the security plugin may run the work as the wrong principal. Always schedule through ThreadPool/threadPool.executor(...).

The cardinal rule: never block coordination threads

Two thread types must never block:

1. The cluster-applier thread (ClusterApplierService) — runs appliers and listeners synchronously when a new cluster state is committed.
2. The cluster-manager update thread (MasterService) — runs cluster-state update tasks.

(Both are introduced in cluster-state-publishing.md.)

Why blocking the applier thread is catastrophic, link by link:

flowchart TD
    Block["applier/listener blocks (I/O, lock, sleep)"] --> NoApply[committed state not finished applying]
    NoApply --> NoAck[node does not ack the commit]
    NoAck --> PubTimeout["publish times out (cluster.publish.timeout)"]
    PubTimeout --> StepDown["cluster manager may step down"]
    StepDown --> Election[new election / churn]
    Election --> ClusterWide[cluster-wide instability]

The same logic applies to the update thread: a blocked execute() stalls all cluster-state progress for the entire cluster, because there is one such thread.

The fix is always the same: do the slow part on a worker pool. Inside an applier or listener, capture what you need and threadPool.generic().execute(...) (or the appropriate pool) the heavy work; return immediately.

Note: This is exactly what Level 4 lab 4.2 and Level 4 lab 4.3 make you internalize by building (and mis-building) a cluster-state callback.

assertBusy and concurrency in tests

Because so much happens asynchronously, tests cannot assert "right now." They poll with assertBusy, which retries an assertion until it passes or times out:

grep -n "public static void assertBusy\|busy" \
  test/framework/src/main/java/org/opensearch/test/OpenSearchTestCase.java

assertBusy(() -> {
    SearchResponse r = client().prepareSearch("idx").get();
    assertHitCount(r, 1);   // becomes true once the refresh makes the doc visible
});

assertBusy is the idiomatic way to wait for refresh, recovery, allocation, or any other eventually-consistent state in tests — never Thread.sleep. Using it correctly (with a bounded timeout) is a reviewable requirement; see Level 2 lab 2.2.

Reading exercise

# 1. The named pools and their default types/sizes.
grep -n "Names\.\|new ScalingExecutorBuilder\|new FixedExecutorBuilder\|new AutoQueueAdjustingExecutorBuilder" \
  server/src/main/java/org/opensearch/threadpool/ThreadPool.java

# 2. Live saturation snapshot.
curl -s "localhost:9200/_cat/thread_pool?v&h=node_name,name,type,active,queue,rejected"

# 3. The permit system that serializes shard writers.
grep -n "acquire\|asyncBlockOperations\|allPermits\|TOTAL_PERMITS" \
  server/src/main/java/org/opensearch/index/shard/IndexShardOperationPermits.java

# 4. ThreadContext stash/restore.
grep -n "stashContext\|newStoredContext\|restore" \
  libs/common/src/main/java/org/opensearch/common/util/concurrent/ThreadContext.java

# 5. Tests.
./gradlew :server:test --tests "org.opensearch.threadpool.ThreadPoolTests"

Answer:

1. Which pool runs each of: a search query phase, a bulk write, a get, a shard refresh, a snapshot copy, a _cat/health?
2. What is the difference between a fixed and a scaling pool, and why is WRITE fixed while GENERIC is scaling?
3. A client sees HTTP 429 rejected_execution on indexing. Which counter confirms it, and what is the correct remediation (and the wrong one)?
4. Explain the single-writer-per-shard invariant and how IndexShardOperationPermits implements both shared and exclusive access.
5. What does ThreadContext carry, and what breaks if you run work on a raw Thread instead of through ThreadPool?
6. Walk the full cascade from "an applier blocks" to "the cluster manager steps down." Name the timeout that triggers the step-down.

Common bugs and symptoms

Validation: prove you understand this

1. From memory, list eight named thread pools and what each runs. Mark which are fixed vs scaling.
2. Explain rejection as back-pressure: which pools reject, what the client sees, and why enlarging the queue is the wrong fix.
3. Describe the single-writer-per-shard model and how shared vs exclusive permits coexist in IndexShardOperationPermits.
4. Explain ThreadContext propagation and the concrete security failure that bypassing ThreadPool causes.
5. Draw the cascade from a blocked applier thread to cluster-manager step-down, naming every link and the governing timeout.
6. Explain why assertBusy (not Thread.sleep) is the correct way to wait for eventually-consistent state in tests.