Workload Management (WLM)

Backpressure sheds load when a node is drowning. Circuit breakers stop a single allocation before it OOMs the JVM. Neither one cares whose query is causing the pain. That is the gap Workload Management fills: it isolates tenants. When your analytics dashboard fires a terms aggregation over a year of data at the same moment your latency-sensitive product search runs, WLM keeps the greedy one from eating the whole node — the noisy-neighbor problem. You define workload groups, give each a slice of the node's CPU and memory, tag incoming searches into a group, and WLM enforces the slice by rejecting requests at admission or cancelling in-flight tasks that overrun.

This is a flagship 2.17+/3.x subsystem, and on OpenSearch 3.8 it is spelled WorkloadGroup everywhere — the feature was renamed from QueryGroup and the old symbol is gone from the main source (some Java constants still carry a legacy QUERYGROUP_ prefix, but every class, REST path, and setting key is WorkloadGroup / workload_group). Verify that yourself before you trust any older doc:

cd ~/src/OpenSearch
# QueryGroup is fully renamed on 3.x — expect ZERO hits in main source.
grep -rln "QueryGroup" server/src/main plugins/workload-management/src/main
# The real thing:
ls server/src/main/java/org/opensearch/wlm/

After this chapter you can: place WLM as the third admission layer alongside backpressure and circuit breakers and say crisply what each protects; describe the workload-group data model (resource limits + resiliency mode); trace how a search is tagged into a group and how its CPU/memory are tracked; explain the two enforcement mechanisms (rejection vs. cancellation) and the node/group modes that gate them; use the CRUD and stats REST APIs; and know which files to open to add a resource type or a selection strategy.

Prerequisites. Read backpressure and admission control first — WLM reuses its machinery (TaskResourceTrackingService, CancellableTask, TaskCancellation, node-duress trackers). It also leans on threadpools and concurrency (the enforcement loop runs on the generic pool) and search execution (the tasks it tracks are search tasks). Circuit breakers are the backstop below all of this: circuit breakers and memory.

Three admission layers, three different jobs

The single most important thing to hold in your head: WLM, search backpressure, and circuit breakers are three distinct layers that fire at different times, at different granularities, for different reasons. Contributors constantly conflate them.

Circuit breakers are blunt and per-operation. SBP is smart but group-blind — it cancels the heaviest task on a stressed node regardless of who owns it. WLM is the only layer that understands tenancy: it can reject or cancel a group that is over its own limit even while the node as a whole is healthy (in ENFORCED mode), which is exactly what isolation requires.

flowchart TD
    Req["search request"] --> Tag["WLM: tag into a workload group"]
    Tag --> Adm{"WLM admission: group over its share?"}
    Adm -->|yes, ENABLED| Rej["reject: OpenSearchRejectedExecutionException"]
    Adm -->|no| Run["execute; task tracked"]
    Run --> WLMc{"WLM loop: group over limit / node in duress?"}
    WLMc -->|yes| Cancel["cancel worst tasks in that group"]
    Run --> SBP{"SBP: node in duress?"}
    SBP -->|yes| SBPc["cancel worst task node-wide"]
    Run --> CB{"CB: allocation over heap limit?"}
    CB -->|yes| CBx["CircuitBreakingException"]

The three do not race blindly — WLM and SBP explicitly divide labor (covered in Composition below).

The data model: workload groups

cd ~/src/OpenSearch
sed -n '1,60p' server/src/main/java/org/opensearch/cluster/metadata/WorkloadGroup.java
grep -n "ResiliencyMode\|SOFT\|ENFORCED\|MONITOR\|validateResourceLimits" \
  server/src/main/java/org/opensearch/wlm/MutableWorkloadGroupFragment.java

A WorkloadGroup (in org.opensearch.cluster.metadata) is a named, cluster-state object. It is stored as a Metadata custom (WorkloadGroupMetadata) so it replicates to every node like any other cluster metadata — grep server/src/main/java/org/opensearch/cluster/metadata/Metadata.java for workloadGroups(). Its schema, from the class Javadoc:

{
  "_id": "fafjafjkaf9ag8a9ga9g7ag0aagaga",
  "name": "analytics",
  "resource_limits": { "cpu": 0.2, "memory": 0.4 },
  "resiliency_mode": "enforced",
  "updated_at": 4513232415
}

The mutable, user-editable part lives in MutableWorkloadGroupFragment. Two fields carry the whole enforcement story:

• resource_limits — a Map<ResourceType, Double> where each value is a fraction of the node, validated to 0 < v <= 1.0 (validateResourceLimits). cpu: 0.2 means "this group may use up to 20% of the node's CPU." The tracked resources are CPU and MEMORY only:

  grep -n "CPU\|MEMORY\|NATIVE_MEMORY\|statsEnabled\|getSortedValues" \
    server/src/main/java/org/opensearch/wlm/ResourceType.java
  grep -n "TRACKED_RESOURCES" \
    server/src/main/java/org/opensearch/wlm/tracker/WorkloadGroupResourceUsageTrackerService.java
  

  ResourceType (annotated @PublicApi(since = "2.17.0")) also declares NATIVE_MEMORY, but note the comment in the enum: it is not tracked per-group (statsEnabled = false, node threshold pinned at 1.0) — it exists to flow an off-heap duress signal to search backpressure. TRACKED_RESOURCES is EnumSet.of(CPU, MEMORY).

• resiliency_mode — one of SOFT, ENFORCED, MONITOR, defined on MutableWorkloadGroupFragment.ResiliencyMode. Read the enum's own Javadoc; it is the clearest statement of intent in the codebase:

  // MutableWorkloadGroupFragment.ResiliencyMode
  // SOFT     - may exceed its limits while the node is NOT in duress
  // ENFORCED - will never breach; cancels as soon as limits are breached
  // MONITOR  - never cancels; only logs the tasks that WOULD be cancelled
  public enum ResiliencyMode { SOFT("soft"), ENFORCED("enforced"), MONITOR("monitor"); }
  

Two different "modes" — do not confuse them

There is a per-group ResiliencyMode (above) and a node-wide master switch WlmMode. They are different enums with different jobs:

grep -n "ENABLED\|MONITOR_ONLY\|DISABLED" server/src/main/java/org/opensearch/wlm/WlmMode.java
grep -n "DEFAULT_WLM_MODE\|WLM_MODE_SETTING" \
  server/src/main/java/org/opensearch/wlm/WorkloadManagementSettings.java

Both gates must line up before anything is actually rejected or cancelled: rejection only happens when WlmMode == ENABLED, and cancellation only executes (rather than merely logging) when WlmMode == ENABLED. The out-of-the-box default is monitor_only, so a fresh cluster tracks and logs but never kills a query until an operator opts in — verify in WorkloadManagementSettings (DEFAULT_WLM_MODE = "monitor_only").

How a request is tagged into a group

WLM enforcement is worthless if it cannot tell which group a query belongs to. Tagging happens at request ingress and rides a thread-context header the whole way down.

cd ~/src/OpenSearch
grep -rn "WORKLOAD_GROUP_ID_HEADER\|DEFAULT_WORKLOAD_GROUP" \
  server/src/main/java/org/opensearch/wlm/WorkloadGroupTask.java
grep -rn "WORKLOAD_GROUP_ID_HEADER\|putHeader" \
  plugins/workload-management/src/main/java/org/opensearch/plugin/wlm/AutoTaggingActionFilter.java \
  server/src/main/java/org/opensearch/action/ActionModule.java

The header name is a constant on WorkloadGroupTask:

// WorkloadGroupTask
public static final String WORKLOAD_GROUP_ID_HEADER = "workloadGroupId";
public static final Supplier<String> DEFAULT_WORKLOAD_GROUP_ID_SUPPLIER = () -> "DEFAULT_WORKLOAD_GROUP";

There are two ways the header gets set:

1. Explicitly — a client sends the workloadGroupId HTTP header. It is a registered REST header (grep RestHeaderDefinition(WorkloadGroupTask.WORKLOAD_GROUP_ID_HEADER, false) in ActionModule — false = not allowed multiple).
2. Automatically — the workload-management plugin's AutoTaggingActionFilter evaluates rules (attribute → group) and, on a match, putHeader(WORKLOAD_GROUP_ID_HEADER, label). This is the rule-based auto-tagging that lets you route by index pattern, username, etc., without touching clients.

Anything untagged falls into the sentinel DEFAULT_WORKLOAD_GROUP, which is never rejected or cancelled — it is the escape hatch for traffic no rule matched.

Once the header exists, WorkloadManagementTransportInterceptor copies it onto the task and runs the admission check:

// WorkloadManagementTransportInterceptor.RequestHandler#messageReceived
if (isSearchWorkloadRequest(task)) {                       // task instanceof WorkloadGroupTask
    ((WorkloadGroupTask) task).setWorkloadGroupId(threadPool.getThreadContext());
    final String workloadGroupId = ((WorkloadGroupTask) task).getWorkloadGroupId();
    workloadGroupService.rejectIfNeeded(workloadGroupId);  // <-- admission gate
}
actualHandler.messageReceived(request, channel, task);

WorkloadGroupTask extends CancellableTask — the same task abstraction search backpressure cancels — which is exactly why the two layers can share tracking infrastructure. The interceptor is wired in Node.java; grep WorkloadManagementTransportInterceptor there to see it join the transport interceptor chain. A parallel entry point, WorkloadGroupRequestOperationListener#onRequestStart, runs the same rejectIfNeeded on the coordinator's search path and applies the group's per-request search settings (timeout, max_concurrent_shard_requests, …).

sequenceDiagram
    participant C as Client
    participant AF as AutoTaggingActionFilter
    participant TC as ThreadContext
    participant I as WLM TransportInterceptor
    participant S as WorkloadGroupService
    C->>AF: search (header, or matched by rule)
    AF->>TC: putHeader("workloadGroupId", group)
    I->>TC: read header -> task.setWorkloadGroupId()
    I->>S: rejectIfNeeded(groupId)
    alt group over share AND WlmMode==ENABLED
        S-->>C: OpenSearchRejectedExecutionException
    else ok
        I->>I: actualHandler.messageReceived (execute)
    end

Resource tracking: whose task burned what

WLM does not invent its own accounting — it reuses the task resource tracking that search backpressure relies on.

cd ~/src/OpenSearch
sed -n '1,90p' server/src/main/java/org/opensearch/wlm/tracker/WorkloadGroupResourceUsageTrackerService.java
grep -rn "getResourceAwareTasks\|refreshResourceStats" \
  server/src/main/java/org/opensearch/tasks/TaskResourceTrackingService.java

WorkloadGroupResourceUsageTrackerService.constructWorkloadGroupLevelUsageViews() pulls every resource-aware task from the shared TaskResourceTrackingService, filters to WorkloadGroupTasks that have a group set, groups them by workloadGroupId, refreshes their stats, and sums CPU + memory per group into a WorkloadGroupLevelResourceUsageView:

// WorkloadGroupResourceUsageTrackerService
taskResourceTrackingService.refreshResourceStats(tasksForGroup.toArray(new WorkloadGroupTask[0]));
for (ResourceType resourceType : TRACKED_RESOURCES) {          // CPU, MEMORY
    double usage = resourceType.getResourceUsageCalculator().calculateResourceUsage(tasksForGroup);
    workloadGroupUsage.put(resourceType, usage);
}

Each ResourceType owns a pluggable ResourceUsageCalculator (CpuUsageCalculator, MemoryUsageCalculator) — grep the tracker/ package. This is the seam you extend to teach WLM about a new resource.

The enforcement path: reject early, or cancel late

WLM enforces at two moments, and the distinction mirrors the reads/writes split you already saw in backpressure:

• Rejection happens at admission (rejectIfNeeded), before the query runs — cheap, based on the group's last recorded usage.
• Cancellation happens during execution, on a periodic loop (WorkloadGroupTaskCancellationService.cancelTasks) — because an expensive query is only known to be expensive after it has started.

Rejection at admission

grep -n "rejectIfNeeded\|getNormalisedRejectionThreshold\|OpenSearchRejectedExecutionException\|isNodeInDuress" \
  server/src/main/java/org/opensearch/wlm/WorkloadGroupService.java

// WorkloadGroupService#rejectIfNeeded (trimmed)
if (workloadManagementSettings.getWlmMode() != WlmMode.ENABLED) return;
if (workloadGroupId == null || workloadGroupId.equals(DEFAULT_WORKLOAD_GROUP)) return;
// SOFT groups are NOT rejected unless the node itself is in duress:
if (group.getResiliencyMode() == ResiliencyMode.SOFT && !nodeDuressTrackers.isNodeInDuress()) return;
for (ResourceType resourceType : TRACKED_RESOURCES) {
    double threshold = getNormalisedRejectionThreshold(limit, resourceType);   // limit * node rejection threshold
    double lastRecordedUsage = state.getResourceState().get(resourceType).getLastRecordedUsage();
    if (threshold < lastRecordedUsage) { reject = true; ...; break; }          // count once, even if both breach
}
if (reject) throw new OpenSearchRejectedExecutionException("WorkloadGroup " + id + " is already contended. ...");

The threshold is normalised: a group's configured limit (say cpu: 0.2) is multiplied by the node-level rejection threshold (default 0.8), so the effective reject line is 0.2 * 0.8 = 0.16. This leaves headroom between "start rejecting" and "start cancelling" (whose node threshold defaults to 0.9). Rejection returns OpenSearchRejectedExecutionException, which surfaces to the client as a retryable 429-class error — the same shed-and-retry contract as backpressure.

Cancellation on the enforcement loop

grep -n "doRun\|scheduleWithFixedDelay\|cancelTasks\|getWorkloadGroupServiceRunInterval" \
  server/src/main/java/org/opensearch/wlm/WorkloadGroupService.java
grep -n "cancelTasks\|ResiliencyMode.ENFORCED\|ResiliencyMode.SOFT\|handleNodeDuress\|taskCancellation.cancel\|MaximumResourceTaskSelectionStrategy" \
  server/src/main/java/org/opensearch/wlm/cancellation/WorkloadGroupTaskCancellationService.java

WorkloadGroupService.doRun() is scheduled every wlm.workload_group.enforcement_interval (default 1000ms) on the generic threadpool, and short-circuits when WlmMode == DISABLED. It delegates to WorkloadGroupTaskCancellationService.cancelTasks, whose policy is the heart of WLM's fairness model:

// WorkloadGroupTaskCancellationService#cancelTasks (trimmed)
workloadGroupLevelResourceUsageViews = resourceUsageTrackerService.constructWorkloadGroupLevelUsageViews();
cancelTasks(ResiliencyMode.ENFORCED, activeWorkloadGroups);      // ALWAYS: enforced groups over limit
handleNodeDuress(isNodeInDuress, activeWorkloadGroups, deletedWorkloadGroups); // only if node in duress:
                                                                 //   1) tasks from deleted groups
                                                                 //   2) SOFT groups over limit
updateResourceUsageInWorkloadGroupState(activeWorkloadGroups);

Two rules to internalize:

1. ENFORCED groups are policed unconditionally. If an enforced group exceeds its normalised cancellation threshold (limit * node cancellation threshold), its worst tasks are cancelled even on a healthy node. That is the isolation guarantee.
2. SOFT groups (and orphaned tasks from deleted groups) are only cancelled when the node is in duress. Soft groups get to borrow idle capacity right up until the node is actually hurting.

Which tasks die is decided by a TaskSelectionStrategy — MaximumResourceTaskSelectionStrategy picks the biggest consumers first, just enough to bring the group back under its share (excessUsage), avoiding double-counting already-selected tasks. And the final gate: cancellation only calls taskCancellation.cancel() when WlmMode == ENABLED; in MONITOR_ONLY it merely logs "eligible for cancellation" — your dry-run before you flip enforcement on.

flowchart TD
    Loop["doRun() every enforcement_interval (generic pool)"] --> M{WlmMode == DISABLED?}
    M -->|yes| Stop["no-op"]
    M -->|no| Views["build per-group CPU/MEM usage views"]
    Views --> Enf["ENFORCED groups over limit -> select worst tasks"]
    Views --> Duress{node in duress?}
    Duress -->|yes| Soft["also: deleted-group tasks + SOFT groups over limit"]
    Duress -->|no| Skip["leave SOFT groups alone"]
    Enf --> Act{WlmMode == ENABLED?}
    Soft --> Act
    Act -->|yes| Kill["taskCancellation.cancel()"]
    Act -->|no| Log["MONITOR_ONLY: log only"]

Settings

All keys use the wlm.workload_group.* namespace (the Java constants keep a legacy QUERYGROUP_ name — the key strings are current). Confirm keys and defaults in your checkout:

grep -n "SETTING_NAME\|doubleSetting\|longSetting\|intSetting\|DEFAULT_" \
  server/src/main/java/org/opensearch/wlm/WorkloadManagementSettings.java
grep -n "WorkloadManagementSettings\." \
  server/src/main/java/org/opensearch/common/settings/ClusterSettings.java

Two invariants enforced in WorkloadManagementSettings: the cancellation threshold must be >= the rejection threshold (ensureRejectionThresholdIsLessThanCancellation), and the maxima above exist so an operator cannot set a threshold so high it invites a node drop. A group also carries a per-request settings block (WorkloadGroupSearchSettings: search.default_search_timeout, search.cancel_after_time_interval, search.max_concurrent_shard_requests, search.batched_reduce_size, search.max_buckets, and override_request_values), applied by WorkloadGroupRequestOperationListener.

REST API

CRUD lives in the workload-management plugin; stats live in server core. Grep the real routes so you never guess a path:

grep -rn "new Route" plugins/workload-management/src/main/java/org/opensearch/plugin/wlm/rest/
grep -n "new Route" server/src/main/java/org/opensearch/rest/action/admin/cluster/RestWlmStatsAction.java

# Create a group, then watch it enforce.
curl -s -XPUT 'localhost:9200/_wlm/workload_group/' -H 'Content-Type: application/json' -d '{
  "name": "analytics",
  "resiliency_mode": "enforced",
  "resource_limits": { "cpu": 0.2, "memory": 0.4 }
}'
# Turn WLM on (default is monitor_only -> tracks but never kills):
curl -s -XPUT 'localhost:9200/_cluster/settings' -H 'Content-Type: application/json' \
  -d '{"persistent":{"wlm.workload_group.mode":"enabled"}}'
# Route a search into it and read the counters:
curl -s -H 'workloadGroupId: <the _id from create>' 'localhost:9200/big-index/_search' -d '...'
curl -s 'localhost:9200/_wlm/stats?pretty'

Stats keys come from WorkloadGroupStats: per group total_completions, total_rejections, total_cancellations, failures; and per resource current_usage, cancellations, rejections (grep WorkloadGroupStats.java for total_cancellations / current_usage).

How it composes with backpressure and breakers

WLM and search backpressure both cancel search tasks under duress, so they must not step on each other. The tie-breaker is WorkloadGroupService.shouldSBPHandle:

grep -n "shouldSBPHandle\|NodeDuressTrackers\|NATIVE_MEMORY" \
  server/src/main/java/org/opensearch/wlm/WorkloadGroupService.java

// WorkloadGroupService#shouldSBPHandle (trimmed)
// SBP handles the task only when WLM ISN'T actively owning it:
return workloadManagementSettings.getWlmMode() != WlmMode.ENABLED || isInvalidWorkloadGroupTask;

So when WLM is ENABLED and the task belongs to a real (non-default) group, WLM owns it and SBP stands down; otherwise SBP takes over. They also share signals: WLM builds NodeDuressTrackers (CPU via ProcessProbe, heap via JvmStats) that feed both, and the NATIVE_MEMORY ResourceType exists purely to route an off-heap duress signal into SBP. Circuit breakers sit underneath all of it, unaware of groups — the last line before OOM. The mental model: WLM decides fairness between tenants, SBP protects the node as a whole, breakers protect the heap.

Where a contributor would touch it

grep -n "WorkloadGroupService\|WorkloadGroupTaskCancellationService\|WorkloadGroupResourceUsageTrackerService\|WorkloadManagementTransportInterceptor" \
  server/src/main/java/org/opensearch/node/Node.java

• Add a tracked resource (e.g. real per-group native memory): extend ResourceType, supply a ResourceUsageCalculator, and add it to TRACKED_RESOURCES — the TODO in WorkloadGroupResourceUsageTrackerService marks the exact spot.
• Change which tasks are cancelled: implement a new TaskSelectionStrategy alongside MaximumResourceTaskSelectionStrategy.
• New REST/transport behavior: the plugins/workload-management module holds the CRUD actions (TransportCreateWorkloadGroupAction, WorkloadGroupPersistenceService, the Rest*WorkloadGroupAction classes) and the auto-tagging rule engine (AutoTaggingActionFilter, WorkloadGroupRuleRoutingService).
• Node wiring: Node.java constructs the tracker, cancellation service, WorkloadGroupService, the interceptor, and the operation listener — the whole graph in one place.

Common bugs and symptoms

Validation: prove you understand this

1. Draw the three admission layers (breakers, SBP, WLM) and, for each, name the granularity, the exception it throws, and the failure it prevents.
2. Explain the difference between WlmMode and ResiliencyMode, and give the exact condition under which a task is actually cancelled (not just logged).
3. A group with resiliency_mode: soft is 3x over its CPU limit on a node that is otherwise idle. Is any task cancelled? Why or why not — cite the code path.
4. Trace one search from ingress to admission: which component sets the workloadGroupId header, which copies it onto the task, and which method decides to reject.
5. Given cpu: 0.2 and default thresholds, compute the effective CPU fractions at which (a) admission rejects and (b) the loop cancels, and explain why rejection is set lower.
6. Explain how WLM and search backpressure avoid cancelling the same task twice, naming the method and the condition.
7. You want WLM to enforce a new resource. List every file/seam you must change and why TRACKED_RESOURCES matters.

Next: Backpressure and Admission Control for the node-level companion, or Circuit Breakers and Memory for the heap-level backstop.