Security — Intensive

Almost every other masterclass in this book treats a request as if it arrives already trusted: the action framework dispatches it, the storage engine writes it, the query engine runs it. This chapter is about the layer that decides whether the request is allowed to exist at all, and — if it is — which bytes the caller is permitted to see.

That layer is the Security plugin: opensearch-project/security, Java package org.opensearch.security. It is not part of server/; it is a separate plugin that ships in the default distribution. It is also the single most instructive plugin in the ecosystem, because to do its job it has to hook every extension point you learned about elsewhere — REST, transport, action filters, the search path at the shard, the cluster-state lifecycle — and it stores its own config in a system index instead of in opensearch.yml. If you understand Security, you understand the plugin SPI (plugin-architecture) better than most committers.

This chapter builds it up from first principles:

1. Where it plugs in — the four interception seams (SecurityRestFilter, SecurityInterceptor, SecurityFilter, the search-time IndexSearcherWrapper).
2. The config system index — why config lives in .opendistro_security and how securityadmin.sh loads YAML into it.
3. Authentication (authn) — BackendRegistry, the authc chain, the backends, the ThreadContext user injection.
4. Authorization (authz) — roles, action groups, role mappings, the PrivilegesEvaluator per-action decision.
5. DLS / FLS / field-masking — DlsFlsValveImpl, the security IndexSearcherWrapper, DlsQueryParser.
6. TLS — the two layers (transport 9300, http 9200) and their settings.
7. Audit — AuditLog.

Note: "cluster manager" is the current name for what was historically called the master node — the elected node that owns cluster-state changes. Security hooks the cluster-state lifecycle (it is a ClusterPlugin) to reload its config index, so the term shows up here.

The labs that go with this chapter:

• Lab SE1: Authentication and TLS — bring up a secured node with the demo config, authenticate with HTTP Basic, inspect the authc chain, and configure transport + http TLS.
• Lab SE2: Authorization, DLS, and FLS — define a role, map a user, prove a 403, then add a DLS filter, FLS field hiding, and field masking and verify each as different users.
• Lab SE3: Build a Security Extension — trace and extend a real security SPI (HTTPAuthenticator / principal extractor) with Java, a unit test, and a contribution walkthrough.

First principles: security is interception, not a wall

There is no single "security gate" in OpenSearch. A request does not hit one firewall and then run free. Instead, Security inserts itself at four distinct seams, each of which you already met as a generic extension point. The whole plugin is the cooperation of those four hooks plus a config store.

flowchart TD
    Client[client request] --> TLSHTTP["http TLS (9200): SecuritySSLNettyTransport"]
    TLSHTTP --> RF["SecurityRestFilter (REST edge)\nauthn happens here"]
    RF --> NC["NodeClient.execute(ActionType, req)"]
    NC --> AF["ActionFilters chain"]
    AF --> SF["SecurityFilter (ActionFilter)\nauthz happens here (PrivilegesEvaluator)"]
    SF --> TA["TransportAction.doExecute"]
    TA -->|node to node| SI["SecurityInterceptor (transport TLS 9300 + ThreadContext propagation)"]
    SI --> Shard["shard-level op"]
    Shard --> ISW["security IndexSearcherWrapper\nDLS/FLS/masking applied here"]
    ISW --> Lucene["Lucene search"]

Read that diagram top to bottom and notice where each decision is made:

This is the single most important mental model in the chapter. Authentication happens at the REST edge; authorization happens at the action filter; data visibility happens at the shard. Three different layers, three different classes. A bug in "user can log in but every search is empty" is a DLS/wrapper bug, not an authn bug — because authn already succeeded two layers up.

# In a security plugin checkout (git clone https://github.com/opensearch-project/security):
grep -rn "implements .*NetworkPlugin\|implements .*ActionPlugin\|implements .*ClusterPlugin" \
  src/main/java/org/opensearch/security/OpenSearchSecurityPlugin.java
grep -n "getActionFilters\|getRestHandlerWrapper\|getTransportInterceptors\|onIndexModule\|getHttpTransports" \
  src/main/java/org/opensearch/security/OpenSearchSecurityPlugin.java

Note: Exact class and method names drift across major versions (2.x → 3.x). Everywhere in this chapter, treat the grep as the source of truth and the quoted name as a signpost. The shape — four seams + a config index — is stable.

The config system index and securityadmin.sh

Here is the design decision that surprises everyone: Security does not read its roles, users, and mappings from opensearch.yml. It reads them from a system index — historically .opendistro_security, newer builds .opensearch_security — where each config file is stored as a single document.

Why an index and not a file? Because config must be cluster-wide and live. A role change must take effect on every node without a rolling restart, and it must survive node replacement. An index gives you replication, durability, and a single source of truth for free. The cost is a bootstrapping problem: you cannot use the REST API to write the config that governs the REST API before any config exists. That is what securityadmin.sh solves — it talks to the transport port (9300) using an admin TLS certificate, bypassing the normal authn/authz path, and writes the YAML directly into the index.

flowchart LR
    YAML["config/opensearch-security/*.yml"] --> SA["securityadmin.sh\n(admin cert on 9300)"]
    SA --> Idx[".opendistro_security index\n(1 doc per config type)"]
    Idx -->|cluster-state listener reloads| Cache["in-memory ConfigModel\n(roles, mappings, action groups)"]
    Cache --> PE["PrivilegesEvaluator / BackendRegistry read this"]

The seven YAML config files (one document each in the index):

# Apply YAML config into the index (run from the plugin's tools/ dir on a node):
./securityadmin.sh \
  -cd ../../../config/opensearch-security/ \
  -icl -nhnv \
  -cacert  ../../../config/root-ca.pem \
  -cert    ../../../config/kirk.pem \
  -key     ../../../config/kirk-key.pem
# -cd: config dir of YAMLs  -icl: ignore cluster name  -nhnv: no hostname verification
# kirk.pem is the demo ADMIN cert; its DN must be listed in plugins.security.authcz.admin_dn

Warning: The admin certificate is god-mode. Any cert whose DN appears in plugins.security.authcz.admin_dn can rewrite the entire security config over the transport port. Treat it like a root SSH key. A "my role changes don't take effect" symptom is almost always securityadmin wrote to the index but the node's admin_dn / cert chain rejected it — covered in the Lab SE1 troubleshooting.

REST admin API mirrors all of this once the cluster is up: /_plugins/_security/api/internalusers, /_plugins/_security/api/roles, /_plugins/_security/api/rolesmapping, /_plugins/_security/api/actiongroups, /_plugins/_security/authinfo (who am I?).

grep -rn "opensearch_security\|opendistro_security\|securityconfig\|CONFIG_INDEX\|getConfigIndexName" \
  src/main/java/org/opensearch/security/support/ConfigConstants.java

Authentication: BackendRegistry and the authc chain

Authentication answers who is the caller? It runs at the REST edge, inside SecurityRestFilter, which delegates to org.opensearch.security.auth.BackendRegistry.

BackendRegistry walks an ordered authc chain defined in config.yml. Each link is one HTTP authenticator (extracts a credential from the request) paired with one or more authentication backends (validates the credential and returns the user's identity + backend roles). The chain tries authenticators in order until one succeeds.

flowchart TD
    Req["HTTP request on 9200"] --> BR["BackendRegistry.authenticate"]
    BR --> A1{"HTTPBasicAuthenticator\n(Authorization: Basic ...)"}
    A1 -->|creds found| B1["internal_users backend\nor LDAP backend"]
    A1 -->|no creds| A2{"HTTPJwtAuthenticator\n(Bearer token)"}
    A2 -->|valid JWT| Done
    A2 -->|none| A3{"OIDC / SAML / Kerberos / PKI"}
    B1 -->|valid| Done["User built:\nname + backend roles"]
    B1 -->|invalid| Fail["401 Unauthorized\n(WWW-Authenticate)"]
    Done --> TC["inject into ThreadContext\ntransient _opendistro_security_user"]

The authenticators (HTTP-side credential extraction)

The backends (credential validation)

A real config.yml authc block (HTTP Basic against internal users, then JWT):

authc:
  basic_internal_auth_domain:
    description: "Authenticate via HTTP Basic against internal users"
    http_enabled: true
    transport_enabled: true
    order: 0
    http_authenticator:
      type: basic
      challenge: true
    authentication_backend:
      type: internal
  jwt_auth_domain:
    http_enabled: true
    order: 1
    http_authenticator:
      type: jwt
      challenge: false
      config:
        signing_key: "base64-encoded-HMAC-or-RSA-public-key"
        jwt_header: "Authorization"
        subject_key: "preferred_username"
        roles_key: "roles"
    authentication_backend:
      type: noop

The crucial output: user in the ThreadContext

Once a chain link succeeds, BackendRegistry builds a User and stores it in the request's ThreadContext as a transient header, _opendistro_security_user. This is the load-bearing fact of the whole plugin: every downstream layer — the action filter doing authz, the search wrapper doing DLS/FLS — reads the authenticated user out of the ThreadContext, not off the request. And because ThreadContext headers are propagated across the transport wire by SecurityInterceptor, the user identity travels with the request to other nodes (coordinator → data node) without re-authenticating.

grep -rn "OPENDISTRO_SECURITY_USER\|_opendistro_security_user\|threadContext.putTransient\|class BackendRegistry\|authenticate(" \
  src/main/java/org/opensearch/security/auth/BackendRegistry.java \
  src/main/java/org/opensearch/security/support/ConfigConstants.java | head -30

Note: "transient" ThreadContext headers are not serialized to the wire by default; Security explicitly arranges for the user to cross nodes via SecurityInterceptor, which serializes the identity into a header the receiving node trusts (it arrives on the already-TLS-authenticated transport channel). This is why transport TLS is mandatory when security is on — node-to-node trust is the substrate the propagated identity rides on.

Authorization: roles, action groups, and PrivilegesEvaluator

Authorization answers may this user perform this action on these indices? It runs in SecurityFilter, an ActionFilter on the transport-action path (recall from action-framework that every action flows through the ActionFilters chain). SecurityFilter delegates the decision to org.opensearch.security.privileges.PrivilegesEvaluator.

The model has four moving parts:

A role granting read on orders-* and using the READ action group:

# roles.yml
orders_reader:
  cluster_permissions:
    - "cluster:monitor/health"
  index_permissions:
    - index_patterns:
        - "orders-*"
      allowed_actions:
        - "READ"            # an action group expanding to indices:data/read/*

# roles_mapping.yml
orders_reader:
  users:
    - "alice"
  backend_roles:
    - "ops-team"            # e.g. an LDAP group

How PrivilegesEvaluator decides, per action

sequenceDiagram
    participant SF as SecurityFilter (ActionFilter)
    participant TC as ThreadContext
    participant PE as PrivilegesEvaluator
    participant CM as ConfigModel (roles + mappings)
    SF->>TC: read User (_opendistro_security_user)
    SF->>PE: evaluate(user, action="indices:data/read/search", request)
    PE->>CM: map user -> set of roles (via roles_mapping)
    PE->>CM: expand action groups -> concrete actions per role
    PE->>PE: resolve request's concrete index names (aliases, wildcards, date-math)
    PE->>PE: does any role permit (action AND every resolved index)?
    alt permitted
        PE-->>SF: ALLOW (+ DLS/FLS rules to apply downstream)
    else not permitted
        PE-->>SF: DENY -> OpenSearchSecurityException -> 403
    end

The subtle, bug-prone part is index resolution: the request might target an alias, a wildcard (orders-*), or date-math (<orders-{now/d}>), and the evaluator must resolve those to concrete index names before checking each against the role's index_patterns. A "should be allowed but I get 403" bug is very often an alias or wildcard that resolves to an index the role's pattern doesn't cover.

grep -rn "class PrivilegesEvaluator\|evaluate(\|class SecurityFilter\|apply(\|resolveIndexPatterns\|ConfigModel" \
  src/main/java/org/opensearch/security/privileges/PrivilegesEvaluator.java \
  src/main/java/org/opensearch/security/filter/SecurityFilter.java | head -30

Note: Cluster-level actions (cluster:*) check cluster_permissions; index-level actions (indices:*) check index_permissions against resolved index names. The action name string is the join key — the same "indices:data/read/search" you saw registered in ActionModule. Authz is literally "is this action name, on these indices, in the union of this user's roles' allowed actions?".

DLS, FLS, and field masking: filtering at the shard

Authz is binary: you can search orders-* or you can't. Document-Level Security (DLS), Field-Level Security (FLS), and field masking are finer: they let a user run the search but transparently restrict which documents and which fields come back. This cannot happen at the action filter — it has to happen inside Lucene, at the shard, per segment. So Security wraps the reader.

The orchestrator is org.opensearch.security.configuration.DlsFlsValveImpl. It runs early in the search path, reads the user's effective DLS/FLS rules from the resolved roles, and installs them so the security IndexSearcherWrapper (an IndexModule reader wrapper) enforces them per shard.

flowchart TD
    Search["search request reaches the shard"] --> Valve["DlsFlsValveImpl\n(reads user's DLS/FLS from roles)"]
    Valve --> Wrap["security IndexSearcherWrapper\nwraps the DirectoryReader / IndexSearcher"]
    Wrap --> DLS["DLS: DlsQueryParser parses the role's\nDLS query -> inject as a filter clause"]
    Wrap --> FLS["FLS: filter FieldInfos +\nFieldVisitor drops excluded fields"]
    Wrap --> Mask["masking: hash field values on read"]
    DLS --> Lucene["Lucene IndexSearcher.search\n(only matching docs)"]
    FLS --> Lucene
    Mask --> Lucene

DLS: a query filter injected per role

A DLS role says "only docs where region == EU":

# roles.yml
eu_reader:
  index_permissions:
    - index_patterns: ["orders-*"]
      dls: '{"term": {"region": "EU"}}'      # injected as a mandatory filter
      allowed_actions: ["READ"]

org.opensearch.security.configuration.DlsQueryParser parses that JSON into a Lucene Query, and the wrapper ANDs it into every search this user runs on orders-*. The user cannot see non-EU docs even with a match_all — the filter is enforced inside the searcher, below where the user's query is applied. (If multiple roles grant overlapping access, the DLS queries are combined with OR — the user sees the union of what any role permits, which is the correct, permissive semantics.)

FLS: include or exclude fields

eu_reader:
  index_permissions:
    - index_patterns: ["orders-*"]
      fls:
        - "~ssn"          # leading ~ = EXCLUDE this field
        - "~card_number"
      allowed_actions: ["READ"]

FLS rewrites _source and filters FieldInfos so excluded fields never reach the response — not in _source, not in doc-values aggregations, not in stored-field fetches. Include-mode (fls: ["region","total"] with no ~) is the inverse: only the listed fields survive.

Field masking: hash instead of hide

eu_reader:
  index_permissions:
    - index_patterns: ["orders-*"]
      masked_fields:
        - "email::SHA-256"     # show a stable hash, not the cleartext
      allowed_actions: ["READ"]

Masking is the middle ground between "show it" (no rule) and "hide it" (FLS exclude): the field still appears, but its value is a one-way hash, so a user can correlate on it (same email → same hash) without seeing the cleartext.

grep -rn "class DlsFlsValveImpl\|class DlsQueryParser\|IndexSearcherWrapper\|maskedFields\|fls\|getDlsQuery\|FieldReader\|FieldInfos" \
  src/main/java/org/opensearch/security/configuration/DlsFlsValveImpl.java \
  src/main/java/org/opensearch/security/configuration/DlsQueryParser.java | head -40

Warning: DLS interacts with caching. The shard request cache and query cache key must include the DLS query (or be disabled for DLS users), or user A could see a cached result computed for user B's broader access. Security handles this, but it is the reason DLS has a performance cost and why a "user sees too many docs intermittently" bug points straight at a cache-key omission. See tiered-caching for the cache layers DLS must be careful around.

TLS: two layers, two settings namespaces

When security is on, transport TLS is mandatory and http TLS is strongly recommended. They are configured independently because they protect different traffic:

# opensearch.yml — transport TLS (mandatory)
plugins.security.ssl.transport.pemcert_filepath:   esnode.pem
plugins.security.ssl.transport.pemkey_filepath:    esnode-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: root-ca.pem
plugins.security.ssl.transport.enforce_hostname_verification: false

# http/REST TLS (recommended)
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath:   esnode.pem
plugins.security.ssl.http.pemkey_filepath:    esnode-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: root-ca.pem

# the admin cert(s) allowed to run securityadmin.sh over transport
plugins.security.authcz.admin_dn:
  - "CN=kirk,OU=client,O=client,L=test,C=de"

Transport TLS does double duty: it encrypts node traffic and establishes the node-to-node trust that lets SecurityInterceptor carry the authenticated user across nodes without re-auth. That is why you cannot run security with transport TLS disabled — there would be no authenticated channel to propagate identity over. Lab SE1 brings both layers up and demonstrates a bad-cert handshake failure.

grep -rn "ssl.transport\|ssl.http\|SSL_TRANSPORT_ENABLED\|SSL_HTTP_ENABLED\|pemcert_filepath\|admin_dn" \
  src/main/java/org/opensearch/security/ssl/util/SSLConfigConstants.java \
  src/main/java/org/opensearch/security/support/ConfigConstants.java | head -30

Audit logging: AuditLog

Every authn/authz decision and (optionally) every read/write can be recorded by org.opensearch.security.auditlog.AuditLog. Audit answers the compliance question who did what, when, and was it allowed? — distinct from authz, which makes the decision.

Sinks are pluggable: an audit index (auditlog-*), a file, an external endpoint. AuditLog is invoked from inside SecurityFilter and SecurityRestFilter, so it sees exactly the same allow/deny decisions described above.

grep -rn "interface AuditLog\|enum Origin\|logFailedLogin\|logMissingPrivileges\|logGrantedPrivileges\|AuditCategory" \
  src/main/java/org/opensearch/security/auditlog/AuditLog.java | head

End-to-end: one search through every seam

Trace GET orders-2026/_search as user alice (mapped to eu_reader with the DLS filter region:EU and ~ssn FLS) on a TLS-enabled cluster:

sequenceDiagram
    participant C as curl -u alice:... --cacert
    participant H as http transport (9200, TLS)
    participant RF as SecurityRestFilter
    participant BR as BackendRegistry
    participant SF as SecurityFilter
    participant PE as PrivilegesEvaluator
    participant DN as data node (9300, TLS)
    participant V as DlsFlsValveImpl + wrapper
    C->>H: TLS handshake (http cert)
    H->>RF: decrypted request
    RF->>BR: authenticate (Basic)
    BR->>BR: internal backend, bcrypt check OK
    BR->>RF: User(alice, roles via mapping) -> ThreadContext
    RF->>SF: dispatch search action via NodeClient
    SF->>PE: evaluate(alice, indices:data/read/search, [orders-2026])
    PE->>SF: ALLOW + DLS(region:EU) + FLS(~ssn)
    SF->>DN: transport (TLS), user propagated by SecurityInterceptor
    DN->>V: search at shard
    V->>V: inject region:EU filter; drop ssn from FieldInfos
    V->>C: only EU docs, no ssn field

Every box in that diagram is one of the four seams plus the config. If you can draw it from memory and name the class at each box, you understand the plugin.

Trade-offs: when each feature helps vs hurts

Common bugs and symptoms

Reading exercise

# 0. Get the source:
git clone https://github.com/opensearch-project/security && cd security

# 1. The four seams — where the plugin hooks core.
grep -n "getActionFilters\|getRestHandlerWrapper\|getTransportInterceptors\|onIndexModule" \
  src/main/java/org/opensearch/security/OpenSearchSecurityPlugin.java

# 2. Authn: the registry and the user injection.
grep -rn "class BackendRegistry\|authenticate(\|putTransient\|_opendistro_security_user" \
  src/main/java/org/opensearch/security/auth/BackendRegistry.java \
  src/main/java/org/opensearch/security/support/ConfigConstants.java

# 3. Authz: the evaluator and the filter.
grep -rn "class PrivilegesEvaluator\|evaluate(\|class SecurityFilter\|apply(" \
  src/main/java/org/opensearch/security/privileges/PrivilegesEvaluator.java \
  src/main/java/org/opensearch/security/filter/SecurityFilter.java

# 4. DLS/FLS at the shard.
grep -rn "class DlsFlsValveImpl\|class DlsQueryParser\|IndexSearcherWrapper\|maskedFields" \
  src/main/java/org/opensearch/security/configuration/

# 5. TLS settings.
grep -rn "ssl.transport\|ssl.http\|admin_dn" \
  src/main/java/org/opensearch/security/support/ConfigConstants.java

Answer:

1. Name the four interception seams and, for each, the plugin SPI method it uses and the one question it answers (who/what/which-bytes/encrypt).
2. Why does security config live in a system index instead of opensearch.yml, and how does securityadmin.sh bootstrap it without using the API it governs?
3. Walk the authc chain: what does an authenticator do vs a backend? Where does the authenticated user end up, and how does it cross to another node?
4. How does PrivilegesEvaluator turn a user + action name + index pattern into an allow/deny? What makes index resolution the bug-prone step?
5. Contrast DLS, FLS, and masking mechanically — which injects a query, which filters fields, which hashes — and where (which class) each is enforced.
6. Why is transport TLS mandatory but http TLS only recommended?
7. Give one bug whose symptom is "search returns 0 hits" and explain why it is a DLS bug and not an authn bug.

Validation: prove you understand this

• Draw the four-seam diagram from memory and name the class at each seam (SecurityRestFilter, SecurityInterceptor, SecurityFilter, the IndexSearcherWrapper) plus the config index.
• In Lab SE1, bring up a secured node, authenticate with curl -u, hit /_plugins/_security/authinfo, and explain which authc-chain link matched and where the User was stored.
• Configure transport + http TLS, then trigger a bad-cert handshake failure on 9200 and read the error; explain why transport TLS is non-optional.
• In Lab SE2, produce a deliberate 403, then a DLS filter that hides docs, an FLS rule that hides a field, and a masked field — verifying each as two different users.
• State, for one search, exactly which class makes the authn decision, which makes the authz decision, and which enforces DLS — and that they are three different layers.
• In Lab SE3, trace the HTTPAuthenticator SPI, extend it, write a unit test, and describe how you would contribute it back (DCO, CHANGELOG, PR) — see pr-quality.