Lab SE1: Authentication and TLS

Prerequisite reading: Security — Intensive, specifically the "four seams" model, the authc chain, and the TLS section.

Background

This lab makes the authentication and TLS seams concrete. You will bring up a single secured OpenSearch node using the security demo configuration (which ships demo certificates and a demo internal_users set), authenticate with HTTP Basic, watch the authc chain pick the matching link, inspect both TLS layers (transport 9300 and http 9200), add a new internal user, add a second authenticator (JWT) to the chain, and finally break TLS on purpose to read the handshake error.

Nothing here mutates a real cluster — it is a throwaway demo node you can delete afterward.

Why this matters for contributors

Every bug report that starts with "I can't connect", "401 on everything", or "securityadmin won't apply" is an authn-or-TLS bug, and the security repo's issue tracker is full of them. To triage one you must be able to: read config.yml's authc chain, know which authenticator should have matched, find where the user lands in the ThreadContext, and tell a TLS-handshake failure apart from an authn failure. This lab builds that exact reflex against running code, and points you at the BackendRegistry source so you can read the path your request takes.

Prerequisites

• A JDK 21+ and a local OpenSearch checkout or a downloaded distribution tarball (the security plugin and its tools/ scripts ship in the default distribution).
• curl, openssl, and keytool on PATH.
• Optional but recommended: a clone of the security source to grep — git clone https://github.com/opensearch-project/security.

# Confirm you have a distribution with the security plugin (not a bare server build):
ls plugins/ | grep -i security        # opensearch-security
ls plugins/opensearch-security/tools/  # install_demo_configuration.sh securityadmin.sh

Note: A bare ./gradlew run of server/ has no security plugin. For this lab use a full distribution tarball (opensearch-min plus plugins, or the standard distribution). If you only have the server checkout, download a matching distribution release to get plugins/opensearch-security/.

Step-by-step tasks

Step 1 — Install the demo configuration

The demo installer generates self-signed demo certs (root-ca.pem, esnode.pem, esnode-key.pem, the admin cert kirk.pem), writes the TLS settings into opensearch.yml, and seeds the demo internal_users (including admin).

cd <distribution-root>
# Non-interactive; -y accept, -i initialize, -c cluster-mode off for a single node:
./plugins/opensearch-security/tools/install_demo_configuration.sh -y -i -s

Inspect what it wrote into opensearch.yml:

grep -n "plugins.security" config/opensearch.yml

You should see (abbreviated):

plugins.security.ssl.transport.pemcert_filepath: esnode.pem
plugins.security.ssl.transport.pemkey_filepath: esnode-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: root-ca.pem
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: esnode.pem
plugins.security.ssl.http.pemkey_filepath: esnode-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: root-ca.pem
plugins.security.allow_default_init_securityindex: true
plugins.security.authcz.admin_dn:
  - 'CN=kirk,OU=client,O=client,L=test,C=de'

• Identify the two TLS namespaces (...ssl.transport.* vs ...ssl.http.*) and the admin DN that will be allowed to run securityadmin.sh.

Step 2 — Start the node and confirm both TLS layers are up

./bin/opensearch &        # or run in another terminal
# Wait for it to bind. Then test http TLS (note: https, and we trust the demo CA):
curl -sk -u admin:<demo-admin-password> https://localhost:9200/ | head

-k skips cert verification (demo CA is self-signed). To do it properly — which is the point of TLS — trust the demo CA explicitly and drop -k:

curl -s --cacert config/root-ca.pem -u admin:<demo-admin-password> \
  https://localhost:9200/_cluster/health?pretty

If you get a JSON cluster health back over https, http TLS works. Transport TLS (9300) is proven by the fact the node formed a cluster at all — it cannot without it when security is on.

• Confirm the URL scheme is https, not http. A plain http://localhost:9200 will fail the handshake — that is http TLS doing its job.

Step 3 — Authenticate with HTTP Basic and inspect the chain

Hit the "who am I?" endpoint. This shows exactly what BackendRegistry built for you and stored in the ThreadContext:

curl -s --cacert config/root-ca.pem -u admin:<demo-admin-password> \
  https://localhost:9200/_plugins/_security/authinfo?pretty

{
  "user": "User [name=admin, backend_roles=[admin], requestedTenant=null]",
  "user_name": "admin",
  "backend_roles": ["admin"],
  "roles": ["all_access", "security_rest_api_access"],
  "tenants": { "global_tenant": true, "admin": true },
  "principal": null,
  "remote_address": "127.0.0.1:xxxxx"
}

Read the demo config.yml to see which authc link matched your Basic credential:

grep -n "authc:\|http_authenticator\|type:\|authentication_backend\|order:" \
  config/opensearch-security/config.yml

The basic_internal_auth_domain (order 0, type: basic, backend type: internal) is the one that matched: HTTPBasicAuthenticator extracted admin:... from the Authorization: Basic header, and InternalAuthenticationBackend bcrypt-checked it against internal_users.yml.

• Now send a wrong password and observe the 401:

curl -s -o /dev/null -w "%{http_code}\n" --cacert config/root-ca.pem \
  -u admin:WRONG https://localhost:9200/_plugins/_security/authinfo
# 401

• Read where the user is injected (in a security source checkout):

grep -rn "putTransient\|OPENDISTRO_SECURITY_USER\|class BackendRegistry\|authenticate(" \
  src/main/java/org/opensearch/security/auth/BackendRegistry.java \
  src/main/java/org/opensearch/security/support/ConfigConstants.java | head

You are looking for the line where the authenticated User is put into the ThreadContext under the transient key _opendistro_security_user. Everything downstream (authz, DLS) reads it from there.

Step 4 — Add a new internal user (two ways)

Way A — via the REST admin API (cluster already up):

curl -s --cacert config/root-ca.pem -u admin:<demo-admin-password> \
  -XPUT https://localhost:9200/_plugins/_security/api/internalusers/alice \
  -H 'Content-Type: application/json' -d'
{
  "password": "Alice-Strong-Passw0rd!",
  "backend_roles": ["ops-team"],
  "attributes": { "dept": "fulfillment" }
}'

Verify Alice can authenticate:

curl -s --cacert config/root-ca.pem -u alice:Alice-Strong-Passw0rd! \
  https://localhost:9200/_plugins/_security/authinfo?pretty | head
# user_name: alice, backend_roles: [ops-team], roles: []  (no roles mapped yet — that's Lab SE2)

Way B — via YAML + securityadmin.sh (the production / GitOps way). Edit config/opensearch-security/internal_users.yml, add a bob block with a bcrypt hash, then load it:

# Generate a bcrypt hash for the password:
./plugins/opensearch-security/tools/hash.sh -p 'Bob-Strong-Passw0rd!'
# Paste the hash under bob: in internal_users.yml, then apply the whole config dir:
./plugins/opensearch-security/tools/securityadmin.sh \
  -cd config/opensearch-security/ \
  -icl -nhnv \
  -cacert config/root-ca.pem \
  -cert   config/kirk.pem \
  -key    config/kirk-key.pem

• Confirm bob now authenticates the same way. Note that securityadmin.sh talked to 9300 with the admin cert — it never used a password.

Step 5 — Add a second authenticator (JWT) to the chain

Append a JWT domain after the basic domain in config.yml, so Basic is tried first and JWT second:

# config/opensearch-security/config.yml, under config.dynamic.authc:
    jwt_auth_domain:
      description: "HMAC-signed JWT bearer tokens"
      http_enabled: true
      transport_enabled: false
      order: 1
      http_authenticator:
        type: jwt
        challenge: false
        config:
          signing_key: "<base64 of your HMAC secret>"
          jwt_header: "Authorization"
          subject_key: "sub"
          roles_key: "roles"
      authentication_backend:
        type: noop

Apply it and mint a token to test:

./plugins/opensearch-security/tools/securityadmin.sh -cd config/opensearch-security/ \
  -icl -nhnv -cacert config/root-ca.pem -cert config/kirk.pem -key config/kirk-key.pem

# Mint an HS256 JWT (sub=alice) with your secret, then:
curl -s --cacert config/root-ca.pem \
  -H "Authorization: Bearer $JWT" \
  https://localhost:9200/_plugins/_security/authinfo?pretty | head
# user_name: alice (resolved from the JWT 'sub' claim) — no Basic creds sent

The chain logic: BackendRegistry tries basic_internal_auth_domain first; with no Authorization: Basic header it falls through to jwt_auth_domain, which validates the signature and reads the sub claim. challenge: false on the JWT domain means it will not send a WWW-Authenticate back — important so the two domains don't fight over the 401 challenge.

• Confirm: a request with Basic creds matches link 0; a request with a Bearer token matches link 1; a request with neither gets 401.

Step 6 — Break TLS on purpose

Demonstrate a bad-cert handshake failure so you can recognize it in the wild. Connect with the wrong CA (use the system CA bundle instead of the demo CA):

# Wrong trust store -> handshake fails BEFORE any authn:
curl -v https://localhost:9200/ -u admin:<demo-admin-password> 2>&1 | grep -i "ssl\|certificate\|handshake" | head
# ... SSL certificate problem: self-signed certificate / unable to get local issuer certificate

Contrast the two failure classes — this distinction is the whole point:

• Produce all three: a TLS failure (wrong CA), a 401 (wrong password), and note that 403 comes in Lab SE2.

Deliverables

• The opensearch.yml TLS block, annotated: which lines are transport, which are http, and which is the admin DN.
• The /_plugins/_security/authinfo output for admin and for alice, showing the difference in backend_roles/roles.
• The config.yml authc chain after adding the JWT domain, with the order: values, and a one-line explanation of which link matches Basic vs Bearer.
• Three captured failures: a TLS handshake error, a 401, and the grep line where BackendRegistry injects the user into the ThreadContext.

Troubleshooting

# securityadmin connectivity is a TLS problem 9 times out of 10 — confirm the chain:
openssl s_client -connect localhost:9300 -cert config/kirk.pem -key config/kirk-key.pem \
  -CAfile config/root-ca.pem </dev/null 2>&1 | grep -i "verify\|return code"

Expected output

• https://localhost:9200/_cluster/health returns JSON with --cacert, fails with the wrong CA.
• authinfo for admin shows roles: [all_access, ...]; for a fresh alice shows roles: [] (no mapping yet — that is Lab SE2's job).
• A wrong password yields 401; a wrong CA yields a TLS error with no HTTP status.

Stretch goals

• Add a PKI (clientcert) authenticator: configure a clientcert authc domain and authenticate with curl --cert client.pem --key client-key.pem and no -u. Read HTTPClientCertAuthenticator to see how the cert DN becomes the username.
• Stand up a tiny OIDC provider (e.g. a local Keycloak) and wire an openid authc domain; watch Security fetch the JWKS and validate the ID token. Trace HTTPJwtKeyByOpenIdConnectAuthenticator.
• Turn on audit logging to an index and re-run your 401/200 requests; query auditlog-* and find the FAILED_LOGIN and AUTHENTICATED events.

Coding Exercises

You drove the authc chain and TLS with curl; now make the behaviour executable. These target a clone of opensearch-project/security (git clone … && cd security). Find every class/method with rg first (rg -l "class BackendRegistry" src/main); the security repo's signatures drift, so never trust a stale line number.

1. (warm-up) Unit-test HTTPBasicAuthenticator credential extraction. Find the existing test (rg -l "HTTPBasicAuthenticator" src/test) and add a case that builds a SecurityRequest mock with an Authorization: Basic <base64(admin:secret)> header, calls extractCredentials(...), and asserts the returned AuthCredentials has username admin. Add a second assertion: a request with no Basic header returns null (so the chain falls through). Verify with ./gradlew test --tests "*HTTPBasicAuthenticator*".

2. (core) Assert InternalAuthenticationBackend rejects a wrong password. Locate the backend and its test (rg -l "class InternalAuthenticationBackend" src; rg -l "InternalAuthenticationBackendTest" src/test). Add a test that seeds an internal user with a bcrypt hash, then asserts authenticate(...) succeeds for the right password and throws OpenSearchSecurityException for the wrong one — the in-code form of the 401 you produced in Step 3.

3. (core) Assert the user lands in the ThreadContext under the right key. In Step 3 you grepped for where BackendRegistry injects the authenticated User. Turn it into a test: drive BackendRegistry.authenticate(...) (or the smallest reachable seam) with a valid Basic credential and assert the ThreadContext transient under ConfigConstants.OPENDISTRO_SECURITY_USER holds a User with name admin. Read the constant name from rg -n "OPENDISTRO_SECURITY_USER" src/main/java/org/opensearch/security/support/ConfigConstants.java.

4. (core) A security integration test: 401 vs 200. The security repo ships an integration-test framework (find it with rg -l "LocalCluster|class SingleClusterTest|extends.*IntegrationTest" src/integrationTest src/test). Write an integ test that boots a secured single node with the demo-style internal_users, then asserts an authenticated request to /_plugins/_security/authinfo returns 200 and a wrong-password request returns 401. This is the chain you exercised in Step 3, now self-contained and CI-runnable.

5. (core) Two-link-chain test: Basic then JWT fall-through. Model Step 5 in code. In an integration test, configure a config.yml with basic_internal_auth_domain (order 0) and a jwt_auth_domain (order 1, challenge: false). Assert that (a) Basic creds authenticate via link 0, (b) a valid Bearer JWT authenticates via link 1, and (c) neither yields a 401. Read how an existing JWT test mints a token (rg -l "Jwts.builder|HS256|signing_key" src/test src/integrationTest).

6. (advanced) Add a clientcert PKI authenticator path and prove DN→username. Advanced challenge: building on the Stretch goal, configure a clientcert authc domain and write an integration test that authenticates with a client cert (no -u, no token) and asserts the resolved user_name equals the username derived from the cert DN. Read HTTPClientCertAuthenticator (rg -l "class HTTPClientCertAuthenticator" src/main) to see which DN field becomes the username, then make the test assert exactly that mapping — and a second case where a cert whose DN does not match the configured pattern is rejected. The deliverable is one integ test proving cert-DN authentication end to end, the same SPI shape Lab SE3 extends.

Issues to Practice On

Authn/TLS issues dominate the opensearch-project/security tracker. Hunt them (labels move; confirm on the tracker):

gh issue list --repo opensearch-project/security --label "good first issue" --state open
gh issue list --repo opensearch-project/security --label "bug" --state open --search "TLS OR certificate OR JWT OR authentication OR 401"
gh issue list --repo opensearch-project/security --label "triage" --state open
gh label list --repo opensearch-project/security | grep -iE "auth|tls|ssl|jwt|good first"

Representative patterns. (1) "securityadmin.sh can't connect / handshake fails" — almost always transport TLS or admin-cert/DN mismatch: reproduce with the openssl s_client check from Troubleshooting, locate the config read via rg, fix the cert/DN, and add a test. (2) "JWT always 401" — bad signing_key/alg/clock skew: reproduce in an integration test, trace the validation in the JWT authenticator, fix, and ship the test. Arc: reproduce → locate via rg → fix → test → PR with a CHANGELOG.md entry and DCO sign-off.

Planted-bug drill. In HTTPBasicAuthenticator.extractCredentials (locate with rg -n "extractCredentials" $(rg -l "class HTTPBasicAuthenticator" src/main)), change the header it reads from Authorization to a typo (e.g. Authorisation). Run ./gradlew test --tests "*HTTPBasicAuthenticator*" and watch which test goes red — that test is the guard for every Basic login. Revert, then add the "no header ⇒ null creds" assertion from Exercise 1 so the fall-through contract is pinned.

Etiquette: claim the issue first, reproduce before theorising, and every PR ships a test + CHANGELOG.md entry + DCO Signed-off-by (git commit -s). See community-interaction.

Validation / self-check

• Explain, for a single curl -u alice:... https://..., the exact order: TLS handshake → SecurityRestFilter → BackendRegistry → which authc link matched → where the User was stored.
• State the difference between a TLS failure and a 401 at the wire level (one has no HTTP status, the other does) and why.
• Show why transport TLS is non-optional by reasoning about how securityadmin and cross-node user propagation depend on it.
• Point at the grep line in BackendRegistry where the user enters the ThreadContext, and name the transient header key.