Lab 1.3: Launch a Single-Node Cluster and Index Data

Background

Reading the engine is abstract until you watch it serve a request. In this lab you launch a real, debuggable OpenSearch node straight from your source checkout with ./gradlew run, then drive it over its REST API: check cluster health, create an index with an explicit mapping, bulk-index documents, run a match query, run a terms aggregation, and inspect the shards and Lucene segments backing your index.

The point is not "learn the REST API" (that is user documentation). The point is to connect every curl you run to the RestHandler that parses it and the TransportAction that executes it — so that when you read those classes in Level 3, you have already seen their effects.

Why This Lab Matters for Contributors

• ./gradlew run is the tool you will use for the rest of the curriculum to exercise code you have just read or changed.
• Mapping a REST call to its handler/action is the foundational request-tracing skill (Level 3 is built entirely on it).
• Watching _cat/segments change as you index, refresh, and merge makes the refresh/flush/merge lifecycle concrete.

Prerequisites

• Lab 1.1 complete: a clean ./gradlew assemble.
• curl and (optionally) jq installed for readable JSON.

Step-by-Step Tasks

Step 1: Launch the Node

From the repo root:

./gradlew run

This task builds a distribution and launches a single node with REST on localhost:9200 and the transport port on 9300. It runs in the foreground, streaming the node's logs — leave it running and open a second terminal for the curl calls. The node's data directory is ephemeral under the build output, so each run starts clean (great for reproducible experiments).

Wait for the line indicating the node has started and recovered:

[INFO ][o.o.n.Node] [runTask-0] started
[INFO ][o.o.g.GatewayService] [runTask-0] recovered [0] indices into cluster_state

Note: ./gradlew run launches with security/auth disabled by default in this dev flow, so plain HTTP on :9200 works without credentials. This is a development convenience — production clusters run the Security plugin (a separate repo). To attach a debugger, use ./gradlew run --debug-jvm and connect your IDE's remote JVM debugger to the printed port.

Step 2: Confirm the Cluster Is Alive

In the second terminal:

curl -s "localhost:9200" | jq .

Expected (versions/names vary):

{
  "name" : "runTask-0",
  "cluster_name" : "runTask",
  "version" : {
    "distribution" : "opensearch",
    "number" : "3.0.0",
    "lucene_version" : "9.x.x"
  },
  "tagline" : "The OpenSearch Project: https://opensearch.org/"
}

Now health:

curl -s "localhost:9200/_cat/health?v"

epoch      timestamp cluster status node.total node.data shards pri relo init unassign ...
1718531200 12:00:00  runTask green           1         1      0   0    0    0        0 ...

A single-node cluster reports green only when there are no unassigned replica shards to place. Watch this column change as you create indices in the next steps.

REST → handler map: GET / is served by RestMainAction; GET /_cat/health by RestHealthAction (one of the _cat family registered in RestController). The _cat handlers ultimately call the cluster-health transport action.

Step 3: Create an Index with an Explicit Mapping

Define the schema instead of relying on dynamic mapping — it makes the field types explicit and the later aggregation deterministic.

curl -s -X PUT "localhost:9200/books" \
  -H 'Content-Type: application/json' -d '{
  "settings": {
    "number_of_shards": 1,
    "number_of_replicas": 0
  },
  "mappings": {
    "properties": {
      "title":  { "type": "text" },
      "author": { "type": "keyword" },
      "year":   { "type": "integer" },
      "tags":   { "type": "keyword" }
    }
  }
}' | jq .

Expected:

{ "acknowledged": true, "shards_acknowledged": true, "index": "books" }

We set number_of_replicas: 0 so a single node stays green (no replica to leave unassigned). The text vs keyword distinction matters: title is analyzed for full-text match queries; author/tags are keyword (exact, not analyzed) so they aggregate cleanly.

REST → handler/action map: PUT /books → RestCreateIndexAction → TransportCreateIndexAction. Creating an index is a cluster-state change: it routes to the elected cluster manager (TransportClusterManagerNodeAction lineage), updates Metadata, and publishes a new ClusterState (the publish path is Level 4 and the cluster-state deep dive).

Step 4: Bulk-Index Documents

The Bulk API takes newline-delimited JSON (NDJSON): alternating action and source lines, with a trailing newline.

curl -s -X POST "localhost:9200/books/_bulk" \
  -H 'Content-Type: application/x-ndjson' --data-binary '
{ "index": { "_id": "1" } }
{ "title": "Lucene in Action", "author": "mccandless", "year": 2010, "tags": ["search","java"] }
{ "index": { "_id": "2" } }
{ "title": "Mastering OpenSearch", "author": "community", "year": 2023, "tags": ["search","ops"] }
{ "index": { "_id": "3" } }
{ "title": "Designing Data-Intensive Applications", "author": "kleppmann", "year": 2017, "tags": ["systems","data"] }
{ "index": { "_id": "4" } }
{ "title": "Elasticsearch: The Definitive Guide", "author": "community", "year": 2015, "tags": ["search","java"] }
' | jq '{errors, items: (.items | length)}'

Expected:

{ "errors": false, "items": 4 }

Make the documents searchable immediately (a refresh opens a new Lucene searcher — see refresh/flush/merge):

curl -s -X POST "localhost:9200/books/_refresh" >/dev/null
curl -s "localhost:9200/books/_count" | jq .
# { "count": 4, ... }

REST → handler/action map: POST /books/_bulk → RestBulkAction → TransportBulkAction → per-shard TransportShardBulkAction → IndexShard.applyIndexOperationOnPrimary(...) → InternalEngine.index(...) → Lucene IndexWriter.addDocument/updateDocument + Translog.add. You will trace this exact path in Level 6, Lab 6.1.

Step 5: Run a match (Full-Text) Query

match analyzes the query text and searches the analyzed title field:

curl -s -X GET "localhost:9200/books/_search" \
  -H 'Content-Type: application/json' -d '{
  "query": { "match": { "title": "search" } }
}' | jq '{hits: .hits.total.value, titles: [.hits.hits[]._source.title]}'

Because title is text, "Lucene in Action" matches nothing for search, but the two titles analyzed to include the term will match. Try a term that clearly hits:

curl -s -X GET "localhost:9200/books/_search" \
  -H 'Content-Type: application/json' -d '{
  "query": { "match": { "title": "opensearch lucene" } }
}' | jq '[.hits.hits[] | {title: ._source.title, score: ._score}]'

Expected (scores vary by BM25 stats):

[
  { "title": "Lucene in Action", "score": 1.4 },
  { "title": "Mastering OpenSearch", "score": 1.1 }
]

REST → handler/action map: GET /books/_search → RestSearchAction → TransportSearchAction. The coordinating node fans out to each shard; each runs SearchService.executeQueryPhase (QueryPhase) then executeFetchPhase (FetchPhase); the coordinator merges shard results in SearchPhaseController. The DSL match becomes a Lucene Query via QueryShardContext. Full trace in Level 7, Lab 7.1 and the search-execution deep dive.

Step 6: Run a terms Aggregation

Aggregate over the author keyword field, returning zero hits (we only want the buckets):

curl -s -X GET "localhost:9200/books/_search" \
  -H 'Content-Type: application/json' -d '{
  "size": 0,
  "aggs": {
    "by_author": { "terms": { "field": "author" } }
  }
}' | jq '.aggregations.by_author.buckets'

Expected:

[
  { "key": "community", "doc_count": 2 },
  { "key": "kleppmann", "doc_count": 1 },
  { "key": "mccandless", "doc_count": 1 }
]

This works because author is a keyword field with DocValues (a columnar, per-document store Lucene maintains). Aggregations read DocValues, not the inverted index. Try aggregating over the text field title and watch it fail — text fields disable DocValues by default:

curl -s -X GET "localhost:9200/books/_search" -H 'Content-Type: application/json' -d '{
  "size": 0, "aggs": { "by_title": { "terms": { "field": "title" } } }
}' | jq '.error.type, .error.reason'
# "illegal_argument_exception"
# "Text fields are not optimised for ... aggregations ... set fielddata=true ..."

That error message is a thing contributors improve — you will study handler validation in Level 2, Lab 2.3. Aggregation internals are the aggregations deep dive.

Step 7: Inspect Shards and Segments

Now look beneath the index at its physical shape.

curl -s "localhost:9200/_cat/shards/books?v"

index shard prirep state   docs store ip        node
books 0     p      STARTED    4  ...  127.0.0.1 runTask-0

One primary shard (p), no replicas (we set number_of_replicas: 0), in STARTED state. That one shard is a single Lucene index. Now the segments inside it:

curl -s "localhost:9200/_cat/segments/books?v"

index shard prirep ip        segment generation docs.count docs.deleted size searchable committed
books 0     p      127.0.0.1 _0               0          4            0  ...  true       false

You likely see one segment after a single bulk + refresh. Index more documents in separate batches (each with its own refresh) and re-run _cat/segments — you will see multiple segments appear. A force-merge collapses them:

curl -s -X POST "localhost:9200/books/_forcemerge?max_num_segments=1" >/dev/null
curl -s "localhost:9200/_cat/segments/books?v"
# now a single, larger segment

This is the merge process the engine runs continuously in the background. You will build a standalone Lucene index and watch the same segment/merge behavior with no OpenSearch at all in Lab 1.4.

Step 8: Shut Down

Stop the node with Ctrl+C in the terminal running ./gradlew run. Because the data directory is ephemeral, the next run starts empty.

Implementation Requirements

This lab has no code to implement. Deliverables:

1. GET /_cat/health returning green with your node running.
2. The books index created with an explicit mapping; _count returning 4.
3. A match query returning the expected titles, and a terms aggregation returning the author buckets.
4. A completed REST → handler → action mapping table (below) filled in from your own reading.
5. _cat/segments output before and after a force-merge, with one sentence explaining the change.

Fill this in by grepping the source (grep -rn "new Route" server/src/main/java/org/opensearch/rest and the action classes):

Troubleshooting

curl: (7) Failed to connect to localhost port 9200

The node has not finished starting, or run failed. Check the ./gradlew run terminal for the started line; if it crashed, read the stack trace there.

./gradlew run exits immediately with a port-in-use error

java.net.BindException: Address already in use

A previous node or another service holds :9200/:9300. Stop the old process (./gradlew --stop won't kill a foreground run — use Ctrl+C; find strays with lsof -i :9200).

Bulk request returns "errors": true

Inspect the per-item errors:

curl -s -X POST "localhost:9200/books/_bulk" -H 'Content-Type: application/x-ndjson' \
  --data-binary @your.ndjson | jq '.items[] | select(.index.error) | .index.error'

The most common cause is a missing trailing newline on the NDJSON body, or a Content-Type other than application/x-ndjson. Use --data-binary, not -d (which strips newlines).

Search returns zero hits right after indexing

You did not refresh. Documents are not searchable until a refresh opens a new searcher. Either POST /books/_refresh or wait for the periodic refresh (default ~1s). This is refresh/flush/merge in action.

Expected Output

The end-to-end happy path, condensed:

$ curl -s localhost:9200/_cat/health?h=status      ->  green
$ curl -s localhost:9200/books/_count | jq .count  ->  4
$ ... match "opensearch lucene"                     ->  2 hits
$ ... terms by_author                               ->  community:2 kleppmann:1 mccandless:1
$ curl -s localhost:9200/_cat/shards/books?h=prirep,state -> p STARTED
$ curl -s localhost:9200/_cat/segments/books?h=segment    -> _0  (one segment after forcemerge)

Stretch Goals

1. Force a yellow cluster. Recreate books with number_of_replicas: 1 and watch _cat/health go yellow (the replica cannot be allocated on a single node) and _cat/shards/books show an UNASSIGNED replica. Explain why — this is the shard-allocation story from Level 4.

2. Watch refresh make data visible. Index a doc with ?refresh=false, immediately search (zero hits), then _refresh and search again (one hit). Confirm the refresh boundary.

3. Profile the query. Add "profile": true to a _search body and read the per-shard, per-Lucene-query timing breakdown. This is the query phase exposed.

4. Find the route registration. For each row in your mapping table, find the new Route(...) or routes() declaration in the handler:

   grep -rn "routes()\|new Route(" server/src/main/java/org/opensearch/rest/action/search/RestSearchAction.java
   

Coding Exercises

You drove the node by hand with curl; now turn that into code — both scripts that index and query, and real tests that assert cluster behavior. Locate every class with find/rg before citing it.

1. (warm-up) Script the happy path. Write index-and-query.sh: it PUTs the books index with the explicit mapping from Step 3, bulk-indexes the four documents, refreshes, then runs the match and terms queries and prints the hit count and the author buckets. Pipe through jq and make the script exit 1 if _count is not 4. Verify: against a running ./gradlew run node the script prints 4, the two match titles, and the three author buckets, and exits 0.

2. (warm-up) Assert cluster health in a script. Add to the script a function that polls GET /_cat/health?h=status until it reads green (or times out after N seconds) — the shell analogue of assertBusy. Verify: it succeeds against your single-node books index (replicas = 0) and fails (stays yellow) if you recreate books with number_of_replicas: 1, exactly the Stretch Goal 1 scenario, now machine-checked.

3. (core) Write a health assertion as a real test. Using OpenSearchSingleNodeTestCase (find a sibling with grep -rln "extends OpenSearchSingleNodeTestCase" server/src/test | head), write a test that creates an index with number_of_replicas = 0, waits for green via the test client's cluster-health request with setWaitForGreenStatus(), and asserts the status is GREEN. Then add a second method that creates the index with one replica and asserts the health goes YELLOW. Verify: both pass under :server:test. This is Step 2's _cat/health as a deterministic test.

4. (core) Verify the REST → handler mapping in code, not prose. Step 7 / Stretch Goal 4 had you grep for new Route(. Now prove the route exists with a test-shaped check: write a script (or a JUnit method) that, for RestSearchAction, asserts its routes() includes GET /{index}/_search. Locate the class with find server -name "RestSearchAction.java" -path "*/main/*" and read routes(); if you go the JUnit route, instantiate the handler in a unit test and assert handler.routes() contains the expected Route. Verify: the assertion is green and you can name the TransportAction it dispatches to (TransportSearchAction).

5. (core) Index from a real test and assert search results. Using OpenSearchSingleNodeTestCase, write a test that indexes the four books programmatically (via the node's client index(...) requests), refreshes, runs a match on title, and asserts the exact set of returned ids. This is Lab 1.4's Lucene queries one layer up — through the engine and the search phases. Verify: green under :server:test; the returned ids match what your curl produced in Step 5.

6. (advanced) Advanced challenge — turn the whole lab into an integration test. Write a single OpenSearchSingleNodeTestCase (or OpenSearchIntegTestCase if you want a real multi-node cluster — see Lab 1.2) that reproduces this lab end to end: create the books mapping, bulk-index, refresh, assert _count == 4, assert the match hits, assert the terms buckets, then assert the text-field aggregation fails with IllegalArgumentException (expectThrows) — capturing the exact error message Step 6 showed. Finally, call force-merge via the client and assert the segment count drops to 1 (read it from IndicesSegmentsRequest / _cat/segments-equivalent API). Verify: the whole flow is one green test, with no curl and no running node — the lab is now reproducible in CI. This is the bridge to Level 7, where you trace these same phases.

Issues to Practice On

Indexing and search behavior generate a steady stream of beginner-accessible issues — especially the error-message quality issues this lab brushed against in Step 6. The right repo is opensearch-project/OpenSearch.

# Good-first-issues, then area filters (labels move; confirm on the tracker):
gh issue list --repo opensearch-project/OpenSearch --label "good first issue" --state open
gh issue list --repo opensearch-project/OpenSearch --label "Search" --state open
gh issue list --repo opensearch-project/OpenSearch --label "Search:Aggregations" --state open
gh issue list --repo opensearch-project/OpenSearch --label "Indexing" --state open

# Confirm the area labels actually exist on the tracker before relying on them:
gh label list --repo opensearch-project/OpenSearch | grep -iE "search|index|aggregation"

Representative issue patterns for this subsystem:

• An unhelpful error/validation message on a REST request. Exactly the "Text fields are not optimised for aggregations" class from Step 6. Reproduce with curl, locate the message with rg "<distinctive fragment>" server/src/main/java, improve it (name the field and the remedy), and add a unit test asserting the new text. This is the whole of Lab 2.3.
• A _cat / health / stats endpoint quirk. A column is mislabeled or a _cat output is confusing. Locate the Rest*Action with rg "new Route" server/src/main/java/org/opensearch/rest, fix, and cover it with a REST-YAML test under rest-api-spec or a handler unit test.

Planted-bug drill. Break the request path in a scratch branch and watch which test catches it:

1. Find RestCreateIndexAction (find server -name "RestCreateIndexAction.java" -path "*/main/*") and the transport action it dispatches to. In a scratch copy, change the route or a default in the create-index path so the index is created with a different default shard count, then run the tests that cover create-index (./gradlew :server:test --tests "*CreateIndex*"). Watch which test goes red, then revert and add an assertion that would have caught your change (e.g. asserting the default number_of_shards).
2. Alternatively, flip the text-field-aggregation guard: locate the check that throws the "not optimised for aggregations" error (rg -n "not optimised\|fielddata" server/src/main/java), weaken it, watch the aggregation tests fail, then restore and add a focused unit test asserting the IllegalArgumentException fires for a text field.

Etiquette: claim before you code, reproduce first (paste the curl and the output), and ship a test + CHANGELOG + DCO Signed-off-by (git commit -s) with every PR. Workflow: Lab 2.2; norms: community interaction.

Validation / Self-check

You are done when you can answer these without notes:

1. Why does a single-node cluster go yellow when you ask for one replica, but green with zero?
2. Which handler and which transport action service POST /_bulk? Where does the write ultimately call into Lucene?
3. Why does a terms aggregation on a keyword field work but the same on a text field fail by default? What underlying Lucene structure does the aggregation read?
4. What does _refresh do at the Lucene level, and why is a document not searchable before it?
5. What is the relationship between an OpenSearch index, a shard, and a Lucene segment?
6. What does a force-merge do to _cat/segments, and why is that the same thing the engine does in the background?

Next: Lab 1.4 — Build a Minimal Lucene Index, where you build a shard by hand.