Lab 1.3: Launch a Single-Node Cluster and Index Data

Background

Reading the engine is abstract until you watch it serve a request. In this lab you launch a real, debuggable OpenSearch node straight from your source checkout with ./gradlew run, then drive it over its REST API: check cluster health, create an index with an explicit mapping, bulk-index documents, run a match query, run a terms aggregation, and inspect the shards and Lucene segments backing your index.

The point is not "learn the REST API" (that is user documentation). The point is to connect every curl you run to the RestHandler that parses it and the TransportAction that executes it — so that when you read those classes in Level 3, you have already seen their effects.

Why This Lab Matters for Contributors

• ./gradlew run is the tool you will use for the rest of the curriculum to exercise code you have just read or changed.
• Mapping a REST call to its handler/action is the foundational request-tracing skill (Level 3 is built entirely on it).
• Watching _cat/segments change as you index, refresh, and merge makes the refresh/flush/merge lifecycle concrete.

Prerequisites

• Lab 1.1 complete: a clean ./gradlew assemble.
• curl and (optionally) jq installed for readable JSON.

Step-by-Step Tasks

Step 1: Launch the Node

From the repo root:

./gradlew run

This task builds a distribution and launches a single node with REST on localhost:9200 and the transport port on 9300. It runs in the foreground, streaming the node's logs — leave it running and open a second terminal for the curl calls. The node's data directory is ephemeral under the build output, so each run starts clean (great for reproducible experiments).

Wait for the line indicating the node has started and recovered:

[INFO ][o.o.n.Node] [runTask-0] started
[INFO ][o.o.g.GatewayService] [runTask-0] recovered [0] indices into cluster_state

Note: ./gradlew run launches with security/auth disabled by default in this dev flow, so plain HTTP on :9200 works without credentials. This is a development convenience — production clusters run the Security plugin (a separate repo). To attach a debugger, use ./gradlew run --debug-jvm and connect your IDE's remote JVM debugger to the printed port.

Step 2: Confirm the Cluster Is Alive

In the second terminal:

curl -s "localhost:9200" | jq .

Expected (versions/names vary):

{
  "name" : "runTask-0",
  "cluster_name" : "runTask",
  "version" : {
    "distribution" : "opensearch",
    "number" : "3.0.0",
    "lucene_version" : "9.x.x"
  },
  "tagline" : "The OpenSearch Project: https://opensearch.org/"
}

Now health:

curl -s "localhost:9200/_cat/health?v"

epoch      timestamp cluster status node.total node.data shards pri relo init unassign ...
1718531200 12:00:00  runTask green           1         1      0   0    0    0        0 ...

A single-node cluster reports green only when there are no unassigned replica shards to place. Watch this column change as you create indices in the next steps.

REST → handler map: GET / is served by RestMainAction; GET /_cat/health by RestHealthAction (one of the _cat family registered in RestController). The _cat handlers ultimately call the cluster-health transport action.

Step 3: Create an Index with an Explicit Mapping

Define the schema instead of relying on dynamic mapping — it makes the field types explicit and the later aggregation deterministic.

curl -s -X PUT "localhost:9200/books" \
  -H 'Content-Type: application/json' -d '{
  "settings": {
    "number_of_shards": 1,
    "number_of_replicas": 0
  },
  "mappings": {
    "properties": {
      "title":  { "type": "text" },
      "author": { "type": "keyword" },
      "year":   { "type": "integer" },
      "tags":   { "type": "keyword" }
    }
  }
}' | jq .

Expected:

{ "acknowledged": true, "shards_acknowledged": true, "index": "books" }

We set number_of_replicas: 0 so a single node stays green (no replica to leave unassigned). The text vs keyword distinction matters: title is analyzed for full-text match queries; author/tags are keyword (exact, not analyzed) so they aggregate cleanly.

REST → handler/action map: PUT /books → RestCreateIndexAction → TransportCreateIndexAction. Creating an index is a cluster-state change: it routes to the elected cluster manager (TransportClusterManagerNodeAction lineage), updates Metadata, and publishes a new ClusterState (the publish path is Level 4 and the cluster-state deep dive).

Step 4: Bulk-Index Documents

The Bulk API takes newline-delimited JSON (NDJSON): alternating action and source lines, with a trailing newline.

curl -s -X POST "localhost:9200/books/_bulk" \
  -H 'Content-Type: application/x-ndjson' --data-binary '
{ "index": { "_id": "1" } }
{ "title": "Lucene in Action", "author": "mccandless", "year": 2010, "tags": ["search","java"] }
{ "index": { "_id": "2" } }
{ "title": "Mastering OpenSearch", "author": "community", "year": 2023, "tags": ["search","ops"] }
{ "index": { "_id": "3" } }
{ "title": "Designing Data-Intensive Applications", "author": "kleppmann", "year": 2017, "tags": ["systems","data"] }
{ "index": { "_id": "4" } }
{ "title": "Elasticsearch: The Definitive Guide", "author": "community", "year": 2015, "tags": ["search","java"] }
' | jq '{errors, items: (.items | length)}'

Expected:

{ "errors": false, "items": 4 }

Make the documents searchable immediately (a refresh opens a new Lucene searcher — see refresh/flush/merge):

curl -s -X POST "localhost:9200/books/_refresh" >/dev/null
curl -s "localhost:9200/books/_count" | jq .
# { "count": 4, ... }

REST → handler/action map: POST /books/_bulk → RestBulkAction → TransportBulkAction → per-shard TransportShardBulkAction → IndexShard.applyIndexOperationOnPrimary(...) → InternalEngine.index(...) → Lucene IndexWriter.addDocument/updateDocument + Translog.add. You will trace this exact path in Level 6, Lab 6.1.

Step 5: Run a match (Full-Text) Query

match analyzes the query text and searches the analyzed title field:

curl -s -X GET "localhost:9200/books/_search" \
  -H 'Content-Type: application/json' -d '{
  "query": { "match": { "title": "search" } }
}' | jq '{hits: .hits.total.value, titles: [.hits.hits[]._source.title]}'

Because title is text, "Lucene in Action" matches nothing for search, but the two titles analyzed to include the term will match. Try a term that clearly hits:

curl -s -X GET "localhost:9200/books/_search" \
  -H 'Content-Type: application/json' -d '{
  "query": { "match": { "title": "opensearch lucene" } }
}' | jq '[.hits.hits[] | {title: ._source.title, score: ._score}]'

Expected (scores vary by BM25 stats):

[
  { "title": "Lucene in Action", "score": 1.4 },
  { "title": "Mastering OpenSearch", "score": 1.1 }
]

REST → handler/action map: GET /books/_search → RestSearchAction → TransportSearchAction. The coordinating node fans out to each shard; each runs SearchService.executeQueryPhase (QueryPhase) then executeFetchPhase (FetchPhase); the coordinator merges shard results in SearchPhaseController. The DSL match becomes a Lucene Query via QueryShardContext. Full trace in Level 7, Lab 7.1 and the search-execution deep dive.

Step 6: Run a terms Aggregation

Aggregate over the author keyword field, returning zero hits (we only want the buckets):

curl -s -X GET "localhost:9200/books/_search" \
  -H 'Content-Type: application/json' -d '{
  "size": 0,
  "aggs": {
    "by_author": { "terms": { "field": "author" } }
  }
}' | jq '.aggregations.by_author.buckets'

Expected:

[
  { "key": "community", "doc_count": 2 },
  { "key": "kleppmann", "doc_count": 1 },
  { "key": "mccandless", "doc_count": 1 }
]

This works because author is a keyword field with DocValues (a columnar, per-document store Lucene maintains). Aggregations read DocValues, not the inverted index. Try aggregating over the text field title and watch it fail — text fields disable DocValues by default:

curl -s -X GET "localhost:9200/books/_search" -H 'Content-Type: application/json' -d '{
  "size": 0, "aggs": { "by_title": { "terms": { "field": "title" } } }
}' | jq '.error.type, .error.reason'
# "illegal_argument_exception"
# "Text fields are not optimised for ... aggregations ... set fielddata=true ..."

That error message is a thing contributors improve — you will study handler validation in Level 2, Lab 2.3. Aggregation internals are the aggregations deep dive.

Step 7: Inspect Shards and Segments

Now look beneath the index at its physical shape.

curl -s "localhost:9200/_cat/shards/books?v"

index shard prirep state   docs store ip        node
books 0     p      STARTED    4  ...  127.0.0.1 runTask-0

One primary shard (p), no replicas (we set number_of_replicas: 0), in STARTED state. That one shard is a single Lucene index. Now the segments inside it:

curl -s "localhost:9200/_cat/segments/books?v"

index shard prirep ip        segment generation docs.count docs.deleted size searchable committed
books 0     p      127.0.0.1 _0               0          4            0  ...  true       false

You likely see one segment after a single bulk + refresh. Index more documents in separate batches (each with its own refresh) and re-run _cat/segments — you will see multiple segments appear. A force-merge collapses them:

curl -s -X POST "localhost:9200/books/_forcemerge?max_num_segments=1" >/dev/null
curl -s "localhost:9200/_cat/segments/books?v"
# now a single, larger segment

This is the merge process the engine runs continuously in the background. You will build a standalone Lucene index and watch the same segment/merge behavior with no OpenSearch at all in Lab 1.4.

Step 8: Shut Down

Stop the node with Ctrl+C in the terminal running ./gradlew run. Because the data directory is ephemeral, the next run starts empty.

Implementation Requirements

This lab has no code to implement. Deliverables:

1. GET /_cat/health returning green with your node running.
2. The books index created with an explicit mapping; _count returning 4.
3. A match query returning the expected titles, and a terms aggregation returning the author buckets.
4. A completed REST → handler → action mapping table (below) filled in from your own reading.
5. _cat/segments output before and after a force-merge, with one sentence explaining the change.

Fill this in by grepping the source (grep -rn "new Route" server/src/main/java/org/opensearch/rest and the action classes):

Troubleshooting

curl: (7) Failed to connect to localhost port 9200

The node has not finished starting, or run failed. Check the ./gradlew run terminal for the started line; if it crashed, read the stack trace there.

./gradlew run exits immediately with a port-in-use error

java.net.BindException: Address already in use

A previous node or another service holds :9200/:9300. Stop the old process (./gradlew --stop won't kill a foreground run — use Ctrl+C; find strays with lsof -i :9200).

Bulk request returns "errors": true

Inspect the per-item errors:

curl -s -X POST "localhost:9200/books/_bulk" -H 'Content-Type: application/x-ndjson' \
  --data-binary @your.ndjson | jq '.items[] | select(.index.error) | .index.error'

The most common cause is a missing trailing newline on the NDJSON body, or a Content-Type other than application/x-ndjson. Use --data-binary, not -d (which strips newlines).

Search returns zero hits right after indexing

You did not refresh. Documents are not searchable until a refresh opens a new searcher. Either POST /books/_refresh or wait for the periodic refresh (default ~1s). This is refresh/flush/merge in action.

Expected Output

The end-to-end happy path, condensed:

$ curl -s localhost:9200/_cat/health?h=status      ->  green
$ curl -s localhost:9200/books/_count | jq .count  ->  4
$ ... match "opensearch lucene"                     ->  2 hits
$ ... terms by_author                               ->  community:2 kleppmann:1 mccandless:1
$ curl -s localhost:9200/_cat/shards/books?h=prirep,state -> p STARTED
$ curl -s localhost:9200/_cat/segments/books?h=segment    -> _0  (one segment after forcemerge)

Stretch Goals

1. Force a yellow cluster. Recreate books with number_of_replicas: 1 and watch _cat/health go yellow (the replica cannot be allocated on a single node) and _cat/shards/books show an UNASSIGNED replica. Explain why — this is the shard-allocation story from Level 4.

2. Watch refresh make data visible. Index a doc with ?refresh=false, immediately search (zero hits), then _refresh and search again (one hit). Confirm the refresh boundary.

3. Profile the query. Add "profile": true to a _search body and read the per-shard, per-Lucene-query timing breakdown. This is the query phase exposed.

4. Find the route registration. For each row in your mapping table, find the new Route(...) or routes() declaration in the handler:

   grep -rn "routes()\|new Route(" server/src/main/java/org/opensearch/rest/action/search/RestSearchAction.java
   

Validation / Self-check

You are done when you can answer these without notes:

1. Why does a single-node cluster go yellow when you ask for one replica, but green with zero?
2. Which handler and which transport action service POST /_bulk? Where does the write ultimately call into Lucene?
3. Why does a terms aggregation on a keyword field work but the same on a text field fail by default? What underlying Lucene structure does the aggregation read?
4. What does _refresh do at the Lucene level, and why is a document not searchable before it?
5. What is the relationship between an OpenSearch index, a shard, and a Lucene segment?
6. What does a force-merge do to _cat/segments, and why is that the same thing the engine does in the background?

Next: Lab 1.4 — Build a Minimal Lucene Index, where you build a shard by hand.