Lab 6.3: Build It — A Custom Analyzer

Background

You have traced the write path (Lab 6.1) and operated the engine (Lab 6.2). The one piece you have only glanced at is analysis — the step where a text field's value is tokenized and filtered into the terms Lucene actually indexes. Analysis is the most plugin-friendly extension point in OpenSearch: tokenizers, token filters, char filters, and analyzers are all registered through AnalysisPlugin, and writing one is the cleanest introduction to the entire plugin model.

In this Build-It lab you create a real, installable OpenSearch plugin from scratch: a Plugin implements AnalysisPlugin that registers a custom token filter. You will write the Lucene TokenFilter (the incrementToken() loop), the AbstractTokenFilterFactory that wires it into OpenSearch, the plugin-descriptor.properties, and the opensearch.opensearchplugin Gradle build. Then you build it, install it into a distribution, and verify it with the _analyze API and a unit test extending OpenSearchTokenStreamTestCase.

The example filter is skip_short_tokens: drop any token shorter than a configurable min_length (with an uppercase variant in the stretch goals). It is small enough to understand fully and real enough to be useful.

Note: Read the plugin architecture and mapping & analysis deep dives alongside this lab. The bundled analysis-common module is your reference implementation — when in doubt, copy its patterns.

Why This Lab Matters for Contributors

Plugins are how OpenSearch is extended in the real world (security, k-NN, SQL, analyzers all ship as plugins). Building one — even a tiny one — demystifies plugin-descriptor.properties, the opensearch.opensearchplugin Gradle plugin, isolated classloaders, and the registration interfaces.
AnalysisPlugin is the gentlest entry point: no cluster state, no transport, no threading — pure Lucene TokenStream mechanics you can fully test in-JVM.
incrementToken() is the heart of Lucene text processing. Writing one teaches you how every analyzer in OpenSearch actually works, which makes analysis bugs and feature requests legible.
A working plugin with a _analyze demo and a token-stream unit test is a complete, portfolio-quality contribution shape.

Prerequisites

A clean OpenSearch build (./gradlew assemble works).
Lab 6.2 done — you know how text becomes terms.
JDK 21; the repo's bundled JDK is fine.

Familiarity with the bundled analysis module as a template:

ls modules/analysis-common/src/main/java/org/opensearch/analysis/common/ | head -30
grep -rln "AbstractTokenFilterFactory" modules/analysis-common/src/main/java | head

Step-by-Step Tasks

Step 1: Understand the four pieces you must write

A token-filter plugin is exactly four artifacts plus a build file. Hold this map in your head:

flowchart LR
    A["plugin-descriptor.properties<br/>(name, classname, versions)"] --> B["SkipShortTokensPlugin<br/>implements AnalysisPlugin"]
    B -->|getTokenFilters() registers| C["SkipShortTokenFilterFactory<br/>extends AbstractTokenFilterFactory"]
    C -->|create() wraps stream| D["SkipShortTokenFilter<br/>extends Lucene TokenFilter<br/>incrementToken()"]

Artifact	Responsibility
`plugin-descriptor.properties`	Tells OpenSearch the plugin's name, entry `classname`, and the OpenSearch/Java versions it targets
`Plugin implements AnalysisPlugin`	The entry point; `getTokenFilters()` returns the name→factory registry
`AbstractTokenFilterFactory` subclass	Reads settings (e.g. `min_length`) and `create(TokenStream)`s the Lucene filter
Lucene `TokenFilter` subclass	The actual logic in `incrementToken()`

Step 2: Scaffold the plugin module

Create the module directory under plugins/ (mirroring the in-repo plugins):

mkdir -p plugins/analysis-skip-short/src/main/java/org/opensearch/analysis/skipshort
mkdir -p plugins/analysis-skip-short/src/test/java/org/opensearch/analysis/skipshort
mkdir -p plugins/analysis-skip-short/src/main/resources
ls plugins/ | head     # confirm your module sits beside analysis-icu etc.

Step 3: The Lucene `TokenFilter` — `incrementToken()`

This is the core. A TokenFilter decorates an input TokenStream; incrementToken() pulls tokens from the input, decides whether to emit each, and returns true for "a token is ready" or false for end-of-stream. We read each token's text via the CharTermAttribute and skip those shorter than minLength:

/*
 * SPDX-License-Identifier: Apache-2.0
 *
 * The OpenSearch Contributors require contributions made to
 * this file be licensed under the Apache-2.0 license or a
 * compatible open source license.
 */

package org.opensearch.analysis.skipshort;

import org.apache.lucene.analysis.TokenFilter;
import org.apache.lucene.analysis.TokenStream;
import org.apache.lucene.analysis.tokenattributes.CharTermAttribute;

import java.io.IOException;

/** Drops tokens whose length is strictly less than {@code minLength}. */
public final class SkipShortTokenFilter extends TokenFilter {

    private final int minLength;
    // CharTermAttribute is the token's text; it is shared/mutated across the stream.
    private final CharTermAttribute termAttr = addAttribute(CharTermAttribute.class);

    public SkipShortTokenFilter(TokenStream input, int minLength) {
        super(input);
        this.minLength = minLength;
    }

    @Override
    public boolean incrementToken() throws IOException {
        // Pull tokens until we find one to keep, or the stream ends.
        while (input.incrementToken()) {
            if (termAttr.length() >= minLength) {
                return true;     // keep this token; its attributes are already populated
            }
            // else: too short — loop and pull the next one, effectively dropping this token
        }
        return false;            // input exhausted
    }
}

Note: Token attributes (CharTermAttribute, OffsetAttribute, PositionIncrementAttribute, …) are shared, mutable objects reused for every token — never cache a token's value across incrementToken() calls without copying it. A filtering filter like this one doesn't need to fix up position increments for downstream phrase queries, but a stricter implementation would bump PositionIncrementAttribute for dropped tokens; note that as a known limitation.

Step 4: The factory — read settings and wire it in

AbstractTokenFilterFactory is OpenSearch's adapter between a Lucene TokenFilter and the analysis registry. It receives the index settings, the analysis settings block, and the filter's name. Read your min_length setting here, with a default and validation:

/* SPDX header omitted for brevity — include it in your real file */
package org.opensearch.analysis.skipshort;

import org.apache.lucene.analysis.TokenStream;
import org.opensearch.common.settings.Settings;
import org.opensearch.env.Environment;
import org.opensearch.index.IndexSettings;
import org.opensearch.index.analysis.AbstractTokenFilterFactory;

public class SkipShortTokenFilterFactory extends AbstractTokenFilterFactory {

    private final int minLength;

    public SkipShortTokenFilterFactory(IndexSettings indexSettings, Environment env, String name, Settings settings) {
        super(indexSettings, name, settings);
        // "min_length" from the analyzer's filter config; default 3, must be >= 1.
        this.minLength = settings.getAsInt("min_length", 3);
        if (minLength < 1) {
            throw new IllegalArgumentException(
                "[min_length] must be >= 1 for filter [" + name + "], got [" + minLength + "]");
        }
    }

    @Override
    public TokenStream create(TokenStream tokenStream) {
        return new SkipShortTokenFilter(tokenStream, minLength);
    }
}

Step 5: The plugin entry point — `AnalysisPlugin.getTokenFilters()`

The Plugin is the classname OpenSearch loads. Implementing AnalysisPlugin and overriding getTokenFilters() registers your filter under a name (the name users put in their analyzer config):

/* SPDX header omitted for brevity — include it in your real file */
package org.opensearch.analysis.skipshort;

import org.opensearch.index.analysis.TokenFilterFactory;
import org.opensearch.indices.analysis.AnalysisModule.AnalysisProvider;
import org.opensearch.plugins.AnalysisPlugin;
import org.opensearch.plugins.Plugin;

import java.util.Map;
import java.util.TreeMap;

public class SkipShortTokensPlugin extends Plugin implements AnalysisPlugin {

    @Override
    public Map<String, AnalysisProvider<TokenFilterFactory>> getTokenFilters() {
        Map<String, AnalysisProvider<TokenFilterFactory>> filters = new TreeMap<>();
        // Registered name "skip_short" -> factory constructor reference.
        filters.put("skip_short", SkipShortTokenFilterFactory::new);
        return filters;
    }
}

Note: The AnalysisProvider<TokenFilterFactory> functional interface matches the factory's 4-arg constructor (IndexSettings, Environment, String name, Settings). The method-reference SkipShortTokenFilterFactory::new satisfies it directly — if it doesn't compile, your constructor signature is off.

Step 6: `plugin-descriptor.properties`

OpenSearch reads this at load time. Place it where the Gradle plugin expects it (it is generated into the zip from your opensearchplugin {} config, but you can also provide a static one). The fields:

# plugins/analysis-skip-short/src/main/resources/plugin-descriptor.properties
description=Adds a skip_short token filter that drops tokens shorter than min_length
version=${version}
name=analysis-skip-short
classname=org.opensearch.analysis.skipshort.SkipShortTokensPlugin
java.version=21
opensearch.version=${opensearchVersion}

Note: In an in-repo plugin you usually do not hand-write this file — the opensearch.opensearchplugin Gradle plugin generates it from the opensearchplugin { ... } block (Step 7) so versions stay in sync. Hand-write it only for an out-of-tree plugin built against the published artifacts.

Step 7: The Gradle build (`opensearch.opensearchplugin`)

Create plugins/analysis-skip-short/build.gradle. The opensearch.opensearchplugin plugin produces an installable plugin zip and generates the descriptor:

/* plugins/analysis-skip-short/build.gradle */
apply plugin: 'opensearch.opensearchplugin'
apply plugin: 'opensearch.yaml-rest-test'   // optional: enables yamlRestTest for the plugin

opensearchplugin {
    description 'Adds a skip_short token filter that drops tokens shorter than min_length'
    classname  'org.opensearch.analysis.skipshort.SkipShortTokensPlugin'
}

// Analysis APIs come from the core 'opensearch' dependency the plugin already builds against.
// Lucene's analysis classes are provided transitively.
restResources {
    restApi {
        includeCore '_common', 'indices', 'index', 'indices.analyze'
    }
}

Wire the module into the build by ensuring it is discovered (in-repo plugins under plugins/ are picked up by the root settings.gradle's project inclusion; confirm):

grep -rn "analysis-skip-short\|project(.*plugins" settings.gradle build.gradle 2>/dev/null | head
# If in-repo plugins are auto-included you'll see the glob; otherwise add:
#   include ':plugins:analysis-skip-short'

Step 8: Build and install the plugin

# Build just your plugin's zip.
./gradlew :plugins:analysis-skip-short:assemble
find plugins/analysis-skip-short/build/distributions -name "*.zip"
#   plugins/analysis-skip-short/build/distributions/analysis-skip-short-<version>.zip

# Easiest path to a node WITH the plugin already loaded: run with the plugin on the classpath.
./gradlew run -PinstalledPlugins="['analysis-skip-short']"

Alternatively, install into a built distribution by hand:

./gradlew localDistro
DISTRO=$(find distribution/archives -maxdepth 2 -name "opensearch-*" -type d | head -1)
"$DISTRO/bin/opensearch-plugin" install \
  "file://$(pwd)/plugins/analysis-skip-short/build/distributions/$(ls plugins/analysis-skip-short/build/distributions | grep '\.zip$')"
"$DISTRO/bin/opensearch"        # start it

Confirm OpenSearch loaded it:

curl -s 'localhost:9200/_cat/plugins?v'
#   name   component             version
#   node-0 analysis-skip-short   <version>

Step 9: Test it with `_analyze`

The _analyze API runs text through a tokenizer + filter chain and returns the resulting tokens — the fastest way to verify your filter. Test it standalone first:

curl -s -XPOST 'localhost:9200/_analyze?pretty' -H 'Content-Type: application/json' -d '{
  "tokenizer": "standard",
  "filter": [ { "type": "skip_short", "min_length": 4 } ],
  "text": "the quick brown fox is up"
}'

Expected: tokens shorter than 4 chars (the, fox, is, up) are dropped; quick and brown remain:

{
  "tokens" : [
    { "token" : "quick", "start_offset" : 4,  "end_offset" : 9,  "type" : "<ALPHANUM>", "position" : 1 },
    { "token" : "brown", "start_offset" : 10, "end_offset" : 15, "type" : "<ALPHANUM>", "position" : 2 }
  ]
}

Then bind it into a real custom analyzer on an index and index a doc through it:

curl -s -XPUT 'localhost:9200/skipdemo?pretty' -H 'Content-Type: application/json' -d '{
  "settings": {
    "analysis": {
      "filter":   { "no_short": { "type": "skip_short", "min_length": 4 } },
      "analyzer": { "long_words": { "type": "custom", "tokenizer": "standard", "filter": ["lowercase","no_short"] } }
    }
  },
  "mappings": { "properties": { "body": { "type": "text", "analyzer": "long_words" } } }
}'

# Verify the named analyzer applies the filter.
curl -s -XPOST 'localhost:9200/skipdemo/_analyze?pretty' -H 'Content-Type: application/json' -d '{
  "analyzer": "long_words", "text": "An OpenSearch token filter in action"
}'
# Expect: opensearch, token, filter, action  (an, in dropped)

Step 10: The unit test — `OpenSearchTokenStreamTestCase`

_analyze is a smoke test; the real test is a fast, deterministic unit test of the token stream. OpenSearchTokenStreamTestCase gives you Lucene's analysis assertions:

/*
 * SPDX-License-Identifier: Apache-2.0
 *
 * The OpenSearch Contributors require contributions made to
 * this file be licensed under the Apache-2.0 license or a
 * compatible open source license.
 */

package org.opensearch.analysis.skipshort;

import org.apache.lucene.analysis.TokenStream;
import org.apache.lucene.analysis.core.WhitespaceTokenizer;
import org.opensearch.test.OpenSearchTokenStreamTestCase;

import java.io.StringReader;

public class SkipShortTokenFilterTests extends OpenSearchTokenStreamTestCase {

    public void testDropsShortTokens() throws Exception {
        WhitespaceTokenizer src = new WhitespaceTokenizer();
        src.setReader(new StringReader("a bb ccc dddd eeeee"));
        TokenStream filtered = new SkipShortTokenFilter(src, 3);   // keep length >= 3

        // assertTokenStreamContents verifies the exact remaining tokens in order.
        assertTokenStreamContents(filtered, new String[] { "ccc", "dddd", "eeeee" });
    }

    public void testKeepsAllWhenMinLengthIsOne() throws Exception {
        WhitespaceTokenizer src = new WhitespaceTokenizer();
        src.setReader(new StringReader("a bb ccc"));
        TokenStream filtered = new SkipShortTokenFilter(src, 1);
        assertTokenStreamContents(filtered, new String[] { "a", "bb", "ccc" });
    }

    public void testEmptyInputProducesNoTokens() throws Exception {
        WhitespaceTokenizer src = new WhitespaceTokenizer();
        src.setReader(new StringReader(""));
        TokenStream filtered = new SkipShortTokenFilter(src, 3);
        assertTokenStreamContents(filtered, new String[0]);
    }
}

Run it:

./gradlew :plugins:analysis-skip-short:test --tests "*SkipShortTokenFilterTests"

assertTokenStreamContents checks not just the token texts but that the stream reset()s, close()s, and end()s correctly — it catches the classic bug of a filter that forgets to propagate end()/reset to its input. Add a YAML REST test (src/yamlRestTest/...) for the _analyze behavior as well if you enabled opensearch.yaml-rest-test.

Step 11: Precommit and CHANGELOG

./gradlew spotlessApply
./gradlew :plugins:analysis-skip-short:precommit

Add a CHANGELOG.md entry and commit with DCO sign-off:

### Added
- New `analysis-skip-short` plugin adding a `skip_short` token filter ([#NNNNN](https://github.com/opensearch-project/OpenSearch/pull/NNNNN))

git checkout -b feature/analysis-skip-short
git add plugins/analysis-skip-short CHANGELOG.md settings.gradle
git commit -s -m "Add analysis-skip-short plugin with a skip_short token filter"

Implementation Requirements / Deliverables

SkipShortTokenFilter extends TokenFilter with a correct incrementToken() loop.
SkipShortTokenFilterFactory extends AbstractTokenFilterFactory reading min_length with a default and validation.
SkipShortTokensPlugin extends Plugin implements AnalysisPlugin, registering skip_short in getTokenFilters().
A build.gradle applying opensearch.opensearchplugin (descriptor generated from it).
The plugin builds (:plugins:analysis-skip-short:assemble) and loads (_cat/plugins lists it).
_analyze demonstrates short tokens dropped, both standalone and via a custom analyzer on an index.
A passing OpenSearchTokenStreamTestCase unit test (assertTokenStreamContents) covering drop, keep-all, and empty-input.
precommit passes; CHANGELOG.md entry added; DCO-signed commit; SPDX headers on every file.

Troubleshooting

Symptom	Likely cause	Fix
`_cat/plugins` doesn't list it	Plugin not installed / not on the run classpath	Use `./gradlew run -PinstalledPlugins="['analysis-skip-short']"` or install the zip
`unknown filter type [skip_short]`	Name in `getTokenFilters()` ≠ name in the analyzer config	Make both `skip_short`
`IllegalArgumentException: classname` on load	`classname` in descriptor wrong	Point it at the FQN of `SkipShortTokensPlugin`
Plugin fails version check on install	Built against a different OpenSearch version	Build the plugin from the same checkout as the node
`SkipShortTokenFilterFactory::new` won't compile	Constructor signature mismatch	Match `(IndexSettings, Environment, String, Settings)` exactly
`assertTokenStreamContents` fails on `end()`/`reset()`	Filter doesn't delegate to input properly	`TokenFilter` base handles it; ensure you call `super(input)` and loop on `input.incrementToken()`
`licenseHeaders` precommit failure	Missing SPDX header	Add it to every `.java` file

Expected Output

# _cat/plugins
name   component             version
node-0 analysis-skip-short   3.0.0-SNAPSHOT

# _analyze with min_length: 4 on "the quick brown fox is up"
tokens: [ "quick", "brown" ]

# Unit test
> Task :plugins:analysis-skip-short:test
org.opensearch.analysis.skipshort.SkipShortTokenFilterTests > testDropsShortTokens PASSED
org.opensearch.analysis.skipshort.SkipShortTokenFilterTests > testKeepsAllWhenMinLengthIsOne PASSED
org.opensearch.analysis.skipshort.SkipShortTokenFilterTests > testEmptyInputProducesNoTokens PASSED
BUILD SUCCESSFUL

Stretch Goals

Add an uppercase filter too. Implement UpperCaseTokenFilter (mutate CharTermAttribute in place with Character.toUpperCase) and register it as to_upper. This teaches mutating a token vs dropping one. Verify with _analyze.
Register a tokenizer. Override getTokenizers() to add a simple custom Tokenizer (e.g. a fixed-length n-gram or a regex split). Tokenizers are the source of the stream, not a decorator — note how the lifecycle differs.
Make it a pre-configured filter. Override getPreConfiguredTokenFilters() so skip_short is usable without an explicit min_length (a sensible default), the way lowercase is built in.
Fix position increments. Make the filter bump PositionIncrementAttribute for dropped tokens so phrase queries across a gap behave correctly. Add a test asserting positions.
Add a YAML REST test. Under src/yamlRestTest/resources/rest-api-spec/test/, write a do: indices.analyze + match: test for the filter and run :plugins:analysis-skip-short:yamlRestTest.

Validation / Self-check

Name the four artifacts a token-filter plugin needs and the single responsibility of each.
In incrementToken(), why is it a while loop and not an if? What does returning false mean?
Why must you never cache a CharTermAttribute's value across incrementToken() calls?
What does AbstractTokenFilterFactory give you, and where do you read the min_length setting?
How does the registered name (skip_short) connect the analyzer config in _settings to your Java factory? Where is that mapping declared?
Why is OpenSearchTokenStreamTestCase + assertTokenStreamContents a better test than the _analyze curl? What does it check that curl can't easily?
Which Gradle plugin builds the installable zip and generates plugin-descriptor.properties, and what does classname in that descriptor point to?

When your plugin builds, loads, drops short tokens through _analyze, and passes its token-stream unit test, you have completed Level 6. You now understand the write path from REST to Lucene, can operate the engine's visibility/durability mechanisms, and have extended OpenSearch with a real plugin. Continue to Level 7: Search and Aggregation Execution, and deepen the theory with the plugin architecture and mapping & analysis deep dives.

Open-Source Engineer & Contributor