Mapping and Analysis

Before a document can be indexed or a field can be searched, OpenSearch must know its schema — what fields exist and how each is typed and stored — and it must know how to analyze text into the terms that go into the inverted index. Those two jobs are mapping and analysis. Mapping (MapperService, DocumentMapper, FieldMapper/MappedFieldType) defines field types and how they are indexed; analysis (Analyzer, Tokenizer, TokenFilter, CharFilter, AnalysisRegistry) defines how text fields are broken into tokens. This chapter covers both, plus dynamic mapping, the _source/_doc model, the built-in and analysis-common analyzers, how an AnalysisPlugin extends the chain, and the _analyze API for inspecting it directly.

After this chapter you should be able to: explain the difference between a FieldMapper and a MappedFieldType; describe how a text field's value is turned into index terms; explain dynamic mapping and why it is both convenient and dangerous; and use _analyze to debug why a query does not match.

Note: Mapping and analysis are index-time and query-time concerns living inside a single index. They feed engine-internals.md (what gets written to Lucene) and query-dsl-querybuilders.md (how a query term is analyzed to match). Get analysis wrong and "matching" documents silently fail to match.

The mapping classes

find server -name "MapperService.java" -o -name "DocumentMapper.java" -o -name "Mapping.java" \
  -o -name "FieldMapper.java" -o -name "MappedFieldType.java"
grep -n "class MapperService\|merge\|documentMapper\|fieldType\|class DocumentMapper" \
  server/src/main/java/org/opensearch/index/mapper/MapperService.java | head

The FieldMapper vs MappedFieldType split is the single most important idea here:

flowchart LR
    Doc["JSON document"] -->|index time| DM[DocumentMapper.parse]
    DM --> FM["FieldMapper per field"]
    FM --> Lucene["Lucene Document (indexed fields, doc values, stored _source)"]
    Query["query value"] -->|query time| MFT["MappedFieldType.termQuery/rangeQuery"]
    MFT --> LQ["Lucene Query"]

Field types

find server modules -name "*FieldMapper.java" | head -40
grep -rn "contentType()\|CONTENT_TYPE = " server/src/main/java/org/opensearch/index/mapper/ | head

The core field types you must know:

Note: The classic gotcha: a text field has no doc values, so you cannot sort or aggregate on it directly — you get an error pointing you at fielddata (docvalues-fielddata.md). The idiomatic fix is a keyword multi-field (text with a .keyword sub-field). This is the most common mapping mistake new users hit.

_source and the document model

find server -name "SourceFieldMapper.java" -o -name "DocumentParser.java"
grep -n "_source\|_id\|_routing\|_field_names\|class .*FieldMapper" \
  server/src/main/java/org/opensearch/index/mapper/SourceFieldMapper.java | head

_source is what makes OpenSearch feel document-oriented: Lucene stores inverted-index terms, but _source keeps the original document so you can return it, reindex it, and run partial updates. Disabling _source saves space but breaks update, reindex, and some highlighting — a frequently regretted optimization.

Dynamic mapping

If a document contains a field with no mapping, OpenSearch can infer one:

grep -n "dynamic\|DynamicTemplate\|parseDynamicValue\|createDynamicUpdate\|Dynamic\b" \
  server/src/main/java/org/opensearch/index/mapper/DocumentParser.java | head

flowchart TD
    Doc["doc has field 'price' (unmapped)"] --> Dyn{index.mapping.dynamic}
    Dyn -->|true| Infer["infer type -> add mapping via cluster-state update"]
    Dyn -->|runtime| RT[add as runtime field]
    Dyn -->|false| Ignore[store in _source only]
    Dyn -->|strict| Reject[reject document]

Warning: Dynamic mapping adds fields by issuing a cluster-state update (the new mapping must be published — see cluster-state-publishing.md). Unbounded dynamic fields cause mapping explosion: thousands of fields bloat the cluster state, slow publishing, and can exhaust the field limit (index.mapping.total_fields.limit). Untrusted/high-cardinality keys (e.g. user-supplied JSON object keys) are the usual culprit. Prefer strict or dynamic: false for such data, or use flattened-style modeling.

The analysis chain

Analysis converts a text value into a stream of index terms. The chain has three stages, applied in order:

find server -name "AnalysisRegistry.java"
find modules/analysis-common -name "*.java" | head
grep -n "class AnalysisRegistry\|getAnalyzer\|buildAnalyzer\|tokenizers\|tokenFilters\|charFilters" \
  server/src/main/java/org/opensearch/index/analysis/AnalysisRegistry.java | head

flowchart LR
    Raw["raw text: 'The Quick-Brown Fox!'"] --> CF[CharFilter: strip/replace chars]
    CF --> TOK[Tokenizer: split into tokens]
    TOK --> TF[TokenFilter chain: lowercase, stop, stem, ...]
    TF --> Terms["index terms: quick, brown, fox"]

An Analyzer is the assembled pipeline (CharFilters → Tokenizer → TokenFilters). Built-in analyzers combine these:

Most non-trivial analyzers (language analyzers, many tokenizers/filters) live in the bundled modules/analysis-common module, not in server:

find modules/analysis-common/src/main/java -name "*TokenFilterFactory.java" | head
find modules/analysis-common/src/main/java -name "*TokenizerFactory.java" | head

Note: Index-time and query-time analysis can differ. A field has an analyzer (index time) and a search_analyzer (query time). If they disagree, a query term can be analyzed into something that never matches the indexed terms — the classic "my search returns nothing" bug. Use _analyze (below) to see both.

Extending analysis with AnalysisPlugin

A plugin contributes new tokenizers/filters/analyzers via AnalysisPlugin:

find server -name "AnalysisPlugin.java"
grep -n "getTokenizers\|getTokenFilters\|getCharFilters\|getAnalyzers" \
  server/src/main/java/org/opensearch/plugins/AnalysisPlugin.java

public class MyAnalysisPlugin extends Plugin implements AnalysisPlugin {
    @Override
    public Map<String, AnalysisProvider<TokenFilterFactory>> getTokenFilters() {
        return Map.of("my_filter", MyTokenFilterFactory::new);
    }
}

The plugins/analysis-icu, analysis-kuromoji, etc. plugins are exactly this: factories registered into the AnalysisRegistry. The mechanism (factories, AnalysisProvider, classloader isolation) is covered in plugin-architecture.md. Building such a plugin is the spirit of the analysis plugin labs and Level 7.

The _analyze API: your debugging tool

The single best tool for understanding and debugging analysis:

# Run an analyzer and see the tokens it produces.
curl -s -X POST "localhost:9200/_analyze?pretty" -H 'Content-Type: application/json' -d '{
  "analyzer": "standard",
  "text": "The Quick-Brown Fox!"
}'

# Build an ad-hoc chain (char filter + tokenizer + filters).
curl -s -X POST "localhost:9200/_analyze?pretty" -H 'Content-Type: application/json' -d '{
  "char_filter": ["html_strip"],
  "tokenizer": "standard",
  "filter": ["lowercase", "stop"],
  "text": "<b>The</b> Quick Fox"
}'

# Analyze with the actual field mapping (uses the field's configured analyzer).
curl -s -X POST "localhost:9200/orders/_analyze?pretty" -H 'Content-Type: application/json' -d '{
  "field": "description",
  "text": "Wireless-Headphones"
}'

The output lists each token with its start_offset/end_offset/position. When a search "should match but doesn't," run the document text through the field's index analyzer and the query text through the search analyzer and compare the token sets — they must overlap.

Reading exercise

# 1. The mapper split.
grep -n "class FieldMapper\|parseCreateField\|class MappedFieldType\|termQuery\|rangeQuery\|hasDocValues" \
  server/src/main/java/org/opensearch/index/mapper/FieldMapper.java \
  server/src/main/java/org/opensearch/index/mapper/MappedFieldType.java | head

# 2. Dynamic mapping inference.
grep -n "createDynamicUpdate\|parseDynamicValue\|Dynamic\." \
  server/src/main/java/org/opensearch/index/mapper/DocumentParser.java | head

# 3. The analysis registry and built-ins.
grep -n "getAnalyzer\|buildMapping\|PreBuiltAnalyzers\|standard\|keyword" \
  server/src/main/java/org/opensearch/index/analysis/AnalysisRegistry.java | head

# 4. analysis-common contents.
ls modules/analysis-common/src/main/java/org/opensearch/analysis/common/ | head -40

# 5. Tests + the API.
./gradlew :server:test --tests "org.opensearch.index.mapper.MapperServiceTests"
./gradlew :modules:analysis-common:test --tests "*Standard*" 2>/dev/null || true
curl -s -X POST "localhost:9200/_analyze?pretty" -H 'Content-Type: application/json' \
  -d '{"analyzer":"standard","text":"Hello World"}'

Answer:

1. Explain the division of labor between FieldMapper and MappedFieldType. Which one is used when you run a query, and which when you index a document?
2. Why can you not sort or aggregate on a text field, and what is the idiomatic fix? Where is the doc-values distinction encoded?
3. Walk the three stages of the analysis chain for the input "<b>The</b> Quick-Brown Fox!" through html_strip + standard + lowercase. List the resulting tokens.
4. What does dynamic mapping do under the hood that makes mapping explosion a cluster-wide (not just index-local) problem?
5. What is the difference between an index analyzer and a search_analyzer, and what bug appears when they disagree?
6. Use _analyze to show that keyword and standard analyzers produce different token sets for the same input.

Common bugs and symptoms

Validation: prove you understand this

1. Draw the index-time path (DocumentMapper → FieldMapper → Lucene Document) and the query-time path (MappedFieldType → Lucene Query), and explain why the split exists.
2. Explain the text vs keyword distinction in terms of analysis and doc values, and give the standard multi-field pattern.
3. From memory, list the three analysis stages and one example component of each, and trace one input string through them.
4. Explain how dynamic mapping interacts with cluster state and why mapping explosion is dangerous; name the setting that bounds it and the dynamic modes that prevent it.
5. Explain index vs search analyzer and demonstrate, with _analyze, a mismatch that causes a non-matching search.
6. Write the AnalysisPlugin method signature that registers a custom token filter, and name two in-repo plugins that work this way.