DocValues: The Columnar Store

The inverted index answers "which docs contain term X." The exact opposite question — "given doc D, what is the value of field F" — is what sorting, aggregating, and scripting need, and the inverted index is terrible at it. Lucene's answer is doc values: a columnar, per-document, on-disk store, written into each segment alongside the postings. This is the structure behind every OpenSearch sort, every terms/stats/cardinality aggregation, and every doc['field'].value script access.

This chapter is the Lucene-level companion to OpenSearch's DocValues and Fielddata deep dive. That chapter covered the OpenSearch view — the IndexFieldData abstraction, the fielddata circuit breaker, global ordinals, why text is special. Here we go a layer down: the Lucene DocValuesType flavors, the DocValuesFormat that encodes them, the per-doc iterator APIs, and how ordinals work at the format level. We do not re-derive the OpenSearch side — read that chapter for it; this one is the storage format underneath.

After this chapter you can: name the five DocValuesType flavors and the Lucene API for each; explain the columnar encoding and why it's disk/page-cache friendly; describe how SORTED/SORTED_SET store ordinals + a sorted term dictionary; explain global ordinals at the format level; and trace how OpenSearch reads a keyword field's values for an aggregation down to the Lucene accessor.

Row store vs column store

Stored fields (.fdt, the _source) are a row store: doc 5's whole document sits together, great for "fetch doc 5," terrible for "read field F across a million docs" (you'd touch a million scattered rows). Doc values are a column store: all values of field F sit together, contiguous, so "read F across all docs" is a sequential scan of one tight array — cache-friendly, off-heap, OS-page-cached.

grep -rn "enum DocValuesType\|class DocValuesFormat" \
  lucene/core/src/java/org/apache/lucene/index/DocValuesType.java \
  lucene/core/src/java/org/apache/lucene/codecs/DocValuesFormat.java

The five DocValuesType flavors

Lucene stores doc values as one of five column types, chosen at index time per field. Each has a matching read API (an iterator that advances doc-by-doc).

The read APIs are all DocValuesIterators — they extend DocIdSetIterator, so you advance(docID) to a doc and then read its value. This iterator model (introduced when doc values became sparse-aware) means a field that's absent on many docs costs nothing for those docs.

// NUMERIC: read a long per doc
NumericDocValues nd = leaf.getNumericDocValues("price");
if (nd.advanceExact(docID)) {
    long v = nd.longValue();
}

// SORTED_SET: read ordinals, resolve to bytes only when needed
SortedSetDocValues ss = leaf.getSortedSetDocValues("tags.keyword");
if (ss.advanceExact(docID)) {
    long ord;
    while ((ord = ss.nextOrd()) != SortedSetDocValues.NO_MORE_ORDS) {
        // bucket by ord (cheap); resolve text only to emit:
        BytesRef term = ss.lookupOrd(ord);
    }
}

grep -rn "class NumericDocValues\|class SortedDocValues\|class SortedSetDocValues\|class SortedNumericDocValues\|class BinaryDocValues" \
  lucene/core/src/java/org/apache/lucene/index/
grep -rn "advanceExact\|nextOrd\|lookupOrd\|getValueCount" \
  lucene/core/src/java/org/apache/lucene/index/SortedSetDocValues.java

Ordinals: the key trick for strings

SORTED/SORTED_SET do not store the strings per doc. They store, per doc, a small integer ordinal into a per-segment sorted dictionary of the field's distinct terms. So a keyword field with a million docs but only 50 distinct values stores a million tiny ordinals + a 50-entry dictionary, not a million strings.

flowchart LR
    Dict["Per-segment sorted dict:<br/>0=apple 1=mango 2=pear"] 
    Doc0["doc0 -> ord 2"] --> Dict
    Doc1["doc1 -> ord 0"] --> Dict
    Doc2["doc2 -> ord {0,2}"] --> Dict
    Dict -->|lookupOrd| Bytes["BytesRef('pear')"]

This is why string aggregations are fast: a terms agg buckets by ordinal (integer compares, integer hash keys), and only calls lookupOrd to turn the winning ordinals back into strings at the very end. Ordinals are also already sorted (the dictionary is sorted), so a sort on a keyword field is a sort on ordinals — no string comparison until the final fetch.

Note: Ordinals are per-segment. Ordinal 2 in segment A and ordinal 2 in segment B may be different terms. To aggregate across a whole shard you need a shard-wide numbering — global ordinals, below.

The DocValuesFormat and the .dvd/.dvm files

The on-disk encoding is defined by the codec's DocValuesFormat sub-format (see Segments and Codecs). It writes two files:

The default format is aggressively compact. Numerics are stored with techniques chosen from the data: delta/GCD encoding (store differences or a common factor), table/ordinal encoding when few distinct values, and bit-packing to the minimum bits per value. Sorted dictionaries are prefix-compressed. None of this is loaded onto the JVM heap — it's mmap'd and served from the OS page cache, which is exactly why doc values do not count against the fielddata circuit breaker (the OpenSearch deep dive's point, now with the format reason underneath).

ls lucene/core/src/java/org/apache/lucene/codecs/lucene*/   # *DocValuesFormat
grep -rln "DocValuesFormat" lucene/core/src/java/org/apache/lucene/codecs/
grep -rn "class .*DocValuesConsumer\|class .*DocValuesProducer" \
  lucene/core/src/java/org/apache/lucene/codecs/lucene*/ | head

The format splits into a consumer (write side, DocValuesConsumer) and a producer (read side, DocValuesProducer) — the same writer/reader split you've now seen for postings (BlockTree) and points (BKD). It's the consistent shape of every Lucene format.

Global ordinals at the format level

A terms agg needs one ordinal space for the whole shard, but each segment has its own. Lucene builds an OrdinalMap: it merges the per-segment sorted dictionaries into one shard-global sorted dictionary and produces, per segment, a mapping localOrd → globalOrd. Now an aggregator can bucket by global ordinal across all segments and resolve to text once at the end.

flowchart LR
    SA["Seg A: 0=apple 1=pear"] --> OM[OrdinalMap]
    SB["Seg B: 0=pear 1=plum"] --> OM
    OM --> G["Global: 0=apple 1=pear 2=plum"]
    OM --> MapA["A: local 0->0, 1->1"]
    OM --> MapB["B: local 0->1, 1->2"]

grep -rn "class OrdinalMap\|build(" \
  lucene/core/src/java/org/apache/lucene/index/OrdinalMap.java
grep -rn "MultiDocValues\|getSortedSetValues\|OrdinalMap" \
  lucene/core/src/java/org/apache/lucene/index/MultiDocValues.java

This OrdinalMap is precisely what the OpenSearch docvalues/fielddata chapter calls "global ordinals" — built lazily on first agg, cached per shard, invalidated when segments change (a refresh). eager_global_ordinals: true just moves the OrdinalMap build to refresh time. The Lucene class is the implementation; the OpenSearch chapter is the operational behavior. Read both.

How OpenSearch enables and reads doc values

By default OpenSearch sets doc_values: true on keyword, numeric, date, boolean, ip, and geo_point fields — and false on text (a token stream has no single value). The field mapper declares the DocValuesType, and at query time OpenSearch's IndexFieldData reads the matching Lucene accessor.

# Where field types declare doc values:
grep -rn "hasDocValues\|DocValuesType\|SortedSetDocValuesField\|SortedNumericDocValuesField\|NumericDocValuesField" \
  server/src/main/java/org/opensearch/index/mapper/KeywordFieldMapper.java \
  server/src/main/java/org/opensearch/index/mapper/NumberFieldMapper.java

# The bridge from OpenSearch fielddata to Lucene doc-values accessors:
grep -rn "getSortedSetDocValues\|getNumericDocValues\|SortedSetDocValues\|LeafFieldData" \
  server/src/main/java/org/opensearch/index/fielddata/ | head

The chain for a keyword terms agg, end to end:

flowchart TD
    Agg["terms agg on tags.keyword"] --> IFD["IndexFieldData (OpenSearch)"]
    IFD --> LFD["LeafFieldData per segment"]
    LFD --> SSDV["SortedSetDocValues (Lucene)"]
    SSDV --> Ord["nextOrd() -> local ordinals"]
    Ord --> GO["OrdinalMap: local -> global ordinal"]
    GO --> Bucket["bucket by global ord; lookupOrd at the end"]

That's the same picture the OpenSearch deep dive draws as MappedFieldType → IndexFieldData → LeafFieldData → Lucene accessor — now you can name the Lucene end of it: SortedSetDocValues + OrdinalMap, encoded in .dvd/.dvm by the DocValuesFormat.

Reading exercise

# 1. The five types and their accessors.
grep -rn "enum DocValuesType" lucene/core/src/java/org/apache/lucene/index/DocValuesType.java
grep -rn "class SortedSetDocValues\|class NumericDocValues\|class SortedNumericDocValues" \
  lucene/core/src/java/org/apache/lucene/index/

# 2. Ordinals.
grep -rn "lookupOrd\|nextOrd\|getValueCount\|ordValue" \
  lucene/core/src/java/org/apache/lucene/index/SortedSetDocValues.java \
  lucene/core/src/java/org/apache/lucene/index/SortedDocValues.java

# 3. The format: consumer/producer and files.
ls lucene/core/src/java/org/apache/lucene/codecs/lucene*/
grep -rn "DocValuesConsumer\|DocValuesProducer" lucene/core/src/java/org/apache/lucene/codecs/ | head

# 4. Global ordinals (OrdinalMap).
grep -rn "class OrdinalMap\|OrdinalMap.build" lucene/core/src/java/org/apache/lucene/index/OrdinalMap.java

# 5. OpenSearch side (cross-link to docvalues-fielddata.md).
grep -rn "SortedSetDocValuesField\|NumericDocValuesField\|hasDocValues" \
  server/src/main/java/org/opensearch/index/mapper/KeywordFieldMapper.java

# 6. The files on disk.
find /path/to/shard/index -name "*.dvd" -o -name "*.dvm"

Answer:

1. Contrast a row store (stored fields) and a column store (doc values) and say which access pattern each is built for.
2. Name the five DocValuesType flavors, the read API for each, and an OpenSearch field type that uses each.
3. Explain ordinals: what is stored per doc for a keyword, why aggregations bucket on ordinals, and when lookupOrd is called.
4. Why do doc values not count against the fielddata circuit breaker? Tie it to where the .dvd bytes live.
5. Explain OrdinalMap (global ordinals) at the format level and how it relates to eager_global_ordinals in OpenSearch.
6. Trace a keyword terms agg from the OpenSearch IndexFieldData to the Lucene SortedSetDocValues and OrdinalMap.

Common bugs and symptoms

Validation: prove you understand this

1. Draw the column layout for one keyword field across three docs, showing the sorted dictionary and the per-doc ordinals.
2. Explain why ordinals make both sorting and terms aggregation fast, and where lookupOrd enters.
3. Name the five DocValuesTypes and give the Lucene accessor and an OpenSearch field type for each.
4. Explain the DocValuesFormat consumer/producer split and the .dvd/.dvm files, and one numeric encoding technique it uses.
5. Define OrdinalMap and connect it precisely to the OpenSearch notion of global ordinals and eager_global_ordinals.
6. Trace, naming every layer, how a terms agg on tags.keyword reads values from OpenSearch down to the Lucene SortedSetDocValues in a .dvd.