SIMD and the Panama Vector API

The HNSW chapter showed which vectors a query compares. This chapter is about the comparison itself — the single hottest loop in all of vector search. Every greedy hop in an HNSW walk, every candidate in the layer-0 beam, every node inserted during a merge, costs one distance computation: a dot product, a squared Euclidean distance, or a cosine over two float[]s of 768, 1024, or more dimensions. A single query touches hundreds or thousands of these. A merge touches millions. If you make that loop 4× faster, you make vector search ~4× faster and vector indexing dramatically faster. That is the entire "vectorization" theme.

The word vector is overloaded here, so be precise:

• An embedding vector is the data — a float[768] representing a document.
• A SIMD vector (the topic of this chapter) is a hardware register that holds several floats at once and operates on all of them in one instruction. SIMD = Single Instruction, Multiple Data.

This chapter covers why distance is the hot loop, the Panama Vector API (jdk.incubator.vector) that lets Java code emit SIMD, how Lucene's VectorUtil + VectorizationProvider selects a SIMD-accelerated implementation or a scalar fallback, how MemorySegment-based scoring works, how to prove SIMD is active, and how OpenSearch and the k-NN plugin inherit all of it for free.

After this chapter you should be able to: write a scalar dot product and its Panama-vectorized twin; explain how the JIT maps Panama lanes onto AVX-512 / AVX2 / ARM NEON; describe how Lucene chooses the implementation at runtime; verify SIMD is engaged on a running node; and name the JDK flags and pitfalls that silently disable it.

Note: This is the layer beneath HNSW and beneath OpenSearch's k-NN engines. The lucene engine gets SIMD through Lucene's VectorUtil; the native faiss engine gets it through its own C++ SIMD kernels (compiled with -mavx2 etc.). Both ultimately ride the same hardware instructions — see k-NN native integration.

Why distance is the hot loop

A dot product over d dimensions is d multiplies and d-1 adds. Look at the call counts:

At 768 dims that is ~1535 float operations per distance. Multiply by the call counts and the CPU spends the overwhelming majority of vector-search time in this one arithmetic kernel. Profilers of vector workloads light up on dotProduct / squareDistance / cosine. Everything else — graph traversal bookkeeping, queue management — is noise by comparison. Therefore: make the kernel fast and you make the system fast. A scalar Java loop processes one float per instruction; a SIMD loop processes 8 (AVX2, 256-bit) or 16 (AVX-512, 512-bit) floats per instruction. That is the 4–8× that matters.

The Panama Vector API

Historically, Java could not emit SIMD directly — you got whatever auto-vectorization the JIT managed, which for reductions like a dot product was unreliable. Project Panama added the Vector API (jdk.incubator.vector), an incubator module that exposes SIMD as portable Java: you write lane-wise operations against a FloatVector, and the JIT compiles them to the actual vector instructions of the host CPU (AVX-512/AVX2 on x86, NEON/SVE on ARM), or falls back to scalar where unsupported.

Because it is an incubator module, it is not on the module path by default. You must opt in:

java --add-modules jdk.incubator.vector ...      # required to load the module

Lucene knows this and probes for the module at startup; if it is absent, Lucene uses its scalar implementation instead of crashing.

# In an apache/lucene checkout, see where Lucene references the incubator module:
grep -rln "jdk.incubator.vector\|FloatVector\|VectorSpecies\|--add-modules" lucene/core/src/java

The key Panama types

Scalar vs vectorized: the dot product

Here is the canonical kernel both ways. First, the scalar version — clear, correct, one float at a time:

/** Scalar dot product: one multiply-add per loop iteration. */
static float dotScalar(float[] a, float[] b) {
    float sum = 0f;
    for (int i = 0; i < a.length; i++) {
        sum += a[i] * b[i];            // one FMA worth of work per iteration
    }
    return sum;
}

Now the Panama-vectorized version. We process SPECIES.length() floats per iteration (8 for AVX2, 16 for AVX-512), accumulating into a SIMD register, then do a horizontal reduce at the end:

import jdk.incubator.vector.FloatVector;
import jdk.incubator.vector.VectorOperators;
import jdk.incubator.vector.VectorSpecies;

/** Vectorized dot product using the Panama Vector API. */
static float dotVectorized(float[] a, float[] b) {
    final VectorSpecies<Float> SPECIES = FloatVector.SPECIES_PREFERRED; // widest lanes the CPU has
    int i = 0;
    // A running accumulator vector: lane j holds the partial sum for positions j, j+L, j+2L, ...
    FloatVector acc = FloatVector.zero(SPECIES);
    int upper = SPECIES.loopBound(a.length);         // largest multiple of L <= length
    for (; i < upper; i += SPECIES.length()) {
        FloatVector va = FloatVector.fromArray(SPECIES, a, i);
        FloatVector vb = FloatVector.fromArray(SPECIES, b, i);
        acc = va.fma(vb, acc);                       // acc += va * vb, lane-wise, one instruction
    }
    float sum = acc.reduceLanes(VectorOperators.ADD); // horizontal sum across lanes
    for (; i < a.length; i++) {                      // scalar tail for the remainder
        sum += a[i] * b[i];
    }
    return sum;
}

The two structural pieces every vectorized kernel has:

1. The main loop strides by SPECIES.length(), doing the work on a full register per iteration. fma (fused multiply-add) does a*b+c in one instruction with one rounding — both faster and more accurate than separate multiply and add.
2. The tail loop handles the leftover length % laneCount elements scalar-style, because the array length is rarely an exact multiple of the lane count.

Squared Euclidean distance and cosine are the same shape — subtract-square-accumulate for L2, two extra accumulators (for |a|² and |b|²) for cosine.

How the JIT maps lanes to AVX/NEON

FloatVector.SPECIES_PREFERRED is resolved at runtime to the host's widest supported shape. The HotSpot JIT then intrinsifies the Panama operations into the matching machine instructions:

Crucially, the same Java source runs on all of them. You do not write AVX intrinsics; you write lane-wise Java and the JIT picks the instruction. This is what makes the Vector API attractive for a portable library like Lucene that must run on x86 and ARM (Graviton) servers.

Warning: Vectorization is not free correctness. Floating-point addition is not associative, so dotVectorized and dotScalar can produce slightly different sums (different summation order). This is expected and within float tolerance — never assert exact equality between a scalar and a vectorized score in a test; assert within delta.

Lucene's VectorUtil and VectorizationProvider

Lucene does not call Panama from a hundred places. It funnels all vector arithmetic through one class, org.apache.lucene.util.VectorUtil, with methods like dotProduct, squareDistance, cosine, and the int8/byte variants. VectorUtil delegates to a VectorUtilSupport chosen once at class-load time by a VectorizationProvider:

# In an apache/lucene checkout:
find lucene -name "VectorUtil.java" -o -name "VectorizationProvider.java" \
  -o -name "*VectorUtilSupport.java"
grep -n "PanamaVectorUtilSupport\|DefaultVectorUtilSupport\|lookup\|isSupported\|runtimeVersion" \
  lucene/core/src/java/org/apache/lucene/internal/vectorization/VectorizationProvider.java

The selection logic:

flowchart TD
    Start["VectorizationProvider.lookup() (once, at class init)"] --> Mod{"jdk.incubator.vector present<br/>AND JDK version supported<br/>AND CPU has vectors?"}
    Mod -->|yes| Panama["PanamaVectorizationProvider<br/>-> PanamaVectorUtilSupport (SIMD)"]
    Mod -->|no| Scalar["DefaultVectorizationProvider<br/>-> DefaultVectorUtilSupport (scalar)"]
    Panama --> VU["VectorUtil.dotProduct / squareDistance / cosine"]
    Scalar --> VU
    VU --> HNSW["HnswGraphSearcher / VectorScorer / merge"]

The two implementations:

The provider is chosen once per JVM and logged. Because the choice is at the VectorUtil boundary, every consumer — HNSW search, exact scoring, graph construction, merge — automatically gets SIMD when available and the scalar path when not. There is no per-call branch in the hot loop; the implementation is selected before the loop ever runs.

MemorySegment-based scoring

Stored vectors live in the .vec file (HNSW chapter). Modern Lucene reads them through MemorySegment (the Panama Foreign Function & Memory API), which lets the Vector API load lanes directly from the mapped file region without first copying into a float[]. The scorer (RandomVectorScorer / the off-heap VectorScorer) computes a distance straight against the segment-backed data. This avoids an allocation and a copy per vector compared and is a meaningful part of why recent Lucene vector search is fast on large indices that do not fit in heap.

grep -rln "MemorySegment\|OffHeap.*Vector\|RandomVectorScorer" lucene/core/src/java | head

The payoff: ~25% indexing speedups

Faster distance has two effects. At query time, lower per-comparison cost directly lowers HNSW search latency. At index/merge time, graph construction is dominated by distance computations (see the merge-rebuilds-the-graph discussion in the HNSW chapter), so a faster kernel plus an improved HNSW graph merger produced ~25% indexing speedups in Lucene's nightly benchmarks. That number is a combination — better merging algorithm and SIMD scoring — not SIMD alone, but SIMD is a load-bearing part of it. The lesson for a contributor: a change to the distance kernel or the merge path is high-leverage because it multiplies across millions of calls.

How to verify SIMD is actually active

A vectorized library that is silently running the scalar fallback is a classic, invisible performance bug. Prove it is on:

# 1. Confirm the incubator module is on the command line of a running OpenSearch node:
jps -lvm | grep -i opensearch | tr ' ' '\n' | grep -i "add-modules"
#   expect: --add-modules jdk.incubator.vector  (OpenSearch's jvm.options enables it on JDK 21)
ps -ef | grep -i opensearch | grep -o "add-modules [^ ]*"

# 2. In a standalone Lucene/Java program, ask the provider what it picked:
#    System.out.println(VectorizationProvider.getInstance().getClass());
#    -> PanamaVectorizationProvider  (good) vs DefaultVectorizationProvider (scalar)

# 3. Lucene logs the chosen provider; enable its logging or check for the panama class in a heap/thread dump.

# 4. Microbench: time dotScalar vs dotVectorized in a JMH harness; a >2x gap on long vectors
#    confirms SIMD. (No speedup => module missing or species is scalar.)

A minimal self-check program:

import jdk.incubator.vector.FloatVector;
import jdk.incubator.vector.VectorSpecies;

public class SimdCheck {
    public static void main(String[] args) {
        VectorSpecies<Float> s = FloatVector.SPECIES_PREFERRED;
        System.out.println("Preferred lane count: " + s.length()
            + " (vector bits: " + s.vectorBitSize() + ")");
        // 16 => AVX-512, 8 => AVX2, 4 => NEON, 1 => no SIMD / module problem
    }
}

javac --add-modules jdk.incubator.vector SimdCheck.java
java  --add-modules jdk.incubator.vector SimdCheck
#   e.g. "Preferred lane count: 8 (vector bits: 256)"  on an AVX2 box

If you omit --add-modules jdk.incubator.vector, the class won't even load — which is exactly why Lucene guards the Panama path behind a runtime check and ships the scalar fallback.

Common pitfalls

How OpenSearch and k-NN benefit

OpenSearch does nothing special to get this — it inherits Lucene's VectorUtil. Two paths:

• k-NN lucene engine: uses Lucene's HNSW and therefore Lucene's VectorUtil. As long as the node runs with --add-modules jdk.incubator.vector (it does, via the bundled jvm.options on JDK 21), vector scoring is SIMD-accelerated for free. An OpenSearch Lucene upgrade that improves VectorUtil improves k-NN's lucene engine with zero plugin changes.
• k-NN faiss engine: does not use Java SIMD — its native C++ library is compiled with AVX2/AVX-512 (and NEON on ARM) kernels. The hardware speedup is the same idea, achieved in C++ rather than Panama. See k-NN native integration and memory.

So "is SIMD on?" is a real operational question for an OpenSearch operator running vector search: check the JVM flags (above) for the lucene engine, and check the faiss build variant for the native engine.

Reading exercise

# In an apache/lucene checkout:

# 1. The funnel: every vector arithmetic op goes through VectorUtil.
grep -n "dotProduct\|squareDistance\|cosine\|IMPL" \
  lucene/core/src/java/org/apache/lucene/util/VectorUtil.java | head

# 2. The runtime selection of SIMD vs scalar.
grep -n "PanamaVector\|DefaultVector\|lookup\|getInstance" \
  lucene/core/src/java/org/apache/lucene/internal/vectorization/VectorizationProvider.java

# 3. The Panama implementation itself — find the FloatVector species and fma.
grep -rn "FloatVector\|SPECIES\|fma\|reduceLanes" \
  $(grep -rln "class PanamaVectorUtilSupport" lucene/core)

# 4. MemorySegment-based scoring.
grep -rln "MemorySegment\|RandomVectorScorer" lucene/core/src/java | head

# 5. Verify on a running node.
ps -ef | grep -i opensearch | grep -o "add-modules [^ ]*"

Answer:

1. Why is the distance kernel the right place to optimize vector search? Quantify roughly how many times it runs for one query vs one merge.
2. Write (from memory) the structure of a Panama dot product: the main lane-striding loop and the tail. What does fma do, and why is the tail necessary?
3. How does the same Panama Java source end up as AVX-512 on one machine and NEON on another?
4. Trace how VectorUtil.dotProduct decides between SIMD and scalar. When is the decision made, and why is there no per-call branch in the hot loop?
5. Give two independent ways to prove SIMD is actually active on a running OpenSearch node.
6. The "~25% indexing speedup" came from two things — name both, and explain why a distance-kernel change is high-leverage.

Validation: prove you understand this

1. Implement squareDistance both scalar and Panama-vectorized; assert they agree within a delta (and explain why exact equality would be wrong).
2. Explain VectorizationProvider selecting PanamaVectorUtilSupport vs DefaultVectorUtilSupport: what triggers each, and when the choice is made.
3. State the single JDK flag that gates the Vector API and what happens to Lucene without it.
4. Explain how MemorySegment-based scoring avoids a copy that an array-based scorer would incur.
5. Describe how a k-NN lucene-engine workload and a k-NN faiss-engine workload each obtain hardware SIMD — and why the answer is different.
6. List three ways SIMD can be silently disabled or negated, and how you would detect each.

When you can do all six, you understand the vectorization theme end to end — from the HNSW graph walk down to the AVX instruction. Now make it concrete: Lab L1 dissects the files these kernels read, and Lab L3 builds and benchmarks a real HNSW index whose every comparison runs this loop. For the native side, see k-NN native integration and memory.