Lab DP3: Heap and Thread Dumps

Background

Lab DP2 profiled active work — where CPU and allocations go. This lab handles the two failures where the node isn't busy doing the wrong thing, it's stuck or out of memory:

• A thread dump (jstack / jcmd Thread.print) is a snapshot of every thread's stack and state. It is how you diagnose a deadlock, a blocked thread pool, or a thread parked on a lock it will never get.
• A heap dump (jcmd GC.heap_dump / jmap) is a snapshot of every live object. Opened in Eclipse Memory Analyzer (MAT), its dominator tree tells you which object is retaining the most heap — the leak or the one giant structure.
• GC logs (-Xlog:gc*) tell you whether GC is keeping up, which decides whether a high-heap symptom is a leak (heap dump) or just thrash (allocate less / size the heap).

You will build a small standalone deadlock, dump it, and read jstack's own deadlock detector; then take a heap dump of an OpenSearch node holding a large fielddata structure and find that culprit in MAT's dominator tree; then read a GC log to tell leak from thrash.

Why This Matters for Contributors

"The node hung" and "the node OOM'd" are among the highest-severity issues in the tracker, and they are un-fixable without these two dumps. A maintainer triaging a hang will ask for a thread dump; one triaging an OOM will ask for the heap-dump analysis. Being able to take both, read jstack's Found one Java-level deadlock block, and walk a MAT dominator tree to a named class is the difference between a useful bug report and "it broke." The fielddata example here is a real OpenSearch footgun — aggregating/sorting a text field loads field data on heap — and recognizing its shape in a heap dump is a transferable skill.

Prerequisites

• A JDK with jstack, jcmd, jmap (all ship with the JDK).
• Eclipse MAT: download from https://eclipse.dev/mat/. Set its own heap (MemoryAnalyzer.ini, -Xmx) at least as large as the dumps you'll open.
• An OpenSearch checkout you can ./gradlew run, with the shop index from Lab DP1.
• Read the intensive sections "Why is it stuck?" and "Why is it OOM?", and the circuit-breakers-memory deep-dive (the breakers that try to stop the OOM first).

Warning: A heap dump is roughly the size of the live heap and pauses the JVM while it writes. On a real cluster, take it on a node you can afford to stall (or one already failing), and write it to fast local disk with room to spare.

Step-by-Step Tasks

Step 1 — Build and run a deterministic deadlock

The cleanest way to learn to read a thread dump is to dump a deadlock you authored. Two threads, two locks, opposite order — the textbook cycle:

// Deadlock.java
public class Deadlock {
  static final Object A = new Object();
  static final Object B = new Object();
  public static void main(String[] args) throws Exception {
    Thread t1 = new Thread(() -> {
      synchronized (A) {
        sleep(200);
        synchronized (B) { System.out.println("t1 got both"); }
      }
    }, "worker-1");
    Thread t2 = new Thread(() -> {
      synchronized (B) {
        sleep(200);
        synchronized (A) { System.out.println("t2 got both"); }
      }
    }, "worker-2");
    t1.start(); t2.start();
    t1.join(); t2.join();   // never returns -- deadlocked
  }
  static void sleep(long ms){ try { Thread.sleep(ms);} catch(Exception e){} }
}

javac Deadlock.java
java Deadlock &        # hangs forever
DPID=$!
sleep 1

Step 2 — Take a thread dump and read the deadlock

jstack -l "$DPID" > /tmp/deadlock.txt      # -l also dumps lock ownership
# (equivalently: jcmd "$DPID" Thread.print > /tmp/deadlock.txt)
sed -n '/Found one Java-level deadlock/,/^$/p' /tmp/deadlock.txt

jstack detects the cycle and prints it for you — this block is the whole answer:

Found one Java-level deadlock:
=============================
"worker-1":
  waiting to lock monitor 0x... (object 0x...a, a java.lang.Object),
  which is held by "worker-2"
"worker-2":
  waiting to lock monitor 0x... (object 0x...b, a java.lang.Object),
  which is held by "worker-1"

Read the cycle: worker-1 holds A, wants B; worker-2 holds B, wants A. The fix is always consistent lock ordering — both threads must acquire A before B. Now find the same shapes in the per-thread section:

grep -A6 '"worker-1"\|"worker-2"' /tmp/deadlock.txt | grep -E 'State|locked|waiting to lock'

kill "$DPID" 2>/dev/null

Note: In OpenSearch this exact pattern appears as "all [search] threads BLOCKED waiting to lock the same <0x...>, held by one stuck thread" — a wedged fixed pool. The dump reading is identical; only the thread names and the held object change. Take three dumps a few seconds apart: if the same threads sit in the same BLOCKED state across all three, it's a real hang, not a momentary contention blip.

Step 3 — Create heap pressure: a large fielddata structure

Now the OOM side. The classic OpenSearch heap eater is fielddata: aggregating or sorting on a text field builds an in-memory, per-segment structure on the JVM heap (unlike keyword/doc-values fields, which read off-heap from disk — see docvalues-fielddata). We'll force it.

# Map a text field with fielddata enabled (normally forbidden for good reason):
curl -s -XPUT 'localhost:9200/fd_demo?pretty' -H 'Content-Type: application/json' -d'
{ "settings": { "number_of_shards": 1, "number_of_replicas": 0 },
  "mappings": { "properties": {
    "text_hi_card": { "type": "text", "fielddata": true }
  }}}'

# Load many docs with high-cardinality text so the fielddata is large:
python3 - <<'PY' > /tmp/fd.ndjson
for i in range(100000):
    print('{"index":{}}')
    print('{"text_hi_card":"token-%d term-%d uniqueish-%d"}' % (i, i%5000, i))
PY
curl -s -XPOST 'localhost:9200/fd_demo/_bulk?refresh=wait_for' \
  -H 'Content-Type: application/x-ndjson' --data-binary @/tmp/fd.ndjson >/dev/null

# Aggregate on the text field -> loads field data onto the heap:
curl -s 'localhost:9200/fd_demo/_search' -H 'Content-Type: application/json' -d'
{ "size":0, "aggs": { "t": { "terms": { "field": "text_hi_card", "size": 5000 } } } }' >/dev/null

# Confirm fielddata is now holding heap:
curl -s 'localhost:9200/_nodes/stats/indices/fielddata?filter_path=nodes.*.indices.fielddata' \
  | python3 -m json.tool
curl -s 'localhost:9200/_cat/fielddata?v'

# And check the breakers -- the fielddata breaker is what guards this:
curl -s 'localhost:9200/_nodes/stats/breaker?filter_path=nodes.*.breakers.fielddata' \
  | python3 -m json.tool

Step 4 — Take a heap dump

PID=$(jps -l | grep -iE 'opensearch|Bootstrap' | grep -iv gradle | awk '{print $1}'); echo $PID

# Preferred: the JVM's own dumper (consistent, includes only live objects on demand):
jcmd "$PID" GC.heap_dump /tmp/node-heap.hprof
# Alternative with an explicit live-GC first:
# jmap -dump:live,format=b,file=/tmp/node-heap.hprof "$PID"

ls -lh /tmp/node-heap.hprof

A fast first look without MAT — the class histogram ranks classes by retained instances/bytes:

jcmd "$PID" GC.class_histogram | head -25
# or: jmap -histo:live "$PID" | head -25
# Expect large counts of long[]/byte[]/Object[] and Lucene/ordinals classes near the top.

Step 5 — Analyze in Eclipse MAT

Open /tmp/node-heap.hprof in MAT. When prompted, run the Leak Suspects report. Then work the views in this order:

In the dominator tree, expand the largest retained object. For this dump the chain is roughly:

IndicesService / IndexShard
  -> fielddata cache entry
     -> a per-segment ordinals / PackedLongValues structure
        -> long[] / byte[]  (the bulk of the retained heap)

The lesson MAT teaches: retained size, not shallow size, is what matters. A small cache-entry object has tiny shallow size but retains megabytes because it dominates (is the sole path to) the big long[]s. The dominator tree sorts by exactly that, so the real culprit floats to the top even though it's a small object.

flowchart TD
    Roots["GC roots"] --> IS["IndicesService"]
    IS --> Shard["IndexShard (fd_demo[0])"]
    Shard --> FD["fielddata cache entry (small shallow size)"]
    FD --> Ord["ordinals / PackedLongValues"]
    Ord --> Arr["long[] / byte[] (HUGE retained size)"]
    Arr -. "dominator tree ranks FD by RETAINED size" .-> Culprit["culprit surfaces at the top"]

# Tie the MAT class names back to source:
find server -name "*Fielddata*" -o -name "*Ordinals*" | grep -i index | head
grep -rn "class .*FieldData\|fielddata\|breaker.*fielddata" \
  server/src/main/java/org/opensearch/index/fielddata/ 2>/dev/null | head

Step 6 — Read a GC log: leak vs thrash

Take the heap dump's verdict and corroborate with GC behavior. Enable GC logging (OpenSearch's default jvm.options already has a variant; to be explicit):

grep -n "Xlog:gc\|gc.log\|HeapDumpOnOutOfMemory" config/jvm.options
# A representative enable line (in jvm.options):
#   -Xlog:gc*,gc+age=trace:file=logs/gc.log:utctime,pid,tags:filecount=16,filesize=64m

# Read the log -- look at the PATTERN across many events, not one line:
find . -name 'gc.log*' 2>/dev/null
grep -E "Pause Young|Pause Full|Humongous|->" $(find . -name 'gc.log*' | head -1) | tail -40

Decide the diagnosis from the pattern:

For the fielddata demo, the heap stays elevated because the fielddata cache legitimately holds it (it's not garbage) — which is exactly the point: it's retention, not a transient allocation spike. Clearing the cache releases it:

# Release the fielddata and watch the breaker estimate drop:
curl -s -XPOST 'localhost:9200/fd_demo/_cache/clear?fielddata=true' >/dev/null
curl -s 'localhost:9200/_cat/fielddata?v'

Step 7 — The right fix (and clean up)

The real fix for the fielddata footgun is don't aggregate on text — use a keyword (doc-values, off-heap) sub-field instead, which never touches the fielddata breaker. That's the contributor takeaway: a heap dump that points at fielddata is usually a mapping bug, not a leak in OpenSearch.

curl -s -XDELETE 'localhost:9200/fd_demo' >/dev/null

Deliverables

• /tmp/deadlock.txt with the Found one Java-level deadlock block, and a one-line statement of the cycle and the fix (consistent lock ordering).
• /tmp/node-heap.hprof and a screenshot/notes of MAT's dominator tree with the retained-memory culprit named (the fielddata/ordinals structure) and its approximate retained size.
• The _cat/fielddata / breaker output before and after _cache/clear, showing the heap was retention, not garbage.
• A GC-log excerpt and your leak-vs-thrash verdict for it.

Expected Output

# jstack (Step 2)
Found one Java-level deadlock:
"worker-1": waiting to lock <0x...a>, which is held by "worker-2"
"worker-2": waiting to lock <0x...b>, which is held by "worker-1"

# fielddata stats (Step 3)
"fielddata": { "memory_size_in_bytes": 41875320, "evictions": 0 }

# class histogram (Step 4)
 num     #instances         #bytes  class name
   1:       ...            18MB     [J            (long[])
   2:       ...             9MB     [B            (byte[])
   ...      org.apache.lucene.util.packed.PackedLongValues...

# MAT dominator tree (Step 5)
IndexShard(fd_demo[0]) -> fielddata entry -> ordinals -> long[]  (retained ~40 MB)

# after cache clear (Step 6)
fielddata memory_size_in_bytes -> 0

Troubleshooting

Stretch Goals

• Reproduce a pool wedge. Make many threads block on one OpenSearch monitor and show all [search] threads BLOCKED in a real node dump (carefully, on a throw- away node) — the production shape of Step 2.
• OnOutOfMemoryError heap dump. Set -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp and force an OOM with a giant agg; confirm the .hprof is written automatically on the crash.
• Compare two heap dumps. Take a dump before and after the fielddata agg; use MAT's "compare to baseline" to show exactly the delta — the new fielddata bytes.
• Request breaker vs fielddata breaker. Trip the request breaker with a huge terms agg size and show the CircuitBreakingException; contrast which breaker guards which structure (circuit-breakers-memory).
• GC log to flame graph. When the GC log says "thrash," confirm with an async-profiler -e alloc graph (Lab DP2) that the allocation rate is the cause, closing the loop.

Coding Exercises

You read a deadlock and a dominator tree by eye; now write code that creates those conditions deterministically and asserts the system reacts correctly — a leak that a test catches is a leak that never ships. Locate every class with rg/find first.

1. (warm-up) A thread-dump deadlock detector. Write find_deadlock.py that reads /tmp/deadlock.txt, finds the Found one Java-level deadlock block, and prints the cycle as worker-1 -> (holds A, wants B) -> worker-2 -> (holds B, wants A). Make it also count threads in BLOCKED (on object monitor) waiting on the same monitor id and flag "possible pool wedge: N threads blocked on <0x...>". Run it on three dumps taken seconds apart (Step 2's note) and have it report whether the wedge persists.

2. (core) A unit test that asserts your lock ordering is deadlock-free. Turn the Deadlock.java into a JUnit test: spin the two threads with consistent lock ordering (both take A before B), join with a timeout, and assertTrue they both finished — then a second test that uses the inconsistent order and asserts they time out (proving the bug exists). Use ThreadMXBean.findDeadlockedThreads() to detect it programmatically. This is the executable version of the lesson "the fix is consistent lock ordering."

3. (core) A fielddata-on-heap regression assertion. Find the breaker that guards fielddata and how stats expose it:

   rg -n "FIELDDATA|fielddata|class FieldDataBreaker|getBreaker|CircuitBreaker.FIELDDATA" \
     server/src/main/java/org/opensearch/indices/breaker/ \
     server/src/main/java/org/opensearch/index/fielddata/
   

   Write an OpenSearchSingleNodeTestCase integration test that maps a text field with fielddata:true, indexes high-cardinality docs, runs a terms agg on it, and asserts the fielddata breaker estimate (client().admin().cluster().prepareNodesStats().setBreaker(true)) is > 0 — then _cache/clear?fielddata=true and asserts it returns to 0. This encodes the "retention, not garbage" finding from Step 6.

4. (advanced) A leak-detector test using MockBigArrays/LeakTracker. OpenSearch ships leak detection for its array and ref-counted allocations. Find it:

   rg -ln "MockBigArrays|LeakTracker|ReleasableBytesReference|leak" server/src/test test/framework
   rg -n "class MockBigArrays|releaseAll|leak detection|finalize" \
     test/framework/src/main/java/org/opensearch/common/util/MockBigArrays.java
   

   Write an AggregatorTestCase test for an aggregation that allocates BigArrays state, then deliberately forget to release one (comment out a close()/Releasables.close) and confirm MockBigArrays' end-of-test leak check fails. Restore the release and confirm it passes. You've used the framework's leak detector as a real test gate — the unit-test analogue of MAT's dominator tree.

5. (Advanced challenge) An automated heap-histogram differ for a retention test. Build a small standalone Java program that: (a) records a jcmd <pid> GC.class_histogram baseline, (b) triggers the fielddata load over the local node, (c) records a second histogram, and (d) diffs them to print the classes whose retained bytes grew most (expect [J / PackedLongValues / ordinals classes) — the CLI equivalent of MAT's "compare to baseline." Then promote it to an OpenSearchIntegTestCase that asserts the fielddata-related class count grew after the agg and shrank back after _cache/clear. Find the cache-clear and stats plumbing with rg -ln "ClearIndicesCacheRequest|IndicesService|FieldDataCache" server/src/main. This single artifact proves you can detect retention growth without ever opening a GUI.

Issues to Practice On

Hangs and OOMs are the highest-severity issues on the tracker — and unfixable without exactly the dumps this lab teaches. Bring the analysis.

There is no single "memory-leak" label; list labels first (gh label list --repo opensearch-project/OpenSearch) and filter by area + read the title for "OOM"/"leak"/"hang".

Representative patterns. (a) "Node OOMs when aggregating/sorting field X" — often a mapping footgun (text with fielddata:true, or a missing keyword sub-field): reproduce, take a heap dump, walk the dominator tree to the fielddata/ordinals structure, and the fix is usually a doc-values path, not a core leak. Back it with the breaker test from exercise 3. (b) "Thread pool wedged / requests hang" — collect three thread dumps, identify the many-threads-BLOCKED-on-one-monitor shape, locate the lock with rg, and the fix is consistent ordering or removing the shared lock.

Planted-bug drill. Find a Releasable/close() site in an aggregation or cache:

rg -n "Releasables.close|@Override\s+public void close|implements Releasable" \
  server/src/main/java/org/opensearch/search/aggregations/bucket/terms/

Remove or skip one release call and run the aggregation's *Tests (with MockBigArrays active). Watch the leak assertion go red, then revert and add a test that closes the aggregator and asserts no bytes remain accounted — the assertion that catches the leak before review.

Etiquette: claim before working, reproduce first; every PR needs a test + a CHANGELOG.md entry + DCO Signed-off-by (git commit -s). See community-interaction.

Validation / Self-check

• You can take a thread dump (jstack -l and jcmd Thread.print) and read the Found one Java-level deadlock block to name the cycle and the fix.
• You can recognize a wedged fixed pool in a dump (many threads BLOCKED on one monitor) and know to take three dumps to confirm it's a real hang.
• You can take a heap dump (jcmd GC.heap_dump) and, in MAT's dominator tree, name the retained-memory culprit and explain retained vs shallow size.
• You can read a GC log and decide leak vs thrash from the pattern, and pick the right next tool for each.
• You can explain why aggregating a text field with fielddata:true lands on heap while a keyword field does not.

Next: Lab DP4 — Reproduce and Bisect a Bug, the "why is it wrong?" workflow on real source.