Reclaim, cgroups, and OOM

Everything so far assumed memory was available. This chapter is about what happens when it is not, which is where the interesting failures live — because the kernel's options are all bad and it has to pick one.

Three concepts in the six-part treatment: reclaim, memory cgroups, and the OOM killer.

Concept 1: Reclaim

1. What problem it solves

An allocation arrives and there are no free pages. The kernel cannot fail it casually — most kernel allocations have no useful failure path, and user allocations failing means programs dying. So it must make free pages by taking them from something else.

Everything resident is a candidate, and every candidate costs something to evict:

Reclaim's job is to choose, and to be right often enough that the machine keeps working.

2. Where it exists in the kernel

rg -n "try_to_free_pages|shrink_node|shrink_lruvec" -A 20 mm/vmscan.c | head -40
rg -n "kswapd\b" mm/vmscan.c | head
rg -n "register_shrinker|struct shrinker \{" -A 20 include/linux/shrinker.h mm/shrinker.c | head -30
grep -E 'CONFIG_LRU_GEN' ~/kernel/build/.config
ls /sys/kernel/mm/lru_gen/ 2>/dev/null

3. Who does it, and when

   WATERMARKS, per zone:
     free > high   ── everything is fine
     free < low    ── wake KSWAPD (background, asynchronous, per node)
     free < min    ── DIRECT RECLAIM: the allocating task reclaims ITSELF,
                      synchronously, before its allocation returns
     nothing works ── the OOM killer

   KSWAPD                          DIRECT RECLAIM
   ──────                          ──────────────
   a kernel thread per node        runs IN the allocating task
   asynchronous                    SYNCHRONOUS -- the task is stalled
   invisible to applications       shows up as latency, and it is the
                                   single most useful thing to measure

Tip: If you learn one thing from this chapter, make it this:

sudo bpftrace -e 'tracepoint:vmscan:mm_vmscan_direct_reclaim_begin { @[comm] = count(); }'
grep -E 'allocstall|pgscan_direct|pgsteal_direct' /proc/vmstat
cat /proc/pressure/memory

Nonzero direct reclaim on a latency-sensitive workload is the answer, not a symptom. It means your task stopped doing its job and did the kernel's instead, at an unpredictable moment.

4. Choosing a victim: LRU and MGLRU

The classic scheme keeps four lists per node (or per cgroup): active/inactive × anon/file.

   NEW page ──▶ INACTIVE list
                    │  referenced again?
                    ├── yes ──▶ ACTIVE list  (promoted)
                    └── no  ──▶ evicted from the tail

   ACTIVE list, under pressure ──▶ demoted back to INACTIVE

   The two-list scheme resists "scan resistance": one sequential read of a
   huge file fills INACTIVE and is evicted from there, WITHOUT displacing
   the working set on ACTIVE.

MGLRU (multi-generational LRU, CONFIG_LRU_GEN, merged 6.1) replaces the two lists with several generations, ages pages by scanning page-table accessed bits rather than by list position, and is generally cheaper and more accurate. It is selectable at runtime:

cat /sys/kernel/mm/lru_gen/enabled 2>/dev/null
grep -E 'nr_active_anon|nr_inactive_anon|nr_active_file|nr_inactive_file' /proc/vmstat

The swappiness knob decides the balance between evicting anonymous pages (to swap) and file pages (to nothing):

cat /proc/sys/vm/swappiness       # 0..200; 60 is a common default

Warning: swappiness=0 does not mean "never swap" and is one of the most counterproductive pieces of received tuning wisdom. It means "strongly prefer evicting page cache", which on a machine with a large working set means throwing away the cache your database needs while keeping anonymous pages nothing has touched in hours. Measure before setting it.

Shrinkers are how subsystems participate: a filesystem's dentry and inode caches, a driver's object cache, anything that can be recomputed.

struct shrinker *s = shrinker_alloc(0, "mylab");
s->count_objects = lab_count;   /* how many could you free? */
s->scan_objects  = lab_scan;    /* free some, return how many */
shrinker_register(s);

rg -n "shrinker_alloc|shrinker_register" -A 10 mm/shrinker.c | head -20
grep -E 'nr_dentry|nr_inodes' /proc/sys/fs/dentry-state /proc/sys/fs/inode-nr 2>/dev/null

5. Experiment

CLAIM. Direct reclaim is visible, measurable, and it is what "the machine got slow" means.

METHOD. In a guest with a small -m:

# Watch the three signals while you apply pressure.
sudo bpftrace -e '
  tracepoint:vmscan:mm_vmscan_direct_reclaim_begin { @direct[comm] = count(); }
  tracepoint:vmscan:mm_vmscan_wakeup_kswapd        { @kswapd = count(); }' &

before=$(grep -E '^(allocstall|pgscan_direct|pgsteal_direct)' /proc/vmstat)

# Fill the page cache, then demand anonymous memory.
find / -type f -exec cat {} + > /dev/null 2>&1 &
sleep 20
python3 -c "b=[bytearray(1<<20) for _ in range(400)]; input()" &
sleep 20

grep -E '^(allocstall|pgscan_direct|pgsteal_direct|pgscan_kswapd)' /proc/vmstat
cat /proc/pressure/memory
kill %1 %2 %3 2>/dev/null

PREDICT FIRST: which happens first — kswapd waking, or direct reclaim? What has to be true for direct reclaim to happen at all, given that kswapd exists?

Then measure the latency cost:

sudo bpftrace -e '
  tracepoint:vmscan:mm_vmscan_direct_reclaim_begin { @s[tid] = nsecs; }
  tracepoint:vmscan:mm_vmscan_direct_reclaim_end /@s[tid]/ {
      @stall_us = hist((nsecs - @s[tid]) / 1000); delete(@s[tid]); }'

PREDICT FIRST: how long does one direct-reclaim stall take — microseconds, or milliseconds?

6. Failure mode

Concept 2: Memory cgroups

1. What problem it solves

Reclaim as described is machine-global: a memory hog forces eviction of everyone else's working set. On a shared machine that is unacceptable — one tenant should not be able to evict another's cache, or trigger an OOM kill of an unrelated process.

Memory cgroups make reclaim, accounting, and OOM per-group rather than per-machine.

2. Where it exists in the kernel

rg -n "struct mem_cgroup \{" -A 40 include/linux/memcontrol.h | head -50
rg -n "try_charge|mem_cgroup_charge\b" mm/memcontrol.c | head
ls /sys/fs/cgroup/
$EDITOR Documentation/admin-guide/cgroup-v2.rst      # the "memory" section

3. The knobs, and what each actually does

The high versus max distinction is the useful one and it is widely misunderstood:

   memory.high = 1G     over it, tasks are THROTTLED and reclaimed against.
                        The workload degrades gracefully. Nothing dies.

   memory.max  = 1G     over it, reclaim runs; if it cannot free enough,
                        something in this cgroup IS KILLED.

   Setting `high` slightly below `max` gives you a warning region: the
   workload slows down and memory.events shows it, before anything dies.

cat /sys/fs/cgroup/<path>/memory.events    # look here FIRST when a container misbehaves
cat /sys/fs/cgroup/<path>/memory.stat | head -20
cat /sys/fs/cgroup/<path>/memory.pressure

4. Experiment

CLAIM. memory.high throttles and memory.max kills, and the difference is observable.

METHOD. In the guest, cgroup v2:

cd /sys/fs/cgroup
echo "+memory" > cgroup.subtree_control
mkdir -p lab/h lab/m

echo 64M > lab/h/memory.high
echo max > lab/h/memory.max

echo max > lab/m/memory.high
echo 64M > lab/m/memory.max

hog() {   # allocate 256 MB in a cgroup and report what happened
  ( echo $BASHPID > "$1/cgroup.procs"
    python3 -c "b=[bytearray(1<<20) for _ in range(256)]; print('SURVIVED')" ) \
    || echo "KILLED (exit $?)"
  echo "--- events:"; cat "$1/memory.events"
}

echo "== memory.high = 64M"; hog lab/h
echo "== memory.max  = 64M"; hog lab/m

PREDICT FIRST: for each group, does the process survive? If it survives, is it slower? What appears in memory.events in each case?

Then watch the throttle:

( echo $BASHPID > lab/h/cgroup.procs
  time python3 -c "b=[bytearray(1<<20) for _ in range(256)]" )
cat lab/h/memory.events lab/h/memory.pressure

5. Failure mode

Concept 3: The OOM Killer

1. What problem it solves

Reclaim has run, everything evictable is gone, and an allocation still cannot be satisfied. The kernel has exactly three options: fail the allocation (most kernel callers cannot handle it), hang forever, or kill something.

It kills something. There is no good answer; there is only a policy.

2. Where it exists in the kernel

rg -n "out_of_memory\b" -A 40 mm/oom_kill.c | head -50
rg -n "oom_badness" -A 30 mm/oom_kill.c
rg -n "oom_score_adj" include/uapi/linux/oom.h include/linux/sched.h | head
$EDITOR Documentation/admin-guide/mm/concepts.rst

3. How the victim is chosen

   oom_badness(task) ≈ (RSS + swap + page tables) , then adjusted by
                       oom_score_adj  (-1000 .. +1000)

     oom_score_adj = -1000  ── EXEMPT. Never chosen.
     oom_score_adj = +1000  ── always chosen first

   Kill the whole thread group, and any process sharing its mm.
   With memory.oom.group=1 in a cgroup, kill EVERY task in that cgroup --
   which is usually what you want for a container, since killing one worker
   out of a pool leaves a broken half-application.

cat /proc/self/oom_score /proc/self/oom_score_adj
for p in $(pgrep -n sshd) $(pgrep -n systemd); do
  printf "%-8s score=%-6s adj=%s\n" "$p" "$(cat /proc/$p/oom_score)" "$(cat /proc/$p/oom_score_adj)"
done

4. Reading an OOM report

The report is long and every section is load-bearing. When one arrives, read it in this order:

   1. WHO ASKED, and for what:
        "<comm> invoked oom-killer: gfp_mask=0x..., order=0, oom_score_adj=0"
        order=0 means a single page. A HIGHER ORDER means fragmentation,
        which is a completely different problem with a different fix.

   2. WHICH constraint:
        "Mem-Info:" for a global OOM
        "memory: usage 65536kB, limit 65536kB" for a MEMCG OOM
        A memcg OOM on a machine with free memory is a LIMIT problem, not a
        memory problem, and people routinely misread this.

   3. WHERE the memory went, from the Mem-Info block:
        active_file / inactive_file  -- page cache
        active_anon / inactive_anon  -- process memory
        slab_reclaimable / slab_unreclaimable -- KERNEL memory
        unevictable, mlocked         -- cannot be reclaimed at all
        A large slab_unreclaimable is a KERNEL leak, not an application bug.

   4. THE TASK TABLE: every task with rss and oom_score_adj. Add up the rss
        column. If it does not account for the memory, it is not a userspace
        problem -- go back to slab.

   5. THE VERDICT:
        "Out of memory: Killed process 1234 (foo) total-vm:..., anon-rss:..."

dmesg | grep -A 60 -i "invoked oom-killer" | head -80

5. Experiment

CLAIM. OOM selection is a scoring function you can steer, and the report tells you where the memory went.

METHOD. In a guest with a small -m — not on your host:

# Make a hog that is NOT the most attractive victim, and one that is.
( echo -1000 > /proc/self/oom_score_adj
  python3 -c "b=[bytearray(1<<20) for _ in range(400)]; input()" ) & PROTECTED=$!
( python3 -c "b=[bytearray(1<<20) for _ in range(400)]; input()" ) & NORMAL=$!

for p in $PROTECTED $NORMAL; do
  printf "%s score=%s adj=%s\n" "$p" "$(cat /proc/$p/oom_score)" "$(cat /proc/$p/oom_score_adj)"
done

# Now force the issue:
python3 -c "b=[bytearray(1<<20) for _ in range(4000)]"
dmesg | grep -A 40 -i "invoked oom-killer" | tail -50

PREDICT FIRST: which of the three dies? Does the oom_score_adj = -1000 process survive even though it is large?

Then the cgroup version, which behaves differently:

mkdir -p /sys/fs/cgroup/lab/oom
echo 128M > /sys/fs/cgroup/lab/oom/memory.max
echo 1    > /sys/fs/cgroup/lab/oom/memory.oom.group
( echo $BASHPID > /sys/fs/cgroup/lab/oom/cgroup.procs
  python3 -c "b=[bytearray(1<<20) for _ in range(64)]; input()" & 
  python3 -c "b=[bytearray(1<<20) for _ in range(256)]" )
cat /sys/fs/cgroup/lab/oom/memory.events

PREDICT FIRST: with memory.oom.group=1, does the small sibling process survive?

6. Failure mode

Validation / Self-check

1. List everything reclaim can evict, and the cost of each.
2. What is the difference between kswapd and direct reclaim, and which one do you measure?
3. Give the three commands that tell you direct reclaim is happening.
4. Why does the active/inactive split make reclaim scan-resistant? Give the concrete scenario.
5. What is MGLRU, roughly when did it arrive, and how does it age pages differently?
6. Why is swappiness=0 usually a mistake? What does it actually mean?
7. What is a shrinker, and name two things a shrinker must never do.
8. Explain memory.high versus memory.max, and give a configuration that produces a warning region.
9. Which single file do you read first when a container is misbehaving, and what is in it?
10. How does oom_badness score a task, and what does oom_score_adj = -1000 do?
11. Give the five things to read from an OOM report, in order. Which two do people skip?
12. An OOM report shows order=3. What does that change about your diagnosis?
13. slab_unreclaimable is 12 GB in an OOM report. Where do you look next?

Next: Lab 11 — Trace a Page Fault — one fault, from the instruction to the page.