Lab 11: Trace a Page Fault

Background

A program dereferences a pointer. The MMU finds no translation, traps to the kernel, and microseconds later — or milliseconds, if a disk was involved — the instruction re-executes and succeeds. The program sees nothing.

This lab makes that visible. You will produce all four kinds of fault deliberately, measure the cost of each, watch the kernel choose between them, and follow one all the way down in a debugger.

Why This Lab Matters

Page faults are the mechanism, not an error path. Everything in mm/ hangs off handle_mm_fault.
Major faults are the difference between a microsecond and a millisecond, and knowing how to count them is how you find the cause of "the machine is slow".
The copy-on-write path is why fork() is affordable and where a surprising number of real bugs live.
Direct reclaim showing up inside a fault is the single most useful memory measurement there is.

Prerequisites

Page Tables and Faults and Allocators and Folios read.
A lab guest with a small -m (512M–1G) so you can create pressure without a huge workload.
perf and bpftrace working in the guest.
GDB attaching to the guest.

Predict First

Six, in writing.

Cost of a minor fault, in nanoseconds. Of a major fault. Ratio?
mmap 64 MB anonymous and read every page. How many pages are allocated?
The same, but write every page. How many now?
After fork(), the child writes to every page of a 64 MB shared mapping. How many faults, and of which kind?
How many faults does a trivial int main(){return 0;} take from execve to exit?
Under memory pressure, what appears inside a fault that is not there otherwise?

The Target

   instruction touches an address
        │
   MMU: no valid translation ──▶ TRAP
        │
   arch/x86/mm/fault.c : do_user_addr_fault()
        │   error code says: read/write, user/kernel, present/absent
        │
   mm/memory.c : handle_mm_fault(vma, addr, flags)
        │
        ├── no VMA ──────────────▶ SIGSEGV
        ├── anonymous, first touch ─▶ do_anonymous_page()   MINOR
        ├── file-backed ───────────▶ do_fault()             MINOR or MAJOR
        ├── swapped out ───────────▶ do_swap_page()         MAJOR
        └── write to a CoW page ───▶ do_wp_page()           MINOR
        │
   install the PTE, return, RE-EXECUTE the instruction

Step-by-Step Tasks

Step 1: The four kinds, counted

// SPDX-License-Identifier: GPL-2.0
/* faultkinds.c -- produce each kind of fault deliberately and count them. */
#define _GNU_SOURCE
#include <fcntl.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/mman.h>
#include <sys/resource.h>
#include <sys/wait.h>
#include <unistd.h>

#define MB (1UL << 20)
#define SZ (64 * MB)

static long last_min, last_maj;

static void mark(const char *tag)
{
	struct rusage r;

	getrusage(RUSAGE_SELF, &r);
	printf("%-34s minor +%-8ld major +%-6ld\n", tag,
	       r.ru_minflt - last_min, r.ru_majflt - last_maj);
	last_min = r.ru_minflt;
	last_maj = r.ru_majflt;
}

int main(void)
{
	volatile long sum = 0;
	char *p;
	int fd;

	mark("baseline");

	/* 1. ANONYMOUS, untouched: no faults at all. */
	p = mmap(NULL, SZ, PROT_READ | PROT_WRITE,
		 MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
	mark("after mmap (untouched)");

	/* 2. ANONYMOUS READ: the shared zero page. */
	for (size_t i = 0; i < SZ; i += 4096)
		sum += p[i];
	mark("anon READ every page");

	/* 3. ANONYMOUS WRITE: real allocation. */
	for (size_t i = 0; i < SZ; i += 4096)
		p[i] = 1;
	mark("anon WRITE every page");

	/* 4. COPY ON WRITE, in the child. */
	if (fork() == 0) {
		last_min = last_maj = 0;
		mark("child: right after fork");
		for (size_t i = 0; i < SZ; i += 4096)
			p[i] = 2;
		mark("child: WRITE every page (CoW)");
		_exit(0);
	}
	wait(NULL);
	munmap(p, SZ);

	/* 5. FILE-BACKED. Cold vs warm is the point; drop caches between runs. */
	fd = open("/tmp/faultfile", O_RDONLY);
	if (fd >= 0) {
		p = mmap(NULL, SZ, PROT_READ, MAP_PRIVATE, fd, 0);
		if (p != MAP_FAILED) {
			mark("after mmap(file)");
			for (size_t i = 0; i < SZ; i += 4096)
				sum += p[i];
			mark("file READ every page");
			munmap(p, SZ);
		}
		close(fd);
	}
	return 0;
}

dd if=/dev/urandom of=/tmp/faultfile bs=1M count=64 status=none
gcc -O0 -o /tmp/faultkinds /tmp/faultkinds.c

sync; sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
/tmp/faultkinds                       # COLD file
/tmp/faultkinds                       # WARM file

PREDICT FIRST: fill in every cell before running (64 MB ÷ 4 KB = 16384 pages).

Stage	minor	major
after `mmap` (untouched)
anon READ every page
anon WRITE every page
child: right after `fork`
child: WRITE every page (CoW)
file READ, cold
file READ, warm

The two rows people get wrong are "anon READ" (the zero page) and "child right after fork" (page tables are copied, but nothing faults until a touch).

Step 2: Confirm the zero page

The anon-READ result claims 16384 faults allocated no memory. Prove it.

/tmp/faultkinds &
PID=$!
sleep 0.2
grep -E 'VmRSS|RssAnon' /proc/$PID/status

Better, read the PTEs directly and see that many virtual pages share one physical frame:

sudo python3 - <<'EOF'
import struct
pid  = int(input("pid: "))
virt = int(input("virt (hex, from /proc/PID/maps): "), 16)
pfns = []
with open(f"/proc/{pid}/pagemap","rb") as f:
    for i in range(64):
        f.seek(((virt >> 12) + i) * 8)
        e, = struct.unpack("<Q", f.read(8))
        pfns.append(e & ((1 << 55) - 1) if e >> 63 else None)
print("distinct physical frames among 64 virtual pages:",
      len({p for p in pfns if p}))
EOF

PREDICT FIRST: after reading (not writing) 64 pages, how many distinct physical frames back them? After writing?

Step 3: Cost, measured

# Per-fault latency, by kind:
sudo bpftrace -e '
kprobe:handle_mm_fault  /comm == "faultkinds"/ { @s[tid] = nsecs; }
kretprobe:handle_mm_fault /@s[tid]/ {
	@ns = hist(nsecs - @s[tid]);
	delete(@s[tid]);
}' &
/tmp/faultkinds; kill %1

PREDICT FIRST: where does the histogram peak, and is it bimodal? If so, what are the two modes?

# The aggregate view:
sudo perf stat -e page-faults,minor-faults,major-faults,dTLB-load-misses -- /tmp/faultkinds

# Which code paths, with stacks:
sudo perf record -e page-faults -g -- /tmp/faultkinds
sudo perf report --stdio | head -30

Step 4: Major faults, and where the time goes

sync; sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
sudo bpftrace -e '
kprobe:do_swap_page  { @swap  = count(); }
kprobe:filemap_fault { @file  = count(); }
kprobe:do_anonymous_page { @anon = count(); }
kprobe:do_wp_page    { @cow   = count(); }' &
/tmp/faultkinds; kill %1

PREDICT FIRST: the four counts, for one cold run.

Then make swap faults happen, which needs pressure:

# In the guest, with swap configured and a small -m:
swapon --show
python3 -c "b=[bytearray(1<<20) for _ in range(600)]; input()" &
sleep 10
grep -E 'pswpin|pswpout' /proc/vmstat
/usr/bin/time -v /tmp/faultkinds 2>&1 | grep -E 'Major|Minor'

Step 5: Watch reclaim happen inside a fault

This is the measurement that matters most in production.

sudo bpftrace -e '
kprobe:handle_mm_fault { @in_fault[tid] = 1; }
kretprobe:handle_mm_fault { delete(@in_fault[tid]); }
tracepoint:vmscan:mm_vmscan_direct_reclaim_begin /@in_fault[tid]/ {
	@reclaim_in_fault[comm] = count();
	@rs[tid] = nsecs;
}
tracepoint:vmscan:mm_vmscan_direct_reclaim_end /@rs[tid]/ {
	@stall_us = hist((nsecs - @rs[tid]) / 1000);
	delete(@rs[tid]);
}' &

# Apply pressure while faulting.
find / -type f -exec cat {} + >/dev/null 2>&1 &
python3 -c "b=[bytearray(1<<20) for _ in range(400)]; input()" &
sleep 20; kill %1 %2 %3 2>/dev/null

PREDICT FIRST: how much does one direct-reclaim stall add to a fault — microseconds, or milliseconds? And what does that do to the fault-latency histogram's shape?

Step 6: Follow one fault in GDB

~/kernel-labs/scripts/gdb-attach.sh handle_mm_fault

(gdb) continue
        ... in the guest, cause exactly one fault ...
(gdb) bt
(gdb) p vma->vm_start
(gdb) p vma->vm_end
(gdb) p/x vma->vm_flags
(gdb) p vma->vm_file
(gdb) p address
(gdb) p $lx_current().comm
(gdb) finish

PREDICT FIRST: what will bt show above handle_mm_fault? Name the frames between it and the user instruction.

Then walk the page tables by hand for that address, using the accessors from the concepts chapter:

(gdb) p/x address
(gdb) p/x (address >> 39) & 0x1ff        ← PGD index
(gdb) p/x (address >> 30) & 0x1ff        ← PUD index
(gdb) p/x (address >> 21) & 0x1ff        ← PMD index
(gdb) p/x (address >> 12) & 0x1ff        ← PTE index

Warning: A breakpoint on handle_mm_fault fires constantly — it is one of the hottest functions in the kernel. Set a condition, or the guest will be unusable:
(gdb) break handle_mm_fault if $lx_current().pid == <your pid>

Implementation Requirements / Deliverables

faultkinds.c built and run, cold and warm.
The seven-row prediction table, filled in before running, with results and a one-sentence note on each miss.
The zero-page result confirmed by counting distinct PFNs, not just by RSS.
A fault-latency histogram, with the modes identified and explained.
Counts for all four fault-handling functions from one cold run.
Major faults produced deliberately, from both a file and swap.
Direct reclaim observed inside a fault, with the added latency measured.
A GDB session at handle_mm_fault with the VMA inspected and the backtrace explained.
One paragraph: given a machine where "some requests are occasionally slow", what would you measure here first, and what would each result rule in or out?

Expected Output

$ sync; sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'; /tmp/faultkinds
baseline                           minor +182      major +0
after mmap (untouched)             minor +0        major +0
anon READ every page               minor +16384    major +0
anon WRITE every page              minor +16384    major +0
child: right after fork            minor +8        major +0
child: WRITE every page (CoW)      minor +16384    major +0
after mmap(file)                   minor +0        major +0
file READ every page               minor +15872    major +512

$ /tmp/faultkinds        # second run, file now warm
file READ every page               minor +16384    major +0

Two things to notice: the anon READ and anon WRITE lines have the same fault count and completely different memory consequences; and the cold file read shows far fewer major faults than pages, because readahead brought most of them in as a side effect of the first few.

@ns:                                 # bimodal, and the modes are the story
[512, 1K)          28104 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[1K, 2K)           19822 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@              |
[2K, 4K)            1120 |@@                                               |
...
[256K, 512K)         498 |@                                                |   ← major
[512K, 1M)            14 |                                                 |

Debugging Steps

`getrusage` counts look wrong

ru_minflt is cumulative for the process. The mark() helper subtracts the previous value — check you did not reset last_min in the parent when you meant the child.

No major faults, ever

The file is in the page cache. sync; echo 3 > /proc/sys/vm/drop_caches before each cold run, and make the file larger than free memory if you want them sustained.

`bpftrace` cannot attach to `handle_mm_fault`

It may be inlined or renamed in your build. Check, then use the tracepoint or the arch entry instead:

sudo grep -w handle_mm_fault /proc/kallsyms
sudo bpftrace -l 'kprobe:*mm_fault*'
sudo bpftrace -l 'tracepoint:exceptions:*'      # x86 page_fault_user/kernel

The guest is unusable with a breakpoint set

Expected. Use a condition on the pid, per the warning in step 6.

Fault latencies are absurd

No KVM: every guest instruction is emulated. ls -l /dev/kvm on the host. Relative shapes still teach you something; absolute numbers do not.

No swap, so no swap faults

fallocate -l 512M /swapfile && chmod 600 /swapfile && mkswap /swapfile && swapon /swapfile in the guest. Remember this is a throwaway VM.

CoW faults show up as zero

You measured the parent, or the child inherited counters. getrusage(RUSAGE_SELF) in the child reports the child's own — reset your baseline right after fork().

Experiment

CLAIM. Readahead makes sequential file access dramatically cheaper, and you can turn it off and measure exactly what it was worth.

METHOD.

DEV=$(df /tmp --output=source | tail -1 | sed 's|/dev/||; s|[0-9]*$||')
cat /sys/block/$DEV/queue/read_ahead_kb

for ra in 128 0; do
  echo $ra | sudo tee /sys/block/$DEV/queue/read_ahead_kb >/dev/null
  sync; sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
  echo "== read_ahead_kb=$ra"
  /usr/bin/time -v /tmp/faultkinds 2>&1 | grep -E 'Major|Minor|Elapsed'
done
echo 128 | sudo tee /sys/block/$DEV/queue/read_ahead_kb >/dev/null

PREDICTION. Before running: with readahead disabled, how many major faults for a 64 MB sequential read? With it at 128 KB? And what is the ratio of wall-clock times?

RESULT. Then use madvise to change the answer without touching a global knob:

madvise(p, SZ, MADV_SEQUENTIAL);   /* aggressive readahead      */
madvise(p, SZ, MADV_RANDOM);       /* none: one fault per page  */
madvise(p, SZ, MADV_WILLNEED);     /* fetch it all, now         */

Measure all three. Then answer the question the experiment exists for: a database does random reads over a large file. Which madvise is right, and what does the default cost it?

Test

cat > ~/kernel-labs/scripts/faultprofile.sh <<'EOF'
#!/usr/bin/env bash
# Fault profile for a command: counts by kind, latency distribution, and
# whether any fault had to reclaim synchronously.
set -euo pipefail
[ $# -ge 1 ] || { echo "usage: faultprofile.sh <command> [args...]"; exit 2; }

sudo bpftrace -e '
kprobe:do_anonymous_page { @anon = count(); }
kprobe:do_wp_page        { @cow  = count(); }
kprobe:filemap_fault     { @file = count(); }
kprobe:do_swap_page      { @swap = count(); }
kprobe:handle_mm_fault   { @s[tid] = nsecs; @in[tid] = 1; }
kretprobe:handle_mm_fault /@s[tid]/ {
	@fault_ns = hist(nsecs - @s[tid]); delete(@s[tid]); delete(@in[tid]);
}
tracepoint:vmscan:mm_vmscan_direct_reclaim_begin /@in[tid]/ {
	@RECLAIM_INSIDE_FAULT = count();
}
END { clear(@s); clear(@in); }' -c "$*"
EOF
chmod +x ~/kernel-labs/scripts/faultprofile.sh
~/kernel-labs/scripts/faultprofile.sh /tmp/faultkinds

Verify it can distinguish: run it on /tmp/faultkinds cold and warm; the @file and @fault_ns tail must differ. Then run it under memory pressure and confirm @RECLAIM_INSIDE_FAULT becomes nonzero — if it never does, you have not created real pressure.

Challenge Extensions

Find the zero page. Get its PFN from pagemap for a read-only anonymous mapping, and confirm two different processes reading untouched anonymous memory share the same physical frame. Then explain what happens to that sharing on the first write.
Measure huge pages. Enable transparent_hugepage=always, rerun, and compare fault counts and dTLB-load-misses. 64 MB should now be ~32 faults instead of 16384. Then measure the cost: how long does one huge-page fault take versus one small one?
Break CoW deliberately. Write a program where parent and child both write to a shared MAP_SHARED mapping and compare with MAP_PRIVATE. Explain the fault counts and the visibility difference.
Add a tracepoint. The fault path has fewer tracepoints than it deserves. Add one to do_anonymous_page that records whether the zero page was used, build it, and consume it with bpftrace. Then consider whether it is worth proposing upstream.
userfaultfd. Handle page faults in user space with userfaultfd(2). This is how live migration and some garbage collectors work, and implementing a toy version makes the whole fault path concrete in a way nothing else does.
Fault around. Find fault_around_bytes in mm/memory.c and in debugfs. Vary it and measure the effect on a file-mapped workload. Then explain the tradeoff it embodies — this is a real tunable with a real cost in both directions.

Validation / Self-check

Name the four fault-handling functions and the case each serves.
Reading 64 MB of untouched anonymous memory produces 16384 faults and allocates how much? Explain.
What exactly does fork() copy, and why is the child's fault count right after it so small?
Distinguish minor and major faults, and give two ways to count each from user space.
Why does a cold 64 MB file read produce far fewer major faults than pages?
Why is the fault-latency histogram bimodal? What are the modes?
What does direct reclaim inside a fault mean, and why is it the most useful memory measurement?
Why must a handle_mm_fault breakpoint be conditional?
Given the address 0x7f3c2a4b1000, compute its four page-table indices.
A database does random reads over a large mapped file. Which madvise is right, and what does the default cost?
Transparent huge pages: what improves, what gets worse, and what does the fault count become?
"Some requests are occasionally slow." Give the three commands you run first, and what each rules in or out.

Next: Storage — where those major faults actually went.

Open-Source Engineer & Contributor