The Warm-Up: An Evening With the Kernel You Already Have

Two to three hours, no code, no build. There is a Linux kernel running on your machine right now. It has been answering every question you never asked it. This chapter makes you look at it — with tools already installed — until the abstraction cracks and you can see the machinery underneath.

You do not need your own kernel yet. You need the habit of interrogating one.

Do these in order, and write down your prediction before each exercise. Keep the results in warmup.md in your workspace; you re-read it at the capstone.

Warning: the escape hatch. Exercises 9–11 turn on tracing, which costs real CPU and keeps costing it until you turn it off. Learn the reset now, before you need it:

T=/sys/kernel/tracing               # or /sys/kernel/debug/tracing on older systems
sudo sh -c "echo 0    > $T/tracing_on"
sudo sh -c "echo nop  > $T/current_tracer"
sudo sh -c "echo      > $T/set_ftrace_filter"
sudo sh -c "echo      > $T/trace"

Nothing here can damage your machine, but leaving the function tracer on will make it feel like it did.

Everything below assumes Linux. If you are on macOS or Windows, do this inside the VM you set up in Overview & Prerequisites.

Exercise 1: Which Kernel Am I Actually Running?

uname -a
cat /proc/version                    # the compiler and the build host, too
cat /proc/cmdline                    # what the bootloader told it
cat /proc/sys/kernel/osrelease
ls /lib/modules/"$(uname -r)"/       # the modules that go with THIS kernel

Predict first: is uname -r a version from kernel.org, or something else? Write down what you expect before you look.

Almost certainly you are running a distro kernel: an upstream base plus hundreds of backports and a config with thousands of choices someone else made. The -generic, -arch, .fc41, or .el9 suffix is the tell.

Now find the configuration it was built with:

# One of these will work:
zcat /proc/config.gz 2>/dev/null | head -20      # needs CONFIG_IKCONFIG_PROC
head -20 /boot/config-"$(uname -r)"

# How many decisions are in there?
{ zcat /proc/config.gz 2>/dev/null || cat /boot/config-"$(uname -r)"; } \
  | grep -c '^CONFIG_'
{ zcat /proc/config.gz 2>/dev/null || cat /boot/config-"$(uname -r)"; } \
  | grep -E '^CONFIG_(PREEMPT|HZ|KASAN|PROVE_LOCKING|DEBUG_INFO_BTF|SLUB|MODULES)[= ]'

What you just learned: "the Linux kernel" is not one artifact. It is a source tree plus a configuration, and the configuration alone changes the scheduler's preemption behavior, the timer frequency, and whether half the debugging you are about to do is even possible. A CONFIG_ symbol is as much a part of "which kernel" as the version number.

Exercise 2: The Boundary Is Measurable

strace -c -f ls /usr/bin > /dev/null

Predict first: how many distinct syscalls does ls make? How many total calls?

Now the interesting half — the calls that do not cross the boundary:

cat > /tmp/vdso.c <<'EOF'
#include <time.h>
#include <unistd.h>
int main(void) {
    struct timespec ts;
    for (int i = 0; i < 200000; i++) clock_gettime(CLOCK_MONOTONIC, &ts);
    for (int i = 0; i < 200000; i++) getpid();
    return 0;
}
EOF
gcc -O2 -o /tmp/vdso /tmp/vdso.c
strace -c /tmp/vdso 2>&1 | tail -12
ldd /tmp/vdso | head -3            # note "linux-vdso.so.1" — with no path

Predict first: 200,000 clock_gettime calls and 200,000 getpid calls. How many of each does strace see?

You will see ~200,000 getpid and zero clock_gettime. The kernel maps a small shared library — the vDSO — into every process, containing implementations of a few calls that need kernel data but not kernel privilege. clock_gettime reads a timekeeping page the kernel updates and returns without ever executing a syscall instruction.

The conclusion to carry forward: the user/kernel boundary is a cost, not just a wall. It is expensive enough that the kernel maintains an entire mechanism to avoid crossing it for three or four hot calls. Everything about io_uring, mmap-based interfaces, and batching syscalls follows from that cost.

Exercise 3: /proc Is Not a Filesystem

ls -l /proc/self/status              # size 0 — but it has content
wc -c /proc/self/status              # ...and now it does not
cat /proc/self/status | head -12

Predict first: run ls -l /proc/self/ twice in a row. Will the number after /proc/ be the same? Why or why not?

ls -l /proc/self
ls -l /proc/self
readlink /proc/self                  # it resolves per READER, not globally

Each of those is a different process (ls itself), so /proc/self resolves differently every time. Nothing is stored. These files are function calls wearing a filesystem costume: read() on them runs kernel code that formats a string on demand.

cat /proc/self/maps                  # this process's address space, generated as you read it
cat /proc/self/limits
cat /proc/uptime; sleep 1; cat /proc/uptime

Why this matters to you as a contributor: every one of those files is a uapi surface. Its format is a promise (see the two rules), which is exactly why adding a column to an existing /proc file is a fight and adding a new file is not.

Exercise 4: Where Did the Memory Go?

free -m
grep -E '^(MemTotal|MemFree|MemAvailable|Buffers|Cached|Slab|SReclaimable)' /proc/meminfo

Predict first: MemFree is probably small and MemAvailable much larger. Which one should you worry about, and what is the difference made of?

Now make the page cache visible by using it:

# Pick a file of a few hundred MB you do not care about; make one if needed.
dd if=/dev/urandom of=/tmp/big bs=1M count=500 status=none

sync; sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'    # drops clean page cache
grep -E '^Cached' /proc/meminfo
time cat /tmp/big > /dev/null                            # COLD: real disk I/O
grep -E '^Cached' /proc/meminfo
time cat /tmp/big > /dev/null                            # WARM: page cache only

Predict first: the ratio between the two cat times. Most people guess 5×. Then look.

Note: drop_caches is safe (it only drops clean cache, and sync flushes dirty data first) but it will make your machine briefly slow while everything is re-read. Do not put it in a script.

And the allocator underneath:

sudo slabtop -o | head -20                 # which kernel objects exist, and how many
head -3 /proc/slabinfo                     # the raw source slabtop reads
cat /proc/buddyinfo                        # free pages by order — the page allocator's free lists
cat /proc/vmstat | grep -E 'pgfault|pgmajfault|pgalloc_normal|pgsteal'

/proc/buddyinfo is the page allocator's free lists, by order: column n is the count of free 2ⁿ-page blocks. Watch it while a large program starts and you are watching fragmentation happen.

Exercise 5: Modules Are Just Code

lsmod | head -15                     # Module, Size, Used by
cat /proc/modules | head -3          # same data, raw — including the load address
modinfo "$(lsmod | awk 'NR==2{print $1}')"

Predict first: the "Used by" column shows a refcount and a list of dependents. What do you think prevents you from rmmod-ing a module with a nonzero count?

Now find a module's tunables, live:

ls /sys/module/ | head -20
# Pick one with parameters:
for m in /sys/module/*/parameters; do echo "== $m"; ls "$m"; done 2>/dev/null | head -30
cat /sys/module/*/parameters/* 2>/dev/null | head

What you just learned: a module parameter is not a command-line curiosity. It is a file in sysfs, sometimes writable at runtime, created by a macro (module_param()) that you will use yourself in Lab 2. The module's Size column in lsmod is its text+data — code loaded into the kernel's address space, running at full privilege, with no isolation of any kind.

Exercise 6: The Device Model, Walked

sysfs is a rendering of the kernel's internal object graph. Learning to walk it is learning the device model.

ls /sys/                             # the top-level views
ls /sys/class/                       # devices grouped by what they DO
ls /sys/bus/                         # devices grouped by how they ATTACH
ls /sys/devices/                     # the actual tree: the physical topology

Predict first: /sys/class/net/ and /sys/devices/ both contain your network interface. Which one is the real object, and which is a view?

# Follow one device from its friendly name down to the hardware:
IFACE=$(ls /sys/class/net | grep -v lo | head -1)
ls -l /sys/class/net/"$IFACE"                    # note: it is a SYMLINK
readlink -f /sys/class/net/"$IFACE"              # ...into /sys/devices/. THAT is the object.
cat /sys/class/net/"$IFACE"/{address,mtu,operstate,speed} 2>/dev/null

# Which driver claimed it, and how?
basename "$(readlink -f /sys/class/net/"$IFACE"/device/driver)" 2>/dev/null
ls /sys/bus/pci/drivers/ 2>/dev/null | head
udevadm info -a -p /sys/class/net/"$IFACE" 2>/dev/null | head -30

Everything in /sys/class and /sys/bus is a symlink into /sys/devices. The tree under /sys/devices is the topology; the others are indexes. That structure is kobjects, and every entry you just read is a struct attribute with a show() function behind it — which is precisely what you will write in the sysfs lab.

Exercise 7: The Kernel's Own Symbol Table

wc -l /proc/kallsyms
head -5 /proc/kallsyms
grep -w " T vfs_read" /proc/kallsyms || grep -w vfs_read /proc/kallsyms

Predict first: run the head above as a normal user, then again with sudo. Will the addresses differ?

head -3 /proc/kallsyms
sudo head -3 /proc/kallsyms
cat /proc/sys/kernel/kptr_restrict

As a normal user you almost certainly saw 0000000000000000. kptr_restrict hides kernel addresses from unprivileged readers, because a leaked kernel address defeats KASLR — the kernel randomizes its own load address at boot precisely so an attacker cannot know where anything is.

sudo grep -c ' T \| t ' /proc/kallsyms     # text symbols: every function in the running kernel
sudo grep ' \[.*\]$' /proc/kallsyms | head -5   # symbols belonging to loaded MODULES

What you just learned: /proc/kallsyms is why an oops can print vfs_read+0x2c/0x180 instead of a bare hex address. It is also why nokaslr will appear in your QEMU command line in Lab 1 — GDB needs the addresses to match the symbols in vmlinux.

Exercise 8: The Ring Buffer

sudo dmesg | head -20                # the earliest boot messages
sudo dmesg | tail -20                # the most recent
sudo dmesg -T --level=err,warn | tail -20
cat /proc/sys/kernel/printk          # current, default, minimum, boot-time console log levels

Predict first: open a second terminal, run sudo dmesg -w, then in the first terminal insert and remove a USB device (or run sudo modprobe -r <some safe module>; sudo modprobe <it>). How many lines appear, and who wrote them?

dmesg is a fixed-size ring buffer in kernel memory (CONFIG_LOG_BUF_SHIFT). It wraps. On a busy machine, boot messages are long gone — which is why the serial console in your QEMU rig matters: it captures output the ring buffer will eventually drop, and output from a kernel too broken to answer dmesg.

The log levels are the eight KERN_* constants you will use in your own printk/pr_info calls:

Exercise 9: ftrace — Watch the Kernel Run

This is the exercise that changes how you think. ftrace is built into your kernel already; no installation, no compilation, no agent.

T=/sys/kernel/tracing
[ -d "$T" ] || T=/sys/kernel/debug/tracing
sudo ls "$T" | head -20
sudo cat "$T"/available_tracers

Predict first: roughly how many kernel functions do you think the function tracer can hook?

sudo sh -c "wc -l $T/available_filter_functions"

Now trace something specific. Filter first — turning on the function tracer unfiltered on a busy machine produces gigabytes per second.

sudo sh -c "echo 0 > $T/tracing_on"
sudo sh -c "echo function_graph > $T/current_tracer"
sudo sh -c "echo do_sys_openat2 > $T/set_graph_function"
sudo sh -c "echo 1 > $T/tracing_on"

cat /etc/hostname > /dev/null           # cause one open()

sudo sh -c "echo 0 > $T/tracing_on"
sudo head -60 "$T/trace"

You are looking at the call graph of a real open() in your running kernel, with per-function durations in microseconds. Read it. Find where it decides the path is absolute. Find the permission check.

Now tracepoints — static, stable, zero-cost-when-off instrumentation the kernel ships on purpose:

sudo sh -c "echo nop > $T/current_tracer"
sudo ls "$T"/events/ | head -20
sudo ls "$T"/events/sched/
sudo sh -c "echo 1 > $T/events/sched/sched_switch/enable"
sudo sh -c "echo 1 > $T/tracing_on"
sudo timeout 2 cat "$T/trace_pipe" | head -20
sudo sh -c "echo 0 > $T/events/sched/sched_switch/enable; echo 0 > $T/tracing_on"

Every line is a context switch: which task left the CPU, in what state, and which task took it. That is the scheduler, live, with no debugger and no reboot.

Tip: trace-cmd record -p function_graph -g do_sys_openat2 -- cat /etc/hostname then trace-cmd report does all of the above in one command, and kernelshark draws it. Learn the raw tracefs interface first anyway — when you are debugging a kernel that will not finish booting, trace-cmd is not available and the tracefs files are.

Reset before moving on (the escape hatch at the top of this page).

Exercise 10: perf — Where Is the Time Actually Going?

sudo perf stat -- sha256sum /usr/bin/* 2>/dev/null

Predict first: what fraction of the cycles will be in user space versus kernel space? Then read the context-switches, page-faults, and IPC lines.

sudo perf top                        # live profile of the whole machine; 'q' to quit

Leave perf top running and, in another terminal, run something I/O-heavy (find / -type f > /dev/null) and something CPU-heavy (openssl speed sha256). Watch the top functions change. Kernel symbols are marked [k].

# A profile of one workload, with call graphs:
sudo perf record -g -- tar cf /dev/null /usr/share 2>/dev/null
sudo perf report --stdio | head -40

# Syscalls, as a summary:
sudo perf trace -s -- ls /usr/bin > /dev/null

Note: If perf refuses, check cat /proc/sys/kernel/perf_event_paranoid. Values of 2 or 3 restrict unprivileged use. sudo is the simple answer; lowering the sysctl is the other one, and you should know why it exists before you do.

What you just learned: the kernel exposes hardware performance counters and a sampling infrastructure that works across the user/kernel boundary in one profile. You will use this again in Performance, where the hard part turns out not to be collecting numbers but making an honest comparison.

Exercise 11: bpftrace — Ask the Kernel a Question

ftrace shows you what happened. bpftrace lets you ask a question in one line and get an answer aggregated in the kernel.

sudo bpftrace -l 'tracepoint:syscalls:sys_enter_open*'
sudo bpftrace -l 'kprobe:vfs_*' | head

Predict first: which process on your machine opens the most files per second? Write down a name.

# Who is calling open(), and how often? Ctrl-C after ~10 seconds.
sudo bpftrace -e 'tracepoint:syscalls:sys_enter_openat { @[comm] = count(); }'

# What are the actual paths?
sudo bpftrace -e 'tracepoint:syscalls:sys_enter_openat { printf("%-16s %s\n", comm, str(args.filename)); }' \
  | head -20

# How long do block I/Os take? A log2 histogram, computed in-kernel.
sudo bpftrace -e '
  tracepoint:block:block_rq_issue    { @start[args.dev, args.sector] = nsecs; }
  tracepoint:block:block_rq_complete /@start[args.dev, args.sector]/ {
      @usecs = hist((nsecs - @start[args.dev, args.sector]) / 1000);
      delete(@start[args.dev, args.sector]);
  }'

Note: bpftrace needs CONFIG_DEBUG_INFO_BTF (check ls -l /sys/kernel/btf/vmlinux). If that file is missing, your kernel was built without BTF and bpftrace will be limited to tracepoints with fixed arguments. Turning BTF on is one line in your own build — the lab rig does it for you.

The point of this exercise is not the tool. It is that a running production kernel can be interrogated, safely, with a verified program loaded into it at runtime and no reboot. Understanding why that is safe — the verifier, the restricted instruction set, the helper allow-list — is a real subsystem, and it is the same infrastructure sched_ext and XDP are built on.

Exercise 12: Interrupts and the Two Halves

cat /proc/interrupts | head -20
cat /proc/softirqs

Predict first: you are about to run ping -c 100 -i 0.01 <your gateway>. Which rows of /proc/interrupts will change, and which rows of /proc/softirqs?

GW=$(ip route | awk '/^default/{print $3; exit}')
cp /proc/interrupts /tmp/irq.before; cp /proc/softirqs /tmp/sirq.before
ping -c 100 -i 0.01 -q "$GW" > /dev/null 2>&1
diff <(cat /tmp/irq.before) /proc/interrupts | head -20
diff <(cat /tmp/sirq.before) /proc/softirqs

What you just observed: the NIC's hardware interrupt count went up and the NET_RX softirq count went up, roughly together. That is the two-halves design in a single measurement:

   packet arrives
       │
       ▼
   HARDWARE IRQ   ── atomic context, interrupts possibly masked, cannot sleep
   (top half)        acknowledge the device, schedule the rest, get out
       │
       ▼
   SOFTIRQ NET_RX ── still atomic (cannot sleep!) but interruptible, runs after
   (bottom half)     the IRQ handler returns, or in ksoftirqd under load
       │
       ▼
   the socket's receive queue → a process wakes up and read()s

The number of columns in /proc/interrupts is your CPU count: interrupt affinity is per-CPU and tunable (/proc/irq/N/smp_affinity). Notice ksoftirqd/N in ps — those threads exist for when softirq load is so high that the kernel must stop processing them inline and fall back to scheduling them like ordinary work.

Exercise 13: The Scheduler, Observed

ps -eo pid,pri,ni,psr,policy,stat,comm --sort=-pri | head -15
chrt -p $$                           # your shell's scheduling policy and priority
nproc; cat /proc/loadavg

Predict first: you will pin a busy loop to one CPU with taskset. Will psr (the CPU it last ran on) stay fixed? What about without taskset?

# A busy task, unpinned:
( while :; do :; done ) & BG=$!
for i in 1 2 3 4 5; do ps -o pid,psr,pcpu,comm -p $BG --no-headers; sleep 1; done

# ...and pinned:
taskset -pc 0 $BG
for i in 1 2 3 4 5; do ps -o pid,psr,pcpu,comm -p $BG --no-headers; sleep 1; done
kill $BG

Now watch the scheduler make a decision:

T=/sys/kernel/tracing; [ -d "$T" ] || T=/sys/kernel/debug/tracing
sudo sh -c "echo 1 > $T/events/sched/sched_wakeup/enable"
sudo sh -c "echo 1 > $T/events/sched/sched_switch/enable"
sudo sh -c "echo 1 > $T/tracing_on"
sleep 0.2
sudo sh -c "echo 0 > $T/tracing_on"
sudo grep -E "sched_wakeup|sched_switch" "$T/trace" | head -20
sudo sh -c "echo 0 > $T/events/sched/sched_wakeup/enable; echo 0 > $T/events/sched/sched_switch/enable"

Each sched_wakeup line names the CPU the scheduler chose for the woken task. That choice — which CPU, and whether to preempt what is running there — is most of kernel/sched/fair.c, and it is a whole subsystem chapter.

If your kernel exposes them, these are worth a look too:

cat /proc/pressure/{cpu,io,memory} 2>/dev/null    # PSI: how much time is lost to contention
sudo cat /proc/sched_debug 2>/dev/null | head -40 # per-runqueue internals, if built in
ls /sys/kernel/debug/sched/ 2>/dev/null

The Debrief

Answer these in writing. If any is uncomfortable, re-run the exercise rather than reading ahead.

1. What are the three things that together determine "which kernel am I running"?
2. What is the vDSO, which experiment revealed it, and what does its existence tell you about the cost of the boundary?
3. Why does ls -l /proc/self/status report size 0? What is actually happening on read()?
4. What is the difference between MemFree and MemAvailable, and which experiment made the page cache visible?
5. What isolates a loaded module from the rest of the kernel? (This is a trick question. Answer it anyway.)
6. In /sys, which directory holds the real objects and which hold views? How did you prove it?
7. Why did /proc/kallsyms show zeros as a normal user, and what security property is that protecting?
8. Name the ftrace file that lists every hookable function, the one that selects a tracer, and the one that produces a live stream.
9. In the ping experiment, which two counters moved together, and what does that pairing tell you about interrupt handling?
10. Which context may sleep: a hardware IRQ handler, a softirq, a workqueue item, or a syscall handler? (Two of the four.)
11. What is a tracepoint, and how is it different from a kprobe?
12. Name one question you asked the kernel tonight that you could not have answered yesterday.

What You Should Now Believe

Ready?

You are ready for The Kernel Mental Model when:

• All thirteen exercises done, with written predictions recorded first.
• The debrief answered without looking anything up.
• You have left the function tracer running long enough to notice, and reset it.
• You can state, in one sentence each: what the vDSO is for, why /proc/kallsyms hides addresses, and why an interrupt handler defers work.
• You have bpftrace working, or you know exactly which CONFIG_ symbol you are missing.

You still cannot build a kernel. But you can now interrogate one, and that is the prerequisite for everything else — including debugging the kernel you are about to build.

Next: The Kernel Mental Model — Milestone 0, and it contains no code.