The Lab Rig

This is the machine you experiment on. Build it once, carefully, and every lab in this curriculum becomes a five-minute loop. Skip it, and every lab becomes an evening of environment debugging.

The rig has one job: change one line of kernel source, and be looking at the result in a debugger in under five minutes. Everything below serves that number.

The Shape of It

  ┌──────────────────────────────────────────────────────────────────────────┐
  │  HOST (your Linux machine or Linux VM)                                   │
  │                                                                          │
  │   ~/kernel/linux/          the source tree — git, never dirty by accident│
  │   ~/kernel/build/          O= output: .config, vmlinux, bzImage, *.ko    │
  │   ~/kernel-labs/           the companion workspace: modules and scripts  │
  │   ~/.ccache/               20 GB of object cache — this is the 5 minutes │
  │                                                                          │
  │   ┌──────────────────┐        ┌──────────────────────────────────────┐   │
  │   │ gdb vmlinux      │───────▶│ QEMU  -s  (gdbstub on tcp:1234)      │   │
  │   │ lx-dmesg, lx-ps  │  :1234 │                                      │   │
  │   │ lx-symbols       │        │   ┌──────────────────────────────┐   │   │
  │   └──────────────────┘        │   │  GUEST: your kernel          │   │   │
  │                               │   │  bzImage / Image             │   │   │
  │   ┌──────────────────┐        │   │  + initramfs.cpio.gz         │   │   │
  │   │ serial console   │◀───────│   │  + your .ko modules          │   │   │
  │   │ (-nographic)     │  stdio │   │  PID 1 = /init = busybox sh  │   │   │
  │   └──────────────────┘        │   └──────────────────────────────┘   │   │
  │                               └──────────────────────────────────────┘   │
  └──────────────────────────────────────────────────────────────────────────┘

Three properties are non-negotiable, and each rules out a tempting shortcut:

1. The Tree and the Build Directory

Keep the source tree pristine and the build output somewhere else. O= costs you one flag on every make and buys you a git status that means something.

mkdir -p ~/kernel && cd ~/kernel
# (clone as described in Overview & Prerequisites)
cd linux

mkdir -p ../build
make O=../build defconfig            # a sane baseline for your architecture
make O=../build -j"$(nproc)"         # the first build: 3–40 minutes depending on the box

Set up ccache before the first build, not after. A cold build populates it; every build after that is mostly cache hits.

ccache -M 20G
ccache -z                            # zero the stats so you can see the win
export KBUILD_BUILD_TIMESTAMP=''     # keeps the build reproducible-ish, helps cache hits
make O=../build -j"$(nproc)" CC="ccache gcc"
ccache -s | grep -E 'cache hit|cache miss'

Tip: Put CC="ccache gcc" (or CC="ccache clang" with LLVM=1) in a shell alias or the workspace's build-kernel.sh and stop thinking about it. Forgetting it on one build silently costs you twenty minutes.

The make targets you will actually use

Warning: make menuconfig is fine for exploring and terrible for reproducibility. Every config decision this curriculum depends on is expressed as a fragment below, applied by a script, so that "which kernel am I running" has a written answer. Change configs in the fragment, not in the UI.

2. The Config Fragment

Kconfig is where most of your rig's behavior is decided. Two fragments: one for the loop you run fifty times a day, one for the paranoid runs you do before you believe anything.

lab-fast.config — the everyday kernel

# ── Debug info and GDB ────────────────────────────────────────────────────
CONFIG_DEBUG_KERNEL=y
CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT=y
CONFIG_DEBUG_INFO_REDUCED=n
CONFIG_GDB_SCRIPTS=y
CONFIG_KALLSYMS_ALL=y
CONFIG_DEBUG_INFO_BTF=y

# ── Modules: you will load, unload, and reload constantly ─────────────────
CONFIG_MODULES=y
CONFIG_MODULE_UNLOAD=y
CONFIG_MODULE_FORCE_UNLOAD=y

# ── Observability ─────────────────────────────────────────────────────────
CONFIG_DEBUG_FS=y
CONFIG_FTRACE=y
CONFIG_FUNCTION_TRACER=y
CONFIG_FUNCTION_GRAPH_TRACER=y
CONFIG_DYNAMIC_FTRACE=y
CONFIG_KPROBES=y
CONFIG_KPROBE_EVENTS=y
CONFIG_UPROBE_EVENTS=y
CONFIG_BPF_SYSCALL=y
CONFIG_BPF_EVENTS=y
CONFIG_MAGIC_SYSRQ=y

# ── Correctness checks that are cheap enough to leave on ──────────────────
CONFIG_DEBUG_ATOMIC_SLEEP=y
CONFIG_PROVE_LOCKING=y
CONFIG_DEBUG_LIST=y
CONFIG_VMAP_STACK=y

# ── Testing ───────────────────────────────────────────────────────────────
CONFIG_KUNIT=y
CONFIG_KUNIT_DEBUGFS=y

# ── The guest hardware QEMU gives you ─────────────────────────────────────
CONFIG_BLK_DEV_INITRD=y
CONFIG_DEVTMPFS=y
CONFIG_DEVTMPFS_MOUNT=y
CONFIG_VIRTIO=y
CONFIG_VIRTIO_PCI=y
CONFIG_VIRTIO_BLK=y
CONFIG_VIRTIO_NET=y
CONFIG_VIRTIO_CONSOLE=y
# Sharing a host directory into the guest (see §5):
CONFIG_NET_9P=y
CONFIG_NET_9P_VIRTIO=y
CONFIG_9P_FS=y

Serial console, which differs by architecture — enable the one you need:

# x86-64 (QEMU's default 8250 UART → console=ttyS0)
CONFIG_SERIAL_8250=y
CONFIG_SERIAL_8250_CONSOLE=y

# arm64 -M virt (PL011 → console=ttyAMA0)
CONFIG_SERIAL_AMBA_PL011=y
CONFIG_SERIAL_AMBA_PL011_CONSOLE=y

lab-paranoid.config — everything above, plus

CONFIG_KASAN=y
CONFIG_KASAN_INLINE=y
CONFIG_UBSAN=y
CONFIG_UBSAN_BOUNDS=y
CONFIG_DEBUG_OBJECTS=y
CONFIG_DEBUG_OBJECTS_FREE=y
CONFIG_DEBUG_KMEMLEAK=y
CONFIG_DEBUG_SPINLOCK=y
CONFIG_DEBUG_MUTEXES=y
CONFIG_DEBUG_RT_MUTEXES=y
CONFIG_DEBUG_WW_MUTEX_SLOWPATH=y
CONFIG_DEBUG_PLIST=y
CONFIG_DEBUG_SG=y
CONFIG_DEBUG_PAGEALLOC=y
CONFIG_SLUB_DEBUG_ON=y
CONFIG_FAULT_INJECTION=y
CONFIG_FAILSLAB=y

Applying a fragment

Do not hand-edit .config. Merge, then reconcile — and then verify, because Kconfig silently drops symbols whose dependencies are unmet.

cd ~/kernel/linux
make O=../build defconfig
./scripts/kconfig/merge_config.sh -O ../build ../build/.config ~/kernel-labs/lab-fast.config
make O=../build olddefconfig

# VERIFY. Every line of this must print the value you asked for.
grep -E '^CONFIG_(GDB_SCRIPTS|DEBUG_INFO_BTF|PROVE_LOCKING|DEBUG_ATOMIC_SLEEP|KUNIT|FUNCTION_GRAPH_TRACER)=' ../build/.config

Warning: A symbol you set that is missing from the output was not silently ignored by accident — its dependencies were not met, and Kconfig dropped it. Find out why:

make O=../build menuconfig      # press / and search the symbol; it shows the dependency chain

CONFIG_DEBUG_INFO_BTF failing because pahole is not installed is the most common instance, and it silently disables bpftrace in the guest.

The cost of the paranoid kernel

Measure it once so you can make an informed choice later.

Use lab-fast for the loop. Run lab-paranoid before you believe a result, before you send a patch, and any time something is behaving strangely — most memory-corruption bugs announce themselves immediately under KASAN and are undebuggable without it.

3. The Root Filesystem

Your kernel needs something to run as PID 1. The smallest useful answer is a busybox initramfs: a compressed cpio archive the bootloader (here, QEMU) hands to the kernel, which the kernel unpacks into a tmpfs and executes /init from.

   QEMU -kernel bzImage -initrd initramfs.cpio.gz
        │
        ▼
   kernel boots, unpacks the cpio into rootfs (a tmpfs)
        │
        ▼
   kernel exec()s /init  ──────▶ PID 1
        │
   your shell script mounts /proc /sys /dev, then exec /bin/sh

Building it

mkdir -p ~/kernel/rootfs && cd ~/kernel/rootfs

# 1. A statically linked busybox: one binary, every tool, no libc to ship.
curl -LO https://busybox.net/downloads/busybox-1.36.1.tar.bz2
tar xf busybox-1.36.1.tar.bz2 && cd busybox-1.36.1
make defconfig
sed -i 's/^# CONFIG_STATIC is not set/CONFIG_STATIC=y/' .config
make oldconfig < /dev/null
make -j"$(nproc)"
make CONFIG_PREFIX=../initramfs install
cd ..

# 2. The directories the kernel and your init script need.
mkdir -p initramfs/{proc,sys,dev,tmp,etc,lib/modules,mnt}

# 3. PID 1.
cat > initramfs/init <<'EOF'
#!/bin/sh
mount -t proc     none /proc
mount -t sysfs    none /sys
mount -t devtmpfs none /dev      2>/dev/null
mount -t tmpfs    none /tmp
mount -t tracefs  none /sys/kernel/tracing 2>/dev/null
mount -t debugfs  none /sys/kernel/debug   2>/dev/null

# Share a host directory (see §5) if the tag is present.
mkdir -p /mnt/host
mount -t 9p -o trans=virtio,version=9p2000.L host /mnt/host 2>/dev/null

echo
echo "=== lab guest up: $(uname -r) ==="
echo
exec /bin/sh
EOF
chmod +x initramfs/init

# 4. Pack it.
( cd initramfs && find . -print0 | cpio --null -o -H newc 2>/dev/null ) | gzip -9 > initramfs.cpio.gz
ls -lh initramfs.cpio.gz          # a couple of megabytes

Predict first, before your first boot: how large is that archive, and how long does the guest take to reach the shell prompt? Write both down.

Note: An initramfs is not the same thing as a disk image. It lives entirely in RAM, it is rebuilt from your directory every time, and nothing you do inside the guest survives a reboot. That is a feature: every boot is identical, and you cannot accidentally accumulate state that explains a result. When a lab needs persistence, it says so and gives you a qemu-img disk.

Getting your modules into the guest

Two ways. Use the first while learning, the second once the loop annoys you.

# (a) Bake them in — simple, requires repacking the cpio on every change.
cp ~/kernel-labs/modules/02-chardev/*.ko ~/kernel/rootfs/initramfs/lib/modules/
( cd ~/kernel/rootfs/initramfs && find . -print0 | cpio --null -o -H newc 2>/dev/null ) \
  | gzip -9 > ~/kernel/rootfs/initramfs.cpio.gz

# (b) Share a host directory over 9p — no repacking; rebuild and re-insmod.
#     Add to the QEMU line:
#       -virtfs local,path=$HOME/kernel-labs,mount_tag=host,security_model=none,id=host
#     The init script above mounts it at /mnt/host.

The shortcut, once you have earned it

virtme-ng boots your freshly built kernel using your host's root filesystem, with no initramfs to build:

vng --build            # build the kernel in the current tree
vng                    # boot it, with your home directory present

It is excellent and most kernel developers use something like it. Build the manual rig first anyway. When vng does not work — a config it does not expect, an early-boot panic, a machine without it installed — you need to know what it was doing for you.

4. Booting It

x86-64

cd ~/kernel
qemu-system-x86_64 \
  -kernel build/arch/x86/boot/bzImage \
  -initrd rootfs/initramfs.cpio.gz \
  -append "console=ttyS0 earlyprintk=serial,ttyS0 nokaslr panic=1 oops=panic" \
  -nographic -no-reboot \
  -m 2G -smp 4 \
  -enable-kvm -cpu host \
  -virtfs local,path="$HOME/kernel-labs",mount_tag=host,security_model=none,id=host \
  -s

arm64

qemu-system-aarch64 \
  -M virt -cpu max -m 2G -smp 4 \
  -kernel build/arch/arm64/boot/Image \
  -initrd rootfs/initramfs.cpio.gz \
  -append "console=ttyAMA0 nokaslr panic=1 oops=panic" \
  -nographic -no-reboot \
  -virtfs local,path="$HOME/kernel-labs",mount_tag=host,security_model=none,id=host \
  -s

On Apple Silicon, replace -cpu max with -accel hvf -cpu host for native-speed arm64.

Every flag, and why

Tip: Ctrl-A X quits. Ctrl-A C switches to the QEMU monitor, where info registers, info mtree, and system_reset work even when the guest is wedged. Learn both now; you will need them the first time your kernel spins with interrupts disabled.

5. Attaching GDB

cd ~/kernel/build
gdb vmlinux

(gdb) target remote :1234
(gdb) lx-version
(gdb) lx-dmesg
(gdb) break do_sys_openat2
(gdb) continue
        ... in the guest:  cat /etc/hostname
(gdb) bt
(gdb) p *filename
(gdb) lx-ps

vmlinux, not bzImage. bzImage is compressed and stripped; it has no symbols. vmlinux is the full ELF image, and it is in the build directory only if CONFIG_DEBUG_INFO is on.

Making lx- commands work

Building with CONFIG_GDB_SCRIPTS=y produces vmlinux-gdb.py next to vmlinux. GDB auto-loads it only if the path is trusted:

cat >> ~/.gdbinit <<EOF
add-auto-load-safe-path $HOME/kernel/build
set history save on
set print pretty on
EOF

Then apropos lx inside GDB lists everything you got. The ones you will use:

Debugging your own module

A module's code is at an address chosen at load time, so GDB needs to be told:

        ... in the guest:  insmod /mnt/host/modules/02-chardev/chardev.ko
(gdb) lx-symbols /home/you/kernel-labs/modules/02-chardev
        ^ scans the directory for .ko files matching loaded modules and adds their symbols
(gdb) break chardev_read
(gdb) continue

Debugging early boot

Add -S to the QEMU line. The guest freezes before executing anything; GDB attaches, you set a breakpoint on start_kernel, then continue.

What will disappoint you

Note: That last row is the most useful thing GDB gives you and its biggest trap. Stopping the guest stops everything — which is what lets you inspect a race — and also means timeouts, watchdogs, and heartbeats all expire the moment you resume. A breakpoint in a network path will produce timeouts that are artifacts of debugging, not bugs. For anything timing-sensitive, reach for ftrace or a printk instead.

6. The Loop

This is the number that decides whether you finish the curriculum. Measure it today.

cd ~/kernel/linux
touch kernel/sched/core.c                       # simulate a one-file change
time make O=../build -j"$(nproc)" CC="ccache gcc"

If your loop is much worse than that, fix it before Lab 1. The usual culprits:

7. The Companion Workspace

book/projects/kernel-labs/ in this repository is the rig, scripted. Copy it out (Overview & Prerequisites) and use it instead of retyping the above.

kernel-labs/
├── README.md              what is complete vs. skeleton; the kernel version it is pinned to
├── Makefile               kbuild for all out-of-tree modules at once
├── scripts/
│   ├── build-kernel.sh    defconfig + fragment + olddefconfig + build, with ccache
│   ├── mkrootfs.sh        the busybox initramfs from §3
│   ├── run-qemu.sh        the QEMU line from §4, arch-detected, -s -S optional
│   ├── gdb-attach.sh      gdb vmlinux + target remote + lx-symbols, preloaded
│   └── checkpatch-all.sh  checkpatch over your commits before you send anything
├── modules/
│   ├── 01-hello/          COMPLETE — the reference to read
│   ├── 02-chardev/        skeleton + a userspace exerciser
│   ├── 03-sysfs-device/   skeleton
│   ├── 04-ramdisk-block/  skeleton
│   ├── 05-netdev/         skeleton
│   ├── 06-toy-lsm/        skeleton
│   └── 07-fake-accel/     skeleton
├── kunit/                 KUnit suites the modules must pass
└── userspace/             ioctl exercisers, XDP loaders, test programs

Every skeleton is /* TODO(Lab N): … */ bodies plus a check that fails until the body is right. The failing check is the specification.

Warning: Out-of-tree modules are not portable across kernel versions — see why there is no stable internal API. The README pins a version. If yours is newer and a module does not compile, do not work around it: find the commit that changed the API and read it.

cd ~/kernel/linux
git log -S'the_function_that_vanished' --oneline -- include/ | head
git show <that commit>          # the commit message explains the new way

That is the actual skill. You will use it every time you rebase a patch.

8. Hygiene That Saves You Later

# Keep the source tree clean. A stray edit that you forget about will waste a day.
cd ~/kernel/linux && git status --short          # must be empty before every experiment

# Work on a branch, always, even for throwaway experiments.
git switch -c lab-03-chardev

# Know exactly what you are running, in the guest:
#   uname -a matches your build, and the config is queryable
grep CONFIG_LOCALVERSION ../build/.config

Set a local version string so uname -r inside the guest tells you which build you booted — you will otherwise waste an hour debugging a kernel you did not rebuild:

./scripts/config --file ../build/.config --set-str LOCALVERSION "-lab"
make O=../build olddefconfig

Finally, a .gitignore habit: the build directory is outside the tree (O=../build), so the tree stays clean by construction. That is the main reason to use O= at all.

Validation / Self-check

You have a working rig when all of these are true. Each maps to a step in Lab 1.

• make O=../build -j$(nproc) completes, and a one-file change rebuilds in under a minute.
• uname -a in the guest shows your build string, including -lab.
• The guest boots to a shell in under 30 seconds and quits cleanly with Ctrl-A X.
• -smp 4 — you verified with nproc in the guest that it has four CPUs.
• GDB attaches, lx-dmesg prints the guest's log, and a breakpoint on a syscall entry fires.
• lx-symbols resolves a symbol from a module you loaded.
• A deliberate BUG() or bad dereference in a test module produces a panic with a readable stack trace, and QEMU exits instead of rebooting forever.
• You can state what each of nokaslr, console=ttyS0, panic=1, -s, and -S does.

And the questions:

1. Why vmlinux and not bzImage for GDB? What is the difference between the two files?
2. Why is nokaslr required for debugging, and what security property are you giving up in the guest?
3. Why does this rig insist on -smp greater than 1?
4. What does an initramfs not give you that a disk image does, and why is that a feature here?
5. Your breakpoint on a module function does not fire even though the module is loaded. Name two causes and the command that distinguishes them.
6. You added CONFIG_DEBUG_INFO_BTF=y to the fragment and it is absent from .config afterwards. What happened, and how do you find out?
7. Why does this chapter tell you to build the busybox initramfs by hand when virtme-ng exists?
8. The guest is frozen at a breakpoint for two minutes. What will happen to timers and watchdogs when you continue, and what class of bug report would that produce if you did not know?

Next: The Roadmap — fifteen milestones, each with completion criteria you can check.