Lab 1: Build, Boot, and Attach a Debugger (Milestone 1)

Background

Everything in this curriculum is an experiment, and an experiment needs a machine you can break. This lab builds it: a kernel you compiled, booting in QEMU against a root filesystem you made, with a debugger attached that can stop the entire machine on a function you chose.

It is not a small lab. It is the one that decides whether you finish the curriculum, because a five-minute edit-test loop kills curiosity and a two-minute one does not.

The Lab Rig is the specification. This lab is you executing it, with predictions and checks at each step.

Why This Lab Matters

You cannot debug what you cannot rebuild. Distro kernels have no vmlinux with debug info and no way to change a line.
Every later lab assumes you can load a module, crash the machine, read the trace, fix it, and reboot — in a loop tight enough that you do it dozens of times an evening.
The QEMU + GDB rig is what makes "stop the whole machine and look at the scheduler's data structures" a thing you can actually do.
Maintainers ask "did you test this?" The honest answer requires this rig.

Prerequisites

Overview & Prerequisites complete: packages installed, tree cloned.
The Lab Rig read.
The Kernel Mental Model read — Milestone 0.
At least 60 GB of free disk and a machine (or VM) you can leave building for a while.

Predict First

Write these down before you start. All five.

How long will the first full build take on your machine? How large will the build directory be?
How large is vmlinux? How large is bzImage/Image? Why is one so much bigger?
How long does the guest take to reach a shell prompt, with KVM and without?
After a one-line change to a .c file in drivers/, how many files does make recompile?
If you set a breakpoint on do_sys_openat2 and continue, how long until it fires?

The Target

   HOST                                        GUEST (QEMU)
   ────                                        ────────────
   ~/kernel/linux/     source, pristine
   ~/kernel/build/     .config, vmlinux, bzImage
   ~/kernel/rootfs/    initramfs.cpio.gz   ─────▶  unpacked into a tmpfs
                                                   /init → busybox sh
   gdb vmlinux ──────── tcp:1234 ─────────────────▶ the whole machine,
     lx-dmesg, lx-ps, breakpoints                   stoppable
   serial console ◀──── stdio ──────────────────── console=ttyS0

Step-by-Step Tasks

Step 1: Build a baseline kernel

cd ~/kernel/linux
mkdir -p ../build
ccache -M 20G && ccache -z

time make O=../build defconfig
time make O=../build -j"$(nproc)" CC="ccache gcc"

ls -lh ../build/vmlinux ../build/arch/x86/boot/bzImage 2>/dev/null || \
ls -lh ../build/vmlinux ../build/arch/arm64/boot/Image
ccache -s | grep -E 'cache hit|cache miss'
du -sh ../build

Compare against predictions 1 and 2. vmlinux will be an order of magnitude larger than the boot image; write down why in one sentence.

Step 2: Apply the lab config fragment

Create ~/kernel-labs/lab-fast.config with the contents from the lab rig, then:

cd ~/kernel/linux
./scripts/kconfig/merge_config.sh -O ../build ../build/.config ~/kernel-labs/lab-fast.config
./scripts/config --file ../build/.config --set-str LOCALVERSION "-lab"
make O=../build olddefconfig

# VERIFY. Every one of these must print the value you asked for.
grep -E '^CONFIG_(GDB_SCRIPTS|DEBUG_INFO_BTF|KALLSYMS_ALL|PROVE_LOCKING|DEBUG_ATOMIC_SLEEP|FUNCTION_GRAPH_TRACER|KUNIT|MODULE_UNLOAD)=' ../build/.config
grep -E '^CONFIG_LOCALVERSION=' ../build/.config

make O=../build -j"$(nproc)" CC="ccache gcc"

Warning: A symbol you asked for that is absent from that grep output was dropped by Kconfig because its dependencies were unmet — silently. The most common one is CONFIG_DEBUG_INFO_BTF failing because pahole is not installed, which then silently disables bpftrace in the guest. Find out why with make O=../build menuconfig, press /, and search the symbol; it shows the dependency chain.

Step 3: Build the root filesystem

Follow the lab rig, §3 to build a static busybox initramfs at ~/kernel/rootfs/initramfs.cpio.gz.

ls -lh ~/kernel/rootfs/initramfs.cpio.gz
# Sanity-check the contents before you boot it:
zcat ~/kernel/rootfs/initramfs.cpio.gz | cpio -t 2>/dev/null | head -20
zcat ~/kernel/rootfs/initramfs.cpio.gz | cpio -t 2>/dev/null | grep -x './init'

If ./init is not in that listing, the kernel will panic with No working init found and you will spend twenty minutes on it. Check now.

Step 4: Boot it

cd ~/kernel
qemu-system-x86_64 \
  -kernel build/arch/x86/boot/bzImage \
  -initrd rootfs/initramfs.cpio.gz \
  -append "console=ttyS0 earlyprintk=serial,ttyS0 nokaslr panic=1 oops=panic" \
  -nographic -no-reboot -m 2G -smp 4 -enable-kvm -cpu host \
  -virtfs local,path="$HOME/kernel-labs",mount_tag=host,security_model=none,id=host \
  -s

(arm64: use the invocation from the lab rig, §4.)

Inside the guest:

uname -a                  # must show YOUR build and the -lab suffix
nproc                     # must be 4 — a 1-CPU guest hides every race
cat /proc/cmdline
ls /sys/kernel/tracing /sys/kernel/debug
ls /mnt/host              # the 9p share

Quit with Ctrl-A then X.

Step 5: Attach GDB

Leave QEMU running. In another terminal:

cat >> ~/.gdbinit <<EOF
add-auto-load-safe-path $HOME/kernel/build
set print pretty on
EOF

cd ~/kernel/build
gdb vmlinux

(gdb) target remote :1234
(gdb) lx-version
(gdb) lx-dmesg
(gdb) lx-ps
(gdb) break do_sys_openat2
(gdb) continue

Now, in the guest: cat /etc/hostname. GDB should stop the machine.

(gdb) bt
(gdb) info threads              # one GDB thread per vCPU
(gdb) p $lx_current().comm      # or: p current->comm
(gdb) finish
(gdb) continue

Step 6: Break early boot

Kill QEMU, restart it with -S added, and before continuing:

(gdb) target remote :1234
(gdb) break start_kernel
(gdb) continue
(gdb) bt

You are now stopped before the kernel has initialized anything. Step around for a minute — this is the view that makes early-boot bugs tractable.

Step 7: Measure the loop

This is the deliverable that matters most.

cd ~/kernel/linux
touch drivers/misc/Makefile          # a small, leaf-ish target
time make O=../build -j"$(nproc)" CC="ccache gcc"

Then the full cycle, end to end, with a stopwatch:

   edit one .c file  →  make  →  QEMU boots  →  breakpoint fires
   ─────────────────────────────────────────────────────────────
   target: under two minutes

Write down your actual number. If it is much worse, fix it now — the debugging table below has the usual causes.

Step 8: Crash it on purpose

Build the trivial crash module, load it, and confirm the whole failure path works:

// SPDX-License-Identifier: GPL-2.0
#include <linux/module.h>
static int __init crash_init(void)
{
        int *p = NULL;
        pr_info("crash: about to dereference NULL\n");
        *p = 1;                         /* deliberate */
        return 0;
}
module_init(crash_init);
MODULE_DESCRIPTION("Deliberate oops, for testing the rig");
MODULE_LICENSE("GPL");

# Makefile
obj-m += crash.o
KDIR ?= $(HOME)/kernel/build
all:
	$(MAKE) -C $(KDIR) M=$(PWD) modules
clean:
	$(MAKE) -C $(KDIR) M=$(PWD) clean

make            # on the host, into the 9p-shared directory
# in the guest:
insmod /mnt/host/crash/crash.ko

With oops=panic and panic=1 and -no-reboot, QEMU should exit, not reboot-loop. That is the behavior you want for every future lab.

Implementation Requirements / Deliverables

A kernel that builds with the lab-fast fragment, verified with grep, not assumed.
uname -a in the guest shows your build string including -lab.
nproc in the guest reports 4 or more.
The guest boots to a shell in under 30 seconds and quits cleanly with Ctrl-A X.
/mnt/host is mounted and you can insmod a module from it without repacking the initramfs.
GDB attaches, lx-dmesg and lx-ps work, and a breakpoint on a syscall entry fires.
A breakpoint on start_kernel hit with -S.
A deliberate oops produces a readable stack trace and QEMU exits.
A written, measured edit-to-breakpoint time. Under two minutes.
All five predictions recorded, with results and a one-sentence note for each one you got wrong.

Expected Output

$ uname -a
Linux (none) 6.x.0-lab #1 SMP PREEMPT_DYNAMIC ... x86_64 GNU/Linux
                  ^^^^ your LOCALVERSION. If this is missing, you booted the wrong image.

$ nproc
4

$ cat /proc/cmdline
console=ttyS0 earlyprintk=serial,ttyS0 nokaslr panic=1 oops=panic

GDB, after continue and a cat in the guest:

Thread 1 hit Breakpoint 1, do_sys_openat2 (dfd=-100, filename=..., how=...) at fs/open.c:...
(gdb) bt
#0  do_sys_openat2 (...)
#1  __do_sys_openat (...)
#2  do_syscall_64 (...)
#3  entry_SYSCALL_64_after_hwframe ()

That backtrace is the syscall entry path you read about, on your own machine, with your own cat stopped inside it.

And the deliberate oops:

BUG: kernel NULL pointer dereference, address: 0000000000000000
#PF: supervisor write access in kernel mode
RIP: 0010:crash_init+0x1e/0xff0 [crash]
Call Trace:
 do_one_initcall+0x...
 do_init_module+0x...
 __do_sys_init_module+0x...
Kernel panic - not syncing: Fatal exception

Debugging Steps

The build fails on a missing header or tool

Re-run the package install from Overview & Prerequisites. The three that fail late and confusingly are libelf-dev (objtool), bc, and dwarves/pahole.

`No working init found` / `Kernel panic - not syncing: No working init`

./init is missing from the cpio, is not executable, or is a dynamically-linked binary with no libc in the image. Check with the cpio -t command in step 3, and confirm busybox is CONFIG_STATIC=y:

file ~/kernel/rootfs/initramfs/bin/busybox     # must say "statically linked"

The boot is silent — no output at all

console=ttyS0 is missing from -append, or the serial driver is not built in. Both are covered by the config fragment; verify:

grep -E '^CONFIG_SERIAL_8250(_CONSOLE)?=' ~/kernel/build/.config       # x86
grep -E '^CONFIG_SERIAL_AMBA_PL011(_CONSOLE)?=' ~/kernel/build/.config # arm64 virt

Add earlyprintk=serial,ttyS0 (x86) or earlycon (arm64) to see output from before the real console registers.

The breakpoint never fires

In order:

nokaslr missing from -append. Symbols will not match addresses.
The function was inlined. Check: sudo grep -w do_sys_openat2 /proc/kallsyms in the guest.
You are debugging a different vmlinux than the one you booted. lx-version in GDB versus uname -a in the guest — they must match exactly.

`lx-dmesg` is undefined

vmlinux-gdb.py was not auto-loaded. Check CONFIG_GDB_SCRIPTS=y, confirm the file exists next to vmlinux, and confirm the add-auto-load-safe-path line is in ~/.gdbinit.

Half my variables are `<optimized out>`

Normal. The kernel is built at -O2 and cannot be built at -O0 — it does not compile, and would not fit or boot if it did. Print the containing struct, use lx- scripts, or add a printk.

QEMU reboot-loops on a panic

You are missing -no-reboot, or panic=1 is not in -append. Both, together, turn a panic into a clean exit.

The guest hangs and Ctrl-A X does nothing

Use the QEMU monitor: Ctrl-A then C, then quit. From there info registers and info cpus also work on a wedged guest — worth knowing before you need it.

A one-file change rebuilds thousands of files

You edited a widely-included header, which is itself worth knowing. If you did not, check that O=../build is consistent between invocations and that your clock is not skewed.

`ccache` hit rate is near zero

CC="ccache gcc" is not being passed on every invocation, or KBUILD_BUILD_TIMESTAMP is changing. ccache -s tells you which.

Experiment

CLAIM. nokaslr is required for source-level debugging, and without it the symbols GDB has and the addresses the kernel uses are unrelated.

METHOD.

# Boot 1: with nokaslr (as above). In GDB:
#   (gdb) p &do_sys_openat2
#   ...and in the guest:
#   # grep -w do_sys_openat2 /proc/kallsyms

# Boot 2: remove nokaslr from -append. Repeat both.

PREDICTION. Before running: (a) will the two addresses match in boot 1? (b) in boot 2? (c) will the breakpoint still fire in boot 2? (d) will the guest's own /proc/kallsyms address change between two boots without nokaslr?

RESULT. Record all four. Then answer the question this raises: KASLR exists to make kernel addresses unguessable to an attacker — so what have you given up in the guest by disabling it, and why is that acceptable here and not in production?

Test

The rig itself needs a regression test, because you will change it.

cat > ~/kernel-labs/scripts/rig-check.sh <<'EOF'
#!/usr/bin/env bash
# Boot the guest, run a command, assert on the output, and exit non-zero on failure.
set -euo pipefail
BUILD=${BUILD:-$HOME/kernel/build}
ROOTFS=${ROOTFS:-$HOME/kernel/rootfs/initramfs.cpio.gz}
KERNEL=$(ls "$BUILD"/arch/*/boot/bzImage "$BUILD"/arch/*/boot/Image 2>/dev/null | head -1)

out=$(timeout 60 qemu-system-"$(uname -m)" \
        -kernel "$KERNEL" -initrd "$ROOTFS" \
        -append "console=ttyS0 nokaslr panic=1 oops=panic rdinit=/init" \
        -nographic -no-reboot -m 1G -smp 4 \
        ${KVM:+-enable-kvm -cpu host} <<< 'uname -r; nproc; poweroff -f' 2>&1) || true

echo "$out" | tail -30

grep -q -- '-lab'   <<< "$out" || { echo "FAIL: LOCALVERSION not present"; exit 1; }
grep -qx '4'        <<< "$out" || { echo "FAIL: guest does not have 4 CPUs"; exit 1; }
grep -q 'Oops\|BUG:\|panic' <<< "$out" && { echo "FAIL: guest oopsed on boot"; exit 1; }
echo "PASS"
EOF
chmod +x ~/kernel-labs/scripts/rig-check.sh
~/kernel-labs/scripts/rig-check.sh

Run this after every config change. It catches the two failures that waste the most time: booting a stale image, and a config change that broke the guest's boot.

Challenge Extensions

Build the paranoid kernel too. A second build directory with lab-paranoid.config (KASAN, UBSAN, kmemleak). Measure the build time, the boot time, and the image size against lab-fast, and write the three ratios down. You will make this trade dozens of times.
Cross-compile. Build an arm64 kernel on an x86-64 host (make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu-) and boot it under qemu-system-aarch64 -M virt. Note every place a path in this book changed (arch/arm64/boot/Image, console=ttyAMA0). Doing this once removes a whole category of "works on my machine".
virtme-ng. Install it and boot the same kernel with vng. Compare the setup cost and the iteration speed against your hand-built rig, then articulate what vng was doing for you that you now understand.
A persistent disk. Add -drive file=disk.img,if=virtio and a mkfs.ext4 filesystem, and mount it from /init. You need this the moment a lab wants state to survive a reboot.
Bisect something. Pick two kernel tags a few releases apart, find any behavioral difference you can script, and run git bisect run against your rig. This is Milestone 12's skill, and doing a trivial version now makes the real one far less intimidating.
Automate the whole loop. One command that rebuilds, repacks if needed, boots, runs a script in the guest, and exits with the guest's status. This is what scripts/run-qemu.sh in the companion workspace does; write your own version and then read theirs.

Validation / Self-check

Why does GDB need vmlinux rather than bzImage? What exactly is in each?
Why is nokaslr required, and what security property are you disabling in the guest?
What does console=ttyS0 do, and what does earlyprintk/earlycon add on top?
What do panic=1 and -no-reboot do together, and why is that the behavior you want?
Why does this rig insist on -smp 4 or more?
What is an initramfs, how does the kernel find /init, and what does not survive a reboot?
You changed a CONFIG_ symbol, and after olddefconfig it is absent from .config. What happened, and how do you find out why?
Your breakpoint does not fire. Give three causes and the command that distinguishes each.
Why can the kernel not be built at -O0, and what do you do instead when a variable is <optimized out>?
The guest is stopped at a breakpoint for two minutes. What happens to timers and watchdogs when you continue, and what class of false bug report does that produce?
What is your measured edit-to-breakpoint time, and what is the slowest step in it?

Next: Lab 2 — Your First Module. The rig is built; now put your own code inside the kernel.

Open-Source Engineer & Contributor