Foundations: The Floor
This section is the floor. Seven concepts and six labs, and at the end of it you can write kernel code, load it, break it, debug it, and test it — which is the prerequisite for every other section and for every patch you will ever send.
It is also the longest section, and the one where people fall behind. That is not an accident: the concepts here are the ones with non-local failure modes. A mistake in context or concurrency does not produce a compiler error or a crash at the offending line. It produces a corrupted list on someone else's 96-core machine, three weeks later, once.
Learning Objectives
| # | You will be able to |
|---|---|
| 1 | Trace a syscall from the syscall instruction to the handler and back, on your own tree |
| 2 | Explain why kernel code cannot dereference a user pointer, and use the four APIs that can |
| 3 | Write C in the kernel's dialect: no libc, no float, a 16 KB stack, negative errnos, goto unwinding |
| 4 | Name your current context and say what it forbids — reflexively, without looking it up |
| 5 | Choose the right synchronization primitive and defend the choice against the two obvious alternatives |
| 6 | Choose the right allocator and the right GFP flags, and say what happens under memory pressure |
| 7 | Split work between an interrupt handler and a deferred half, and pick the right deferral mechanism |
| 8 | Write a driver that binds to a device through the device model, with correct probe/remove and devres |
| 9 | Build, boot, and debug a kernel under QEMU with GDB, on demand, in minutes |
| 10 | Read an oops, a lockdep splat, and a KASAN report line by line |
| 11 | Write a KUnit test and a kselftest, and say which a given bug belongs in |
Before You Start
This section assumes a working lab rig. Specifically:
[ ] A kernel tree that builds, with ccache, in an O= directory
[ ] lab-fast.config and lab-paranoid.config applied and VERIFIED in .config
[ ] A guest that boots to a shell in under 30 seconds, with -smp 4 or more
[ ] GDB attaching, lx-dmesg working, breakpoints firing
[ ] A way to get a .ko into the guest without rebuilding the initramfs
If any of those is not true, stop and finish Lab 1 first — it is the lab that builds them. Everything after it assumes you experiment freely, and you will not experiment freely if each experiment costs five minutes.
Warning: Run every lab in this section with
-smp 4at minimum, and re-run the concurrency work with-smp 8. A single-CPU guest makes broken locking look correct. That is the single most effective way to waste a month.
The Seven Concepts, and Why in This Order
flowchart TD
UKB["1. The user/kernel boundary<br/>what crosses, and how"]
KC["2. Kernel C<br/>the dialect and its rules"]
CTX["3. Context and atomicity<br/>may this code sleep?"]
CON["4. Concurrency<br/>who else is touching this?"]
MEM["5. Memory<br/>which allocator, which flags"]
DEF["6. Deferred work<br/>splitting the interrupt"]
DM["7. The device model<br/>how code meets hardware"]
UKB --> KC
KC --> CTX
CTX --> CON
CTX --> MEM
CON --> DEF
MEM --> DEF
DEF --> DM
| # | Chapter | The question it answers | Why it is here |
|---|---|---|---|
| 1 | The User/Kernel Boundary | What crosses the line, and what must never be trusted across it? | Every kernel security bug is a boundary bug. Start here. |
| 2 | Kernel C | Why does my C not compile / not work here? | You cannot read the tree until you can read its dialect. |
| 3 | Context and Atomicity | May this code sleep? | The #1 newbie bug, and it is a category, not a mistake. |
| 4 | Concurrency | Who else can touch this data right now? | The #2 bug, and the one that survives review. |
| 5 | Memory | Which allocator, which flags, and can a device reach it? | Wrong answers here fail only under pressure — i.e. in production. |
| 6 | Deferred Work | The interrupt is 2 µs and the work is 2 ms. Now what? | Half of driver design is this decision. |
| 7 | The Device Model | How does a driver find its hardware, and who cleans up? | The scaffolding under ~70% of the tree. |
Chapters 3 and 4 are the ones to slow down on. If you have limited time, read those two twice and skim the rest — the others you can look up, and those two you cannot, because you have to notice you need them.
The Six Labs
| Lab | Milestone | You build | You learn it works when |
|---|---|---|---|
| Lab 1: Build and Boot | M1 | The rig itself | uname -a shows your build, and a breakpoint fires |
| Lab 2: First Module | M2 | An out-of-tree module with parameters | Ten load/unload cycles leak nothing |
| Lab 3: Character Device | M3 | A real device node with ioctl | Two concurrent processes cannot corrupt it |
| Lab 4: Debugging | M4 | Nothing — you break things and diagnose them | You can name a bug from its dmesg output alone |
| Lab 5: A Syscall | M5 | One syscall, and the argument against it | The ABI survives a struct change; the old binary does not break |
| Lab 6: Testing | M6 | A KUnit suite and a kselftest | Both fail informatively when you reintroduce the bug |
Labs 2, 3, 5, and 6 build on each other: the char device from Lab 3 is what Lab 4 debugs and what Lab 6 tests. Lab 5 is independent and can be done any time after Lab 2.
The Structures and APIs You Will Meet
A quick-reference index. Every name here should be grepped, not memorized — this table exists so you recognize a name when you hit it, not so you can recite it.
| Area | Names | Find it |
|---|---|---|
| Tasks | task_struct, current, pid/tgid, TASK_RUNNING/TASK_INTERRUPTIBLE | rg -n "struct task_struct \{" include/linux/sched.h |
| Boundary | SYSCALL_DEFINEn, copy_to_user, copy_from_user, get_user, put_user, __user | rg -n "define SYSCALL_DEFINE3" include/linux/syscalls.h |
| Errors | -EINVAL…, ERR_PTR, IS_ERR, PTR_ERR, dev_err_probe | rg -n "define IS_ERR\b" include/linux/err.h |
| Context | might_sleep, in_task, in_hardirq, in_atomic, preempt_disable | rg -n "define might_sleep\b" include/linux/kernel.h include/linux/sched.h |
| Locking | spinlock_t, raw_spinlock_t, struct mutex, rw_semaphore, seqlock_t, atomic_t, refcount_t | ls include/linux/spinlock.h include/linux/mutex.h include/linux/refcount.h |
| RCU | rcu_read_lock, rcu_dereference, rcu_assign_pointer, synchronize_rcu, kfree_rcu | ls Documentation/RCU/ |
| Memory | kmalloc, kzalloc, kvmalloc, alloc_pages, vmalloc, kmem_cache_create, gfp_t | $EDITOR Documentation/core-api/memory-allocation.rst |
| DMA | dma_alloc_coherent, dma_map_single, dma_set_mask_and_coherent, dma_addr_t | ls Documentation/core-api/dma-* |
| Deferred | request_irq, request_threaded_irq, INIT_WORK, queue_work, timer_setup, hrtimer | rg -n "int request_threaded_irq" include/linux/interrupt.h |
| Device model | struct device, device_driver, bus_type, probe, devm_kzalloc, of_match_table | rg -n "struct device_driver \{" include/linux/device/driver.h |
| Files | file_operations, struct file, struct inode, cdev, miscdevice | rg -n "struct file_operations \{" include/linux/fs.h |
| Modules | module_init, module_exit, MODULE_LICENSE, module_param, THIS_MODULE | rg -n "define module_init" include/linux/module.h |
Tip: Two in-tree documents carry more of this section than any chapter here does, and both are in your checkout. Read them once now and again after Lab 3:
$EDITOR Documentation/core-api/memory-allocation.rst # which allocator, which flags $EDITOR Documentation/process/coding-style.rst # short, opinionated, enforced
The Bugs This Section Exists to Prevent
Every one of these is common, and every one of them is caught by something you will turn on.
| Bug | What it looks like | What catches it | Chapter |
|---|---|---|---|
| Sleeping in atomic context | BUG: sleeping function called from invalid context at … | CONFIG_DEBUG_ATOMIC_SLEEP | 3 |
| Dereferencing a user pointer | Oops, or a silent read of kernel memory into a user buffer | SMAP/PAN hardware, plus sparse on __user | 1 |
Ignoring copy_from_user's return | Uninitialized data used as if it were valid | Review; checkpatch will not catch it | 1 |
| Missing lock, or the wrong one | Corruption that only appears with more CPUs | lockdep, KCSAN, -smp 8 | 4 |
| ABBA deadlock | Hang, or a lockdep splat before the hang | CONFIG_PROVE_LOCKING | 4 |
GFP_KERNEL under a spinlock | Works until memory is tight, then deadlocks | CONFIG_DEBUG_ATOMIC_SLEEP | 3, 5 |
DMA to vmalloc or stack memory | Silent corruption, or an IOMMU fault | CONFIG_DMA_API_DEBUG | 5 |
| Use after free | Anything at all, later, somewhere else | KASAN | 5, Lab 4 |
| Leaking on the error path | Slow leak; nothing at all until it matters | kmemleak, CONFIG_FAILSLAB | 2, Lab 2 |
| Work still queued at teardown | Use-after-free on rmmod, intermittently | KASAN + rmmod under load | 6 |
rmmod while a device is open | Use-after-free in a userspace-triggered path | Try it. Deliberately. | 7, Lab 3 |
Deliverables
You have finished Foundations when all of these exist and are committed:
- A lab rig with a measured, written-down edit-to-breakpoint time under two minutes.
-
modules/01-helloread line by line, with your annotations. - Your own module with parameters, a failure path, and clean teardown — ten cycles, no leak.
-
A character device with
ioctl, a userspace exerciser, and a locking fix you can justify. - Four diagnostic artifacts, each annotated line by line: an oops, a lockdep splat, a KASAN report, and a sleeping-in-atomic BUG.
- A syscall, its written ABI specification, and a paragraph arguing it should not exist.
- A KUnit suite and a kselftest, both verified to fail when the bug returns.
-
predictions.mdwith a written prediction for every experiment in this section, and a one-sentence post-mortem for each one you got wrong. -
Every lab's code passing
checkpatch.pl --strict,make C=1, andmake W=1.
How to Verify You Are Actually Done
Not "did I do the labs" — can you do these cold?
# 1. Given an arbitrary kernel function, determine which contexts may call it.
# Try it on three you have never read:
rg -n "^\w+ \w+\(" mm/vmalloc.c | shuf -n 3
# 2. Read a splat you did not produce. Search lore for a recent one:
# https://lore.kernel.org — search "BUG: sleeping function called"
# Can you say what the author did wrong, from the message alone?
# 3. Given an allocation site, name the allocator and flags and defend it.
rg -n "kmalloc\(|kzalloc\(|kvmalloc\(|vmalloc\(" drivers/ | shuf -n 5
# For each: why that one? What would break with the alternatives?
# 4. Given a driver, find its bind path end to end.
ls drivers/misc/*.c | shuf -n 1
# Which bus? What matches it? What does probe() do, in order? What does devres free?
If those four feel like work but not like guessing, you are done.
Contributor Profile: Foundations Graduate
This is what a maintainer can assume about you once this section is behind you.
| Dimension | What you can do |
|---|---|
| Reading | Open an unfamiliar .c file in drivers/ and follow it without stopping at an idiom |
| Writing | Produce kernel C that passes checkpatch --strict, sparse, and W=1 on the first try |
| Context discipline | Know what context every line of your code runs in, and say so in review |
| Locking | Choose a primitive, name what it protects, and say what would break without it |
| Memory | Choose an allocator and GFP flags deliberately, and handle allocation failure everywhere |
| Teardown | Write an error path and a remove() that unwind completely, in reverse order |
| Debugging | Diagnose from dmesg, ftrace, and KASAN before reaching for printk |
| Testing | Cover a change with the right framework, and prove the test fails without the fix |
| What you cannot do yet | Send a patch. That is the next section, and it is deliberately separate. |
The gap between here and a merged patch is workflow, not knowledge — which is exactly why Contribution comes next, and why it is short.
Next: The User/Kernel Boundary — what crosses the line, and why nothing that comes across it may ever be trusted.