Foundations: The Floor

This section is the floor. Seven concepts and six labs, and at the end of it you can write kernel code, load it, break it, debug it, and test it — which is the prerequisite for every other section and for every patch you will ever send.

It is also the longest section, and the one where people fall behind. That is not an accident: the concepts here are the ones with non-local failure modes. A mistake in context or concurrency does not produce a compiler error or a crash at the offending line. It produces a corrupted list on someone else's 96-core machine, three weeks later, once.


Learning Objectives

#You will be able to
1Trace a syscall from the syscall instruction to the handler and back, on your own tree
2Explain why kernel code cannot dereference a user pointer, and use the four APIs that can
3Write C in the kernel's dialect: no libc, no float, a 16 KB stack, negative errnos, goto unwinding
4Name your current context and say what it forbids — reflexively, without looking it up
5Choose the right synchronization primitive and defend the choice against the two obvious alternatives
6Choose the right allocator and the right GFP flags, and say what happens under memory pressure
7Split work between an interrupt handler and a deferred half, and pick the right deferral mechanism
8Write a driver that binds to a device through the device model, with correct probe/remove and devres
9Build, boot, and debug a kernel under QEMU with GDB, on demand, in minutes
10Read an oops, a lockdep splat, and a KASAN report line by line
11Write a KUnit test and a kselftest, and say which a given bug belongs in

Before You Start

This section assumes a working lab rig. Specifically:

  [ ] A kernel tree that builds, with ccache, in an O= directory
  [ ] lab-fast.config and lab-paranoid.config applied and VERIFIED in .config
  [ ] A guest that boots to a shell in under 30 seconds, with -smp 4 or more
  [ ] GDB attaching, lx-dmesg working, breakpoints firing
  [ ] A way to get a .ko into the guest without rebuilding the initramfs

If any of those is not true, stop and finish Lab 1 first — it is the lab that builds them. Everything after it assumes you experiment freely, and you will not experiment freely if each experiment costs five minutes.

Warning: Run every lab in this section with -smp 4 at minimum, and re-run the concurrency work with -smp 8. A single-CPU guest makes broken locking look correct. That is the single most effective way to waste a month.


The Seven Concepts, and Why in This Order

flowchart TD
    UKB["1. The user/kernel boundary<br/>what crosses, and how"]
    KC["2. Kernel C<br/>the dialect and its rules"]
    CTX["3. Context and atomicity<br/>may this code sleep?"]
    CON["4. Concurrency<br/>who else is touching this?"]
    MEM["5. Memory<br/>which allocator, which flags"]
    DEF["6. Deferred work<br/>splitting the interrupt"]
    DM["7. The device model<br/>how code meets hardware"]
    UKB --> KC
    KC --> CTX
    CTX --> CON
    CTX --> MEM
    CON --> DEF
    MEM --> DEF
    DEF --> DM
#ChapterThe question it answersWhy it is here
1The User/Kernel BoundaryWhat crosses the line, and what must never be trusted across it?Every kernel security bug is a boundary bug. Start here.
2Kernel CWhy does my C not compile / not work here?You cannot read the tree until you can read its dialect.
3Context and AtomicityMay this code sleep?The #1 newbie bug, and it is a category, not a mistake.
4ConcurrencyWho else can touch this data right now?The #2 bug, and the one that survives review.
5MemoryWhich allocator, which flags, and can a device reach it?Wrong answers here fail only under pressure — i.e. in production.
6Deferred WorkThe interrupt is 2 µs and the work is 2 ms. Now what?Half of driver design is this decision.
7The Device ModelHow does a driver find its hardware, and who cleans up?The scaffolding under ~70% of the tree.

Chapters 3 and 4 are the ones to slow down on. If you have limited time, read those two twice and skim the rest — the others you can look up, and those two you cannot, because you have to notice you need them.


The Six Labs

LabMilestoneYou buildYou learn it works when
Lab 1: Build and BootM1The rig itselfuname -a shows your build, and a breakpoint fires
Lab 2: First ModuleM2An out-of-tree module with parametersTen load/unload cycles leak nothing
Lab 3: Character DeviceM3A real device node with ioctlTwo concurrent processes cannot corrupt it
Lab 4: DebuggingM4Nothing — you break things and diagnose themYou can name a bug from its dmesg output alone
Lab 5: A SyscallM5One syscall, and the argument against itThe ABI survives a struct change; the old binary does not break
Lab 6: TestingM6A KUnit suite and a kselftestBoth fail informatively when you reintroduce the bug

Labs 2, 3, 5, and 6 build on each other: the char device from Lab 3 is what Lab 4 debugs and what Lab 6 tests. Lab 5 is independent and can be done any time after Lab 2.


The Structures and APIs You Will Meet

A quick-reference index. Every name here should be grepped, not memorized — this table exists so you recognize a name when you hit it, not so you can recite it.

AreaNamesFind it
Taskstask_struct, current, pid/tgid, TASK_RUNNING/TASK_INTERRUPTIBLErg -n "struct task_struct \{" include/linux/sched.h
BoundarySYSCALL_DEFINEn, copy_to_user, copy_from_user, get_user, put_user, __userrg -n "define SYSCALL_DEFINE3" include/linux/syscalls.h
Errors-EINVAL…, ERR_PTR, IS_ERR, PTR_ERR, dev_err_proberg -n "define IS_ERR\b" include/linux/err.h
Contextmight_sleep, in_task, in_hardirq, in_atomic, preempt_disablerg -n "define might_sleep\b" include/linux/kernel.h include/linux/sched.h
Lockingspinlock_t, raw_spinlock_t, struct mutex, rw_semaphore, seqlock_t, atomic_t, refcount_tls include/linux/spinlock.h include/linux/mutex.h include/linux/refcount.h
RCUrcu_read_lock, rcu_dereference, rcu_assign_pointer, synchronize_rcu, kfree_rculs Documentation/RCU/
Memorykmalloc, kzalloc, kvmalloc, alloc_pages, vmalloc, kmem_cache_create, gfp_t$EDITOR Documentation/core-api/memory-allocation.rst
DMAdma_alloc_coherent, dma_map_single, dma_set_mask_and_coherent, dma_addr_tls Documentation/core-api/dma-*
Deferredrequest_irq, request_threaded_irq, INIT_WORK, queue_work, timer_setup, hrtimerrg -n "int request_threaded_irq" include/linux/interrupt.h
Device modelstruct device, device_driver, bus_type, probe, devm_kzalloc, of_match_tablerg -n "struct device_driver \{" include/linux/device/driver.h
Filesfile_operations, struct file, struct inode, cdev, miscdevicerg -n "struct file_operations \{" include/linux/fs.h
Modulesmodule_init, module_exit, MODULE_LICENSE, module_param, THIS_MODULErg -n "define module_init" include/linux/module.h

Tip: Two in-tree documents carry more of this section than any chapter here does, and both are in your checkout. Read them once now and again after Lab 3:

$EDITOR Documentation/core-api/memory-allocation.rst   # which allocator, which flags
$EDITOR Documentation/process/coding-style.rst         # short, opinionated, enforced

The Bugs This Section Exists to Prevent

Every one of these is common, and every one of them is caught by something you will turn on.

BugWhat it looks likeWhat catches itChapter
Sleeping in atomic contextBUG: sleeping function called from invalid context at …CONFIG_DEBUG_ATOMIC_SLEEP3
Dereferencing a user pointerOops, or a silent read of kernel memory into a user bufferSMAP/PAN hardware, plus sparse on __user1
Ignoring copy_from_user's returnUninitialized data used as if it were validReview; checkpatch will not catch it1
Missing lock, or the wrong oneCorruption that only appears with more CPUslockdep, KCSAN, -smp 84
ABBA deadlockHang, or a lockdep splat before the hangCONFIG_PROVE_LOCKING4
GFP_KERNEL under a spinlockWorks until memory is tight, then deadlocksCONFIG_DEBUG_ATOMIC_SLEEP3, 5
DMA to vmalloc or stack memorySilent corruption, or an IOMMU faultCONFIG_DMA_API_DEBUG5
Use after freeAnything at all, later, somewhere elseKASAN5, Lab 4
Leaking on the error pathSlow leak; nothing at all until it matterskmemleak, CONFIG_FAILSLAB2, Lab 2
Work still queued at teardownUse-after-free on rmmod, intermittentlyKASAN + rmmod under load6
rmmod while a device is openUse-after-free in a userspace-triggered pathTry it. Deliberately.7, Lab 3

Deliverables

You have finished Foundations when all of these exist and are committed:

  • A lab rig with a measured, written-down edit-to-breakpoint time under two minutes.
  • modules/01-hello read line by line, with your annotations.
  • Your own module with parameters, a failure path, and clean teardown — ten cycles, no leak.
  • A character device with ioctl, a userspace exerciser, and a locking fix you can justify.
  • Four diagnostic artifacts, each annotated line by line: an oops, a lockdep splat, a KASAN report, and a sleeping-in-atomic BUG.
  • A syscall, its written ABI specification, and a paragraph arguing it should not exist.
  • A KUnit suite and a kselftest, both verified to fail when the bug returns.
  • predictions.md with a written prediction for every experiment in this section, and a one-sentence post-mortem for each one you got wrong.
  • Every lab's code passing checkpatch.pl --strict, make C=1, and make W=1.

How to Verify You Are Actually Done

Not "did I do the labs" — can you do these cold?

# 1. Given an arbitrary kernel function, determine which contexts may call it.
#    Try it on three you have never read:
rg -n "^\w+ \w+\(" mm/vmalloc.c | shuf -n 3

# 2. Read a splat you did not produce. Search lore for a recent one:
#    https://lore.kernel.org — search "BUG: sleeping function called"
#    Can you say what the author did wrong, from the message alone?

# 3. Given an allocation site, name the allocator and flags and defend it.
rg -n "kmalloc\(|kzalloc\(|kvmalloc\(|vmalloc\(" drivers/ | shuf -n 5
#    For each: why that one? What would break with the alternatives?

# 4. Given a driver, find its bind path end to end.
ls drivers/misc/*.c | shuf -n 1
#    Which bus? What matches it? What does probe() do, in order? What does devres free?

If those four feel like work but not like guessing, you are done.


Contributor Profile: Foundations Graduate

This is what a maintainer can assume about you once this section is behind you.

DimensionWhat you can do
ReadingOpen an unfamiliar .c file in drivers/ and follow it without stopping at an idiom
WritingProduce kernel C that passes checkpatch --strict, sparse, and W=1 on the first try
Context disciplineKnow what context every line of your code runs in, and say so in review
LockingChoose a primitive, name what it protects, and say what would break without it
MemoryChoose an allocator and GFP flags deliberately, and handle allocation failure everywhere
TeardownWrite an error path and a remove() that unwind completely, in reverse order
DebuggingDiagnose from dmesg, ftrace, and KASAN before reaching for printk
TestingCover a change with the right framework, and prove the test fails without the fix
What you cannot do yetSend a patch. That is the next section, and it is deliberately separate.

The gap between here and a merged patch is workflow, not knowledge — which is exactly why Contribution comes next, and why it is short.


Next: The User/Kernel Boundary — what crosses the line, and why nothing that comes across it may ever be trusted.