The Linux Kernel: Build It, Read It, and Get a Patch Merged

Welcome to the Linux Kernel Curriculum — a project-based engineering apprenticeship in which you build and boot your own kernel, write drivers and modules by hand, debug them under QEMU with GDB attached, learn to read any subsystem cold, and then send a patch to a mailing list and see it merged into mainline.

The primary goal is not the patch. The primary goal is that you can open any directory of the Linux source tree — mm/, net/core/, drivers/gpu/drm/, security/landlock/ — and read it without hitting a layer you do not understand, then change it in a way a maintainer will take.

What This Curriculum Is

This is the book's first build-it and contribute curriculum, and that dual nature is not an accident of scope. The other curricula pick a side:

You must build by hand, because there is no way to understand what GFP_ATOMIC means by reading about it — you learn it the first time your module deadlocks in an interrupt handler. And you must master a contribution workflow that exists nowhere else in open source: no pull requests, no web UI, no CI badge on a green PR. Plain-text email, to a mailing list, threaded correctly, reviewed by a person who has said no to a thousand patches before yours.

Neither half works alone. A person who can write a char device but cannot send a patch has a hobby. A person who can format a patch but cannot explain why their change is safe in atomic context will get one trivial fix merged and never a second.

You will finish with:

• A kernel tree you build, boot, break, and debug daily.
• A set of out-of-tree modules — a char device, a sysfs device, a ramdisk block driver, a netdev, a toy LSM, a fake accelerator — each written from a skeleton with a failing test.
• A ~/kernel-labs workspace with a QEMU rig, a GDB setup, and scripts you actually use.
• Working knowledge of eight subsystems, and depth in one.
• At least one patch merged into mainline Linux, with your name in git log.

Note: That last line is the only deliverable in this book whose completion date is not under your control. A kernel release cycle is roughly nine to ten weeks. A patch sent today might land in linux-next in three weeks and in a tagged release two months after that. Plan for it — see the weekly plan.

Learning Priorities

These are the outcomes, in priority order. Everything in this curriculum exists to serve one of them. When you must choose between finishing a feature and understanding a layer, choose understanding — in kernel work, the code you cannot explain is the code that corrupts memory on someone else's machine six months later.

Do not optimize for the merged patch alone. Optimize for the ability to inspect any layer of a running kernel and explain what you see.

1. The Complete Stack

Here is the whole system. Every box is a chapter of this curriculum. The doubled line is the user space / kernel space boundary — the single most important line in the diagram, because almost every kernel misconception comes from putting a behavior on the wrong side of it.

  ┌─────────────────────────────────────────────────────────────────────────────┐
  │  YOUR PROGRAM      fd = open("/data/f", O_RDONLY);  n = read(fd, buf, 4096); │
  │  libc (glibc/musl) wraps it — and sometimes does NOT cross the line at all   │
  │                    (clock_gettime, getcpu → the vDSO, mapped into your proc) │
  └─────────────────────────────────────────────────────────────────────────────┘
                    │  x86-64: `syscall`      arm64: `svc #0`
                    │  THE ONLY DOOR. Registers carry the number and the args.
 ═══════════════════│═════════════════════════════════════ USER / KERNEL ═══════
                    ▼
  ┌─────────────────────────────────────────────────────────────────────────────┐
  │  ARCHITECTURE ENTRY      arch/x86/entry/   ·   arch/arm64/kernel/entry.S     │
  │  switch stacks · save registers into `struct pt_regs` · look up the number   │
  │  in the syscall table · call the handler defined by SYSCALL_DEFINEn()        │
  └─────────────────────────────────────────────────────────────────────────────┘
                    ▼
  ┌─────────────────────────────────────────────────────────────────────────────┐
  │  GENERIC KERNEL CODE                                                        │
  │                                                                             │
  │   fs/          VFS: struct file, inode, dentry, address_space               │
  │   mm/          page faults, the page cache, folios, the allocators, reclaim │
  │   kernel/      scheduler, signals, timers, workqueues, RCU, tracing, cgroups│
  │   net/         sockets, sk_buff, protocol stacks, netfilter, qdiscs, XDP    │
  │   block/       bio, request, blk-mq, the I/O schedulers                     │
  │   security/    LSM hooks: SELinux, AppArmor, Landlock, Smack, BPF-LSM       │
  │   ipc/  lib/  crypto/  io_uring/  virt/  init/  rust/                       │
  └─────────────────────────────────────────────────────────────────────────────┘
                    ▼
  ┌─────────────────────────────────────────────────────────────────────────────┐
  │  DRIVERS AND BUSES       drivers/  ·  sound/  ·  ~70% of the whole tree      │
  │  the device model: bus → driver → device · probe() · devres · DT / ACPI      │
  │  drivers/gpu/drm  drivers/net  drivers/nvme  drivers/accel  drivers/firmware │
  └─────────────────────────────────────────────────────────────────────────────┘
                    ▼
 ═══════════════════════════════════════════════════════ KERNEL / HARDWARE ═════
  ┌─────────────────────────────────────────────────────────────────────────────┐
  │  HARDWARE      MMU · caches · interrupt controller · timers · DMA engines    │
  │                PCIe · USB · I²C · SPI · NVMe · NIC · GPU · NPU               │
  │  FIRMWARE      UEFI / device tree / ACPI / PSCI — what runs before you do    │
  └─────────────────────────────────────────────────────────────────────────────┘

Two arrows are missing from that picture, and they are the ones that cause trouble:

  INTERRUPTS come UP from the hardware at an arbitrary moment, on any CPU,
  possibly while the code you are reading holds a lock.
        hardware IRQ ──▶ handler (atomic, cannot sleep)
                          └─▶ softirq / tasklet (atomic, cannot sleep)
                          └─▶ threaded IRQ / workqueue (process ctx, MAY sleep)

  PREEMPTION can take the CPU away from your task between any two instructions,
  unless you have disabled it — which spin_lock() does for you, and which is
  half of the reason spin_lock() exists at all.

Note: Write those two paragraphs somewhere you will see them. Roughly half of all beginner kernel bugs are one of: sleeping in atomic context, or touching shared data without synchronization because you did not notice an interrupt or another CPU could reach it. Context and Atomicity and Concurrency exist entirely for this.

2. The Words, Defined Against Each Other

These terms get used interchangeably, including by people who should know better. They are different things, and blurring them is how you end up asking a question on a mailing list that reveals you have not read the tree.

The code

The trees

There is no single "the Linux source". There are thousands of trees, and knowing which one you are looking at is a prerequisite for asking a coherent question.

Three sentences that clear up most confusion:

1. Your patch is not "in Linux" until it is in Linus's tree. Being in linux-next means it will probably be in the next release. Probably.
2. The kernel has no stable internal API on purpose. Function signatures change between releases because refactoring the whole tree at once is cheaper than freezing a bad interface forever. This is an argued position, not an accident.
3. The kernel has an absolutely stable external API. "We do not break userspace" is enforced by Linus personally, with reverts. The asymmetry between rules 2 and 3 is the kernel's design philosophy.

Checkpoint question (answer before you continue): A patch adds a field to struct file and changes an exported function's signature. Is that allowed? Now: a patch changes the value returned by an existing syscall in a case that previously returned -EINVAL. Is that allowed? Write both answers down. You will be asked again in the mental model.

3. What Happens When You Call read()

This is the trace the whole curriculum is built around. Read it once now — it will not fully land. Read it again after Lab 1, when you have stepped through part of it in a debugger, and again after Lab 3, when you have written the bottom of it yourself.

The example: a program calls read(fd, buf, 4096) on a file on an NVMe SSD, and the data is not in the page cache.

Phase A — crossing the boundary

 1. Your program puts the syscall number and arguments in registers and executes
    `syscall` (x86-64) or `svc #0` (arm64). The CPU switches privilege level and
    jumps to a fixed address the kernel installed at boot.
       x86-64: MSR_LSTAR points at the entry stub in arch/x86/entry/
       arm64:  VBAR_EL1 points at the exception vector table

 2. The entry stub does the unglamorous, architecture-specific work:
       - switch to the kernel stack for this task (never trust the user stack)
       - on x86, `swapgs` so per-CPU data is reachable
       - if KPTI is on, switch page tables (this is the Meltdown mitigation)
       - save every user register into a `struct pt_regs` on the kernel stack

 3. Generic C code takes over: bounds-check the syscall number, look it up in the
    syscall table, call the handler. Handlers are declared with SYSCALL_DEFINE3(read, ...)
    — a macro, which is why grepping for "sys_read" finds less than you expect.

Find all of that on your own tree, rather than believing this page:

rg -n "SYSCALL_DEFINE3\(read" fs/
rg -n "do_syscall_64|el0_svc" arch/x86/entry/ arch/arm64/kernel/
ls arch/x86/entry/syscalls/          # the .tbl files the table is generated from

Phase B — the VFS, and the first place it can sleep

 4. fdget(fd) → the `struct file`. Wrong fd → -EBADF, and you are already done.
 5. vfs_read() checks permissions and the file's mode, then dispatches through
    `struct file_operations`:  file->f_op->read_iter(...)
       ^ THIS is the indirection every filesystem and every char driver plugs into.
         It is also exactly what you implement in Lab 3.
 6. For a regular file on ext4/xfs/btrfs, read_iter lands in the generic page-cache
    read path. It asks: is this data already in the page cache?
       HIT  → copy_to_user() and return. Microseconds. No I/O at all.
       MISS → we must go to the device, which means we must SLEEP.

Warning: Step 6 is the first hard rule of this curriculum. read() on a cache miss blocks, which means the calling task is put to sleep and the scheduler runs something else. Any code path that can reach a sleep is forbidden in atomic context — and "can reach a sleep" includes kmalloc(..., GFP_KERNEL), copy_to_user(), and mutex_lock(). Learning to see this in code you did not write is Context and Atomicity.

Phase C — down to the block layer and the hardware

 7. The filesystem maps the file offset to physical blocks (extent lookup) and builds
    one or more `struct bio` — the block layer's "read these sectors into these pages".
 8. The bio goes to blk-mq: a per-CPU software queue, then a hardware queue that maps
    to a real NVMe submission queue.
 9. The NVMe driver writes a command into a submission queue in host memory and rings
    a doorbell register.  Now the CPU has nothing to do.
10. The task is marked TASK_UNINTERRUPTIBLE and schedule() is called. Another task runs.
    Your process is now one entry on a wait queue. It does not exist to the CPU.

Phase D — the interrupt, and why "bottom halves" exist

11. Microseconds later the SSD finishes and raises an MSI-X interrupt.
12. The CPU takes the interrupt, on whatever it was doing, in ATOMIC context:
       - cannot sleep, cannot call anything that might sleep
       - other interrupts may be masked; every microsecond here adds latency everywhere
    So the handler does the minimum: acknowledge, note completion, and defer.
13. The deferred half (softirq / threaded IRQ / workqueue, depending on the driver)
    completes the bio, marks the page cache pages Uptodate, and wakes the sleeping task.
14. wake_up() moves the task to a runqueue and possibly asks for a reschedule.

Phase E — coming back out

15. The scheduler eventually picks your task. It resumes inside schedule(), exactly
    where it stopped, with its kernel stack intact. This is why every task has one.
16. copy_to_user(buf, page_data, n) — the ONLY legal way to touch a user pointer.
    It is not a memcpy: it handles page faults, checks the address range, and can
    itself sleep. A raw dereference of a user pointer is a bug and often an exploit.
17. Update the file position, return n.
18. On the way out, before returning to user mode, the kernel checks the task's flags:
       need_resched?    → schedule() (this is the preemption point)
       pending signals? → deliver them, possibly restarting or aborting the syscall
       then restore registers from pt_regs and execute `sysretq` / `eret`.
19. Your program's next instruction runs. It has no idea any of this happened.

That is the whole system. Every remaining page of this curriculum is one of those numbered steps in detail, with code you write and an experiment that proves it.

Tip: You can watch most of this right now, before you build anything:

sudo perf trace -e read -- head -c 1 /etc/hostname
sudo bpftrace -e 'kprobe:vfs_read { @[comm] = count(); }'   # Ctrl-C to print

The Warm-Up is an evening of exactly this.

4. The Eight Domains

Section 3 covers eight subsystems. You will read all eight and go deep in one — depth in one subsystem is what makes you useful; breadth across eight is what lets you tell which one.

Every subsystem chapter opens by making you run these two commands, because a table in a book goes stale and your tree does not:

git log --oneline --since="6 months ago" -- mm/ | head -30
./scripts/get_maintainer.pl -f mm/

5. The First Three Milestones

The full fifteen-milestone sequence with completion criteria is in the roadmap. Here are the first three.

Warning: M1 is the gate for everything. Do not proceed on the theory that you will "set up QEMU later". A kernel you cannot rebuild and reboot in under five minutes is a kernel you will not experiment with, and this curriculum is entirely experiments. Budget a full evening; it will take two.

6. The First Hands-On Exercise

Start here: Lab 1 — Build, Boot, and Attach a Debugger.

You will clone the tree, configure it, build it, boot it in QEMU against a tiny initramfs you also build, and attach GDB with the kernel's own lx- helper commands loaded. Then you will set a breakpoint in the syscall path and watch your own ls stop the entire machine.

It is not a small lab and it is not supposed to be. It is the difference between reading about the kernel and having one on the bench.

Before running it, you will be asked to predict several things — including how long the build takes and how big the resulting image is. Do the prediction in writing. The gap between your prediction and the result is the actual lesson.

7. Questions to Answer Before You Write Kernel Code

You are ready to move from M0 to M1 when you can answer all of these without looking anything up. They are re-asked as the validation gate at the end of the mental model.

1. Name every way control can enter the kernel from outside it. (There are more than one; "syscall" is an incomplete answer.)
2. Why can kernel code not simply dereference a pointer that came from user space? Name three distinct things that can go wrong, and the function you must use instead.
3. What is "context" in kernel terms? List the kinds, and for each, state whether you may sleep.
4. What does might_sleep() do, and what class of bug does it exist to catch early?
5. Name three functions that can sleep but do not look like they can.
6. What is the difference between GFP_KERNEL and GFP_ATOMIC, and what happens if you get it backwards in each direction?
7. What is the difference between a spinlock and a mutex — not "one spins" but which one may be taken in which context, and why?
8. What is preemption, what does spin_lock() do about it, and why does that matter for correctness rather than just performance?
9. Your driver's interrupt handler needs to do something that takes 2 ms. Name three mechanisms for deferring it, and the tradeoff between them.
10. Why does the kernel deliberately have no stable internal API, and what would be worse if it did?
11. What does "we do not break userspace" actually forbid? Give one example of a change it allows and one it forbids that look superficially similar.
12. A patch is in linux-next. Is it "in Linux"? What has to happen next, and roughly when?

Tip: Write your answers into answers-m0.md in your workspace and commit it. At the end of the curriculum you will re-answer them and diff. That diff is your progress report, and the capstone asks you for it.

Who This Is For

This curriculum is designed for engineers who:

• Are fluent in C. Pointers, pointer arithmetic, function pointers, struct layout, macros, the preprocessor, undefined behavior, and reading a linker error. The kernel is C with unusual rules, and this book teaches the unusual rules — not C. Rust in the kernel is discussed (rust/) but no lab requires it.
• Have solid systems fundamentals: processes, virtual memory, interrupts, caches, what an MMU is for. If "TLB" or "DMA" are fuzzy, you will still make it, but slow down in Foundations.
• Are comfortable at a Linux command line and can drive git beyond commit and push — rebase -i, format-patch, bisect, and range-diff are working tools here, not trivia.
• Have a Linux machine or a Linux VM they can reboot, crash, and reinstall without consequence. See Overview & Prerequisites — this is a hard requirement, and macOS users need a specific setup.
• Want to be able to explain a system, not just change it.

You do not need prior kernel experience, driver experience, or any knowledge of the mailing-list workflow. That is what you are here to build.

Restrictions (Read These; They Are the Discipline)

These constraints are what make the curriculum work. They are enforced by the ordering of the material, and breaking them produces a person with a merged patch who cannot write a second one.

• Do not start by trying to find a bug in mainline. Build first, boot first, break your own code first.
• Do not test kernel changes on your daily-driver machine. Everything happens in a VM until Lab 1 is behind you and you have said out loud why that rule exists.
• Do not use a distro's prebuilt kernel headers as a substitute for a source tree you built. You need vmlinux with debug info, and you need to be able to change any line in it.
• Do not copy a driver from Stack Overflow. The kernel API it uses is probably from 2014, and the reason it does not compile is the lesson.
• Do not send a patch before Lab 7. Not one. The list is a real place with real people, and your first impression is a real thing.
• Do not send a patch that ./scripts/checkpatch.pl complains about, and do not "fix" checkpatch complaints you do not understand.
• Do not skip the experiments. In this domain the text is a hypothesis and ftrace is the evidence.
• Do not believe any line number, function signature, or config symbol printed in this book — including in this paragraph. Run the rg/git log command next to it. The tree moved while this page was being written.
• Do not move to the next milestone until the current behavior can be inspected and explained.

Convenience layers — devm_*, module_platform_driver(), b4, virtme-ng — are introduced, but every time one is, this book first shows you what it does by hand. That ordering is never reversed.

What You Will Be Able to Do

How This Curriculum Is Organized

The concept chapters are not optional background — they are where the depth lives. A lab says "take the lock before touching the list"; the concurrency chapter is where you learn why the same code is correct in a workqueue and a deadlock in a timer callback. Treat the labs as the spine and the concept chapters as the muscle.

Begin with Overview & Prerequisites and clear the setup gate. Then read The Hitchhiker's Guide for why the kernel and its community are the way they are, spend an evening on The Warm-Up with the kernel you are already running, and work through The Kernel Mental Model — that is Milestone 0, and it contains no code — before starting Foundations. The lab rig you will build, and the companion workspace that comes with this book, are specified in The Lab Rig.