The Teaching Method

This chapter describes the shape of every concept chapter and every lab in this curriculum. Read it once so the structure of what follows is not a surprise, and come back to it when you are designing your own experiments — because by the end, designing your own is the actual skill.

The Phase Template

Every unit of work here — every concept chapter, every lab — follows this order. The order is not decorative; each step exists because skipping it produces a specific failure.

The Six-Part Concept Treatment

Every concept chapter in this curriculum presents each concept in exactly six parts. When you see this shape, it is deliberate; when you write your own notes on a new subsystem, copy it.

┌─────────────────────────────────────────────────────────────────────────┐
│ 1.  WHAT PROBLEM IT SOLVES                                              │
│       The thing that would be impossible or wrong without it.           │
│       Not "what it is" — what it is FOR.                                │
│                                                                         │
│ 2.  WHERE IT LIVES IN THE KERNEL                                        │
│       Which directory, which layer, which side of the boundary.         │
│       Always with the `rg` / `find` command that shows you.             │
│                                                                         │
│ 3.  WHO OWNS OR INTERACTS WITH IT                                       │
│       Which subsystem owns it, who calls in, who it calls out to,       │
│       what refcounts or lifetimes are involved. Usually a table.        │
│                                                                         │
│ 4.  THE STRUCTURES, SYSCALLS, AND CODE PATHS                            │
│       The actual nouns and verbs: struct names, ops tables, the         │
│       function that is the entry point, the flags that change behavior. │
│                                                                         │
│ 5.  AN EXPERIMENT THAT DEMONSTRATES IT                                  │
│       Something you run on YOUR kernel that makes the concept visible:  │
│       ftrace, /proc, /sys, bpftrace, a printk, GDB. With a prediction.  │
│                                                                         │
│ 6.  THE FAILURE MODE WHEN IT IS DONE WRONG                              │
│       The symptom, the message, the class of bug. Usually a table of    │
│       "mistake → what you will see". This is the part you will reread.  │
└─────────────────────────────────────────────────────────────────────────┘

Part 6 is the one that pays off years later. When you hit BUG: scheduling while atomic or a lockdep splat at work, what you will retrieve is not the explanation — it is the failure-mode table.

Note: Part 5 is non-negotiable and it is why this curriculum insists on the lab rig. Every assertion about kernel behavior in this book is paired with a way to observe it. If a chapter ever tells you something is true without telling you how to see it, that is a defect in the chapter.

The Eleven-Part Lab Template

Every lab has these eleven parts, in this order. When a lab omits one, it is because the step genuinely has none — not as a shortcut.

┌──────────────────────────────────────────────────────────────────────────┐
│  1.  Background                    what this is and why it exists        │
│  2.  Why This Lab Matters          the contributor-facing reason         │
│  3.  Prerequisites                 what must already work                │
│  4.  Predict First                 written predictions, BEFORE step 5    │
│  5.  Step-by-Step Tasks            real commands and real code, with a   │
│                                    target-architecture diagram and a     │
│                                    deliverables checklist                │
│  6.  Expected Output               so you know whether it worked         │
│  7.  Debugging Steps               symptom → cause, for when it did not  │
│  8.  One Experiment                a claim you could disprove            │
│  9.  One Test                      KUnit, kselftest, or a reproducer     │
│ 10.  One Challenge Extension       past the lesson                       │
│ 11.  Validation / Self-check       gating questions, answered unaided    │
└──────────────────────────────────────────────────────────────────────────┘

Two properties of that ordering matter more than the list itself.

Part 4 comes before part 5. You predict before you build, not after. The gap is the lesson.

Parts 8, 9, and 10 are not optional extras. A lab you finished through part 7 is a lab where you typed something in. The experiment is where you find out whether your model is right; the test is what keeps it right; the challenge is where the depth compounds.

Warning: Avoid large unexplained code dumps — including your own. If a code block in this curriculum runs past about 80 lines, it is followed by a walkthrough of the parts that carry meaning. If you find yourself copying a block you cannot annotate line by line, stop and annotate it. The annotation is the exercise. In kernel code this is not a study tip: an unannotated block is where the missing spin_unlock on the error path lives.

The Predict-First Protocol

Throughout the curriculum you are asked to predict behavior before revealing the result. This is not a gimmick. Prediction converts a passive read into a test of your model, and the gap between prediction and observation is where learning happens. A confirmed prediction teaches you almost nothing; a wrong one teaches you exactly which belief was false.

The protocol:

1. Read the question.
2. Write your prediction down — in a file, in a comment, on paper. Writing is required. A prediction you kept in your head will silently rewrite itself when you see the answer. This is hindsight bias and you are not immune to it.
3. Include your confidence: high / medium / guessing.
4. Run the experiment.
5. If you were wrong, write one sentence naming the false belief. Not "I forgot" — the actual belief.

Keep these in predictions.md in your workspace. At the capstone you review it.

Example questions you will be asked

• Your module calls kmalloc(GFP_KERNEL) while holding a spinlock. What happens — always, sometimes, or never?
• You dereference a user pointer directly. What happens for a valid pointer, for NULL, and for a kernel address?
• Your driver works on your 4-CPU VM and corrupts a list on a 96-CPU server. Which three causes are most likely, in order?
• A use-after-free runs ten times without KASAN. How many of those ten show any symptom at all?
• You add a field to the middle of a struct in a uapi header and rebuild only the kernel. What does the old userspace binary do?
• You set a breakpoint in the network receive path and leave the guest stopped for two minutes. What happens when you continue?
• Your patch builds with defconfig. Name two configurations under which it will not.

Warning: The four predictions most people get wrong on first encounter are: which functions can sleep, what a use-after-free looks like without KASAN, how much of a stack trace to trust, and how large your measurement noise is. If you predict all four correctly, you may move faster through Foundations and Engineering.

What "Instrumentation" Means Here

Instrumentation is not printk scattered in a loop. It is a deliberate, switchable view of one layer, and the kernel gives you an unusually good set of them.

Design rules for kernel instrumentation, learned the hard way:

1. printk is not free and it is not neutral. It can be slow, it serializes on the console, and at high rates it changes the timing of the bug you are chasing — sometimes making it disappear. Use pr_debug (off by default, switchable at runtime) or a tracepoint. Use printk_ratelimited() for anything in a path that can repeat.
2. Never printk from a path the console itself uses. You will deadlock or recurse, and the output that would have told you why is the thing that is broken.
3. It must be switchable without a rebuild. A rebuild is two minutes you spend fifty times. Dynamic debug, ftrace filters, and bpftrace all cost zero rebuilds.
4. Prefer tracepoints for anything permanent. They are zero-cost when off, they have a stable format that tooling can consume, and adding one is a legitimate upstream contribution.
5. Ask the tooling before you instrument. KASAN, lockdep, and dmesg may already know the answer. The instinct to reach for printk first is the instinct this curriculum is trying to replace.

What "One Experiment" Means Here

An experiment has four parts, and it is not an experiment without all four:

CLAIM        A falsifiable statement about the system.
             "A spinlock held across kmalloc(GFP_KERNEL) is detected by
              CONFIG_DEBUG_ATOMIC_SLEEP before it deadlocks."

METHOD       The exact commands, in order, that would show it.
             "Build lab-fast. Load a module that takes a spinlock and then
              calls kmalloc(GFP_KERNEL). Read dmesg."

PREDICTION   What you expect to observe, written before you run it.
             Include: will it BUG, WARN, hang, or appear to work?

RESULT       What you observed, and — if it differs — which belief was wrong.

Every experiment in this book is written in that shape. When you invent your own, keep it. The habit transfers: this is how you will debug a kernel bug in production five years from now, and it is also what a maintainer means when they ask "how did you test this?"

Tip: Kernel experiments have one extra requirement that userspace experiments do not: state the configuration. "It worked" is meaningless without the config, the CPU count, the preemption model, and whether the debug options were on. Record uname -a, nproc, and the CONFIG_ symbols that matter, next to the result. Every result in this curriculum that you cannot reproduce later will be missing one of those four.

What "One Test" Means Here

The kernel has two test frameworks and they are for different things. Choosing wrongly is a common review comment.

The four properties every test in this curriculum has:

1. It names what it is testing in a comment, including the rule. Six months later assert(ret == -EINVAL) is unreadable; /* unknown flag bits must be rejected so the flags field stays extensible */ is not.
2. It asserts one thing. A test that checks the return value and the side effect and the refcount tells you nothing when it fails.
3. Its failure message states the rule, not just the numbers.
4. It fails when you reintroduce the bug. Verify this. A test you have never seen fail is a test you have not written.

For every kernel interface you implement, this book asks for six things:

1. The happy path works.
2. Every error path returns the right errno — and you exercised each one.
3. The behavior under concurrency: two callers at once, with PROVE_LOCKING on.
4. The behavior under allocation failure: use CONFIG_FAILSLAB / fault injection.
5. The behavior at the boundaries: zero, one, maximum, off-by-one, NULL.
6. What happens when the module is unloaded while in use.

Item 6 is the one people skip, and it is the one that produces a use-after-free in a real user's kernel a year later.

What "One Challenge Extension" Means Here

Each lab ends with something past the lesson: a feature the lab did not need, a correctness edge the happy path avoided, or the thing a maintainer would ask about. Challenges are optional; they are also where the depth compounds. A representative sample:

• Make your char device's read() correct when two processes read concurrently at different offsets.
• Convert your module's manual cleanup to devres and explain what changed about the failure paths.
• Add a tracepoint to your driver and consume it with bpftrace.
• Make your module survive rmmod while a userspace process has the device open. (This is genuinely hard. That is the point.)
• Run your module under CONFIG_FAILSLAB with a 10% failure rate and fix everything that breaks.
• Build your module for a different architecture and find the endianness or alignment assumption you did not know you had made.

Why This Book Never Cites a Line Number

You will not find kernel/sched/fair.c:4212 anywhere in this curriculum. This is a rule, not an oversight, and it has a reason you should adopt for your own notes.

The tree merges on the order of 1,500 commits a week. A line number is wrong within days. A function name is wrong within releases. What is stable is the role a thing plays — "the function that picks the next task in the fair class" — and the command that finds it today.

So every reference in this book has this shape:

# NOT: "see pick_next_task_fair() at kernel/sched/fair.c:8123"
# BUT:
rg -n "pick_next_task_fair" kernel/sched/          # where is it now?
git log --oneline -S'pick_next_task_fair' | head   # when did it change, and why?

Adopt this in your own notes, your own commit messages, and your own review comments. A review comment that says "line 412" is unreadable in v2; one that quotes the line is readable forever. This is also why kernel review happens by quoting inline: the quoted text is the anchor, because line numbers move.

Warning: The corollary applies to this book. Any struct name, function signature, config symbol, or directory named here may have changed. Run the command next to it. If the command finds nothing, the tree moved — use git log -S to find out where it went, and treat that as a free exercise in the skill you are here to learn.

Common Failure Modes of Learners (Not of Code)

How to Ask Yourself a Debugging Question

When something is wrong, ask in this order. This ordering is the single most valuable transferable skill in the curriculum.

 1. WHAT DOES THE KERNEL ALREADY KNOW?
    dmesg first. A KASAN report, a lockdep splat, or a WARN has already
    done the analysis for you — and people skip it constantly.
    → dmesg -T --level=err,warn | tail -50

 2. WHICH CONTEXT AM I IN?
    Process? Softirq? Holding a spinlock? Almost every "impossible" kernel
    bug is a context violation.
    → in_task(), in_atomic(), preempt_count(); CONFIG_DEBUG_ATOMIC_SLEEP

 3. WHICH SIDE OF THE BOUNDARY?
    Is the bad value coming from user space? Then it is untrusted, and the
    bug is a missing validation, not a wrong computation.
    → check every copy_from_user return; check every bound

 4. WHO ELSE CAN TOUCH THIS DATA RIGHT NOW?
    Another CPU? An interrupt? A timer? A workqueue? If the answer is
    "nothing", prove it — do not assume it.
    → lockdep; -smp 8; CONFIG_DEBUG_SPINLOCK

 5. IS IT DETERMINISTIC?
    Reproduces every time  → logic. Read the code.
    Reproduces sometimes   → concurrency, timing, or memory corruption.
                             Reach for KASAN and lockdep, not printk.

 6. IS IT MY BUG, OR MY CONFIG?
    → Try the other config. Try defconfig. Try a released tag.
      A bug that vanishes under defconfig is telling you something.

 7. HAS SOMEONE ELSE ALREADY FIXED IT?
    → git log --oneline --since="1 year ago" -- <the file>
      lore.kernel.org search for the function name
      This is the step that saves whole days, and it costs two minutes.

Validation / Self-check

1. Name the six parts of the concept treatment, and say which one you will still be using in five years.
2. Name the eleven parts of the lab template. Which three come after the code, and why does the ordering matter?
3. What are the four parts of an experiment, and which one do people skip?
4. Why must a prediction be written rather than held in your head?
5. What extra thing must a kernel experiment record that a userspace experiment need not, and name its four components.
6. When should a test be a KUnit test and when a kselftest? Give one example of each from your own work.
7. Name the six things this book asks for with every kernel interface you implement. Which one do people skip, and what does it cause a year later?
8. Give five rules for kernel instrumentation, and say which one printk in an interrupt handler violates.
9. Why does this book never cite line numbers? Give the rule you should apply in your own review comments as a result.
10. Give the seven-question debugging order. Now apply it to: "my module works on my laptop and corrupts a list on the CI machine."

Next: Foundations — the floor. Seven concepts and six labs, and by the end of it you can write, load, debug, and test kernel code.