The Roadmap: Fifteen Milestones

Every milestone has a goal, an observable behavior, concrete completion criteria, an experiment, and a checkpoint question. You do not advance until the criteria are met and you can answer the checkpoint question without notes.

The rule that governs the whole sequence:

Do not move to the next milestone until the current behavior can be inspected and explained.

"It works" is not a completion criterion. "I can show you, at any layer, exactly what is happening and why" is. In kernel work this is not pedantry: code that works for reasons you cannot state is code that will fail on a machine with a different CPU count, a different preemption model, or an interrupt at a different moment.

The Sequence at a Glance

flowchart TD
    M0["M0 Mental model"] --> M1["M1 Build & boot"]
    M1 --> M2["M2 First module"]
    M2 --> M3["M3 Char device"]
    M3 --> M4["M4 Debugging"]
    M4 --> M5["M5 A syscall"]
    M4 --> M6["M6 Testing"]
    M5 --> M7["M7 Read a subsystem"]
    M6 --> M7
    M7 --> M8["M8 First patch sent"]
    M8 --> M9["M9 A patch series"]
    M8 --> M10["M10 Review a patch"]
    M7 --> M11["M11 Subsystem depth"]
    M9 --> M12["M12 Regression & perf"]
    M11 --> M12
    M10 --> M13["M13 In a maintainer tree"]
    M12 --> M13
    M13 --> M14["M14 Merged to mainline"]

Note: M13 and M14 are the only milestones whose timing is not yours. See the weekly plan. Start M8 early — the clock on the community's side is measured in weeks, and it runs in parallel with everything else you are doing.

Milestone 0 — Kernel Mental Model

Goal. Explain the boundary, the contexts, and the tree without code.

Observable behavior. None. This is the one milestone with no program.

Completion criteria.

• You can name all four ways control enters the kernel, and for each, the context it produces and whether it may sleep.
• You can draw a process's address space with the kernel in it, and say why the kernel is mapped into every process.
• You can state the context rules from memory: which of process context, spinlock-held, softirq, hardirq, and NMI may sleep, take a mutex, or call kmalloc(GFP_KERNEL).
• answers-m0.md exists, contains your answers to the fifteen questions in the mental model, and is committed.
• You can name the purpose of at least fifteen top-level directories in the tree.

Experiment. From the warm-up: enable the sched_switch tracepoint for 200 ms and read the output. For each line, say which CPU, which task left, in what state, and which took over. Explain every column before reading further.

Checkpoint question. You are inside a syscall handler and you take a spinlock. Which of these are now forbidden, and why: kmalloc(GFP_KERNEL), mutex_lock(), copy_to_user(), printk()?

Milestone 1 — Build, Boot, and Debug

Goal. Compile a kernel, boot it under QEMU, and stop it in a debugger.

Observable behavior. A shell prompt inside a kernel you built, and a GDB session that can halt the machine on a function you chose.

Completion criteria.

• uname -a in the guest shows your build, including a LOCALVERSION you set.
• The guest has more than one CPU (nproc inside it) — a single-CPU guest hides everything.
• A one-file change rebuilds and reboots in under two minutes, measured.
• GDB attaches, lx-dmesg works, and a breakpoint on a syscall entry point fires when you trigger it from the guest shell.
• You can boot with -S and break on start_kernel.
• A deliberate NULL dereference in a test module produces a readable oops, and QEMU exits rather than reboot-looping.
• You can state what nokaslr, console=ttyS0, panic=1, -s, and -S each do.

Experiment. Boot once with nokaslr and once without. In each, break on a function and compare the address GDB reports with the address in the guest's /proc/kallsyms (as root). Predict, before running, whether the breakpoint fires in the second case. Then explain the result in terms of KASLR.

Checkpoint question. Why does GDB need vmlinux rather than bzImage, and what exactly is in one that is not in the other?

Milestone 2 — First Module

Goal. Write, load, parameterize, and unload an out-of-tree module.

Observable behavior. insmod produces your message in dmesg; lsmod lists your module; modinfo shows your metadata and parameters; rmmod succeeds and leaves nothing behind.

Completion criteria.

• module_init/module_exit, MODULE_LICENSE("GPL"), MODULE_AUTHOR, MODULE_DESCRIPTION.
• At least one module_param() with a permission mask, readable (and writable) under /sys/module/<name>/parameters/.
• The init function has a failure path that is exercised (e.g. a parameter value you reject) and unwinds everything it allocated, in reverse order.
• rmmod then insmod again works ten times in a row with no leak (/proc/slabinfo or kmemleak shows nothing accumulating).
• You can explain why the module loaded successfully at full privilege with no verification of any kind.
• checkpatch.pl --no-tree -f your_module.c is clean, and you understand each thing it flagged before you fixed it.

Experiment. Add a pr_info() in the init path that prints smp_processor_id(), current->pid, current->comm, and in_task(). Load the module from the shell, then load it from a script piped into the shell, then from a sh -c. Predict what comm will be in each case before you look.

Checkpoint question. Your module's init allocates three things and the third allocation fails. What must happen before you return, and what is the symptom on the running system if you get it wrong?

Milestone 3 — Character Device

Goal. A real device node with real file_operations, including ioctl and user-pointer handling — and a deliberate race you then find and fix.

Observable behavior. cat /dev/mylab returns data; echo x > /dev/mylab is accepted; a userspace program drives your ioctls; two concurrent writers do not corrupt your state.

Completion criteria.

• A registered major/minor (alloc_chrdev_region + cdev_add, or the misc device API) and a node that appears automatically via class_create/device_create.
• open, release, read, write, llseek, and unlocked_ioctl implemented.
• Every user pointer goes through copy_to_user/copy_from_user, with the return value checked and -EFAULT returned on partial copies.
• Your ioctl numbers use the _IO/_IOR/_IOW/_IOWR macros with a documented magic number, and the struct they pass has no implicit padding.
• A userspace exerciser in userspace/ that drives every operation, including the error paths.
• A concurrency test — two processes reading and writing simultaneously — that fails before you add locking and passes after. Run it under CONFIG_PROVE_LOCKING.
• You can say which lock you chose, and why a mutex rather than a spinlock (or the reverse).

Experiment. Remove the copy_from_user and dereference the user pointer directly. Predict what happens for (a) a valid pointer, (b) NULL, (c) a kernel address passed from userspace. Run all three. Then explain why (a) sometimes "works", and why that is more dangerous than (b).

Checkpoint question. Name three distinct things that can go wrong when kernel code dereferences a user-supplied pointer, and say which one copy_from_user handles that a range check alone would not.

Milestone 4 — Debugging a Kernel

Goal. Diagnose failures with the tools rather than by guessing.

Observable behavior. Given a broken module, you can name the bug from its output alone.

Completion criteria.

• You have produced, read, and explained line by line: an oops with a call trace, a lockdep splat, a KASAN use-after-free report, and a "sleeping function called from invalid context" BUG.
• You can decode a stack trace: scripts/decode_stacktrace.sh and scripts/faddr2line both used successfully on your own oops.
• ftrace used to answer a question you actually had: function_graph filtered to your module, plus at least one tracepoint.
• A kprobe placed on a function you did not write, printing an argument.
• CONFIG_PROVE_LOCKING catches an ABBA deadlock you wrote deliberately — before it deadlocks.
• Magic SysRq used from the QEMU monitor to dump task state on a wedged guest.
• You can state what each of KASAN, UBSAN, kmemleak, KFENCE, and lockdep catches, and roughly what each costs.

Experiment. Write a module with a use-after-free. Run it under lab-fast (no KASAN) ten times and record what happens. Then run it under lab-paranoid. Predict, before running, how many of the ten no-KASAN runs will show any symptom. This experiment is the argument for the paranoid config, and it is more convincing than any paragraph.

Checkpoint question. A user reports a crash with a stack trace containing ? __kmalloc+0x.... What does the leading ? mean, and what does that tell you about how much to trust that frame?

Milestone 5 — A Syscall and Its ABI

Goal. Add a syscall, understand the ABI you just created, and be able to argue against having added it.

Observable behavior. A userspace program calls your syscall by number and gets a result.

Completion criteria.

• The syscall is wired up for your architecture: the .tbl entry, SYSCALL_DEFINEn, and the uapi header.
• It validates every argument before use, returns proper negative errnos, and handles copy_from_user failure.
• You have written down its ABI: argument types and their sizes, struct layout with explicit padding, error codes, and what a future extension would look like.
• You can explain compat/32-bit concerns for your arguments, even if you did not implement them.
• A flags argument exists and unknown bits are rejected with -EINVAL — and you can say why that specific choice makes the syscall extensible.
• A written paragraph arguing that this should have been an ioctl, a sysfs file, a prctl(), or nothing at all — and a rebuttal.

Experiment. Change your syscall's struct by adding a field at the end, rebuild the kernel but not the userspace program, and run the old binary. Predict what happens. Then move the field to the middle and repeat. This is the ABI lesson, and it costs ten minutes.

Checkpoint question. You shipped a syscall with a struct containing three ints and a pointer. Name two ways that struct's layout differs between a 32-bit and a 64-bit userspace, and what the kernel must do about it.

Milestone 6 — Testing Like the Kernel Does

Goal. Write tests in the two frameworks the kernel actually uses, and know which is for what.

Observable behavior. ./tools/testing/kunit/kunit.py run passes your suite; make -C tools/testing/selftests runs your selftest.

Completion criteria.

• A KUnit suite for a pure-logic function, run under kunit.py (which builds and boots a UML or QEMU kernel for you).
• A kselftest that exercises your char device or syscall from user space, following the selftests harness conventions and its exit-code protocol.
• Both tests fail, visibly and informatively, when you reintroduce the bug they cover.
• You can state the difference between KUnit and kselftest in one sentence each, and say which one a given bug should be covered by.
• An allmodconfig build completes (or you can list exactly what broke and why) — this is what the 0-day bot will do to your patch.
• make C=1 (sparse) and make W=1 produce no new warnings for the files you touched.

Experiment. Introduce a deliberate __user annotation error — assign a __user pointer to a plain pointer and dereference it. Predict whether gcc complains, whether sparse complains, and whether it crashes at runtime. Run all three checks.

Checkpoint question. Your patch adds a function used only when CONFIG_FOO=y. Your build passes. Name two configurations under which it will not, and the single make target that would have found both.

Milestone 7 — Read a Subsystem Cold

Goal. Open an unfamiliar subsystem and produce a map of it, unaided, in a day.

Observable behavior. A written document another engineer could use to orient themselves.

Completion criteria. Pick a subsystem you have never read. Produce, using only the tree:

• Its MAINTAINERS entry, decoded: who, which list, which tree, what status.
• Its Documentation/ entry points, read.
• The 3–6 core data structures and a diagram of how they reference each other.
• Every *_ops structure: who fills it in, who calls through it, and what the contract is.
• One complete control path traced from its entry point to the hardware or to the return.
• Its test coverage: KUnit suites, selftests, and what is not covered.
• Six months of git log summarized: what is active, what is dead, who is doing the work.
• Three questions you could not answer from the tree alone, phrased well enough to ask on the list.

Experiment. Do the same exercise for a second subsystem, timeboxed to two hours. Compare how long each step took. The gap between the first and second is the skill you just acquired.

Checkpoint question. What is the fastest reliable way to find out whether a subsystem is actively maintained, and why is the S: line in MAINTAINERS not a sufficient answer on its own?

Milestone 8 — First Patch, Sent for Real

Goal. Send one correct, trivial, genuinely useful patch to a real mailing list.

Observable behavior. Your patch appears on lore.kernel.org with a message-ID you can link to.

Completion criteria.

• The change is real: a documentation fix that corrects an actual error, a Fixes:-worthy typo in a user-visible string, a dead-code removal you verified. Not a whitespace change; not a checkpatch cleanup in code you have not read.
• ./scripts/get_maintainer.pl used, and everyone it named is on the To:/Cc:.
• The commit message explains why, in the imperative mood, wrapped at 72 columns, with a subject of the form subsystem: short description.
• Signed-off-by: present, and you have read the DCO text it certifies.
• checkpatch.pl --strict -g HEAD clean.
• The patch applies to the right tree — the one the MAINTAINERS T: line names.
• Sent with git send-email (or b4 send), plain text, correctly threaded, and it survives a round trip: you mailed it to yourself and git am applied it cleanly.
• Sent at a sensible point in the cycle (see the release cycle).

Experiment. Before sending, run b4 am on someone else's recent series from lore and apply it to your tree. Read how the trailers were collected. Then look at your own patch as it appears on lore after you send it, and compare.

Checkpoint question. You sent a patch and there has been no reply for two weeks. What are the four possible reasons, and what is the correct action for each?

Milestone 9 — A Patch Series

Goal. Split one change into an ordered series that is bisectable at every step.

Observable behavior. A [PATCH 0/N] cover letter and N patches, correctly threaded, that build and boot at every intermediate commit.

Completion criteria.

• Each patch is one logical change with its own justification.
• The tree builds and boots after every patch in the series — verified, not assumed: git rebase --exec 'make -j$(nproc)' <base>.
• A cover letter that states the problem, the approach, what was tested, and what was not.
• No patch depends on a later one. Refactors come before the change they enable.
• A v2 produced after review, with a changelog under the --- line, In-Reply-To the v1 thread, and Reviewed-by: trailers collected from v1 correctly.
• git range-diff v1..v2 used to check that v2 contains exactly the changes you claim.

Experiment. Take your series and deliberately reorder two patches so the tree does not build in the middle. Then run git bisect on a bug introduced by the last patch. Observe how the broken intermediate commit corrupts the bisection. This is why bisectability is a hard rule and not a style preference.

Checkpoint question. Why must a series be bisectable? Name the specific tool that breaks, and the specific person it inconveniences, when it is not.

Milestone 10 — Review Someone Else's Patch

Goal. Provide useful public review of a patch you did not write.

Observable behavior. Your review is in the thread on lore, and it says something the author acted on.

Completion criteria.

• You picked a patch in an area you actually understand — ideally the one from M7 or M11.
• You applied and built it, and said so. A review that did not compile the patch says so explicitly.
• Your reply quotes the specific lines it is about, inline, below the quoted text, trimmed.
• Your comments are about correctness, locking, error paths, or the ABI — not style that checkpatch already covers.
• You used the right trailer, and you can define each: Reviewed-by:, Acked-by:, Tested-by:. You did not use one you had not earned the right to give.
• Where you were unsure, you asked a question rather than asserting.

Experiment. Before writing your review, read the last twenty reviews on the same list from the subsystem's maintainer. Note the length, the tone, and what they choose to comment on versus let pass. Then write yours. The calibration is the exercise.

Checkpoint question. What is the difference between Acked-by and Reviewed-by, who is entitled to give each, and what is a maintainer entitled to assume when they see one?

Milestone 11 — Depth in One Subsystem

Goal. Know one of the eight domains well enough to have an opinion a maintainer would engage with.

Observable behavior. You can predict, before running it, what an experiment in that subsystem will show — and you are usually right.

Completion criteria. For your chosen subsystem:

• All of its concept chapters read, with the validation questions answered.
• Its labs completed, including the experiments.
• You are subscribed to its list (or reading it on lore) and can name three threads from the last month.
• You can name its five most active contributors and what each of them works on.
• You have found and written down three plausible contribution targets: an open bug, a TODO with a real reason behind it, or a documentation gap you can prove is a gap.
• You have traced one full control path end to end with ftrace or GDB, and your written trace matches what the code says.

Experiment. Take a claim from this book's chapter on your subsystem and try to falsify it with an experiment on your own kernel. Whether you succeed or not, you now know that claim in a way reading cannot produce. If you do falsify one, the book has a bug — which is worth reporting.

Checkpoint question. For your subsystem: what is the one data structure that, if you did not understand it, would make everything else unreadable? Defend the choice.

Milestone 12 — Regression and Performance

Goal. Find a regression by bisection, and measure a performance change honestly.

Observable behavior. A commit hash you identified as the cause of a behavior change, and a benchmark result you would defend on a mailing list.

Completion criteria.

• A full git bisect completed on a real behavior difference between two kernel versions, automated with git bisect run and a script that returns the right exit codes — including 125 for "cannot test this commit".
• You can explain what a bisection does when the tree does not build at some middle commit, and what you do about it.
• A benchmark of a kernel change with: a stated hypothesis, a control, at least five runs, reported variance, and a named source of noise you controlled for (CPU frequency scaling, thermal, ASLR, NUMA placement, other load).
• A perf record/perf report profile that supports (or contradicts) your explanation of the result.
• A flame graph, and a written statement of what it does not show.
• You have read a syzbot report end to end and can say what a reproducer is and why "no reproducer" changes how it is handled.

Experiment. Benchmark the same workload on the same kernel five times, changing nothing. Report the spread. Then benchmark it with lab-paranoid versus lab-fast. Predict both numbers first. Most people discover their measurement noise is larger than the effects they were about to claim.

Checkpoint question. You measured a 3% improvement. What must be true about your measurement before that number means anything, and what is the single most common reason such a claim is wrong?

Milestone 13 — Applied to a Maintainer's Tree

Goal. A patch of yours is picked up by a maintainer and appears in their tree.

Observable behavior. git log in the subsystem tree contains your commit, with your Signed-off-by and the maintainer's beneath it.

Completion criteria.

• The patch went through at least one round of review and you revised it.
• You can find your commit in the maintainer's tree: git fetch <subsystem-remote> && git log --author="Your Name" <branch>.
• It appears in a linux-next tag shortly after.
• You handled whatever the automated bots said about it — the 0-day/LKP bot, and any CI the subsystem runs.
• The trailers on the applied commit are correct: your Signed-off-by, any Reviewed-by you collected, a Link: to the list posting, and — if applicable — Fixes: and Cc: stable.
• You know which release it is targeted at, and why.

Experiment. Diff the commit as applied against the patch as you sent it (git range-diff or by hand). Maintainers frequently adjust the subject line, the commit message, or whitespace. Every difference is a lesson about what they wanted that you did not give them.

Checkpoint question. Your patch is in a subsystem for-next branch at -rc6. When will it appear in a released kernel, and what could still prevent that?

Milestone 14 — Merged Into Mainline

Goal. Your commit is in Linus's tree, in a tagged release.

Observable behavior.

cd ~/kernel/linux
git fetch origin && git log --author="Your Name" origin/master
git describe --contains <your commit sha>     # the first tag containing it

Completion criteria.

• The commit is in origin/master on git.kernel.org/torvalds/linux.
• git describe --contains names a release tag, not an -rc.
• You can walk the full path your patch took: which tree applied it, which pull request carried it, which merge window it went through.
• The capstone write-up is complete, and you have scored yourself against the rubric.
• You have re-answered the fifteen M0 questions and diffed against answers-m0.md.

Experiment. Find the merge commit that brought your patch into mainline (git log --merges --ancestry-path <your sha>..origin/master | tail -1) and read its message. It is a maintainer's pull request to Linus, summarizing a cycle's work — including yours. Read the whole thing.

Checkpoint question. Count the people whose action was required between you pressing send and your commit appearing in a tagged release. For each one, what did they assert by acting?

Validation / Self-check

1. For each milestone, name its single most important completion criterion.
2. Which milestones can be done in a different order, and which are strictly sequential? Which two should be run in parallel, and why?
3. Which milestone first makes each of these possible: reading a lockdep splat; sending a patch; defending a performance claim; explaining what happens on a cache miss?
4. Which milestones would be blocked if you had never built the lab rig, and which could you still do?
5. Name the three milestones whose completion date is not under your control, and what you should do about that in week one.
6. Which milestone's checkpoint question do you currently find hardest? That is where to spend the extra evening.

Next: The Weekly Learning Plan — the same sequence, on a calendar, with the parts you cannot schedule marked as such.