The Hitchhiker's Guide to Unix, Linux & the Patch

Nothing about kernel development makes sense until you know why it is like this. Why patches go by email in 2026. Why there is no stable driver API when every other operating system has one. Why the license cannot be changed. Why a release takes nine weeks whether or not anything is ready. Why one person's Acked-by is worth more than a hundred GitHub stars.

None of it is arbitrary, and none of it is nostalgia. Every one of those is a decision someone made for a reason, usually after the alternative was tried and failed. This chapter is that record.

Read it once for the story. Come back to it when the community seems to be doing something inexplicable, because it never is.

1969–1983: Unix, and the Thing That Was Missing

Unix was written at Bell Labs starting in 1969, and rewritten in C in 1973 — which is the fact that mattered, because it made the operating system portable and made the source readable. AT&T was barred by a 1956 antitrust consent decree from selling software, so it licensed Unix to universities for a nominal fee, with source.

An entire generation learned operating systems by reading the actual code of a real one. That generation wrote BSD at Berkeley, wrote the papers, and taught the courses.

Then in 1984 AT&T was broken up, the consent decree no longer applied, and Unix became a commercial product with a source license priced for corporations. The teaching material vanished behind a paywall, mid-decade, for everyone.

Two responses:

So in 1991 there was a free userland with no kernel, and a readable kernel you were not free to change. The shape of the hole was very specific.

1991: The Announcement

On 25 August 1991 a student in Helsinki posted to comp.os.minix:

"I'm doing a (free) operating system (just a hobby, won't be big and professional like gnu) for 386(486) AT clones."

Version 0.01 appeared in September. It was not under the GPL at first — the original license forbade commercial distribution. In February 1992, version 0.12 switched to the GNU GPL version 2, and Linus has since called that the single best decision he made. The free userland now had a kernel and the kernel had a license that guaranteed it would stay free.

Three properties of that origin still govern the project:

1. It was a hobby project that got contributions, not a designed system with a plan. The architecture is what survived, not what was specified. This is why you will find, in the same tree, code of extraordinary sophistication next to a driver someone wrote in 1998 for hardware nobody owns.

2. It was monolithic on purpose, and stayed that way after an argument. In 1992 Andrew Tanenbaum opened a Usenet thread titled "LINUX is obsolete", arguing that monolithic kernels were a dead end and microkernels were the future. The debate is still worth reading. Linux won on engineering pragmatics: in-kernel calls are function calls, and message-passing overhead is real. The consequence you inherit is that everything in the kernel shares one address space and one privilege level — which is why a bug in a sound driver can corrupt a filesystem, and why review is as strict as it is.

3. It was distributed by patch from day one. There was no shared repository. People mailed diffs to Linus, who applied them by hand. That is not a historical footnote — it is the origin of the entire contribution workflow, and it is section 4 of this chapter.

1991–2005: The Patch Flow, and Why Git Exists

For fourteen years the workflow was: produce a unified diff with diff -u, mail it to Linus or a subsystem maintainer, and hope. Everything about the modern process is a load-bearing fossil of that arrangement.

The scaling problem became acute around 2002, when the patch volume exceeded what one person could apply by hand. Linus adopted BitKeeper, a proprietary distributed VCS whose vendor granted the kernel a free license. It worked, and it was politically radioactive — a free kernel developed with proprietary tools.

In April 2005 the BitKeeper license was withdrawn after a developer reverse-engineered its protocol. The kernel had no version control, and the largest software project in the world had roughly two weeks of runway.

Linus wrote git. The first commit was 7 April 2005; it was self-hosting within days and managing the kernel within a month.

Note: This is why git looks the way it does. It was not designed as a general-purpose VCS with a nice interface — it was designed to be a fast, distributed, cryptographically-verifiable content store for a project with thousands of contributors and a hierarchy of trusted maintainers. Every feature you find strange (the index, the plumbing/porcelain split, the emphasis on rewriting local history) is that requirement showing through. The kernel is not a git project. Git is a kernel tool that escaped.

2. Why Email, Still

This is the question every newcomer asks, and it deserves a serious answer rather than "tradition."

The scale the workflow has to handle

Run these against your own clone. The numbers will be larger than you expect.

git log --since="1 year ago" --oneline | wc -l           # commits merged in a year
git log --since="1 year ago" --format='%aN' | sort -u | wc -l   # distinct authors
git log --since="1 year ago" --format='%cN' | sort | uniq -c | sort -rn | head -20  # committers
grep -c '^M:' MAINTAINERS                                 # maintainer entries

A mainline kernel merges on the order of 70,000–80,000 commits a year from ~4,000 developers, through roughly 200 subsystem trees, with no company able to overrule any other. Any proposed replacement workflow has to handle that, and the ones people usually suggest do not.

What email actually buys

The last row is stronger than it looks. When Linus pulls from a maintainer, he is pulling a signed tag from a tree he has decided to trust, and that maintainer is vouching for every Signed-off-by beneath. The chain is cryptographic at the top and social all the way down.

The honest costs

This book will not pretend the workflow is friendly.

The community's answer has been tooling, not migration:

Tip: Read Documentation/process/email-clients.rst before you configure anything. Then use git send-email anyway, which is what everyone actually does, and verify by sending a patch to yourself and running git am on the result. If it applies, your setup is correct. If it does not, your mail path is corrupting the patch and it would have corrupted it on the list too.

3. Why There Is No Stable Internal API

Every other mainstream operating system publishes a stable driver API. Windows drivers written for one release load on the next. Linux does the opposite, deliberately, and it is written down: Documentation/process/stable-api-nonsense.rst.

$EDITOR Documentation/process/stable-api-nonsense.rst   # read it in your own tree

The argument, compressed:

1. A frozen interface freezes its mistakes. Every API embeds assumptions about the hardware and the problem. Hardware changes. If the interface cannot change, you accumulate a permanent layer of workarounds around an interface nobody would design today. Linux instead changes the interface and fixes every caller in the same commit or series.

2. Fixing every caller is possible here and impossible elsewhere, because all the callers are in one tree. This is the core economic asymmetry:

  WITH a stable API                        WITHOUT one, but with one tree
  ─────────────────                        ──────────────────────────────
  interface is wrong                       interface is wrong
      │                                        │
  cannot change it — out-of-tree           change it, and mechanically fix
  code would break                         all 400 callers in one series
      │                                        │
  add a v2 alongside v1                    tree is consistent; the old
      │                                    interface no longer exists
  now maintain both, forever                   │
      │                                    out-of-tree code breaks — and
  ...and eventually a v3                   that is its owner's problem

You can watch this happen. Pick any tree-wide conversion and read it:

git log --oneline --grep="convert.*to folio" -- mm/ | head -20
git log --oneline -- Documentation/process/stable-api-nonsense.rst
git log --oneline --grep="BKL" | tail -20      # the multi-year big-kernel-lock removal

3. It makes upstreaming the rational choice. Out-of-tree code must be re-fixed every release, by you, forever. In-tree code gets fixed by whoever changes the interface, as part of their patch. The absence of a stable API is not hostility toward out-of-tree drivers — it is a price on staying out of tree, and the price is deliberately high.

4. It is also a license mechanism. EXPORT_SYMBOL_GPL() marks a symbol as available only to modules declaring a GPL-compatible MODULE_LICENSE(). Loading a non-GPL module sets a taint flag that appears in every subsequent oops report, and maintainers will not debug a tainted kernel.

rg -c "EXPORT_SYMBOL_GPL" kernel/ mm/ | head
cat /proc/sys/kernel/tainted        # 0 on a clean system; decode it in Documentation/admin-guide/tainted-kernels.rst
rg -n "MODULE_LICENSE" samples/ | head -5

Warning: The corollary lands on you in Lab 2. Your out-of-tree module will stop compiling one day because someone changed a function you call. That is the system working as designed. The fix is always the same: git log -S'<function>' to find the commit, read its message to learn the new way, update your call. Do it once and the whole philosophy stops being abstract.

4. Why the License Cannot Change

Linux is GPL-2.0-only. Not "GPLv2 or later" — Linus removed the "or any later version" clause explicitly, so that no future FSF decision could change the kernel's terms. This is why Linux is not and will never be GPLv3.

There is one carve-out, and it is the reason your proprietary software runs on Linux at all:

The syscall note says, in effect: using the kernel's system-call interface does not make your program a derivative work. Without it, every binary on every Linux system would arguably be a derivative of the kernel. With it, the boundary between "kernel" and "userspace" is also the boundary of the copyleft.

head -20 COPYING                                    # the note itself
ls LICENSES/                                        # every license text used in the tree
head -1 kernel/sched/core.c                         # the SPDX identifier
$EDITOR Documentation/process/license-rules.rst     # the rules, in full
rg -c "SPDX-License-Identifier" --type c | head -3

Since roughly 2017 every file carries a machine-readable SPDX identifier on its first line, which replaced tens of thousands of hand-written license boilerplate blocks. Your new file needs one, and checkpatch.pl will tell you so.

Why it cannot be relicensed: the kernel has tens of thousands of copyright holders, many unreachable and some deceased. Relicensing requires all of them to agree. This is not a bug; it is the same ratchet that keeps the code free — the mechanism that makes it impossible to close is the mechanism that makes it impossible to change at all.

5. The Release Cycle

Linux ships on a clock, not on a feature list. The clock has not meaningfully changed since 2005.

  day 0        vX.Y released ─────────── and the MERGE WINDOW opens the same day
   │
   │  ~2 WEEKS: Linus pulls from subsystem maintainers.
   │            ONLY material that was already in linux-next is eligible.
   │            Roughly 10,000–14,000 commits land in these two weeks.
   ▼
  day 14       -rc1  ── the merge window CLOSES. No new features after this point.
   │
   │  ~6–7 WEEKS of stabilization, one -rc per week:
   │            -rc2  -rc3  -rc4  -rc5  -rc6  -rc7  [-rc8 if Linus is unhappy]
   │            Fixes only. The bar rises every week.
   ▼
  day ~63–70   vX.Y+1 released ───────── and the next merge window opens. Repeat.

Nine to ten weeks, roughly six releases a year. Check where you are right now:

git describe --tags                     # e.g. v6.x-rc4-123-gabcdef01
git log --oneline --tags --simplify-by-decoration -12   # the recent tag cadence

What this means for your patch

The cycle position is the single biggest predictor of how your patch is received.

And the arithmetic that governs the capstone: a feature patch sent at -rc3 lands in a maintainer's tree at -rc5, appears in linux-next shortly after, is pulled during the next merge window, and appears in a tagged release two to three months after you sent it.

Note: linux-next is the gate you cannot skip. Stephen Rothwell has published a daily integration of ~200 subsystem trees since 2008, and Linus's rule is that material should have had exposure there before the merge window. "This wasn't in linux-next" is a complete and sufficient reason to defer a pull request by an entire cycle.

6. The Maintainer Model

There is no committer list and no permission bit. There is a chain of trust, and it is implemented in a text file.

                        Linus Torvalds
                        pulls signed tags from ~100–200 people
                              │
        ┌─────────────────────┼─────────────────────┐
        ▼                     ▼                     ▼
   net (Jakub/Paolo/…)   mm (Andrew Morton)    drm (Dave/Simona)
   tree: net-next        tree: mm-stable       tree: drm-next
        │                     │                     │
        ▼                     ▼                     ▼
   drivers/net/ethernet/…  memcg, slab, …      i915, amdgpu, msm, …
   sub-maintainers         sub-maintainers     group maintainership
        │                     │                     │
        └──────────── individual contributors ──────┘
                        that is you: you send patches,
                        someone with a tree applies them

MAINTAINERS is the index, and it is a real database, not documentation:

./scripts/get_maintainer.pl -f mm/page_alloc.c
./scripts/get_maintainer.pl --scm -f drivers/gpu/drm/vkms/
head -80 MAINTAINERS                 # the field-letter legend lives at the top
tail -12 MAINTAINERS                 # "THE REST" — Linus, the catch-all

The field letters you will use constantly:

Warning: Subsystems have genuinely different cultures, and applying one's norms to another is the most common avoidable mistake. netdev runs a strict Patchwork queue, closes net-next during the merge window, and wants you to say which tree you targeted in the subject. The tip tree (x86, scheduler, timers) has exacting commit-message standards and will send you back for prose. DRM uses group maintainership with a shared commit right. Check for a profile before assuming: ls Documentation/process/maintainer-*.rst

Signed-off-by is a legal statement

It is not a signature line or a courtesy. It certifies the Developer's Certificate of Origin — a short statement that you wrote the patch, or received it under a compatible license, and that you understand it will be redistributed under the project's license and recorded permanently.

It came out of the 2004 SCO litigation, when the project needed a provenance chain for every line. Read the exact text once, in your tree — you are asserting it on every patch you ever send:

rg -n -A 30 "Developer's Certificate of Origin" Documentation/process/submitting-patches.rst

7. The Two Rules

Everything above is process. These two are the actual constitution, and they pull in opposite directions on purpose.

The second is the stronger rule in the entire project. It means: if a program worked on an older kernel and stops working on a newer one, the kernel is wrong — even if the program was relying on a bug, even if the old behavior was clearly unintended, even if fixing it is ugly.

The consequence for you is one sentence long: uapi is forever. An ioctl number you add today, with a struct whose padding you got wrong, is a maintenance burden for the rest of the project's life. This is why Lab 5 spends more time on the ABI than on the code, and why the honest answer to "should I add a syscall?" is almost always no.

ls include/uapi/linux/ | head -20
$EDITOR Documentation/process/adding-syscalls.rst
rg -n "we do not break user" Documentation/ | head

The Layer Cake, Annotated with Dates

 ┌────────────────────────────────────────────────────────────────────────┐
 │ YOUR PATCH                                                     (today) │
 ├────────────────────────────────────────────────────────────────────────┤
 │ b4, Patchwork, lore, syzbot, 0-day, KernelCI      the tooling  (2008→) │
 ├────────────────────────────────────────────────────────────────────────┤
 │ THE EMAIL WORKFLOW    plain-text patches to a list              (1991) │
 │                       Signed-off-by / the DCO                   (2004) │
 ├────────────────────────────────────────────────────────────────────────┤
 │ GIT                   written in 10 days for exactly this       (2005) │
 ├────────────────────────────────────────────────────────────────────────┤
 │ THE RELEASE CYCLE     merge window + weekly -rc                 (2005) │
 │                       linux-next as the integration gate        (2008) │
 ├────────────────────────────────────────────────────────────────────────┤
 │ THE MAINTAINER MODEL  MAINTAINERS, a chain of pulls             (1990s) │
 ├────────────────────────────────────────────────────────────────────────┤
 │ THE LICENSE           GPL-2.0-only + Linux-syscall-note         (1992) │
 │                       SPDX identifiers tree-wide                (2017) │
 ├────────────────────────────────────────────────────────────────────────┤
 │ THE ARCHITECTURE      monolithic, one address space             (1991) │
 │                       no stable internal API                    (1991) │
 │                       "we do not break userspace"               (1991) │
 ├────────────────────────────────────────────────────────────────────────┤
 │ THE SUBSTRATE         Unix: files, processes, fds, signals      (1969) │
 │                       written in C                              (1973) │
 └────────────────────────────────────────────────────────────────────────┘

Every layer is still load-bearing. Not legacy to be cleaned up — structure. The unified diff format from the 1980s is the interchange format for a project that merged 80,000 commits last year, and it works.

Twelve "Why Is It Like This?" Answers

Keep these. They answer most of what people ask.

The Documents to Actually Read

They are in your tree. That is the point — they ship with the code they describe, so they cannot go stale the way a blog post does. Full annotated list in Primary Sources.

Warning: Prefer in-tree Documentation/ over any external source, including this book. It ships with the code you have checked out, it is reviewed as part of the same patch flow, and when it disagrees with a tutorial you found, the tutorial is wrong. When it disagrees with the code, that is a documentation bug — and fixing one is an excellent first patch.

Validation / Self-check

1. What specific gap existed in 1991 that Linux filled, and what were the two failed attempts to fill it?
2. Why is the unified diff still the interchange format for kernel development?
3. Give three concrete properties of the email workflow that a centralized forge does not provide, and one real cost of it.
4. Explain, in terms of who pays, why a frozen internal API would be more expensive than the current arrangement.
5. What does EXPORT_SYMBOL_GPL do, and what happens to an oops report after a non-GPL module loads?
6. Why can Linux never move to GPLv3? Name both reasons.
7. What is the Linux-syscall-note and what would be true without it?
8. Draw the release cycle with its durations. When in the cycle should you send a new feature, and why?
9. What is linux-next, who runs it, and what happens to a pull request whose contents never appeared there?
10. Describe the chain of trust from your patch to a released kernel. How many people must act, and what does each of them assert?
11. State both governing rules. Give one change each rule allows and one it forbids that look superficially similar.
12. Why is Signed-off-by a legal statement rather than a signature, and where is its exact text?

Next: The Warm-Up — an evening with the kernel you are already running, before you build one of your own.