Lab 6: Testing Like the Kernel Does (Milestone 6)

Background

"Did you test this?" is the question every patch gets. The useful answer is not "yes" — it is a named test, in the framework the subsystem uses, that you can point at, and that you have watched fail.

The kernel has two testing frameworks for two different jobs, plus a set of static checkers and a set of robots that will run all of it on your patch whether you asked or not. This lab uses all of them on code you already wrote.

Why This Lab Matters

  • A patch with a test is reviewed differently from one without.
  • The bots will run these checks on your patch anyway. Finding out from the 0-day robot, in public, that your code does not build on allmodconfig is avoidable.
  • Choosing the wrong framework is a common review comment, and the rule is simple once you know it.
  • Watching a test fail is the only way to know it works.

Prerequisites

  • Lab 3 — you have a char device with an ioctl.
  • Lab 5 — you have a syscall (or skip its half of this lab).
  • The in-tree documentation, skimmed first:
$EDITOR ~/kernel/linux/Documentation/dev-tools/testing-overview.rst
ls ~/kernel/linux/Documentation/dev-tools/kunit/
ls ~/kernel/linux/tools/testing/selftests/ | head -20

Predict First

  1. KUnit runs your test inside a kernel. Which kernel — the one you booted, a new one it builds, or a user-mode one?
  2. A selftest cannot run because the feature is not configured in. Should it report pass, fail, or something else? What exit code?
  3. Your patch builds fine for you. Name two make invocations that would likely break it.
  4. make C=1 and make C=2 differ. How, and which do you want after a small edit?
  5. You reply to a syzbot report with a patch. How does syzbot know whether it worked?

The Two Frameworks

   KUNIT                                  KSELFTEST
   ─────                                  ─────────
   Tests kernel code IN the kernel        Tests the kernel FROM user space
   Runs in a purpose-built kernel         Runs on a booted kernel
   (UML by default, or QEMU)              (yours, or CI's)

   Use for:                               Use for:
     pure logic                             syscall behavior
     a parser, a state machine              /proc and /sys formats
     allocator arithmetic                   device semantics
     anything with no I/O                   the ABI, end to end

   Where:                                 Where:
     next to the code, foo_test.c           tools/testing/selftests/<area>/

   Runs in:                               Runs in:
     seconds                                seconds to minutes

The rule: if the thing you want to assert is a function's behavior, KUnit. If it is the kernel's behavior as user space observes it, kselftest. Almost every ABI test is a kselftest; almost every "does this helper compute the right thing" test is KUnit.

Step-by-Step Tasks

Step 1: Something worth unit-testing

Pull a piece of pure logic out of your Lab 3 driver so it can be tested in isolation. This refactoring is part of the lesson: code that is hard to unit-test is usually code doing too many things.

lab_range.h — a helper that computes the clamped read window:

/* SPDX-License-Identifier: GPL-2.0 */
#ifndef _LAB_RANGE_H
#define _LAB_RANGE_H

#include <linux/types.h>

/*
 * How many bytes may be read at @pos from a buffer holding @len valid
 * bytes, when the caller asked for @want?
 *
 * Returns 0 at or past the end (EOF). Never returns more than @want.
 */
static inline size_t lab_read_len(loff_t pos, size_t len, size_t want)
{
	if (pos < 0 || (u64)pos >= len)
		return 0;
	return min_t(size_t, want, len - (size_t)pos);
}

#endif /* _LAB_RANGE_H */

Step 2: A KUnit suite

lab_range_test.c:

// SPDX-License-Identifier: GPL-2.0
#include <kunit/test.h>
#include "lab_range.h"

static void lab_read_len_normal(struct kunit *test)
{
	/* Reading 4 bytes from offset 0 of a 10-byte buffer gives 4. */
	KUNIT_EXPECT_EQ(test, lab_read_len(0, 10, 4), 4);
	KUNIT_EXPECT_EQ(test, lab_read_len(3, 10, 4), 4);
}

static void lab_read_len_clamps_at_end(struct kunit *test)
{
	/* The read must be clipped to what is actually there. */
	KUNIT_EXPECT_EQ(test, lab_read_len(8, 10, 4), 2);
	KUNIT_EXPECT_EQ(test, lab_read_len(9, 10, 100), 1);
}

static void lab_read_len_eof(struct kunit *test)
{
	/* At or past the end is EOF, not an error and not a huge number. */
	KUNIT_EXPECT_EQ(test, lab_read_len(10, 10, 4), 0);
	KUNIT_EXPECT_EQ(test, lab_read_len(11, 10, 4), 0);
}

static void lab_read_len_degenerate(struct kunit *test)
{
	/* The boundary cases where an unsigned underflow would hide. */
	KUNIT_EXPECT_EQ(test, lab_read_len(0,  0, 4), 0);   /* empty buffer  */
	KUNIT_EXPECT_EQ(test, lab_read_len(0, 10, 0), 0);   /* zero-length   */
	KUNIT_EXPECT_EQ(test, lab_read_len(-1, 10, 4), 0);  /* negative pos  */
	KUNIT_EXPECT_EQ(test, lab_read_len(0, 10, SIZE_MAX), 10);
}

static struct kunit_case lab_range_cases[] = {
	KUNIT_CASE(lab_read_len_normal),
	KUNIT_CASE(lab_read_len_clamps_at_end),
	KUNIT_CASE(lab_read_len_eof),
	KUNIT_CASE(lab_read_len_degenerate),
	{}
};

static struct kunit_suite lab_range_suite = {
	.name = "lab_range",
	.test_cases = lab_range_cases,
};
kunit_test_suite(lab_range_suite);

MODULE_DESCRIPTION("KUnit tests for lab_read_len()");
MODULE_LICENSE("GPL");

Note: KUNIT_EXPECT_* records a failure and continues; KUNIT_ASSERT_* records a failure and aborts the test case. Use ASSERT for preconditions whose failure would make the rest of the case meaningless (a NULL from an allocation), and EXPECT for everything else — so one run tells you about all the failures, not just the first.

Step 3: Run it

cd ~/kernel/linux

# The simplest form: kunit.py builds a kernel and runs it for you.
./tools/testing/kunit/kunit.py run --kunitconfig=path/to/.kunitconfig lab_range

# Or, with the suite built into your lab kernel as a module:
#   CONFIG_KUNIT=y and your test's CONFIG_LAB_RANGE_KUNIT_TEST=m
# then in the guest:
modprobe lab_range_test
cat /sys/kernel/debug/kunit/lab_range/results
dmesg | grep -A 20 "lab_range"

A .kunitconfig next to the test declares the minimum config the suite needs:

CONFIG_KUNIT=y
CONFIG_LAB_RANGE_KUNIT_TEST=y

Output is TAP:

    KTAP version 1
    # Subtest: lab_range
    1..4
    ok 1 lab_read_len_normal
    ok 2 lab_read_len_clamps_at_end
    ok 3 lab_read_len_eof
    ok 4 lab_read_len_degenerate
ok 1 lab_range

Step 4: A kselftest

This is the one that tests your interface, from outside. tools/testing/selftests/lab/lab_test.c:

// SPDX-License-Identifier: GPL-2.0
#include <fcntl.h>
#include <string.h>
#include <sys/ioctl.h>
#include <unistd.h>

#include "../kselftest_harness.h"
#include "lab_uapi.h"

#define DEV "/dev/mylab"

FIXTURE(labdev) {
	int fd;
};

FIXTURE_SETUP(labdev)
{
	self->fd = open(DEV, O_RDWR);
	/* If the module is not loaded, SKIP — do not FAIL. A test that
	 * cannot run is not a test that found a bug.                   */
	if (self->fd < 0)
		SKIP(return, "%s not present (module not loaded?)", DEV);
	ASSERT_EQ(0, ioctl(self->fd, LAB_IOC_CLEAR));
}

FIXTURE_TEARDOWN(labdev)
{
	if (self->fd >= 0)
		close(self->fd);
}

TEST_F(labdev, write_then_read_roundtrips)
{
	char buf[16] = {0};

	ASSERT_EQ(5, write(self->fd, "hello", 5));
	ASSERT_EQ(0, lseek(self->fd, 0, SEEK_SET));
	ASSERT_EQ(5, read(self->fd, buf, sizeof(buf)));
	EXPECT_STREQ("hello", buf);
}

TEST_F(labdev, stats_reserved_field_is_zeroed)
{
	struct lab_stats st;

	memset(&st, 0xAA, sizeof(st));
	ASSERT_EQ(0, ioctl(self->fd, LAB_IOC_GET_STATS, &st));
	/* An unzeroed reserved field is an information leak AND breaks
	 * forward compatibility. Assert it explicitly.                 */
	EXPECT_EQ(0, st.__reserved);
}

TEST_F(labdev, unknown_ioctl_returns_ENOTTY)
{
	EXPECT_EQ(-1, ioctl(self->fd, _IO(0x99, 1)));
	EXPECT_EQ(ENOTTY, errno);
	EXPECT_EQ(-1, ioctl(self->fd, _IO(LAB_IOC_MAGIC, 99)));
	EXPECT_EQ(ENOTTY, errno);
}

TEST_F(labdev, oversize_len_is_rejected)
{
	__u32 big = 0xFFFFFFFF;

	EXPECT_EQ(-1, ioctl(self->fd, LAB_IOC_SET_LEN, &big));
	EXPECT_EQ(EINVAL, errno);
}

TEST_F(labdev, bad_user_pointer_returns_EFAULT)
{
	EXPECT_EQ(-1, read(self->fd, (void *)0x1, 16));
	EXPECT_EQ(EFAULT, errno);
}

TEST_HARNESS_MAIN

tools/testing/selftests/lab/Makefile:

# SPDX-License-Identifier: GPL-2.0
CFLAGS += -Wall -O2 $(KHDR_INCLUDES)
TEST_GEN_PROGS := lab_test

include ../lib.mk

Run it:

cd ~/kernel/linux
make -C tools/testing/selftests TARGETS=lab
make -C tools/testing/selftests TARGETS=lab run_tests

# Or from the top level:
make kselftest TARGETS=lab

# Or install a runnable bundle to copy into the guest:
make -C tools/testing/selftests TARGETS=lab install INSTALL_PATH=/tmp/ksft

TAP version 13
1..5
# Starting 5 tests from 1 test cases.
#  RUN           labdev.write_then_read_roundtrips ...
#            OK  labdev.write_then_read_roundtrips
ok 1 labdev.write_then_read_roundtrips
...
# PASSED: 5 / 5 tests passed.

Step 5: Exit codes, and why SKIP matters

ConstantValueMeans
KSFT_PASS0It worked
KSFT_FAIL1A real failure
KSFT_XFAIL2Expected failure
KSFT_XPASS3Unexpectedly passed
KSFT_SKIP4Could not run — feature absent, not root, wrong arch
rg -n "KSFT_(PASS|FAIL|SKIP)" tools/testing/selftests/kselftest.h

Getting SKIP right is what makes a test useful in CI. A test that FAILs on a machine where the feature is not configured trains everyone to ignore it, and then it is worthless the day it finds a real bug.

Step 6: The static checkers

Run all of these on the files you touched, before anyone else does.

cd ~/kernel/linux

make O=../build C=1 M=/path/to/module      # sparse, on files being recompiled
make O=../build C=2 M=/path/to/module      # sparse, on everything
make O=../build W=1 M=/path/to/module      # extra warnings maintainers care about
make O=../build W=2 M=/path/to/module      # more; expect noise from existing code

# Coccinelle: semantic patterns, including "you leaked this on the error path"
make O=../build coccicheck MODE=report M=/path/to/module

# smatch, if installed: flow-sensitive, finds unchecked user data and
# error-path bugs sparse cannot see
make O=../build CHECK=smatch C=1 M=/path/to/module

./scripts/checkpatch.pl --strict -g HEAD   # the commits you are about to send

Step 7: Build the way the bots build

This is the step people skip and the bots do not.

# Everything as a module: compiles code your config never touches.
make O=../build-all allmodconfig
make O=../build-all -j"$(nproc)"

# A randomized config: finds "this only builds when CONFIG_X is set" bugs.
make O=../build-rand randconfig
make O=../build-rand -j"$(nproc)"

# A different compiler exercises different warnings.
make O=../build-clang LLVM=1 defconfig
make O=../build-clang LLVM=1 -j"$(nproc)"

Predict first: which of the three is most likely to break code that builds cleanly for you, and why?

Step 8: Make the tests fail

The deliverable is not a green run. It is a green run you have seen turn red.

1. Change lab_read_len's clamp to `want` instead of `min`.
     → the KUnit clamp test must FAIL, and the message must name the values.

2. Remove the memset from LAB_IOC_GET_STATS.
     → stats_reserved_field_is_zeroed must FAIL — on a busy kernel.
       Note whether it fails EVERY time. This is the lesson about
       nondeterministic tests for information leaks.

3. Return -EINVAL instead of -ENOTTY for an unknown ioctl.
     → unknown_ioctl_returns_ENOTTY must FAIL.

4. Remove the `if (self->fd < 0) SKIP(...)`.
     → run it without the module loaded. It now FAILs instead of SKIPping.
       Which of those two would you rather see in a CI dashboard of 400 tests?

Implementation Requirements / Deliverables

  • A KUnit suite with at least four cases, covering normal, boundary, and degenerate inputs.
  • The suite runs under kunit.py run and as a module in your lab guest.
  • A kselftest using kselftest_harness.h with a fixture, covering the happy path and every error path of your ioctl.
  • The selftest SKIPs cleanly when the module is not loaded, with exit code 4.
  • make -C tools/testing/selftests TARGETS=lab run_tests passes and emits TAP.
  • make C=2, make W=1, and coccicheck clean for your files; every finding either fixed or explained in writing.
  • An allmodconfig build completed, or a written list of exactly what broke and why.
  • A clang (LLVM=1) build completed, or the same.
  • All four "make it fail" cases performed and recorded.
  • One paragraph: for each of your two tests, why is it in that framework and not the other?

Expected Output

$ ./tools/testing/kunit/kunit.py run lab_range
[00:00:12] Starting KUnit Kernel (1/1)...
[00:00:14] ============================================================
[00:00:14] ===================== lab_range (4 subtests) ===============
[00:00:14] [PASSED] lab_read_len_normal
[00:00:14] [PASSED] lab_read_len_clamps_at_end
[00:00:14] [PASSED] lab_read_len_eof
[00:00:14] [PASSED] lab_read_len_degenerate
[00:00:14] ===================== [PASSED] lab_range ===================
[00:00:14] Testing complete. Ran 4 tests: passed: 4

And a failure, which is what you actually want to see once:

[00:00:14] [FAILED] lab_read_len_clamps_at_end
[00:00:14]     # lab_read_len_clamps_at_end: EXPECTATION FAILED at lab_range_test.c:21
[00:00:14]     Expected lab_read_len(8, 10, 4) == 2, but
[00:00:14]         lab_read_len(8, 10, 4) == 4

Note that the message names the expression, the expected value, and the actual value. That is the standard your own failure messages should meet.

Debugging Steps

kunit.py cannot find the suite

The name is the .name field of the kunit_suite, not the file name. And the config symbol must be enabled in the .kunitconfig you passed.

The KUnit build fails on UML

Some code does not build for UML (anything with architecture-specific dependencies). Run under QEMU instead:

./tools/testing/kunit/kunit.py run --arch=x86_64 lab_range

The selftest does not build: linux/lab_stat.h: No such file

$(KHDR_INCLUDES) in the Makefile points at the installed uapi headers. Run make headers_install first, or include the header via a relative path for an out-of-tree experiment.

run_tests reports a failure with no output

The harness catches signals; a segfaulting test shows as a failure with no assertion message. Run the binary directly to see the crash, and consider whether an ASSERT should have caught the precondition first.

make W=1 produces hundreds of warnings from files you did not touch

Expected. Restrict it: build only your module (M=), or diff the warning list before and after your change. What matters is that your files add none.

allmodconfig fails in code you did not write

Also expected on a moving tree. Confirm with a clean checkout at the same commit; if it fails there too, it is not yours — and reporting it is a legitimate small contribution.

Sparse warns about __user in your selftest

Selftests are user-space programs. Do not run kernel checkers on them.

Experiment

CLAIM. A config you do not build is a config you have broken. The kernel's config space is large enough that "it builds for me" carries almost no information.

METHOD.

  1. Add a function to your module that is only referenced under #ifdef CONFIG_LAB_FEATURE, but defined unconditionally.
  2. Build with your normal config. Note the result.
  3. Build with W=1. Note the result.
  4. Build with allmodconfig. Note the result.
  5. Now make the definition conditional and the use unconditional, and repeat all three.

PREDICTION. Fill in a 2 × 3 table before running: for each of the two arrangements, does the build succeed under each of the three invocations?

RESULT. Then answer the question the 0-day robot exists to ask: given the size of the kernel's config space, what is the minimum set of builds you should run before sending a patch — and what do you accept that the bots will find for you?

Test

The test for this lab is the tests. Verify each one can fail, per step 8, and then verify the harness behaves:

# Exit codes, checked explicitly:
rmmod lab_chardev
./lab_test; echo "exit=$?"          # want 4 (SKIP), not 1 (FAIL)
insmod ./lab_chardev.ko
./lab_test; echo "exit=$?"          # want 0
# Break something, rebuild:
./lab_test; echo "exit=$?"          # want 1

Challenge Extensions

  1. Add a KUnit init/exit. Use .init and .exit on the suite (and KUNIT_CASE_PARAM for table-driven cases) to test a function that needs allocated state. Note how KUnit's kunit_kzalloc frees automatically at case end — and compare it with devres.

  2. Test the syscall from Lab 5. Write a selftest for it, including both compatibility directions (small usize, oversize usize). This is what an upstream syscall patch is expected to ship with.

  3. Read a real syzbot report. Find one on lore.kernel.org, download its C reproducer, and run it against your lab kernel. Then read Documentation/dev-tools/syzbot.rst and find out what #syz test: does.

  4. Run syzkaller yourself. Point it at your char device with a hand-written description of your ioctls (sys/linux/*.txt). Predict how long before it finds something; most people are surprised.

  5. Set up a local CI loop. A script that, for a given commit: applies it, runs checkpatch, builds defconfig + allmodconfig + clang, runs C=2 W=1, boots the guest, and runs both test suites. This is a small version of what the bots do, and running it before you send is what makes Lab 7 uneventful.

  6. Find an untested path. Build with CONFIG_GCOV_KERNEL=y, exercise your driver, and look at the coverage. Then write a test for the least-covered branch — which will be an error path, because it always is.

Validation / Self-check

  1. State the rule for choosing between KUnit and kselftest, and classify these: a bounds-check helper; the errno for a bad ioctl; a red-black tree's rebalance; the format of a /proc file.
  2. Where does a KUnit test run, and what does kunit.py build?
  3. What is the difference between KUNIT_EXPECT_* and KUNIT_ASSERT_*? When is each right?
  4. What are the kselftest exit codes, and why does SKIP matter more than it looks?
  5. What does a good failure message contain? Name the three parts from the KUnit output above.
  6. What does make C=1 do, and how does it differ from C=2?
  7. What does make W=1 add, and why is it noisy on files you did not touch?
  8. What does coccicheck find that sparse cannot? Give an example.
  9. Why does allmodconfig break code that builds fine for you?
  10. Name four automated systems that will look at your patch without being asked, and what each one does.
  11. Why must you watch a test fail before trusting it?
  12. Your information-leak test passes on a freshly booted machine and fails on a busy one. What does that tell you about the bug, and about the test?

Foundations Complete

You can now write, load, debug, and test kernel code. That is the floor, and everything after it stands on it.

What you cannot do yet is get any of it merged — and the gap is workflow, not knowledge. It is smaller than you think and it blocks more people than anything in this section.

Next: Contribution — MAINTAINERS, patch craft, email, and review. Start Lab 7 early: the community's clock runs in weeks, and you want it running while you read.