Lab 3: A Character Device (Milestone 3)

Background

Until now your code ran only when you loaded it. This lab gives user space a way in: a device node in /dev that open, read, write, lseek, and ioctl reach through a table of function pointers you fill in.

That table — struct file_operations — is the same indirection the VFS uses for every file on the system. Filling it in is how a driver, a filesystem, and /dev/null all present the same interface.

You will also, deliberately, write a race, observe it, and fix it. That part is not decoration: a character device is the smallest program where two processes can be inside your kernel code at the same time, and seeing it happen is worth more than reading the concurrency chapter twice.

Why This Lab Matters

• file_operations is the interface. Learn it once and every char driver, block driver, and filesystem is a variation.
• Every user pointer that crosses into your code is untrusted, and this is where you handle that for real.
• ioctl numbers and their structs are permanent uapi. Getting the shape right here is the rehearsal for Lab 5.
• The race you write and fix is the shape of the bug that survives review in real drivers.

Prerequisites

• Lab 2 complete.
• The User/Kernel Boundary read — all three concepts.
• Concurrency read as far as spinlocks vs. mutexes.
• A lab-paranoid kernel available (you will want KASAN and lockdep for the race).

Predict First

Write these down before you write code.

1. Two processes write() to your device at the same offset simultaneously, with no locking. What are the possible outcomes for the buffer contents? Is a torn mixture of both payloads possible?
2. You increment a u64 statistics counter from read() with no lock. After 4 processes each do 100,000 reads, what will the counter say?
3. You call copy_to_user() while holding a mutex. Legal or not? While holding a spinlock?
4. Your ioctl handler is passed a command number your driver does not implement. What errno should it return, and why that one specifically?
5. A process has your device open. You run rmmod. What happens — with .owner = THIS_MODULE, and without?

The Target

   user space                        kernel
   ──────────                        ──────
   open("/dev/mylab")  ─────────────▶ lab_open      → f->private_data = dev
   read(fd, buf, n)    ─────────────▶ lab_read      → mutex, copy_to_user
   write(fd, buf, n)   ─────────────▶ lab_write     → mutex, copy_from_user
   lseek(fd, off, w)   ─────────────▶ lab_llseek
   ioctl(fd, CMD, arg) ─────────────▶ lab_ioctl     → validate, copy both ways
   close(fd)           ─────────────▶ lab_release

              struct file_operations  ← the table that connects them

Step-by-Step Tasks

Step 1: The uapi header

This file is compiled by both the kernel and your test program. Every rule from the boundary chapter applies.

lab_uapi.h:

/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
#ifndef _LAB_UAPI_H
#define _LAB_UAPI_H

#include <linux/types.h>
#include <linux/ioctl.h>

/* Registered magic numbers live in
 * Documentation/userspace-api/ioctl/ioctl-number.rst. 0xBA is used here
 * only because this driver is never going upstream; a real one must claim
 * a number in that file as part of the patch.                            */
#define LAB_IOC_MAGIC	0xBA

struct lab_stats {
	__u64	reads;		/* fixed width, uapi spelling            */
	__u64	writes;
	__u32	len;		/* bytes currently held                  */
	__u32	__reserved;	/* EXPLICIT padding: zeroed and checked,
				 * so it can become a real field later   */
};

#define LAB_IOC_GET_STATS	_IOR(LAB_IOC_MAGIC, 1, struct lab_stats)
#define LAB_IOC_CLEAR		_IO (LAB_IOC_MAGIC, 2)
#define LAB_IOC_SET_LEN		_IOW(LAB_IOC_MAGIC, 3, __u32)
#define LAB_IOC_MAXNR		3

#endif /* _LAB_UAPI_H */

_IOR/_IOW/_IOWR encode the direction and the argument size into the number itself, which is how the kernel can reject a command built against a different version of the struct.

Step 2: The racy version — write this one first, on purpose

lab_chardev.c, version A. Note the deliberate absence of any locking.

// SPDX-License-Identifier: GPL-2.0
#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt

#include <linux/fs.h>
#include <linux/miscdevice.h>
#include <linux/module.h>
#include <linux/mutex.h>
#include <linux/slab.h>
#include <linux/uaccess.h>

#include "lab_uapi.h"

#define LAB_BUF_SIZE	4096

struct lab_dev {
	struct mutex	lock;		/* unused in version A */
	char		*buf;
	size_t		len;
	u64		reads;
	u64		writes;
};

static struct lab_dev *labdev;

static int lab_open(struct inode *ino, struct file *f)
{
	f->private_data = labdev;
	return 0;
}

static ssize_t lab_read(struct file *f, char __user *ubuf,
			size_t count, loff_t *ppos)
{
	struct lab_dev *d = f->private_data;

	if (*ppos >= d->len)
		return 0;				/* EOF */
	count = min_t(size_t, count, d->len - *ppos);

	if (copy_to_user(ubuf, d->buf + *ppos, count))
		return -EFAULT;

	*ppos += count;
	d->reads++;					/* RACY */
	return count;
}

static ssize_t lab_write(struct file *f, const char __user *ubuf,
			 size_t count, loff_t *ppos)
{
	struct lab_dev *d = f->private_data;

	if (*ppos >= LAB_BUF_SIZE)
		return -ENOSPC;
	count = min_t(size_t, count, LAB_BUF_SIZE - *ppos);

	if (copy_from_user(d->buf + *ppos, ubuf, count))
		return -EFAULT;

	*ppos += count;
	if (*ppos > d->len)
		d->len = *ppos;				/* RACY */
	d->writes++;					/* RACY */
	return count;
}

Step 3: ioctl, with every check

static long lab_ioctl(struct file *f, unsigned int cmd, unsigned long arg)
{
	struct lab_dev *d = f->private_data;
	struct lab_stats st;
	u32 newlen;

	/* Reject commands that are not ours BEFORE looking at the number.
	 * -ENOTTY, not -EINVAL: userspace uses it to mean "this device does
	 *  not implement that ioctl", which is a different thing from "bad
	 *  arguments".                                                     */
	if (_IOC_TYPE(cmd) != LAB_IOC_MAGIC)
		return -ENOTTY;
	if (_IOC_NR(cmd) > LAB_IOC_MAXNR)
		return -ENOTTY;

	switch (cmd) {
	case LAB_IOC_GET_STATS:
		/* memset the WHOLE struct: the compiler is not required to
		 * initialize padding, and padding copied out is an info leak. */
		memset(&st, 0, sizeof(st));
		st.reads  = d->reads;
		st.writes = d->writes;
		st.len    = d->len;
		if (copy_to_user((void __user *)arg, &st, sizeof(st)))
			return -EFAULT;
		return 0;

	case LAB_IOC_CLEAR:
		memset(d->buf, 0, LAB_BUF_SIZE);
		d->len = 0;
		return 0;

	case LAB_IOC_SET_LEN:
		if (copy_from_user(&newlen, (void __user *)arg, sizeof(newlen)))
			return -EFAULT;
		if (newlen > LAB_BUF_SIZE)		/* validate the KERNEL copy */
			return -EINVAL;
		d->len = newlen;
		return 0;

	default:
		return -ENOTTY;
	}
}

static loff_t lab_llseek(struct file *f, loff_t off, int whence)
{
	return fixed_size_llseek(f, off, whence, LAB_BUF_SIZE);
}

static const struct file_operations lab_fops = {
	.owner		= THIS_MODULE,	/* holds a module ref while open */
	.open		= lab_open,
	.read		= lab_read,
	.write		= lab_write,
	.llseek		= lab_llseek,
	.unlocked_ioctl	= lab_ioctl,	/* "unlocked" = without the long-dead
					 *  big kernel lock. A historical name. */
	.compat_ioctl	= compat_ptr_ioctl,	/* 32-bit userspace on a 64-bit
						 * kernel: our args are a pointer
						 * and a __u32, so this generic
						 * thunk is sufficient.          */
};

static struct miscdevice lab_misc = {
	.minor	= MISC_DYNAMIC_MINOR,
	.name	= "mylab",		/* → /dev/mylab, created by udev/devtmpfs */
	.fops	= &lab_fops,
	.mode	= 0666,
};

static int __init lab_init(void)
{
	int ret;

	labdev = kzalloc(sizeof(*labdev), GFP_KERNEL);
	if (!labdev)
		return -ENOMEM;
	mutex_init(&labdev->lock);

	labdev->buf = kzalloc(LAB_BUF_SIZE, GFP_KERNEL);
	if (!labdev->buf) {
		ret = -ENOMEM;
		goto err_free_dev;
	}

	/* LAST: after this, user space can call in, so everything it
	 * touches must already be valid.                              */
	ret = misc_register(&lab_misc);
	if (ret)
		goto err_free_buf;

	pr_info("registered /dev/%s\n", lab_misc.name);
	return 0;

err_free_buf:
	kfree(labdev->buf);
err_free_dev:
	kfree(labdev);
	labdev = NULL;
	return ret;
}

static void __exit lab_exit(void)
{
	misc_deregister(&lab_misc);	/* FIRST: no new callers */
	kfree(labdev->buf);
	kfree(labdev);
	labdev = NULL;
	pr_info("unregistered\n");
}

module_init(lab_init);
module_exit(lab_exit);

MODULE_DESCRIPTION("Foundations Lab 3: a character device");
MODULE_AUTHOR("You <you@example.com>");
MODULE_LICENSE("GPL");

Note: misc_register() is the convenience layer. It allocates a minor number, registers a cdev, and creates the /dev node — three things you will do by hand in challenge 1. Per this book's rule, use the convenience only after you can write what it hides; challenge 1 is where you pay that debt.

Step 4: The userspace exerciser

lab_test.c:

// SPDX-License-Identifier: GPL-2.0
#define _GNU_SOURCE
#include <fcntl.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/ioctl.h>
#include <sys/wait.h>
#include <unistd.h>

#include "lab_uapi.h"

#define DEV "/dev/mylab"

static int check_basic(void)
{
	struct lab_stats st;
	char buf[64] = {0};
	int fd, rc = 0;

	fd = open(DEV, O_RDWR);
	if (fd < 0) { perror("open"); return 1; }

	if (ioctl(fd, LAB_IOC_CLEAR) < 0) { perror("CLEAR"); rc = 1; }

	if (write(fd, "hello", 5) != 5) { perror("write"); rc = 1; }
	lseek(fd, 0, SEEK_SET);
	if (read(fd, buf, sizeof(buf)) != 5) { perror("read"); rc = 1; }
	if (memcmp(buf, "hello", 5)) { fprintf(stderr, "content mismatch\n"); rc = 1; }

	if (ioctl(fd, LAB_IOC_GET_STATS, &st) < 0) { perror("GET_STATS"); rc = 1; }
	printf("reads=%llu writes=%llu len=%u reserved=%u\n",
	       (unsigned long long)st.reads, (unsigned long long)st.writes,
	       st.len, st.__reserved);
	if (st.__reserved != 0) { fprintf(stderr, "reserved not zeroed!\n"); rc = 1; }

	/* Error paths. */
	if (ioctl(fd, _IO(0x99, 1)) != -1)   { fprintf(stderr, "bad magic accepted\n"); rc = 1; }
	if (ioctl(fd, _IO(LAB_IOC_MAGIC, 99)) != -1) { fprintf(stderr, "bad nr accepted\n"); rc = 1; }
	{ __u32 big = 999999;
	  if (ioctl(fd, LAB_IOC_SET_LEN, &big) != -1) { fprintf(stderr, "oversize len accepted\n"); rc = 1; } }
	if (read(fd, (void *)0x1, 16) != -1) { fprintf(stderr, "bad pointer accepted\n"); rc = 1; }

	close(fd);
	return rc;
}

/* The race: N processes each doing ITERS reads. If the counter is correct,
 * the total is exactly N * ITERS.                                        */
#define NPROC 4
#define ITERS 50000

static int check_race(void)
{
	struct lab_stats st;
	char buf[8];
	int fd, i;

	fd = open(DEV, O_RDWR);
	if (fd < 0) { perror("open"); return 1; }
	ioctl(fd, LAB_IOC_CLEAR);
	write(fd, "abcdefgh", 8);
	close(fd);

	for (i = 0; i < NPROC; i++) {
		if (fork() == 0) {
			int f = open(DEV, O_RDONLY), j;
			for (j = 0; j < ITERS; j++) {
				lseek(f, 0, SEEK_SET);
				if (read(f, buf, sizeof(buf)) < 0) _exit(1);
			}
			close(f);
			_exit(0);
		}
	}
	for (i = 0; i < NPROC; i++) wait(NULL);

	fd = open(DEV, O_RDWR);
	ioctl(fd, LAB_IOC_GET_STATS, &st);
	close(fd);

	printf("expected %d reads, counter says %llu (lost %lld)\n",
	       NPROC * ITERS, (unsigned long long)st.reads,
	       (long long)(NPROC * ITERS) - (long long)st.reads);
	return st.reads == (unsigned long long)NPROC * ITERS ? 0 : 1;
}

int main(void)
{
	int rc = check_basic();
	rc |= check_race();
	puts(rc ? "FAIL" : "PASS");
	return rc;
}

Step 5: Observe the race

# In the guest, on lab-fast, with -smp 4:
insmod /mnt/host/chardev/lab_chardev.ko
/mnt/host/chardev/lab_test

Predict before running: how many of the 200,000 reads will the counter lose? Write a number.

Then run it at -smp 1 and at -smp 8 and compare. This is the experiment from Concurrency, Concept 1 in the shape of a real device, and the -smp 1 result is the one that teaches the lesson.

Step 6: Fix it, and defend the choice

Add the mutex. Every field of struct lab_dev is now protected by d->lock:

static ssize_t lab_read(struct file *f, char __user *ubuf,
			size_t count, loff_t *ppos)
{
	struct lab_dev *d = f->private_data;
	ssize_t ret;

	/* _interruptible so a Ctrl-C can break a stuck reader out.
	 * -ERESTARTSYS asks the syscall layer to restart or return -EINTR;
	 *  it must never reach user space as-is.                          */
	if (mutex_lock_interruptible(&d->lock))
		return -ERESTARTSYS;

	if (*ppos >= d->len) {
		ret = 0;
		goto out;
	}
	count = min_t(size_t, count, d->len - *ppos);

	/* copy_to_user CAN SLEEP (the user page may not be resident). That
	 * is legal here because we hold a MUTEX. Under a spinlock this line
	 * would be a BUG — which is exactly why this driver uses a mutex.  */
	if (copy_to_user(ubuf, d->buf + *ppos, count)) {
		ret = -EFAULT;
		goto out;
	}

	*ppos += count;
	d->reads++;
	ret = count;
out:
	mutex_unlock(&d->lock);
	return ret;
}

Apply the same treatment to lab_write and to every ioctl case that touches d. Rebuild, reload, re-run the test. The counter must now be exact.

Write down why a mutex and not a spinlock. The answer must mention copy_to_user.

Step 7: Run it under the paranoid kernel

# lab-paranoid: KASAN + PROVE_LOCKING
insmod /mnt/host/chardev/lab_chardev.ko
/mnt/host/chardev/lab_test
dmesg | tail -30           # must be silent: no KASAN, no lockdep splat

Implementation Requirements / Deliverables

• /dev/mylab appears on insmod and disappears on rmmod.
• open, read, write, llseek, unlocked_ioctl, compat_ioctl, and release implemented.
• Every user pointer goes through copy_to_user/copy_from_user, with the return checked.
• Structs copied out are memset first; the test asserts __reserved == 0.
• Unknown ioctl magic and unknown command numbers both return -ENOTTY.
• Every ioctl argument is validated on the kernel copy, never on user memory.
• The racy version was built, run, and its loss recorded at -smp 1, 4, and 8.
• The fixed version passes the concurrency test exactly, with PROVE_LOCKING silent.
• A written justification for mutex-over-spinlock that mentions copy_to_user.
• .owner = THIS_MODULE, and the rmmod-while-open behavior demonstrated.
• make check (C=2 W=1) and checkpatch --strict -f clean.

Expected Output

# insmod /mnt/host/chardev/lab_chardev.ko
[  102.3] lab_chardev: registered /dev/mylab

# ls -l /dev/mylab
crw-rw-rw-    1 root  root  10, 122 Jan  1 00:01 /dev/mylab
^ 'c' = character device.  10 = misc major.  122 = the minor it was given.

# echo -n hello > /dev/mylab
# cat /dev/mylab
hello

# /mnt/host/chardev/lab_test        (RACY version, -smp 4)
reads=1 writes=1 len=5 reserved=0
expected 200000 reads, counter says 172431 (lost 27569)
FAIL

# /mnt/host/chardev/lab_test        (FIXED version, -smp 4)
reads=1 writes=1 len=5 reserved=0
expected 200000 reads, counter says 200000 (lost 0)
PASS

And the lifetime check:

# exec 3</dev/mylab
# lsmod | grep lab_chardev
lab_chardev            16384  1
                              ^ the open fd
# rmmod lab_chardev
rmmod: ERROR: Module lab_chardev is in use
# exec 3<&-
# rmmod lab_chardev

Debugging Steps

/dev/mylab does not appear

devtmpfs is not mounted in the guest. Your /init must do mount -t devtmpfs none /dev, and the kernel needs CONFIG_DEVTMPFS=y. Check:

grep -E '^CONFIG_DEVTMPFS' ~/kernel/build/.config
mount | grep devtmpfs
cat /proc/misc | grep mylab       # the driver registered even if the node did not appear

open() returns -ENODEV or -ENXIO

misc_register failed, or the node's minor does not match. dmesg after insmod, and compare /proc/misc with ls -l /dev/mylab.

read() returns -EFAULT for a pointer that looks fine

You are dereferencing instead of copying, or copying the wrong length. Also check make check — sparse catches a __user pointer used directly.

ioctl returns -ENOTTY for a command you implemented

The command number encodes the argument type and size. If the header the kernel was built with and the header your test program included have different struct sizes, the numbers differ. Rebuild both from the same header — and note that this is exactly the ABI failure mode Lab 5 is about.

The concurrency test passes at -smp 1 and fails at -smp 4

That is not a flaky test. That is the bug, and the -smp 1 run is the misleading one.

lockdep complains about mutex_lock in your ioctl

Read the splat's second stack trace — you are probably taking the mutex twice on one path, or calling a _locked helper without the lock. Split the function.

rmmod succeeds while the device is open, and the next read() panics

.owner = THIS_MODULE is missing from fops.

KASAN: use-after-free in lab_read

misc_deregister() is not the first thing in lab_exit, so a caller entered after you freed the buffer. Order matters: stop new callers, then free.

Experiment

CLAIM. copy_to_user can sleep, so the choice of lock is not a style preference — a spinlock here is a bug that the debug kernel catches immediately.

METHOD. Take the fixed driver and convert d->lock from a struct mutex to a spinlock_t, mechanically:

static DEFINE_SPINLOCK(...);          /* instead of a mutex */
spin_lock(&d->lock);
...  copy_to_user(...);               /* unchanged */
spin_unlock(&d->lock);

PREDICTION. Before building: (a) does it compile? (b) does it load? (c) does a read() work? (d) does dmesg say anything, and on which kernel — lab-fast or only lab-paranoid?

RESULT. Record the exact BUG message and identify, from it, the line that made you atomic — using the Preemption disabled at: field from the context chapter.

Then answer the design question this raises: suppose you genuinely needed a spinlock — because an interrupt handler also touched this data. How would you restructure read() so that the copy_to_user happens outside the critical section? (Hint: copy into a kernel bounce buffer under the lock, release it, then copy out. And then say what that costs and what it changes about consistency.)

Test

The exerciser above is the test. Two things make it a good one:

It asserts on error paths, not just the happy path. Bad ioctl magic, bad command number, oversize length, and a bad user pointer are each checked to fail.

It can fail. Verify that:

# 1. Remove the mutex from lab_read only. Rebuild. The race check must FAIL.
# 2. Remove the memset from the GET_STATS case. Rebuild. Run the test on a
#    kernel that has had some activity — does __reserved come back nonzero?
# 3. Change -ENOTTY to -EINVAL for unknown commands. The test must FAIL.

A test you have never seen fail is a test you have not written. Lab 6 rewrites this as a proper kselftest with the harness conventions.

Challenge Extensions

1. Do it without miscdevice. Rewrite with alloc_chrdev_region, cdev_init, cdev_add, class_create, and device_create. Diff the two files and enumerate every line that appeared — each one is something misc_register did for you. This is the debt this lab deferred; pay it.

2. Per-open state. Give each open() its own private buffer and position in f->private_data, allocated in open and freed in release. Then answer: which of your locks is now unnecessary, and which is still required?

3. Make it a blocking device. read() on an empty buffer should sleep until a writer arrives, using wait_event_interruptible and wake_up_interruptible. Handle O_NONBLOCK (-EAGAIN), and then implement .poll so select/epoll work. This is where a toy becomes a device.

4. Add an extensible ioctl. Use copy_struct_from_user() so that a program built against an older header still works, and one built against a newer one gets -E2BIG rather than silent truncation. Then test both directions by compiling the test program against two versions of the header.

5. Find the double fetch. Rewrite LAB_IOC_SET_LEN to read the length from user space twice — once to validate, once to use — and write a userspace program with two threads that flips the value between the fetches until it wins. Then fix it and explain the structural reason the fixed version cannot lose.

6. Convert to a real device. Register a platform driver as in the device model chapter, attach the char device to it, add a sysfs attribute via dev_groups, and use devm_kzalloc. Then unbind and rebind it from sysfs and confirm nothing leaks.

Validation / Self-check

1. What is struct file_operations, and name three completely different things in the kernel that fill one in.
2. Why is unlocked_ioctl called that? What is compat_ioctl for, and when is compat_ptr_ioctl sufficient?
3. What must an ioctl handler check before it looks at the command number, and what errno does it return for a command it does not implement — and why not -EINVAL?
4. Why is a struct memset before copy_to_user, when every field is assigned?
5. Why must ioctl arguments be validated on the kernel copy rather than in user memory? Name the bug class.
6. Why is copy_to_user legal under a mutex and a bug under a spinlock?
7. Your counter lost 27,569 of 200,000 increments at -smp 4 and zero at -smp 1. Explain both numbers.
8. What does mutex_lock_interruptible buy you, and what must you return when it fails?
9. Why is misc_deregister() the first statement in the exit function?
10. Why is misc_register() the last statement in the init function?
11. What does .owner = THIS_MODULE prevent? Demonstrate the failure without it.
12. fixed_size_llseek versus a hand-written llseek: what does the helper get right that people get wrong?
13. You need a spinlock because an interrupt handler touches this data, but you also need copy_to_user. Describe the restructuring, and what it costs.

Next: Lab 4 — Debugging a Kernel, where you break this driver on purpose and learn to read what the kernel tells you.