Deferred Work

A device raises an interrupt. The handler runs in atomic context, possibly with other interrupts masked, on a CPU that was doing something else, and every microsecond it spends adds latency to the entire machine. The work that actually needs doing takes two milliseconds and needs to allocate memory.

Half of driver design is deciding what to do about that gap. This chapter is the menu of options, the criteria for choosing, and the teardown discipline that turns a working driver into one that can be unloaded.

Six concepts: the two halves, softirqs and tasklets, workqueues, threaded IRQs, timers, and teardown.

Concept 1: The Interrupt and the Two Halves

1. What problem it solves

An interrupt handler must acknowledge the device quickly — the device may be waiting, the line may be level-triggered and will re-assert forever until serviced, and on many architectures other interrupts are masked while it runs. But the response to the event is usually substantial work.

The classic Unix answer, unchanged in shape since the 1980s: split it. A top half that runs immediately in interrupt context and does the minimum, and a bottom half that runs later, in a context with fewer restrictions.

   device asserts IRQ
        │
        ▼
   ┌──────────────────────────────────────────────────────────┐
   │  TOP HALF — hardirq context                              │
   │  • acknowledge the device (stop it re-asserting)         │
   │  • read the minimum state that will be lost otherwise    │
   │  • schedule the bottom half                              │
   │  • return IRQ_HANDLED                                    │
   │  CANNOT: sleep, allocate with GFP_KERNEL, take a mutex,  │
   │          copy to/from user, take an unbounded amount of  │
   │          time                                            │
   └──────────────────────────────────────────────────────────┘
        │
        ▼
   ┌──────────────────────────────────────────────────────────┐
   │  BOTTOM HALF — one of:                                   │
   │    softirq / tasklet   still atomic, but interruptible   │
   │    threaded IRQ        process context, MAY SLEEP        │
   │    workqueue           process context, MAY SLEEP        │
   └──────────────────────────────────────────────────────────┘

2. Where it exists in the kernel

ls kernel/irq/ include/linux/interrupt.h
rg -n "int request_threaded_irq|devm_request_threaded_irq" include/linux/interrupt.h
cat /proc/interrupts | head -10
cat /proc/softirqs
$EDITOR Documentation/core-api/genericirq.rst

3. Who owns or interacts with it

4. Registering a handler

/* Simple: one handler, atomic context. */
ret = request_irq(irq, my_isr, IRQF_SHARED, "mylab", ml);
/*                                ^ shared lines REQUIRE a unique dev_id (ml)
 *                                  and the handler must check whether the
 *                                  interrupt was actually yours              */

/* Threaded: a small atomic top half plus a sleepable thread. */
ret = request_threaded_irq(irq, my_isr, my_isr_thread,
                           IRQF_ONESHOT, "mylab", ml);

/* In a driver, prefer the devres versions — see the device model chapter. */
ret = devm_request_threaded_irq(dev, irq, my_isr, my_isr_thread,
                                IRQF_ONESHOT, "mylab", ml);

static irqreturn_t my_isr(int irq, void *dev_id)
{
        struct mylab *ml = dev_id;
        u32 status = readl(ml->regs + REG_STATUS);

        if (!(status & STATUS_MINE))
                return IRQ_NONE;        /* shared line: not ours. MUST return this,
                                         * or the kernel's spurious-interrupt
                                         * detection will disable the line.      */

        writel(status, ml->regs + REG_STATUS);   /* acknowledge */

        return IRQ_WAKE_THREAD;         /* run my_isr_thread in process context */
}

static irqreturn_t my_isr_thread(int irq, void *dev_id)
{
        struct mylab *ml = dev_id;

        /* Process context. Sleeping is allowed here. */
        mutex_lock(&ml->lock);
        process_the_event(ml);          /* may allocate with GFP_KERNEL     */
        mutex_unlock(&ml->lock);

        return IRQ_HANDLED;
}

5. Experiment

CLAIM. The kernel detects an interrupt line nobody handles and disables it — and the message it prints is one you will one day have to recognize.

METHOD. Read the mechanism first, then watch a real line:

rg -n "nobody cared|note_interrupt|spurious" kernel/irq/spurious.c | head
cat /proc/interrupts                      # note the per-CPU counts

# Generate interrupts and watch a specific line's delta:
IRQ=$(awk -F: '/eth|virtio|nvme/{gsub(/ /,"",$1); print $1; exit}' /proc/interrupts)
before=$(grep "^ *$IRQ:" /proc/interrupts)
ping -c 200 -i 0.01 -q "$(ip route | awk '/^default/{print $3;exit}')" >/dev/null 2>&1
after=$(grep "^ *$IRQ:" /proc/interrupts)
echo "before: $before"; echo "after : $after"

PREDICT FIRST: will the interrupt count go up by roughly 200, much less, or much more? (The answer is "much less", and the reason is NAPI — a driver that switches from interrupts to polling under load. That mechanism is a softirq, which is the next concept.)

6. Failure mode

Concept 2: Softirqs and Tasklets

1. What problem it solves

Some bottom halves must still be fast and must not sleep — network receive, block I/O completion, timer expiry. Making them full process-context threads would add a context switch to every packet. Softirqs are the answer: still atomic, but running with interrupts enabled and preemptible by hardirqs, so a long one does not block interrupt delivery.

2. Where it exists in the kernel

rg -n "enum.*NR_SOFTIRQS|open_softirq" include/linux/interrupt.h kernel/softirq.c | head
cat /proc/softirqs
ps -eo pid,comm | grep ksoftirqd

3. The fixed set, and why you cannot add one

Softirq types are a compile-time enum: HI, TIMER, NET_TX, NET_RX, BLOCK, IRQ_POLL, TASKLET, SCHED, HRTIMER, RCU. open_softirq() is not available to modules, and adding a type to the enum is a core-kernel change with a very high bar.

This is deliberate. Softirqs are a scarce, global, latency-critical resource; if every driver could add one, the return-from-interrupt path would become unbounded.

   WHEN DO SOFTIRQS RUN?

   1. On return from a hardware interrupt (irq_exit), if any are pending.
   2. On local_bh_enable(), if any are pending.
   3. In ksoftirqd/N, when (1) has already processed too many in a row —
      which is the kernel refusing to let softirq load starve user space.

   That third case is why you see ksoftirqd at 100% CPU on a machine under
   heavy network load: the softirq work exceeded what the return-from-interrupt
   path is willing to do inline, so it was handed to a schedulable thread.

4. Tasklets

A tasklet is a way to run a function in softirq context without owning a softirq type. Two properties distinguish it: a given tasklet never runs on two CPUs at once, and it is dynamically created.

static void lab_tasklet_fn(struct tasklet_struct *t) { /* softirq context */ }
static DECLARE_TASKLET(lab_tasklet, lab_tasklet_fn);
/* ... in the ISR: */    tasklet_schedule(&lab_tasklet);
/* ... at teardown: */   tasklet_kill(&lab_tasklet);

Warning: Tasklets are on the way out. They have no priority control, they run in a context that cannot sleep, and their semantics interact badly with PREEMPT_RT. There has been a long-running effort to convert existing users to threaded IRQs or workqueues, and a patch adding a new tasklet will be questioned. Check the current state of that effort in your tree before you reach for one:

rg -n -i "tasklet" Documentation/ | head
git log --oneline --grep="tasklet" --since="2 years ago" | head -20

For new code: use a threaded IRQ or a workqueue.

5. Experiment

CLAIM. Softirq load that exceeds the inline budget migrates to ksoftirqd, and you can watch the handoff.

METHOD.

# Terminal 1: watch NET_RX and the ksoftirqd threads.
watch -n1 'grep NET_RX /proc/softirqs; ps -eo comm,pcpu | grep ksoftirqd'

# Terminal 2: generate enough receive load to matter.
ping -f -c 20000 "$(ip route | awk '/^default/{print $3;exit}')" >/dev/null 2>&1 &
# or, on a loopback-only guest:
iperf3 -s >/dev/null 2>&1 & sleep 1; iperf3 -c 127.0.0.1 -t 20 >/dev/null

PREDICT FIRST: at what point does ksoftirqd appear in top? Does NET_RX grow linearly with packet count?

6. Failure mode

Concept 3: Workqueues

1. What problem it solves

You need to do something that can sleep — allocate with GFP_KERNEL, take a mutex, do an I²C transfer, wait for firmware — from a context that cannot. A workqueue runs your function in a kernel thread, in process context, with all restrictions lifted.

This is the default answer for driver bottom halves that are not latency-critical.

2. Where it exists in the kernel

ls kernel/workqueue.c include/linux/workqueue.h
$EDITOR Documentation/core-api/workqueue.rst
ps -eo pid,comm | grep kworker | head
ls /sys/bus/workqueue/devices/ 2>/dev/null      # for queues created with WQ_SYSFS

The implementation is concurrency-managed workqueues (cmwq): a shared pool of kworker threads per CPU, sized dynamically, so that a thousand drivers queuing work do not create a thousand threads.

3. The API

struct mylab {
        struct work_struct         work;
        struct delayed_work        poll;
        struct workqueue_struct   *wq;
        ...
};

static void lab_work_fn(struct work_struct *w)
{
        struct mylab *ml = container_of(w, struct mylab, work);

        /* Process context. Everything is allowed. */
        mutex_lock(&ml->lock);
        ...
        mutex_unlock(&ml->lock);
}

/* setup */
INIT_WORK(&ml->work, lab_work_fn);
INIT_DELAYED_WORK(&ml->poll, lab_poll_fn);

/* Use the shared queue for short, ordinary work: */
schedule_work(&ml->work);
schedule_delayed_work(&ml->poll, msecs_to_jiffies(500));

/* Or your own, when you need specific properties: */
ml->wq = alloc_workqueue("mylab", WQ_UNBOUND | WQ_MEM_RECLAIM, 0);
queue_work(ml->wq, &ml->work);

Note: WQ_MEM_RECLAIM is the flag people omit and reviewers catch. If your work item is on a path that memory reclaim can depend on — a block driver, a filesystem, a storage path — then under memory pressure the workqueue code may be unable to create a worker thread, and your work never runs, and reclaim never completes. The rescuer thread exists for exactly that deadlock.

Two behaviors that surprise people:

queue_work() returns false if the item is already queued, and does not queue it twice. A work_struct is a single slot, not a queue. If you need to remember that another event happened, set a flag the work function checks.

A work item may run on any CPU, and a plain work_struct may run concurrently with itself on two CPUs if requeued — unless you use an ordered queue (alloc_ordered_workqueue). Do not assume serialization you did not ask for.

4. Experiment

CLAIM. A work_struct cannot be queued twice, and understanding that changes how you write the handler.

METHOD.

static atomic_t runs = ATOMIC_INIT(0);

static void lab_work_fn(struct work_struct *w)
{
        atomic_inc(&runs);
        msleep(100);                    /* legal here — process context */
}

static int __init wq_init(void)
{
        int i, queued = 0;

        INIT_WORK(&lab_work, lab_work_fn);
        for (i = 0; i < 10; i++)
                queued += schedule_work(&lab_work);   /* returns bool */
        msleep(1000);
        pr_info("queue_work returned true %d times; the fn ran %d times\n",
                queued, atomic_read(&runs));
        return 0;
}

PREDICT FIRST: how many times does schedule_work return true? How many times does the function run? Write both numbers, then explain the gap.

5. Failure mode

Concept 4: Threaded IRQs

1. What problem it solves

A workqueue is a general mechanism with general scheduling. For an interrupt, you usually want something more specific: a dedicated thread, one per IRQ, that the interrupt directly wakes, whose priority you can set, and which is automatically torn down with the IRQ.

That is what request_threaded_irq() gives you, and on PREEMPT_RT it is what almost every handler becomes anyway.

2. Where it exists in the kernel

rg -n "request_threaded_irq|irq_thread\b" kernel/irq/manage.c | head
ps -eo pid,rtprio,comm | grep irq/

Look for irq/NN-<name> in ps. Each is a thread created for a threaded handler.

3. How to use it

ret = devm_request_threaded_irq(dev, irq,
                                lab_isr,          /* hardirq: quick check + ack */
                                lab_isr_thread,   /* process context: the work  */
                                IRQF_ONESHOT,
                                "mylab", ml);

Threaded IRQ vs. workqueue — the decision:

4. Experiment

CLAIM. A threaded handler runs in process context and can do everything the top half cannot.

METHOD. In a driver with a threaded IRQ, print the context from both halves:

static irqreturn_t lab_isr(int irq, void *d)
{
        pr_info("top:    in_task=%d in_hardirq=%d preempt=%08x\n",
                !!in_task(), !!in_hardirq(), preempt_count());
        return IRQ_WAKE_THREAD;
}
static irqreturn_t lab_isr_thread(int irq, void *d)
{
        pr_info("thread: in_task=%d in_hardirq=%d preempt=%08x\n",
                !!in_task(), !!in_hardirq(), preempt_count());
        msleep(1);                      /* proves it: this would BUG up top */
        return IRQ_HANDLED;
}

PREDICT FIRST: fill in both rows before running. Then check ps -eo pid,comm | grep irq/ and find your thread by name.

5. Failure mode

Concept 5: Timers

1. What problem it solves

"Do this later" and "poll this every N milliseconds" — without a thread spinning.

2. Where it exists in the kernel

ls kernel/time/
rg -n "timer_setup|mod_timer|timer_delete_sync|timer_shutdown_sync" include/linux/timer.h
rg -n "hrtimer_start|enum hrtimer_restart" include/linux/hrtimer.h | head
grep CONFIG_HZ= ~/kernel/build/.config

Warning: The timer teardown API was renamed during the 6.x series — del_timer_sync() became timer_delete_sync(), and timer_shutdown_sync() was added for final teardown. Grep before you write it, and if your out-of-tree module stops compiling on a newer kernel, this is exactly the kind of change to find with git log -S'del_timer_sync'.

3. The two kinds

/* timer_list */
static void lab_timer_fn(struct timer_list *t)
{
        struct mylab *ml = from_timer(ml, t, timer);
        /* SOFTIRQ context: no sleeping, no mutex, no GFP_KERNEL. */
        schedule_work(&ml->work);       /* defer anything real to a workqueue */
        mod_timer(&ml->timer, jiffies + msecs_to_jiffies(500));
}

timer_setup(&ml->timer, lab_timer_fn, 0);
mod_timer(&ml->timer, jiffies + msecs_to_jiffies(500));

/* hrtimer */
static enum hrtimer_restart lab_hr_fn(struct hrtimer *h)
{
        /* HARDIRQ context by default. Even more restricted. */
        hrtimer_forward_now(h, ms_to_ktime(1));
        return HRTIMER_RESTART;
}

And the ways to wait, which are not timers:

Tip: usleep_range() rather than msleep() for short sleeps, and give a real range. A range lets the timer subsystem coalesce your wakeup with another one, which matters enormously for idle-power behavior. A driver that wakes the CPU every 10 ms with a zero-slack timer is a laptop battery complaint.

4. Experiment

CLAIM. msleep(1) does not sleep for one millisecond, and the error depends on CONFIG_HZ.

METHOD.

for (i = 0; i < 10; i++) {
        u64 t0 = ktime_get_ns();
        msleep(1);
        pr_info("msleep(1)          took %llu us\n", (ktime_get_ns() - t0) / 1000);
}
for (i = 0; i < 10; i++) {
        u64 t0 = ktime_get_ns();
        usleep_range(1000, 1100);
        pr_info("usleep_range(1000) took %llu us\n", (ktime_get_ns() - t0) / 1000);
}

PREDICT FIRST: how long does msleep(1) actually take on a CONFIG_HZ=250 kernel? On CONFIG_HZ=1000? Check yours with grep CONFIG_HZ= ~/kernel/build/.config before you run it.

5. Failure mode

Concept 6: Teardown — The Part Everyone Gets Wrong

1. What problem it solves

Every deferred mechanism above creates something that will run later, referencing your data. When your module unloads or your device is removed, "later" may be after that data is freed and after your code is unmapped.

This is the single most common source of use-after-free in a first driver, it is nondeterministic, and KASAN is what turns it from a mystery into a report.

2. The ordering, and it is not negotiable

   TEARDOWN, IN ORDER. Every step exists because skipping it is a bug.

   1. STOP THE SOURCE.
        Tell the hardware to stop generating interrupts.
        free_irq() / devm handles it — this also WAITS for a running handler.

   2. PREVENT NEW WORK FROM BEING QUEUED.
        Set a `dying` flag under the lock the queuers take, or unregister the
        interface user space uses. A cancel is useless if something can requeue
        after it.

   3. CANCEL AND WAIT FOR WHAT IS ALREADY PENDING.
        cancel_work_sync(&w)            NOT cancel_work()
        cancel_delayed_work_sync(&dw)
        timer_delete_sync(&t)  /  timer_shutdown_sync(&t)
        hrtimer_cancel(&h)
        tasklet_kill(&tl)
        destroy_workqueue(wq)           (drains, then frees)

   4. NOW free the memory those callbacks referenced.

   5. Unregister anything user space can still reach — or rather, do this
      FIRST if user space can trigger new work. See below.

The _sync suffix is the whole point: cancel_work() removes it from the queue if it has not started, and returns immediately if it is running. cancel_work_sync() waits.

The requeue trap, which is why step 2 exists:

/* BROKEN: the work function requeues itself, so cancel races with it. */
static void poll_fn(struct work_struct *w)
{
        struct mylab *ml = container_of(to_delayed_work(w), struct mylab, poll);
        do_poll(ml);
        schedule_delayed_work(&ml->poll, HZ);      /* ← requeues */
}
static void lab_remove(...)
{
        cancel_delayed_work_sync(&ml->poll);       /* may cancel, then it requeues */
        kfree(ml);                                 /* → use-after-free */
}

/* CORRECT: */
static void poll_fn(struct work_struct *w)
{
        struct mylab *ml = container_of(to_delayed_work(w), struct mylab, poll);

        do_poll(ml);
        if (!READ_ONCE(ml->stopping))
                schedule_delayed_work(&ml->poll, HZ);
}
static void lab_remove(...)
{
        WRITE_ONCE(ml->stopping, true);            /* 2. no new work */
        cancel_delayed_work_sync(&ml->poll);       /* 3. drain */
        kfree(ml);                                 /* 4. safe */
}

And the module-lifetime trap: if user space holds an open file descriptor to your device, the module's reference count is held by fops.owner = THIS_MODULE and rmmod will fail with -EBUSY — which is the good case. The bad case is a callback with no such reference, running after the module's text is gone, jumping into unmapped memory.

3. Experiment

CLAIM. Wrong teardown order produces a use-after-free that is invisible without KASAN and obvious with it.

METHOD. Write the broken version above — a self-requeueing delayed work, cancelled without a stopping flag — and run it in a loop:

# lab-fast (no KASAN):
for i in $(seq 50); do insmod ./lab.ko; sleep 0.05; rmmod lab; done; dmesg | tail

# lab-paranoid (KASAN):
for i in $(seq 50); do insmod ./lab.ko; sleep 0.05; rmmod lab; done; dmesg | tail -40

PREDICT FIRST: of 50 iterations, how many produce a visible symptom without KASAN? With KASAN?

RESULT. Without KASAN you will usually see nothing at all. With KASAN you get:

BUG: KASAN: slab-use-after-free in poll_fn+0x2c/0x120 [lab]
Read of size 8 at addr ffff888... by task kworker/1:2/89

Allocated by task 412:   ... lab_init+...
Freed by task 415:       ... lab_exit+...

Three stack traces — where it was used, where it was allocated, where it was freed. That is the report that turns a week of guessing into a five-minute fix, and it is why lab-paranoid exists.

4. Failure mode

Choosing: The Decision Table

Validation / Self-check

1. Why must an interrupt handler be short? Name three distinct costs of a long one.
2. What must a shared-line handler return when the interrupt was not its device's, and what happens if it gets that wrong?
3. Name the ten softirq types (or the mechanism for finding them) and explain why you cannot add an eleventh.
4. When does a softirq run in ksoftirqd rather than inline, and what does seeing ksoftirqd at 100% tell you?
5. Why are tasklets being phased out? What should new code use instead?
6. What does queue_work() return when the item is already queued, and how do you handle "another event happened while it was pending"?
7. What is WQ_MEM_RECLAIM for? Describe the deadlock it prevents.
8. Compare a threaded IRQ and a workqueue on four axes. When is each right?
9. What does IRQF_ONESHOT do, and when is it mandatory?
10. In what context does a timer_list callback run? An hrtimer callback? What does each forbid?
11. Why is usleep_range() preferred over msleep() for short sleeps?
12. Give the five steps of correct teardown, in order, and say what breaks if you skip each.
13. Why is cancel_work_sync() not sufficient on its own for a self-requeueing work item?
14. Your module unloads cleanly 49 times out of 50 and oopses once. What is the most likely structure of the bug, and which config option would have told you on the first run?

Next: The Device Model — how a driver finds its hardware, and who cleans up after it.