The Block Layer

Three concepts in the six-part treatment: the bio, blk-mq, and schedulers, merging, and the device.

The block layer sits between filesystems, which think in files and offsets, and drivers, which think in sectors and hardware queues. Its job is to turn one into the other while merging, ordering, and accounting along the way.

Concept 1: The bio

1. What problem it solves

A filesystem wants to say: "read these 64 KB of this file into these pages". The pages are scattered in physical memory — the page cache allocated them wherever it could — and the device wants a list of physical addresses and a starting sector.

The bio is that request: a target device, a starting sector, and a vector of (page, offset, length) segments. It exists so nothing has to copy data into a contiguous buffer first.

2. Where it exists in the kernel

rg -n "struct bio \{" -A 40 include/linux/blk_types.h
rg -n "struct bio_vec \{" -A 8 include/linux/bvec.h
rg -n "submit_bio\b" -A 30 block/blk-core.c | head -40
rg -n "enum req_op|REQ_FUA|REQ_PREFLUSH" include/linux/blk_types.h | head -25

3. The structure

   struct bio
     ├── bi_bdev         which block device
     ├── bi_opf          the operation (READ/WRITE/DISCARD/FLUSH) OR'd with
     │                   flags: REQ_SYNC, REQ_FUA, REQ_PREFLUSH, REQ_META...
     ├── bi_iter         { bi_sector, bi_size, bi_idx, bi_bvec_done }
     │                   ← a CURSOR. Advancing it is how a bio gets split
     │                     without copying anything.
     ├── bi_io_vec[]     the segments:
     │       [0] (page A, offset 0,    len 4096)
     │       [1] (page B, offset 0,    len 4096)   ← physically scattered
     │       [2] (page C, offset 0,    len 4096)
     ├── bi_end_io       the completion callback
     ├── bi_private      the caller's context
     └── bi_status       the result

   A bio is SUBMITTED, not called. submit_bio() returns immediately; the
   completion callback runs later, in interrupt or softirq context.

The bi_iter cursor is the elegant part: a bio can be split (the device has a maximum transfer size) or chained by advancing a cursor and cloning a small header, with no data movement.

Flags worth knowing, because they are how durability is expressed at this layer:

REQ_PREFLUSH and REQ_FUA are how fsync() becomes a hardware operation, and they are the subject of the durability chapter.

4. Experiment

CLAIM. Bios are visible, they carry flags that explain what the filesystem was doing, and they get split and merged on the way to the device.

METHOD.

# Every bio, with its operation and flags:
sudo bpftrace -e '
  tracepoint:block:block_bio_queue {
      @[args.rwbs, args.nr_sector] = count();
  }' &
dd if=/dev/zero of=/tmp/b bs=1M count=64 oflag=direct 2>/dev/null
sync
kill %1

The rwbs field is a compact string of the flags: R read, W write, S sync, F flush, FUA, A readahead, M metadata. Reading it tells you what the layer above intended.

# Splitting and merging, both visible:
sudo bpftrace -e '
  tracepoint:block:block_split          { @splits = count(); }
  tracepoint:block:block_bio_backmerge  { @merges = count(); }
  tracepoint:block:block_rq_issue       { @issued[args.rwbs] = count(); }' &
dd if=/dev/zero of=/tmp/b bs=4k count=20000 2>/dev/null; sync; kill %1

# The device's limits, which are why splits happen:
DEV=$(lsblk -no PKNAME "$(df --output=source /tmp | tail -1)" 2>/dev/null || echo sda)
cat /sys/block/$DEV/queue/max_sectors_kb \
    /sys/block/$DEV/queue/max_segments \
    /sys/block/$DEV/queue/logical_block_size 2>/dev/null

PREDICT FIRST: 20,000 4 KB buffered writes. How many bios reach block_bio_queue? How many block_rq_issue events? The ratio is the merging you just measured.

5. Failure mode

Concept 2: blk-mq

1. What problem it solves

An NVMe device has dozens of hardware queues and can service a million IOPS. A single request queue with one lock — which is what the block layer had until 2013 — cannot feed that: every CPU contends for one cache line on every I/O.

blk-mq (multi-queue) replaces it with per-CPU software queues feeding a small number of hardware queues that map onto the device's real queues.

2. Where it exists in the kernel

ls block/blk-mq*.c
rg -n "struct blk_mq_tag_set \{" -A 25 include/linux/blk-mq.h
rg -n "struct blk_mq_ops \{" -A 30 include/linux/blk-mq.h
rg -n "blk_mq_submit_bio" -A 40 block/blk-mq.c | head -50
$EDITOR Documentation/block/blk-mq.rst

3. The structure

   submit_bio()
        │
   blk_mq_submit_bio()
        │
        ├── try to MERGE into an existing request (plug, software queue,
        │   or the I/O scheduler) -- merging is the single biggest win
        │
        ├── otherwise allocate a REQUEST, which requires a TAG
        │      Tags are a bounded per-device resource: the number in
        │      flight is capped by the tag set. Running out means WAITING,
        │      and that is where queue-depth backpressure comes from.
        │
        ├── SOFTWARE QUEUE (ctx): per CPU. No lock contention.
        │
        ├── HARDWARE QUEUE (hctx): maps to a real device queue.
        │      The ctx -> hctx mapping is usually by CPU, so a request
        │      is usually issued on the CPU that submitted it, and its
        │      completion interrupt lands on that CPU too.
        │
        └── mq_ops->queue_rq()  ── the DRIVER. Writes a command into the
                                   device's submission queue, rings a
                                   doorbell.
   ...
   completion interrupt ──▶ blk_mq_complete_request() ──▶ bio->bi_end_io()

The plug is worth knowing about: a task can accumulate requests in a per-task list (blk_start_plug() / blk_finish_plug()) so they can be merged before being handed to the queue. Filesystems plug around writeback for exactly this reason.

rg -n "blk_start_plug|blk_finish_plug" mm/ fs/ | head
DEV=nvme0n1     # or your device
ls /sys/block/$DEV/mq/ 2>/dev/null
cat /sys/block/$DEV/queue/nr_requests /sys/block/$DEV/queue/scheduler 2>/dev/null

4. Experiment

CLAIM. The queue structure is visible, and queue depth changes latency in a way you can measure.

METHOD.

DEV=$(lsblk -no PKNAME "$(df --output=source /tmp | tail -1)" 2>/dev/null || echo vda)
ls /sys/block/$DEV/mq/            # one directory per hardware queue
cat /sys/block/$DEV/mq/0/nr_tags 2>/dev/null
cat /sys/block/$DEV/queue/nr_requests

# Latency vs. queue depth:
sudo bpftrace -e '
  tracepoint:block:block_rq_issue    { @s[args.dev, args.sector] = nsecs; @inflight = count(); }
  tracepoint:block:block_rq_complete /@s[args.dev, args.sector]/ {
      @us = hist((nsecs - @s[args.dev, args.sector]) / 1000);
      delete(@s[args.dev, args.sector]); }' &

fio --name=d1  --rw=randread --bs=4k --size=256M --filename=/tmp/io --direct=1 --iodepth=1  2>/dev/null | grep -E 'IOPS|lat'
fio --name=d32 --rw=randread --bs=4k --size=256M --filename=/tmp/io --direct=1 --iodepth=32 2>/dev/null | grep -E 'IOPS|lat'
kill %1

PREDICT FIRST: going from queue depth 1 to 32, what happens to (a) IOPS and (b) per-I/O latency? They move in opposite directions, and understanding why is the point.

5. Failure mode

Concept 3: Schedulers, Merging, and the Device

1. What problem it solves

Even with per-CPU queues, which requests to issue and in what order still matters — differently for a rotating disk (seek time dominates) and an NVMe SSD (there is no seek, and the device reorders internally anyway).

Hence: pluggable I/O schedulers, and none being a perfectly good choice.

2. Where it exists in the kernel

ls block/*iosched* block/mq-deadline.c block/bfq-iosched.c block/kyber-iosched.c 2>/dev/null
cat /sys/block/*/queue/scheduler
$EDITOR Documentation/block/deadline-iosched.rst 2>/dev/null || ls Documentation/block/

3. The choices

DEV=vda
cat /sys/block/$DEV/queue/scheduler                # [brackets] mark the active one
echo mq-deadline | sudo tee /sys/block/$DEV/queue/scheduler
ls /sys/block/$DEV/queue/iosched/                  # per-scheduler tunables

Merging is where the real win is, not ordering. Two adjacent 4 KB writes becoming one 8 KB request halves the per-request overhead, and a sequential write of 1 MB in 4 KB pieces should reach the device as a handful of large requests, not 256 small ones.

   FRONT MERGE: the new bio ends where an existing request begins
   BACK MERGE:  the new bio begins where an existing request ends  (common)

   Merging happens at three places, cheapest first:
     1. the task's PLUG list
     2. the software queue
     3. the I/O scheduler's queue

4. Where the request finally goes

For NVMe, which is worth knowing because it is the shape of all modern storage:

   nvme_queue_rq()                         drivers/nvme/host/pci.c
     ├── build an NVMe command (opcode, LBA, length, PRP/SGL for the data)
     ├── map the bio's pages for DMA        ← dma_map_sg()
     ├── write the command into the submission queue in HOST memory
     └── ring a DOORBELL register           ← a single MMIO write

   ...the device DMAs the data, then posts a completion...

   nvme_irq() / polled completion
     └── blk_mq_complete_request() ──▶ bio->bi_end_io() ──▶ the filesystem

The device reads its own commands out of host memory by DMA. The doorbell write is the only MMIO on the fast path, which is why NVMe scales the way it does.

5. Experiment

CLAIM. The scheduler choice matters enormously on some devices and not at all on others, and merging is measurable.

METHOD.

DEV=$(lsblk -no PKNAME "$(df --output=source /tmp | tail -1)")

for sched in none mq-deadline bfq; do
  echo "$sched" | sudo tee /sys/block/$DEV/queue/scheduler >/dev/null 2>&1 || continue
  echo "== $sched"
  # A latency-sensitive reader competing with a bulk writer:
  dd if=/dev/zero of=/tmp/bulk bs=1M count=512 oflag=direct 2>/dev/null &
  fio --name=r --rw=randread --bs=4k --size=64M --filename=/tmp/io \
      --direct=1 --runtime=10 --time_based 2>/dev/null | grep -E 'IOPS|clat.*99'
  wait
done

PREDICT FIRST: on an NVMe device, how much does the scheduler change the reader's p99 latency? On a rotating disk (or a throttled virtual one)?

And merging:

sudo bpftrace -e '
  tracepoint:block:block_bio_queue      { @bios = count(); }
  tracepoint:block:block_bio_backmerge  { @merged = count(); }
  tracepoint:block:block_rq_issue       { @requests = count(); }' &
dd if=/dev/zero of=/tmp/seq bs=4k count=50000 2>/dev/null; sync; kill %1

PREDICT FIRST: 50,000 sequential 4 KB writes. How many bios, how many merges, how many requests actually issued to the device?

6. Failure mode

Validation / Self-check

1. What is a bio, and why does it hold a vector of segments rather than a buffer?
2. What is bi_iter for, and how does it make splitting free?
3. Name five REQ_* flags and what each tells the layer below.
4. Why did blk-mq replace the single request queue? What specifically did not scale?
5. What is a tag, and what happens when they run out?
6. What is a plug, and which layer uses it and why?
7. Where can a bio be merged into an existing request? List the three places in order of cost.
8. Compare none, mq-deadline, and bfq. When is each right?
9. Why is none a reasonable choice on NVMe when it does no ordering at all?
10. Trace an NVMe write from queue_rq to the completion callback, naming the DMA and the MMIO.
11. Going from queue depth 1 to 32: what happens to IOPS and to per-I/O latency? Why do they move oppositely?
12. Why does block-layer ordering not imply on-media ordering, and what does constrain it?

Next: Durability and Filesystems — when is the data actually safe?