The virtio Block Device

The block device is your first concrete virtio device, and the one whose data path is easiest to hold in your head: a guest reads and writes a virtual disk, and on the host side those reads and writes become pread/pwrite calls against a single backing file. Everything you learned about virtqueues — the descriptor chain, the kick, the used ring, the interrupt — applies here with one queue and one canonical chain shape: header, data, status. If you can trace that three-descriptor chain from a guest dd to a host pwrite and back, you understand the block device.

This chapter covers the request queue and the request chain; the four request types; how a request becomes file I/O at byte offset sector * 512; the two I/O engines (synchronous pread/pwrite vs io_uring); rate limiting; how activate wires the device into the VMM thread's epoll loop via an ioeventfd; and the is_root_device / is_read_only flags. After it you will be able to read Firecracker's process_queue, explain why a malformed request returns a status byte rather than crashing the host, and reason about which io_engine to pick under which workload.

Note: The block device is virtio device type ID 2, and it exposes exactly one request virtqueue. Every read, write, flush, and ID query for that disk multiplexes over that single queue. The guest serialises concurrency into the queue (many in-flight descriptor chains); the host serialises completion back through the one used ring. There is no second "control" queue as on some other devices — confirm the queue count on your branch with the rg below.

Where the block device lives

# The device's source tree (may split into virtio/ and vhost_user/ — verify on your branch).
find . -type d -path '*devices/virtio/block*'
rg -n "virtio/block" --files-with-matches src/vmm/src | head

# The device struct, its feature set, and queue count.
rg -n "struct (Block|VirtioBlock)\b|TYPE_BLOCK|NUM_QUEUES|QUEUE_SIZE|avail_features" \
  src/vmm/src/devices/virtio/block/

Confirm three things before reading further: the device type constant resolves to 2, the queue count is 1, and the struct holds a backing-file handle plus a configuration block. The block module may be split into a "real" virtio backend (block/virtio/) and a vhost_user backend (block/vhost_user/) where the data path lives in a separate process — verify on your branch; this chapter describes the in-process virtio backend, which is the one Firecracker drives by default.

 guest kernel (virtio-blk driver)
        │  builds desc chain, kicks QueueNotify
        ▼
 ioeventfd ──► VMM thread epoll loop ──► Block::process_queue
        │                                      │
        │                                      ▼
        │                          io_engine: pread/pwrite (Sync)
        │                                or  io_uring submit/reap (Async)
        │                                      │
        │                                      ▼
        │                          host file at path_on_host
        ▼                                      │
 irqfd ◄────────── add_used + interrupt ◄──────┘

The request queue and the request chain

# The chain layout: header struct, request type constants, status byte.
rg -n "virtio_blk_outhdr|VIRTIO_BLK_T_IN|VIRTIO_BLK_T_OUT|VIRTIO_BLK_T_FLUSH|VIRTIO_BLK_T_GET_ID|\
VIRTIO_BLK_S_OK|VIRTIO_BLK_S_IOERR|VIRTIO_BLK_S_UNSUPP" src/vmm/src/devices/virtio/block/
# Where the chain is parsed into a Request.
rg -n "struct Request\b|fn parse\b|RequestType|fn process_queue|status" src/vmm/src/devices/virtio/block/

A block request is a single descriptor chain, and it always has the same three-part shape. The first descriptor is the header, device-readable (no WRITE flag): a fixed C struct the guest fills in. The middle is one or more data descriptors. The last is a 1-byte status, device-writable (WRITE flag set), into which the device reports the outcome.

struct virtio_blk_outhdr {
    le32 type;      // VIRTIO_BLK_T_* — what kind of request
    le32 reserved;  // ioprio, historically; ignored
    le64 sector;    // starting sector, in 512-byte units
};

flowchart LR
    Avail["avail.ring[i] = head"] --> H
    subgraph chain
      H["desc[0]: header\nvirtio_blk_outhdr (16B)\ndevice-readable"] --> D["desc[1..]: data buffer(s)\nread ⇒ WRITE flag (device fills)\nwrite ⇒ no flag (guest fills)"]
      D --> S["desc[last]: status (1B)\nWRITE flag (device fills)"]
    end
    S --> Used["used.ring[j] = {id:head, len:bytes_written}"]

The direction bit on the data descriptor is the crux, and it is exactly the VIRTIO_DESC_F_WRITE confusion from virtqueues: for a read (VIRTIO_BLK_T_IN) the data buffer is device-writable (the device fills it from disk); for a write (VIRTIO_BLK_T_OUT) the data buffer is device-readable (the guest filled it, the device flushes it to disk). The status byte is always device-writable. The device parses the header, validates that the chain's descriptor directions match the request type, performs the I/O, writes one of VIRTIO_BLK_S_OK / VIRTIO_BLK_S_IOERR / VIRTIO_BLK_S_UNSUPP into the status byte, and calls add_used with the byte count it produced.

Warning: The header sector, the data len, and the descriptor directions are all guest-controlled. The device must validate them — a too-long write, a read into a non-writable buffer, an out-of-range sector — and respond with VIRTIO_BLK_S_IOERR, never trust them into a host syscall blindly. A bug here is a guest→host issue, not a guest inconvenience. Read the parse path as security-critical code.

Request types

rg -n "RequestType|VIRTIO_BLK_T_|GetDeviceID|Flush|VIRTIO_BLK_F_FLUSH|fn execute\b" \
  src/vmm/src/devices/virtio/block/

Read and write are the workhorses. FLUSH is how the guest forces durability and is only honoured if the VIRTIO_BLK_F_FLUSH feature was negotiated — without it the guest assumes writes are already durable. GET_ID lets the guest read the configured drive_id (this is why your disk shows a serial in the guest). Any type the device does not recognise gets VIRTIO_BLK_S_UNSUPP in the status byte — a clean rejection, not a panic.

From request to file: pread/pwrite at sector × 512

# The backing file and the offset math.
rg -n "path_on_host|pread|pwrite|read_at|write_at|seek|SECTOR_SHIFT|SECTOR_SIZE|sector" \
  src/vmm/src/devices/virtio/block/

The backend is a single host file named by path_on_host (the rootfs image, or an extra disk). There is no block layer, no partition logic, no LVM — virtio-blk hands the guest a flat array of 512-byte sectors, and the host maps sector N to byte offset N * 512 in that file. A read of sector 2048 for 8 sectors is pread(fd, buf, 4096, 2048 * 512). The data buffers come straight out of guest memory via bounds-checked accessors — the device never copies into a host bounce buffer for the common path; it reads/writes the guest pages directly through vm-memory.

 guest sees:  [ sector 0 ][ sector 1 ][ sector 2 ] ...   (512B each)
                    │            │            │
 host file:   byte 0       byte 512      byte 1024  ...   path_on_host
                    └─ pread/pwrite(fd, buf, len, sector * 512) ─┘

Tip: SECTOR_SIZE is 512 in the virtio-blk protocol regardless of the host file's real block size. The guest's logical sector size, the host filesystem's block size, and the image's internal layout are three different things. Capacity advertised in config space is also in 512-byte sectors (see below), so a 1 GiB disk advertises 2097152.

The disk capacity is exposed in the device's config space at offset 0x100, as a count of 512-byte sectors, computed from the backing file's length at activation. The guest reads it to size the block device.

rg -n "0x100|config_space|capacity|len\(\)|fn read_config|fn write_config|metadata" \
  src/vmm/src/devices/virtio/block/

io_engine: Sync vs Async (io_uring)

rg -n "io_engine|IoEngine|enum (Engine|FileEngine)|\bSync\b|\bAsync\b|io_uring|FixedVec|submit|reap|complete" \
  src/vmm/src/devices/virtio/block/

Two implementations turn a parsed request into actual disk I/O. They produce identical guest-visible behaviour; they differ entirely in how the host syscalls are issued.

The Sync engine is the simple one: in process_queue the device does the pread/pwrite right there on the VMM thread, writes the status byte, adds the used element, and moves to the next chain. The cost is that a slow syscall stalls the one thread that drives every device on that VM.

The Async engine wraps io_uring. Instead of issuing the syscall inline, process_queue submits the I/O (a submission-queue entry referencing the guest buffer) and returns; the request is tracked in flight (look for a fixed-capacity ring such as FixedVec holding pending requests). io_uring signals a completion eventfd, which is also registered with the EventManager, so a second epoll wakeup drives the reap path: pull completion-queue entries, match each back to its pending request, write the status byte, and add the used element. This is why the async path needs two registered fds — the queue ioeventfd and the io_uring completion fd. See io engines for the full tradeoff discussion.

sequenceDiagram
    participant G as Guest driver
    participant E as EventManager (VMM thread)
    participant B as Block::process_queue
    participant U as io_uring
    G->>E: kick (ioeventfd)
    E->>B: process_queue()
    B->>U: submit SQE (pread/pwrite on guest buffer)
    Note over B: returns without blocking
    U-->>E: completion eventfd fires
    E->>B: process_completions()
    B->>B: reap CQE → write status byte → add_used
    B->>G: interrupt (irqfd)

Activation: how the device joins the run loop

rg -n "fn activate|ACTIVATE|DRIVER_OK|ioeventfd|register|EventManager|fn process_queue|rate_limiter|timer" \
  src/vmm/src/devices/virtio/block/

A virtio device is inert until the guest driver finishes negotiation and writes DRIVER_OK through the MMIO transport. That write triggers device.activate(). Activation is the moment the device wires its fds into the VMM thread's epoll loop:

1. The transport set up an ioeventfd so the guest's QueueNotify write (the kick) becomes an eventfd signal instead of a VM exit. activate registers that eventfd with the EventManager.
2. If a rate limiter is configured, its timer eventfd is registered too, so the device can be re-woken when tokens replenish.
3. For the async engine, the io_uring completion eventfd is registered.

After activation, processing is entirely event-driven: the guest kicks, the ioeventfd fires, the epoll loop dispatches to the device's handler, and process_queue drains the available ring — parse, execute, status, add_used — until the queue is empty, then injects one interrupt via irqfd. No polling, no dedicated thread; the block device shares the one VMM thread with every other device, which is exactly why a blocking Sync syscall is a tax on the whole VM.

Note: Pre-DRIVER_OK, a kick that somehow arrives is ignored — the device has no registered handler yet. If a guest "hangs" before its disk appears, suspect feature negotiation or the transport's activate, not the block code itself.

Rate limiting

rg -n "rate_limiter|RateLimiter|TokenBucket|bandwidth|ops|consume|fn process_queue|timer" \
  src/vmm/src/devices/virtio/block/

Each block device may carry a rate limiter with two independent token buckets: ops/s (IOPS) and bandwidth (bytes/s). Before the device executes a request it asks the limiter for tokens. If a bucket is empty, process_queue stops — it does not spin or drop the request. The limiter's timer eventfd (registered at activation) re-arms; when it fires, the epoll loop calls back into the device, which resumes draining the queue from where it paused. The guest sees backpressure as latency, exactly as a slow physical disk would. This is how one noisy microVM is prevented from saturating shared host I/O.

Read-only and root flags

rg -n "is_root_device|is_read_only|VIRTIO_BLK_F_RO|root_device|read_only" \
  src/vmm/src/devices/virtio/block/

A drive is configured through the API before boot, and the path or rate limiter can be PATCHed at runtime (e.g. to swap a rootfs or change throttling) — the is_root_device / is_read_only flags are fixed at create time.

API=/tmp/firecracker.socket
# Create the root drive (canonical shape).
curl -X PUT --unix-socket $API \
  --data '{"drive_id":"rootfs","path_on_host":"./ubuntu-24.04.ext4","is_root_device":true,"is_read_only":false}' \
  http://localhost/drives/rootfs
# An extra, read-only data disk with an explicit io_engine.
curl -X PUT --unix-socket $API \
  --data '{"drive_id":"scratch","path_on_host":"./data.ext4","is_root_device":false,"is_read_only":true,"io_engine":"Async"}' \
  http://localhost/drives/scratch
# Hot-swap the backing file at runtime.
curl -X PATCH --unix-socket $API \
  --data '{"drive_id":"scratch","path_on_host":"./data-v2.ext4"}' \
  http://localhost/drives/scratch

Snapshot and metrics

rg -n "Persist|fn save\b|fn restore\b|BlockState|BlockConstructorArgs" src/vmm/src/devices/virtio/block/
rg -n "BlockDeviceMetrics|block.*metrics|METRICS|read_count|write_count|flush_count" src/vmm/src/devices/virtio/block/

The device implements the Persist trait so it can be serialised into a snapshot and reconstructed on restore — the saved state holds the configuration (path, flags, rate-limiter config) and the virtqueue cursors, not the disk contents (those stay in the backing file). It also emits per-device metrics (BlockDeviceMetrics): read/write/flush counts, byte counters, error counts, and queue-event counts, keyed by drive_id, so a busy or erroring disk is visible in telemetry (metrics).

Reading exercise

# 1. The device struct, queue count, type ID, advertised features.
rg -n "struct (Block|VirtioBlock)\b|TYPE_BLOCK|NUM_QUEUES|avail_features|VIRTIO_BLK_F_" \
  src/vmm/src/devices/virtio/block/

# 2. The request header, the four request types, the status byte values.
rg -n "virtio_blk_outhdr|VIRTIO_BLK_T_|VIRTIO_BLK_S_|struct Request|fn parse" \
  src/vmm/src/devices/virtio/block/

# 3. The parse → execute → status → add_used path.
rg -n "fn process_queue|fn execute|fn parse|add_used|status|sector" \
  src/vmm/src/devices/virtio/block/

# 4. The two io engines and the io_uring submit/reap split.
rg -n "io_engine|IoEngine|\bSync\b|\bAsync\b|io_uring|submit|reap|FixedVec" \
  src/vmm/src/devices/virtio/block/

# 5. Activation: which fds get registered with the EventManager.
rg -n "fn activate|register|ioeventfd|completion|timer|rate_limiter" \
  src/vmm/src/devices/virtio/block/

# 6. On a booted guest, watch real requests:
#    in guest:  dd if=/dev/vdb of=/dev/null bs=1M count=64 iflag=direct
#    on host:   strace -p <firecracker-pid> -e pread64,pwrite64,io_uring_enter

Answer:

1. What is the block device's virtio type ID, and how many virtqueues does it expose? Why is one enough?
2. Lay out the three parts of a request chain. For a read vs a write, which descriptor carries the VIRTIO_DESC_F_WRITE flag, and why?
3. Take sector = 4096, an 8-sector read. What host syscall, at what byte offset, for how many bytes?
4. Sync vs Async: what does process_queue do differently in each, and how many epoll-registered fds does the async path require?
5. A rate-limited disk runs out of bandwidth tokens mid-drain. What does the device do, and what re-wakes it?
6. With is_read_only set, name both mechanisms that stop the guest from writing.

Common bugs and symptoms

Validation: prove you understand this

1. Draw the header/data/status chain and annotate, for a block read, which descriptors are device-readable and which are device-writable. Tie each back to a VIRTIO_DESC_F_WRITE flag.
2. Trace a single VIRTIO_BLK_T_OUT (write) of one sector from the guest kick to the host pwrite and back to the guest interrupt, naming the ioeventfd, process_queue, add_used, and the irqfd.
3. Given path_on_host pointing at a 2 GiB image, what capacity (in sectors) does the device advertise, and where does the guest read it?
4. Explain the Sync→Async difference in terms of threads and fds: what blocks where, and why the async path needs a completion eventfd registered with the EventManager.
5. A reviewer sees a PR where process_queue issues a pwrite but, on the error path, returns without writing the status byte or calling add_used. Describe the two distinct ways the guest now misbehaves.
6. Explain how is_read_only is enforced in two layers, and why advertising VIRTIO_BLK_F_RO alone would be insufficient against a hostile guest.

Next: The virtio Net Device and TAP — two queues instead of one, packets instead of sectors, and a host TAP fd instead of a backing file; the same activate/process/used-ring spine you just learned, applied to the network data path.