Lab 1: Virtio-Block, End to End

Background

This is a trace-it-and-measure lab. You will build a backing file from scratch, attach it as a second drive to a microVM, drive real I/O against it with dd and fio from inside the guest, and then follow a single block request through every hop: the guest driver's descriptor chain, the MMIO kick, the ioeventfd, the device's queue handler on the VMM thread, the host pread/pwrite on the backing file's fd, the used-ring completion, and the injected interrupt. Then you will switch the block device's io_engine from Sync (plain pread/pwrite) to io_uring (Async) and measure the difference with real numbers and real syscall traces.

You traced block once in Level 7 Lab 7.1. This lab goes further: you control the backing file (so you can correlate exact offsets), you run a proper fio workload (so the numbers mean something), and you compare the two io engines head to head. The goal is that "a guest disk read" becomes, in your head, a precise sequence of named functions and one host syscall at a known offset.

Why This Lab Matters for Contributors

  • Almost every block-device issue — a hang, a wrong byte count, a rate-limiter accounting bug, an io_uring-vs-Sync discrepancy, a partial-read edge case — is debugged by walking exactly this path and reading exactly these syscalls. The issue-roadmap virtio stage is full of them.
  • The Sync-vs-io_uring comparison is a real engineering decision operators make; you should be able to explain the trade-off with measurements, not adjectives. See the io-engines engineering chapter.
  • It cements three deep dives: virtqueues (the rings), virtio-transport-mmio (the kick), and virtio-block (the request semantics and engines).

Prerequisites

  • The masterclass overview read; Level 7 complete.
  • A built firecracker and a kernel + rootfs you can boot (Lab 1.3).
  • fio available inside the guest (most Ubuntu rootfs images have it or can apt install fio; dd always works as a fallback).
  • Verify the block device, the queue, the io engines, and the host I/O calls exist on your branch:
# All must return hits. If a path is empty, the layout moved — find by role.
rg -l "impl VirtioDevice for"            src/vmm/src/devices/virtio/block/
rg -n "fn process_queue|fn process|fn handle_.*event|request_queue" \
   src/vmm/src/devices/virtio/block/
rg -n "io_engine|IoEngine|FileEngine|Sync|Async|io_uring" src/vmm/src/devices/virtio/block/
rg -n "pread|pwrite|read_exact_at|write_all_at|read_at|write_at" \
   src/vmm/src/devices/virtio/block/

The Path You Are Tracing

sequenceDiagram
    participant GK as Guest kernel (virtio-blk driver)
    participant GM as Guest memory (virtqueue)
    participant KVM as KVM
    participant EM as EventManager (VMM thread)
    participant BLK as Block device (VirtioDevice)
    participant FS as Backing file (host fd)

    GK->>GM: 1. build chain: header | data | status
    GK->>GM: 2. avail.ring[idx]=head; avail.idx++
    GK->>KVM: 3. MMIO write to QueueNotify (the kick)
    KVM-->>EM: 4. ioeventfd signals → epoll wakes VMM thread
    EM->>BLK: 5. queue handler runs (process_queue)
    BLK->>GM: 6. Queue::pop → walk chain, bounds-check
    BLK->>BLK: 7. parse virtio_blk_req (type, sector)
    BLK->>FS: 8. pread/pwrite (Sync) OR io_uring submit (Async)
    FS-->>BLK: 9. bytes / completion
    BLK->>GM: 10. write data into WRITE descriptor (on a read)
    BLK->>GM: 11. write status byte = VIRTIO_BLK_S_OK
    BLK->>GM: 12. add_used(head, len); used.idx++
    BLK->>KVM: 13. raise IRQ via irqfd; InterruptStatus |= USED_RING
    KVM-->>GK: 14. guest interrupt; driver reaps used ring

Each numbered step below maps to this diagram.

Step-by-Step Tasks

Step 1: Build a backing file you control

A virtio-block drive in Firecracker is just a flat file on the host exposed to the guest as a block device. Make one, give it recognizable contents so you can spot it in a trace, and format it.

WORK=/tmp/blk-lab; mkdir -p "$WORK"; cd "$WORK"

# A 256 MiB scratch disk, pre-allocated (not sparse) so reads actually hit the file.
dd if=/dev/zero of=scratch.ext4 bs=1M count=256 status=progress

# Put a filesystem on it and seed a marker file at a known place.
mkfs.ext4 -q scratch.ext4
mkdir -p mnt && sudo mount -o loop scratch.ext4 mnt
echo "FIRECRACKER-BLOCK-LAB-MARKER" | sudo tee mnt/marker.txt >/dev/null
# A 16 MiB file of a repeating byte pattern, for predictable read offsets.
sudo dd if=/dev/zero bs=1M count=16 2>/dev/null | tr '\0' '\125' | sudo tee mnt/pattern.bin >/dev/null
sudo umount mnt

ls -l scratch.ext4
# Note the absolute path; you will pass it to /drives and watch its fd in strace.
echo "BACKING=$WORK/scratch.ext4"

Note: Use a non-sparse file (dd of real zeros, not truncate/fallocate --punch-hole) so that guest reads actually transfer bytes from disk and your strace offsets are real. A sparse hole would be served from the page cache as zeros and muddy the measurement.

Step 2: Boot a microVM with a logger and the scratch drive

Boot from your build, configure a logger so trace lines are visible, attach the rootfs and the scratch drive, and start. Keep the io_engine unset for now (defaults to Sync — verify on your branch).

ARCH=$(uname -m)
BIN=build/cargo_target/${ARCH}-unknown-linux-musl/debug/firecracker   # from your repo root
API=/tmp/fc-blk.sock
LOG=/tmp/fc-blk.log
rm -f "$API"; : > "$LOG"
sudo "$BIN" --api-sock "$API" &

curl -X PUT --unix-socket "$API" --data \
  '{"log_path":"'"$LOG"'","level":"Warning","show_level":true,"show_log_origin":true}' \
  http://localhost/logger

curl -X PUT --unix-socket "$API" --data \
  '{"kernel_image_path":"./vmlinux-6.1.x","boot_args":"console=ttyS0 reboot=k panic=1"}' \
  http://localhost/boot-source

curl -X PUT --unix-socket "$API" --data \
  '{"drive_id":"rootfs","path_on_host":"./ubuntu-24.04.ext4","is_root_device":true,"is_read_only":false}' \
  http://localhost/drives/rootfs

# The scratch drive. Sync engine is the default; we set it explicitly for clarity.
curl -X PUT --unix-socket "$API" --data \
  '{"drive_id":"scratch","path_on_host":"/tmp/blk-lab/scratch.ext4","is_root_device":false,"is_read_only":false,"io_engine":"Sync"}' \
  http://localhost/drives/scratch

curl -X PUT --unix-socket "$API" --data \
  '{"vcpu_count":2,"mem_size_mib":1024}' http://localhost/machine-config

curl -X PUT --unix-socket "$API" --data '{"action_type":"InstanceStart"}' http://localhost/actions

Tip: Confirm the API accepts io_engine on your branch before you depend on it:

rg -n "io_engine|IoEngine|Sync|Async" src/firecracker/swagger/firecracker.yaml

If io_engine is rejected, your branch may name it differently or gate it — rg the swagger and the vmm_config drive parsing (rg -n "io_engine" src/vmm/src/vmm_config/).

Inside the guest, the scratch drive appears as the second virtio-block device (/dev/vdb; the rootfs is /dev/vda):

# On the guest serial console:
lsblk
#   vda  ...  /         <- rootfs
#   vdb  ...            <- your scratch drive
ls -l /sys/block/vdb

Step 3: Instrument the device's queue handler

Add three temporary log points to the block device so you can see each I/O. Find the exact function and field names first — they drift between branches.

# The queue handler the EventManager calls on a kick.
rg -n "fn process_queue|fn process\b|fn handle_.*event|request_queue" \
   src/vmm/src/devices/virtio/block/
# The request struct/fields (type, sector) and the status constants.
rg -n "struct Request\b|struct RequestHeader|sector|request_type|r#type|VIRTIO_BLK_T_IN|VIRTIO_BLK_T_OUT|VIRTIO_BLK_S_OK" \
   src/vmm/src/devices/virtio/block/

Add minimal log lines at three points (adjust field names to what rg showed):

#![allow(unused)]
fn main() {
// 1. Top of the queue handler — the EventManager just woke us (steps 4–5).
log::warn!("[trace] block kick: queue event fired");

// 2. After parsing each request header (step 7).
log::warn!(
    "[trace] block req: type={:?} sector={} len={}",
    request.request_type, request.sector, data_len
);

// 3. Right after the host I/O completes, before add_used (steps 9–11).
log::warn!("[trace] block io done: status=OK len={}", len);
}

Rebuild and re-boot with the new binary (re-run Step 2):

tools/devtool build      # debug build is fine for tracing

Warning: These log lines run on the VMM thread, which is the data plane for every device. Keep them to one cheap log::warn! each — no allocation, no formatting of large buffers, nothing blocking. And remove them before any PR; trace instrumentation is never part of a contribution.

Step 4: Generate a controlled read and correlate it in the trace

In another terminal, tail the log. Then issue a read you control from inside the guest, bypassing the page cache so each block truly hits the device.

# Host terminal:
tail -f /tmp/fc-blk.log

# Guest serial console — direct read of the scratch device, 256 × 4 KiB blocks:
dd if=/dev/vdb of=/dev/null bs=4096 count=256 iflag=direct

In the log you should see a burst of:

[Warning] [trace] block kick: queue event fired
[Warning] [trace] block req: type=In sector=0 len=4096
[Warning] [trace] block io done: status=OK len=4096
[Warning] [trace] block req: type=In sector=8 len=4096
...

The type is a read (In), the sector values ascend by 8 (4096 bytes ÷ 512 bytes/sector = 8 sectors), and each len is 4096. This is steps 6–12 of the diagram, made visible.

Step 5: Confirm the host syscall and match the offset

Now nail the host side: which syscall served that read, on which fd, at what offset. With the firecracker process running, strace the I/O syscalls while you repeat the guest read.

# Host: find the process, then strace its threads' I/O syscalls.
pgrep -a firecracker
sudo strace -f -e trace=pread64,pwrite64,preadv,pwritev,preadv2,pwritev2,io_uring_enter \
  -p "$(pgrep -n firecracker)" 2>&1 | grep -E 'pread|pwrite|io_uring' | head -40

# Guest: a read at a known LBA so you can compute the host offset.
#   skip=1024 blocks of 4096 = byte offset 1024*4096 = 4194304 in the device,
#   which is host file offset 4194304 in scratch.ext4.
dd if=/dev/vdb of=/dev/null bs=4096 count=4 skip=1024 iflag=direct

You should see, on the Sync engine, lines like:

[pid NNNN] pread64(14, "...", 4096, 4194304) = 4096
[pid NNNN] pread64(14, "...", 4096, 4198400) = 4096
...

Map it: guest device byte offset skip*bs = 1024*4096 = 4194304 equals the host pread64 offset 4194304 on fd 14 (the scratch file's fd). That is step 8 — the guest's read() became a host pread at exactly sector * 512 on a flat file. Confirm fd 14 is your backing file:

sudo ls -l /proc/$(pgrep -n firecracker)/fd | grep scratch.ext4

Note: Without the jailer the backing-file path and fd are visible directly under /proc/<pid>/fd. Under the jailer the process is chrooted and possibly PID-namespaced; that is a later concern. Run firecracker directly for this lab.

Step 6: Trace a write, and notice the descriptor-direction flip

Writes are the mirror image. Drive a write into the scratch device and watch type=Out and host pwrite64.

# Guest: write 4 MiB of a pattern straight to the device (DESTROYS the fs on vdb — that's fine,
# it's a scratch disk; remount/reformat if you want it back).
dd if=/dev/zero of=/dev/vdb bs=4096 count=1024 oflag=direct

# Your trace:
[Warning] [trace] block req: type=Out sector=... len=4096
# strace:
[pid NNNN] pwrite64(14, "...", 4096, ...) = 4096

The key conceptual point: on a read, the data descriptor carries the WRITE flag (the device writes the bytes it read into guest memory); on a write, the data descriptor is device-readable (no WRITE flag — the device reads the bytes out of guest memory and pwrites them). This is the direction-bit confusion from the virtqueues deep dive. Add a temporary trace of the chain's descriptor flags to see it for yourself.

Step 7: Switch to the io_uring engine and re-trace

Now change the engine. Tear down the VM, and re-attach the scratch drive with io_engine: "Async" (the io_uring backend — verify the spelling on your branch).

# Stop the VM (Ctrl-C the firecracker process or send a reset), reformat the scratch disk,
# then re-boot exactly as in Step 2 but with the scratch drive's io_engine set to Async:
curl -X PUT --unix-socket "$API" --data \
  '{"drive_id":"scratch","path_on_host":"/tmp/blk-lab/scratch.ext4","is_root_device":false,"is_read_only":false,"io_engine":"Async"}' \
  http://localhost/drives/scratch

Re-run the strace from Step 5 during a guest read. On the Async engine you will see io_uring_enter instead of pread64/pwrite64 — the device submits SQEs (submission queue entries) to a kernel io_uring ring and reaps CQEs (completions), rather than issuing one blocking syscall per I/O:

[pid NNNN] io_uring_enter(15, 1, 0, 0, NULL, 8) = 1
[pid NNNN] io_uring_enter(15, 4, 0, 0, NULL, 8) = 4   # multiple I/Os submitted in one syscall

Find how Firecracker drives the ring — there is a completion eventfd registered with the EventManager so reaped completions wake the VMM thread the same way a kick does:

rg -n "io_uring|IoUring|submit|EnterFlags|completion|cqe|sqe|Async" src/vmm/src/devices/virtio/block/
rg -n "io_uring" src/vmm/src/io_uring/ 2>/dev/null   # FC has historically vendored a small io_uring wrapper — rg to confirm

Step 8: Benchmark Sync vs io_uring with fio

Numbers, not adjectives. Run the same fio workload under each engine and compare. Use a queue depth

1 so io_uring's batching can show — at QD=1 the engines are near-identical because there is no inflight parallelism to exploit.

# Inside the guest, on the scratch device (vdb). Random 4k reads, queue depth 32, direct I/O:
fio --name=randread --filename=/dev/vdb --rw=randread --bs=4k --iodepth=32 \
    --ioengine=libaio --direct=1 --size=128m --runtime=30 --time_based --group_reporting

Run it once with the host-side scratch drive on Sync, reboot with Async, run it again. Record IOPS, bandwidth, and average/99th-percentile latency from the fio summary for each. Then re-run the strace during each fio and count syscalls:

# Host, during each fio run: how many I/O syscalls in 5 seconds?
timeout 5 sudo strace -f -c -e trace=pread64,pwrite64,preadv,pwritev,io_uring_enter \
  -p "$(pgrep -n firecracker)" 2>&1 | tail -20

You are looking for the structural difference: Sync issues roughly one pread64/pwrite64 per I/O on the VMM thread (each a blocking syscall in the critical path); Async issues far fewer io_uring_enter calls because it batches many in-flight requests per syscall and reaps completions asynchronously. Whether that translates to higher IOPS depends on the workload, the queue depth, and the host storage — which is exactly the point of measuring rather than assuming.

Implementation Requirements / Deliverables

  • A non-sparse backing file you created, formatted, and seeded; its absolute path and the guest device it became (/dev/vdb).
  • A reading log naming the file:function for each hop: the Block struct, activate, the queue handler, the request parser, and the host I/O call — each with the rg you used.
  • Trace output from a controlled dd read showing type=In, ascending sectors, and 4096-byte lengths.
  • strace output showing the host pread64 at an offset you computed from the guest LBA, on the fd you confirmed is the scratch file.
  • Trace output from a write showing type=Out and host pwrite64, plus a one-sentence explanation of the descriptor-direction flip.
  • A Sync-vs-io_uring comparison table: the strace syscall structure (pread64/pwrite64 vs io_uring_enter) and the fio IOPS / bandwidth / latency for each.
  • All trace instrumentation removed from your working tree.

Troubleshooting

No [trace] lines appear

The logger must be configured before InstanceStart, with level at or below Warning. Confirm Firecracker's own startup lines are in the log first. Then check your trace is in the queue handler, not in activate (which runs once). If you rebuilt but did not re-launch the new binary, you are still running the old one — re-check the path under build/cargo_target/....

strace -p shows nothing on the backing file

Three causes: you are on the io_uring engine (trace io_uring_enter too); the reads are served from the guest page cache (use iflag=direct / --direct=1); or you are stracing the wrong thread (use -f to follow all threads). The host I/O happens on the VMM thread, not the API thread.

io_engine: "Async" is rejected

io_uring requires a host kernel new enough to support the operations Firecracker uses, and the field may be gated or spelled differently on your branch. rg -n "io_engine|Async|io_uring" src/firecracker/swagger/firecracker.yaml src/vmm/src/vmm_config/ to see the exact accepted values and any version guard. On an old host kernel, fall back to Sync and note the limitation.

fio shows no difference between engines

You are probably at queue depth 1, where there is no in-flight parallelism for io_uring to batch. Raise --iodepth to 32+ and ensure --direct=1. Also confirm the host storage is not the bottleneck (a slow disk hides any VMM-side difference) — try a tmpfs-backed file to isolate the device path.

The guest sees the scratch disk as read-only / missing

Check is_read_only in the /drives/scratch PUT and that path_on_host is correct and readable by the firecracker process. lsblk in the guest must show vdb; if not, the drive PUT failed — re-issue it and read the curl response.

Expected Output

# Trace during a controlled read (Sync engine):
[Warning] [trace] block kick: queue event fired
[Warning] [trace] block req: type=In sector=8192 len=4096
[Warning] [trace] block io done: status=OK len=4096

# strace, Sync engine — one pread per I/O at sector*512:
[pid NNNN] pread64(14, "...", 4096, 4194304) = 4096

# strace, Async engine — batched submissions:
[pid NNNN] io_uring_enter(15, 8, 0, 0, NULL, 8) = 8

# A representative fio comparison table you produce (numbers are host-dependent):
Engine     IOPS(4k randread QD32)   BW        avg lat   p99 lat   I/O syscalls/5s
Sync       ~K                       ~MB/s     ~us       ~us       ~ one pread per I/O
io_uring   ~K (often higher)        ~MB/s     ~us       ~us       ~ far fewer io_uring_enter

Stretch Goals

  1. Count descriptors per chain. Trace the number of descriptors and their flags per request. Confirm the three-descriptor header/data/status shape, and find a request that uses multiple data descriptors (a large contiguous read the guest split).
  2. Watch the interrupt side. Find where the device raises the IRQ (the IrqTrigger / InterruptStatus / irqfd path), add a trace there, and correlate it to cat /proc/interrupts | grep virtio counts in the guest moving. See interrupts-and-irqchip.
  3. Apply a rate limiter. PATCH the scratch drive with a bandwidth rate limiter and watch the fio bandwidth cap and the syscall cadence change. Read rate-limiting-token-bucket; this is the same limiter Lab 2 applies to net.
  4. read-only and flush. Attach the drive is_read_only: true, attempt a guest write, and trace the VIRTIO_BLK_S_IOERR status path. Then trace a fsync from the guest and find the VIRTIO_BLK_T_FLUSH request and its host fsync/fdatasync.
  5. Measure the no-exit kick. Reason quantitatively: how many VM exits would a naive (no-ioeventfd) design take per I/O versus the ioeventfd design? Tie it to the the-event-manager epoll loop.

Validation / Self-check

Answer without notes; these gate completion.

  1. Which ring does the guest write to hand the device a request, and which does the device write on completion? Who bumps avail.idx and who bumps used.idx?
  2. What are the three descriptors in a block-read chain, and which carry the WRITE flag? Why does the data descriptor carry WRITE on a read but not on a write?
  3. Given a guest dd ... bs=4096 skip=1024, what host file offset do you expect in the pread64, and why?
  4. What exactly happens between the guest writing QueueNotify and the block device's queue handler running? Where does ioeventfd remove a VM exit?
  5. Structurally, how does the host syscall pattern differ between the Sync and io_uring engines, and under what workload does that difference show up in fio?
  6. If you removed the add_used call from the handler, what would the guest observe, and why?
  7. Why must the queue handler stay fast and non-blocking, and which thread runs it?

Cross-references: virtqueues, virtio-transport-mmio, virtio-block, io-engines, the-event-manager.

Next: Lab 2 — Virtio-Net and TAP: stand up a host TAP, route it, SSH into the guest, and trace an RX/TX frame across the TAP fd and the two virtqueues.