Lab 3: Benchmark the I/O Engines

Background

A virtio-block device has two halves. The front half — the virtqueue, the descriptor chain, the MMIO transport — is how the guest hands Firecracker a read or write. The back half is what this lab is about: once Firecracker has a block request, how does it move the bytes between guest memory and the host file that backs the drive? That back half is the I/O engine, and Firecracker ships two — a synchronous engine built on blocking pread/pwrite on the VMM thread, and an asynchronous engine built on io_uring. The choice is a textbook systems trade-off: latency vs. throughput vs. CPU vs. complexity, with a host-kernel-surface (security) dimension on top. The I/O engines engineering essay makes the argument; this lab makes you produce the numbers.

You will read the engine-selection code, configure two drives — one Sync, one Async — drive each with fio across queue depths inside the guest, measure latency, throughput, and CPU, interpret the result, and connect it to the rule that decides which engine is appropriate when. Then you will find how the project guards this performance in its own test suite, so a regression in the block path fails CI rather than shipping. The headline you must end up able to demonstrate, not assert: io_uring wins on concurrency and on not blocking the VMM thread; sync can win on raw latency for shallow, cache-friendly workloads — and "io_uring is always faster" is a benchmarking mistake.

This is a measure-it lab, and it lives or dies by methodology (Lab 1's discipline: baseline, iterate, report a distribution, explain disagreements).

Why This Lab Matters for Contributors

• The block I/O path is performance-sensitive, security-sensitive (io_uring widens the host-kernel surface, which the minimal device model philosophy makes you weigh), and actively maintained — fertile contribution ground. You cannot touch it credibly without being able to benchmark both engines.
• The async engine is Firecracker's own in-tree io_uring with registered restrictions, not a general crate. Understanding its submit/harvest/drain state machine is required to review robustness fixes, drain-on-snapshot edge cases, and restriction-tightening PRs.
• "Which engine, and why" is exactly the kind of measured, explained judgment the maintainers ask for. A contributor who answers it with a distribution and a critical path — not a vibe — is one who can be trusted near the block device.

Prerequisites

cd ~/firecracker
B=build/cargo_target/$(uname -m)-unknown-linux-musl/release
test -x $B/firecracker && echo "firecracker built"

# The engines and the selector exist on your branch — locate, don't assume the path.
ls src/vmm/src/devices/virtio/block/virtio/io/    # mod.rs sync_io.rs async_io.rs (verify)
rg -n "enum FileEngine|FileEngineType|io_engine|Sync|Async" \
  src/vmm/src/devices/virtio/block/virtio/io/mod.rs \
  src/vmm/src/vmm_config/drive.rs

Warning: This benchmark is meaningless without --direct=1 in fio. Without it, the guest page cache absorbs reads and you measure the cache, not the engine. Direct I/O forces requests down to the block device where the engine actually runs. Forgetting this is the classic block-benchmarking error — a reviewer will catch it.

Step-by-Step Tasks

Step 1: Read the engine selector and confirm io_uring is available

Before measuring, read the dispatch. A small FileEngine enum routes every operation to one of two implementations; the drive's io_engine field picks the variant.

# The dispatch shape and per-op methods.
rg -n "enum FileEngine|FileEngine::Sync|FileEngine::Async|fn read|fn write|fn flush|push_read|push_write|push_flush" \
  src/vmm/src/devices/virtio/block/virtio/io/mod.rs

# The sync engine: blocking seek + read/write, completes inline on the VMM thread.
rg -n "fn read|fn write|fn flush|seek|read_volatile|write_volatile|fsync" \
  src/vmm/src/devices/virtio/block/virtio/io/sync_io.rs

# The async engine: io_uring submission + completion via eventfd, plus drain.
rg -n "push|submit|completion_evt|process_completion|fn drain|register_restrictions|Restriction" \
  src/vmm/src/devices/virtio/block/virtio/io/async_io.rs src/vmm/src/io_uring/mod.rs | head -25

# CRITICAL: io_uring has a host-kernel dependency. Firecracker probes the ops it needs.
rg -n "probe|EOPNOTSUPP|unsupported|register|OpCode" src/vmm/src/io_uring/probe.rs
# Verify the DEFAULT engine on your branch — do not assume.
rg -n "file_engine_type|FileEngineType|io_engine|Default|default" \
  src/vmm/src/vmm_config/drive.rs src/vmm/src/devices/virtio/block/virtio/device.rs

State, from the code: the sync engine blocks the VMM thread (the EventManager epoll loop) on each I/O, so a slow read stalls all device emulation until it returns; the async engine submits and returns immediately, harvesting completions later via an eventfd, so the thread never parks and many requests fly concurrently. That single difference — does a slow I/O block the event loop? — is what the benchmark will expose.

Step 2: Configure two drives, one per engine

Boot a microVM with a root drive plus two extra data drives — one Sync, one Async — backed by two equally-sized image files. Benchmarking both in one guest controls for everything except the engine.

# Two backing files of equal size (large enough NOT to fit in cache — see the --direct note).
fallocate -l 2G /img/sync.ext4 ; mkfs.ext4 -q /img/sync.ext4
fallocate -l 2G /img/async.ext4 ; mkfs.ext4 -q /img/async.ext4

API=/tmp/fc-io.sock; rm -f $API
sudo $B/firecracker --api-sock $API &
for i in $(seq 1 100); do [ -S $API ] && break; done
curl -sX PUT --unix-socket $API --data \
 '{"kernel_image_path":"./vmlinux-6.1.x","boot_args":"console=ttyS0 reboot=k panic=1 nomodule"}' \
 http://localhost/boot-source
curl -sX PUT --unix-socket $API --data \
 '{"drive_id":"rootfs","path_on_host":"./ubuntu-fio.ext4","is_root_device":true,"is_read_only":false}' \
 http://localhost/drives/rootfs

# The two engines, on two data drives:
curl -sX PUT --unix-socket $API --data \
 '{"drive_id":"sync_blk","path_on_host":"/img/sync.ext4","is_root_device":false,"is_read_only":false,"io_engine":"Sync"}' \
 http://localhost/drives/sync_blk
curl -sX PUT --unix-socket $API --data \
 '{"drive_id":"async_blk","path_on_host":"/img/async.ext4","is_root_device":false,"is_read_only":false,"io_engine":"Async"}' \
 http://localhost/drives/async_blk

curl -sX PUT --unix-socket $API --data '{"vcpu_count":2,"mem_size_mib":1024}' http://localhost/machine-config
curl -sX PUT --unix-socket $API --data '{"action_type":"InstanceStart"}' http://localhost/actions

In the guest, the two data drives appear as /dev/vdb (sync) and /dev/vdc (async) — confirm the order with lsblk. If io_engine:"Async" is rejected, your host kernel failed the io_uring probe (Step 1) — see Troubleshooting.

Step 3: Benchmark latency (shallow queue) — where sync can win

Run fio at iodepth=1: one request at a time, latency-bound. This is the regime where the async engine's submission/completion plumbing is pure overhead and sync's inline completion can win.

# Inside the guest. --direct=1 is mandatory (bypass guest page cache).
# Sync drive:
fio --name=lat-sync --filename=/dev/vdb --direct=1 --rw=randread --bs=4k \
    --iodepth=1 --runtime=30 --time_based --output-format=normal | tee lat-sync.txt
# Async drive:
fio --name=lat-async --filename=/dev/vdc --direct=1 --rw=randread --bs=4k \
    --iodepth=1 --runtime=30 --time_based --output-format=normal | tee lat-async.txt

# Pull the numbers that matter: median (p50) and tail (p99) latency, IOPS.
grep -E 'lat .*avg|clat percentiles|iops' lat-sync.txt lat-async.txt

At iodepth=1 expect the gap to be small, possibly favoring sync. Record p50, p99, and IOPS for each. If async is meaningfully slower here, that is expected and is the whole point of the next step's contrast — do not "fix" it.

Step 4: Benchmark throughput (deep queue) — where async wins

Now run with deep queues and parallelism (iodepth=64, numjobs=4): many requests in flight, throughput-bound. This is where async pulls ahead because the kernel services many requests at once and the VMM thread never blocks.

# Sync drive, deep:
fio --name=tput-sync --filename=/dev/vdb --direct=1 --rw=randread --bs=4k \
    --iodepth=64 --numjobs=4 --runtime=30 --time_based --group_reporting | tee tput-sync.txt
# Async drive, deep:
fio --name=tput-async --filename=/dev/vdc --direct=1 --rw=randread --bs=4k \
    --iodepth=64 --numjobs=4 --runtime=30 --time_based --group_reporting | tee tput-async.txt

grep -E 'iops|bw=|lat .*avg' tput-sync.txt tput-async.txt

At depth 64 expect async to win on IOPS/throughput, often substantially, because the sync engine completes serially (one request occupies the VMM thread until it returns) while async submits a batch and harvests completions together. Record both.

Step 5: Measure CPU and VMM-thread blocking

Throughput and latency are half the story; CPU per request and whether the VMM thread blocks are the other half — and they matter for density (every CPU-second is multiplied across the host).

# On the HOST, while each fio run is in flight, sample the firecracker process's CPU
# and its VMM thread specifically.
FC=$(pgrep -n firecracker)
# Per-thread CPU — the VMM thread is the one running the event loop.
top -H -b -n3 -p $FC | grep -A8 'PID' | head -40
# Total CPU consumed by firecracker during a fixed-work run:
pidstat -p $FC 1 5

What to look for, tied to the engine mechanics:

The sync engine blocking the VMM thread is the subtle cost that throughput numbers alone hide: under sync, a slow drive doesn't just slow that drive — it stalls every device on the VMM thread (other queues, MMDS). That is a density and isolation argument, not just a throughput one.

Step 6: Interpret, and connect to "which engine when"

Assemble the four runs into the head-to-head and write the rule. A benchmark you cannot explain is not a result — for any number that surprises you, find the cause (wrong default engine? a file that fit in cache despite --direct? an idle VMM thread that never contended?).

The rule: async/io_uring for deep or slow or concurrent I/O and to keep the VMM thread unblocked; sync for shallow, fast, cache-friendly workloads, or where the narrower host-kernel surface and simplicity matter. There is no universally correct choice — it depends on queue depth, backing storage, and VMM-thread contention from other devices.

Step 7: Find how the performance is guarded

A measured win that isn't guarded regresses silently. Find the block-performance tests in the suite — the regression gate — and read what they pin and assert.

# The block/io performance tests.
rg -rln "io_engine|Async|Sync|fio|block|iops|throughput" tests/integration_tests/performance/ | head
rg -n "io_engine|Async|Sync|fio|iodepth|assert|threshold|baseline" \
  tests/integration_tests/performance/test_block*.py 2>/dev/null | head -30

# Run them.
./tools/devtool test -- integration_tests/performance/ -k block 2>&1 | tail -20

Note what the suite does that your hand runs didn't: it pins the kernel/rootfs, the engine, the fio profile, and the host class; runs multiple iterations; and asserts the result against a baseline so a block-path regression fails CI. That is the difference between a benchmark and a guard — and it's what a maintainer will ask your block-path PR to extend.

Implementation Requirements / Deliverables

• The FileEngine selector and both engine implementations located in the source, with the io_uring probe.rs dependency noted and the default engine verified.
• Two drives configured, one Sync and one Async, on equal backing files.
• A latency run (iodepth=1, --direct=1) for both engines, with p50/p99/IOPS.
• A throughput run (iodepth=64, numjobs=4, --direct=1) for both engines, with IOPS/bandwidth.
• A CPU/VMM-thread measurement showing the sync engine blocking the VMM thread under slow/deep I/O.
• The filled-in head-to-head table and a one-paragraph "which engine when" rule, with every surprising number explained.
• The block-performance test located in the suite, with what it pins and asserts.

Troubleshooting

io_engine:"Async" is rejected at drive configuration

Your host kernel failed Firecracker's io_uring probe. rg -n "probe|EOPNOTSUPP|register" src/vmm/src/io_uring/probe.rs to see which ops it needs, and check your host kernel version. On too-old a kernel the async engine is simply unavailable — that's a feature (a fail-closed probe), not a bug. Test on a newer host.

Async isn't faster than sync (or is slower)

Most likely you benchmarked the shallow regime (iodepth=1), where sync legitimately wins — that's expected, not a defect. Or you omitted --direct=1 and measured the guest page cache. Or the backing file fit in the host page cache (use a file bigger than RAM, or drop caches). Async's advantage is concurrency; if there's no concurrency to exploit, it won't show.

Numbers swing wildly between runs

Noise — same as Lab 1. Pin CPUs (taskset), set the governor to performance, use a quiet host, run ≥5 iterations, report median + spread. Block I/O adds storage-device variance on top; prefer a fast SSD/NVMe and a warmed device.

fio not present in the guest

The CI rootfs may not ship it. Use a rootfs with fio installed, or build a small image that has it. The benchmark must run inside the guest to exercise the virtio-block path.

The two data drives map to unexpected /dev/vdX names

Drive letters follow attachment order, not drive_id. Confirm with lsblk in the guest and match by size/order; don't assume sync_blk is /dev/vdb.

Expected Output

# iodepth=1 (latency) — gap small, may favor sync:
sync :  p50 ~120 us  p99 ~210 us   ~8k IOPS
async:  p50 ~150 us  p99 ~260 us   ~7k IOPS      <-- async slower here: EXPECTED

# iodepth=64, numjobs=4 (throughput) — async wins:
sync :  ~28k IOPS   ~110 MiB/s
async:  ~95k IOPS   ~370 MiB/s     <-- async pulls ahead with concurrency

# Host CPU / VMM thread during a deep sync run:
the VMM thread spends time BLOCKED in pread/pwrite (sync) vs free (async)

Your host's absolute numbers will differ; the shape — sync competitive or better at depth 1, async ahead at depth 64 — is the result to reproduce and explain.

Stretch Goals

1. Sweep queue depth. Run iodepth ∈ {1, 4, 16, 64, 256} for both engines and plot IOPS vs depth. Find the crossover point where async overtakes sync — that depth is the practical decision boundary for your storage.
2. Mixed read/write and fsync. Re-run with --rw=randrw --rwmixread=70 and with --fsync=1. Flushes hit the engines differently; explain how flush/drain differ between sync and async (rg -n "fn flush|fn drain" src/vmm/.../io/).
3. Rate limiter interaction. Add a rate limiter to each drive (rate-limiting deep dive) and benchmark how the token bucket interacts with the async submission path. Does the limiter throttle at submit or at completion?
4. Drain on snapshot. Take a snapshot while async I/O is in flight and confirm completions are drained cleanly (rg -n "fn drain|process_completion" src/vmm/.../io/async_io.rs). A lost completion is a snapshot-correctness bug — this is a real contribution edge.
5. Find a block-perf issue. gh issue list --repo firecracker-microvm/firecracker --search "io_uring OR block OR io_engine in:title,body state:open" — reproduce one and propose a measurement or a regression test.

Validation / Self-check

Answer without notes; these gate completion:

1. What does the io_engine field select, and what are the two variants? Where does the default get decided?
2. On which thread does the sync engine block, and why does that stall more than just the one drive?
3. Why does async win at iodepth=64 but not necessarily at iodepth=1? State the mechanism, not the result.
4. Why is --direct=1 mandatory for this benchmark, and what do you measure without it?
5. What host-kernel dependency does the async engine have, and how does Firecracker handle a host that can't satisfy it?
6. Give the "which engine when" rule, including the security/surface dimension.
7. What does the block-performance test pin and assert, and why is that the difference between a benchmark and a guard?

Next: you have measured the per-microVM boot cost (Lab 1), the aggregate density cost (Lab 2), and the per-request I/O cost (Lab 3) — the full performance argument. Return to the performance & density masterclass index to consolidate, and read the I/O engines, oversubscription & density, and boot-time optimization engineering essays, where these measurements become design arguments you can contribute against.