I/O Engines: Sync vs. io_uring

A virtio-block device has two halves. The front half — the virtqueue, the descriptor chain, the MMIO transport — is how the guest hands Firecracker a read or write request, and it is covered in the virtio-block deep dive. The back half is what this chapter is about: once Firecracker has pulled a block request off the virtqueue, how does it actually move the bytes between the guest's memory and the host file that backs the drive? That back half is the I/O engine, and Firecracker ships two of them — a synchronous engine built on blocking syscalls, and an asynchronous engine built on io_uring.

The choice between them is a textbook systems trade-off: latency vs. throughput vs. CPU vs. complexity, with a security dimension layered on top (io_uring widens the host-kernel surface, which the minimal device model philosophy makes you think twice about). This chapter grounds the trade-off in the real code so you can reason about which engine wins for which workload, and so you can benchmark it yourself instead of taking anyone's word.

Note: Read the virtio-block deep dive and the virtqueues deep dive first. This chapter starts where a block request already exists and asks only how it gets serviced against the host file.

Where the engines live

Find the code first; do not trust paths from memory.

cd ~/src/firecracker
ls src/vmm/src/devices/virtio/block/virtio/io/    # mod.rs sync_io.rs async_io.rs
# The selector enum: which engine a drive uses.
rg -n "enum FileEngine|Sync\(|Async\(|enum FileEngineType" \
  src/vmm/src/devices/virtio/block/virtio/io/mod.rs \
  src/vmm/src/devices/virtio/block/virtio/device.rs

The architecture is a small enum that dispatches every operation to one of two implementations:

                    block request (from the virtqueue)
                                  │
                                  ▼
                  ┌──────────── FileEngine ────────────┐
                  │  enum { Sync(SyncFileEngine),       │
                  │         Async(AsyncFileEngine) }    │
                  └───────────────┬────────────────────┘
                 read/write/flush │ dispatched by variant
              ┌───────────────────┴────────────────────┐
              ▼                                          ▼
     SyncFileEngine                            AsyncFileEngine
     (sync_io.rs)                              (async_io.rs)
     blocking seek + read/write                io_uring submission queue
     completes inline, in the VMM thread       completes via completion_evt

# Confirm the dispatch shape and the per-op methods.
rg -n "FileEngine::Sync|FileEngine::Async|fn read|fn write|fn flush|push_read|push_write|push_flush" \
  src/vmm/src/devices/virtio/block/virtio/io/mod.rs

The drive's engine is chosen by the file_engine_type field on the drive config (io_engine in the API / Sync vs Async). Verify the default on your branch — do not assume it:

rg -n "file_engine_type|FileEngineType|io_engine|Default" \
  src/vmm/src/vmm_config/drive.rs \
  src/vmm/src/devices/virtio/block/virtio/device.rs

The synchronous engine

The sync engine is exactly what it sounds like: for each request, seek to the offset and do a blocking read/write against the file, then complete the request inline. There is no queue, no batching, no separate completion path.

rg -n "fn read|fn write|fn flush|seek|SeekFrom|read_volatile|write_volatile|fsync" \
  src/vmm/src/devices/virtio/block/virtio/io/sync_io.rs

The key consequence is where the blocking happens. The block device's event handler runs on the VMM thread (the EventManager epoll loop). A synchronous read that misses the host page cache and has to hit the disk blocks that thread until the syscall returns. While it blocks, the VMM thread is not servicing other devices, other queues, or MMDS.

sequenceDiagram
    participant G as guest
    participant VQ as virtqueue
    participant VMM as VMM thread (event loop)
    participant K as host kernel
    G->>VQ: kick (block request)
    VQ->>VMM: queue event
    VMM->>K: pread/pwrite (BLOCKS)
    Note over VMM,K: VMM thread parked here<br/>until syscall returns
    K-->>VMM: bytes done
    VMM->>VQ: write used ring, raise IRQ
    VQ-->>G: completion

This is simple, correct, and low-overhead per request — and it is fine when I/O is fast (cache hits, fast SSD, light load). It hurts when I/O is slow or deep: a single high-latency request stalls the whole VMM thread, and you cannot have many requests in flight at once because each one occupies the thread until it finishes.

The asynchronous engine: io_uring

The async engine decouples submission from completion using Linux's io_uring. Instead of blocking, Firecracker pushes the operation onto an io_uring submission queue, returns immediately to the event loop, and learns about completions later via an eventfd. Many requests can be in flight at once, and the VMM thread never parks on a slow disk.

Firecracker maintains its own io_uring implementation in-tree, rather than pulling a general-purpose crate — consistent with the minimal-surface philosophy, it exposes only the operations it needs and registers restrictions to lock the ring down.

# Firecracker's in-tree io_uring. Note the restriction registration.
ls src/vmm/src/io_uring/
rg -n "push|submit|register_restrictions|Restriction|Cqe|OpCode|completion_evt|EFD_NONBLOCK" \
  src/vmm/src/devices/virtio/block/virtio/io/async_io.rs \
  src/vmm/src/io_uring/mod.rs | head -25

The flow has two independent phases:

sequenceDiagram
    participant G as guest
    participant VMM as VMM thread (event loop)
    participant SQ as io_uring submission queue
    participant K as host kernel
    participant CQ as io_uring completion queue
    G->>VMM: block request
    VMM->>SQ: push_read/push_write (returns immediately)
    Note over VMM: VMM thread free to service<br/>other events / more requests
    SQ->>K: kernel processes async
    K->>CQ: completion entry
    CQ->>VMM: completion_evt (eventfd) wakes the loop
    VMM->>VMM: pop CQEs, finish requests, update used ring
    VMM->>G: completions (possibly batched)

The two phases are: submit (push_read/push_write/push_flush enqueue a submission entry and return — non-blocking), and harvest (the completion_evt eventfd is registered with the event loop; when the kernel posts completions, the loop wakes, pops the completion entries, and finishes the corresponding block requests). drain flushes outstanding completions, which matters for clean device shutdown and snapshotting.

rg -n "fn drain|do_pop|process_completion|completion_evt|register" \
  src/vmm/src/devices/virtio/block/virtio/io/async_io.rs

Tip: Because io_uring is a Linux kernel feature, the async engine has a kernel version dependency. Firecracker probes for the operations it needs (src/vmm/src/io_uring/probe.rs) — on a host kernel too old to support them, the async engine is unavailable. Check probe.rs and the kernel support policy before assuming Async works.

The trade-off, head to head

The honest summary: async/io_uring wins on throughput and on not blocking the VMM thread under deep or slow I/O; sync can win on raw latency for shallow, fast, cache-friendly workloads and is simpler and narrower. There is no universally correct choice — it depends on the workload's queue depth, the backing storage, and the contention on the VMM thread from other devices.

Warning: "io_uring is always faster" is wrong and a common benchmarking mistake. For a workload that issues one small synchronous read at a time against a warm page cache, the submission/completion overhead can make async slower. The async engine's advantage is concurrency and non-blocking, which only pays off when there is concurrency to exploit.

Benchmark it yourself

Do not believe the table — measure. Use fio inside the guest against drives configured with each engine and compare.

# Configure two drives, one per engine (via API or config file).
#   {"drive_id":"sync_blk","path_on_host":"/img/sync.ext4","io_engine":"Sync", ...}
#   {"drive_id":"async_blk","path_on_host":"/img/async.ext4","io_engine":"Async", ...}

# Inside the guest, vary queue depth — this is where the engines diverge.
# Shallow, latency-bound:
fio --name=lat --filename=/dev/vdb --rw=randread --bs=4k --iodepth=1 --runtime=30 --time_based
# Deep, throughput-bound:
fio --name=tput --filename=/dev/vdb --rw=randread --bs=4k --iodepth=64 --numjobs=4 --runtime=30 --time_based

# Watch the engine pick on the host side and find the perf tests in the suite.
rg -rln "io_engine|Async|Sync|fio|block" tests/integration_tests/performance/ | head
./tools/devtool test -- integration_tests/performance/ -k block 2>&1 | tail -20

The result you should expect to see (and then explain): at iodepth=1 the gap is small and may favor sync; as queue depth and parallelism rise, async pulls ahead because the kernel services many requests at once and the VMM thread never blocks. If your numbers disagree, find out why — wrong default engine, a too-small file that fits in cache, or a VMM thread that is idle anyway. A benchmark you cannot explain is not a result.

Where to contribute

The block I/O path is performance-sensitive, security-sensitive, and actively maintained — fertile ground.

gh issue list --repo firecracker-microvm/firecracker \
  --search "io_uring OR block OR io_engine in:title,body state:open" --limit 40
gh issue list --repo firecracker-microvm/firecracker --label "Type: Performance" --state open

On-ramps: io_uring engine robustness (completion harvesting, drain-on-snapshot edge cases); tightening the registered io_uring restrictions (surface reduction); block performance regression tests; clear documentation of when each engine wins; rate-limiter interaction with the async path (see rate limiting).

Next: I/O performance is one memory lever; the page size behind guest RAM is another — Huge Pages & Memory Performance. Or go hands-on in the io-engines benchmark masterclass lab.