vmm-sys-util

Underneath every crate you've read in this section is a layer of plain Linux plumbing: an eventfd to wake a thread, the ioctl() macros that kvm-ioctls expands into, the epoll wrappers event-manager builds on, the flexible-array- member handling kvm-bindings needs for kvm_cpuid2, an errno wrapper so a failed syscall becomes a Rust Error. None of that is VMM-specific — it is the generic unsafe glue any Rust program talking to the Linux kernel needs. rust-vmm collects it into vmm-sys-util, and Firecracker uses it everywhere, most visibly through EventFd, which shows up as the serial console's Trigger, the virtio fast-path ioeventfd, the irqfd that injects interrupts, and the API wake-up that nudges the VMM thread's epoll loop.

After this chapter you can: name what vmm-sys-util provides (EventFd, the ioctl_with_* macros, epoll wrappers, FamStructWrapper, errno::Error, TempFile/TempDir, terminal/signal helpers); explain how each underpins a crate or feature you've already met; and find where Firecracker uses EventFd to wire KVM, devices, and the API together.

Note — it moved out of the tree. vmm-sys-util was historically vendored inside Firecracker; it is now an external rust-vmm dependency (per the fact sheet — verify on your branch). When you see use vmm_sys_util::..., that is the shared crate, not in-tree code.

# Confirm the dependency and pinned version (verify on your branch):
rg -n "vmm-sys-util" Cargo.toml src/vmm/Cargo.toml Cargo.lock
cargo doc -p vmm-sys-util --no-deps --open

docs.rs: docs.rs/vmm-sys-util.

What's in the box

vmm-sys-util is a grab-bag, organized into modules. The ones that matter for Firecracker:

The crate is the floor. It adds no abstraction beyond "make this raw Linux facility safe and ergonomic in Rust." Everything else in rust-vmm — and in Firecracker — stands on it.

                Firecracker / rust-vmm crates
   ┌──────────────┬───────────────┬───────────────┬──────────────┐
   │ kvm-ioctls   │ event-manager │ vm-superio    │ FC devices   │
   │ (ioctl_with_)│ (Epoll)       │ (Trigger=EFd) │ (EventFd)    │
   └──────┬───────┴───────┬───────┴───────┬───────┴──────┬───────┘
          ▼               ▼               ▼              ▼
   ┌─────────────────────────────────────────────────────────────┐
   │                       vmm-sys-util                            │
   │  EventFd · ioctl_with_*! · Epoll · FamStructWrapper · errno   │
   └─────────────────────────────────────────────────────────────┘
                              ▼  raw syscalls
                       Linux kernel

EventFd: the one type you'll meet constantly

An eventfd is a kernel-provided counter you can write() (increment) and read() (drain), exposed as a single fd you can put in an epoll set. It is the universal "poke a thread / signal an event" primitive, and vmm-sys-util's EventFd is the safe wrapper. Firecracker uses it for four distinct jobs:

#![allow(unused)]
fn main() {
use vmm_sys_util::eventfd::EventFd;

// A non-blocking eventfd (the usual flavor for an epoll-driven loop).
let evt = EventFd::new(libc::EFD_NONBLOCK).unwrap();

// Signalling side (e.g. the API thread, or a Serial Trigger):
evt.write(1).unwrap();                 // increment the counter -> fd becomes readable

// Draining side (e.g. a device subscriber's process()):
match evt.read() {                     // read the counter, reset to 0
    Ok(count) => { /* count notifications coalesced */ }
    Err(e) if e.raw_os_error() == Some(libc::EAGAIN) => { /* nothing pending */ }
    Err(e) => panic!("{e}"),
}

// Clone the fd to register the same eventfd with KVM and with epoll:
let kvm_copy = evt.try_clone().unwrap();   // give one to KVM (irqfd/ioeventfd),
                                           // keep one for your loop
}

try_clone is the detail that makes the fast path work: the same eventfd is handed to KVM (so the kernel signals it) and kept by the VMM (so epoll waits on it). The eventfd is the rendezvous point between the kernel side and the userspace loop. Notice also that read coalesces — multiple writes before a read show up as one readiness with a counter, which is why a subscriber drains and then processes whatever work is pending, not exactly one item per readiness.

Tip: When you read the virtio transport or interrupts & irqchip, every "an eventfd is signalled" sentence is a vmm_sys_util::eventfd::EventFd. Searching rg -n "EventFd" src/vmm/src/ is the fastest map of Firecracker's async seams.

The ioctl macros, FamStructWrapper, and errno

Three more pieces you should recognize even though you rarely call them directly:

The ioctl_with_*! macros. kvm-ioctls doesn't hand-write ioctl() syscalls; it uses vmm-sys-util's macros — ioctl_with_ref!, ioctl_with_mut_ref!, ioctl_with_val! — which expand to the correct unsafe { libc::ioctl(...) } with the right argument shape (a pointer to a struct, a value, etc.). When you read kvm-ioctls source you'll see these macros at every ioctl. If you ever add a KVM ioctl wrapper, this is the tool.

find ~/.cargo/registry/src -maxdepth 2 -type d -name 'kvm-ioctls-*' \
  -exec rg -n "ioctl_with_ref|ioctl_with_mut_ref|ioctl_with_val|ioctl_expr" {} +

FamStructWrapper. Several KVM structs are flexible-array-member structs: a fixed header followed by a runtime-length array. kvm_cpuid2 is a count nent followed by nent kvm_cpuid_entry2 entries; kvm_msrs is similar. You can't represent that cleanly as a fixed Rust struct, so vmm-sys-util provides FamStructWrapper<T> — a safe owner of the header-plus-array allocation, with length bookkeeping. kvm-bindings defines the FAM struct types; vmm-sys-util provides the wrapper that makes them usable. Firecracker touches these whenever it reads or sets CPUID/MSRs — i.e. all over CPU templates.

rg -n "FamStructWrapper|CpuId|kvm_cpuid2|Msrs|kvm_msrs|fam" src/vmm/src/ 2>/dev/null

errno::Error. A raw syscall returns -1 and sets errno. vmm-sys-util's errno::Error (and errno_result()) capture that into a typed Rust error, which is the base of the error types in kvm-ioctls and others. It is why a failed KVM_RUN surfaces as a proper Err you can match on, not a silent -1.

How Firecracker uses it

# EventFd — the most-used type; this rg is a tour of FC's async seams:
rg -n "EventFd::new|EventFd|try_clone|\.write\(1\)|\.read\(\)" src/vmm/src/ | head -40

# The KVM registrations that consume those eventfds:
rg -n "register_irqfd|register_ioevent|KVM_IRQFD|KVM_IOEVENTFD|ioeventfd|irqfd" src/vmm/src/

# FAM structs for CPUID/MSRs:
rg -n "FamStructWrapper|CpuId|Msrs|kvm_cpuid2" src/vmm/src/

# Terminal raw-mode for the console, and signal handling:
rg -n "Terminal|set_raw_mode|termios" src/vmm/src/
rg -n "vmm_sys_util::signal|register_signal_handler|sigaction" src/vmm/src/signal_handler.rs

This is the crate that ties the section together. Trace any async path in Firecracker far enough down — a guest kick, an interrupt, an API call, a vCPU pause — and you reach a vmm_sys_util::eventfd::EventFd. It is the smallest, lowest, and most pervasive of the rust-vmm crates Firecracker depends on.

Reading exercise

# 1. The dependency and pinned version.
rg -n "vmm-sys-util" Cargo.toml Cargo.lock

# 2. The modules and types (pinned version).
cargo doc -p vmm-sys-util --no-deps --open

# 3. Every EventFd use in the VMM — a map of the async seams.
rg -n "EventFd" src/vmm/src/ | head -40

# 4. The KVM registrations that consume eventfds.
rg -n "register_irqfd|register_ioevent|KVM_IRQFD|KVM_IOEVENTFD" src/vmm/src/

# 5. FAM structs for CPUID/MSRs.
rg -n "FamStructWrapper|CpuId|Msrs|kvm_cpuid2|kvm_msrs" src/vmm/src/

# 6. The ioctl macros inside kvm-ioctls.
find ~/.cargo/registry/src -maxdepth 2 -type d -name 'kvm-ioctls-*' \
  -exec rg -n "ioctl_with_ref|ioctl_with_val" {} +

Answer:

1. What is an eventfd, and why is it the right primitive for waking an epoll-driven loop? What does read coalescing mean for a subscriber?
2. Name the four distinct jobs Firecracker uses EventFd for, and the KVM registration (if any) behind each.
3. Why does try_clone matter for the ioeventfd/irqfd fast path? Who gets each copy?
4. What is a FAM struct, give a KVM example, and explain the division of labor between kvm-bindings and vmm-sys-util for it.
5. What do the ioctl_with_*! macros do, and which crate's source is full of them?
6. Trace one async seam (e.g. a virtio kick or an interrupt) down to the vmm-sys-util primitive at the bottom.

Common bugs and symptoms

You've reached the floor of the stack. From here, go build: start with Lab R1: Build a KVM VM (kvm-ioctls + vm-memory + EventFd), then Lab R2: Load a Kernel (linux-loader), Lab R3: Drive a Virtqueue (virtio-queue), and Lab R4: Contribute to rust-vmm — your first upstream PR. Then return to the deep dives: every crate here is the foundation under one of them.