Project 2: A Production-Quality UFFD Page-Fault Handler

When Firecracker restores a microVM from a snapshot, it has to get guest RAM back into the address space. The simple way is MAP_PRIVATE over the memory file: the kernel faults pages in lazily, copy-on-write, as the guest touches them. The powerful way is userfaultfd (UFFD): Firecracker registers guest memory with the kernel's userfaultfd mechanism and hands the fault-handling file descriptor to a separate process you write. Now every first-touch of a guest page traps into your handler, which decides where the bytes come from — the local memory file, a remote store, a compressed blob, a prefetch buffer — and installs them with UFFDIO_COPY. This is how restore-from-snapshot works at scale: you control the memory backend.

This project asks you to build that handler to production quality: a standalone UFFD memory backend with lazy loading, prefetching, and metrics, and then to benchmark restore latency against the naive MAP_PRIVATE path and against a naive page-at-a-time handler. It is the cleanest "land it as an example/tool" project in the portfolio, because Firecracker ships an example UFFD handler precisely so the community can build better ones, and the maintainers welcome improvements and diagnostics there.

Note: Read the snapshotting deep dive and do the snapshotting masterclass, especially Lab 2: a UFFD page-fault handler. This brief assumes you know the two-file snapshot model (state file + memory file), the mem_backend{backend_path, backend_type: File|Uffd} load option, and what track_dirty_pages / diff snapshots are. It also assumes you understand userfaultfd(2), UFFDIO_REGISTER, UFFDIO_COPY, and UFFDIO_ZEROPAGE. If those are fuzzy, do the lab first.

Problem & motivation

The UFFD path exists because, at Lambda/Fargate scale, you do not want to read an entire multi-hundred-MiB memory file off disk (or the network) before a function can respond. You want the guest to start immediately and pay for memory pages only as they are actually touched — and you want to prefetch the pages you know it will touch, so the guest rarely stalls on a fault.

Firecracker makes this possible but deliberately does not ship the policy: it hands you a UFFD and an ordered description of the memory regions, and your handler decides everything else. The shipped example handler is intentionally simple. The interesting engineering lives in the policy:

• Naive handlers fault one page at a time, synchronously. Every fault is a round trip; the guest blocks on each. Under a workload that touches thousands of cold pages early (kernel init, language runtime warm-up), the synchronous fault storm dominates restore-to-responsive latency.
• A good handler prefetches. When a fault lands at page N, the pages around it (and the pages a previous run learned were hot) are likely next. Installing a run of pages on one fault, or proactively populating known-hot regions before the guest asks, collapses the fault count.
• A good handler is observable. How many faults? How many were served from a prefetch buffer (a "hit") vs. a cold read (a "miss")? What is the fault-service latency distribution? Without metrics you cannot tell a good policy from a lucky one.

The motivation is leverage: restore latency is the headline number for snapshot-based serverless, and the page-fault policy is where you move it.

What you'll build

A standalone UFFD handler process (a separate binary; it is not part of the Firecracker process — that is the whole point of UFFD's privilege separation) that:

1. Receives the UFFD and the memory-region layout from Firecracker over a Unix socket at restore time.
2. Serves page faults from a backing memory file with UFFDIO_COPY (and UFFDIO_ZEROPAGE for known-zero pages).
3. Implements at least one prefetch policy (e.g. fault-around / run-length, plus an optional access-trace-guided warm set).
4. Emits metrics: fault count, prefetch hits/misses, bytes installed, and a fault-service latency histogram.
5. Comes with a benchmark comparing restore-to-responsive latency across MAP_PRIVATE, a naive single-page UFFD handler, and your handler.

The Firecracker side is unchanged — you configure mem_backend{backend_type: "Uffd", backend_path: "<your socket>"} on PUT /snapshot/load. The build is the handler and the measurement.

Prerequisites

• Level 9, especially Lab 2: snapshot/restore.
• Snapshotting deep dive and the snapshotting masterclass + UFFD lab.
• Guest memory management deep dive and rust-vmm vm-memory (how Firecracker lays out guest RAM and where the regions come from).
• Engineering essay snapshotting at scale.
• Working knowledge of userfaultfd(2) and SCM_RIGHTS fd-passing over a Unix socket.

Phased plan

Phase 0 — Run the shipped example handler and trace the protocol (1 day)

Find and run Firecracker's own example UFFD handler, then read how Firecracker hands it the UFFD and the region layout.

# The example handler + the docs that describe the handshake:
rg -rn "uffd|userfaultfd|UFFDIO_COPY|GuestRegionUffdMapping|SCM_RIGHTS" src/ tests/ docs/ | head -40
find . -iname '*uffd*' -o -iname '*userfault*' | head
# The Firecracker side: how the UFFD socket is wired on snapshot load:
rg -n "Uffd|backend_type|backend_path|create_guest_memory_uffd|register_uffd" src/vmm/src/

Anti-staleness: the example handler has moved between src/firecracker/examples/, a tests/host_tools/ helper, and a standalone crate across releases. Do not trust a path — find it on your branch, and read the matching page under docs/snapshotting/. Confirm the exact JSON shape of the region mapping message Firecracker sends; that wire contract is what your handler must parse.

Produce capstone-work/uffd-protocol.md: the exact handshake — Firecracker connects to your socket, sends the ordered guest-memory region mappings, passes the UFFD via SCM_RIGHTS; then for each fault your handler reads a uffd_msg, finds the region, and installs pages. Cite the code.

Phase 1 — A correct single-page handler (the baseline you will beat)

Build the smallest correct handler: accept the connection, receive the UFFD and regions, loop on read() of the UFFD, and for each UFFD_EVENT_PAGEFAULT install exactly the faulting page from the memory file with UFFDIO_COPY.

#![allow(unused)]
fn main() {
// Sketch — use the `userfaultfd` crate or raw ioctls; rg the example for the real shape.
loop {
    let event = uffd.read_event()?;                 // blocks until a fault
    if let Event::Pagefault { addr, .. } = event {
        let page = align_down(addr, PAGE_SIZE);
        let region = self.region_for(page);          // from the mapping handshake
        let src = region.file_offset_for(page);      // bytes in the memory file
        unsafe { uffd.copy(src_ptr(src), page as *mut _, PAGE_SIZE, true)?; }
        self.metrics.faults += 1;
    }
}
}

Milestone 1: restore a real microVM through your handler and get a login prompt. This is the correctness gate — if the guest boots and runs, your region math and UFFDIO_COPY are right. Restore through MAP_PRIVATE too and confirm identical guest behavior.

Phase 2 — Prefetching

Now beat the baseline. Implement at least one prefetch policy and make it configurable so you can compare:

The right design separates mechanism (install a run of pages) from policy (which run). Make the run length and warm-set a parameter so the benchmark can sweep them.

Tip: Beware over-prefetching. Every page you install that the guest never touches is wasted I/O and resident memory — the exact thing UFFD exists to avoid. Your metrics (Phase 3) must let you see the prefetch waste ratio, not just the fault count.

Milestone 2: the prefetch handler restores correctly and demonstrably reduces the fault count on a real workload.

Phase 3 — Metrics and observability

Instrument the handler so a policy can be judged:

• Total faults, total bytes installed, UFFDIO_COPY vs. UFFDIO_ZEROPAGE counts.
• Prefetch hits (a faulting page was already installed by prefetch — i.e. it never faulted) vs. waste (prefetched pages never touched).
• A fault-service latency histogram (time from read_event to copy complete).
• Emit as JSON to stderr/a file on shutdown, and optionally a live counter.

This observability is itself a contribution: a diagnostic UFFD handler that reports why a restore was slow is something the community wants.

Phase 4 — Benchmark restore latency

The headline deliverable. Measure restore-to-responsive latency: time from PUT /snapshot/load + resume to the guest doing observable work (a serial-console marker, or a guest agent pinging back). Compare three backends on the same snapshot and workload:

backend                  restore→responsive   faults   prefetch waste   p99 fault (µs)
MAP_PRIVATE (File)             <a> ms            n/a        n/a              n/a
naive single-page UFFD         <b> ms          <high>       0%             <c>
your prefetch UFFD             <d> ms          <low>       <e>%            <f>

Run each ≥20 times, report median and p99, and state the host, kernel, CPU, page size, snapshot size, and whether the memory file was warm or cold in the page cache (this matters enormously — characterize it).

Key code areas

The userfaultfd rust crate is the ergonomic way to do the ioctls; the raw path through vmm-sys-util/libc is the educational one. Decide and document.

Design considerations & trade-offs

• Privilege separation is the point. The handler is a separate process with its own seccomp story; it can be more privileged or differently sandboxed than Firecracker. Do not collapse it into the VMM.
• Latency vs. residency. Aggressive prefetch lowers fault latency but raises resident memory and I/O — the opposite of why UFFD exists. The right policy is workload-dependent; your benchmark must show the trade-off curve, not a single point.
• Cold vs. warm page cache. If the memory file is in the host page cache, every backend looks fast and UFFD's advantage shrinks. Benchmark cold (drop caches) to see the real picture, and say which you measured.
• Zero pages. Guest RAM is mostly zero at restore. UFFDIO_ZEROPAGE is far cheaper than copying zeros from a file. Detecting and special-casing zero pages is a real win — but it interacts with diff snapshots, so verify correctness.
• Diff snapshots. With track_dirty_pages, the memory file is a base plus a diff (rebase-snap rebases them). Your handler's region math must account for whatever layout you load against. Start with full snapshots; treat diff as a stretch.
• Robustness. A handler that panics or deadlocks hangs the guest forever. Handle UFFD_EVENT_REMOVE/UFFD_EVENT_UNMAP, partial reads, and a Firecracker that disconnects. Production quality means it does not wedge.

How to test & validate

• Correctness: restore the same snapshot through MAP_PRIVATE and through your handler; the guest must reach an identical state (same boot log, same workload output). A guest that boots correctly is strong evidence your region/offset math is right.

• Fault accounting: a unit-level test that drives a synthetic UFFD over a small mmap, faults known pages, and asserts your handler installs exactly the right bytes and counts faults/prefetches correctly.

• Integration: use the Firecracker pytest snapshot tests as a template —

  rg -n "uffd|snapshot|restore|def test_" tests/integration_tests/functional/ | head
  

  — and add a test that loads a snapshot with backend_type: Uffd pointing at your handler and asserts the guest resumes.

• Benchmark rigor: scripted, repeatable, cache state controlled, ≥20 iterations, median + p99 reported. The harness is a deliverable — make it one command.

Stretch goals

• Trace-guided prefetch: record the fault order on a first restore, persist it, and replay it as a warm-set on subsequent restores; show the hit-rate climb.
• Remote backend: serve pages from a network store (or a compressed local blob) instead of a plain file, and measure the latency hit — this is the real Lambda-scale shape.
• Diff-snapshot support: handle a base+diff memory layout correctly.
• Concurrency: serve faults from a small thread pool and show whether it helps or whether the UFFD read is the bottleneck.
• Upstream a diagnostic improvement to the shipped example handler (better metrics output, a zero-page fast path) — a clean, mergeable PR.

What a strong deliverable looks like

A strong deliverable is a standalone UFFD handler that restores real microVMs correctly, implements a configurable prefetch policy, emits actionable metrics, and comes with a reproducible benchmark showing it beats both MAP_PRIVATE and a naive single-page handler on restore-to-responsive latency — with the cold/warm cache caveat stated and the prefetch-waste trade-off shown.

The upstreaming path is the cleanest in the portfolio for a "tool":

1. The shipped example handler is the target. Improvements to it — better metrics, a zero-page fast path, a documented prefetch option, clearer region handling — are mergeable on their own merits as they help every UFFD user. Find the live state: gh issue list --repo firecracker-microvm/firecracker --search "uffd OR userfaultfd OR snapshot restore" and check docs/snapshotting/.
2. Comment before you code. Snapshot/UFFD behavior is compatibility-sensitive; confirm the maintainers want the change.
3. Bring the numbers. A restore-latency improvement with a reproducible benchmark and a clear methodology is exactly what gets a snapshotting PR reviewed.

Even if nothing lands upstream, a handler with a benchmark and a findings write-up is a portfolio-grade artifact: it demonstrates you understand the single most important performance path in snapshot-based serverless. A finished version at 90+ on the rubric is maintainer-grade memory-subsystem work.

Next: Project 3 — a custom rate-limiting policy for another performance-policy build, or Project 5 — a CPU template to follow snapshots into the cross-CPU portability problem.