Lab 2: Build a UFFD Page-Fault Handler

Background

In Lab 1 you restored with backend_type: File: Firecracker mmaps the memory file MAP_PRIVATE and lets the kernel page guest RAM in on demand. That is simple and fast, but the host kernel decides when and how pages arrive, and the page bytes must already be a file the kernel can mmap. The production serverless story needs more control than that — pages might live in a remote store, a compressed tier, or a deduplicated cache; you might want to fetch them over the network, prefetch around a fault, or count exactly which pages a workload touches. That control is what the Uffd backend buys, and the price is that you — not the kernel — must serve every page fault.

userfaultfd is a Linux mechanism (man 2 userfaultfd) that lets a userspace process register a memory range and receive its page faults as events on a file descriptor, then satisfy them with UFFDIO_COPY (copy bytes into the faulting page) or UFFDIO_ZEROPAGE. With backend_type: Uffd, Firecracker does not map the memory file itself. Instead it maps guest RAM as an empty (anonymous) region, registers that region with userfaultfd, and hands the resulting fd — plus a description of the guest memory layout — to a separate handler process over a Unix socket. From then on, every time the guest touches an un-populated page, the kernel blocks the faulting vCPU and delivers a fault message to your handler, which reads the right bytes out of the memory file and copies them into place. The vCPU resumes only after your handler answers.

This is an advanced build-it lab, and it is honest about being hard. You are writing a program that participates in the host kernel's fault path for a privileged VMM; a bug means a hung guest (you never answered a fault), a corrupted guest (you copied the wrong bytes), or a crash. You will model it on Firecracker's own UFFD example handler, which ships in the tree precisely so you do not have to invent the socket protocol or the fd-passing from scratch. You will read that example, write a minimal handler of your own that serves pages from the memory file, load a Lab-1 snapshot against it, and then watch individual pages fault in, one at a time, and measure the difference from the File backend.

This is a build-it (advanced) lab.

Why This Lab Matters for Contributors

• The UFFD backend is the foundation of snapshotting at scale (engineering/snapshotting-at-scale). You cannot review or extend Firecracker's UFFD support, the socket protocol, or the memory-layout message it sends to the handler if the handler is a black box to you. Writing one makes every part of that interface concrete.
• The handler is the single most important reason restore is lazy and cheap: a microVM resumes after touching a few hundred pages, not after reading gigabytes. Owning the demand-paging path is what lets you reason about restore latency, working sets, and prefetch.
• It is a real contribution surface. The example handler, the layout message, the fault-handling edge cases (removed/zeroed pages, UFFD_EVENT_REMOVE from the balloon's madvise), and the docs are all places where good PRs land. A contributor who has written a handler spots a protocol bug a reader never would.

Prerequisites

# Readiness check.
cd ~/firecracker
B=build/cargo_target/x86_64-unknown-linux-musl/release
test -x $B/firecracker && echo "firecracker built"
ls -lh base.state base.mem 2>/dev/null || echo "produce these in Lab 1 first"

# userfaultfd must be available. On many distros it is gated.
sysctl vm.unprivileged_userfaultfd 2>/dev/null   # 0 = root-only userfaultfd (fine, FC runs privileged)
grep -i userfaultfd /boot/config-$(uname -r) 2>/dev/null || zcat /proc/config.gz 2>/dev/null | grep -i userfault

# Find Firecracker's example handler and the UFFD docs — your model. Do NOT trust a path; locate it.
find . -iname '*uffd*' -o -ipath '*examples*uffd*' 2>/dev/null
ls docs/snapshotting/ ; rg -ln "uffd|userfaultfd|UFFD" docs/snapshotting/

Warning: This lab makes your code part of the guest's memory fault path. A handler that fails to answer a fault hangs the guest forever; one that answers with wrong bytes silently corrupts it. Develop with a small guest (the 256 MiB Lab-1 VM), log every fault, and never point a half-finished handler at a workload you care about. And as always: only restore snapshots you produced.

Step-by-Step Tasks

Step 1: Read Firecracker's UFFD example handler — it is your template

Firecracker ships an example UFFD handler so you do not reinvent the wire format. Read it before writing a line. Locate it (path drifts — it has lived under src/firecracker/examples/ and the docs reference it):

# Find the example and read its three jobs: accept the socket, receive the fd + layout,
# loop on fault events and serve pages.
find . -iname '*uffd*' 2>/dev/null
EX=$(find . -ipath '*examples*uffd*' -name '*.rs' 2>/dev/null | head -1); echo "$EX"
rg -n "UnixListener|recv|SCM_RIGHTS|userfaultfd|UffdMsg|EVENT_PAGEFAULT|copy|zeropage|GuestRegionUffdMapping|mappings" "$EX"

# The Firecracker side: what it sends the handler, and what the layout message contains.
rg -n "Uffd|userfaultfd|UffdManager|backend_type|GuestRegionUffdMapping|send_fds|page_size|base_host_virt_addr|offset" src/vmm/src/
rg -n "uffd|userfaultfd" docs/snapshotting/*.md

From the example, extract the contract — this is what your handler must implement:

Note: The exact field names of the layout message are version-sensitive — read them off the example and the Firecracker source (rg -n "GuestRegionUffdMapping" src/vmm/src/), do not copy them from here. The shape (a list of regions, each with a host base address, a length, and an offset into the backing file) is stable; the spelling is not. (Verify on your branch.)

Step 2: Write a minimal handler that serves pages from the memory file

Now write your own, deliberately minimal. The userfaultfd crate wraps the ioctls; you may also use raw libc if you prefer to see every syscall. The skeleton below is illustrative — the layout-message parsing and the socket/fd-receive must match what you read in Step 1 on your branch.

# Cargo.toml for the handler (a tiny standalone bin).
[package]
name = "uffd-handler"
version = "0.1.0"
edition = "2021"

[dependencies]
userfaultfd = "0.8"        # verify a current version
libc = "0.2"
serde = { version = "1", features = ["derive"] }
serde_json = "1"

// src/main.rs — a minimal UFFD memory backend for Firecracker restore.
// MODEL THIS ON THE IN-TREE EXAMPLE. The wire format (fd-passing + layout message)
// must match the Firecracker version you restore with — read it from the source.
use std::fs::File;
use std::os::unix::io::{FromRawFd, RawFd};
use std::os::unix::net::UnixListener;
use std::ptr;

use userfaultfd::Uffd;

/// One guest memory region, as Firecracker describes it to the handler.
/// (Field names per the in-tree GuestRegionUffdMapping — VERIFY on your branch.)
#[derive(serde::Deserialize, Debug)]
struct Region {
    base_host_virt_addr: u64, // where this region is mapped in FC's address space
    size: usize,              // region length in bytes
    offset: u64,              // byte offset of this region inside the memory file
}

fn main() {
    let sock_path = std::env::args().nth(1).expect("usage: uffd-handler <socket> <mem_file>");
    let mem_path = std::env::args().nth(2).expect("usage: uffd-handler <socket> <mem_file>");

    let _ = std::fs::remove_file(&sock_path);
    let listener = UnixListener::bind(&sock_path).expect("bind socket");
    eprintln!("handler: listening on {sock_path}");

    // 1) Accept Firecracker, receive the userfaultfd (SCM_RIGHTS) + the layout message.
    let (stream, _) = listener.accept().expect("accept");
    let (uffd_raw, regions): (RawFd, Vec<Region>) = recv_uffd_and_layout(&stream);
    let uffd = unsafe { Uffd::from_raw_fd(uffd_raw) };
    eprintln!("handler: got uffd + {} region(s)", regions.len());

    // 2) mmap the memory file read-only — our source of truth for page bytes.
    let mem = File::open(&mem_path).expect("open mem file");
    let mem_len = mem.metadata().unwrap().len() as usize;
    let mem_ptr = unsafe {
        libc::mmap(ptr::null_mut(), mem_len, libc::PROT_READ,
                   libc::MAP_PRIVATE, std::os::unix::io::AsRawFd::as_raw_fd(&mem), 0)
    };
    assert!(mem_ptr != libc::MAP_FAILED, "mmap mem file");

    let page_size = unsafe { libc::sysconf(libc::_SC_PAGESIZE) } as usize;

    // 3) The fault loop: every guest page touch blocks a vCPU until we answer.
    loop {
        let event = uffd.read_event().expect("read uffd event");
        match event {
            Some(userfaultfd::Event::Pagefault { addr, .. }) => {
                let fault_addr = addr as u64 & !((page_size as u64) - 1); // align down
                // Find which region contains the fault, compute the file offset.
                let region = regions.iter()
                    .find(|r| fault_addr >= r.base_host_virt_addr
                           && fault_addr < r.base_host_virt_addr + r.size as u64)
                    .expect("fault outside known regions");
                let in_region = fault_addr - region.base_host_virt_addr;
                let file_off = region.offset + in_region;
                let src = unsafe { mem_ptr.add(file_off as usize) };

                // Copy one page from the memory file into the guest page. UFFDIO_COPY.
                unsafe {
                    uffd.copy(src, fault_addr as *mut _, page_size, true)
                        .expect("UFFDIO_COPY");
                }
                eprintln!("served fault @ {:#x} (file off {:#x})", fault_addr, file_off);
            }
            // The balloon's madvise(MADV_DONTNEED) shows up as a REMOVE event; you must
            // handle it or a later touch of that page can deadlock. (Verify behavior.)
            Some(userfaultfd::Event::Remove { .. }) => { /* mark range as zero-on-fault */ }
            other => eprintln!("unhandled uffd event: {other:?}"),
        }
    }
}

// Receive the passed userfaultfd via SCM_RIGHTS and read the JSON layout that follows.
// IMPLEMENT THIS to match the in-tree example's exact framing (it is the fiddly part).
fn recv_uffd_and_layout(_stream: &std::os::unix::net::UnixStream) -> (RawFd, Vec<Region>) {
    unimplemented!("port the fd-receive + layout parse from the in-tree example handler")
}

The hard, error-prone parts — and where almost all bugs live — are: (1) receiving the fd from ancillary SCM_RIGHTS data correctly; (2) parsing the layout with the exact field names your branch uses; (3) page alignment (you must UFFDIO_COPY a page-aligned address and a page-sized length); and (4) the file-offset math (host fault address → region → offset into base.mem). Get those four right and the rest is a loop.

Tip: Start by only logging faults and serving them with UFFDIO_ZEROPAGE (zero-filled) to prove the plumbing works end to end, then switch to UFFDIO_COPY from the real memory file. A guest restored on zero pages will obviously misbehave, but it proves your socket, fd-receive, layout parse, and fault loop are correct before you add the file math.

Step 3: Build the handler and start it before restore

cargo build --release          # in the handler's directory
HANDLER=./target/release/uffd-handler

# Start the handler FIRST — it must own the socket before Firecracker connects.
SOCK=/tmp/uffd.sock
$HANDLER $SOCK ./base.mem &
HPID=$!
sleep 0.2
test -S $SOCK && echo "handler listening on $SOCK"

The ordering matters: the handler binds the socket; Firecracker, during PUT /snapshot/load, connects to it as a client, sends the fd and layout, and expects the handler to start answering faults immediately on resume.

Step 4: Load the snapshot with backend_type: Uffd

Restore in a fresh Firecracker process, pointing backend_path at the handler's socket (not the memory file — the handler owns the file now):

API=/tmp/fc-uffd.sock; rm -f $API
sudo $B/firecracker --api-sock $API &
for i in $(seq 1 100); do [ -S $API ] && break; done

curl -sX PUT --unix-socket $API --data '{
  "snapshot_path": "./base.state",
  "mem_backend": { "backend_path": "/tmp/uffd.sock", "backend_type": "Uffd" },
  "resume_vm": true
}' http://localhost/snapshot/load

If everything is wired correctly, the restore returns success and the guest resumes — but unlike the File backend, no guest RAM has been read yet. The first thing you should see is your handler logging a burst of faults as the resumed guest touches its working set.

Note: backend_path for Uffd is the Unix socket path, not a file path. This is the single most common configuration mistake. Confirm the field meaning in the source: rg -n "backend_path|backend_type|Uffd|MemBackendType" src/vmm/src/.

Step 5: Observe on-demand page faults, one at a time

This is the payoff. With the File backend the kernel pages RAM in invisibly; with your handler, every fault is a line you printed. Watch them:

# Your handler is logging each served fault to stderr. Count and rate them.
# As the guest does work in its serial console, faults stream in.
# In another terminal, drive a touch in the guest and watch the handler:
#   guest$ cat /proc/meminfo ; dd if=/dev/zero of=/tmp/x bs=1M count=8   # touches pages

# Count served faults over the first few seconds of resume:
# (your handler prints "served fault @ ..."; pipe its stderr to a file)
$HANDLER $SOCK ./base.mem 2>uffd.log &
# ... restore, exercise the guest ...
wc -l uffd.log
sort -u uffd.log | wc -l          # unique pages served

You are now seeing the working set materialize. A freshly resumed microVM that touches a few hundred pages served a few hundred faults — not the whole 256 MiB. That gap is exactly why UFFD restore is cheap: you pay for the pages the guest actually uses.

Step 6: Measure — UFFD vs File, and the working set

Compare the two backends honestly. Restore the same snapshot once with File (Lab 1) and once with your Uffd handler, and measure both the resume latency and the pages actually loaded.

# Resume latency (time the load call) for each backend — reuse Lab 1's restore_once timer.
# File:  backend_type File, backend_path ./base.mem
# Uffd:  backend_type Uffd, backend_path /tmp/uffd.sock (handler running)

# Pages loaded: File backend faults are kernel-internal (read /proc/<fc_pid>/smaps_rollup
# for Referenced/Rss); Uffd faults are COUNTABLE — your handler logged each one.
FC=$(pgrep -n firecracker)
grep -E 'Rss|Referenced' /proc/$FC/smaps_rollup 2>/dev/null

Interpret it:

The takeaway you must be able to state: UFFD does not make restore faster than the File backend — it makes restore programmable. The File backend is faster per fault (no userspace round-trip), but the UFFD handler can fetch pages from places the kernel cannot, prefetch around faults, dedup, and report exactly which pages a workload needs — which is the input to every snapshot-at-scale optimization.

Implementation Requirements / Deliverables

• A reading of the in-tree example handler with the wire contract written out: what fd Firecracker passes, what the layout message contains, how faults are served.
• A working handler binary that binds a socket, receives the userfaultfd and layout, and serves faults from base.mem with UFFDIO_COPY.
• A successful PUT /snapshot/load with backend_type: Uffd against your handler, with the guest resuming and your in-guest Lab-1 marker present.
• A log showing individual page faults served, with a count of total and unique pages served during resume + light guest work.
• A comparison of Uffd vs File: resume latency for each and the number of pages served (UFFD: counted directly; File: inferred from RSS/smaps), plus a sentence on why you would choose UFFD despite no latency win.

Troubleshooting

PUT /snapshot/load returns immediately but the guest hangs on resume

Your handler isn't serving faults. Either it never received the fd/layout (the SCM_RIGHTS receive is wrong), or it received them but is blocked/crashed. Confirm the handler is still alive (kill -0 $HPID), that it logged "got uffd + N region(s)", and that it's printing faults. A hang with zero faults logged means the fd/layout handshake failed; a hang after some faults means your offset math walked off the region (re-check the alignment and the find-the-region logic).

"connection refused" / Firecracker can't reach the socket

The handler must own the socket before restore. Start the handler, confirm test -S $SOCK, then call /snapshot/load. Also confirm backend_path is the socket path, not the memory file.

Guest resumes but is corrupted / panics shortly after

You served the wrong bytes — almost always the file-offset math. The offset field is per-region; a faulting host address maps to region.offset + (fault_addr - region.base_host_virt_addr). If you used the raw fault address as a file offset, or ignored multiple regions, you copied garbage. Log (fault_addr, region, file_off) for the first few faults and verify by hand against the layout message.

UFFDIO_COPY fails with EINVAL

The address or length isn't page-aligned, or you're copying into a page that's already populated (a double-fault from a race). Align the fault address down to page_size and copy exactly one page_size. If you see EEXIST, the page was already filled — that's benign; skip it.

Faults stop and the guest deadlocks after the balloon inflates

The balloon's madvise(MADV_DONTNEED) removes pages, which userfaultfd reports as a REMOVE event. If you ignore it, a later touch of that range can wedge. Handle UFFD_EVENT_REMOVE (or serve those re-faults with UFFDIO_ZEROPAGE). See the example handler for how it tracks removed ranges, and the balloon deep dive.

userfaultfd syscall fails with EPERM

Unprivileged userfaultfd is disabled on your host (vm.unprivileged_userfaultfd=0). Firecracker runs privileged so this is usually fine in the real path; if your handler needs it, run it with privilege or enable the sysctl on a dev host (not in production).

Expected Output

# Handler, on connect:
handler: listening on /tmp/uffd.sock
handler: got uffd + 1 region(s)

# Firecracker:
$ curl ... /snapshot/load   ->  (success; guest resumes)

# Handler, as the guest touches its working set:
served fault @ 0x7f3a40001000 (file off 0x1000)
served fault @ 0x7f3a40002000 (file off 0x2000)
served fault @ 0x7f3a40010000 (file off 0x10000)
...
$ wc -l uffd.log
   327 uffd.log              # ~hundreds of faults for resume + light work
$ sort -u uffd.log | wc -l
   318                       # nearly all unique pages — that's the working set

# restored guest:
root@guest:~# cat /run/marker     # the Lab-1 value — resume, not reboot
1718700000

A few hundred served faults to resume a 256 MiB guest — proof that you paid only for the pages the guest touched.

Stretch Goals

1. Prefetch around faults. When you serve a fault at offset O, also UFFDIO_COPY the next K pages. Measure the change in fault count and resume-to-steady-state time. Where does prefetch help and where does it waste work?
2. Serve pages over the network. Replace the mmap'd file with an HTTP/vsock fetch of each page from a remote base.mem. This is the snapshot-at-scale pattern (engineering/snapshotting-at-scale) in miniature. Measure the latency cost per fault.
3. Handle the balloon properly. Inflate the balloon in the resumed guest and confirm your REMOVE-event handling keeps it alive. Reproduce the deadlock first by ignoring the event, then fix it — and explain the bug.
4. Working-set fingerprint. Dump the ordered list of pages your handler served on first resume. That ordered fingerprint is the input to a real prefetch/warming strategy: a future restore could UFFDIO_COPY exactly that set up front.
5. Compare against the in-tree example head to head. Run Firecracker's own example handler against the same snapshot and diff its served-page order against yours.

Validation / Self-check

Answer without notes. These gate completion.

1. What does backend_type: Uffd change about how Firecracker maps guest RAM compared to File? What does Firecracker hand the handler, and over what?
2. Walk a single guest page fault end to end: from the guest touch to your handler's UFFDIO_COPY and the vCPU resuming.
3. What is in the layout message, and how do you turn a faulting host address into an offset in the memory file?
4. Why is backend_path a socket path for Uffd but a file path for File?
5. Why must the handler own the socket before /snapshot/load?
6. UFFD restore is not faster per fault than File restore. So why use it? Give two concrete capabilities it unlocks.
7. The guest deadlocks after the balloon inflates. What event did you fail to handle, and why does ignoring it wedge a later access?

When you can serve a real restore from your own handler, watch faults stream in, and explain why UFFD trades raw speed for programmability, you've completed Lab 2. Continue to Lab 3 — Snapshot Compatibility, where the subject stops being the memory and becomes the format — the serialized state file as a public compatibility surface that you must never break casually.