Boot-Time Optimization

The headline number in Firecracker's design is < 125 ms to application code, and it is not marketing — it is a hard requirement that falls directly out of the serverless use case. A function that is invoked once and then idles must pay its startup cost on the first request, in the user's latency budget. If the microVM takes a second to boot, the product is unusable; if it takes tens of milliseconds, the product is invisible. Boot time is therefore a first-class engineering concern, not an afterthought, and it is one of the most measurable, most-watched, and most contributable areas in the project.

This chapter is about what is actually on that critical path, the techniques that keep it short, how snapshots sidestep boot entirely, and — crucially — how the project measures boot time and catches regressions, because an optimization you cannot measure is a guess.

Note: This chapter builds on the boot sequence deep dive and the guest memory deep dive. It does not re-derive the zero page or the e820 map — it asks the performance question: what costs time, and how do you cut it?

What is on the critical path

A microVM boot is three phases, and time spent in each is qualitatively different.

  InstanceStart
        │
        ▼
┌───────────────────────┐   Firecracker (host userspace) — you control this
│ 1. VMM setup          │   • parse + copy the vmlinux ELF into guest RAM
│    (host-side)        │   • build boot_params / zero page / e820 (x86)
│                       │   • create vCPUs, set initial registers (long mode)
│                       │   • attach the minimal device set (MMIO, no PCI probe)
│                       │   • set up guest memory regions, register with KVM
└──────────┬────────────┘
           ▼
┌───────────────────────┐   the guest kernel — you influence this via config
│ 2. Guest kernel init  │   • decompress? (NO — vmlinux is uncompressed)
│                       │   • probe devices (FAST — few devices, MMIO not PCI)
│                       │   • init drivers, mount rootfs, page-table setup
└──────────┬────────────┘
           ▼
┌───────────────────────┐   userspace — the workload's own concern
│ 3. Userspace / init   │   • init → your application → first byte of work
└───────────────────────┘

The first phase is Firecracker's code and is where the VMM's own efficiency shows. The second is the guest kernel, which Firecracker shapes through the device model, the boot protocol, and the kernel command line. The third is the workload's. The < 125 ms budget is dominated by phases 1 and 2, and almost every boot-time technique attacks one of them.

cd ~/src/firecracker
# Phase 1: kernel load + boot config + vCPU setup. Find the load path.
rg -n "fn load_kernel|Elf|load_cmdline|configure_system|setup_boot|build_microvm_for_boot" \
  src/vmm/src/builder.rs src/vmm/src/arch/x86_64/ | head

The techniques: how the path is kept short

Every entry in this table is a deliberate design choice you have seen elsewhere in this section, now viewed through the lens of boot latency. This is the minimal device model paying a performance dividend on top of its security one.

The kernel command line is a surprisingly large lever. Look at the boot args the project's own boot-time test uses — every token is there to cut something:

rg -n "DEFAULT_BOOT_ARGS|boot_args|nomodule|nr_uarts|i8042|swiotlb|cryptomgr" \
  tests/integration_tests/performance/test_boottime.py

Tokens like 8250.nr_uarts=0, i8042.noaux/nomux/nopnp, nomodule, and cryptomgr.notests each disable a guest-kernel init path that the microVM does not need — serial-probe delays, keyboard-controller probing, module loading, crypto self-tests. They are the guest-side mirror of the host-side minimalism: don't initialize what you'll never use.

# The guest kernel configs are also part of the boot-time story.
ls resources/guest_configs/
rg -rn "CONFIG_" resources/guest_configs/ | rg -i "module\|pci\|usb" | head

Snapshots: bypassing boot entirely

The most dramatic boot-time technique is not booting at all. A snapshot restore skips phases 1, 2, and 3 by reconstructing a microVM that has already booted and is sitting ready at the first request. Restore latency is bounded by the (tiny) state file plus the handful of guest pages touched before the first response — single-digit milliseconds, an order of magnitude under even the 125 ms boot budget.

cold boot:     [ VMM setup ][ kernel init ][ userspace init ] → ready  (~tens–125 ms)
snapshot:      [ map state + on-demand pages ]                → ready  (~ms)

This is why, for production serverless, boot-time optimization and snapshotting are two halves of one strategy: optimize the boot you do pay (for the first ever launch and for building the snapshot), then amortize it across every subsequent cold start via restore. A contributor reasoning about cold-start latency must hold both in view — see snapshotting at scale.

Measuring boot time: the harness

You cannot optimize what you cannot measure, and the project measures boot time with a clever, low-overhead mechanism: a pseudo "boot timer" device. The guest writes a magic byte to a known MMIO address the instant it finishes booting; Firecracker catches that MMIO write, diffs the current timestamp against the microVM's start time, and logs the result.

# The boot-timer pseudo device — find the magic value and the log line.
rg -n "BootTimer|MAGIC_VALUE_SIGNAL_GUEST_BOOT_COMPLETE|Guest-boot-time|start_ts" \
  src/vmm/src/devices/pseudo/boot_timer.rs

The mechanism is elegantly minimal: the device only handles a single-byte write at offset zero; if the byte is the magic value (123 — verify), it computes now - start_ts for both wall-clock and CPU time and emits a Guest-boot-time = N us / N ms log line. The guest is instrumented to write that byte at the end of boot, so the measured interval is true boot time from InstanceStart to guest-ready, with negligible measurement overhead.

sequenceDiagram
    participant FC as Firecracker (start_ts captured)
    participant Guest as guest kernel/init
    participant BT as BootTimer pseudo device
    FC->>Guest: InstanceStart (vCPU enters KVM_RUN)
    Note over Guest: kernel + userspace init...
    Guest->>BT: MMIO write byte=123 at offset 0
    BT->>BT: boot_time = now - start_ts
    BT->>FC: log "Guest-boot-time = N us / N ms"

The pytest performance suite drives this end to end and asserts the result is within spec:

# The boot-time integration test — the regression gate.
sed -n '1,60p' tests/integration_tests/performance/test_boottime.py
./tools/devtool test -- integration_tests/performance/test_boottime.py 2>&1 | tail -25

Tip: The test pins specific guest kernels and a known boot-args string, then parses the Guest-boot-time log line. When you benchmark, always record the exact kernel, rootfs, boot args, and host — a boot-time number without that context is unfalsifiable. The harness records them for exactly this reason.

How regressions are caught

Boot time is a defended number, not just a measured one. A change that adds 10 ms to boot is a regression even if it adds a useful feature, and the project treats it that way:

This is why a PR that touches the boot path — kernel loading, device attachment, the cmdline defaults, vCPU setup — draws boot-time scrutiny. If you add a device or an init step, the maintainers will ask what it does to the boot-time test, and "I didn't check" is not an answer. The discipline is the same as the minimal device model bar: new work must justify its cost on a number the project cares about.

Warning: Boot-time benchmarks are noisy. Host CPU frequency scaling, a cold page cache, a busy host, or NUMA effects (huge pages) can swamp the signal you are trying to measure. Run multiple iterations, pin CPUs, warm the cache, and report a distribution — not a single number. A "regression" that is within run-to-run noise is not a regression.

Where to contribute

Boot time is measurable, valued, and bounded — an ideal area for a contributor who likes performance work with a tight feedback loop.

gh issue list --repo firecracker-microvm/firecracker \
  --search "boot OR boottime OR latency in:title,body state:open" --limit 40
gh issue list --repo firecracker-microvm/firecracker --label "Type: Performance" --state open

On-ramps: profiling phase-1 VMM setup to find avoidable work; tightening the default guest kernel configs; boot-time benchmarks across kernel versions and page sizes; reducing boot-time benchmark noise; documenting the cmdline tokens and what each one buys. Pair any of these with the boot-time masterclass lab to build the hands-on intuition first.

This is the last chapter of the Engineering at Scale section. Take what you have built here — the ability to reason about Firecracker's design tensions — back to Real Issues & Roadmap to pick an area to own, then turn it into a contribution via the Capstone. The performance-density masterclass is where this reasoning becomes hands-on work.