Firecracker Warm-Up: From User to Contributor

Before you read a single line of vstate/vcpu/mod.rs or devices/virtio/net/, you need to have sat in the seat of the person whose workload Firecracker serves: someone booting a tiny, fast, isolated Linux VM, giving it a network and a disk, snapshotting it, and feeding it metadata. The engineers who built Firecracker's boot path, virtio devices, and snapshot machinery were solving specific, painful problems that show up in serverless production every day. If you skip that context and go straight to the source, you will memorize struct names without understanding why the design exists.

This is the missing first mile, and it is the most important page in this section. You will run Firecracker from the outside — as a user — across five real scenarios: boot, networking, a second drive, snapshot/restore, and the metadata service. After each scenario, the chapter maps what you observed back to the crate/module that owns it, with a real rg/find you run yourself. By the end, every internal name will feel like an old acquaintance.

Everything here assumes the setup from the prerequisites: a Linux host with /dev/kvm, a firecracker binary you built with tools/devtool build, and a vmlinux + rootfs.ext4 from docs/getting-started.md. If you have not read the Hitchhiker's Guide yet, read it first — it gives you the vocabulary every "under the hood" section below leans on.

# Anchor a few variables you'll reuse in every scenario.
cd ~/src/firecracker
FC=$(find build/cargo_target -type f -name firecracker | grep release | head -1)
API=/tmp/firecracker.socket
echo "binary: $FC"   # if empty: tools/devtool build --release

What Firecracker Actually Is (Two Sentences)

Firecracker is a userspace Virtual Machine Monitor (VMM) that uses KVM to run microVMs — minimal, fast-booting Linux VMs with a tiny virtio device model — and is controlled entirely through a REST API over a Unix domain socket. It is the engine: one Firecracker process is exactly one microVM, and density at scale comes from running thousands of these processes, each jailed and seccomp-filtered, which is how it powers AWS Lambda and AWS Fargate.

Hold that boundary the whole curriculum: Firecracker is the VMM, KVM is the kernel module beneath it, the guest is an unmodified Linux running inside, and orchestrators like firecracker-containerd or Kata sit above the API socket. When something misbehaves, your job as a contributor is to attribute it correctly — VMM bug, guest-kernel config, KVM/host setup, or orchestrator. This warm-up keeps that line sharp.

Where Firecracker Sits in the Isolation Spectrum

┌──────────────────────────────────────────────────────────────────────────────┐
│                         Isolation / Sandboxing Spectrum                       │
│                                                                              │
│  Weaker isolation, lighter  ◄──────────────────────────►  Stronger, heavier   │
│                                                                              │
│  runc          gVisor          Firecracker / Cloud Hypervisor      QEMU       │
│  (namespaces,  (userspace      (KVM microVM, minimal devices)      (full      │
│   shared       kernel /                                            device     │
│   kernel)      Sentry)         Kata can sit on top of either ──►   model)     │
│                                                                              │
│  ──────────────────────────────────────────────────────────────────────────  │
│  Beneath Firecracker:  KVM (Linux module)  →  VT-x / AMD-V / ARM hardware     │
│  Above Firecracker:    firecracker-containerd · Kata · Fly.io · Vercel ...    │
│  Built from:           rust-vmm crates (kvm-ioctls, vm-memory, linux-loader)  │
└──────────────────────────────────────────────────────────────────────────────┘

Firecracker sits where you want a real hardware-virtualization boundary (the guest kernel is untrusted) but with container-grade density and startup. gVisor is a different model (intercepts guest syscalls in userspace, no hardware virt by default). Cloud Hypervisor shares the same rust-vmm crates but targets full cloud guests (PCI, ACPI, hotplug). Knowing the neighbors tells you when a reported "Firecracker is missing feature X" is really "you want a general-purpose VMM, not a microVM."

Firecracker vs. its closest relatives

The key insight you carry throughout: a VM is strongly isolated because the guest is confined behind CPU virtualization extensions — but the VMM itself is privileged host code and part of the attack surface. A guest that compromises the VMM escapes. That is exactly why a minimal VMM in a memory-safe language matters. The full table lives in Firecracker vs. Other VMMs.

The Control Surface: One Socket, A Few PUTs

Everything you do as a user goes through the API socket. There is no firecracker run vm.img command. You start the process, it listens on the socket, you configure the machine with PUTs (most of them pre-boot only), and then you InstanceStart.

   you  ──curl --unix-socket /tmp/fc.sock──►  ┌──────────────────────────────┐
                                              │  Firecracker process         │
   PUT /boot-source     (kernel + cmdline)    │   • API thread  (this socket)│
   PUT /drives/{id}     (virtio-block)        │   • VMM thread  (devices,    │
   PUT /network-interfaces/{id}  (virtio-net) │       event loop, MMDS)      │
   PUT /machine-config  (vcpus, mem)          │   • vCPU thread × N (KVM_RUN) │
   PUT /mmds, /vsock, /balloon, ...           └──────────────────────────────┘
   PUT /actions  {InstanceStart}  ───────────────────►  the microVM boots
   PATCH /vm {Paused|Resumed}, PUT /snapshot/create|load

The OpenAPI definition for the whole surface lives in the repo — read it once:

# The authoritative API spec (every endpoint, every field). Locate it, don't guess.
find . -name 'firecracker.yaml' -path '*swagger*'
#   src/firecracker/swagger/firecracker.yaml
rg -n 'paths:|/boot-source|/drives|/network-interfaces|/snapshot|/mmds' src/firecracker/swagger/firecracker.yaml | head -40

Tip: A config file is the batch alternative to the socket. firecracker --no-api --config-file vm.json boots a microVM from a single JSON document with kebab-case sections (boot-source, drives[], machine-config, network-interfaces[], mmds-config, …). Config-file boot starts the VM immediately — handy for reproducible labs.

Scenario A: Boot a microVM and Watch the Serial Console

What the user does — start the VMM, configure a kernel and rootfs, start the instance, and watch a Linux boot log scroll past on the serial console.

rm -f "$API"
# Start the VMM in the foreground IN ITS OWN TERMINAL so you SEE the serial console.
sudo "$FC" --api-sock "$API"

In a second terminal, configure and start:

API=/tmp/firecracker.socket

curl -X PUT --unix-socket "$API" \
  --data '{"kernel_image_path":"./vmlinux","boot_args":"console=ttyS0 reboot=k panic=1"}' \
  http://localhost/boot-source

curl -X PUT --unix-socket "$API" \
  --data '{"drive_id":"rootfs","path_on_host":"./rootfs.ext4","is_root_device":true,"is_read_only":false}' \
  http://localhost/drives/rootfs

curl -X PUT --unix-socket "$API" \
  --data '{"vcpu_count":2,"mem_size_mib":1024}' http://localhost/machine-config

# Inspect instance state before starting (Not started → Running).
curl --unix-socket "$API" http://localhost/   # GET /  → id, state, vmm_version

curl -X PUT --unix-socket "$API" \
  --data '{"action_type":"InstanceStart"}' http://localhost/actions

In the first terminal a kernel log scrolls and you reach a login prompt. You just booted a Linux VM with no BIOS, no bootloader, in well under a second. That is the entire user story most people ever see — and everything in this curriculum is what happens underneath these four calls.

Note: Notice what you did not do: no disk image with a partition table, no GRUB, no ISO. You handed Firecracker a raw vmlinux and a flat ext4, and it loaded the kernel straight into 64-bit mode and pointed /dev/vda at your ext4. That is the "no BIOS" boot from the Hitchhiker's Guide, made real.

Where this lives in the source

The API request → machine-configuration → boot path threads through several modules. Locate them:

cd ~/src/firecracker

# 1. The HTTP API server (lives in the `firecracker` BINARY, not vmm).
find src/firecracker -type d -name api_server
rg -n 'ParsedRequest|boot-source|/actions' src/firecracker/src/api_server/ | head

# 2. Each parsed request becomes a VmmAction enum, dispatched to the VMM thread.
rg -n 'enum VmmAction' src/vmm/src/rpc_interface.rs

# 3. Pre-boot config is accumulated in VmResources; boot args / kernel config too.
rg -n 'struct VmResources' src/vmm/src/resources.rs
rg -n 'boot_source|BootSourceConfig|boot_args' src/vmm/src/vmm_config/

# 4. InstanceStart triggers the builder that loads the kernel and starts vCPUs.
rg -n 'fn build_and_boot_microvm|fn build_microvm_for_boot' src/vmm/src/builder.rs

# 5. The kernel is loaded as an uncompressed ELF via rust-vmm linux-loader.
rg -n 'Elf|load_kernel|linux_loader|kernel_loader' src/vmm/src/

# 6. The serial console is a 16550 UART from rust-vmm vm-superio.
rg -n 'Serial|vm-superio|ttyS0' src/vmm/src/devices/ | head

This whole path is Level 3 (API → VMM action channel) and Level 6 (boot + guest memory). Deep dives: the API server and action channel, the boot sequence.

Scenario B: Add virtio-net + a Host TAP, Then SSH In

What the user does — create a host TAP device, configure a virtio-net interface, boot, give the guest an IP, and SSH into it. This is the moment the microVM stops being a sealed box.

# 1. Create a host TAP device and give the host side an address.
TAP=tap0
GUEST_IP=172.16.0.2
HOST_IP=172.16.0.1
sudo ip tuntap add "$TAP" mode tap 2>/dev/null || true
sudo ip addr add "${HOST_IP}/30" dev "$TAP"
sudo ip link set "$TAP" up
# Optional: enable forwarding/NAT if you want the guest to reach the internet.
sudo sysctl -w net.ipv4.ip_forward=1

Now configure the interface before InstanceStart (network config is pre-boot):

rm -f "$API"; sudo "$FC" --api-sock "$API" &   # background for this scenario
sleep 0.3

curl -X PUT --unix-socket "$API" \
  --data '{"kernel_image_path":"./vmlinux","boot_args":"console=ttyS0 reboot=k panic=1 ip='"$GUEST_IP"'::'"$HOST_IP"':255.255.255.252::eth0:off"}' \
  http://localhost/boot-source

curl -X PUT --unix-socket "$API" \
  --data '{"drive_id":"rootfs","path_on_host":"./rootfs.ext4","is_root_device":true,"is_read_only":false}' \
  http://localhost/drives/rootfs

# The virtio-net interface: bind it to the host TAP, assign a guest MAC.
curl -X PUT --unix-socket "$API" \
  --data '{"iface_id":"net1","guest_mac":"06:00:AC:10:00:02","host_dev_name":"'"$TAP"'"}' \
  http://localhost/network-interfaces/net1

curl -X PUT --unix-socket "$API" --data '{"action_type":"InstanceStart"}' http://localhost/actions

# Once booted (the rootfs must permit it), SSH in over the TAP:
ssh root@"$GUEST_IP"        # may need a key baked into the rootfs; see docs/getting-started.md

Inside the guest, ip addr shows an eth0 (the virtio-net device), and ethtool -i eth0 reports the virtio_net driver. Traffic flows guest eth0 ↔ host tap0. You now have a networked microVM.

Warning: TAP setup needs CAP_NET_ADMIN (hence sudo for ip). In production this is exactly what the jailer confines: it sets up the network namespace and the device nodes so Firecracker itself runs unprivileged. You will meet that in Level 9 and the jailer deep dive.

Where this lives in the source

# 1. The /network-interfaces config type.
rg -n 'NetworkInterfaceConfig|host_dev_name|guest_mac' src/vmm/src/vmm_config/net.rs

# 2. The virtio-net device: RX/TX queues, TAP backend, rate limiting.
find src/vmm/src/devices/virtio/net -maxdepth 1 -type f
rg -n 'struct Net|fn process_rx|fn process_tx|Tap' src/vmm/src/devices/virtio/net/ | head

# 3. The TAP wrapper itself (/dev/net/tun).
rg -n 'struct Tap|/dev/net/tun|open_named' src/vmm/src/devices/virtio/net/ src/vmm/src/ | head

This is Level 7 territory. Deep dives: virtio-net and TAP, rate limiting, and the networking masterclass.

Scenario C: Attach a Second virtio-block Drive (/dev/vdb)

What the user does — attach a second block device. The guest sees /dev/vdb appear next to the rootfs /dev/vda.

# Make a small ext4 image on the host to attach as a second disk.
dd if=/dev/zero of=./data.ext4 bs=1M count=64
mkfs.ext4 -F ./data.ext4

rm -f "$API"; sudo "$FC" --api-sock "$API" &
sleep 0.3
curl -X PUT --unix-socket "$API" \
  --data '{"kernel_image_path":"./vmlinux","boot_args":"console=ttyS0 reboot=k panic=1"}' \
  http://localhost/boot-source
curl -X PUT --unix-socket "$API" \
  --data '{"drive_id":"rootfs","path_on_host":"./rootfs.ext4","is_root_device":true,"is_read_only":false}' \
  http://localhost/drives/rootfs

# The SECOND drive — not the root device. This becomes /dev/vdb in the guest.
curl -X PUT --unix-socket "$API" \
  --data '{"drive_id":"data","path_on_host":"./data.ext4","is_root_device":false,"is_read_only":false}' \
  http://localhost/drives/data

curl -X PUT --unix-socket "$API" --data '{"action_type":"InstanceStart"}' http://localhost/actions

Inside the guest:

ls /dev/vd*          # /dev/vda (root)  /dev/vdb (the new drive)
mount /dev/vdb /mnt  # mount it; it's your ext4 from the host

Each PUT /drives/{id} is one virtio-block device, in order: vda, vdb, vdc, … The block device serves one request queue; each request walks a descriptor chain and the device does a host pread/pwrite against the backing file (via the configured I/O engine — Sync or io_uring).

Tip: PATCH /drives/{id} can swap the backing path or adjust the rate limiter on a running microVM — useful for hot-attaching a new rootfs in snapshot/clone workflows. Try it after boot.

Where this lives in the source

# 1. The /drives config type.
rg -n 'BlockDeviceConfig|drive_id|is_root_device|path_on_host' src/vmm/src/vmm_config/drive.rs

# 2. The virtio-block device: request queue, parse, the host read/write.
find src/vmm/src/devices/virtio/block -type f | head
rg -n 'struct Block|fn process_queue|RequestType|pread|pwrite|read_exact_at|write_all_at' \
   src/vmm/src/devices/virtio/block/ | head

# 3. The I/O engine (Sync vs io_uring/Async).
rg -n 'io_engine|Async|Sync|FileEngine' src/vmm/src/devices/virtio/block/ | head

This is Level 7 and the virtio-block deep dive, with the descriptor mechanics in virtqueues and engine choices in I/O engines.

Scenario D: Snapshot, Then Restore

What the user does — pause a running microVM, snapshot its full state to two files, kill it, then restore a fresh process from those files and resume — picking up exactly where it left off. This is the capability behind fast clones and warm starts.

# Assume a microVM is booted and running on $API (Scenario A).
# 1. Pause it.
curl -X PATCH --unix-socket "$API" --data '{"state":"Paused"}' http://localhost/vm

# 2. Write the two snapshot files: machine/device state + guest RAM.
curl -X PUT --unix-socket "$API" \
  --data '{"snapshot_path":"./snap.file","mem_file_path":"./mem.file","snapshot_type":"Full"}' \
  http://localhost/snapshot/create

# 3. Resume the original (or shut it down — you have the snapshot now).
curl -X PATCH --unix-socket "$API" --data '{"state":"Resumed"}' http://localhost/vm

ls -lh snap.file mem.file    # the microVM state file (small) + the memory file (= guest RAM size)

Now restore into a brand-new Firecracker process:

RAPI=/tmp/fc-restore.socket
rm -f "$RAPI"; sudo "$FC" --api-sock "$RAPI" &
sleep 0.3

curl -X PUT --unix-socket "$RAPI" \
  --data '{"snapshot_path":"./snap.file","mem_backend":{"backend_path":"./mem.file","backend_type":"File"},"resume_vm":true}' \
  http://localhost/snapshot/load

The restored microVM resumes mid-execution — same processes, same in-memory state. There was no boot: the kernel never re-initialized. That is why snapshot-restore is the basis for sub-millisecond "clone from a warm template" in production.

Note: Two snapshot flavors. Full (GA) captures everything. Diff (dev-preview) needs track_dirty_pages enabled and stores only pages changed since a base — smaller, but you layer it onto a base. The field was renamed enable_diff_snapshots → track_dirty_pages; standalone mem_file_path on load is deprecated in favor of mem_backend (verify on your branch).

Tip: For lazy, on-demand memory loading at restore, use "backend_type":"Uffd" and run a userfaultfd page-fault handler that serves guest pages as the VM touches them. This is how you restore enormous microVMs without reading all their RAM up front — covered in the UFFD masterclass lab.

Where this lives in the source

# 1. The snapshot create/load actions and config types.
rg -n 'CreateSnapshot|LoadSnapshot|SnapshotType|mem_backend|backend_type' src/vmm/src/vmm_config/snapshot.rs

# 2. The persistence machinery: the Persist trait every device implements.
rg -n 'trait Persist|fn save|fn restore' src/vmm/src/persist.rs
find src/vmm/src/snapshot -type f 2>/dev/null
rg -n 'Snapshot|Versionize|serialize' src/vmm/src/snapshot/ src/vmm/src/persist.rs | head

# 3. Restore goes through a dedicated builder path.
rg -n 'fn build_microvm_from_snapshot' src/vmm/src/builder.rs

# 4. UFFD / userfaultfd memory backing.
rg -n 'userfaultfd|Uffd|uffd' src/vmm/src/ | head

This is Level 9 and the snapshotting deep dive, with scale considerations in snapshotting at scale and the full snapshotting masterclass.

Scenario E: Configure MMDS and Read Metadata From Inside the Guest

What the user does — populate the microVM Metadata Service (MMDS) with a JSON document, then read it from inside the guest over a link-local HTTP endpoint — the IMDS pattern, but served by Firecracker itself over a tiny in-VMM TCP/IP stack.

rm -f "$API"; sudo "$FC" --api-sock "$API" &
sleep 0.3

# 1. Tell MMDS which network interface and version to use (pre-boot).
curl -X PUT --unix-socket "$API" \
  --data '{"version":"V2","network_interfaces":["net1"],"ipv4_address":"169.254.169.254"}' \
  http://localhost/mmds/config

# 2. Put a metadata document into MMDS.
curl -X PUT --unix-socket "$API" \
  --data '{"latest":{"meta-data":{"instance-id":"i-warmup-001","local-hostname":"fc-guest"}}}' \
  http://localhost/mmds

# 3. Boot with a network interface (MMDS rides on the guest's network).
curl -X PUT --unix-socket "$API" \
  --data '{"kernel_image_path":"./vmlinux","boot_args":"console=ttyS0 reboot=k panic=1 ip=172.16.0.2::172.16.0.1:255.255.255.252::eth0:off"}' \
  http://localhost/boot-source
curl -X PUT --unix-socket "$API" \
  --data '{"drive_id":"rootfs","path_on_host":"./rootfs.ext4","is_root_device":true,"is_read_only":false}' \
  http://localhost/drives/rootfs
curl -X PUT --unix-socket "$API" \
  --data '{"iface_id":"net1","guest_mac":"06:00:AC:10:00:02","host_dev_name":"tap0"}' \
  http://localhost/network-interfaces/net1
curl -X PUT --unix-socket "$API" --data '{"action_type":"InstanceStart"}' http://localhost/actions

Inside the guest (after adding a route to 169.254.169.254 via eth0), with MMDS V2 you first fetch a session token, then read metadata — exactly like IMDSv2:

# (run inside the guest)
ip route add 169.254.169.254 dev eth0
TOKEN=$(curl -s -X PUT "http://169.254.169.254/latest/api/token" \
  -H "X-metadata-token-ttl-seconds: 21600")
curl -s -H "X-metadata-token: $TOKEN" "http://169.254.169.254/latest/meta-data/instance-id"
#   i-warmup-001

The guest never knows it is talking to its own VMM. The "server" at 169.254.169.254 is dumbo, Firecracker's purpose-built minimal TCP/IP stack, answering on the VMM thread.

Warning: MMDS V2 is token/session-based (the IMDSv2 model) and is what you should use; V1 is deprecated. In production you also drop guest egress to 169.254.169.254 from real networks so the metadata channel cannot be reached from outside (see docs/prod-host-setup.md).

Where this lives in the source

# 1. MMDS config + the data store.
rg -n 'MmdsConfig|version|network_interfaces' src/vmm/src/vmm_config/mmds.rs
find src/vmm/src/mmds -type f
rg -n 'struct Mmds|put|patch|MmdsVersion|token' src/vmm/src/mmds/ | head

# 2. dumbo — the tiny in-VMM TCP/IP stack that serves MMDS over the guest network.
find src/vmm/src/dumbo -type f | head
rg -n 'tcp|Ipv4|Ethernet|handle_frame' src/vmm/src/dumbo/ | head

Deep dive: the MMDS metadata service, plus the networking masterclass MMDS lab.

The Bridge: User Scenario → Source (Master Table)

Use this whenever you observe a behavior and need the code that owns it. Paths drift between branches — when in doubt, run the rg. Always locate, never memorize line numbers.

Each row has a deep dive — see the Deep Dives index — and the underlying shared crates are covered in the rust-vmm section.

Mapping the Warm-Up to the Curriculum

Every scenario above is a doorway into a level (and its deep dives). This is your map of where each thing goes deep:

What to Verify Before Starting Level 1

This takes 30–45 minutes and proves both your environment and your understanding.

cd ~/src/firecracker
FC=$(find build/cargo_target -type f -name firecracker | grep release | head -1)
API=/tmp/firecracker.socket

# Build + binary exist
[ -n "$FC" ] && "$FC" --version

# Scenario A round-trip: boot, check state, stop.
rm -f "$API"; sudo "$FC" --api-sock "$API" & sleep 0.3
curl -s -X PUT --unix-socket "$API" --data '{"kernel_image_path":"./vmlinux","boot_args":"console=ttyS0 reboot=k panic=1"}' http://localhost/boot-source
curl -s -X PUT --unix-socket "$API" --data '{"drive_id":"rootfs","path_on_host":"./rootfs.ext4","is_root_device":true,"is_read_only":false}' http://localhost/drives/rootfs
curl -s --unix-socket "$API" http://localhost/    # state: Not started
curl -s -X PUT --unix-socket "$API" --data '{"action_type":"InstanceStart"}' http://localhost/actions
curl -s --unix-socket "$API" http://localhost/    # state: Running

# Locate the five subsystems you exercised.
rg -n 'enum VmmAction' src/vmm/src/rpc_interface.rs
find src/vmm/src/devices/virtio/net src/vmm/src/devices/virtio/block -maxdepth 1 -type d
rg -n 'trait Persist' src/vmm/src/persist.rs
find src/vmm/src/mmds src/vmm/src/dumbo -maxdepth 0 -type d

You are ready when, without notes, you can:

• Explain the boundary: Firecracker (VMM) vs KVM (kernel module) vs the guest vs an orchestrator above the socket.
• Describe the user control surface: start the process, PUT pre-boot config, InstanceStart, and that one process = one microVM.
• Name the three thread classes (API, VMM, one-per-vCPU) and what each does.
• Say where each warm-up scenario lives in the source: net → devices/virtio/net + vmm_config, block → devices/virtio/block, snapshot → persist.rs/snapshot/, MMDS → mmds/dumbo, boot → builder.rs + linux-loader.
• Run a boot from memory and read instance state with GET /.
• Locate any of those subsystems with an rg/find rather than a remembered path.

If any box is unchecked, re-run the scenario it maps to before moving on. A user who cannot boot, network, and snapshot a microVM has no business reading the run loop yet.

Where to Go Next

Continue to the 16-Week Plan and Milestones to see how this maps onto a schedule and the competence gates. Then begin Level 1: Virtualization and Firecracker Foundation. The internals you previewed here are covered in full in the Deep Dives and the rust-vmm section.