The Hitchhiker's Guide to Virtualization, KVM & microVMs

Read this before Level 1. It is a teaching chapter, not a reference — its job is to build the mental model that makes every later page click. By the end you should be able to explain, from first principles, what a virtual machine is, why hardware virtualization exists, what /dev/kvm actually gives you, the three jobs every VMM must do, how a guest kernel boots with no BIOS, and what a virtqueue is. None of this requires you to have read Firecracker's source yet. When you do open the source in Level 1, you will recognize the shapes.

Note: This chapter deliberately stays at the concept level. It names KVM ioctls, virtio registers, and memory-layout ideas you will meet again — but it does not point at Firecracker source. The deep dives (KVM fundamentals, the vCPU run loop, guest memory, the boot sequence, virtio transport, virtqueues) do that. Here, just build the picture.

What "Virtualization" Means, and Why

A virtual machine (VM) is a software-constructed illusion of a whole computer: a CPU, RAM, and a handful of devices, convincing enough that an unmodified operating system boots inside it and behaves as if it owns real hardware. The program that constructs and runs that illusion is the Virtual Machine Monitor (VMM), also called a hypervisor's userspace, or simply "the monitor." Firecracker is a VMM.

Why build this illusion at all? Three reasons drive nearly all virtualization:

The hard part is doing this fast and safely. Interpreting every guest instruction in software is correct but a hundred times too slow. The breakthrough that makes modern virtualization practical is letting guest code run directly on the physical CPU at near-native speed, and only "trapping" out to the VMM for the rare operations that must be controlled.

Trap-and-Emulate: The Central Idea

The whole field rests on one mechanism: trap-and-emulate.

Most guest instructions — arithmetic, branches, memory loads and stores within the guest's own RAM — are harmless. They can run natively on the real CPU at full speed. But a few instructions are sensitive: they touch global machine state (talk to a device, change CPU mode, read a hardware timer, access an I/O port). If a guest ran those directly, it could see or break the host. So the CPU is configured to trap on exactly those instructions: it stops the guest, hands control to the VMM, and the VMM emulates the intended effect (returns a fake device register, advances a virtual clock, services the I/O), then resumes the guest.

        guest runs natively on the real CPU
        ────────────────────────────────────────────────►  time
        add, mov, jmp, loop, ...        (full speed, no VMM)
                                   │
              "out 0x3f8, al"  (write to a serial port)  ← SENSITIVE
                                   │  TRAP  →  control transfers to the VMM
                                   ▼
        ┌─────────────────────────────────────────┐
        │ VMM: "the guest wrote a byte to the      │   emulate the device,
        │ serial port — append it to the console"  │   then resume the guest
        └─────────────────────────────────────────┘
                                   │  resume
        guest continues natively ──┘────────────────────►

Two things make this work in practice: hardware support (so the trap is cheap and the CPU can switch between "guest mode" and "host mode" safely) and a VMM that knows how to emulate exactly the devices the guest expects. The fewer sensitive things the guest can do, the fewer traps, the faster it runs — and the smaller the VMM's emulation code, the smaller the attack surface. Hold that last sentence; it is Firecracker's entire design thesis.

Type-1 vs Type-2 Hypervisors, and Where KVM/Firecracker Sit

Hypervisors are traditionally split into two types:

KVM is the interesting case that breaks the binary. KVM (Kernel-based Virtual Machine) is a Linux kernel module that turns the Linux kernel itself into a Type-1-class hypervisor: it programs the CPU's virtualization extensions and runs guest code directly on the hardware. But it does this from inside a full Linux host, so the arrangement also looks Type-2. The practical model is:

┌──────────────────────────────────────────────────────────────────┐
│  Userspace VMM  (Firecracker, QEMU, Cloud Hypervisor, crosvm)     │  ← decides what the VM IS:
│   • creates the VM, vCPUs, guest memory via ioctl()               │     memory, devices, boot
│   • emulates devices, loads the kernel, runs the vCPU loop        │
└──────────────────────────────────────────────────────────────────┘
                    │  ioctl() on /dev/kvm
                    ▼
┌──────────────────────────────────────────────────────────────────┐
│  KVM  (Linux kernel module)                                       │  ← does the privileged work:
│   • programs VT-x / AMD-V / ARM EL2                               │     runs guest on real CPU,
│   • runs guest code, catches VM exits, returns them to userspace  │     handles the CPU plumbing
└──────────────────────────────────────────────────────────────────┘
                    │
                    ▼   physical CPU virtualization extensions
┌──────────────────────────────────────────────────────────────────┐
│  Hardware: Intel VT-x / AMD-V / ARM virtualization               │
└──────────────────────────────────────────────────────────────────┘

The division of labour is the thing to internalize: KVM runs the guest; the VMM decides what the guest is. KVM does not know what a network card or a disk looks like — it knows how to run guest instructions and how to stop and hand control back when the guest does something that needs userspace. Everything about what devices exist, what kernel boots, how memory is laid out is the VMM's job. Firecracker is a small, opinionated VMM; QEMU is an enormous, general one. Both drive KVM through the same /dev/kvm interface.

What the Hardware Actually Provides

Trap-and-emulate needs the CPU's cooperation. The virtualization extensions add a new, more privileged execution context for the hypervisor and a controlled way to enter/leave "guest mode."

Intel VT-x / VMX

VT-x adds two operating modes: VMX root (the hypervisor/KVM) and VMX non-root (the guest). Two new transitions move between them:

• VM entry — hypervisor → guest. KVM executes the guest.
• VM exit — guest → hypervisor. The CPU traps back to KVM, recording why.

State is mediated by the VMCS (Virtual Machine Control Structure), a per-vCPU region that holds guest register state, host state to restore on exit, and a big set of execution controls that decide which guest actions cause a VM exit (which I/O ports trap, whether cpuid exits, whether interrupts are intercepted, etc.). For memory, EPT (Extended Page Tables) adds a second layer of address translation: the guest's own page tables map guest-virtual → guest-physical, and EPT maps guest-physical → host-physical. The guest can never address host memory it was not given, and it does so without the hypervisor trapping every memory access — the MMU does it in hardware.

AMD-V / SVM

AMD's equivalent. The control structure is the VMCB (Virtual Machine Control Block); the second-level paging is NPT (Nested Page Tables), the analog of EPT. vmrun enters the guest. Conceptually identical to VT-x; the field names and instruction mnemonics differ.

ARM (aarch64)

ARM provides exception levels: EL0 (user), EL1 (kernel), EL2 (hypervisor), EL3 (firmware). The hypervisor runs at EL2, the guest kernel at EL1. Stage-2 translation is ARM's second-level paging (guest-physical → host-physical), the EPT/NPT analog; interrupt virtualization uses the GIC (Generic Interrupt Controller). The shapes match x86; the vocabulary differs.

Note: You will rarely touch these directly — KVM abstracts them behind a uniform interface. But knowing that "a VM exit" and "EPT" exist explains why the KVM API looks the way it does, and why the same Firecracker Vcpu::run works on both Intel and AMD. The KVM fundamentals deep dive goes deeper.

What /dev/kvm Gives You: The ioctl API

KVM exposes everything through one character device, /dev/kvm, driven by ioctl(). The API is a hierarchy of three file-descriptor levels, each obtained from the one above:

open("/dev/kvm")            ── the SYSTEM fd  (capabilities, KVM_GET_API_VERSION)
        │  ioctl(KVM_CREATE_VM)
        ▼
   the VM fd                ── one virtual machine  (memory regions, IRQ chip, devices)
        │  ioctl(KVM_CREATE_VCPU)
        ▼
   a vCPU fd  (× N)         ── one virtual CPU      (registers, KVM_RUN)

The ioctls you must recognize (you will see Firecracker issue every one of these through the rust-vmm kvm-ioctls crate):

KVM_RUN is the heartbeat. It and the struct kvm_run shared page are the subject of the next two sections.

The Three Jobs of a VMM

Strip away every feature and a VMM does exactly three things. Everything in Firecracker's source is one of these three, or plumbing around them.

                ┌──────────────────────────────────────────────────────┐
                │                     The VMM                          │
                ├──────────────────┬──────────────────┬────────────────┤
                │  1. vCPUs        │  2. Guest memory │  3. Devices     │
                │  create + run    │  allocate + map  │  emulate the    │
                │  the KVM_RUN     │  host RAM as     │  handful the    │
                │  loop, one       │  guest-physical  │  guest needs    │
                │  thread each     │  RAM via KVM     │  (net/block/... │
                │                  │                  │   serial, etc.) │
                └──────────────────┴──────────────────┴────────────────┘

1. vCPUs. For each virtual CPU, create a vCPU fd, set its initial register state, and run its KVM_RUN loop in a dedicated thread. When the guest exits for I/O, the same thread services it and re-enters. Firecracker runs one thread per vCPU.

2. Guest memory. Allocate a big host buffer (typically mmap of anonymous memory) and register it with KVM via KVM_SET_USER_MEMORY_REGION so the guest sees it as physical RAM at some guest-physical address. The VMM reads and writes guest memory through this host mapping — that is how it loads the kernel, sets up boot parameters, and how devices DMA data in and out.

3. Device emulation. Provide the devices the guest's drivers expect. A real VMM emulates only what is necessary. Firecracker emulates a serial console (a 16550 UART), a partial keyboard controller (just enough to catch a reboot), and a small set of virtio devices (block, net, vsock, rng, balloon). When the guest touches a device register, the CPU traps, KVM returns the exit, and the VMM's device model handles it.

A fourth, cross-cutting job — control and lifecycle (configure the machine, start it, pause, snapshot, shut down) — is what Firecracker's REST API and threading model are about. But the three above are the irreducible core.

Guest-Physical → Host-Virtual: The Memory Mapping

This is the single most confusing point for newcomers, so make it concrete. There are three address spaces in play, and you must keep them straight:

   Guest sees:                         Host (the VMM process) sees:
   ┌────────────────────────┐
   │ Guest VIRTUAL address  │  guest page tables (guest's own)
   └───────────┬────────────┘
               ▼
   ┌────────────────────────┐          ┌────────────────────────────────┐
   │ Guest PHYSICAL address │  ◄══════► │  Host VIRTUAL address          │
   │  (what the guest calls │   the     │  (a pointer INTO the mmap'd    │
   │   "RAM at 0x100000")   │  mapping  │   buffer the VMM allocated)    │
   └───────────┬────────────┘          └───────────────┬────────────────┘
               ▼ EPT / NPT / stage-2                    ▼ host page tables
   ┌────────────────────────┐          ┌────────────────────────────────┐
   │ Host PHYSICAL address  │  ◄──────  │  Host PHYSICAL address (RAM)   │
   └────────────────────────┘  (same physical DRAM)

When the VMM calls KVM_SET_USER_MEMORY_REGION, it tells KVM: "guest-physical address guest_phys_addr for memory_size bytes corresponds to this pointer in my process, userspace_addr." From then on:

• The guest, running natively, uses its page tables to turn guest-virtual into guest-physical, and the hardware's second-level paging (EPT/NPT/stage-2) turns guest-physical into host-physical — transparently, at full speed.
• The VMM, to read or write the guest's RAM, simply does pointer arithmetic into its own mmap'd buffer: host_virtual = userspace_addr + (guest_phys - guest_phys_addr). That is how it loads the kernel into "guest physical 0x100000," how it writes boot parameters at "guest physical 0x7000," and how a device reads a buffer the guest handed it.

So when you later read that Firecracker copies kernel segments to HIMEM_START (1 MiB) or writes the "zero page" at 0x7000, those are guest-physical addresses, and the VMM reaches them by indexing its host buffer. rust-vmm's vm-memory crate (GuestMemoryMmap, GuestAddress) wraps exactly this.

The vCPU Run Loop and VM Exits (in concept)

Here is the loop that is a running VM, stripped to its essence. Each vCPU thread does this forever until shutdown:

   ┌─────────────────────────────────────────────────────────────┐
   │  loop {                                                      │
   │      ioctl(vcpu_fd, KVM_RUN)        ← runs guest natively;   │
   │                                       BLOCKS until a VM exit │
   │      switch (kvm_run->exit_reason) {                         │
   │          KVM_EXIT_IO:    handle a PIO read/write  ──┐        │
   │          KVM_EXIT_MMIO:  handle an MMIO read/write  │ emulate│
   │          KVM_EXIT_HLT:   guest halted              │ then   │
   │          KVM_EXIT_SHUTDOWN / FAIL_ENTRY: stop      │ loop   │
   │          ...                                       ┘        │
   │      }                                                       │
   │  }                                                           │
   └─────────────────────────────────────────────────────────────┘

KVM and the vCPU thread share one page of memory — struct kvm_run — that you mmap once (its size comes from KVM_GET_VCPU_MMAP_SIZE). After every KVM_RUN returns, you read exit_reason from that page and dispatch:

sequenceDiagram
    participant T as vCPU thread (VMM)
    participant K as KVM (kernel)
    participant G as Guest code (on real CPU)
    T->>K: ioctl(KVM_RUN)
    K->>G: VM entry (run natively)
    G-->>K: guest writes a device register → VM exit
    K-->>T: KVM_RUN returns; kvm_run.exit_reason = KVM_EXIT_MMIO
    Note over T: emulate the device,<br/>write result into kvm_run page
    T->>K: ioctl(KVM_RUN) again
    K->>G: VM entry (resume)

The exit reasons you will meet constantly:

The crucial optimization: not every device interaction takes an exit. For the virtio fast path, the guest "kicking" a queue is a write that KVM routes to an eventfd via KVM_IOEVENTFD — the vCPU thread doesn't even have to stop; the VMM thread wakes up on its event loop and processes the queue. That split — vCPU threads in KVM_RUN, a separate VMM thread on an epoll loop handling device work — is exactly Firecracker's threading model, which you meet in Level 3.

How a Kernel Boots, and the "No BIOS" Problem

On a real PC, power-on doesn't jump straight into Linux. The BIOS/UEFI firmware runs first: initializes hardware, builds tables describing the machine (memory map, ACPI), finds a bootloader (GRUB), which loads and decompresses the kernel and hands it a structure describing the environment. The CPU starts in ancient 16-bit real mode and the bootloader walks it up to 64-bit long mode.

A microVM has no BIOS, no bootloader, no firmware. That is a feature — firmware is slow, large, and a security surface. But it means the VMM must do the bootloader's and the firmware's job itself, in code, before the very first guest instruction runs:

   Real PC boot                          microVM boot (the VMM does it all, in memory)
   ───────────                           ─────────────────────────────────────────────
   power on → 16-bit real mode           VMM loads an UNCOMPRESSED vmlinux ELF directly
   BIOS/UEFI: init HW, build tables       into guest-physical memory (parse PT_LOAD, copy
   firmware finds bootloader              segments to ~1 MiB)
   GRUB loads + decompresses kernel       VMM builds the "zero page"/boot_params: e820 memory
   GRUB builds boot info, jumps           map, cmdline pointer, initrd pointer
   kernel walks 16-bit → 64-bit           VMM sets initial vCPU regs in 64-bit LONG MODE
   kernel runs                            directly (rip = ELF entry, rsi → zero page), then
                                          KVM_RUN — kernel runs from its first instruction

So on x86_64 the VMM must, before KVM_RUN:

1. Load the kernel. Firecracker loads an uncompressed vmlinux ELF (via rust-vmm's linux-loader): parse the PT_LOAD segments, copy them into guest memory, note the entry point e_entry.
2. Build boot parameters. The Linux x86 boot protocol expects a boot_params structure (the "zero page") containing an e820 memory map (which guest-physical ranges are RAM vs. reserved), a pointer to the kernel command line, and an optional initrd pointer. Firecracker writes this into guest memory at a known address (ZERO_PAGE_START, around 0x7000 — verify on your branch).
3. Set initial CPU state. Put the vCPU directly into 64-bit long mode (paging on, identity-map the low memory), set rip to the kernel entry and rsi to point at the zero page, then KVM_RUN.

On aarch64 there is no zero page. Instead the VMM builds a Flattened Device Tree (FDT/DTB) — a binary tree describing CPUs, memory, and devices — and passes its address in register x0; the kernel is an arm64 Image (PE). Same idea, different mechanism. CPU topology and interrupt routing are described to the guest via ACPI tables (RSDP/MADT) on x86 and the FDT/GIC on aarch64.

Note: This is why Firecracker boots so fast: it skips firmware entirely and jumps a stripped kernel straight into 64-bit mode. The flip side is that the VMM owns a boot protocol contract — get the e820 map or the entry point wrong and the guest triple-faults before it prints a thing. Level 6 and the boot sequence deep dive live here.

Virtio in One Page

A VMM could emulate a real network card or SATA controller, but that is slow (every register poke is a trap) and large (lots of code = lots of attack surface). Virtio is the standard answer: paravirtualized devices. Instead of pretending to be real hardware, the device and a virtio-aware guest driver agree on a simple, shared-memory protocol. The guest knows it is virtualized and cooperates, which is dramatically faster and simpler.

The core data structure is the virtqueue — a ring of buffers in guest memory that both sides read and write:

                       GUEST MEMORY (shared, both sides read/write)
   ┌─────────────────────────────────────────────────────────────────────┐
   │  Descriptor table   virtq_desc[]  { addr, len, flags, next }         │  buffers + chaining
   │  Available ring     driver → device:  "buffers 3,7,2 are ready"      │  guest fills, device reads
   │  Used ring          device → driver:  "buffer 3 done, wrote 512 B"   │  device fills, guest reads
   └─────────────────────────────────────────────────────────────────────┘
        ▲ guest driver puts a request here          ▲ device puts the result here
        │                                            │
        │  "KICK": guest writes QueueNotify  ───────►│  device processes the queue
        │                                            │
        │◄─────── INTERRUPT: device updates used ring, raises IRQ

The choreography for one I/O (say, "read a disk block"):

1. The guest driver places a buffer (or a chain of buffers via the next field) in the descriptor table and adds its index to the available ring.
2. The driver kicks the device — writes the QueueNotify register. (Via KVM_IOEVENTFD this becomes an eventfd wake, not a full vCPU exit.)
3. The device (in the VMM) reads the available ring, walks the descriptor chain, does the work — reads the disk file into the guest's buffer — and records completion in the used ring.
4. The device raises an interrupt (via KVM_IRQFD); the guest driver wakes, reads the used ring, sees its buffer is done.

Before any of that, the two sides perform feature negotiation and a status handshake (ACKNOWLEDGE → DRIVER → FEATURES_OK → DRIVER_OK) so they agree on capabilities.

The transport is how the guest finds the device and its registers. Firecracker's default is virtio-MMIO: each device occupies a fixed block of memory-mapped registers plus one IRQ, and the guest is told the address on the kernel command line (virtio_mmio.device=SIZE@ADDR:IRQ on x86) or via an FDT node (aarch64). There is no PCI enumeration by default — that legacy machinery is exactly the kind of surface Firecracker omits. (A virtio-PCI transport now exists behind --enable-pci; verify on your branch.) The key MMIO registers — MagicValue (0x74726976, "virt"), DeviceID, QueueSel, QueueNotify, InterruptStatus, Status — are how the guest driver and the VMM's device model talk.

Level 7 and the virtqueues / virtio-block deep dives take this all the way down to the descriptor flags.

Where Firecracker's microVM Fits — and What It Deliberately Omits

Now place Firecracker in the picture. It is a VMM that does the three jobs above and almost nothing else, on purpose. Compared to a general-purpose VMM like QEMU:

What Firecracker deliberately omits — and why — is the heart of its philosophy:

• No BIOS/firmware, no PCI legacy, no USB, no GPU, no sound, no SCSI, no full ACPI device tree. Every one of those is host code a malicious guest could attack. Less code, less surface.
• No general-OS compatibility. It boots Linux microVMs for serverless, not Windows desktops.
• No interactive monitor, no migration of arbitrary device state. Snapshot/restore is purpose-built and minimal, not a general live-migration framework.

This minimalism is a security argument, not aesthetics. The threat model is explicit: the guest, including the guest kernel, is untrusted, and the job is to protect the host. Defense in depth stacks four layers — the KVM/hardware boundary, the jailer (chroot + namespaces + cgroups + privilege drop), seccomp-BPF (whitelisting the ~40-odd syscalls Firecracker may make), and Rust memory safety. A minimal device model makes the second-from-bottom layer — the VMM's own emulation code — as small as it can be. When you later read a GitHub thread where a maintainer says no to a device proposal, this is the reasoning behind it. "QEMU has it" is explicitly not an argument.

Vocabulary You Now Own

Keep this near your desk through Level 1. Every term here recurs constantly.

Where to Go Next

You now have the model. The next page makes it physical: you will run Firecracker as a user and feel every concept above — the boot with no BIOS, the virtio-net TAP, a second virtio-block drive, a snapshot, the MMDS — then bridge each to the source.

Continue to the Firecracker Warm-Up: From User to Contributor. Then skim the 16-Week Plan and Milestones, and begin Level 1: Virtualization and Firecracker Foundation. The hardware and KVM mechanics sketched here are taken all the way down in the KVM fundamentals and vCPU run loop deep dives.