Cross-Repo & Integration Labs

Firecracker is not one repository, and it is almost never run by a human typing curl at a Unix socket. In production it is the bottom of a stack: an orchestrator written in Go starts it through the jailer, drives its REST API with an SDK, feeds it a guest kernel and a rootfs someone else built, and runs it on a host someone else hardened. The firecracker-microvm/firecracker Rust workspace you have spent the rest of this curriculum inside is one layer. The bugs that reach maintainers — and a large fraction of the issues you will triage on the way to maintainership — live at the boundaries between that layer and everything around it: the guest kernel, KVM, the host config, and the orchestrator.

This section builds the single most valuable skill a serious Firecracker contributor can have: given a microVM that misbehaves, attribute the failure to the right component — Firecracker vs. the guest kernel vs. KVM vs. the host vs. the orchestrator — and produce a minimal reproduction that a maintainer can actually run. A "Firecracker bug" that is really a guest-kernel config problem, or a "boot hang" that is really a missing /dev/kvm permission in the jailer chroot, wastes the maintainer's time and yours. These six labs make the attribution mechanical, and they make you fluent in the Go ecosystem (firecracker-containerd, firecracker-go-sdk, firectl) that almost everyone actually uses to drive Firecracker.

This curriculum will not hold your hand here either. It points you at the right repos, gives you the right boundary questions, and makes you build and run real wrappers, real reproductions, and real diagnostics. Where it names a binary, a flag, or a runtime string, you will confirm it against a live checkout — because these repos move on their own release cadences and a contributor who trusts a memorized version string is already wrong.

Note: This section assumes you have done Level 1 (you can boot a microVM by hand and you have a working /dev/kvm), read the jailer deep dive and the threading model, and understand the REST API and action channel. The bug-attribution labs (I4–I6) also lean on the debugging skills from Level 8.

The Firecracker ecosystem map

                      ┌──────────────────────────────────────────────────────────┐
   USERS (operators)  │ AWS Lambda · AWS Fargate · Fly.io · Vercel Sandbox ·      │
                      │ Northflank · Koyeb · Qovery · Kata Containers · flintlock  │
                      └──────────────────────────────────────────────────────────┘
                                        │ each runs its own orchestrator
                                        ▼
   ORCHESTRATION    ┌───────────────────────────────────────────────────────────────┐
   (mostly Go)      │ firecracker-containerd   firecracker-go-sdk   firectl  Kata    │
                    │  (containerd runtime)      (Go library)       (Go CLI) runtime │
                    └───────────────────────────────────────────────────────────────┘
                                        │ HTTP/JSON over a Unix socket (the REST API)
                                        │ + AF_VSOCK control channel (containerd path)
                                        ▼
   ISOLATION        ┌───────────────────────────────────────────────────────────────┐
   BARRIER          │ jailer  ── chroot + cgroups v2 + namespaces + priv-drop ──►    │
   (Rust, in repo)  │ firecracker (the VMM): API thread · VMM thread · vCPU threads  │
                    │   seccomp-BPF on every thread                                  │
                    └───────────────────────────────────────────────────────────────┘
                                        │ uses rust-vmm crates (kvm-ioctls, vm-memory, …)
                                        │ ioctl() on /dev/kvm
                                        ▼
   KERNEL           ┌───────────────────────────────────────────────────────────────┐
                    │ KVM (host Linux kernel module)  ── runs the guest on VT-x/SVM  │
                    └───────────────────────────────────────────────────────────────┘
                                        ▼
   GUEST            ┌───────────────────────────────────────────────────────────────┐
                    │ guest kernel (vmlinux) + rootfs + in-VM agent / your workload  │
                    └───────────────────────────────────────────────────────────────┘

Read the map top to bottom: users pick an orchestrator; the orchestrator drives firecracker through the jailer; firecracker uses rust-vmm crates and KVM; KVM runs the guest. Every one of those horizontal lines is a boundary where bugs hide and where attribution decisions get made. The repos that own each layer:

Warning: Firecracker does not own the guest kernel, the host config, or the orchestrator. A reflex of new contributors is to file every microVM misbehavior on the firecracker repo. Most of the boundaries above are owned by someone else, and the attribution labs (I4–I6) exist precisely to stop that reflex. The threat model is the mirror image: Firecracker does not trust the guest kernel, but it is also not responsible for it.

Why cross-boundary skill matters

A single-repo contributor can read src/vmm/ end to end and still be useless on the issues operators actually file, because those issues span boundaries:

• A microVM hangs at boot. Is it Firecracker failing to load the kernel, the guest kernel panicking because the rootfs is wrong, KVM rejecting a CPUID the guest needs, or the jailer chroot missing /dev/net/tun? The serial console, dmesg, the FC log, and a host strace each answer a different one of those.
• A container won't start under firecracker-containerd but a hand-launched microVM with the same kernel boots fine. Is the bug in Firecracker, in the runtime shim, in the vsock control channel, in the in-VM agent, or in the rootfs the snapshotter built?
• A microVM that restores from a snapshot fails only on a different host. Is that a Firecracker snapshot-compat bug, or a CPU-feature/KVM difference between the two hosts that a CPU template should have normalized?

None of these can be answered from one repo. The contributor who can say "this is a guest-kernel config bug, here is the 15-line repro with a stock kernel that boots and your kernel that doesn't, and here is why it is not Firecracker" is worth ten who can only say "the VM is broken."

The six labs

The control surface in one screen

There are two distinct ways the orchestration layer drives Firecracker, and you must keep them separate when attributing a bug.

The REST API (the universal path). Everything configures Firecracker by sending JSON over a Unix domain socket: PUT /boot-source, PUT /drives/{id}, PUT /machine-config, PUT /actions {InstanceStart}, and so on (the full surface is in the API endpoint map). The Go SDK (I3) and firectl are typed wrappers over exactly these calls; firecracker-containerd's shim (I1) builds the same requests in Go. If you can reproduce a configuration bug with curl, the SDK and the shim are off the hook — the bug is in Firecracker or below.

The vsock control channel (the containerd path). firecracker-containerd additionally runs an agent inside the guest and talks to it over AF_VSOCK (a host Unix socket multiplexed to a guest CID) to launch and manage the actual container with runc. That channel is not the REST API and not something a hand-launched microVM uses. When a container fails but a bare microVM with the same kernel boots, the fault has almost certainly moved into the shim ↔ agent ↔ runc path, which lives in the firecracker-containerd repo, not in firecracker. Keeping "REST control plane" and "vsock agent plane" mentally separate is half of containerd-path attribution.

flowchart LR
  ORCH[Orchestrator code] -->|REST/JSON over UDS| FC[firecracker VMM]
  ORCH -.->|AF_VSOCK, containerd only| AGENT[in-VM agent → runc]
  FC -->|ioctl| KVM[KVM]
  KVM --> GUEST[guest kernel]
  AGENT --> GUEST

How to use these labs

Do them in order. I1–I3 build fluency in the ecosystem (you will run a container in a microVM, write a production jailer wrapper, and launch a microVM from Go). I4 is the keystone — the layered diagnostic method and the attribution decision tree — and it depends on the boundary intuition I1–I3 gave you. I5 and I6 are the contributor-facing payoff: turning an attributed bug into a minimal cross-component repro (I5), and into a diagnostics patch that makes the next person's attribution trivial (I6).

If you arrived here from Level 8, the debugging/profiling masterclass, or the capstone, I4 and I5 are the most directly relevant.

Validation for the section

You have absorbed this section when, given a freshly-failing microVM in a realistic deployment (an orchestrator, a jailer, a guest kernel, a host you do not fully control), you can:

1. Within 10 minutes, name which component most likely owns the failure (Firecracker / guest kernel / KVM / host / orchestrator) and say why.
2. Within 30 minutes, capture the right evidence for that boundary — serial console, guest dmesg, the Firecracker log + metrics, a host strace, or a KVM trace — and not waste time on the wrong instrument.
3. Within an hour, reproduce the symptom with the suspect component removed or replaced (a stock kernel, no jailer, raw curl instead of the orchestrator).
4. Within a day, produce a minimal repro that pins the exact Firecracker, guest-kernel, rootfs, and host versions, and that a maintainer can run unchanged.
5. File the issue in the right repo, or — better — open a diagnostics PR that makes the boundary explain itself next time.

That routine, executed crisply, is what gets cross-component Firecracker issues resolved. The labs build the muscle.

Begin with Lab I1: firecracker-containerd.