Lab I1: firecracker-containerd — Containers in microVMs

Background

firecracker-containerd is the project that makes Firecracker look like a container runtime. It lets containerd — the same container supervisor that backs Docker and Kubernetes — run ordinary OCI container images, except each container (or group of containers) lives inside its own Firecracker microVM instead of sharing the host kernel through namespaces. This is the shape AWS uses to run customer code with VM-grade isolation but a container-grade developer experience, and it is the canonical "Firecracker as a building block" project.

It is written in Go, and its architecture is a layer cake of processes that talk across boundaries: containerd hands work to a runtime shim, the shim launches a Firecracker microVM, an agent running inside that microVM receives container commands over a vsock channel, and the agent runs the container with runc — the exact same runc a normal containerd setup uses, just inside the guest. Your job in this lab is not to become a firecracker-containerd maintainer; it is to trace a single ctr run from your shell all the way down to the guest and build a mental model precise enough to attribute bugs to the right process when you reach Lab I4.

Note: This lab is the most setup-heavy in the section, and it is honest about that. firecracker-containerd needs a thin-pool device-mapper snapshotter, a purpose-built rootfs containing the agent, and several binaries on your PATH. If your goal today is only attribution intuition, you can read the architecture and the trace and skip the live build — but a contributor who has run the stack reasons about it far better than one who has only read about it.

Why This Lab Matters for Contributors

• Most production Firecracker bugs are reported through an orchestrator like this one. Knowing where containerd ends and Firecracker begins is the difference between a useful bug report and a misfiled one.
• It teaches the two control planes (section index): the REST API that configures the microVM, and the vsock agent plane that runs the container. They fail differently and live in different repos.
• It makes the vsock deep dive concrete: vsock is not an abstract device here, it is the literal wire firecracker-containerd uses for its control RPC.
• It connects to Stage 8 of the roadmap and the minimal-device-model philosophy: firecracker-containerd is the biggest real consumer that constrains what Firecracker's API can change.

Prerequisites

Verify your starting point and pin the versions you are about to depend on — do not trust this lab's numbers, read them off the repo:

# Confirm you can still boot a bare microVM (the REST path must work before the containerd path).
ls -l /dev/kvm

# Clone and read the actual required Go version and component list — do not memorize ours.
git clone https://github.com/firecracker-microvm/firecracker-containerd
cd firecracker-containerd
grep -m1 "^go " go.mod                       # the Go toolchain it wants
rg -n "containerd-shim-aws-firecracker|aws.firecracker|firecracker-ctr" Makefile docs/ | head

The architecture

   host userspace                                          │  guest (one microVM)
   ─────────────────────────────────────────────────────  │  ───────────────────────────
   ┌────────────┐   ttrpc    ┌──────────────────────────┐  │
   │ firecracker│──────────► │ containerd-shim-aws-      │  │
   │ -ctr (CLI) │   (gRPC-   │ firecracker  (the runtime │  │
   └────────────┘    like)   │ shim, one per microVM)    │  │
        │                    └──────────────────────────┘  │
        ▼ gRPC                         │      │             │
   ┌────────────────────────┐          │      │ REST/JSON   │
   │ firecracker-containerd │          │      │ over UDS    │
   │ daemon (+ devmapper    │          │      ▼             │
   │ snapshotter, control   │          │  ┌──────────────┐  │
   │ plugin)                │          │  │ firecracker  │──┼─► KVM ─► guest kernel
   └────────────────────────┘          │  │ (the VMM)    │  │
                                       │  └──────────────┘  │
                                       │      ▲             │
                                       │      │ AF_VSOCK    │   ┌─────────────────────┐
                                       └──────┼─────────────┼──►│ agent (in-VM)       │
                                              │ (control RPC │   │  → containerd-shim- │
                                              │  to the      │   │    runc-v1 → runc   │
                                              │  in-VM agent)│   │  → the container    │
                                              └─────────────┼──►└─────────────────────┘

Five processes, four boundaries. Name each one — you will refer to them by role for the rest of the section. The exact binary names are stable enough to learn, but confirm them on your checkout (rg -n "aws.firecracker|containerd-shim-aws-firecracker|/agent" docs/ runtime/ agent/):

The single most important structural fact: the shim drives Firecracker over the REST API (the same PUT /boot-source, /drives, /actions you do by hand), and separately drives the in-VM agent over vsock. Those are two different channels into the same microVM, and bugs land on one or the other.

Where Firecracker's vsock control channel fits

In a bare microVM, vsock is an optional device you may not configure at all. In firecracker-containerd it is load-bearing infrastructure: it is how the host-side shim sends "create this container, start it, send it this signal, collect its exit code" to the agent living inside the guest, because once the VM is running there is no other in-band control path. Firecracker's job is only to transport AF_VSOCK between a host Unix socket and the guest's vsock CID — the protocol spoken over it (containerd's ttrpc) is entirely firecracker-containerd's concern.

sequenceDiagram
  participant Shim as runtime shim (host)
  participant FC as firecracker (vsock device)
  participant Agent as in-VM agent (guest)
  participant Runc as runc (guest)
  Shim->>FC: connect host Unix socket (vsock backend)
  FC->>Agent: AF_VSOCK stream to guest CID:port
  Shim->>Agent: ttrpc: CreateContainer / Start (over vsock)
  Agent->>Runc: exec runc create / start
  Runc-->>Agent: container running, pid
  Agent-->>Shim: task started (over vsock)

Attribution consequence (you will use this in I4): if the microVM boots but the container never starts, the REST/boot path worked and the fault is on the vsock agent plane — the agent didn't come up, the vsock connection failed, or runc rejected the bundle. That is a firecracker-containerd problem, not a Firecracker one, unless Firecracker's vsock device itself dropped the stream — which you confirm by checking FC's vsock metrics rather than guessing.

Step-by-Step Tasks

Step 1: Read the build, then build the components

Do not blindly run make; read what it produces first, because each artifact maps to a box in the diagram above.

cd firecracker-containerd
# What does the build actually produce? Map each target to a process.
rg -n "^[a-zA-Z0-9_-]+:" Makefile | head -40
# The image-builder produces the rootfs that CONTAINS the agent — this is the part people miss.
ls tools/image-builder/
cat tools/image-builder/Makefile | rg -n "agent|rootfs|squashfs|ext4" | head

Build the binaries and the agent-bearing rootfs (commands and target names vary by release — read them, then run them):

make all                              # builds the daemon, runtime shim, firecracker-ctr, agent
sudo make install                     # places binaries on PATH (verify the prefix)
make image                            # builds the rootfs that embeds the agent (image-builder)

Warning: The rootfs is special. A normal Firecracker rootfs has no agent and will boot fine standalone but do nothing under firecracker-containerd, because there is nothing inside the guest to answer the vsock control RPC. If you ever see "VM boots, container hangs forever," the very first thing to check is whether the rootfs actually contains the agent (unsquashfs -l rootfs.img | rg agent).

Step 2: Configure the devmapper snapshotter and the daemon

Container images need a snapshotter, and firecracker-containerd uses the devmapper thin-pool snapshotter. The config file location and contents are documented in the repo — read docs/getting-started.md and create /etc/firecracker-containerd/config.toml. The load-bearing fields (verify against the current docs):

version = 2
disabled_plugins = ["io.containerd.grpc.v1.cri"]
root  = "/var/lib/firecracker-containerd/containerd"
state = "/run/firecracker-containerd"
[grpc]
  address = "/run/firecracker-containerd/containerd.sock"
[plugins]
  [plugins."io.containerd.snapshotter.v1.devmapper"]
    pool_name = "fc-dev-thinpool"
    base_image_size = "10GB"
    root_path = "/var/lib/firecracker-containerd/snapshotter/devmapper"
[debug]
  level = "debug"

There is a second runtime-level config (the firecracker-runtime config: which kernel, which rootfs, default vCPU/mem, the firecracker binary path). Find its expected path and shape:

rg -n "firecracker-runtime.json|RuntimeConfig|KernelImagePath|RootDrive|FirecrackerBinaryPath" \
  runtime/ docs/ | head

The devmapper thin-pool is itself host setup that frequently breaks the demo. The repo ships a script for it — read it before running, because it creates loopback devices:

rg -n "thinpool|dmsetup|losetup" tools/ docs/ | head

Step 3: Start the daemon and run a container in a microVM

# Start the firecracker-containerd daemon with your config.
sudo PATH=$PATH /usr/local/bin/firecracker-containerd \
  --config /etc/firecracker-containerd/config.toml &

# Run a container. Note the two flags that select the FC path:
#   --snapshotter devmapper   (thin-pool images)
#   --runtime aws.firecracker (route the task to the FC shim, not runc-on-host)
sudo firecracker-ctr --address /run/firecracker-containerd/containerd.sock \
  run \
  --snapshotter devmapper \
  --runtime aws.firecracker \
  --rm --tty --net-host \
  docker.io/library/busybox:latest busybox-test

If it works you get a busybox shell inside a Firecracker microVM. Confirm that — the whole point — from inside the guest:

# Inside the container shell:
dmesg | head                 # a Firecracker microVM kernel boot, not the host kernel
cat /proc/cpuinfo | head     # the vCPU count you configured, not the host's
mount | rg ' / '             # the devmapper-backed rootfs

The two flags are the attribution hinge. --runtime aws.firecracker is the string that tells containerd to dispatch the task to containerd-shim-aws-firecracker instead of the default runc runtime. If you drop it, you get a normal host container and prove nothing.

Step 4: Trace the request from ctr to the guest

Now walk the boundaries with real evidence at each hop. The goal is to see each process do its part, so that when one fails you know which log to read.

Hop 1 — CLI → daemon (gRPC). Bump the daemon log level (already debug above) and watch the task be created:

journalctl -f 2>/dev/null | rg "firecracker-containerd" &   # or tail the daemon's stdout
# In another shell, run the ctr command from Step 3; watch the create/start RPCs appear.

Hop 2 — daemon → shim (ttrpc), shim → firecracker (REST). The shim drives Firecracker over the same REST API you know. Find where it builds those requests:

rg -n "PutGuestBootSource|PutGuestDriveByID|CreateSyncAction|InstanceStart|machine-config" \
  runtime/ | head
# The shim uses the Go SDK (Lab I3) to make these calls — confirm the dependency:
rg -n "firecracker-go-sdk" go.mod runtime/

This is the bridge to Lab I3: the shim is a consumer of the Go SDK, which is a wrapper over the REST API. The same PUT /boot-source you typed by hand in Lab 1.3 is happening here, generated by Go.

Hop 3 — shim → agent (vsock), agent → runc. After boot, control crosses into the guest over vsock. Find the vsock wiring on both sides:

# Host side: how the shim opens the vsock control connection.
rg -n "vsock|VsockDevices|UDSPath|CID" runtime/ | head
# Guest side: the agent receiving RPC and invoking runc.
rg -n "runc|containerd-shim-runc-v1|TaskService|CreateContainer" agent/ | head

Hop 4 — runc → the container. Inside the guest the agent runs runc exactly as a normal containerd node would; nothing Firecracker-specific happens below the agent. That symmetry is the point: the container part is ordinary; the VM part is the shim + firecracker.

Draw the full trace for yourself and label, at each hop, the channel and the repo that owns it. That labeled trace is the deliverable.

Step 5: Deliberately break one boundary

The fastest way to internalize the boundaries is to break one and watch which log shows the failure:

# Point the runtime config at a kernel path that does not exist, then run a container.
# Expected: firecracker fails at the REST boot stage; the shim surfaces an InstanceStart
# error. The container never reaches the agent — proving this is a boot-plane (FC) failure.

# Now restore the kernel but use a rootfs WITHOUT the agent.
# Expected: the microVM boots fine (FC is happy), but the container hangs — proving an
# agent-plane (firecracker-containerd) failure with an identical-looking symptom from ctr.

Two superficially identical "my container won't start" symptoms, two different repos. That is the entire attribution lesson of the section, in miniature.

Implementation Requirements / Deliverables

• firecracker-containerd built from source, with the agent-bearing rootfs.
• One container run inside a microVM via --runtime aws.firecracker, confirmed from inside the guest (dmesg, /proc/cpuinfo).
• A labeled five-process trace of a single ctr run, naming the channel and owning repo at each of the four boundaries.
• The rg output showing the shim builds REST requests via the Go SDK and opens the vsock control connection.
• The two deliberate failures from Step 5, each attributed to the correct plane.

Troubleshooting

ctr hangs and no microVM appears

The shim or the daemon failed before launching firecracker. Check the daemon log and confirm --runtime aws.firecracker is spelled exactly (a typo silently falls back to the default runtime, which then fails differently). Confirm the runtime config's FirecrackerBinaryPath points at a real firecracker binary.

microVM boots but the container never starts

The boot plane worked; the agent plane did not. In order: confirm the rootfs contains the agent (unsquashfs -l), confirm vsock is configured for the microVM, then read the agent's output. This is the most common firecracker-containerd issue and it is not a Firecracker bug.

devmapper / thin-pool errors

Snapshotter setup, not Firecracker. Confirm the thin-pool exists (sudo dmsetup ls), that the loopback files are present, and that root_path is writable. The repo's thin-pool script must have run successfully first.

failed to get kvm / permission denied

The daemon/shim could not open /dev/kvm. Confirm ls -l /dev/kvm and group membership. If you are running jailed (see Lab I2), confirm the jailer mknod'd /dev/kvm into the chroot.

Expected Output

A busybox shell inside a Firecracker microVM, started by containerd, where dmesg shows a fresh guest kernel boot and /proc/cpuinfo shows your configured vCPU count — plus a written, labeled trace from firecracker-ctr down to runc that names each process, channel, and repo.

Stretch Goals

1. Run two containers and confirm whether they share a microVM or each get their own; find where that policy lives in runtime/.
2. Configure firecracker-containerd to launch FC through the jailer and reconcile it with Lab I2 — what does the orchestrator now own?
3. Capture the actual vsock traffic conceptually: add the in-VM agent's debug logging and correlate one ctr exec with the agent RPC it triggers.
4. Read how firecracker-containerd uses snapshots for fast container start and connect it to the snapshotting deep dive and Stage 8.

Validation / Self-check

1. Name the five processes in a running firecracker-containerd container, and the channel between each adjacent pair.
2. Which channel does the shim use to configure and boot the microVM, and which does it use to run the container? Why does it need both?
3. A ctr run produces a booted microVM with no container. Which plane failed, which repo owns it, and what is the first thing you check?
4. What does --runtime aws.firecracker actually select, and what happens if you omit it?
5. Where does Firecracker's vsock device's responsibility end and firecracker-containerd's ttrpc protocol begin?
6. Why is the agent-bearing rootfs not interchangeable with a normal Firecracker rootfs?
7. How does this lab's shim relate to Lab I3's Go SDK?

Next: Lab I2: The Jailer in Production — what the orchestrator must do, per microVM, to launch Firecracker safely.