Lab I2: The Jailer in Production

Background

In every demo — including Lab 1.3 — you run firecracker directly. No serious deployment does that. In production, an orchestrator launches each microVM through the jailer: a small Rust binary, shipped in the same repo, that runs as root, builds an isolation barrier around the about-to-run VMM (a chroot, cgroups, namespaces, an explicit UID/GID), drops its own privileges, and only then execs firecracker as an unprivileged process inside that barrier. The jailer is the difference between "a microVM is contained by KVM" and "a microVM is contained by KVM and the host is defended even if the VMM is compromised." It is the second pillar of the threat model: the guest is untrusted, the VMM is privileged host code and therefore also part of the attack surface, and the jailer is what limits the blast radius if that surface is breached.

This lab is build-it. You will read docs/jailer.md and [docs/prod-host-setup.md] from the real repo, then write a small wrapper script that launches a properly jailed microVM the way an orchestrator would: per-instance UID/GID, cgroups v2 resource limits, a network namespace, a chroot the jailer builds, and the seccomp filter Firecracker applies. The goal is to understand, concretely, what an orchestrator must do per microVM — because that list is exactly what you must be able to subtract when attributing a bug in Lab I4. Half of "is this Firecracker or the host?" is "did the jailer set up the environment correctly?"

Note: The jailer applies the process-level isolation (chroot, cgroups, namespaces, priv-drop). It does not apply seccomp — firecracker installs its own seccomp-BPF filters at startup. Keep that split clear; it surprises people.

Why This Lab Matters for Contributors

• The jailer is where a large class of "Firecracker won't start in production" issues actually live — a missing mknod for /dev/kvm in the chroot, a cgroup that throttled the VMM, a netns the TAP device wasn't moved into. Knowing the jailer cold lets you attribute these in seconds.
• Reviewing jailer PRs and prod-host-setup changes is high-trust maintainer work; you cannot review what you have not operated.
• It grounds the jailer deep dive and the security masterclass in a runnable artifact.
• Every orchestrator in Lab I1 and Lab I3 does some version of what your script does; you will recognize their setup code.

Prerequisites

Confirm cgroups v2 and locate the jailer binary and its docs — read, do not assume:

# cgroups v2 unified hierarchy? (the jailer's --cgroup-version default has changed over time — verify)
mount | rg cgroup2 && stat -fc %T /sys/fs/cgroup    # "cgroup2fs" => unified v2

# Find the jailer binary you built and the authoritative docs.
find . -name jailer -type f -path '*release*'
ls docs/jailer.md docs/prod-host-setup.md

# Read the jailer's REAL flag set — do not trust this lab's list, the flags drift.
./jailer --help 2>&1 | head -60
rg -n "uid|gid|--cgroup|--resource-limit|--netns|--chroot-base-dir|--new-pid-ns|--daemonize" docs/jailer.md

What the jailer does (and what's left to the orchestrator)

   orchestrator (root)                jailer (root → unpriv)            firecracker (unpriv)
   ────────────────────               ──────────────────────           ────────────────────
   pick UID/GID, id, netns    ──►   build chroot at                ──►  open /dev/kvm
   create cgroup v2 limits          <base>/<exec>/<id>/root             API thread + VMM thread
   create the netns + TAP           pivot_root / chroot                 + vCPU threads
   set resource limits              mknod /dev/kvm, /dev/net/tun        install seccomp-BPF
   exec the jailer  ───────────►    join cgroup, join netns
                                    setuid(uid) / setgid(gid)
                                    exec firecracker  ───────────────►  (runs, now unprivileged)

The division of labor — memorize this table, it is the attribution checklist:

The crucial insight: the jailer does a fixed set of mechanical things, but the orchestrator owns the policy — which UID, which cgroup limits, which netns, whether to disable SMT host-wide. When a jailed microVM misbehaves, the question is almost always "did the orchestrator hand the jailer the right policy?" before it is ever "is firecracker broken?"

Step-by-Step Tasks

You will build launch-jailed-microvm.sh, incrementally. Each step adds one layer of the barrier and verifies it.

Step 1: Lay out the per-instance identity and chroot

An orchestrator assigns each microVM a unique id and a dedicated UID/GID so a guest escape cannot touch other instances' files.

#!/usr/bin/env bash
# launch-jailed-microvm.sh — a teaching-grade production-style launcher.
set -euo pipefail

JAILER=${JAILER:-./jailer}
FIRECRACKER_BIN=$(realpath ./firecracker)
ID="vm-$(date +%s)-$$"               # unique per microVM
UID_FC=10000; GID_FC=10000           # per-instance, unprivileged — pick uniquely per VM in prod
CHROOT_BASE=/srv/jailer              # jailer default base; chroot lands at $CHROOT_BASE/firecracker/$ID/root
JROOT="$CHROOT_BASE/firecracker/$ID/root"

# Stage the kernel + rootfs INTO the chroot — the jailed firecracker can only see paths under root/.
sudo install -d -o "$UID_FC" -g "$GID_FC" "$JROOT"
sudo cp ./vmlinux "$JROOT/vmlinux"
sudo cp ./rootfs.ext4 "$JROOT/rootfs.ext4"
sudo chown "$UID_FC:$GID_FC" "$JROOT/vmlinux" "$JROOT/rootfs.ext4"

Warning: Inside the jail, all API paths are relative to the chroot. The microVM's kernel path is vmlinux, not /abs/path/vmlinux. Forgetting this is the #1 jailer mistake — see Troubleshooting. The jailer mknods /dev/kvm for you, so do not copy a device node yourself.

Step 2: Create a cgroups v2 limit set

Production limits each microVM's CPU and memory so one tenant cannot starve the host. With cgroups v2 the jailer takes --cgroup controller.file=value pairs.

# Append to the script: build the cgroup args the jailer will apply.
CGROUP_ARGS=(
  --cgroup-version 2                  # verify the flag/default on your branch
  --cgroup "memory.max=536870912"     # 512 MiB ceiling for the whole jailed process tree
  --cgroup "cpu.max=50000 100000"     # 50% of one CPU (quota/period); quoting varies — verify
)
# A wall-clock / fd resource limit, applied via setrlimit by the jailer:
RLIMIT_ARGS=( --resource-limit "no-file=1024" )   # verify the exact key name

Verify the controllers you intend to use are actually enabled in the unified hierarchy before depending on them:

cat /sys/fs/cgroup/cgroup.controllers        # must list memory cpu io ...

Step 3: Create a network namespace and a TAP device for the microVM

Each microVM gets its own netns so guest traffic is isolated and the host's main network is untouched. The orchestrator creates the netns and the TAP; the jailer only joins it.

NETNS="fcnet-$ID"
TAP="tap0"                            # name is per-netns, so "tap0" is fine in each
sudo ip netns add "$NETNS"
sudo ip netns exec "$NETNS" ip tuntap add "$TAP" mode tap
sudo ip netns exec "$NETNS" ip addr add 172.16.0.1/30 dev "$TAP"
sudo ip netns exec "$NETNS" ip link set "$TAP" up
sudo ip netns exec "$NETNS" ip link set lo up
NETNS_PATH="/var/run/netns/$NETNS"

The microVM's /network-interfaces/eth0 will reference host_dev_name: "tap0", which resolves inside the netns the jailer joined — not the host's default namespace. This is exactly the kind of indirection that produces "the microVM has no network" bugs that are really orchestrator/netns bugs, not Firecracker bugs.

Step 4: Assemble the jailer invocation

Now combine identity, cgroups, and netns into the real jailer call. Everything after -- is passed through to firecracker.

sudo "$JAILER" \
  --id "$ID" \
  --exec-file "$FIRECRACKER_BIN" \
  --uid "$UID_FC" --gid "$GID_FC" \
  --chroot-base-dir "$CHROOT_BASE" \
  --netns "$NETNS_PATH" \
  "${CGROUP_ARGS[@]}" "${RLIMIT_ARGS[@]}" \
  --new-pid-ns \
  --daemonize \
  -- \
  --api-sock "/run/firecracker.socket"      # this path is INSIDE the chroot

After this returns, the socket exists at $JROOT/run/firecracker.socket on the host filesystem. Confirm the process is jailed:

FC_PID=$(pgrep -f "firecracker --api-sock")
ls -l /proc/$FC_PID/root            # symlink target is the chroot, not /
cat /proc/$FC_PID/status | rg -i "Uid|Gid"     # the unprivileged UID/GID you set
cat /proc/$FC_PID/cgroup            # the cgroup the jailer placed it in
ip netns identify $FC_PID           # the netns it joined

Those four checks are the proof that the barrier exists. An orchestrator that cannot produce them has not actually jailed anything.

Step 5: Configure and boot through the jailed socket

The API is identical to the un-jailed case — only the paths are chroot-relative and the socket lives under the chroot.

API="$JROOT/run/firecracker.socket"
sudo curl -X PUT --unix-socket "$API" --data \
  '{"kernel_image_path":"vmlinux","boot_args":"console=ttyS0 reboot=k panic=1"}' \
  http://localhost/boot-source
sudo curl -X PUT --unix-socket "$API" --data \
  '{"drive_id":"rootfs","path_on_host":"rootfs.ext4","is_root_device":true,"is_read_only":false}' \
  http://localhost/drives/rootfs
sudo curl -X PUT --unix-socket "$API" --data \
  '{"iface_id":"eth0","guest_mac":"06:00:AC:10:00:02","host_dev_name":"tap0"}' \
  http://localhost/network-interfaces/eth0
sudo curl -X PUT --unix-socket "$API" --data '{"action_type":"InstanceStart"}' \
  http://localhost/actions

Note the relative paths (vmlinux, rootfs.ext4) — that is the chroot at work.

Step 6: Prove seccomp is on, and where it comes from

The jailer did not apply seccomp; firecracker did. Confirm the filter is active on the running process:

grep Seccomp /proc/$FC_PID/status           # Seccomp: 2  => a BPF filter is installed
# And know where the filter is defined — baked in at build from JSON:
ls resources/seccomp/                        # <arch>.json per category (vmm/api/vcpu)
rg -n "vmm|api|vcpu|default_action" resources/seccomp/*.json | head

Read the seccomp deep dive for what the three per-thread filters allow. The point for this lab: in production you ship the default baked-in filter; --no-seccomp is a debugging aid only and must never reach prod.

Step 7: Apply the prod-host-setup hardening checklist

The jailer protects per-instance; docs/prod-host-setup.md protects the host. Read it and apply (or at least audit) each item — these are host-wide, orchestrator/operator responsibilities, not Firecracker code:

rg -n "SMT|hyperthread|KSM|swap|169.254.169.254|Rowhammer|ECC|TRR" docs/prod-host-setup.md

None of these is a Firecracker setting. That is the lesson: a "side-channel" or "guest reached the host IMDS" report is a host/orchestrator finding, attributed in Lab I4 to the host layer, not to firecracker.

Implementation Requirements / Deliverables

• launch-jailed-microvm.sh that launches a microVM with: a per-instance UID/GID, a chroot the jailer builds, cgroups v2 memory+cpu limits, and a dedicated netns+TAP.
• The four jail proofs from Step 4 (/proc/$PID/root, status UID/GID, cgroup, netns).
• Seccomp: 2 confirmed on the running firecracker process, plus the location of the filter JSON.
• A booted, networked microVM reachable only through the chroot-relative socket.
• A short written audit of the prod-host-setup.md checklist against your host.

Troubleshooting

No such file or directory for the kernel after InstanceStart

You passed an absolute kernel path to a jailed firecracker. Inside the jail the path is relative to the chroot — stage the file under root/ and reference it as vmlinux, not /path/vmlinux. This is the single most common jailer error.

Could not open /dev/kvm from the jailed process

The chroot is missing the device node. The jailer normally mknods /dev/kvm and /dev/net/tun; confirm ls -l $JROOT/dev/. If they're absent, your jailer flags or version differ — re-read ./jailer --help.

The microVM has no network

Almost always a netns/TAP problem, not Firecracker. Confirm the TAP exists inside the netns the jailer joined (ip netns exec $NETNS ip link), that host_dev_name matches the TAP name, and that the jailer was given the correct --netns path.

The VMM is being throttled / OOM-killed

A cgroup limit is too tight. Inspect cat /proc/$FC_PID/cgroup and the controller files; memory.max includes the VMM's own footprint plus guest RAM — size it accordingly.

cgroup args rejected

The controller isn't enabled in the unified hierarchy, or the v1/v2 flag is wrong for your branch. Check cat /sys/fs/cgroup/cgroup.controllers and re-verify the jailer's --cgroup-version semantics in docs/jailer.md.

Expected Output

A daemonized, jailed firecracker process whose /proc/$PID/root is the chroot, whose UID/GID are your per-instance unprivileged ids, that sits in a constrained cgroup and a dedicated netns, with Seccomp: 2 set — running a microVM you configured and booted through the chroot-relative socket. In other words: the exact thing an orchestrator launches thousands of times per host.

Stretch Goals

1. Add --new-pid-ns and confirm the VMM sees a fresh PID namespace (ls /proc/$FC_PID/ns/pid); reason about why an orchestrator might or might not want it.
2. Wire the TAP to a host bridge with NAT so the guest reaches the internet, then remove guest access to 169.254.169.254 per the hardening checklist and verify it's blocked.
3. Launch two jailed microVMs with distinct UIDs and confirm one cannot read the other's chroot files — the per-instance UID isolation in action.
4. Compare your script to how firecracker-containerd or the Go SDK's jailer support sets up the same barrier.

Validation / Self-check

1. List every isolation mechanism the jailer applies, and the one it does not (and who does that one).
2. Why are kernel/rootfs paths relative inside the jail, and what breaks if you forget?
3. Which jail properties does the orchestrator decide, and which does the jailer merely execute?
4. How do you prove, from /proc, that a firecracker process is actually jailed?
5. A jailed microVM has no network. Name three orchestrator/netns causes you'd check before suspecting Firecracker.
6. Why is --no-seccomp a debugging-only flag, and where does the real filter come from?
7. Pick three prod-host-setup.md items and say which boundary owns each (Firecracker / host / orchestrator).

Next: Lab I3: The Go SDK and firectl — drive Firecracker programmatically and see how an SDK maps to the REST API.