Lab 1: The Jailer in Depth

Prerequisite reading: Security — Intensive (the four-layer model and boundary B), and The Jailer (deep dive) (the barrier, step by step).

Background

In production you never run firecracker directly. You run jailer, a separate root-privileged binary that builds an isolation barrier — a chroot, cgroups, namespaces, two device nodes — drops to an unprivileged UID/GID, and only then execs firecracker inside that confinement. From that moment the process you call "the VMM" is jailed, unprivileged, and (once Firecracker installs its own seccomp filter) syscall-restricted. The jailer is boundary B from the intensive overview made concrete: it assumes the guest may eventually compromise the VMM, and it makes that compromise worthless.

This lab is a trace-it and break-it exercise. You will run a real microVM under the real jailer, construct every piece of the barrier, and then observe each piece from outside the jail — its process tree, its namespaces, its cgroup, its near-empty filesystem. Then you will deliberately remove or skip individual steps and read the exact failure each one produces, so that you understand not just what the jailer does but what would happen if it didn't. Finally you will map every jailer action to the precise Linux mechanism (unshare, setns, pivot_root, mknod, setgid/setuid, cgroup file writes) it is built from, reading src/jailer/ to confirm.

Why this matters for contributors

• The jailer is one of the most readable security-critical files in the project, and PRs that touch it get the most scrutiny. To review one credibly you must know the order of operations cold — privilege drop after the chroot, gid before uid — because a reordering is a silent hole.
• Real issues in this area read like "VMM can't open /dev/kvm inside the jail," "microVM has no network under the jailer," "cgroup writes rejected on cgroup v2." Each maps to one barrier step. After this lab you will diagnose them by inspection, not guesswork.
• Orchestrators (firecracker-containerd, Kata) do enormous file-staging work because of the chroot. Understanding the jail directory layout is understanding why that plumbing exists. See the integration lab on the jailer in production.

Prerequisites

• Completed Level 9, Lab 9.1 and read the two deep dives above.
• A disposable host you own with /dev/kvm, root/sudo, and iproute2 (ip), util-linux (lsns, unshare, nsenter), and a Firecracker + jailer build.
• A kernel image and a rootfs you can boot by hand (from Level 1, Lab 1.3).

# Verify the build and locate both binaries — the jailer is separate from firecracker.
ARCH=$(uname -m)
TARGET=build/cargo_target/${ARCH}-unknown-linux-musl/release
ls -l $TARGET/firecracker $TARGET/jailer
# Confirm you have a kernel + rootfs staged somewhere you control:
ls -l ./vmlinux-* ./*.ext4 2>/dev/null || echo "stage a kernel + rootfs first (Lab 1.3)"

Step-by-step tasks

Step 1 — Read the jailer's barrier-building sequence before you run it

You will run the jailer in a moment, but read what it is going to do first. The whole binary is small and worth reading end to end; for now, locate the major phases by role.

# The jailer is its own crate — read it top to bottom; every line is a privileged op.
find src/jailer/src -name "*.rs" | sort
wc -l src/jailer/src/*.rs

# The central type and the run/setup methods (names by ROLE — confirm on your branch).
rg -n "struct Env|fn run\b|fn setup_jailed_folder|pivot_root|chroot|fn enter_chroot" src/jailer/src/

# The privilege-drop sequence — note the ORDER (gid before uid).
rg -n "setgid|setuid|setgroups" src/jailer/src/

# The two device nodes and the cgroup/namespace machinery.
rg -n "mknod|/dev/kvm|/dev/net/tun" src/jailer/src/
rg -n "cgroup|Cgroup|/sys/fs/cgroup|inherit_from_parent" src/jailer/src/
rg -n "unshare|CLONE_NEWNS|CLONE_NEWPID|setns|netns|new_pid_ns" src/jailer/src/

The jailer's ordered job, which you are about to reproduce and observe:

  (running as root)
        │ parse --id --exec-file --uid --gid --chroot-base-dir --netns --cgroup ...
        ▼  build chroot:  <chroot-base>/<exec_file>/<id>/root     (default base /srv/jailer)
  create + own the jail dir, copy firecracker in
        ▼  cgroups:  create the cgroup, write --cgroup K=V values
        ▼  namespaces:  unshare(mount); setns into --netns; optional new PID ns (--new-pid-ns)
        ▼  mknod /dev/kvm and /dev/net/tun inside the jail; chown to target uid/gid
        ▼  pivot_root / chroot into the jail; chdir to /
        ▼  setgid(gid); setuid(uid)     ← privileges dropped here, IRREVERSIBLY
        ▼  exec ./firecracker            (now unprivileged, jailed)
  ── Firecracker then installs its seccomp filter (Lab 2) ──

• Write out, from the rg output, the default jail path the jailer will build for --id intensive01 --exec-file <abs path>/firecracker. Which two flags determine the directory components? (Answer in your notes; you will create exactly this path.)

Step 2 — Set up the network namespace and a TAP device for the microVM

The jailer does not create networking; an orchestrator does, and points the jailer at it with --netns. Reproduce that: create a named network namespace, put a TAP device in it, and give the guest side an address. This is the same plumbing you will go deeper on in Networking Lab 1; here you only need it to exist so the jailer can setns into it.

# Create a named netns the jailer will join.
sudo ip netns add fcjail

# Create a TAP device INSIDE that netns and bring it up with a host-side address.
sudo ip netns exec fcjail ip tuntap add dev tap0 mode tap
sudo ip netns exec fcjail ip addr add 172.16.0.1/24 dev tap0
sudo ip netns exec fcjail ip link set tap0 up
sudo ip netns exec fcjail ip link set lo up

# Confirm the TAP lives in fcjail and NOT on the host.
sudo ip netns exec fcjail ip -br addr show
ip -br addr show | grep -q tap0 && echo "BUG: tap0 leaked to host" || echo "good: tap0 is only inside fcjail"

Note: The TAP device lives inside fcjail. When the jailer does setns(--netns), the jailed VMM inherits exactly this namespace — it sees tap0 and lo and nothing else. This is how a microVM gets exactly one network interface and no path to the host's other interfaces. The netns is the network half of the sandbox.

Step 3 — Run Firecracker under the jailer

Now construct the barrier for real. Pick an unprivileged UID/GID that exists on your box (e.g. 1000), choose a microVM id, and invoke the jailer. Everything after -- is passed through to firecracker.

ARCH=$(uname -m)
TARGET=$(pwd)/build/cargo_target/${ARCH}-unknown-linux-musl/release
ID=intensive01
JAIL=/srv/jailer/firecracker/${ID}/root      # the jailer will create this

sudo $TARGET/jailer \
  --id $ID \
  --exec-file $TARGET/firecracker \
  --uid 1000 --gid 1000 \
  --chroot-base-dir /srv/jailer \
  --netns /var/run/netns/fcjail \
  --cgroup cpu.max="100000 100000" \
  --cgroup pids.max=512 \
  --new-pid-ns \
  --daemonize \
  -- \
  --api-sock /run/api.socket

What each argument constructs:

Tip: The API socket path /run/api.socket is relative to the jail root once the VMM is chrooted — the real path on the host is <JAIL>/run/api.socket. This is the single most common "where did my socket go" confusion. The jailer chroots, so every path the VMM is handed must be reachable inside the jail.

Step 4 — Find the jailed process and inspect the jail directory

# Find the firecracker PID (it's a child of the jailer machinery; --new-pid-ns means it's PID 1
# in its OWN namespace, but the HOST still sees it under a host PID).
FCPID=$(pgrep -f "firecracker --api-sock /run/api.socket" | head -1)
echo "host-visible PID: $FCPID"

# Inspect the jail directory the jailer built. It should be near-empty.
sudo ls -la /srv/jailer/firecracker/${ID}/root
sudo ls -la /srv/jailer/firecracker/${ID}/root/dev    # only kvm and net/tun
sudo ls -la /srv/jailer/firecracker/${ID}/root/run    # the api socket

You should see a chroot containing essentially: the firecracker binary, a dev/ with exactly kvm and net/tun, and the API socket. That is the entire world the VMM can see. Contrast it with the host root — the difference is the chroot barrier.

• Confirm the device nodes are owned by uid/gid 1000 (the target), not root:

sudo ls -ln /srv/jailer/firecracker/${ID}/root/dev/kvm \
            /srv/jailer/firecracker/${ID}/root/dev/net/tun
# Expect:  crw-------  1 1000 1000  10, 232 ... kvm   (major/minor will match your host's /dev/kvm)

Step 5 — Observe the namespaces from outside (lsns, /proc/<pid>/ns)

This is the heart of the lab: prove, from outside, that the VMM is confined. Each namespace is a link in /proc/<pid>/ns/; if two processes share a namespace, the links point to the same inode.

# List the namespaces the jailed process is in.
sudo lsns -p $FCPID

# The raw namespace links — compare these inode numbers to your shell's.
sudo ls -l /proc/$FCPID/ns/
ls -l /proc/$$/ns/        # your interactive shell, for comparison

# Specifically: is the VMM in a DIFFERENT mount, network, and pid namespace than the host shell?
for NS in mnt net pid; do
  A=$(sudo readlink /proc/$FCPID/ns/$NS)
  B=$(readlink /proc/$$/ns/$NS)
  [ "$A" = "$B" ] && echo "$NS: SHARED (not isolated)" || echo "$NS: isolated  ($A vs $B)"
done

You should see mnt, net, and (because of --new-pid-ns) pid reported as isolated — the VMM's namespace inodes differ from your shell's. The net namespace inode should match fcjail:

# Prove the VMM's net namespace IS the fcjail namespace you created in Step 2.
sudo readlink /proc/$FCPID/ns/net
sudo ip netns exec fcjail readlink /proc/self/ns/net   # same inode → same namespace

   HOST shell                          JAILED firecracker (FCPID)
  /proc/$$/ns/mnt  → mnt:[4026531840]   /proc/FCPID/ns/mnt → mnt:[4026532XXX]   ◄─ DIFFERENT (unshare)
  /proc/$$/ns/net  → net:[4026531992]   /proc/FCPID/ns/net → net:[4026532YYY]   ◄─ == fcjail (setns)
  /proc/$$/ns/pid  → pid:[4026531836]   /proc/FCPID/ns/pid → pid:[4026532ZZZ]   ◄─ DIFFERENT (--new-pid-ns)

• From inside the PID namespace, confirm the VMM is PID 1 and cannot see host processes:

# Enter the VMM's namespaces and list processes — you should see almost nothing.
sudo nsenter --target $FCPID --pid --mount --net ps -e 2>/dev/null || \
sudo nsenter --target $FCPID --pid --net ps -e
# A handful of processes at most; the host's hundreds of PIDs are invisible.

Step 6 — Read the cgroup from outside

The jailer placed the VMM in a cgroup so one microVM cannot starve the host. Find it and read the limits you set.

# Which cgroup is the jailed process in? (cgroup v2 single hierarchy shown.)
cat /proc/$FCPID/cgroup
# e.g. 0::/firecracker/intensive01   (path depends on host + jailer version — verify)

CG=/sys/fs/cgroup$(awk -F: '{print $3}' /proc/$FCPID/cgroup | head -1)
sudo cat $CG/cpu.max     # the "100000 100000" you passed → 100% of one CPU
sudo cat $CG/pids.max    # 512
sudo cat $CG/pids.current
# Watch live accounting:
sudo cat $CG/cpu.stat | head

Note: cgroup v1 and v2 have different layouts and file names (cpu.cfs_quota_us vs cpu.max, separate hierarchies vs a unified one). The jailer supports both; which one you get depends on your host (stat -fc %T /sys/fs/cgroup — cgroup2fs means v2). Verify on your branch and host; never assume a path.

Step 7 — Break the barrier deliberately, one piece at a time

This is where the lab earns its keep. Each sub-step removes one barrier piece and shows the failure — proving what that piece was for. Tear the current microVM down first, then break things on fresh runs.

# Tear down cleanly between experiments.
sudo pkill -f "firecracker --api-sock /run/api.socket" 2>/dev/null
sudo rm -rf /srv/jailer/firecracker/${ID}

Break A — remove the /dev/kvm node after the jail is built. The VMM needs /dev/kvm to talk to KVM; without it, it cannot create the VM at all.

# Re-run the jailer (Step 3), then immediately delete the kvm node inside the jail:
sudo rm -f /srv/jailer/firecracker/${ID}/root/dev/kvm
# Now drive the API to boot — it will fail to open /dev/kvm.
# Expected: firecracker errors out unable to open /dev/kvm (ENOENT) — the chroot has no other path to it.
sudo cat /srv/jailer/firecracker/${ID}/root/run/firecracker.log 2>/dev/null | tail

Lesson: the chroot means the VMM has no fallback path to KVM. The two mknod'd nodes are its only window to the host kernel's device interface — by design.

Break B — point --netns at nothing (or omit it). Without the netns, the VMM runs in the host network namespace and its TAP lookup fails — or worse, it could see host interfaces.

# Run the jailer WITHOUT --netns and with a net interface configured to use tap0.
# tap0 lives in fcjail, so the host-namespace VMM cannot find it:
#   API error: "could not open TAP device tap0" / ENODEV, because tap0 isn't in this namespace.

Lesson: --netns is the network boundary. The TAP must be in the namespace the VMM joins, or there is no path between guest and host networking — and dropping the netns risks exposing host interfaces.

Break C — loosen the cgroup. Remove pids.max and spawn pressure to feel the difference (do this gently; you are stressing your own box).

# With pids.max=512 the VMM's task count is hard-capped. Read it under load:
sudo cat $CG/pids.current $CG/pids.max
# Without a pids cap (omit --cgroup pids.max), a buggy/hostile VMM could fork without bound,
# starving the host scheduler. The cgroup is the DoS boundary; seccomp + jail don't bound CPU/PIDs.

Lesson: namespaces and chroot isolate what the VMM can see; the cgroup bounds how much it can consume. They are orthogonal — you need both. A guest that cannot escape can still try to exhaust the host; the cgroup is the answer to that.

Break D — reason about privilege-drop order (do not actually run a broken jailer). The jailer calls setgid then setuid. Convince yourself why the reverse is a bug:

rg -n -B2 -A6 "setgid|setuid|setgroups" src/jailer/src/
# After setuid(1000) you are no longer root, so a later setgid(1000) may fail or be a no-op,
# leaving the process in the root group. gid-before-uid is the only correct order.

Step 8 — Map every jailer action to its Linux mechanism (the synthesis)

Close the lab by filling in this table from the source — every entry must be backed by a line you can point at with rg. This is the artifact you keep.

# Confirm the one thing the jailer does NOT do — it never touches seccomp.
rg -ni "seccomp" src/jailer/src/ || echo "correct: seccomp is Firecracker's job, after exec (Lab 2)"

Deliverables

• The exact jail directory path the jailer built, with the two flags that determined its components named.
• A listing of <jail>/root and <jail>/root/dev showing the near-empty filesystem and exactly the two device nodes, owned by the target uid/gid.
• The lsns//proc/<pid>/ns comparison proving mnt, net, and pid are isolated, plus the readlink showing the VMM's net namespace is fcjail.
• The cgroup path and the two limit files you set, read from /sys/fs/cgroup.
• Three captured failures from Step 7 (missing /dev/kvm, missing/empty netns, and the privilege-drop-order reasoning), each with the lesson it teaches.
• The completed action→mechanism table from Step 8, every row backed by an rg hit.

Troubleshooting

The jailer exits immediately with a permission error

You must start the jailer as root — it needs privilege to mknod, chown, unshare, setns, write cgroups, and finally drop privileges. sudo the jailer itself; do not sudo only the inner firecracker.

mknod fails or the device node is missing inside the jail

The jailer creates /dev/kvm and /dev/net/tun with specific major/minor numbers read from the host. If your host's /dev/kvm has unusual numbers, confirm with ls -l /dev/kvm /dev/net/tun and compare to what landed in the jail. A container or restricted host may block mknod entirely (check capabilities).

setns into --netns fails (ENOENT / EINVAL)

The path must be a real netns handle, typically /var/run/netns/<name> after ip netns add <name>. Confirm it exists (ip netns list) and that you created the TAP inside it (Step 2). An empty or nonexistent netns path is Break B.

cgroup writes rejected (ENOENT / EINVAL)

Almost always a cgroup v1-vs-v2 mismatch. Check stat -fc %T /sys/fs/cgroup (cgroup2fs = v2) and match the controller file names you pass to --cgroup (cpu.max on v2 vs cpu.cfs_quota_us on v1). Verify the controller is enabled in the parent cgroup's cgroup.subtree_control.

The VMM dies with SIGSYS right after exec

This is not a jailer bug — it is seccomp firing, which means the jailer handed off correctly. The VMM tried a syscall the filter denies. That is Lab 2's entire subject; see Seccomp Filtering.

The API socket "doesn't exist"

It does — at <jail>/root/run/api.socket, not /run/api.socket on the host. The chroot relocates every path. Talk to it at the real on-host path: curl --unix-socket /srv/jailer/firecracker/<id>/root/run/api.socket ....

Expected output

• The jail directory under /srv/jailer/firecracker/<id>/root exists, is near-empty, and contains exactly the firecracker binary, dev/kvm, dev/net/tun, and the API socket.
• lsns -p <pid> shows the VMM in its own mnt and (with --new-pid-ns) pid namespace, and in the fcjail net namespace.
• /proc/<pid>/cgroup resolves to a cgroup whose cpu.max/pids.max match what you passed.
• Removing /dev/kvm from the jail makes the VMM fail to open KVM; a missing netns makes the TAP unreachable; both prove what their barrier piece is for.

Stretch goals

1. Run two microVMs side by side with different --ids and different --netns (fcjail0, fcjail1). Prove with lsns and readlink that they share nothing — different mount, net, and pid namespaces, different jail directories, different cgroups. This is the multi-tenant posture.
2. Pin one microVM to a single CPU with --cgroup cpuset.cpus=0 (plus cpuset.mems=0) and verify with taskset -p or by reading /proc/<pid>/status Cpus_allowed_list that the VMM and all its threads are confined to CPU 0.
3. Add an rlimit: pass --resource-limit no-file=1024 and confirm via cat /proc/<pid>/limits that Max open files is 1024. Reason about why bounding file descriptors matters for a host running thousands of microVMs.
4. Read docs/jailer.md end to end (sed -n '1,200p' docs/jailer.md) and diff its recommended invocation against yours — note every production flag you omitted and why.

Validation / self-check

Answer these without notes; they gate completion.

• List, in order, every privileged action the jailer performs from "running as root" to "exec firecracker," and name the single step that makes the VMM unprivileged.
• Given --id web1 --exec-file /usr/bin/firecracker, write the default jail path and name the two flags that determine its components.
• You ran lsns -p <pid> and three namespaces showed as isolated. Name them, and say which CLI flag created each.
• Why exactly /dev/kvm and /dev/net/tun and nothing else in the jail's /dev? What happens if /dev/kvm is missing, and why is there no fallback?
• Distinguish what the namespaces + chroot bound from what the cgroup bounds, with one concrete failure each barrier prevents.
• Why must setgid precede setuid, and why is the drop irreversible? What protection does running firecracker without the jailer lose entirely?
• State precisely which sandbox layer the jailer owns and which it does not — and name the binary and the moment responsible for the layer it does not.

Next: Lab 2: Seccomp filters — the syscall-level sandbox Firecracker installs on itself immediately after the jailer's exec. You will deny a syscall and watch the VMM die with SIGSYS.