The Jailer

The jailer is a separate binary that runs before Firecracker and exists for one reason: to build an isolation barrier around the VMM and then drop every privilege it does not need. In production you do not run firecracker directly — you run jailer, which sets up a chroot, cgroups, namespaces, and the two device nodes the VMM needs, switches to an unprivileged UID/GID, and then execs firecracker inside that confinement. From that point on, the process you call "the VMM" is unprivileged, jailed, and — once Firecracker installs its own seccomp filter — syscall-restricted.

This chapter covers what the jailer is, the exact barrier it constructs (pivot_root/chroot, cgroups, namespaces, mknod, privilege drop, --daemonize), the CLI surface that controls it, where it sits in the threat model, and the one thing it deliberately does not do.

Note: The jailer does not apply seccomp. That is Firecracker's job, after the exec. Keep this division straight: the jailer owns the process-level sandbox (filesystem, cgroups, namespaces, privileges); Firecracker owns the syscall-level sandbox (seccomp-BPF). The two are complementary layers of defense in depth, applied by two different binaries at two different times.

Where the jailer lives and what it is

# The jailer is its own crate / binary, separate from `vmm` and `firecracker`.
find src/jailer -name "*.rs" | sort
rg -n "fn main|struct Env|fn run|setup_jailed_folder|pivot_root|chroot" src/jailer/src/

The jailer is a small, self-contained program in src/jailer/. Its central type is an Env struct (locate it with rg -n "struct Env" src/jailer/src/) built from parsed CLI arguments; the real work happens in its setup/run methods. Read it top to bottom — it is one of the most readable security-critical files in the project, and every line is a privileged operation you should be able to justify.

The jailer's job, in order, is:

  (running as root)
        │
        ▼
  parse args (--id, --exec-file, --uid, --gid, --chroot-base-dir, --netns, ...)
        │
        ▼  build chroot:  <chroot-base>/<exec_file>/<id>/root   (default base /srv/jailer)
  create + own the jail directory, copy the firecracker binary in
        │
        ▼  cgroups:  create cgroup, write --cgroup K=V and --resource-limit values
        │
        ▼  namespaces:  unshare(mount); setns into --netns; optional new PID ns (--new-pid-ns)
        │
        ▼  mknod /dev/kvm and /dev/net/tun inside the jail; mount what's needed
        │
        ▼  pivot_root / chroot into the jail; chdir to /
        │
        ▼  setgid(gid); setuid(uid)   ← privileges dropped here, irreversibly
        │
        ▼  exec  ./firecracker  (now unprivileged, jailed)
  ── from here Firecracker installs its seccomp filter and runs the microVM ──

The chroot barrier: pivot_root and the jail directory

rg -n "chroot_base|/srv/jailer|exec_file|chroot_dir|pivot_root|fn setup_jailed_folder|copy" src/jailer/src/

The jailer constructs a private root filesystem for the VMM. By default the jail lives at:

/srv/jailer/<exec_file_name>/<id>/root
└────────┬────────┘ └───┬───┘ └┬┘ └─┬─┘
 --chroot-base-dir   exec name  id  the new "/"

--id is the unique microVM identifier; --exec-file is the path to the firecracker binary (its basename becomes a path component). The jailer creates this directory, copies the firecracker binary into it, creates the device nodes inside it, and then makes it the process root via pivot_root (the kernel-preferred mechanism) or chroot. After this the VMM can see nothing on the host filesystem except what is inside the jail. The API socket, the kernel image, and the rootfs must therefore be placed (or bind-mounted/hard-linked) inside the jail by whoever launches the jailer.

Tip: This is why orchestrators (firecracker-containerd, Kata) do a lot of file shuffling: they must stage the kernel, rootfs, and socket paths into <jail>/root because the jailed VMM cannot reach out. The jail is the host↔VMM trust boundary made concrete in the filesystem.

Everything after a -- separator is passed through to firecracker (the API socket path, seccomp flags, etc.). Find the separator handling with rg -n '"--"|extra_args|app_args' src/jailer/src/.

Device nodes: mknod /dev/kvm and /dev/net/tun

rg -n "mknod|makedev|/dev/kvm|/dev/net/tun|dev_kvm|tuntap|major|minor" src/jailer/src/

Inside the jail there is no /dev, but the VMM needs exactly two character devices: /dev/kvm (to talk to KVM) and /dev/net/tun (to open the TAP backend for virtio-net). The jailer creates these with mknod using the correct major/minor numbers, then chowns them to the target UID/GID so the unprivileged VMM can open them. This is deliberate minimalism: the jailed VMM gets only these two device nodes and nothing else. A guest that compromises the VMM finds a filesystem with almost nothing in it.

cgroups and resource limits

rg -n "cgroup|Cgroup|cpuset|/sys/fs/cgroup|resource_limit|rlimit|setrlimit|fn inherit_from_parent" src/jailer/src/

The jailer places the VMM into a cgroup so a single microVM cannot starve the host. --cgroup K=V writes arbitrary cgroup controller files (e.g. --cgroup cpuset.cpus=0 pins the microVM to one CPU); the jailer supports both cgroup v1 and v2 layouts (verify which on your branch and host). --resource-limit sets process rlimits such as no-file (open file descriptors) and fsize (max file size) via setrlimit. These are blunt but effective: the cgroup bounds CPU/memory at the kernel scheduler/accounting layer, the rlimits bound per-process resource counts.

Namespaces: mount, network, and (optionally) PID

rg -n "unshare|CLONE_NEWNS|CLONE_NEWPID|setns|netns|new_pid_ns|fork|clone" src/jailer/src/

The jailer uses Linux namespaces to further isolate the VMM:

• Mount namespace — unshare(CLONE_NEWNS) so the jail's mount changes (and the pivot_root) do not affect the host mount table.
• Network namespace — --netns <path> makes the jailer setns into a pre-created network namespace before exec. This is how you give a microVM exactly one TAP interface and nothing else: the orchestrator creates a netns, puts a TAP device in it, and points the jailer at it. The VMM then only sees that namespace's networking.
• PID namespace — --new-pid-ns runs the VMM in a fresh PID namespace, so the VMM becomes PID 1 in its own namespace and cannot see or signal host processes.

flowchart TD
    Root["jailer (root, host namespaces)"] --> Mnt["unshare mount ns"]
    Mnt --> Net["setns into --netns (TAP lives here)"]
    Net --> Pid{"--new-pid-ns?"}
    Pid -->|yes| NewPid["new PID namespace, VMM = PID 1"]
    Pid -->|no| HostPid["host PID namespace"]
    NewPid --> Chroot["pivot_root into jail"]
    HostPid --> Chroot
    Chroot --> Drop["setgid + setuid (drop privileges)"]
    Drop --> Exec["exec ./firecracker (unprivileged, jailed)"]
    Exec --> Seccomp["Firecracker installs seccomp-BPF"]

Privilege drop and exec — the point of no return

rg -n "setuid|setgid|setgroups|set_uid|set_gid|cap|securebits|exec|execve|fn enter_chroot|fn run" src/jailer/src/

After the jail, cgroups, namespaces, and device nodes are in place, the jailer drops privileges. It sets the supplementary groups, then setgid(gid), then setuid(uid) — order matters, because once you have dropped the UID you can no longer change the GID. This drop is irreversible: there is no saved-set-uid to climb back to root. Only after privileges are gone does the jailer exec the firecracker binary. The VMM therefore starts life unprivileged. Firecracker then installs its seccomp-BPF filter (see seccomp-filtering.md) as its own first act, closing the syscall surface.

Warning: If you run firecracker directly (no jailer), you get none of this: no chroot, no cgroups, no namespaces, no privilege drop — only seccomp (and only if you do not pass --no-seccomp). That configuration is fine for local development and the labs, but it is not the production posture. Production = jailer + seccomp + the host hardening in docs/prod-host-setup.md.

Reading exercise

# 1. The whole jailer is small — read it end to end.
find src/jailer/src -name "*.rs" | sort
wc -l src/jailer/src/*.rs

# 2. Locate the privilege-drop sequence and confirm the order (gid before uid).
rg -n "setgid|setuid|setgroups" src/jailer/src/

# 3. Find where the two device nodes are created.
rg -n "mknod|/dev/kvm|/dev/net/tun" src/jailer/src/

# 4. Find the chroot/pivot_root and the default base path.
rg -n "pivot_root|chroot|/srv/jailer|chroot_base" src/jailer/src/

# 5. The official docs and a real invocation.
sed -n '1,80p' docs/jailer.md

# 6. Confirm the jailer does NOT touch seccomp (expect no real seccomp setup here).
rg -ni "seccomp" src/jailer/src/ || echo "no seccomp in jailer — correct"

Answer:

1. Write out the default jail path for --id web1 --exec-file /usr/bin/firecracker. Which two CLI flags determine the directory components?
2. In what order are setgid and setuid called, and why does the order matter?
3. Which two device nodes does the jailer create inside the jail, and why exactly those two?
4. What does --netns do, and how does it combine with an orchestrator-created TAP device to give a microVM exactly one network interface?
5. The jailer drops privileges but never installs a seccomp filter. Who installs seccomp, and when?
6. If you run firecracker without the jailer, which protections do you lose and which do you keep?

Common bugs and symptoms

Validation: prove you understand this

1. List, in order, every privileged action the jailer performs from "running as root" to "exec firecracker," and say which step makes the VMM unprivileged.
2. Explain the jail directory naming scheme and why orchestrators must stage files into it.
3. Why does the jailer create exactly /dev/kvm and /dev/net/tun and nothing else in /dev?
4. Describe how mount, network, and PID namespaces each narrow what the VMM can see, and which CLI flag enables each.
5. Explain the cgroup-vs-rlimit split: what does each bound, and give one concrete --cgroup and one --resource-limit example.
6. State precisely which sandbox layer the jailer owns and which it does not, and name the binary responsible for the layer it does not.

Next: seccomp-filtering.md — the syscall-level sandbox Firecracker installs on itself immediately after the jailer's exec.