Lab 6.3: Build It — Boot Configuration and the Kernel Command Line

Background

The kernel command line is the single most leverage-per-byte interface between Firecracker and the guest. A handful of space-separated tokens decide whether you see boot output, how the guest reacts to a panic, which init runs, and — crucially — where the guest finds its virtio devices. In this lab you stop reading and start operating: you drive boot_args from the API, add an initrd, watch each change land in the guest's behavior and in the kernel's printed command line, and then read the code that assembles that command line — linux-loader's Cmdline — and the place where Firecracker appends the virtio_mmio.device=SIZE@ADDR:IRQ advertisements that make the guest's devices discoverable at all.

This is a build-it / experiment-it lab. You will make controlled changes to the boot configuration, predict their effect, confirm it empirically, and finally (optionally) thread a harmless custom parameter through the config path yourself so you understand exactly how a boot_args string becomes bytes at CMDLINE_START and a pointer in the zero page.

Note: Everything you set in boot_args is delivered to the guest kernel through the structures you traced in Lab 6.1 (the cmdline at CMDLINE_START, pointed to by the zero page) and placed in the layout you mapped in Lab 6.2. This lab closes the loop: config → code → guest behavior.

Why This Lab Matters for Contributors

• boot-source validation, default-cmdline behavior, and the device-advertisement logic are all live contribution areas — small, well-scoped, and impactful. A wrong virtio_mmio.device= string means a device the guest can't see.
• Understanding how the cmdline is constructed (not just configured) is the prerequisite for any change to it — and for diagnosing the common "my custom param had no effect / broke boot" reports.
• The interaction between transport (virtio-mmio default vs --enable-pci) and how devices are advertised (cmdline string vs PCI enumeration vs FDT node) is exactly the kind of cross-cutting knowledge maintainers expect.
• Building a real initrd and reading how it's wired teaches the second pointer the zero page carries. See the linux-loader chapter and the virtio-MMIO transport deep dive.

Prerequisites

• Completed Lab 6.1 and Lab 6.2.
• A built Firecracker, a vmlinux, and an ext4 rootfs (from Lab 1.3).
• The serial console reachable (so you can read guest output). Verify a baseline boot works:

# Confirm the boot-source config struct (the API surface you'll be driving).
rg -n "struct BootSourceConfig|boot_args|initrd_path|kernel_image_path" \
  src/vmm/src/vmm_config/boot_source.rs

Step-by-Step Tasks

Step 1: Establish a baseline and capture the default cmdline

Boot once with a minimal boot_args and capture exactly what the kernel prints as its command line. That printed line is your ground truth for every subsequent change.

API=/tmp/fc.sock
rm -f $API
sudo ./firecracker --api-sock $API &
curl -X PUT --unix-socket $API --data \
 '{"kernel_image_path":"./vmlinux-6.1.x","boot_args":"console=ttyS0 reboot=k panic=1"}' \
 http://localhost/boot-source
curl -X PUT --unix-socket $API --data \
 '{"drive_id":"rootfs","path_on_host":"./ubuntu-24.04.ext4","is_root_device":true,"is_read_only":false}' \
 http://localhost/drives/rootfs
curl -X PUT --unix-socket $API --data '{"vcpu_count":1,"mem_size_mib":256}' \
 http://localhost/machine-config
curl -X PUT --unix-socket $API --data \
 '{"iface_id":"net1","guest_mac":"06:00:AC:10:00:02","host_dev_name":"tap0"}' \
 http://localhost/network-interfaces/net1
curl -X PUT --unix-socket $API --data '{"action_type":"InstanceStart"}' http://localhost/actions

In the serial output find the line:

Command line: console=ttyS0 reboot=k panic=1 root=... virtio_mmio.device=...@...:... ...

Notice it is longer than what you supplied. Firecracker appended things — at minimum the virtio_mmio.device= advertisements for the block and net devices, and possibly defaults. Record the full printed line.

Tip: What you put in boot_args is the user part of the cmdline. Firecracker prepends/appends its own machinery (device advertisements, sometimes defaults). The kernel sees the concatenation.

Step 2: Understand the default cmdline

Even with an empty boot_args, Firecracker has a notion of a sensible default. Find it and the tokens it implies.

rg -n "DEFAULT_KERNEL_CMDLINE|reboot=k|panic=1|pci=off|nomodule|8250.nr_uarts|console=ttyS0" \
  src/vmm/src/

The conventional Firecracker default is approximately reboot=k panic=1 pci=off nomodule 8250.nr_uarts=0 (plus console=ttyS0 when serial is wanted) — but this varies by version and config; verify on your branch. Read what each token does:

Record: reboot=k + panic=1 together are why a panicking microVM cleanly exits via the i8042 reset path instead of hanging forever — important for serverless, where a stuck VM is wasted capacity.

Step 3: Experiment — change boot_args and observe

Now make controlled changes. For each, predict the effect first, then boot and confirm. Re-run the Step 1 sequence with a different boot_args each time (restart firecracker between runs).

# Example: the custom-marker run. After boot, in the guest:
cat /proc/cmdline      # the guest's view — should contain fc.lab=hello

Record each prediction-vs-observation. The fc.lab=hello case is the cleanest demonstration that boot_args text travels verbatim into the guest: you put it in the API JSON, Firecracker wrote it at CMDLINE_START, the zero page pointed the kernel at it, and /proc/cmdline reflects it back. That is the entire round trip from Labs 6.1–6.2 made observable.

Warning: panic=0 + a real panic will wedge the microVM (no auto-reboot). Kill firecracker manually. This is why the default is panic=1.

Step 4: Add an initrd and observe the second zero-page pointer

The zero page carries two pointers: the cmdline and the initrd. Wire one up.

# Build a trivial initramfs (any small cpio.gz works; or use a CI initrd if you have one).
mkdir -p initramfs/bin
# (populate a minimal busybox + init if you want it to actually run; for observation, even an
#  empty-ish cpio is enough to see the pointer get set and the kernel report it.)
( cd initramfs && find . | cpio -o -H newc 2>/dev/null | gzip > ../initrd.img )

API=/tmp/fc.sock; rm -f $API; sudo ./firecracker --api-sock $API &
curl -X PUT --unix-socket $API --data \
 '{"kernel_image_path":"./vmlinux-6.1.x","initrd_path":"./initrd.img","boot_args":"console=ttyS0 reboot=k panic=1"}' \
 http://localhost/boot-source
# ... drives / machine-config / InstanceStart as before ...

In the serial output, find the kernel's acknowledgment of the initrd:

Trying to unpack rootfs image as initramfs...
Freeing initrd memory: ...K

Record: initrd_path caused Firecracker to copy the image high in RAM (Lab 6.1, Step 4) and store its address+size in the zero page — the kernel found it and unpacked it. Now both zero-page pointers are exercised: cmdline (Step 3) and initrd (here).

Step 5: Read how the cmdline is built — linux-loader's Cmdline

Time to read the construction code. The cmdline is not a raw string Firecracker hands to the kernel; it is built with linux-loader's Cmdline type, which enforces length limits and escaping, then is serialized into guest memory.

# Where Firecracker creates and populates the Cmdline.
rg -n "Cmdline|cmdline::Cmdline|Cmdline::new|\.insert\(|\.insert_str\(|\.as_cstring|write_to|CMDLINE_START" \
  src/vmm/src/ | head -40
# The boot_args string the user supplied flows in here:
rg -n "boot_args|cmdline.*insert|kernel_cmdline" src/vmm/src/builder.rs src/vmm/src/vmm_config/

Read for these facts and record them:

1. A Cmdline is created (with a max length tied to the layout — the cmdline region has a bounded size).
2. The user's boot_args is inserted, and Firecracker inserts additional key/value tokens (insert/insert_str).
3. The finished Cmdline is converted to a C string and written into guest memory at CMDLINE_START; its guest address is what the zero page (or FDT, on aarch64) records.

# Confirm the cmdline size bound / where overflow is rejected.
rg -n "CMDLINE_MAX_LEN|MAX_.*CMDLINE|cmdline.*too.*long|TooLarge|capacity" src/vmm/src/

Step 6: Read how virtio-MMIO devices are advertised

This is the most important appended tokens. On x86_64 with the virtio-MMIO transport (the default), the guest has no way to discover devices — there is no PCI to enumerate. So Firecracker tells it, on the cmdline, exactly where each device's MMIO window and IRQ are.

rg -n "virtio_mmio.device|virtio_mmio\.device|add_virtio_mmio|mmio.*cmdline|fdt.*virtio|device=.*@.*:" \
  src/vmm/src/ | head -40

The format is virtio_mmio.device=SIZE@ADDR:IRQ, e.g. virtio_mmio.device=4K@0xd0000000:5. Record the mapping:

On aarch64, there is no cmdline advertisement for this — instead Firecracker writes a virtio_mmio node into the FDT, with reg (address/size) and interrupts properties. Find it:

rg -n "virtio_mmio|FdtWriter|property|reg|interrupts" src/vmm/src/arch/aarch64/

Record the contrast: x86 advertises via the cmdline string; aarch64 advertises via FDT nodes. Same information, different transport, because the discovery mechanism differs by architecture. This is why Lab 6.2's virtio_mmio.device= cmdline entries only appear on x86. See the virtio-MMIO transport deep dive and (for the PCI alternative) note that --enable-pci changes discovery to PCI enumeration entirely.

Step 7 (optional build-it): thread a harmless custom parameter through the config path

To prove you understand the path, add one harmless cmdline token that Firecracker always appends — not via boot_args, but by inserting it where the cmdline is built. Pick something the kernel ignores, e.g. fc.curriculum=l6.

# Locate the place where the cmdline is assembled (Step 5) and where Firecracker already inserts tokens.
rg -n "\.insert\(|\.insert_str\(|cmdline" src/vmm/src/builder.rs

Add a single insert next to the existing ones, for example (adapt to the real API on your branch — verify the method signature with cargo doc -p linux-loader):

#![allow(unused)]
fn main() {
// In the boot cmdline assembly, alongside Firecracker's own inserts.
// (Names/signatures vary by branch — confirm against linux-loader's Cmdline API.)
cmdline
    .insert("fc.curriculum", "l6")
    .map_err(/* the existing cmdline error variant */)?;
}

tools/devtool build --debug      # rebuild
# boot as in Step 1, then in the guest:
cat /proc/cmdline                 # should now ALWAYS contain fc.curriculum=l6

Record: you changed construction, not configuration — every microVM this build boots now carries the token, regardless of boot_args. Then revert it (git checkout -- src/vmm/src/builder.rs); this was a learning probe, not a real feature. The lesson is the seam: the difference between user-supplied boot_args and VMM-appended machinery, and exactly where in the code that seam is.

Tip: This is a real contribution shape in miniature. A genuine PR here would add a validated, documented, tested parameter with a clear reason — not a marker. But the mechanics are identical, and you've now done them.

Step 8: Verify the full round trip

Tie config to code to guest in one observation:

# Config (API) → what you asked for:
curl -s --unix-socket $API http://localhost/boot-source 2>/dev/null | jq . || true
# Guest (/proc/cmdline) → what the kernel got:    [run inside the guest]
cat /proc/cmdline

The guest's /proc/cmdline should equal: your boot_args, plus Firecracker's appended virtio_mmio.device= entries (x86), plus any default/inserted tokens — exactly the Cmdline you read being assembled in Steps 5–6, written at CMDLINE_START, found via the zero page. Round trip complete.

Implementation Requirements / Deliverables

• The default/baseline cmdline captured from the kernel's Command line: line, with each token explained (Steps 1–2).
• A prediction-vs-observation table for at least four boot_args changes (Step 3), including the fc.lab=hello round-trip via /proc/cmdline.
• An initrd boot with the kernel's "unpack initramfs" acknowledgment, demonstrating the second zero-page pointer (Step 4).
• A reading note on linux-loader's Cmdline: where it's created, where boot_args is inserted, the length bound, and where it's written to CMDLINE_START (Step 5).
• A reading note on device advertisement: the virtio_mmio.device=SIZE@ADDR:IRQ format on x86 and the FDT-node equivalent on aarch64, with the field-to-source mapping (Step 6).
• (Optional build-it) a harmless appended token observed in /proc/cmdline and then reverted (Step 7).

Troubleshooting

No Command line: line in the serial output

The console isn't wired. Ensure boot_args contains console=ttyS0 and you're reading Firecracker's stdout / the serial log. If you removed console=ttyS0 for Step 3, that's expected — the boot is silent.

InstanceStart returns 400 after editing boot_args

A malformed boot_args (e.g. an unescaped/invalid token, or one exceeding the cmdline length bound) is rejected. Read the error body; check the length against the bound you found in Step 5 (rg -n "TooLarge|CMDLINE_MAX|capacity" src/vmm/src/).

The guest panics and never returns to a prompt

You likely set panic=0 (Step 3) and hit a panic, or init=/bin/sh against a rootfs without /bin/sh. Kill firecracker, restore reboot=k panic=1, and use a valid init.

My custom param (Step 7) doesn't appear in /proc/cmdline

Either the insert is on a code path the boot didn't take, the build didn't pick up your change (rebuild), or the insert errored and was swallowed. Add a log::debug! next to the insert and confirm it runs; check the method's Result.

virtio_mmio.device= entries are missing on the cmdline

You may be on --enable-pci (PCI enumeration replaces cmdline advertisement) or on aarch64 (FDT nodes instead). Confirm the transport and architecture.

Expected Output

A guest /proc/cmdline that decomposes cleanly into its three origins:

console=ttyS0 reboot=k panic=1   fc.lab=hello   virtio_mmio.device=4K@0xd0000000:5 virtio_mmio.device=4K@0xd0001000:6
└────────── your boot_args ──────┘ └ your marker ┘ └──────── Firecracker-appended device advertisements ────────────┘

Each segment is traceable: the first to the API JSON you sent, the marker to your insert (or boot_args), the virtio_mmio.device= entries to the device manager's window allocation (Lab 6.2, Step 5) appended by the cmdline builder (Step 6).

Stretch Goals

1. Map every appended token to its source. For a fully-configured microVM (block + net + vsock), diff the kernel's Command line: against your boot_args and account for every extra token by finding the code that inserts it.
2. Hit the length limit. Construct a boot_args long enough to exceed the cmdline bound and confirm the API rejects it with the error you found in Step 5. Then read how the bound relates to the layout (the cmdline region size).
3. aarch64 FDT advertisement. On aarch64 hardware, dump the guest FDT (/sys/firmware/fdt or dtc) and find the virtio_mmio@... nodes with their reg/interrupts — the aarch64 equivalent of the x86 cmdline entries.
4. PCI transport. If your branch supports --enable-pci, boot with it and observe that the virtio_mmio.device= entries disappear (the guest enumerates PCI instead). Note that the CVE-flagged PCI transport is newer (verify your version is patched).
5. A real, mergeable cmdline change. Sketch (don't necessarily submit) what a legitimate PR adding a validated boot-source field would need: the config struct change, validation, the cmdline insert, an integration test asserting /proc/cmdline, docs, and a CHANGELOG entry. This is the Level 6 graduate PR profile.

Validation / Self-check

Answer without notes; these gate completion:

1. What is Firecracker's conventional default cmdline, and what does each token do? Why reboot=k and panic=1 specifically?
2. Trace one boot_args token from the API JSON to /proc/cmdline inside the guest, naming every place it lives (config struct → Cmdline → CMDLINE_START → zero page → guest).
3. The kernel's printed Command line: is longer than your boot_args. What did Firecracker append, and why is it required (not optional)?
4. What is the format of a virtio-MMIO device advertisement on x86, and what does each of its three fields come from?
5. How does aarch64 advertise the same devices, and why is the mechanism different?
6. What is the second pointer the zero page carries (besides the cmdline), and how did you exercise it?
7. In Step 7 you appended a token in the cmdline construction rather than boot_args. What is the practical difference between those two seams, and which one a real feature would use?

When you can decompose a guest's /proc/cmdline into its three origins and explain how each token got there, you have completed Level 6. Proceed to Level 7: The Virtio Device Model, and consolidate with the boot sequence deep dive, the guest memory management deep dive, and the linux-loader chapter.