Lab R2: Load a Kernel with linux-loader

Background

Lab R1 ran 16 bytes of hand-assembled real-mode code. A real microVM runs an entire Linux kernel. The leap from one to the other is the single most arch-heavy part of any VMM, and it has two halves: (1) parse an uncompressed vmlinux ELF and copy its PT_LOAD segments into guest memory at the right physical addresses, and (2) build the boot protocol structures — the kernel command line, the e820 memory map, and the boot_params "zero page" — and write them into guest memory so the kernel knows what hardware it has and where its cmdline is. Get either half wrong and the kernel either refuses to boot or panics in the first microsecond.

rust-vmm packages both halves in one crate, linux-loader, and Firecracker uses it verbatim. loader::Elf::load does half (1); the configurator module (LinuxBootConfigurator, BootParams) and the cmdline module (Cmdline) do half (2). In this lab you build a standalone program that performs the entire boot preparation — load a real vmlinux, build a cmdline, write a zero page with an e820 map — and then prints the resulting memory layout: where the kernel landed, its entry point, where the cmdline and zero page sit. You do not have to boot it (booting is the stretch goal). The deliverable is that you can answer "what is in guest memory the instant before the first KVM_RUN?" — which is exactly the question Firecracker's build_microvm_for_boot answers.

Note — be honest about the hard part. The genuinely fiddly thing in this lab is getting a matching vmlinux. linux-loader's Elf loader wants an uncompressed ELF vmlinux, not a bzImage, not a vmlinuz, not a distro kernel. Step 1 is dedicated to obtaining one, and the cleanest source is the same place Firecracker's CI gets its test kernels.

Why This Lab Matters for Contributors

• The boot sequence deep dive and guest memory layout deep dive describe the zero page, e820, CMDLINE_START, and HIMEM_START. This lab makes those constants concrete by writing bytes at them.
• Firecracker's entire x86_64 boot path (src/vmm/src/arch/x86_64/, builder.rs) is "call linux-loader, then patch the boot_params for our specific devices." You cannot review a change to that code without having driven linux-loader yourself.
• The linux-loader chapter is the API tour; this is the lab that wires it up and shows you the gap between "ELF loaded" and "kernel actually boots" (the regs/long-mode setup Firecracker adds).
• When a boot-config issue lands — wrong e820, truncated cmdline, bad load address — you will recognize the symptom because you produced each structure by hand here.

Prerequisites

• Lab R1 complete: you can build a KVM VM and register guest memory.
• A Rust toolchain and an x86_64 Linux host. (You only need KVM for the boot stretch goal; the core lab — load + layout-print — runs anywhere x86_64 Linux.)
• A Firecracker checkout to rg against.

rustc --version
git clone https://github.com/firecracker-microvm/firecracker.git ~/firecracker 2>/dev/null || true
# See exactly how Firecracker calls the loader you're about to use:
rg -n "linux_loader::|Elf::load|loader::Elf|LinuxBootConfigurator|Cmdline" ~/firecracker/src/vmm/src/

Step-by-Step Tasks

Step 1: Get a real uncompressed vmlinux (the hard part)

linux-loader's Elf loader needs an uncompressed ELF kernel image. The simplest reliable source is a Firecracker CI test kernel — these are plain vmlinux-X.Y.Z ELF files built exactly for this loader. Fetch one and confirm its type:

mkdir -p ~/kvm-kernel-lab && cd ~/kvm-kernel-lab
# Firecracker CI kernels live in the spec.ccfc.min S3 bucket; the exact key/version
# changes over time — get the current path from docs/getting-started.md in your checkout.
rg -n "spec.ccfc.min|vmlinux-" ~/firecracker/docs/getting-started.md
# Example (verify the version/URL against that doc; do NOT trust this literal):
curl -fsSLO "https://s3.amazonaws.com/spec.ccfc.min/firecracker-ci/v1.11/x86_64/vmlinux-6.1.128"
file vmlinux-6.1.128
#  -> ELF 64-bit LSB executable, x86-64, ... statically linked, ...   ← MUST say ELF, not bzImage

Warning: If file reports Linux kernel x86 boot executable bzImage you have the wrong artifact — that is the compressed, self-extracting format. linux-loader's Elf loader will reject it. You need the uncompressed vmlinux ELF. Building your own: make vmlinux in a kernel tree produces vmlinux at the root (large, with symbols); the CI kernel above is the no-fuss path. (verify the current CI kernel version/URL — it moves.)

Step 2: Create the project and pin the crates

cd ~/kvm-kernel-lab && cargo new --bin kernel-load-lab && cd kernel-load-lab

Cargo.toml:

[package]
name = "kernel-load-lab"
version = "0.1.0"
edition = "2021"

[dependencies]
# Versions current to mid-2026 — verify on crates.io and match your Firecracker Cargo.lock.
linux-loader = { version = "0.13", features = ["elf", "bzimage"] }
vm-memory = { version = "0.17", features = ["backend-mmap"] }
# Only needed for the boot stretch goal:
kvm-ioctls = "0.25"
kvm-bindings = "0.14"

Note: linux-loader re-exports the bootparam C structs (the zero page header) so you do not add a separate crate for them. The elf feature enables loader::Elf; backend-mmap on vm-memory gives you GuestMemoryMmap, the same type Firecracker uses for guest RAM. (verify feature names on docs.rs — they occasionally change.)

Step 3: Understand the x86_64 boot layout you are about to build

Firecracker's layout constants live in src/vmm/src/arch/x86_64/layout.rs (verify on your branch — they drift). The ones this lab uses:

guest physical address
  0x0000_0000 ┌──────────────────────────────┐
              │ real-mode IVT / low scratch   │
  0x0000_7000 │ ZERO_PAGE_START (boot_params) │  ← the "zero page": e820, cmdline ptr, etc.
  0x0000_9000 │ ... (page tables, boot stack) │
  0x0002_0000 │ CMDLINE_START (kernel cmdline)│  ← NUL-terminated cmdline string
  0x0010_0000 │ HIMEM_START (1 MiB)           │  ← uncompressed vmlinux loads here (e_entry ~here)
              │ ... kernel image + guest RAM  │
              └──────────────────────────────┘

The boot contract on x86_64: the VMM loads the kernel high (≥1 MiB), writes a filled-in boot_params ("zero page") somewhere low, and on first KVM_RUN sets rsi = ZERO_PAGE_START and rip = e_entry, in 64-bit long mode. The kernel reads boot_params through rsi. Your job in this lab: produce the kernel image in memory, the cmdline, and the zero page. (Setting rsi/long mode is the boot stretch goal — that part is pure Lab R1 mechanics.)

Step 4: Write the loader program

Replace src/main.rs. This loads the kernel, builds the cmdline, builds and writes the zero page, and prints the full layout.

use linux_loader::cmdline::Cmdline;
use linux_loader::configurator::linux::x86_64::LinuxBootConfigurator;
use linux_loader::configurator::{BootConfigurator, BootParams};
use linux_loader::loader::bootparam::boot_params;
use linux_loader::loader::{Elf, KernelLoader};
use std::fs::File;
use vm_memory::{Address, GuestAddress, GuestMemory, GuestMemoryMmap};

// Firecracker's x86_64 layout constants (verify on your branch in arch/x86_64/layout.rs).
const ZERO_PAGE_START: u64 = 0x7000;
const CMDLINE_START: u64 = 0x2_0000;
const HIMEM_START: u64 = 0x10_0000; // 1 MiB
const MEM_SIZE: usize = 256 << 20; // 256 MiB of guest RAM

// e820 region types from the boot protocol.
const E820_RAM: u32 = 1;
const EBDA_START: u64 = 0x9_FC00; // top of usable low RAM before the BIOS area

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let kernel_path = std::env::args()
        .nth(1)
        .expect("usage: kernel-load-lab <path-to-vmlinux>");

    // 1. Build guest memory: one region [0, MEM_SIZE). Same type Firecracker uses.
    let guest_mem = GuestMemoryMmap::from_ranges(&[(GuestAddress(0), MEM_SIZE)])?;

    // 2. Load the vmlinux ELF. The loader parses PT_LOAD segments and copies them in.
    //    kernel_offset = None  -> load at the ELF's own p_paddr (typically 1 MiB).
    let mut kernel_file = File::open(&kernel_path)?;
    let load_result = Elf::load(
        &guest_mem,
        None,                            // kernel_offset: use the ELF's physical addresses
        &mut kernel_file,                // the image (Read + Seek)
        Some(GuestAddress(HIMEM_START)), // highmem_start_address
    )?;

    // 3. Build the kernel command line and write the string into guest memory.
    let mut cmdline = Cmdline::new(0x1_0000)?; // 64 KiB capacity
    cmdline.insert_str("console=ttyS0 reboot=k panic=1 pci=off nomodule")?;
    let cmdline_cstr = cmdline.as_cstring()?;
    let cmdline_bytes = cmdline_cstr.as_bytes_with_nul();
    guest_mem.write_slice(cmdline_bytes, GuestAddress(CMDLINE_START))?;

    // 4. Build the boot_params "zero page". Start zeroed, then fill the fields the
    //    kernel requires: the magic header, cmdline pointer, and the e820 map.
    let mut params = boot_params::default();
    // Header magic: hdr.type_of_loader is conventionally 0xff for "unknown/undefined" loaders.
    params.hdr.type_of_loader = 0xff;
    // Tell the kernel where its command line is.
    params.hdr.cmd_line_ptr = CMDLINE_START as u32;
    params.hdr.cmdline_size = cmdline_bytes.len() as u32;

    // 5. Build the e820 memory map: low RAM [0, EBDA) and high RAM [1 MiB, end).
    add_e820_entry(&mut params, 0, EBDA_START, E820_RAM);
    let himem_len = MEM_SIZE as u64 - HIMEM_START;
    add_e820_entry(&mut params, HIMEM_START, himem_len, E820_RAM);

    // 6. Hand the filled boot_params to the configurator, which writes the zero page
    //    (and the e820 table) into guest memory at ZERO_PAGE_START.
    let boot = BootParams::new(&params, GuestAddress(ZERO_PAGE_START));
    LinuxBootConfigurator::write_bootparams(&boot, &guest_mem)?;

    // 7. Print the layout — the whole point of the lab.
    println!("=== microVM boot preparation complete ===");
    println!("guest RAM           : [0x0, {:#x})  ({} MiB)", MEM_SIZE, MEM_SIZE >> 20);
    println!("kernel loaded at    : {:#x}", load_result.kernel_load.raw_value());
    println!("kernel image ends   : {:#x}", load_result.kernel_end);
    println!(
        "kernel entry (rip)  : {:#x}",
        // For an ELF kernel the entry to jump to is kernel_load (e_entry); the
        // setup_header path is only populated for bzImage. Print what we have:
        load_result.kernel_load.raw_value()
    );
    println!("zero page (rsi)     : {:#x}", ZERO_PAGE_START);
    println!("cmdline at          : {:#x}  ({} bytes incl NUL)", CMDLINE_START, cmdline_bytes.len());
    println!("cmdline             : {:?}", cmdline_cstr);
    println!("e820 entries        : {}", params.e820_entries);
    for i in 0..params.e820_entries as usize {
        let e = params.e820_table[i];
        println!(
            "  e820[{i}]  addr={:#012x}  size={:#012x}  type={}",
            e.addr, e.size, e.type_
        );
    }
    println!("\nReady for first KVM_RUN: set rip={:#x}, rsi={:#x}, long mode.",
        load_result.kernel_load.raw_value(), ZERO_PAGE_START);
    Ok(())
}

/// Append one entry to the boot_params e820 table.
fn add_e820_entry(params: &mut boot_params, addr: u64, size: u64, mem_type: u32) {
    let idx = params.e820_entries as usize;
    params.e820_table[idx].addr = addr;
    params.e820_table[idx].size = size;
    params.e820_table[idx].type_ = mem_type;
    params.e820_entries += 1;
}

Step 5: Run it

cargo run -- ~/kvm-kernel-lab/vmlinux-6.1.128

Expected output (addresses depend on the kernel build; the entry near 0x100000 is the tell):

=== microVM boot preparation complete ===
guest RAM           : [0x0, 0x10000000)  (256 MiB)
kernel loaded at    : 0x1000000
kernel image ends   : 0x2a7e000
kernel entry (rip)  : 0x1000000
zero page (rsi)     : 0x7000
cmdline at          : 0x20000  (45 bytes incl NUL)
cmdline             : "console=ttyS0 reboot=k panic=1 pci=off nomodule"
e820 entries        : 2
  e820[0]  addr=0x000000000000  size=0x00000009fc00  type=1
  e820[1]  addr=0x000000100000  size=0x00000ff00000  type=1

Note: Modern x86_64 vmlinux images often have e_entry/p_paddr at 0x100_0000 (16 MiB), not exactly 0x10_0000 (1 MiB), because of the kernel's physical alignment. Read the actual kernel_load from load_result — never hard-code it. That "read it, don't assume it" discipline is the whole anti-staleness ethos.

Step 6: Map it to Firecracker

Firecracker does exactly these steps, then adds the device-specific patches and the long-mode vCPU setup. Confirm each with rg:

The crucial insight: Firecracker's arch/x86_64/ is mostly the same code you just wrote, plus two things — it appends virtio_mmio.device=SIZE@ADDR:IRQ clauses to the cmdline for each MMIO device (so the guest finds them without PCI enumeration), and it sets up long-mode page tables and the initial registers. The kernel-loading and zero-page work is linux-loader, used the same way you used it.

Implementation Requirements / Deliverables

• You obtained a genuine uncompressed vmlinux ELF (file confirms ELF, not bzImage).
• kernel-load-lab loads it and prints the layout: kernel load address, kernel end, zero-page address, cmdline address and contents, and a 2-entry e820 map.
• You can explain why the kernel loads high (≥1 MiB) and the zero page/cmdline sit low.
• You can state what rsi and rip must be at the first KVM_RUN, and where each value came from in your program.
• The Step 6 mapping table, each rg run against Firecracker, with the file noted.
• (If you did the boot stretch goal) a screenshot/log of the first serial bytes from the kernel.

Troubleshooting

Elf::load returns an error / "Invalid ELF" / wrong magic

You almost certainly handed it a bzImage or vmlinuz. Re-run file on the image — it must say ELF 64-bit. Get the uncompressed vmlinux (Step 1).

error[E0599]: no method named ... on Cmdline / boot_params

API drift between linux-loader versions. The Cmdline methods (new, insert_str, as_cstring) and the boot_params/e820_table field names are version-sensitive. cargo doc --open -p linux-loader and match the method/field names to your resolved version. (verify on docs.rs.)

Elf::load wants F: Read + Seek + ReadVolatile

Newer linux-loader adds a ReadVolatile bound to the kernel-image reader. A std::fs::File satisfies it (vm-memory implements ReadVolatile for File); if the compiler complains, ensure vm-memory is a dependency (it is) and that its version matches linux-loader's expected vm-memory. (verify the bound on your linux-loader version.)

e820 entries print as 0 / the kernel later complains about memory

add_e820_entry did not increment e820_entries, or you exceeded the fixed e820_table array size. Confirm the function increments the counter and that you add ≤ the array capacity (128 on most builds).

The boot stretch goal: kernel loads but prints nothing

The serial console is not wired, or rsi/long mode is wrong. The kernel writes early boot to COM1 (0x3f8) — you must handle VcpuExit::IoOut(0x3f8, ..) (Lab R1) and set rsi = ZERO_PAGE_START, rip = kernel_load, and full long mode (cr0 PE|PG, cr4 PAE, efer LME|LMA, identity page tables). This is genuinely involved; treat it as a multi-hour stretch, not a quick add.

Expected Output

$ cargo run -- ~/kvm-kernel-lab/vmlinux-6.1.128
    Finished `dev` profile [unoptimized + debuginfo] target(s)
     Running `target/debug/kernel-load-lab .../vmlinux-6.1.128`
=== microVM boot preparation complete ===
guest RAM           : [0x0, 0x10000000)  (256 MiB)
kernel loaded at    : 0x1000000
kernel image ends   : 0x2a7e000
kernel entry (rip)  : 0x1000000
zero page (rsi)     : 0x7000
cmdline at          : 0x20000  (45 bytes incl NUL)
cmdline             : "console=ttyS0 reboot=k panic=1 pci=off nomodule"
e820 entries        : 2
  e820[0]  addr=0x000000000000  size=0x00000009fc00  type=1
  e820[1]  addr=0x000000100000  size=0x00000ff00000  type=1

Ready for first KVM_RUN: set rip=0x1000000, rsi=0x7000, long mode.

Stretch Goals

1. Actually boot it. Combine this program with Lab R1's KVM machinery: register guest_mem with KVM_SET_USER_MEMORY_REGION, set up long mode (cr0 PE|PG, cr4 PAE, efer LME|LMA, identity-mapped page tables at a low address), set rip = kernel_load, rsi = ZERO_PAGE_START, and run. Handle VcpuExit::IoOut(0x3f8, ..) to print the kernel's serial output. Catching the kernel's first Linux version ... banner is one of the most satisfying milestones in the whole curriculum. This is essentially the masterclass trace-kernel-load lab in miniature.

2. Append a virtio MMIO clause. Add virtio_mmio.device=4K@0xd0000000:5 to the cmdline (the format Firecracker uses to tell the guest where an MMIO device lives without PCI). Print the final cmdline and note how the device location travels to the guest purely as a string.

3. Add an initrd. Load a small initramfs into high memory, set params.hdr.ramdisk_image and params.hdr.ramdisk_size, and add an e820/region for it. Confirm BootParams::set_modules (or the header fields) and print the initrd's load address. This mirrors Firecracker's initrd support.

4. Diff against Firecracker's layout. rg Firecracker's actual arch/x86_64/layout.rs constants and compare to the ones you hard-coded. Note any that differ on your branch (they drift) and adjust your program to match — then your output is byte-for-byte what Firecracker produces.

Validation / Self-check

You are done when you can answer these without notes:

1. What image format does linux-loader's Elf loader require, and how do you tell it apart from a bzImage with one command?
2. What are the three things you must place in guest memory before the first KVM_RUN, and at which addresses (kernel, cmdline, zero page)?
3. What is the "zero page," what does it contain (name three fields), and how does the kernel find it at boot?
4. What is the e820 map, why does the kernel need it, and what two regions did you add and why the gap between them?
5. At the first KVM_RUN, what must rip and rsi be, and where did each value come from in your program?
6. Which two things does Firecracker's arch/x86_64/ add on top of the linux-loader work you did here (cmdline device clauses; long-mode/register setup)?
7. Why did you read kernel_load from the loader result instead of hard-coding 0x100000?

When you can load a real kernel and lay out its boot environment in guest memory — and explain every address — you understand the boot half of a VMM. Next, learn the device half from the ground up: Lab R3: Drive a Virtqueue with virtio-queue + vm-memory.