Lab 1: Trace the Kernel Load

Background

"Firecracker loads an uncompressed vmlinux ELF and jumps to e_entry." You have read that sentence in the boot-sequence deep dive and the intensive index. It hides three distinct operations, and this lab pulls them apart:

1. Parse. An ELF file has a header (e_entry, e_phoff, …) and a program header table of segments. Only PT_LOAD segments contain bytes that must be placed in memory; each says what file offset to read, how many bytes, where in physical memory it goes (p_paddr), and how much zero-fill to add (p_memsz - p_filesz, for .bss).
2. Place. Firecracker (via rust-vmm's linux-loader Elf loader) walks those PT_LOAD segments and copies each one into the GuestMemoryMmap at the address the segment dictates, then records the kernel's entry point.
3. Enter. That recorded entry becomes the rip of the first vCPU. The bytes you copied at e_entry's physical address are the first instruction the guest ever executes.

This is a trace-it lab. You will inspect a real vmlinux with readelf, read Firecracker's loader and arch/ placement code with rg, optionally instrument the load with a printf-style log, and finish by tying a specific byte on disk to a specific guest-physical address to the rip the vCPU starts at. Nothing here is hand-waved — you will have the addresses.

Why This Lab Matters for Contributors

• A boot that dies before the first console line almost always died in load, layout, or long-mode setup — the three things this lab and the KVM intensive's long-mode lab make concrete. You cannot debug "the guest never prints anything" if "load the kernel" is a black box to you.
• The kernel load path crosses the rust-vmm boundary into linux-loader. The linux-loader chapter and Lab R2 (load a kernel) are where you'll have driven that crate directly; here you see Firecracker's production use of it. Contributors who touch boot routinely touch or upgrade linux-loader.
• The placement is governed by layout constants (HIMEM_START, the load address) that are a real compatibility surface — change where the kernel loads and you can break the boot protocol or the snapshot memory layout. Knowing exactly what those constants control is the difference between a safe change and a regression.

Prerequisites

• Level 6 and the boot-sequence deep dive.
• The KVM intensive Lab 1: you have hand-set rip and watched a vCPU start at it. The kernel's e_entry is the same idea at scale.
• A Firecracker checkout, a built firecracker, and a CI vmlinux (an uncompressed kernel — readelf must recognize it as ELF).

cd ~/firecracker
# You need an UNCOMPRESSED vmlinux ELF. A bzImage will NOT show ELF program headers.
KERNEL=$(ls vmlinux-* 2>/dev/null | head -1); echo "using $KERNEL"
file "$KERNEL"                       # must say: ELF 64-bit LSB executable, x86-64
readelf -h "$KERNEL" | grep -E "Type|Machine|Entry"
rg -q "fn build_microvm_for_boot" src/vmm/src/builder.rs && echo "builder present"

Note: If file says "bzImage" or "gzip compressed", you have the wrong artifact. Firecracker on x86 wants the uncompressed vmlinux ELF. The CI kernels from the spec.ccfc.min bucket (vmlinux-X.Y.Z) are uncompressed — that is what Lab 1.3 had you download.

Step-by-Step Tasks

Step 1: Read the ELF header — find the entry point

The ELF header is 64 bytes at the start of the file. The field that matters most for boot is e_entry: the virtual address of the first instruction.

readelf -h "$KERNEL"

ELF Header:
  ...
  Type:                              EXEC (Executable file)
  Machine:                           Advanced Micro Devices X86-64
  Entry point address:               0x1000000          <-- e_entry (example; YOURS WILL DIFFER)
  Start of program headers:          64 (bytes into file)
  Number of program headers:         5
  ...

Write down your Entry point address. That number is e_entry. By the end of this lab you will have followed it all the way to the vCPU's rip.

Note: The exact e_entry value depends on how the kernel was linked (CONFIG_PHYSICAL_START, relocation). Do not assume 0x1000000 — read yours. The relationship (rip = e_entry) is invariant; the value is not.

Step 2: Read the program headers — find the PT_LOAD segments

readelf -l prints the program header table. The LOAD entries are the ones Firecracker copies into guest memory.

readelf -l "$KERNEL"

Program Headers:
  Type   Offset             VirtAddr           PhysAddr           FileSiz   MemSiz    Flg Align
  LOAD   0x0000000000200000 0xffffffff81000000 0x0000000001000000 0x... 0x... R E 0x200000
  LOAD   0x0000000001000000 0xffffffff82000000 0x0000000002000000 0x... 0x... RW  0x200000
  ...
  NOTE   ...

Read these columns carefully — they are the whole story of placement:

The critical distinction: the kernel is linked at a high virtual address (0xffffffff8...) but physically loaded low, near 0x1000000. The loader places bytes by physical address. The high virtual addresses become real only once the kernel installs its own page tables — until then it runs identity-mapped (see the long-mode setup from the KVM intensive).

# Pull just the LOAD lines and their phys addresses for your kernel:
readelf -l "$KERNEL" | awk '/LOAD/{getline addr; print} /LOAD/'
# Or, more robustly, the wide form:
readelf -lW "$KERNEL" | grep -E "LOAD"

Step 3: Find Firecracker's loader call — the linux-loader Elf loader

Now the source side. Firecracker does not parse ELF itself; it calls linux-loader. Locate the call — by role, not by a line number that will have moved:

cd ~/firecracker
# The kernel-loading entry. linux-loader exposes a loader keyed off the kernel type;
# for vmlinux it's the Elf loader.
rg -n "Elf|load_kernel|KernelLoader|Loader::load|load_cmdline|kernel_entry|entry_addr" src/vmm/src/

# Narrow to where the ELF loader is actually invoked and the entry recorded:
rg -n "linux_loader|loader::|Elf::load|::load\(" src/vmm/src/

You are looking for the place that:

1. opens the kernel file,
2. calls the linux-loader Elf loader with the GuestMemoryMmap and a load offset, and
3. captures the returned KernelLoaderResult (which carries kernel_load address and the entry address).

# The loader result type and the entry/load fields it returns:
rg -n "KernelLoaderResult|kernel_load|kernel_end|entry_addr|setup_header" src/vmm/src/

Note (anti-staleness): The exact function name has been load_kernel, load_kernel_image, or a method on a config struct across releases, and the load address may be passed as HIMEM_START or computed. Run the rg and read what your branch does. The shape is invariant: file → linux-loader Elf loader → KernelLoaderResult{ entry_addr, .. }.

Step 4: Find the load address constant and confirm it matches the ELF

Firecracker passes a load offset into the loader. On x86 that is HIMEM_START (1 MiB), and the kernel's p_paddr values sit at or above it.

rg -n "HIMEM_START|GUEST_MEM_START|kernel_load|load_addr" src/vmm/src/arch/x86_64/ src/vmm/src/
sed -n '1,80p' src/vmm/src/arch/x86_64/layout.rs   # read the constants in context

Cross-check: your kernel's lowest PT_LOAD PhysAddr from Step 2 should be >= HIMEM_START. If your e_entry was 0x1000000 (16 MiB), it is comfortably above the 1 MiB watermark, and the low region (0x0–0x100000) is reserved for the boot params, cmdline, page tables, and the real-mode legacy area Firecracker doesn't use for the image.

Step 5: Tie e_entry to the first vCPU's rip

The kernel's entry address — returned by the loader as entry_addr — is what Firecracker writes into the vCPU's rip before the first KVM_RUN. Find that hand-off:

# Where the entry address becomes rip (and rsi becomes the zero-page address):
rg -n "rip|entry|setup_regs|configure\b|RIP|kernel_entry" src/vmm/src/arch/x86_64/regs.rs src/vmm/src/arch/x86_64/
rg -n "entry_addr|kernel_load|\.rip\b" src/vmm/src/builder.rs src/vmm/src/arch/x86_64/

You should find a path where the loader's entry address flows into the register setup that sets regs.rip = <entry> and regs.rsi = ZERO_PAGE_START. This closes the loop you opened in the KVM intensive, where you set regs.rip by hand:

  vmlinux file              GuestMemoryMmap                 vCPU
  ┌───────────────┐  copy   ┌───────────────────┐  rip =   ┌──────────┐
  │ PT_LOAD @off  │ ──────► │ bytes @ p_paddr    │ ──────►  │ rip=e_entry
  │  ...          │         │  ...               │  entry   │ rsi=ZERO_PAGE
  │ e_entry=0x... │         │ first insn @e_entry│          │ (long mode)
  └───────────────┘         └───────────────────┘          └──────────┘
        Step 2                    Steps 3–4                   Step 5

Step 6: Disassemble the entry point — see the first instruction

You can now show the actual first instruction the guest runs. Disassemble the ELF starting at e_entry:

# Disassemble around the entry point. Replace 0x1000000 with YOUR e_entry.
ENTRY=$(readelf -h "$KERNEL" | awk '/Entry point/{print $4}')
echo "entry = $ENTRY"
objdump -d --start-address=$ENTRY --stop-address=$((ENTRY + 0x40)) "$KERNEL" | sed -n '1,40p'

For a 64-bit vmlinux, the entry lands in the kernel's 64-bit startup (startup_64 / secondary_startup_64 lineage). The first instructions assume exactly the machine state Firecracker set up: long mode on, identity-mapped low memory, rsi pointing at boot_params. If long mode were not on (the failure you induced in the KVM intensive), these very bytes would decode as garbage and the vCPU would Shutdown.

Step 7 (optional, powerful): Instrument the load

Make the placement observable. Add a temporary log right after the loader returns, then rebuild and boot.

# Find the loader-return site again and add a log line there. Example shape —
# adapt to the real variable names you found in Step 3:
rg -n "KernelLoaderResult|entry_addr|kernel_load" src/vmm/src/

#![allow(unused)]
fn main() {
// Temporary instrumentation — REMOVE before any commit. Place it right after the
// linux-loader call returns its result (names per your branch):
log::info!(
    "KERNEL LOAD: entry_addr={:#x} kernel_load={:#x}",
    loader_result.kernel_end,   // or .entry_addr / .kernel_load — match your branch
    0
);
}

tools/devtool build
# Boot a microVM (Lab 1.3) with the logger enabled and grep your line:
#   curl ... /logger {"log_path":"/tmp/fc.log","level":"Info"}
grep "KERNEL LOAD" /tmp/fc.log

The entry_addr your log prints must equal the e_entry from readelf -h in Step 1. If it does, you have proven, end to end, that the bytes on disk become the vCPU's first instruction at the address the ELF declared.

Warning: This is a temporary debugging change. Do not commit it. tools/devtool checkstyle and clippy (-D warnings) will reject stray log:: calls anyway, but the discipline matters: instrument to learn, then revert.

Implementation Requirements / Deliverables

• The e_entry, every PT_LOAD segment's Offset/PhysAddr/FileSiz/MemSiz for your kernel, written down from readelf.
• A one-paragraph explanation of why VirtAddr is high-half but PhysAddr is low, and which one drives placement.
• The rg output locating (a) the linux-loader Elf load call, (b) the HIMEM_START/load-address constant, and (c) where the entry address becomes regs.rip — each with the file noted.
• A disassembly of the first ~16 bytes at e_entry, with a sentence on what machine state those bytes assume.
• (Optional) The instrumented entry_addr log line matching readelf -h's entry, then reverted.

Troubleshooting

readelf -l shows no program headers / "not an ELF file"

You have a compressed kernel (bzImage) or a gzip blob, not the uncompressed vmlinux. file "$KERNEL" will confirm. Get the uncompressed CI kernel (vmlinux-X.Y.Z) per Lab 1.3.

The rg for load_kernel finds nothing

The function has a different name on your branch (the crate merge and refactors renamed boot routines). Broaden the search: rg -n "linux_loader|Elf|::load\(|KernelLoaderResult|entry_addr" src/vmm/src/. Follow KernelLoaderResult — it is the loader's return type and a stable anchor.

My e_entry is 0xffffffff8... (a virtual address)

Some kernels report a high-half e_entry. Firecracker/linux-loader handle the virt→phys mapping for the entry as part of the x86 boot protocol; what reaches rip is the physical entry. Read how your branch derives the physical entry from the loader result, and don't assume — rg -n "entry_addr|GuestAddress|to_phys" src/vmm/src/arch/x86_64/.

objdump disassembles nonsense at the entry

You passed the wrong start address (decimal vs hex) or a stripped kernel. Ensure $ENTRY is hex (0x...) and that the kernel is not stripped of its text.

Expected Output

$ readelf -h "$KERNEL" | grep Entry
  Entry point address:               0x1000000

$ readelf -lW "$KERNEL" | grep LOAD
  LOAD  0x200000  0xffffffff81000000  0x0000000001000000  0x0e8e000 0x0e8e000 R E 0x200000
  LOAD  ...                           0x0000000001e8e000  0x...     0x...     RW  0x200000

# After instrumentation + boot:
$ grep "KERNEL LOAD" /tmp/fc.log
... KERNEL LOAD: entry_addr=0x1000000 kernel_load=0x...

The headline: entry_addr from the log == Entry point address from readelf. That equality is the lab — the file declared an entry, the loader honored it, and the vCPU started there.

Stretch Goals

1. The initrd, too. If you configured an initrd/initramfs (Lab 6.3), find where it is loaded (rg -n "initrd|load_initrd|InitrdConfig" src/vmm/src/) and where its address/size are recorded for the boot params. You'll write the pointer to it into the zero page in Lab 2.
2. Drive the loader yourself. In a scratch Rust binary, depend on linux-loader and vm-memory, build a GuestMemoryMmap, and call the Elf loader on your vmlinux. Print the KernelLoaderResult. This is exactly rust-vmm Lab R2 — do it now with the kernel you just dissected and compare the entry to readelf.
3. Break placement on purpose. In your scratch binary, pass a load offset that collides with the zero-page region (< 0x100000) and observe the loader error or the resulting boot failure. Understanding what enforces the layout is as valuable as knowing the layout.
4. bzImage vs vmlinux. Read how linux-loader distinguishes the Elf loader from the bzImage loader and which one Firecracker selects (rg -n "bzImage\|BzImage\|Elf\|PE\|loader" src/vmm/src/). Why does Firecracker prefer uncompressed vmlinux for boot time? (Hint: no decompression step.)

Validation / Self-check

Answer without notes. These gate completion.

1. What is e_entry, where do you read it, and what register does it end up in before the first KVM_RUN?
2. For a PT_LOAD segment, what do Offset, PhysAddr, FileSiz, and MemSiz each tell the loader to do? What happens to the MemSiz - FileSiz bytes?
3. Why is the kernel linked at a high virtual address but loaded at a low physical address, and which of the two governs where the bytes actually land?
4. Which rust-vmm crate parses the ELF, and what type carries the entry address back to Firecracker?
5. What does HIMEM_START control, and how would you confirm a given kernel's load segments respect it?
6. Trace one byte: pick the byte at e_entry. Where is it in the file (offset), where is it in guest memory (phys addr), and how does the vCPU come to execute it first?
7. If long mode were not set up, what would happen when the vCPU executed the bytes at e_entry, and why?

When you can place every PT_LOAD segment by guest-physical address from a readelf dump, locate the loader and the rip hand-off in the source, and prove entry_addr == e_entry by instrumentation, you've completed Lab 1. Continue to Lab 2 — The Zero Page and the e820 Map, where you dissect what the kernel reads from the address in rsi.