Firecracker Capstone Project

The Capstone is the bridge from "I have read the Firecracker codebase" to "I have shipped a fix that two AWS maintainers approved and merged into main." Everything in Levels 1–9 was preparation — building with tools/devtool, reading the vCPU run loop, tracing a VmmAction from the API socket to a device, driving a virtqueue, reasoning about seccomp and snapshots. This is the work those levels were for.

You will pick one real, open issue from github.com/firecracker-microvm/firecracker/issues, build the exact version it was reported against, reproduce it deterministically, trace the failure through the vmm/firecracker/jailer crates, identify the root cause, write a minimal idiomatic-Rust fix with a pytest integration test (and a unit test where it fits), get it green through tools/devtool checkstyle and checkbuild --all, open a Pull Request, sign every commit under the DCO (git commit -s — Firecracker has no CLA), add a CHANGELOG.md entry, survive two maintainer approvals and the review rounds that precede them, land the change, and write it up so the next engineer can learn from your investigation.

Firecracker contribution is GitHub-native and AWS-maintained. There is no JIRA, no mailing-list patch, no separate steering committee. You fork, branch, push, open a PR against main, sign off, add the CHANGELOG line, and iterate in the PR conversation until two maintainers approve and one of them merges. Keep that model in your head — every step below assumes it. The threat model assumes it too: this is privileged host code that runs untrusted guests for AWS Lambda and Fargate, so "it works on my machine" is the floor, not the goal.

This chapter is the table of contents. The ten step-chapters that follow are the work itself.

The Goal

Two artifacts, both real:

1. A merged-quality Pull Request against firecracker-microvm/firecracker:main — DCO-signed, with an integration test that is red before your fix and green after, a CHANGELOG entry, and a description a maintainer can review without asking you a single clarifying question. "Merged-quality" is the bar even if the merge itself slips past your Capstone window; review latency is outside your control, craft is not.
2. An engineering write-up — a structured postmortem (problem → investigation → root cause → fix → alternatives → testing → lessons) that demonstrates maintainer-level understanding of the subsystem you touched. This is the portfolio artifact that travels with you into a maintainer nomination or your next job. See Step 10.

One clean, small, merged-quality PR plus a sharp write-up beats an ambitious branch that never lands. There is no honor in scope; there is honor in shipping.

Prerequisites

Do not start the Capstone until you can answer "yes" to every one of these. If any is "no," stop and go back — the Capstone is unforgiving of partial preparation, and you will spend three weeks confused instead of three weeks shipping.

• Levels 1–9 complete. You can read the vCPU run loop, the API→VMM action channel, the EventManager epoll loop, a virtio device's queue handling, the builder/boot path, and the seccomp/jailer model without a guide open. If those are not familiar, return to Level 8 and Level 9 and the deep dives.
• You can build from source. tools/devtool build succeeds on your machine and produces build/cargo_target/<arch>-unknown-linux-musl/debug/firecracker (verify the arch/libc on your branch). See Lab 1.1.
• You can run the test suites. tools/devtool test runs the pytest integration harness in tests/ green (modulo known flakes), and cargo test unit tests pass. See Lab 1.2 and Lab 5.1.
• You have booted a microVM by hand. You have driven the API socket with curl through boot-source / drives / machine-config / InstanceStart and gotten a login prompt on the serial console. See Lab 1.3.
• You have DCO configured. git commit -s produces a Signed-off-by: line with your real name and the email tied to your GitHub account. The DCO bot matches that line per commit — get it right locally, once, before you ever push.
• You have read the repo hygiene files: CONTRIBUTING.md, docs/, and the CHANGELOG.md convention. You have skimmed the contributor-mindset essays on PR quality and responding to feedback.
• gh is authenticated. gh auth status is green and gh issue list works against the repo. You will use the GitHub CLI throughout instead of clicking around the web UI.

Note: Firecracker is Linux-only and needs a working /dev/kvm (ls -l /dev/kvm; on AWS use a *.metal bare-metal instance — ordinary nested EC2 will not run it). The whole Capstone is impossible without it. Confirm this before Step 1.

The 10-Step Flow

flowchart TD
    A[Step 1: Issue Selection] --> B[Step 2: Reproduction]
    B --> C[Step 3: Execution-Path Analysis]
    C --> D[Step 4: Root-Cause Identification]
    D --> E[Step 5: Implementation]
    E --> F[Step 6: Testing]
    F --> G[Step 7: Validation]
    G --> H[Step 8: PR Preparation]
    H --> I[Step 9: GitHub & Docs]
    I --> J[Step 10: Engineering Write-Up]
    G -.checkstyle/checkbuild fail.-> E
    F -.test fails wrong way.-> D
    D -.hypothesis wrong.-> C
    H -.review round.-> E
    I -.CI red on a runner.-> F

The dotted arrows are the loops you will actually run. Nobody nails root cause on the first hypothesis. Nobody passes checkbuild --all on the first try (clippy is warnings-as-errors). Nobody clears two-approval review in one round. Plan for two or three iterations through Steps 4–9 before the merge button is even reachable.

Here is the same flow as a pipeline, with the thing you are mainly reasoning about at each step — the subsystem map you built across Levels 3–9:

 ISSUE        REPRO         TRACE          ROOT CAUSE     FIX
 (gh)   ──►   (curl/        (rg + log-     (fix site vs   (minimal
              config/        instrument)    symptom site)  Rust)
              pytest)            │              │            │
                                 ▼              ▼            ▼
                    API thread · VMM thread · vCPU thread · virtio · KVM
                    seccomp · jailer · snapshot/Persist · rate limiter
                                 │
 TEST  ──►  VALIDATE  ──►  PR  ──►  GITHUB/DOCS  ──►  WRITE-UP
 (pytest    (re-repro,    (DCO,    (≥2 approvals,    (postmortem,
  integ +    perf/snap/   CHANGELOG, amend+         portfolio
  unit)      seccomp      CI)       force-push)      artifact)
             regress)

Deliverables

By the time you mark the Capstone done, every one of these artifacts exists. Track them with a checklist:

• Deterministic repro captured and confirmed
• Execution-path doc with located citations
• Root-cause statement written
• Minimal fix, one-logical-change commits, signed off
• Integration test red-before / green-after
• Unit test where reachable
• Validation report: re-repro + green checkstyle/checkbuild + no regressions
• CHANGELOG entry under the correct heading
• PR opened, issue linked, reviewers requested
• Write-up published

Every one. The write-up is not optional — it is how the community (and your future self) learns from the investigation.

How It's Evaluated

The full 100-point rubric lives in evaluation-rubric.md. It is calibrated to what Firecracker maintainers actually look for on a PR, not to what feels good to read. Headline weighting:

Tier thresholds: 80+ = a credible Firecracker contributor who can sustain a PR flow; 90+ = maintainer-ready work a MAINTAINERS.md reviewer would do without hand-holding; 95+ = the quality that, sustained over months, gets you noticed by the team. You self-grade in Step 10 — honestly; inflated self-grades are visible from orbit the moment a maintainer reads your PR.

Timeline

The Capstone is a ~2–3 week effort at one focused evening per weekday plus weekend mornings. Less continuity than that and you lose the execution-path context between sessions — which is expensive for KVM, virtio, and snapshot code.

The phases overlap on purpose. If you blow past three weeks, that is a signal, not a failure: either the issue is bigger than it looked (renegotiate scope in the thread) or you are stuck on one step (ask in the community channels, or re-read the step chapter). The ≥2-approval wait is the one part you cannot compress — plan the write-up and your next issue search to fill it.

Success and Failure Signals

You will know it is working when:

1. A maintainer comments approval and CI on the PR is green across the Buildkite matrix (both architectures).
2. A second maintainer approves, and one of them merges — your Signed-off-by: line appears in git log origin/main.
3. The issue closes via the linked PR, with your name on the merge.
4. Your write-up gets traffic — someone hitting the same symptom finds it.
5. The next issue you pick reaches root cause in days, not weeks.

You will know it is failing when:

1. You are still editing files in Step 5 with no failing repro from Step 2 in hand.
2. Your PR description says "I think this fixes it."
3. You have not run checkbuild --all in over a week.
4. You are arguing in PR comments instead of changing code or asking a sharp question.
5. CI has been red on one runner for days and you are pushing blind "fix CI" commits instead of reproducing locally.

If you spot a failure signal, do not push through. Stop, re-read the relevant step, and reset.

How to Use This Chapter

Read all ten step-chapters once, end to end, before you start Step 1. You need the shape of the whole journey in your head: Step 4 (root cause) makes choices that Step 6 (testing) depends on; Step 8 (PR) assumes artifacts from Steps 2 and 7; Step 9 assumes your tests cleared CI. Skim now, deep-read each as you arrive.

Then go to Step 1: Issue Selection. The clock starts when you comment "I'd like to work on this" on the GitHub issue.

Validation / Self-check

Before starting Step 1, confirm:

1. You can produce, from memory, the role and rough location of Vmm, VmResources, VmmAction, the builder boot fns, MMIODeviceManager, and the EventManager loop — and the rg you would run to find each.
2. tools/devtool build and tools/devtool test both succeed on your clone.
3. You have booted a microVM by hand and seen a login prompt.
4. git commit -s produces a correct Signed-off-by: line with your GitHub email.
5. gh auth status is green and gh issue list --repo firecracker-microvm/firecracker returns issues.
6. You have a capstone-work/ directory ready for repro.md, execution-path.md, root-cause.md, and validation.md.
7. You have skimmed every step-chapter once and set aside 2–3 calendar weeks.

Then go to Step 1: Issue Selection. Pick the issue.