Evaluation Rubric
A 100-point self-grading rubric for the Capstone. Score yourself honestly after you finish Step 10. The scoring is calibrated against what Firecracker maintainers actually look for on a Pull Request to privileged, untrusted-guest host code — not against what feels good to read.
The point of the rubric is not the score. It is the diagnostic: a low score on one dimension tells you exactly where to invest the next contribution.
Scoring Dimensions
Eight dimensions, weighted by how much they matter for a Firecracker review outcome. The weights match the headline table in the capstone index.
| # | Dimension | Points |
|---|---|---|
| 1 | Reproduction | 12 |
| 2 | Execution-path analysis | 18 |
| 3 | Root cause | 12 |
| 4 | Fix quality | 22 |
| 5 | Tests | 16 |
| 6 | PR craft | 12 |
| 7 | Communication | 10 |
| 8 | Write-up | 10 |
| Total | 112 → normalize to 100 |
Note: The eight band tables below sum to 112 raw points so each dimension has a clean internal scale; multiply your raw total by
100/112(≈ 0.893) for the 0–100 score the tier thresholds use. Root cause (12) is split out from execution-path analysis here for diagnostic precision; in the index's seven-row summary it is folded into execution-path analysis. Grade against these eight.
1. Reproduction (12 pts)
Deterministic, minimal, version-pinned — a maintainer can run it unchanged.
| Score | What it looks like |
|---|---|
| 11-12 | A deterministic repro that a maintainer runs verbatim: a tight curl/config-file sequence or a failing pytest, with exact versions pinned — Firecracker commit/tag, guest kernel (vmlinux-X.Y.Z), rootfs, host kernel where relevant. States the affected version range and whether it's a regression. Reproduces on a stock CI kernel + raw firecracker (the orchestrator/guest/host suspects already excluded per Lab I4). |
| 8-10 | Reproduces reliably but under-pinned (no exact kernel/rootfs version) OR larger than minimal (extra config the bug doesn't need). |
| 5-7 | Reproduces only sometimes ("intermittent" with no documented trigger), or only in your bespoke setup (custom kernel/host/orchestrator) without excluding those suspects. |
| 2-4 | A description of the symptom, not a runnable reproduction. |
| 0-1 | "It crashes sometimes." No steps, no versions. |
Look for: the word "intermittent" without a trigger; a repro that needs the contributor's exact host/kernel/orchestrator and so wasn't attributed (see Lab I4).
2. Execution-Path Analysis (18 pts)
Did you trace the code, or guess? This is the heaviest analysis dimension because
Firecracker bugs cross threads (API → VMM → vCPU) and crates (firecracker →
vmm).
| Score | What it looks like |
|---|---|
| 16-18 | execution-path.md maps the full path from entry to failure — e.g. curl → ParsedRequest → VmmAction over the mpsc channel → PrebootApiController/RuntimeApiController → builder/device — with rg-located citations by role (file + struct/fn, not fabricated line numbers) at every hop, and a mermaid/ASCII diagram with the bug node annotated. Identifies the exact point where the value flips from correct to wrong, confirmed by a log-instrument probe / serial / metric, not assumed. A reviewer could follow it without asking. |
| 13-15 | Most hops cited but one skipped ("then it reaches the device"); diagram present but missing a critical thread boundary (e.g. the API↔VMM channel or a vCPU exit). |
| 9-12 | Bug location cited correctly but no trace of how execution reached it; no diagram; observation point asserted, not confirmed. |
| 5-8 | Vague references ("the VMM handles it") with no rg-located citations. |
| 0-4 | No execution-path document, or a paragraph of prose. |
Look for: rg/find commands that locate each cited struct/fn by role (the
anti-staleness discipline), a confirmed observation point, and the thread/crate
boundaries named (API thread vs. VMM thread vs. vCPU thread).
3. Root Cause (12 pts)
Mechanism, not location; fix site distinguished from symptom site.
| Score | What it looks like |
|---|---|
| 11-12 | root-cause.md states the cause as a mechanism, distinguishes the fix site (where the bad state is created or should be rejected) from the symptom site (where it crashes), and explains why the fix site is correct. Regression status settled by git bisect/blame (introducing PR cited, or an explicit "original gap"). At least two competing hypotheses recorded as disproved, each with the one-command experiment that killed it. The four Firecracker dimensions (API/snapshot/attack-surface/perf) explicitly considered. |
| 8-10 | Correct cause and fix site, but no disproved alternatives, or regression status not investigated. |
| 5-7 | Identifies where it breaks but not why; fix site = symptom site (a band-aid). |
| 2-4 | A guess dressed as a cause, no experiments. |
| 0-1 | "Found where it crashes, added a check there." |
Look for: a fix site different from the crash site with a stated reason; a bisect result; disproved hypotheses with the deciding command.
4. Fix Quality (22 pts)
The heaviest dimension. Minimal diff, idiomatic Rust, attack-surface and compatibility discipline.
| Score | What it looks like |
|---|---|
| 20-22 | Minimum-diff fix — production change in tens of lines, not hundreds. Every changed line justifiable in one sentence; no drive-by refactor, rename, or reformat (git diff --stat shows only the predicted files). Typed errors (not panic/unwrap/expect) on every input- or guest-reachable path; guest-supplied data bounds-checked as hostile. Attack surface held or shrunk — no new device, syscall, or guest-reachable path; seccomp filters byte-identical (or deliberately, testably changed). API/snapshot compatibility preserved or versioned with a story. No new cost on the hot path. One-logical-change commits, each DCO-signed and building on its own. checkstyle + checkbuild --all green with no #[allow] papering over a real lint. |
| 16-19 | Mostly minimal but one or two stray changes; idioms mostly matched; a minor nit a reviewer would flag; fix is correct and at the right site. |
| 11-15 | Fix works but broader than necessary (scope creep), OR a new unwrap/panic on a reachable path, OR a symptom-site patch that happens to work. |
| 6-10 | Significant scope creep; widened attack surface or changed seccomp without justification; unguarded snapshot/API change. |
| 0-5 | Diff so large a reviewer asks it be split; breaks compatibility or adds a guest-reachable hazard silently. |
Look for: scope-creep tells in git diff origin/main --stat; a new unwrap/panic
on the data plane; an unexplained resources/seccomp/ diff; a symptom-site fix.
5. Tests (16 pts)
Red-before/green-after, the right layer, determinism.
| Score | What it looks like |
|---|---|
| 14-16 | A test at the lowest viable layer (unit if in-process, pytest integration if it needs a real microVM), red on main, green with the fix — verified by stashing the fix. The integration test CONTRIBUTING requires is present in the right subsystem file. A negative control proves the fix is scoped. Every trigger condition encoded; for guest-reachable bugs, multiple adversarial inputs. Deterministic (no sleep-waits, clean teardown, both arches considered). Coverage not reduced. |
| 11-13 | Test present and red-before/green-after, but no negative control, OR an integration test where a unit test would do (with a weak/absent unit test), OR coverage drops. |
| 7-10 | Test is non-deterministic (sleep-based wait, order-dependent) or incomplete in covering the trigger conditions. |
| 3-6 | Test only checks the happy path and would have passed on main. |
| 0-2 | No new tests, or tests that fail on both main and the fix. |
Look for: proof the test was stashed and went red on main; a scenario-named test
(test_negative_machine_config_zero_vcpus), not a method-named one; a condition
wait instead of a sleep.
6. PR Craft (12 pts)
DCO, CHANGELOG, description, commit hygiene, CI.
| Score | What it looks like |
|---|---|
| 11-12 | Every commit DCO-signed with a matching Signed-off-by:; DCO check green. One-logical-change commits, imperative ≤72-char titles, why-focused bodies wrapped at 72. CHANGELOG.md entry under the correct [Unreleased] heading, leading with the PR link, in the project's format. PR description links the issue (Fixes #NNNN), explains the why, and states attack-surface/snapshot/perf impact. CI green across DCO, style, build (both arches), tests. Right reviewers requested. |
| 8-10 | CHANGELOG present but wrong heading or off-format; OR a description that's thin on the why; OR CI green but reviewers not requested. |
| 5-7 | Mechanics partly followed: a missing CHANGELOG link, a stray wip commit, or one architecture untested. |
| 2-4 | DCO or CHANGELOG check red; commit history is wip/fix CI noise. |
| 0-1 | No CHANGELOG, no issue link, no sign-off. |
Look for: a Signed-off-by: on every commit; a CHANGELOG line under a heading you
can defend; a description that pre-empts the maintainer's first questions.
7. Communication (10 pts)
Claiming, review responsiveness, the ≥2-approval patience.
| Score | What it looks like |
|---|---|
| 9-10 | Commented on the issue before starting ("working on this"). Every review comment addressed in code or with a substantive reply; iterated via amend + --force-with-lease with the DCO preserved and a one-line summary after each push. CI green on every revision, not just the first. Disagreements made the technical case with evidence, without defensiveness. Waited out the ≥2-approval cycle without nagging; thanked reviewers. |
| 7-8 | Addresses comments correctly but slowly, OR lets a couple of nits slide unacknowledged, OR force-pushed without summarizing. |
| 5-6 | Defensive on at least one comment ("but my way is fine"); OR broke CI on a later push and didn't notice; OR a bare --force. |
| 2-4 | Needed multiple reminders; comments addressed sloppily; nagged the maintainers. |
| 0-1 | Worked silently then dropped a PR with no context, or argued every comment. |
Look for: an issue comment before the first push; threads resolved by a pushed
commit (not just a reply); green CI on every revision; --force-with-lease, not
--force.
8. Write-Up (10 pts)
Postmortem depth, alternatives, lessons.
| Score | What it looks like |
|---|---|
| 9-10 | A published 500–1000-word write-up following the template. The Investigation Log shows at least two ruled-out hypotheses with their disproving experiments. Root Cause distinguishes fix site from symptom site. Alternatives Considered names ≥2 rejected designs with reasons. Lessons Learned has 3–5 reusable bullets. Issue, PR, merged commit linked (plus introducing PR if a regression). Any security finding handled via private disclosure, not the post. |
| 7-8 | Solid write-up but only the winning hypothesis (no dead ends), OR no alternatives section, OR missing the merged-commit link. |
| 5-6 | A sanitized "I knew it all along" narrative; thin investigation; mechanics-only. |
| 2-4 | A few sentences; personal narrative dominating the engineering; no links. |
| 0-1 | No write-up beyond the PR description. |
Look for: honest dead ends; alternatives with reasons; three minimum links; no leaked security finding.
Tier Thresholds
Scored on the normalized 0–100 total. Where you land tells you what to do next.
| Score | Tier | Interpretation |
|---|---|---|
| 95-100 | Maintainer-track | The quality that, sustained across many contributions over months, gets you noticed by the AWS Firecracker team. You operate at the level the project would trust to steer an area. |
| 90-94 | Maintainer-ready | Work a MAINTAINERS.md reviewer would approve without hand-holding. With several such PRs across subsystems over 6–12 months plus review participation on others' PRs, a nomination becomes plausible. |
| 80-89 | Credible contributor | A reliable contributor whose PRs need minimal review iteration. This is where maintainers start to look forward to reviewing your work. |
| 65-79 | Contributor | Solid bug-fix-grade work that lands with normal review iteration. Most contributions to most projects live here, and it is honorable work. |
| 50-64 | Learning | PRs eventually land but with significant maintainer guidance. Use the next contribution to focus on your lowest dimension. |
| < 50 | Foundational gap | The PR may have merged, but the process skipped enough corners that a maintainer paid a tax. Restart with a smaller bug and apply the rubric end to end. |
The tier is not a personality assessment. It is calibrated to the artifact you produced for this one Capstone. The same person can score 65 on one contribution and 95 on the next.
How to Self-Grade
Block 30 minutes. Open this rubric beside your own artifacts: the issue, the PR,
the diff, capstone-work/repro.md, execution-path.md, root-cause.md,
validation.md, and the write-up. Score each dimension by reading the band
descriptions and picking the one that most honestly matches what you produced.
Two rules:
- No interpolation upward. Between two bands and unsure? Take the lower. The optimist's tax — and inflated self-grades are visible from orbit the moment a maintainer reads your PR.
- One independent reviewer. Ask a peer (ideally another contributor) to score independently. If your scores differ by more than 10 points on any dimension, talk about it — the difference is where the calibration lives.
Record both scores in capstone-work/self-grade.md, with one sentence per
dimension on what would have moved the score up one band. That becomes the input
for the next contribution's plan.
What to Do With a Low Score
| Lowest dimension | Next contribution focus |
|---|---|
| Reproduction | Pick a bug you can pin to exact FC/kernel/rootfs/host versions; practice the Lab I5 minimal-repro discipline. |
| Execution-path analysis | Pick a bug in a path you've never traced (you did the API path; now do a vCPU exit or a virtqueue), and write execution-path.md before reading the existing tests. |
| Root cause | Practice on a regression: bisect it, find the introducing PR, and write a fix-site-vs-symptom-site paragraph. |
| Fix quality | Pick a bug whose minimum fix is < 10 lines; leave all surrounding code untouched. |
| Tests | Pick a flaky-test-class or test-gap issue — the whole bug is testing discipline. |
| PR craft | Re-do the Lab 2.2 DCO/CHANGELOG drill until it's muscle memory. |
| Communication | Review three other contributors' PRs substantively before opening your next one. |
| Write-up | Write the postmortem for your previous merged PR you skipped; practice the Investigation Log. |
Validation / Self-check
Before declaring the Capstone complete:
capstone-work/self-grade.mdexists with a raw score per dimension, the normalized total, and a tier.- The total is honest, not aspirational — you can defend each score with a citation to your own artifact.
- At least one independent reviewer also scored; disagreements > 10 points on any dimension were discussed.
- The lowest dimension is identified and the next contribution's focus is written down.
- You understand that the tier label describes this one piece of work, not you.
That closes the Capstone. Return to the capstone index to confirm every deliverable is checked, then take what you built into the capstone-projects portfolio — the next eight project briefs each apply this exact cycle to a different Firecracker subsystem, and your self-grade tells you which one to pick first.