Step 10: The Engineering Write-Up

The PR is merged, or it is merged-quality and parked on review latency. The issue is closed, your Signed-off-by: line is in git log origin/main. Most contributors stop here. The ones who become maintainers write the post. The write-up is the artifact that travels with you when you change jobs, request a maintainer nomination, or get cited by the next engineer who hits the same symptom. It is the second of the Capstone's two deliverables — and the one that proves you can communicate engineering reasoning, not just produce a diff.

Five hundred to a thousand words, most of it written in the few hours right after merge, while the dead ends are still fresh. Wait a week and you'll write the sanitized "I knew the answer all along" version, which is both false and useless.

Goal

A structured engineering write-up — problem → investigation → root cause → fix → alternatives → testing → lessons — at a shareable URL, that demonstrates maintainer-level understanding of the Firecracker subsystem you touched. Honest about the dead ends. Cites the issue, the PR, the merged commit, and (if a regression) the introducing PR.

Why It Matters

Three audiences, all real:

  1. Future you. Six months from now you'll touch the same subsystem — the virtqueue handler, the builder boot path, the seccomp filter — and want to remember what you tried and why the fix is shaped the way it is.
  2. The next contributor with the same symptom. They'll find your post by searching the error string ("firecracker vcpu_count 0 InstanceStart") and shortcut a week of work. That is how reputation compounds in this ecosystem.
  3. The maintainers evaluating you. Firecracker is AWS-maintained with a high bar; a maintainer nomination is a judgment about whether you can reason and communicate at the level of the team. The write-up is the evidence — more legible than a diff, more durable than a good PR thread.

A good write-up is a postmortem, not a press release: honest about the approaches that failed, because those are the work.

The Template

Sections in order, with target word counts. The running example follows the vcpu_count == 0 capstone bug so you can see the shape; substitute your own.

Title (one line)

Fixing #NNNN: <one-line technical summary>

Technical and specific, with the symptom phrase a user would search:

  • "Fixing #NNNN: machine-config accepted vcpu_count=0 and failed late at InstanceStart"
  • "Fixing #NNNN: a virtio-block descriptor length that wasn't bounds-checked against guest memory"
  • "Fixing #NNNN: a vsock TX descriptor lost across snapshot restore"

Not "My first Firecracker contribution." Write that post separately if you want; the engineering post stands alone, gets cited, and follows you into a nomination.

Problem (100–150 words)

What broke, for whom, under what conditions. Plain English, precise. Symptom, trigger condition, affected version range. No code yet.

Configuring a microVM with vcpu_count: 0 via PUT /machine-config was accepted by the API, then failed opaquely at InstanceStart when the builder tried to create zero vCPUs — instead of being rejected at configuration time with a clear error. The swagger spec documents a minimum of 1, so this was an unenforced contract: an operator (or an SDK passing a bad value through) got a confusing late failure with no actionable message, rather than a 400 at the boundary. Reproducible on main with a three-line curl sequence; present back to v1.A.0 (an original gap, not a regression). Operator-reachable only — not guest-reachable — so a usability and correctness bug, not a security one.

Investigation Log (200–300 words)

The most valuable section. Walk through what you tried, including the hypotheses that were wrong — three to five, each with the one observation that suggested it and the one experiment that disproved or confirmed it. This is the difference between contributor-grade and maintainer-grade write-ups.

My first hypothesis was that the API parser mangled the value. I added a probe in update_machine_config (located with rg -n "fn update_machine_config" src/vmm/src/resources.rs) and printed the stored config: the value arrived and was stored as 0, correct and unmangled. That ruled out the parse path and pointed at the consumer.

Second hypothesis: the builder mishandled a valid count. I re-ran with vcpu_count: 1 — the builder was correct; only 0 failed, and 0 is never a valid machine config. So the builder's assumption (>= 1) was sound; the defect was that nothing enforced it.

Third hypothesis (confirmed): the documented lower bound was never checked in the Rust path. rg -n "vcpu_count" src/firecracker/swagger/firecracker.yaml showed the spec declares a minimum of 1; rg -n "vcpu_count" src/vmm/src/ showed no bound check at the config boundary. git bisect between v1.A.0 (good — same late failure) and main confirmed this was original behavior, not a regression: the bound was simply never enforced. The fix site was therefore the API-validation boundary, not the builder where it crashed.

Root Cause (50–100 words)

One paragraph, the truth as you now understand it. Distinguish fix site from symptom site; cite the introducing PR if it's a regression.

VmResources::update_machine_config accepts any vcpu_count, including 0, even though firecracker.yaml documents a minimum of 1. The value flows unchecked into build_microvm_for_boot, which assumes >= 1 and fails creating zero vCPUs — the symptom site. The builder's assumption is correct; the defect is the missing lower-bound check at the configuration boundary. Original gap, not a regression (bisect is good to v1.A.0). The right fix is to reject the bad value at the API edge so internal code can keep assuming a well-formed config.

Final Design (150–200 words)

What you changed and why this design over the others. Show the diff size and the principle; name the Firecracker constraints you respected.

The fix adds one lower-bound check in update_machine_config, returning the existing MachineConfigError::InvalidVcpuCount (HTTP 400) — a typed error on an operator-reachable path, never a panic. Behavior for vcpu_count >= 1 is byte-for-byte identical; only the previously-undefined 0 case changes, and it now fails fast and clearly at configuration time. No new error mechanism (I extended the existing enum), no API-shape change (the spec already declared the bound — I enforced it), no snapshot-format change, and no new syscall or emulated device, so the attack surface is unchanged. The production diff is a few lines plus a one-line comment citing the issue and naming the builder's >= 1 invariant. Validated by re-running the original repro (fails on main, passes here) and the full checkstyle / checkbuild --all gates.

Alternatives Considered (100–150 words)

Two or three rejected designs with reasons. This section is what separates contributor-quality from maintainer-quality.

Validate in the builder instead (special-case 0 at the crash site). Rejected: it patches the symptom site, leaves the bad value flowing through the rest of the config plumbing, and gives a late error instead of a clean 400 at the boundary — the exact band-aid Firecracker reviewers reject.

Clamp 0 up to 1 silently. Rejected: silently changing operator input violates the API contract and hides a misconfiguration; an explicit 400 is the correct, documented behavior.

Change the swagger spec to allow 0. Rejected: zero vCPUs is meaningless; the spec is right and the code was wrong, not the reverse.

Testing & Validation (50–100 words)

What proves the fix, and what proves it didn't regress anything.

A unit test in resources.rs asserts vcpu_count == 0 is rejected with the typed error (red on main, green here), with a negative control for vcpu_count == 1. An integration test in tests/integration_tests/functional/test_api.py drives the real API socket and asserts a 400. Regression axes: no resources/seccomp/ diff (attack surface unchanged), no Persist/snapshot change, and a plain boot of a valid config is unaffected. No hot-path cost — the check fires only on invalid input.

Lessons Learned (100–150 words)

Three to five bullets, each concrete enough to reuse.

  • Reject bad input at the boundary, not where it crashes. A panic deep in the builder was a missing validation at the API edge. When you see a late opaque failure, look upstream for the missing check.
  • A documented contract is not an enforced one. The swagger said >= 1; the code never checked it. Specs and enforcement drift — grep both.
  • Bisect to release tags settles "regression vs. original gap" fast. A few git bisect steps proved it was never enforced, which reframed the PR (no regression blame, a clearer "why was this never caught" for the test).
  • The negative control is non-negotiable for a behavior fix — it's the only thing proving the change is scoped, not a blanket flip.
- Issue:                https://github.com/firecracker-microvm/firecracker/issues/NNNN
- PR:                   https://github.com/firecracker-microvm/firecracker/pull/NNNN
- Merged commit:        <SHA>
- (if a regression) Introducing PR #MMMM: <SHA>

Where to Publish

Three venues, in roughly decreasing effort and impact.

  1. Personal or company engineering blog — the full ~1000-word write-up, with an SEO-friendly title carrying the symptom phrase a user would search. This is the version that follows you across jobs and into a maintainer nomination.
  2. The PR description itself — it already carries the engineering reasoning (you wrote it in Step 8). For a non-obvious or widely-felt fix, this is the version maintainers read; keep it sharp.
  3. A community mention — for a fix that affects many users (a snapshot or device behavior, a security-relevant hardening), a one-line note in the Firecracker community channels linking the PR lets maintainers and watchers see the reasoning without reading the whole diff. Optional; earns goodwill; is how people start to know your name.

Note: Security-relevant findings follow a different path. Firecracker takes vulnerability reports privately via AWS Security (firecracker-maintainers@amazon.com / the SECURITY.md process), never as a public issue or a public write-up before a fix ships. If your capstone wandered into a genuine guest-escape or attack-surface vulnerability, stop, report it privately, and write it up only after coordinated disclosure. See SECURITY.md.

Anti-Patterns

What separates write-ups that help from ones that don't:

  • "I learned so much!" — We know. Cut it. The artifact is the engineering.
  • Personal narrative dominating the engineering. Save the "my journey into open source" angle for a separate post; engineering posts get reread and cited.
  • The sanitized "I knew the answer all along" version. Nobody believes it, and it misleads new contributors into thinking their messy investigation is abnormal. The dead ends are the work.
  • No code and no log line. A write-up that never shows the diff or the symptom is unfalsifiable.
  • No links. Issue, PR, merged commit — three minimum. Without the PR link it's unreviewable.
  • Leaking a security issue. Never publish a guest-escape or attack-surface finding before coordinated disclosure. See the note above.
  • Padding to look thorough. A tight 600 words that respects the reader beats a 2000-word slog.

Deliverable for Step 10

  • A published write-up at a shareable URL, 500–1000 words, following the template (problem → investigation → root cause → fix → alternatives → testing → lessons).
  • An Investigation Log with at least two hypotheses you ruled out, each with the experiment that killed it.
  • Alternatives Considered naming at least two rejected designs with reasons.
  • Lessons Learned: three to five bullets, each reusable by a peer.
  • Issue, PR, and merged-commit SHA linked (plus the introducing PR if a regression); security findings handled via private disclosure, not the post.

Rubric Hooks

This is the Write-up dimension (10 pts): postmortem depth, a real Investigation Log with ruled-out hypotheses, alternatives with reasons, and reusable lessons. A write-up that shows the dead ends, distinguishes fix site from symptom site, and links the artifacts scores high; "I found the bug and fixed it," no alternatives, no links, scores low. See the evaluation rubric.

Validation / Self-check

Before declaring the Capstone artifacts complete:

  1. The write-up is published at a shareable URL, 500–1000 words — not 200 (thin), not 3000 (padding).
  2. The Investigation Log contains at least two hypotheses you ruled out, not only the winning one, each with its disproving experiment.
  3. Alternatives Considered names at least two rejected designs with reasons.
  4. Lessons Learned has three to five concrete, reusable bullets.
  5. Issue, PR, and merged commit are all linked (plus the introducing PR if a regression); no security-sensitive finding was published before disclosure.
  6. It reads as a postmortem a peer engineer would respect, not a triumph lap.

Then close the loop with the Evaluation Rubric and grade yourself honestly.