Evaluation Rubric
A 100-point self-grading rubric for the Capstone. Score yourself honestly after you finish Step 10. The scoring is calibrated against what OpenSearch maintainers actually look for on a Pull Request — not what feels good to read.
The point of the rubric is not the score. It is the diagnostic: a low score on one dimension tells you exactly where to invest the next contribution.
Scoring Dimensions
Seven dimensions, weighted by how much they matter for review outcomes.
| # | Dimension | Points |
|---|---|---|
| 1 | Problem articulation | 20 |
| 2 | Execution-path mastery | 20 |
| 3 | Implementation quality | 20 |
| 4 | Testing | 15 |
| 5 | Review responsiveness | 10 |
| 6 | Documentation | 10 |
| 7 | Community interaction | 5 |
| Total | 100 |
1. Problem Articulation (20 pts)
Can you state, in one paragraph, what was broken, for whom, under what conditions?
| Score | What it looks like |
|---|---|
| 18-20 | Crisp one-paragraph statement covering symptom, trigger conditions, affected version range (2.x/main), and operational impact (e.g. Dashboards zero-rows). Distinguishes "what the user sees" from "the underlying mechanism." Could be read aloud at a standup and a peer would grasp the bug. |
| 14-17 | Clear symptom but trigger conditions vague ("happens sometimes under load"). OR trigger clear but conflates symptom with root cause. |
| 10-13 | Reader must ask follow-ups to understand what broke. Uses jargon (reduce, seqno) without grounding it in user-visible behavior. |
| 5-9 | Mostly restates the GitHub issue title. No conditions, no version impact. |
| 0-4 | "It was broken and I fixed it." |
Look for: the word "intermittent" without a documented trigger; conflation of symptom (empty buckets) with cause (shard-local fast path).
2. Execution-Path Mastery (20 pts)
Did you actually trace the code, or did you guess?
| Score | What it looks like |
|---|---|
| 18-20 | Step-3 doc maps the full path from REST handler → transport action → shard/engine/coordination with org.opensearch.* file:line citations at every layer, pinned to a commit SHA. Includes a mermaid (or arrow) diagram with the bug node annotated. Identifies the exact line where the value flips from correct to wrong, confirmed by a breakpoint or TRACE log. A reviewer could open each file at each line and follow it without asking. |
| 14-17 | Most layers cited but one or two skipped ("then it reaches the aggregator"). Diagram present but missing a critical hop. |
| 10-13 | Bug location cited correctly but no trace of how execution reached it. No diagram. |
| 5-9 | Vague references ("the search service handles it") without file:line. |
| 0-4 | No execution-path document, or just a paragraph of prose. |
Look for: server/src/main/java/org/opensearch/...-style paths with line numbers
pinned to a recorded commit, plus the confirmed observation point.
3. Implementation Quality (20 pts)
Diff hygiene, scope discipline, convention compliance.
| Score | What it looks like |
|---|---|
| 18-20 | Minimum-diff fix — production change in tens of lines, not hundreds. Every changed line justifiable in one sentence. No drive-by refactors, no opportunistic renames, no reformatting. Public API and wire format unchanged unless required; any serialization change is Version-guarded and registered. SPDX headers intact. Thread-safety reasoned (what thread, what lock). spotlessJavaCheck and precommit green without manual overrides. |
| 14-17 | Mostly minimum-diff but one or two stray changes that don't belong. Conventions mostly followed; minor style nits a reviewer would flag. |
| 10-13 | Fix works but broader than necessary. Scope creep ("while I was here..."). Conventions inconsistently applied. |
| 5-9 | Significant scope creep. Public API or wire format changed unnecessarily/unguarded. Style violations would block precommit. |
| 0-4 | Diff so large reviewers would ask it be broken up. OR breaks BWC/public API silently. |
Look for: scope-creep tells via git diff origin/main --stat showing files
unrelated to the bug; an unguarded writeTo/StreamInput change.
4. Testing (15 pts)
Coverage, determinism, regression value.
| Score | What it looks like |
|---|---|
| 14-15 | New test reproduces the bug deterministically (red on main, green with fix) at the lowest viable level. A negative control proves the fix is scoped. Every trigger condition encoded. Serialization/BWC test present iff the wire changed. No Thread.sleep, no wall-clock, no order-dependent assertions; uses assertBusy/ensureGreen. Ran ≥ 50 iters and multiple seeds without a flake. |
| 11-13 | Test present and deterministic but no negative control. OR an integration test where a unit test would do, with a weak unit test. |
| 7-10 | Test uses Thread.sleep or is otherwise non-deterministic. Coverage of the fix path incomplete. |
| 3-6 | Test only checks the happy path; would have passed before the fix. |
| 0-2 | No new tests, or tests that fail on both main and the fix. |
Look for: assertBusy(...) rather than Thread.sleep; a scenario-named test
(testMissingWithMinDocCountZeroOnEmptyShard) not a method-named one
(testReduce).
5. Review Responsiveness (10 pts)
How well you ran the review cycle.
| Score | What it looks like |
|---|---|
| 9-10 | Every reviewer comment addressed in code or with a substantive reply. Iteration cadence < 48h on most comments. Disagreements made the technical case without defensiveness. Pushed fixup commits during review (no silent force-push); when a force-push was needed, used --force-with-lease and summarized the change. Updated the PR description after material changes so the top stays accurate. CI re-validated and green after every push. |
| 7-8 | Addresses comments correctly but slowly (multi-day gaps). OR lets a few stylistic comments slide without acknowledgement. |
| 5-6 | Defensive on at least one comment ("but my way is fine"). OR force-pushed without summarizing for reviewers. |
| 2-4 | Required multiple reminders. Comments not addressed cleanly. Broke CI on a later push and didn't notice. |
| 0-1 | PR went silent > 2 weeks without explanation, or argued every comment. |
Look for: review threads resolved by the contributor with a substantive pushed commit, not just a reply; green CI on every revision, not only the first.
6. Documentation (10 pts)
Issue/PR hygiene, CHANGELOG, code comments, write-up.
| Score | What it looks like |
|---|---|
| 9-10 | CHANGELOG entry under the correct heading (Fixed vs Changed), user-facing, PR-linked. PR uses Resolves #NNNN. Issue updated with repro + root cause. Docs implication resolved (a documentation-website PR linked, or an explicit "no docs change needed"). In-code comment cites #NNNN at the non-obvious line. Write-up exists at a public URL. |
| 7-8 | CHANGELOG present but wrong heading, or docs implication unaddressed on a user-visible change. Code comments present but don't cite the issue. |
| 5-6 | Mechanics followed but fields incomplete. No write-up beyond the PR description. |
| 2-4 | CHANGELOG missing or DCO/CHANGELOG checks red. Comments absent at surprising lines. |
| 0-1 | No CHANGELOG, no issue link, no write-up. |
Look for: the CHANGELOG line under a heading you can defend, and an explicit docs decision rather than silence.
7. Community Interaction (5 pts)
Forum/Slack/issue etiquette, claiming and handoff hygiene.
| Score | What it looks like |
|---|---|
| 5 | Commented on the issue before starting ("working on this, PR incoming"). Requested the right reviewers (area maintainers from MAINTAINERS.md + the introducing-PR author). Posted to the forum/Slack only when meaningful (a design question, a summary after merge). Thanked reviewers explicitly. If stuck, posted clearly: "stuck on X, considering A/B/C, leaning A because Y." |
| 3-4 | Mostly good etiquette; one minor slip (claimed late, or one low-signal forum post). |
| 1-2 | Did not signal intent before working. OR sent low-signal channel traffic ("does anyone know...?"). |
| 0 | Worked silently for weeks, then dropped a PR with no context and no reviewers requested. |
Look for: an issue comment by the contributor before the first push, and an explicit thank-you to reviewers in the PR.
Tier Thresholds
Where you land tells you what to do next.
| Score | Tier | Interpretation |
|---|---|---|
| 95-100 | TSC-track | The quality that, sustained across many contributions over months, puts you on the path the Technical Steering Committee notices. You operate at the level the project would trust to steer an area. |
| 90-94 | Maintainer-ready | You write PRs at maintainer quality. With several such contributions across areas over 6–12 months plus demonstrated review participation on others' PRs, a MAINTAINERS.md nomination is plausible. |
| 80-89 | Credible contributor | A reliable contributor whose PRs need minimal review iteration. Keep building the track record; this is where maintainers actively look forward to reviewing your work. |
| 65-79 | Contributor | Solid bug-fix-grade work. PRs land with normal review iteration. Most contributions to most projects live here, and it is honorable work. |
| 50-64 | Learning | PRs eventually land but with significant reviewer guidance. Use the next contribution to focus on your lowest dimension. |
| < 50 | Foundational gap | The PR may have merged, but the process skipped enough corners that a maintainer paid a tax. Restart with a smaller bug and apply the rubric end to end. |
The tier is not a personality assessment. It is calibrated to the artifact you produced for this one Capstone. The same person can score 65 on one contribution and 95 on the next.
How to Self-Grade
Block 30 minutes. Open this rubric beside your own artifacts: the issue, the PR,
the diff, capstone-work/root-cause.md, capstone-work/execution-path.md,
capstone-work/validation.md, and the write-up. Score each dimension by reading
the band descriptions and picking the one that most honestly matches what you
produced.
Two rules:
- No interpolation upward. Between 14 and 17 and unsure? Score 14. The optimist's tax.
- One independent reviewer. Ask a peer (ideally another contributor) to score independently on this rubric. If your scores differ by more than 10 points on any dimension, talk about it — the difference is where the calibration lives.
Record both scores in capstone-work/self-grade.md, with one sentence per
dimension on what would have moved the score up one band. This becomes the input
for the next contribution's plan.
What to Do With a Low Score
| Lowest dimension | Next contribution focus |
|---|---|
| Problem articulation | Pick a smaller, sharper bug. Write the one-paragraph statement before you comment on the issue, and post it for feedback. |
| Execution-path mastery | Pick a bug in a layer you've never traced (you did search/agg; now do coordination or the engine). Write the path doc before reading the existing tests. |
| Implementation quality | Pick a bug whose minimum fix is < 10 lines. Practice leaving surrounding code untouched. |
| Testing | Pick a flaky-test-labeled issue. The whole bug is testing discipline. |
| Review responsiveness | Pick a bug in a high-traffic area where you'll get more reviewers. Set a 24h SLA on every comment. |
| Documentation | Pick a fix that needs a documentation-website PR. Write the docs change before the code is done. |
| Community interaction | Review three other contributors' PRs substantively before opening your next one. |
Validation / Self-check
Before declaring the Capstone done:
capstone-work/self-grade.mdexists with a score per dimension and a total.- The total is honest, not aspirational — you can defend each score with a citation to your own artifact.
- At least one independent reviewer also scored, and disagreements > 10 points on any dimension were discussed.
- The lowest dimension is identified and the next contribution's focus is written down.
- The score is recorded where you'll see it again in three months.
- You understand that the tier label ("Contributor", "Maintainer-ready") describes this one piece of work, not you.
- You have a candidate next bug picked, with the focus dimension in mind.