Step 10: Engineering Write-Up

The PR is merged. The issue is closed. The backport is green. Most contributors stop here. The ones who become maintainers write the post. The write-up is the artifact that travels with you when you change jobs, request a maintainer nomination, or get cited by another contributor working a similar bug.

Five hundred to a thousand words. Most of it written in the few hours right after merge, while the dead ends are still fresh in your memory.

Why It Matters

Three audiences:

  1. Future you. Six months from now you'll touch TermsAggregator again and want to remember what you tried and why the fix is shaped the way it is.
  2. The next contributor working a similar bug. They'll find your post via search ("OpenSearch terms agg missing bucket min_doc_count") and shortcut a week of work.
  3. The maintainers / TSC evaluating you. A maintainer nomination is a judgment about whether you can communicate engineering reasoning, not just produce diffs. The write-up is the evidence.

A good write-up is not a press release. It is a postmortem: honest about what you tried, including the approaches that failed.

The Template

Sections in order, with suggested word counts. A worked excerpt follows the aggregation Capstone bug throughout, so you can see the shape.

Title (one line)

Fixing #NNNN: <one-line technical summary>

Examples:

  • "Fixing #NNNN: terms aggregation dropped the missing bucket on empty shards"
  • "Fixing #NNNN: a ClusterApplierService listener that leaked on coalesced states"
  • "Fixing #NNNN: a seqno race in InternalEngine under concurrent same-_id updates"

Technical and specific. Not "My first OpenSearch contribution" — write that post separately. The engineering post stands on its own and gets cited; the journey post gets one-time clicks.

Problem (100–150 words)

What broke, for whom, under what conditions. Plain English, but precise. Symptom, trigger condition, affected version range. No code yet.

A terms aggregation requested with min_doc_count: 0 and a missing value would return an empty buckets array instead of the synthetic bucket, whenever at least one queried shard held zero matching documents. The simplest trigger is a single-shard index where the aggregated field exists in the mapping but in no document: the response is "buckets": [] rather than the documented [{ "key": "N/A", "doc_count": 1 }]. Dashboards visualizations that rely on "show empty buckets" silently lost their zero rows. Reproducible on main and on the 2.x line; the behavior regressed in #4567, which optimized the shard-local empty-segment path.

Investigation Log (200–300 words)

The most valuable section. Walk through what you tried, including the hypotheses that were wrong. Three to five hypotheses, in order, each with one sentence on what suggested it and one on what disproved it.

My first hypothesis was that the reduce step (InternalTerms.reduce) was discarding the bucket while merging shards. I added a unit test driving reduce directly with two hand-built shard results, one containing the missing bucket — reduce kept it. That ruled out the coordinating node and pointed back at the shards.

Second hypothesis: the bucket was never produced on the empty shard. TRACE logging on org.opensearch.search.aggregations for a one-doc, one-shard repro confirmed it: the shard-local aggregator emitted zero buckets for the empty-segment case, so reduce had nothing to merge. The bug was upstream of reduce, in shard-local collection.

Reading TermsAggregator.doPostCollection() top to bottom surfaced an empty-segment fast path: if (!hasValuesForField()) return;. A git blame placed it in #4567 — a performance optimization that skipped bucket-building when a segment had no ordinals for the field. The assumption ("no values ⇒ no buckets to emit") is true for min_doc_count >= 1, where empty buckets are discarded anyway, but false for min_doc_count == 0, where the synthetic bucket must be emitted by every shard. git bisect between 2.11.0 (good) and main (bad) confirmed #4567 as the introducing change in four steps.

Root Cause (50–100 words)

One paragraph, the truth as you now understand it. Cite the introducing PR and the bisect commit.

The shard-local terms aggregator short-circuits bucket construction when a segment has no values for the field. That fast path (introduced in #4567, commit a1b2c3d4) is correct for min_doc_count >= 1 but skips the synthetic and missing buckets that min_doc_count == 0 requires every shard to emit. With no shard producing the bucket, the coordinating-node reduce has nothing to merge and the response omits it.

Final Design (150–200 words)

What you changed and why this design over the alternatives. Show the diff size and the principle.

The fix gates the fast path on min_doc_count:

if (!hasValuesForField() && bucketCountThresholds.getMinDocCount() != 0) {
    return;
}

One guard condition in doPostCollection(). Behavior for min_doc_count >= 1 is byte-for-byte identical (the extra clause is true in that case, so the branch is unchanged). The synthetic bucket is now emitted per shard exactly when min_doc_count == 0, restoring the input reduce expects. No serialization change, so no Version guard and no BWC test — only the intended observable change (the bucket reappears). Public API surface unchanged; no new setting. The production diff is two lines plus a comment citing the issue.

Alternatives Considered (100–150 words)

Two or three alternatives you rejected, with the reason. This is the section that separates contributor-quality from maintainer-quality write-ups.

Synthesize the bucket at reduce time when min_doc_count == 0 and no shard produced it. Rejected: reduce lacks the per-shard missing-value context to build a correct bucket without risking double-counting; it would push correctness into the merge path that runs for every terms agg.

Revert #4567's fast path entirely. Rejected: it's a real and measured optimization for the common min_doc_count >= 1 case; reverting it regresses performance to fix a narrow parameterization.

Document the combination as unsupported. Rejected: there is a clear correct answer; documenting around a fixable bug taxes every user.

Performance / Behavior Impact (50–100 words)

If perf-relevant, numbers. Otherwise, one honest sentence.

No measurable performance impact. The added clause is a single integer comparison on a path already gated by hasValuesForField(), and it only changes behavior in the min_doc_count == 0 case, which previously did less work incorrectly. The common path is unchanged. Validated by re-running the aggregation microbenchmark suite: no statistically significant delta across 10 runs.

Lessons Learned (100–150 words)

Three to five bullets, each concrete enough to be reusable by a peer.

  • Fast-path optimizations are where correctness goes to die. Any if (cheapCheck) return; added "for performance" is a candidate for a missed parameterization. When you find one in blame, ask which inputs it didn't consider.
  • Bisect against release tags is cheaper than archaeology. Four git bisect steps between 2.11.0 and main found #4567 faster than reading the file's history would have.
  • Prove the layer before you fix the layer. Driving reduce directly in a unit test ruled out the coordinating node in 20 minutes and saved me from "fixing" the wrong place.
  • A negative-control test is non-negotiable for behavior fixes — it's the only thing proving your change is scoped, not a blanket flip.
- Issue:               https://github.com/opensearch-project/OpenSearch/issues/NNNN
- PR:                  https://github.com/opensearch-project/OpenSearch/pull/NNNN
- Merged commit:       <SHA>
- Backport (2.x):      https://github.com/opensearch-project/OpenSearch/pull/MMMM
- Introducing PR #4567: <SHA>

Where to Publish

Three venues, in roughly decreasing order of effort and impact.

1. Personal or company engineering blog

The full ~1000-word write-up. SEO-friendly title with the symptom phrase a user would search ("OpenSearch terms aggregation missing bucket"). Link prominently to the issue and PR. This is the version that follows you across jobs and into a maintainer nomination.

2. The OpenSearch community forum

Post a shorter version (300–500 words) on forum.opensearch.org, focused on the lesson and the user-facing behavior, under the relevant category. This reaches operators who hit the symptom and search the forum before the code. Link the blog post for the deep version.

3. The PR itself + a Slack/community-meeting mention

The PR description already carries the engineering reasoning (you wrote it in Step 8). For a non-obvious or widely-felt fix, a one-line mention in the OpenSearch public Slack (opensearch.org/slack) or a community meeting — "fixed a terms-agg empty-bucket bug that affected Dashboards zero-rows, details in #NNNN" — lets maintainers and watchers see the reasoning without reading the whole diff. Optional, earns goodwill, and is how people start to know your name.

Anti-Patterns

What separates write-ups that help from ones that don't:

  • "I learned so much!" — We know. Cut it. The artifact is the engineering.
  • Personal narrative dominating the engineering. Save the "my journey into open source" angle for a separate post. Engineering posts get reread and cited.
  • The sanitized "I knew the answer all along" version. Nobody believes it, and it misleads new contributors into thinking their messy investigation is abnormal. Be honest about the dead ends — they are the work.
  • No code and no log line. A write-up that never shows the diff or the symptomatic response is unfalsifiable.
  • No links. Issue, PR, merged commit — three minimum. Without the PR link the write-up is unreviewable.
  • Padding to look thorough. A tight 600 words that respects the reader beats a 2000-word slog.

Validation / Self-check

Before declaring the Capstone complete:

  1. The write-up is published at a shareable URL (blog, forum, or a public capstone-work/writeup.md), 500–1000 words — not 200 (thin), not 3000 (padding).
  2. The Investigation Log contains at least two hypotheses you ruled out, not only the winning one.
  3. Alternatives Considered names at least two rejected designs with reasons.
  4. Lessons Learned has three to five bullets, each reusable by another contributor.
  5. Issue, PR, and merged-commit SHA are all linked (plus the introducing PR).
  6. The post reads as something a peer engineer would respect — a postmortem, not a triumph lap.
  7. You picked a publish venue and actually posted (or have it queued), not "I'll write it later."

Then close the loop with the Evaluation Rubric and grade yourself honestly.