Performance Engineering

This chapter is a protocol, not a list of tricks. Section 7 contains the tricks; this chapter is what makes them legitimate.

The rule for the rest of the curriculum:

No optimization ships without a baseline, a hypothesis, a post-change measurement, and a stated tradeoff. An optimization with no measured win is reverted — and the reversion is recorded too.

The Protocol

1. BASELINE      Measure. Commit the numbers. `git show` them later.
2. HYPOTHESIS    "X is slow BECAUSE Y." A causal claim, written down, before changing anything.
3. CHANGE        The smallest change that tests the hypothesis.
4. MEASUREMENT   Same machine, same conditions, same benchmark.
5. TRADEOFF      What did this cost in readability, memory, `unsafe`, or maintenance?
6. DECISION      Keep, or revert. RECORD EITHER WAY.

Step 2 is the one people skip, and skipping it is what makes optimization feel like guesswork. A hypothesis you can be wrong about is what makes step 4 informative; without one, any change that happens to be faster looks like insight.

Step 6's second half matters more than it looks. A reverted optimization with a measurement is a permanent record that the obvious idea does not work here — which saves the next person (you, in March) from trying it again.

The Benchmark Suite

Five benchmarks, each isolating one cost. "The VM is 20× faster" is useless; "local access is 40×, calls are 12×, arithmetic is 3×" is a design document.

Plus one that is not a microbenchmark:

| policy | the capstone's actual workload | 10,000 candidates, a realistic scoring policy |

The last one is the only one that matters for the product, and the first five are how you find out why it moved. Keep both; optimizing a microbenchmark that the real workload does not exercise is the most common way to waste a week.

Measuring Honestly

cargo bench -- --save-baseline before
# ... make the change ...
cargo bench -- --baseline before

Six sources of lying to yourself, and what to do about each:

That last one is worth a specific check: if doubling the input does not roughly double the time, something was optimized away and your benchmark measures nothing.

The Two Kinds of Measurement

   WALL CLOCK                         COUNTERS
   ──────────                         ────────
   criterion, perf, flamegraph        instructions executed, allocations, GC runs
   noisy, machine-dependent           DETERMINISTIC, machine-independent
   what users experience              what changed, exactly
   answers "is it faster?"            answers "why?"

Ember's counters are deterministic, which makes them unusually powerful here: a compiler change that reduces executed instructions by 12% is a fact, not a measurement, and it can be a CI gate with no timing noise at all.

#![allow(unused)]
fn main() {
#[test]
fn the_corpus_instruction_count_has_not_regressed() {
    // A performance test with ZERO timing noise. If a change makes the corpus
    // execute more instructions, CI fails and names the program.
    for case in corpus() {
        let n = Engine::new().run_counting(&case.src);
        let baseline = BASELINE_COUNTS[&case.name];
        assert!(n <= baseline, "{}: {} instructions, baseline {}", case.name, n, baseline);
    }
}
}

Use both: counters to find out what changed, wall clock to find out whether anyone cares.

Where the Time Actually Goes

Before Section 7 hands you techniques, spend an hour finding out where your runtime spends its time. The answers are usually not what people guess:

cargo flamegraph --bench policy -- --bench
# or, without perf:
ember run --stats --profile-opcodes examples/policy.ember

--- opcode profile (policy.ember, 10k candidates) ---
  GET_FIELD      4,120,441   31%     ← userdata field access
  CALL             820,110   14%
  GET_LOCAL      2,041,882   12%
  LT               410,055    9%
  ADD              620,331    8%
  JUMP_IF_FALSE    410,055    6%
  ...

A profile like that decides Section 7's order. If GET_FIELD is 31% of your instructions, an inline cache on field access is the first thing to build and a specialized ADD_INT is the fourth. Without the profile you would probably have built ADD_INT first, because it is the example everyone uses.

What Not to Optimize

Three things Ember deliberately does not optimize, each for a different reason:

Saying which things you have chosen not to optimize is as much a part of a performance strategy as saying which you have.

Things to Notice

• A hypothesis is what makes a measurement informative. Without one you are collecting numbers.
• Record reverted optimizations. They stop the next person repeating them.
• Five microbenchmarks tell you why; one realistic benchmark tells you whether it mattered.
• Deterministic counters make a zero-noise performance regression test, which wall-clock benchmarking cannot.
• Profile before choosing techniques. The profile, not the literature, decides the order.
• Naming what you will not optimize is part of the strategy.

Validation / Self-check

1. Give the six steps of the protocol. Which is most often skipped, and what does skipping it cost?
2. Why record a reverted optimization?
3. Name the five microbenchmarks and the one realistic benchmark, and say what each answers.
4. Give three ways a benchmark can lie to you and the mitigation for each.
5. What check tells you LLVM deleted the code you were measuring?
6. Why are Ember's instruction counters more useful than wall clock for CI, and what can they not tell you?
7. What did your opcode profile say, and how does that change the order of Section 7?
8. Name three things Ember does not optimize, each with a different reason.

Next: The Hardening Checklist.