Lab 27: Benchmarks (Milestone 14)

Background

You will build the benchmark suite, commit a baseline, wire the regression gate into CI, and produce the opcode profile that decides Section 7's order.

No optimizations in this lab. The point is to be able to tell whether one worked.

Why This Lab Matters

• Section 7 refuses optimizations without a baseline. This is the baseline.
• The profile, not the literature, decides which optimization to build first. Most people build ADD_INT first because it is the textbook example; the profile usually says field access.
• Deterministic instruction counts give you a performance regression test with zero timing noise, which is better than a benchmark for catching accidental pessimization.

Prerequisites

• Lab 26 complete; CI green.
• Performance Engineering read.
• Nothing running on the machine. Close the browser.

Predict First

1. fib(25) — VM versus tree walker. What ratio? (You predicted this in Lab 7. Look it up.)
2. Which opcode do you expect to dominate the capstone policy's profile?
3. t.field versus t[i] — which is faster, and by how much?
4. What fraction of fib_25's time is dispatch? How would you find out?
5. Does --stats cost anything measurable?

Step 1: The Six Benchmarks

#![allow(unused)]
fn main() {
// benches/dispatch.rs
fn dispatch(c: &mut Criterion) {
    c.bench_function("loop_10m", |b| {
        let src = "local n = 0 for i = 1, 10000000 do n = n + 1 end return n";
        let proto = compile(black_box(src)).unwrap();
        b.iter(|| Vm::new().run_proto(black_box(proto.clone())));
    });
}
// benches/calls.rs      fib_25 (~243k calls)
// benches/tables.rs     field_access, array_index, table_build, hash_heavy
// benches/strings.rs    the four from Lab 16
// benches/gc.rs         alloc_churn, with the PAUSE DISTRIBUTION reported
// benches/policy.rs     THE REALISTIC ONE: 10k candidates, the capstone's scoring policy
}

policy.rs is the only one that matters for the product. The other five tell you why it moved. Keep both; optimizing a microbenchmark the real workload does not exercise is the most common way to waste a week.

Six hazards, from the chapter, and the one that catches the silent failure:

#![allow(unused)]
fn main() {
#[test]
fn benchmarks_scale_with_input_size() {
    // If doubling the input does not roughly double the time, LLVM deleted the
    // thing you are measuring and your benchmark measures nothing.
    let t1 = time_loop(1_000_000);
    let t2 = time_loop(2_000_000);
    let ratio = t2.as_secs_f64() / t1.as_secs_f64();
    assert!((1.5..2.5).contains(&ratio), "benchmark does not scale: ratio {ratio:.2}");
}
}

Step 2: The Baseline, Committed

cargo bench -- --save-baseline v0.1
cp -r target/criterion docs/learning/baselines/v0.1/
git add docs/learning/baselines/v0.1
git commit -m "lab-27: benchmark baseline v0.1 (machine: <cpu>, rustc <version>)"

Record the machine and the toolchain in the commit message. A number without them is not comparable to anything, and six months from now you will not remember.

Then the summary table, in docs/learning/14-performance.md, filled in with your numbers:

Compare against your Lab 7 predictions and write one sentence about the one you got most wrong. That sentence is worth more than the table.

Step 3: The Opcode Profile

#![allow(unused)]
fn main() {
// Behind a feature so the counter array costs nothing by default.
#[cfg(feature = "profile-opcodes")]
self.stats.by_opcode[op.discriminant() as usize] += 1;
}

$ cargo run --release --features profile-opcodes -- run --stats benches/policy.ember
--- opcode profile (policy.ember, 10,000 candidates) ---
  GET_FIELD       4,120,441   31.2%    ← userdata field access
  GET_LOCAL       2,041,882   15.5%
  CALL              820,110   12.4%
  LT                410,055    8.1%
  ADD               620,331    7.9%
  JUMP_IF_FALSE     410,055    6.2%
  MUL               410,055    5.1%
  GET_INDEX         205,027    3.9%
  ...                         
  total          13,204,882

This table decides Section 7's order, and it is worth pausing on:

Most people build ADD_INT first because it is the example everyone uses. Your profile probably says field access, and following the profile rather than the literature is the entire lesson.

Step 4: The Deterministic Regression Gate

Wall-clock benchmarks are too noisy for CI. Instruction counts are not — they are exact:

#![allow(unused)]
fn main() {
#[test]
fn the_corpus_instruction_count_has_not_regressed() {
    // A performance test with ZERO timing noise. A compiler change that makes
    // the corpus execute more instructions fails CI and names the program.
    let baseline: HashMap<String, u64> = read_json("tests/baselines/instructions.json");
    let mut regressions = Vec::new();
    for case in corpus() {
        let n = Engine::new().run_counting(&case.src);
        let b = baseline[&case.name];
        if n > b { regressions.push(format!("{}: {} > {} (+{:.1}%)",
                                            case.name, n, b, 100.0 * (n as f64 / b as f64 - 1.0))); }
    }
    assert!(regressions.is_empty(), "instruction-count regressions:\n{}", regressions.join("\n"));
}
}

This is strictly better than a benchmark for catching accidental pessimization, because it cannot be noise. It also catches the improvements: when a change reduces counts, you update the baseline deliberately, and the diff records what you gained.

# Regenerate deliberately, never automatically:
cargo test --features regen-baselines the_corpus_instruction_count
git diff tests/baselines/instructions.json     # read this before committing

Step 5: Where the Time Goes

cargo flamegraph --bench policy -- --bench
# or, without perf:
perf stat -e instructions,branches,branch-misses,cache-misses \
    ./target/release/ember run benches/policy.ember

Compute the one ratio that decides whether dispatch is worth optimizing:

   branch-misses / VM instructions executed

If it is a small fraction, dispatch is not your problem and threading would buy you nothing — regardless of what the 2003 papers say about 2003 hardware. Two minutes, and it settles a question people argue about for years.

Step 6: Observability Overhead

cargo bench --features no-stats -- fib_25
cargo bench                     -- fib_25

If the delta is in the noise — it should be; the counters are one += 1 on a value already in cache — then delete the no-stats feature. Observability with no measurable cost should not have an off switch, and having measured it is what lets you say so.

Same argument as the instruction budget's. Two features deleted for the same reason is a pattern worth noticing: an off switch for a zero-cost safety or observability feature is a liability, because someone will flip it.

The Trace

$ cargo bench 2>&1 | tee docs/learning/baselines/v0.1/summary.txt
fib_25/interp           time:   [<your number>]
fib_25/vm               time:   [<your number>]
loop_10m/interp         time:   [...]
loop_10m/vm             time:   [...]
policy_10k/vm           time:   [...]

--- YOUR summary table, filled in ---
| Benchmark      | Interp | VM   | Ratio | Predicted in Lab 7 |
|----------------|--------|------|-------|--------------------|
| fib_25         |        |      |       |                    |
| loop_10m       |        |      |       |                    |
| locals_deep    |        |      |       |                    |
| globals        |        |      |       |     (the control)  |

globals is the control. Both backends do a hash lookup on a string key, so it should improve least — and if it improved as much as locals_deep, something else changed and your attribution is wrong. A benchmark suite without a control tells you a number; one with a control tells you a number you can believe.

And the GC pause distribution, which is the number that decides capstone project 3:

$ cargo bench --bench gc 2>&1 | grep -A6 'pause distribution'
--- gc pause distribution (alloc_churn, 60s) ---
  runs: 184   p50: 1.1ms   p95: 6.8ms   max: 31.4ms

Write the max down. It is what lands in your p99 latency, and it is the honest input to "should I build an incremental collector?" — a question that should be answered by a number, not by whether incremental collectors sound advanced.

Expected Output

$ cargo bench -- --baseline v0.1
fib_25/vm    time: [...]  change: [-0.4% +0.2%] (p = 0.61 > 0.05)  No change in performance
$ cargo test the_corpus_instruction_count_has_not_regressed
test ... ok
$ cargo test benchmarks_scale_with_input_size
test ... ok

Debugging Steps

Run-to-run variance is 30%

Close everything, disable turbo/boost if you can, use criterion's statistics rather than a single run, and interleave baseline and change rather than running them sequentially (thermal throttling makes the second one slower regardless of what it is).

A benchmark is suspiciously fast

black_box the inputs and the outputs. Then run the scaling test — if doubling the input does not double the time, LLVM deleted your loop.

The VM is slower than the tree walker on something

Genuinely possible and worth understanding: on a benchmark dominated by a single large allocation, compilation overhead can exceed the execution saving. Report it; a benchmark suite that only shows wins is not measuring.

CI's numbers do not match yours

Different machine. CI compares CI against CI; your machine compares against your baseline. Never compare across.

The instruction-count test fails after a compiler improvement

That is the gate working in the other direction. Regenerate deliberately and read the diff.

The opcode profile is dominated by an opcode you did not expect

Good — that is why you profiled. Follow it.

Experiment

CLAIM. The opcode profile of a realistic workload differs substantially from that of a microbenchmark, and following the microbenchmark leads you to optimize the wrong thing.

METHOD. Profile fib_25, loop_10m, and policy_10k. Rank the top five opcodes in each.

PREDICTION. How much overlap? Which opcode is top in the policy and where does it rank in fib_25?

RESULT. Three ranked lists, side by side, in docs/learning/14-performance.md. The expected shape: fib_25 is dominated by CALL and GET_LOCAL; loop_10m by ADD and JUMP; policy_10k by GET_FIELD. Three different answers to "what should I optimize?" — and only the third one is about the product.

Test

#![allow(unused)]
fn main() {
#[test]
fn benchmarks_scale_with_input_size() { /* Step 1 */ }

#[test]
fn the_corpus_instruction_count_has_not_regressed() { /* Step 4 */ }

#[test]
fn instruction_counts_are_identical_across_runs_and_seeds() {
    // The property that makes the regression gate possible.
    for case in corpus() {
        let a = Engine::with_seed(1).run_counting(&case.src);
        let b = Engine::with_seed(0xdeadbeef).run_counting(&case.src);
        let c = Engine::with_seed(1).run_counting(&case.src);
        assert_eq!(a, b, "{} count depends on the hash seed", case.name);
        assert_eq!(a, c, "{} count is not reproducible", case.name);
    }
}

#[test]
fn the_baseline_file_records_its_machine() {
    // A number without its machine is not comparable to anything.
    let meta: BaselineMeta = read_json("docs/learning/baselines/v0.1/meta.json");
    assert!(!meta.cpu.is_empty() && !meta.rustc.is_empty() && !meta.profile.is_empty());
    assert_eq!(meta.profile, "release");
}

#[test]
fn every_benchmark_has_a_control_or_is_marked_as_one() {
    // A suite without a control tells you a number you cannot believe.
    let suite = benchmark_manifest();
    assert!(suite.iter().any(|b| b.is_control),
            "no control benchmark: add one that should NOT move");
}
}

Challenge Extensions

1. A CI benchmark bot. Run the suite on a dedicated machine per PR, post the criterion diff as a comment, and fail on a >5% regression in policy_10k.
2. Compare against real Lua. lua fib.lua versus ember run fib.ember. Expect Ember to be slower — Lua is a register machine with twenty years of tuning. Write down the factor; it is your honest position, and Section 7 is about narrowing it.
3. Memory benchmarks. Peak RSS and allocation count for the policy workload, alongside time. Allocation count is deterministic, so it can join the regression gate.
4. A cargo bench --features nan-boxing comparison, once capstone project 2 exists. This is the measurement ADR-004 promised.
5. Instruction counts per candidate. policy_10k divided by 10,000 is "instructions per candidate", which is the number a capacity plan actually needs.

Deliverables

• Six benchmark files, including the realistic policy one and at least one control.
• The scaling test, proving the benchmarks measure something.
• A committed baseline with machine and toolchain metadata.
• The summary table filled in, compared against Lab 7's predictions, with one sentence on the biggest miss.
• The opcode profile for all three workloads, and a stated decision about Section 7's order.
• The deterministic instruction-count regression gate in CI, with deliberate regeneration.
• The branch-misses / instructions ratio measured and recorded.
• The observability-overhead measurement, and the no-stats feature deleted if it says so.
• The GC pause distribution, with the max recorded.
• docs/learning/14-performance.md written.
• Milestone 14, half one, complete. Section 6 done.

Validation / Self-check

1. Why is policy_10k the only benchmark that matters for the product, and why keep the other five?
2. What is a control benchmark, and which of yours is it? What did it tell you?
3. What test proves a benchmark was not optimized away?
4. Why are instruction counts a better CI gate than wall clock? What can they not tell you?
5. What did your opcode profile say, and how does that change Section 7's order?
6. What is the branch-misses / instructions ratio for, and what did yours say about dispatch?
7. Which two features did you delete after measuring, and what is the shared principle?
8. Which GC pause number decides the incremental-collector question, and what was yours?

Next: Section 7 — Making It Fast.