Project 7: A Terminal Benchmark and Conformance Suite

1–2 weeks · ●●●○○ · touches measurement, across implementations

The project with the widest audience. Nobody maintains a current cross-terminal conformance table, and building one is a genuine contribution to the ecosystem.

1. The Problem

Two questions have no good answer today:

1. "Is my terminal fast?" — vtebench measures throughput on a handful of workloads. Nothing measures latency, memory, or behavior under a flood in a comparable way.
2. "Does my terminal behave correctly?" — there is no conformance suite. The only way to know whether your CSI 999 C matches xterm's is to try it in xterm.

The second is the more valuable half, and it is what makes this a conformance suite rather than just a benchmark.

2. Why It Is Hard

Measuring a terminal is measuring a pipeline you do not control both ends of.

3. The Design

Two halves, sharing a corpus.

Half 1: The conformance suite

   For each behavior:
     1. Emit a sequence.
     2. Ask the terminal what happened, MACHINE-READABLY.
     3. Compare against a reference capture.

   The only general-purpose probe is CSI 6n — "where is the cursor?" — which
   turns most behavioral questions into a number you can diff.

#!/usr/bin/env bash
# conformance.sh — emit, probe, report. Runs IN a terminal, reports on stdout.

probe() {   # $1 = label, $2 = sequence
    printf '\033[H\033[2J'                       # home + clear
    printf '%b' "$2"
    printf '\033[6n'
    IFS='[;' read -rsd R _ row col
    printf '%-44s %s;%s\n' "$1" "$row" "$col"
}

echo "# terminal-conformance v1  TERM=$TERM  size=$(stty size)"
probe "baseline"                    'x'
probe "CUF clamps at the margin"    '\033[999C'
probe "CUF default is 1"            '\033[C'
probe "CUP is 1-based"              '\033[5;10H'
probe "CUP clamps"                  '\033[999;999H'
probe "ED2 does not move the cursor" '\033[5;5H\033[2J'
probe "pending wrap at exactly cols" "$(printf 'x%.0s' $(seq 1 $(tput cols)))"
probe "one past cols"               "$(printf 'x%.0s' $(seq 1 $(( $(tput cols) + 1 ))))"
probe "DECAWM off does not wrap"    "\033[?7l$(printf 'x%.0s' $(seq 1 200))\033[?7h"
probe "wide chars advance by 2"     '日本'
probe "combining mark does not advance" 'e\xcc\x81'
probe "ZWJ family emoji width"      '\xf0\x9f\x91\xa8\xe2\x80\x8d\xf0\x9f\x91\xa9'
probe "ambiguous width (degree)"    '°'
probe "box drawing width"           '┌'
probe "tab to next stop"            '\t'
probe "tab at the last stop"        "\033[$(( $(tput cols) - 1 ))G\t"
probe "BS does not erase"           'abc\b'
probe "REP repeats"                 'x\033[5b'
probe "ECH does not move"           'abcdef\033[1;3H\033[2X'
probe "DCH shifts left"             'abcdef\033[1;3H\033[2P'
probe "scroll region homes cursor"  '\033[10;20H\033[5;15r'
probe "origin mode is relative"     '\033[5;15r\033[?6h\033[1;1H'

# Capture a reference, then diff:
xterm -e './conformance.sh > /tmp/xterm.txt'
./conformance.sh > /tmp/mine.txt          # inside your terminal
diff /tmp/xterm.txt /tmp/mine.txt

Every differing line is a finding, and it is classified as BUG, DELIBERATE, or REFERENCE-IS-WRONG (which does happen).

Half 2: The benchmark

Measure four separable things, because conflating them is the usual mistake:

#![allow(unused)]
fn main() {
// benches/parse.rs — the layer you can measure with total precision.
fn bench_parse(c: &mut Criterion) {
    let corpus = load_corpus();
    let mut group = c.benchmark_group("parse");
    for (name, bytes) in &corpus {
        group.throughput(Throughput::Bytes(bytes.len() as u64));
        group.bench_function(*name, |b| {
            b.iter(|| {
                // A fresh Terminal each iteration: reusing one means measuring
                // a warm scrollback, which is a different (and easier) workload.
                let mut t = Terminal::new(24, 80, Default::default());
                t.advance(black_box(bytes));
            })
        });
    }
}
}

Workloads worth having, because they stress different code:

4. Milestones

Milestone 1 alone is useful to you immediately — it turns "does my terminal behave right?" into a diff.

5. The Tests

A test suite needs its own tests, and it needs them more than most code.

#![allow(unused)]
fn main() {
#[test]
fn conformance_probes_are_deterministic() {
    // A probe that reports different results on consecutive runs is measuring
    // noise, and it will waste someone's afternoon.
    for probe in ALL_PROBES {
        let a = run_probe_headless(probe);
        let b = run_probe_headless(probe);
        assert_eq!(a, b, "probe '{}' is nondeterministic", probe.name);
    }
}

#[test]
fn every_probe_has_a_documented_expectation() {
    // "xterm does X" is not enough. WHY does it do X, and where is that stated?
    for probe in ALL_PROBES {
        assert!(!probe.rationale.is_empty(), "probe '{}' has no rationale", probe.name);
        assert!(probe.source.is_some(), "probe '{}' cites no source", probe.name);
    }
}

#[test]
fn our_own_terminal_passes_its_own_probes_headlessly() {
    // The suite must be runnable against terminal-core with no terminal at all,
    // or it cannot run in CI.
    for probe in ALL_PROBES {
        let mut t = Terminal::new(24, 80, cfg());
        t.advance(&probe.sequence);
        t.advance(b"\x1b[6n");
        let reply = t.take_replies();
        assert_eq!(parse_cpr(&reply), probe.expected_cursor,
                   "probe '{}': {}", probe.name, probe.rationale);
    }
}

#[test]
fn benchmark_corpus_is_stable() {
    // Benchmarks compared across versions must use IDENTICAL input.
    for (name, bytes) in load_corpus() {
        assert_eq!(sha256(&bytes), EXPECTED_HASHES[name],
                   "corpus '{name}' changed — old numbers are no longer comparable");
    }
}
}

That last one matters more than it looks. A benchmark corpus that silently changes makes every historical number a lie.

6. The Measurement — and the Honesty Rules

The output is a table. Its value depends entirely on how carefully you qualify it.

## Parse throughput (MB/s, higher is better)

| Core                  | ASCII | Dense SGR | Scrolling | Unicode | Adversarial |
|-----------------------|-------|-----------|-----------|---------|-------------|
| mini-terminal         |       |           |           |         |             |
| alacritty_terminal    |       |           |           |         |             |
| vte (parse only)      |       |           |           |         |             |

Method: criterion, 100 samples, fresh Terminal per iteration, 24x80.
Machine: <cpu>, <ram>, rustc <version>, --release, lto=thin.
Corpus: tests/golden/cases, sha256 pinned in corpus.lock.

The four honesty rules, and they are what separate a useful benchmark from a misleading one:

1. State the machine, the compiler, and the flags. Every number is meaningless without them.
2. Pin the corpus by hash. Otherwise version-to-version comparisons are fiction.
3. Report where you lose. A table where you win every column is a table nobody believes.
4. Say what you are not measuring. If you did not measure input latency, say so — do not let a throughput number imply it.

Warning: Do not publish a comparison that ranks other people's terminals without extreme care about methodology. Terminals make different trade-offs deliberately: foot optimizes for Wayland-native memory use, kitty for feature richness, alacritty for raw throughput. A table that reports "X is fastest" without saying "at this specific workload, on this machine, with this font" is not a measurement — it is a fight, and an uninformative one.

7. Known Traps

What Makes This Contribution-Worthy

The ecosystem genuinely lacks:

• A current cross-terminal conformance table
• A benchmark that separates parse from render from round trip
• A machine-readable capability matrix (which terminals implement ?2026? ?2027? OSC 133?)
• A regression suite terminal authors can run before releasing

You will have all four as a by-product. Publish them with an honest methodology section and they are useful to people who will never read your code.

Deliverables

• conformance.sh with 20+ probes, each with a rationale and a cited source.
• Reference captures for at least five terminals.
• compare.sh producing a markdown matrix with findings classified.
• criterion benchmarks separating parse / render / round trip / end-to-end.
• A hash-pinned corpus.
• The results table, with the full methodology block.
• A capability matrix: which terminals implement which modern proposals.
• Published, with a README that states plainly what is and is not measured.

Next: Project 8 — An Upstream Contribution