Testing Strategy

Six kinds of test. The point of this chapter is that each one catches something the others cannot, and knowing which is what lets you decide where to spend an afternoon.

The Six Kinds

Read the "cannot catch" column. It is the useful one, and it is the reason all six exist.

Where Each One Lives

src/**.rs           #[cfg(test)] unit tests — the invariant nearest the code
tests/golden/       *.ember + *.expected  — the specification
tests/harness.rs    the Backend trait — names NO backend
tests/golden.rs     the corpus, per backend
tests/differential.rs  eight lines; both backends over the whole corpus
tests/property.rs   proptest: round-trip, no-panic, metamorphic, backend agreement
tests/lua_compat.rs behind --features lua-compat; skips documented divergences
fuzz/fuzz_targets/  lex, parse, compile, run, bytecode_validate
benches/            criterion; not a test, but it fails CI on a regression

The One That Multiplies

Differential testing is the highest-leverage of the six because it is the only one whose value grows with every other test you write. A new golden case is automatically a differential case, forever, at zero marginal cost.

That property exists only because the harness was written before the second backend. It is the clearest example in the curriculum of an ordering decision paying compound interest.

And its blind spot is precise: shared code. Ember's backends share the lexer, the parser, Value, raw_eq, the arithmetic helpers, the table, the heap, and the collector. A bug in any of those is invisible to the comparison, and the mitigations are the other five kinds — specifically compatibility testing against lua, and property tests over the shared helpers.

Property Testing: What to Assert

Four properties, in increasing order of value:

#![allow(unused)]
fn main() {
// 1. Round-trip. Catches PRECEDENCE bugs automatically — the class hand-written
//    tests miss, because you test what you believe.
prop_assert_eq!(parse(&print_fully_parenthesized(&e)), e);

// 2. No panic, ever, on any input.
let _ = run_with_limits(&arbitrary_string, &Limits::strict());

// 3. Metamorphic: a relation between two runs, without knowing either answer.
//    Wrapping a program in `if true then … end` must not change its output.
//    Adding an unused local must not. `do … end` around a block must not.
prop_assert_eq!(run(&p), run(&format!("if true then\n{p}\nend")));

// 4. Differential over GENERATED programs — the union of 1–3 and the corpus.
prop_assert_eq!(interp.run(&p), vm.run(&p));
}

Metamorphic testing is the one people have not met. It asserts a relation rather than a value, so it works when you cannot predict the answer — which is exactly the situation for a generated program. SQLite uses it heavily (running a query with and without an index and requiring identical results), and it is the technique that partially covers differential testing's blind spot, because both sides of the relation go through the same shared code but along different paths.

Two rules for the generator, and violating the second makes the whole exercise worthless:

1. Terminate. Bound every loop with a literal; use the budget as a backstop, not a guarantee.
2. Produce output. Every generated program ends in print(...). A property test that passes because both runs printed nothing reports confidence you have not earned — so add the meta-test that asserts a sample of generated programs is non-silent.

Fuzzing: Five Targets, Three Assertions

fuzz_targets/lex.rs                arbitrary bytes → tokenize
fuzz_targets/parse.rs              arbitrary bytes → parse
fuzz_targets/compile.rs            arbitrary bytes → compile
fuzz_targets/run.rs                arbitrary bytes → execute under STRICT limits
fuzz_targets/bytecode_validate.rs  arbitrary bytes → deserialize + validate + (if valid) run

Each asserts exactly three things: no panic, no hang, no unbounded memory. None of them checks correctness — that is the other five kinds' job. Fuzzing checks that a runtime fed hostile bytes does not become a liability.

bytecode_validate is the most interesting one, and its assertion is a pairing:

#![allow(unused)]
fn main() {
fuzz_target!(|data: &[u8]| {
    if let Ok(proto) = deserialize(data) {
        if validate(&proto).is_ok() {
            // THE property: anything the validator accepts, the VM can execute
            // without panicking. That pairing is the actual security guarantee,
            // and neither half means much alone.
            let _ = Vm::new().run_proto(proto, &Limits::strict());
        }
    }
});
}

Commit the corpora. A crash reproducer is a regression test; a corpus that starts from zero on every CI run finds the same shallow bugs forever.

Compatibility Testing

#![allow(unused)]
fn main() {
#[test] // --features lua-compat
fn corpus_agrees_with_real_lua_where_we_do_not_diverge() {
    for case in corpus().iter().filter(|c| !c.diverges_from_lua) {
        assert_eq!(run_ember(&case.src), run_reference_lua(&case.src), "{}", case.name);
    }
}
}

lua is an independent oracle — written by other people, from the same specification — which makes it strictly stronger than a second implementation you wrote. It is also the only thing that covers differential testing's shared-code blind spot.

The coupling that keeps it honest: a case may opt out only by naming a documented divergence.

#![allow(unused)]
fn main() {
#[test]
fn every_lua_divergence_is_documented() {
    for case in corpus().iter().filter(|c| c.diverges_from_lua) {
        assert!(divergence_ids_in_appendix().contains(&case.divergence_id),
                "{} opts out with an undocumented id", case.name);
    }
}
}

Without that test, "skip the Lua comparison" becomes the path of least resistance and appendix/lua-differences.md rots.

What Good Coverage Looks Like

cargo llvm-cov --workspace --html

Do not chase a percentage. Chase specific uncovered things:

• Every arm of the VM's dispatch match. An uncovered arm is an opcode nobody tested.
• Every arm of arith::binary and cmp::*. These are the shared helpers differential testing cannot check, so coverage is the substitute.
• Every ErrorKind construction site. An error you never produced in a test is an error whose message you have never read.
• Every branch of trace_children. An untraced edge is a GC bug.

Four targeted lists beat one number.

Things to Notice

• The "cannot catch" column is the useful one. It tells you which test to write next.
• Differential testing's value compounds, and only because of an ordering decision made in Lab 8.
• Metamorphic properties partially cover the shared-code blind spot, because both sides traverse the same code by different paths.
• A property test over silent programs is worse than no test. Add the meta-test.
• The validator/VM pairing is the actual security guarantee. Neither half means much alone.
• lua is an independent oracle, which is a strictly stronger thing than a second implementation you wrote.
• Chase uncovered things, not a coverage number.

Validation / Self-check

1. Name the six kinds of test and, for each, one thing it cannot catch.
2. Why does differential testing's value compound, and which decision made that possible?
3. What is metamorphic testing, and why does it partially cover the shared-code blind spot?
4. Give the two rules for a program generator, and the symptom of violating the second.
5. What three things does a fuzz target assert, and what does it deliberately not check?
6. State the validator/VM pairing property and why neither half suffices.
7. Why is lua a stronger oracle than your own tree walker? What does it not cover?
8. Give four specific coverage targets that are more useful than a percentage.

Next: Observability.