Project 7: A Lua Compatibility Harness

Run a real Lua test corpus against Ember, classify every failure, and turn appendix/lua-differences.md from a list you maintained by hand into a list that is checked.

Effort: a week, most of it classification. Value: an independent oracle at scale — the only thing that covers differential testing's blind spot.

Why an Independent Oracle Matters

Your differential test compares two implementations you wrote from one understanding. If you misread the manual, both backends are wrong together and the comparison passes.

Lua's own test suite was written by other people, from the specification, to find bugs in an implementation that is not yours. That is a categorically stronger check, and it is the reason this project is worth a week.

The Corpora, in Increasing Order of Value

Start with the third and work toward the fourth. The official suite will fail in hundreds of places on a partial implementation, and triaging that from zero is demoralizing. Rosetta Code gives you a gradient.

The Harness

#![allow(unused)]
fn main() {
pub enum Outcome {
    Match,                                    // identical stdout
    Divergence { id: String },                // a DOCUMENTED difference
    Unsupported { feature: String },          // a documented LIMITATION
    Bug { ember: String, lua: String },       // ← the ones that matter
    Skipped { reason: String },               // uses debug/io/os/coroutine
}

fn classify(case: &Path) -> Outcome { /* run both, compare, categorize */ }
}

The classification is the work, and it must be mechanical rather than by hand, or it rots:

#![allow(unused)]
fn main() {
// Every Divergence must name an id that exists in the appendix. Every
// Unsupported must name a line in docs/limitations.md. A failure that fits
// neither is a BUG, and the harness must not let you quietly reclassify it.
#[test]
fn every_non_bug_outcome_cites_a_document() {
    for (case, outcome) in run_corpus() {
        match outcome {
            Outcome::Divergence { id } =>
                assert!(divergence_ids().contains(&id), "{case:?} cites unknown divergence {id}"),
            Outcome::Unsupported { feature } =>
                assert!(limitations().contains(&feature), "{case:?} cites unknown limitation"),
            Outcome::Skipped { reason } =>
                assert!(SKIP_REASONS.contains(&reason.as_str()), "{case:?} ad-hoc skip"),
            _ => {}
        }
    }
}
}

That test is the project's spine. Without it, "this test fails because we're different" becomes the path of least resistance and the compatibility claim becomes meaningless.

The Report

$ cargo run --bin lua-compat -- corpus/

--- Ember / Lua 5.4 compatibility report ---
  total cases            1,247
  match                    904   72.5%
  documented divergence     38    3.0%     ← each cites an appendix id
  unsupported feature      241   19.3%     ← each cites docs/limitations.md
  skipped (needs io/debug)   61    4.9%
  BUGS                       3    0.2%     ← ← ← the ones that matter

--- unsupported, by feature ---
  coroutine.*              118
  string patterns (%b %f)   47
  goto / labels             31
  utf8.*                    22
  weak tables               15
  os.* / io.*                8

--- bugs ---
  rosetta/sort_stability.lua   table.sort is stable in Ember, not in Lua
  wiki/varargs_nil.lua         select('#') differs with trailing nils in a nested call
  suite/strings_fmt.lua        string.format('%5.2f') width handling

That report is a genuinely useful artifact. It tells a prospective user exactly what fraction of real Lua code runs, and — more usefully — which features account for the gap. "19% unsupported, of which half is coroutines" is actionable; "72% compatible" is not.

What You Will Find

Predictions, and finding out which are right is the exercise:

The last row is the trap. A harness that compares stderr will report thousands of failures and teach you nothing. Compare stdout; for error cases, compare whether it errored and, where you can, the error kind.

Deliverables

• A harness running a corpus under both runtimes, classifying each result.
• The citation test: every non-bug outcome names a document, mechanically checked.
• A corpus of at least 500 cases, committed or fetched reproducibly.
• The report, generated by a command, committed as docs/lua-compat.md.
• Every Bug fixed or promoted to a documented divergence with a reason. Three outcomes, no fourth.
• appendix/lua-differences.md reconciled: nothing documented that the harness does not exercise, nothing diverging that is not documented.
• A compatibility statement in the README: "Ember runs N% of the Lua 5.4 corpus at docs/lua-compat.md; the gap is mostly X and Y."

Where It Gets Hard

1. Test isolation. Lua's own suite has files that depend on earlier files' globals and on collectgarbage internals. Run them in the intended order or skip with a reason.
2. Nondeterminism in the reference. Lua's pairs order and #-with-holes are unspecified, so a case relying on either is not a valid comparison. Detect and skip, with a reason.
3. Timing and environment. os.clock and os.time cases cannot be compared. Skip.
4. The temptation to reclassify. A Bug that is inconvenient is not a Divergence until you write the reason. The citation test is what stops you.

Where to Read

git clone https://github.com/lua/lua
ls lua/testes/          # the official suite: all.lua, strings.lua, closure.lua, ...

• lua/testes/ — the official suite. Read all.lua first to see how it is structured, and note how much of it probes implementation internals rather than language semantics.
• The Lua 5.4 Reference Manual, §8 ("Incompatibilities with the Previous Version") — a model for how to write a divergence list.
• mlua's and hematita's compatibility notes, for how other Rust projects state their scope.

Validation / Self-check

1. Why is lua a stronger oracle than your own tree walker? What exactly does it cover?
2. What are the four outcome categories, and which one must never be silently reclassified?
3. Why must the citation test exist? What happens without it?
4. Why compare stdout rather than stderr?
5. Which two Lua behaviors are unspecified, and how did you handle cases that depend on them?
6. What is your compatibility percentage, and — more usefully — what accounts for the gap?
7. How many genuine bugs did the harness find that your own tests did not?

Next: Project 8 — An Upstream Contribution.