The Evaluation Rubric
How to grade yourself. Six dimensions, four levels each. Be harsh — the point is to find the thing you skipped, not to feel finished.
How to Use It
Score each dimension honestly, then look only at your lowest one. The weakest dimension is your next two weeks, and a runtime that is excellent in five dimensions and absent in the sixth is a runtime nobody should depend on.
| Level | Meaning |
|---|---|
| 0 — Absent | Not done, or done and untested |
| 1 — Works | It runs on the happy path |
| 2 — Correct | Tested, including edge cases and failure modes |
| 3 — Defensible | You can explain every decision, its alternatives, and what you gave up |
Level 3 is the target for the whole curriculum, and it is not about polish. A level-2 garbage
collector collects cycles and passes --gc-stress. A level-3 one also has an ADR explaining why not
reference counting, a measured pause distribution, and a written statement of what an incremental
collector would change.
Dimension 1: The Language
| Level | Criteria |
|---|---|
| 0 | Arithmetic and variables |
| 1 | Control flow, functions, tables, closures |
| 2 | + upvalues, metatables, multiple returns, varargs; the golden corpus passes; every case in warm-up Experiments 1–8 matches Lua |
| 3 | + every divergence from Lua 5.4 is documented with a reason in appendix/lua-differences.md, and the lua-compat test enforces the coupling |
The level-3 test: pick three divergences at random from your appendix and explain each without
looking. If one of them is "we ran out of time", it is a limitation, not a divergence, and it belongs
in docs/limitations.md instead.
Dimension 2: The Implementation
| Level | Criteria |
|---|---|
| 0 | A tree-walking interpreter |
| 1 | + a bytecode compiler and VM that runs the corpus |
| 2 | + the two backends agree on the whole corpus and on generated programs; the validator passes on every chunk; no panics under fuzzing |
| 3 | + you can explain, for each of stack-vs-register, tagged-union-vs-NaN-boxing, handles-vs-pointers, and tracing-vs-refcounting, what you chose, what you rejected, and the measurement or principle that decided it |
The level-3 test: the differential test found N bugs. Can you name three of them and say which backend was wrong and why? If differential testing found nothing, either your corpus is too small or you did not run it — because it always finds something.
Dimension 3: Correctness Discipline
| Level | Criteria |
|---|---|
| 0 | It works on the examples |
| 1 | Unit tests and a golden corpus |
| 2 | + differential, property, and fuzz testing; corpora committed; CI runs all of it |
| 3 | + you can state what each kind of test cannot catch, and name the mitigation for each blind spot |
The level-3 test: what does differential testing not catch, and what covers that gap? If the
answer is not "shared code — covered by the lua comparison and property tests over the shared
helpers", re-read
the reference-implementation chapter.
Dimension 4: Production Readiness
| Level | Criteria |
|---|---|
| 0 | It runs scripts |
| 1 | + limits and a capability-gated library |
| 2 | + diagnostics with spans and tracebacks, a REPL that cannot crash, all 15 threat tests, docs/sandboxing.md with non-goals |
| 3 | + a stated production profile in the README, docs/limitations.md written continuously, a semver policy, and an answer to "is this production-ready?" in two halves |
The level-3 test: state your production profile from memory, then state one thing it explicitly does not cover. If the second half is hard, the first half is marketing.
Dimension 5: Performance Engineering
| Level | Criteria |
|---|---|
| 0 | It is fast enough |
| 1 | Benchmarks exist |
| 2 | + a committed baseline, an opcode profile, and at least one optimization measured both ways |
| 3 | + at least one optimization reverted and recorded, and every kept one documented as baseline → hypothesis → change → measurement → tradeoff |
The level-3 test: name an optimization you reverted and the number that made you revert it. If you have never reverted one, you have either been extraordinarily lucky or you have not been measuring.
Dimension 6: Understanding
The one that matters, and the one no test can check. Answer these out loud, without notes.
| # | Question | Level 3 means |
|---|---|---|
| 1 | How does a programming language execute? | You name every representation and what each deletes |
| 2 | Why is bytecode faster than a tree walk? | Your answer is about name resolution and memory layout, not "it's compiled" |
| 3 | What is a call frame? | Three numbers, and you can draw the stack across a call |
| 4 | How does a closure outlive its frame? | You can draw the open→closed transition and say what moves |
| 5 | Why can't reference counting collect a cycle? | You draw one, and you name the closure case that makes it unavoidable |
| 6 | What are the roots of a collector? | All nine, and the symptom of missing one |
| 7 | Why is dynamic field access expensive, and what fixes it? | Guard plus cached answer; mono/poly/megamorphic |
| 8 | What is deoptimization for? | Speculation requires a way back, and the way back is the engineering |
| 9 | How do a GC and a JIT cooperate? | Safepoints and stack maps; and why a non-moving collector halves the problem |
| 10 | How does a Rust host own a cyclic object graph safely? | Handles, generations, and the three ways past the borrow checker |
| 11 | What can a sandbox promise? | Your controls, and your non-goals, and why the second list is the credible one |
| 12 | When should you not embed a language? | The four questions, in order |
Twelve questions. If you can answer all twelve without notes, the curriculum worked — and that is a stronger claim than anything in your repository.
The Capstone Itself
| Level | Criteria |
|---|---|
| 0 | A policy runs |
| 1 | + 10,000 candidates within a budget |
| 2 | + live reload with validation, deterministic ranking across replicas, per-evaluation stats |
| 3 | + A/B comparison by instruction delta, an honest p99, and the end-to-end trace |
The end-to-end trace is mandatory and it is not a summary. It is the single artifact that demonstrates every subsystem working together, produced by your own tools, on your own script.
The Portfolio
The eight capstone projects are optional and they are where the depth compounds. Two of them are the natural next steps for most readers:
- Register VM — because ADR-002 promised a measurement and this is where you take it.
- Upstream contribution — because a
merged PR to
mlua,rune,koto, orpiccolois the thing that turns "I built a toy" into "I contribute to language runtimes".
Scoring Yourself
Language 0 1 2 3
Implementation 0 1 2 3
Correctness 0 1 2 3
Production 0 1 2 3
Performance 0 1 2 3
Understanding 0 1 2 3
Capstone 0 1 2 3
Look only at the lowest number. That is the next two weeks. A 3-3-3-3-3-3-1 runtime is a runtime with a hole, and the hole is where it will fail.
And if the lowest is Understanding, go back to the twelve questions and find which one you cannot answer. Then re-read that chapter, redo that lab's Trace, and answer it again. The code is the scaffolding; the answers are the building.
Next: The Capstone Projects.