The Roadmap: Fifteen Milestones

Fifteen milestones, M0 through M14, delivered by thirty labs across seven sections. Each milestone has a definition of done that is observable — something you can run and look at, not "I understand it now."

Track them in docs/learning/progress.md. Milestones are the unit you tell other people about; labs are the unit you work in.

The Whole Thing on One Page

flowchart TD
  M0["M0 · Mental model"] --> M1["M1 · Arithmetic end to end"]
  M1 --> M2["M2 · Values and type errors"]
  M2 --> M3["M3 · Variables and scope"]
  M3 --> M4["M4 · Control flow"]
  M4 --> M5["M5 · Functions, frames, recursion — REFERENCE INTERPRETER DONE"]
  M5 --> M6["M6 · Bytecode and disassembler"]
  M6 --> M7["M7 · The compiler"]
  M7 --> M8["M8 · The VM — and it agrees with M5"]
  M8 --> M9["M9 · Tables"]
  M9 --> M10["M10 · Closures and upvalues"]
  M10 --> M11["M11 · Garbage collection and strings"]
  M11 --> M12["M12 · Multiple returns, varargs, metatables — LANGUAGE DONE"]
  M12 --> M13["M13 · Embedding, host objects, stdlib, modules"]
  M13 --> M14["M14 · Sandboxing, diagnostics, REPL, observability, performance, JIT"]
  M14 --> C["Capstone · Recommendation Policy Engine + end-to-end trace"]

Note: M5 and M8 are the two checkpoints that matter most. M5 means you have a complete, correct language you can trust. M8 means you have a second implementation that provably agrees with the first. Everything after M8 is built on that agreement — which is why Lab 12, the least glamorous lab in the curriculum, is also the highest-leverage one.

Section 1 — From Characters to Trees

M1 · Arithmetic, end to end

Labs 1–3. Lexer, Pratt parser, tree-walking evaluator.

$ echo '10 + 20 * 3' > t.ember && ember run t.ember
70

Done when:

• ember tokens, ember ast, and ember run all work on 10 + 20 * 3.
• Precedence and associativity are correct for + - * / % ^ and unary minus, with a test per rule. 2 ^ 3 ^ 2 is 512, not 64 — ^ is right-associative.
• Every token and every AST node carries a Span.
• A syntax error prints the source line with a caret under the offending span.
• docs/learning/01-lexer.md, 02-parser.md, 03-ast.md written.

M2 · Values and honest type errors

Lab 4. nil, booleans, integers, floats, strings; comparison; truthiness; the coercion table.

There are no function calls until M5, so print does not exist yet — ember run -e prints the value of the returned expression, and --types shows its type and numeric subtype.

$ ember run --types -e 'return 1 == 1.0'
true (boolean)
$ ember run --types -e 'return 3 // 2'
1 (number: integer)
$ ember run -e 'return "a" .. "b"'
ab
$ ember run -e 'return "x" * 2'
<argv>:1:8: error: attempt to multiply a string value

   1 │ return "x" * 2
     │        ^^^

Done when:

• Value is defined, is Copy, and size_of::<Value>() is asserted in a test.
• The integer/float rules match the warm-up's observations, with a test table covering every operator/subtype pair.
• Integer/float equality is exact — i64::MAX == i64::MAX + 0.0 is false.
• Type errors name the operator's verb and the offending type, and their span covers the offending operand, not the whole expression.
• docs/adr/ADR-004-value-representation.md, ADR-005-integer-float-split.md, and ADR-009-bytewise-string-ordering.md written.

Section 2 — The Reference Interpreter

M3 · Variables and scope

Lab 5. local, globals, assignment, blocks, shadowing.

Done when:

• Shadowing works: an inner local x does not disturb the outer one, and the inner is visible only inside its block.
• local x = x reads the outer x — the initializer is evaluated before the new binding exists. (Get this wrong and you have written JavaScript's temporal dead zone by accident.)
• Reading an undefined global yields nil; reading an undefined local is a compile error, because there is no such thing.
• You can state, in writing, why the environment being a HashMap is wrong, before Section 3 tells you.

M4 · Control flow

Lab 6. if/elseif/else, while, numeric for, generic for (deferred to M9), break, return.

Done when:

• Only nil and false are falsy. 0 and "" are true. Tested explicitly, because every language disagrees here and yours must be documented.
• and/or short-circuit and return operands, not booleans: nil or 5 is 5.
• A numeric for creates a fresh binding per iteration (warm-up Experiment 3).
• break exits only the innermost loop; nested-loop test present.

M5 · Functions, frames, recursion — the reference interpreter is complete

Labs 7–8. Function declarations, calls, parameters, return, recursion, and a depth limit.

$ ember run --interp fib.ember      # fib(25) by naive recursion
75025
$ ember run --interp deep.ember     # 100_000-deep recursion
error: stack overflow (call depth limit 200 exceeded)
  in function 'recurse'   deep.ember:2

Done when:

• Recursion works, mutual recursion works, and a runaway recursion returns an ErrorKind::Limit error rather than aborting the process.
• The golden corpus exists (tests/golden/) with at least 25 programs, each with expected output.
• cargo test is green and the corpus is the thing that proves it.
• docs/adr/ADR-003-keep-the-tree-walker.md written — before you build the VM, so it is a prediction rather than a justification.

This is the checkpoint. You have a working language. It is slow and it has no tables, but its semantics are defined, tested, and — critically — frozen as a reference. Everything from here is a second implementation that must agree with this one.

Section 3 — Bytecode and the Virtual Machine

M6 · Bytecode and a disassembler

Lab 9. The Op enum, the Chunk, the constant pool, the line table, and the disassembler.

$ ember disassemble t.ember
== chunk: t.ember ==
constants: [0] 10  [1] 20  [2] 3
offs  line  op            operands  comment
0000     1  LOAD_CONST    0         ; 10
0002     1  LOAD_CONST    1         ; 20
0004     1  LOAD_CONST    2         ; 3
0006     1  MUL
0007     1  ADD
0008     1  RETURN

Done when:

• Every opcode is documented in the opcode reference with operands, stack before, stack after, and possible errors.
• The disassembler is written before the VM, and used to check the compiler's output.
• A Chunk round-trips through the disassembler for every golden program without panicking.

M7 · The compiler

Lab 10. AST → Chunk. Scopes, slot allocation, jump emission and patching, constant pooling.

Done when:

• Every golden program compiles without error.
• Jump patching is correct for if/elseif/else, while, and break — verified by reading the disassembly, by hand, for at least three programs. Do this once manually; it is worth an hour.
• Local slots are reused after a block ends (check with ember disassemble on nested blocks).
• Constants are deduplicated: 1 + 1 has one constant, not two.

M8 · The VM — and it agrees with the tree walker

Labs 11–12. The dispatch loop, frames, the value stack, ember trace, and differential testing.

$ ember trace t.ember
ip    op            stack before      stack after
0000  LOAD_CONST 0  []                [10]
0002  LOAD_CONST 1  [10]              [10, 20]
...
$ cargo test --test differential
running 47 tests ... ok

Done when:

• Every golden program produces byte-identical output under --interp and the VM.
• The differential test runs automatically over the entire corpus, so every future test is a differential test.
• A proptest generator emits random valid programs and asserts backend agreement.
• ember trace prints ip, opcode, and the stack before and after each instruction.
• The instruction budget check lives in the fetch position and has a test proving a while true do end terminates with ErrorKind::Limit.
• docs/adr/ADR-002-stack-vm.md written.

Section 4 — Objects, the Heap, and the Collector

M9 · Tables

Lab 13. The hybrid table: array part plus insertion-ordered hash part. Constructors, t.k, t[k], #t, pairs, ipairs.

Done when:

• t.name and t["name"] take the same path and are indistinguishable.
• The array part is used for dense integer keys, the hash part for everything else, and a test shows the transition on rehash.
• A float key with an exact integer value normalizes: t[1.0] and t[1] are the same slot.
• pairs iterates in insertion order, deterministically, and there is a test that would fail under a randomly-seeded hasher.
• Table identity works: {} ~= {}, and a = b makes them alias.
• docs/adr/ADR-008-deterministic-iteration.md written.

M10 · Closures and upvalues

Lab 14. Closures, capture analysis in the compiler, open/closed upvalues in the VM.

Done when:

• Warm-up Experiment 3 produces identical results in Ember and Lua.
• The compiler resolves each free variable to a local, an upvalue, or a global, and the disassembly shows GET_UPVAL/SET_UPVAL where you expect them.
• Open upvalues are shared while the frame lives, and closed when it returns — with a test that creates two closures over one variable and asserts they still share it after the return.
• ember trace --upvalues shows the open-upvalue list and the moment each one closes.

M11 · Garbage collection and strings

Labs 15–16. The slot-table heap, handles, mark and sweep, allocation accounting, thresholds; then string interning, benchmarked.

$ ember run --trace-gc alloc.ember
gc: begin   heap=2.4MB objects=51203
gc: marked  roots=1841 reachable=1205
gc: swept   freed=50000 heap=0.4MB  in 3.1ms

Done when:

• A cycle is collected. This is the test that justifies the whole subsystem.
• The "forgot a root" bug is deliberately introduced, observed, and fixed — and the fix is a root-set enumeration that lives in one function so it cannot drift.
• Allocation accounting drives the collection threshold, and the threshold policy is documented.
• collectgarbage("count") and a heap census by object type are available to scripts and to ember --stats.
• Interning is added after a benchmark showed string comparison or hashing mattered, and the benchmark delta is recorded in docs/learning/14-performance.md.
• docs/adr/ADR-006-tracing-gc-with-handles.md and ADR-007-string-interning.md written.

M12 · Multiple returns, varargs, metatables — the language is done

Labs 17–18.

Done when:

• Every case in warm-up Experiment 4 matches Lua exactly, including (f()) truncation and select('#', ...).
• __index (table and function forms), __newindex, __call, __tostring, __eq, __lt, and the arithmetic metamethods work, with a metamethod-lookup depth limit that prevents an __index chain from hanging the VM.
• An Account-style object-oriented example from the warm-up runs unchanged.
• Differential tests still pass across the whole corpus. (They will break. That is the point: multiple returns are where the two backends are most likely to diverge.)

Section 5 — The Host Boundary

M13 · Embedding, host objects, the standard library, modules

Labs 19–22.

#![allow(unused)]
fn main() {
let mut engine = Engine::new();
engine.register_function("log", |_ctx, args| { println!("{args:?}"); Ok(Value::Nil) })?;
engine.execute(POLICY_SOURCE)?;
let score: f64 = engine.call("score", (user, article))?;
}

Done when:

• Engine::new/execute/call/set_global/get_global/register_function all work and are documented with #[doc] examples that run under cargo test --doc.
• A registered Rust function can call back into Ember (re-entrancy) without a borrow-checker workaround that leaks unsafe.
• Host values are marshaled both ways via a ToValue/FromValue pair, with a clear error when a conversion fails.
• A Rust struct is exposed as userdata with field access from script, and the GC traces it correctly.
• print, type, assert, error, pcall, tostring, tonumber, ipairs, pairs, and math.*, string.*, table.* exist, and no filesystem, network, process, or environment access is reachable by default.
• require works through a host-supplied resolver, with a cache and a cyclic-import error.
• docs/adr/ADR-010-host-controlled-modules.md and ADR-011-send-sync.md written.

Section 6 & 7 — Production and Performance

M14 · Sandboxing, diagnostics, tooling, observability, performance, JIT

Labs 23–30. This is the largest milestone and it splits naturally into two halves.

Half one — production (Labs 23–27), done when:

• Every threat in the model has a countermeasure or a written "we do not defend against this": infinite loops, infinite recursion, huge allocations, huge strings, error amplification, expensive host callbacks.
• Diagnostics render source, span, caret, and a traceback — for compile and runtime errors.
• ember repl works, with multi-line input, .help, .disasm, .stats, and no way to panic the process.
• Fuzz targets for lexer, parser, compiler, VM, and bytecode validation each run 10 minutes clean, corpus committed.
• No panic from any script input. A fuzz-found panic is a bug, not a curiosity.
• engine.stats() reports instructions executed, allocations, live objects, GC runs, GC pause total, call count, and peak depth.
• docs/limitations.md and the stated production profile are written and honest.

Half two — performance (Labs 28–30), done when:

• benches/ has a baseline recorded before any optimization, in the repo.
• Each optimization is documented as: baseline → hypothesis → change → measurement → tradeoff. An optimization with no measured win is reverted, and the reversion is recorded too.
• An inline cache for table field access exists, with monomorphic/polymorphic/megamorphic states and a correctness test for cache invalidation on table shape change.
• At least one specialized opcode (ADD_INT or similar) exists with type-feedback-driven rewriting and a guard.
• A Cranelift JIT compiles at least function add(a, b) return a + b end to native code, with a guard and a working deoptimization path.
• Three-way benchmark recorded: tree interpreter vs. VM vs. JIT, on the same programs, with the command that reproduces it.
• docs/adr/ADR-014-cranelift.md written.

The Capstone

The Recommendation Policy Engine. Rust owns candidate retrieval, user and article data, metrics, execution limits, logging, and policy loading. Ember owns ranking, boosts, penalties, business rules, and experiments.

Plus the mandatory end-to-end trace: one non-trivial script, followed through tokens, AST, bytecode, constants, stack, frames, table accesses, closure captures, allocations, GC roots, and the return into Rust. If you cannot produce that document, you have not finished, regardless of what runs.

And the honest chapter: when not to embed a language — the cases where TOML is enough, or where plain Rust is better. Being able to argue against the thing you just spent four months building is the mark of an engineer rather than an enthusiast.

Lab Index

Validation / Self-check

1. Which two milestones are the checkpoints, and what property does each one establish?
2. Why must ADR-003 be written before the VM exists?
3. What makes M8's definition of done stronger than "the VM runs my test programs"?
4. In M11, why is interning added only after a benchmark?
5. Which milestone would you cut if you had half the time, and what would you lose?
6. What is the mandatory deliverable that is not code, and why is it mandatory?

Next: The Weekly Learning Plan.