The End-to-End Trace
This document is mandatory. If you cannot produce it for your own runtime, you have not finished the curriculum, regardless of what runs.
One non-trivial script, followed through every representation and every subsystem: characters, tokens, AST, bytecode, constants, protos, upvalue descriptors, the value stack, call frames, the heap, closure captures, open and closed upvalues, userdata field access, GC roots, allocation accounting, and the return into Rust.
Every command below is one you run. Every listing is one your own tools produce.
The Script
-- scorer.ember
local BOOST = 1.4
local function make_scorer(threshold)
local hits = 0
return function(article)
local s = article.semantic_score
if article.age_hours < threshold then
s = s * BOOST
hits = hits + 1
end
return s, hits
end
end
local score = make_scorer(6)
return score(candidates[1])
Fourteen lines, and they exercise: locals, a float constant, a closure factory, three kinds of upvalue capture, a captured variable that is mutated, userdata field access, arithmetic, a conditional, multiple returns, an indexed global, and a return into Rust.
The host supplies candidates as userdata
(Lab 20) holding one Article:
#![allow(unused)] fn main() { Article { id: 42, semantic_score: 0.5, age_hours: 2.0, topic: "sports" } }
Layer 1: Characters
byte 0 22 32
│ │ │
l o c a l B O O S T = 1 . 4 \n \n l o c a l f u n c t i o n …
Nothing has been decided yet. This is the ground truth, and it is also the last representation that contains the comment on line 1.
Layer 2: Tokens
$ ember tokens scorer.ember | head -12
# span kind text
0 17..22 Local "local"
1 23..28 Ident("BOOST") "BOOST"
2 29..30 Assign "="
3 31..34 Float(1.4) "1.4"
4 36..41 Local "local"
5 42..50 Function "function"
6 51..62 Ident("make_scorer") "make_scorer"
7 62..63 LParen "("
8 63..72 Ident("threshold") "threshold"
9 72..73 RParen ")"
10 76..81 Local "local"
...
Lost: the comment, the whitespace, the line structure.
Gained: classification, and a Span on every token — the thread that will reach a caret in
Layer 12.
Note token 3: the lexer already decoded 1.4 to an f64. Nothing downstream re-parses it, and
the "is it an integer or a float?" decision was made here, at
Lab 1's number scanner.
Layer 3: The AST
$ ember ast scorer.ember
Block @17..262
├── Local BOOST @17..34
│ └── Float(1.4) @31..34
├── LocalFunction make_scorer @36..205
│ └── Function [threshold] vararg=false @51..205
│ └── Block @76..201
│ ├── Local hits @76..90
│ │ └── Int(0) @89..90
│ └── Return @93..197
│ └── Function [article] vararg=false @100..197
│ └── Block @122..191
│ ├── Local s @122..156
│ │ └── Index @132..156
│ │ ├── Name article @132..139
│ │ └── Str "semantic_score" @140..156
│ ├── If @161..…
│ │ ├── arm 0 cond: Binary(Lt) @164..191
│ │ │ ├── Index @164..181
│ │ │ │ ├── Name article
│ │ │ │ └── Str "age_hours"
│ │ │ └── Name threshold @184..193
│ │ └── arm 0 body: Block
│ │ ├── Assign s = Binary(Mul)[Name s, Name BOOST]
│ │ └── Assign hits = Binary(Add)[Name hits, Int(1)]
│ └── Return [Name s, Name hits]
├── Local score @207..236
│ └── Call @221..236
│ ├── Name make_scorer
│ └── args: [Int(6)]
└── Return @238..262
└── Call
├── Name score
└── args: [Index[Name candidates, Int(1)]]
Lost: keywords, parentheses, end, the token order.
Gained: structure. article.semantic_score became Index[Name, Str] — desugared at parse time
(the AST chapter's decision 2) — and
the Str node's span points at semantic_score, not at the whole expression. That is what makes the
caret land correctly in Layer 12.
Three things to notice. LocalFunction is a distinct node from Local + Function, because
the binding must exist before the body is compiled.
The If has a flat arms vector. And there is no Paren node anywhere, because nothing here needs
truncation.
Layer 4: Capture Analysis
Before any bytecode exists, the compiler resolves every free name. This is the pass that produces the upvalue descriptors, and it runs during code generation (Lab 14).
main chunk locals: BOOST(0) make_scorer(1) score(2)
BOOST is marked CAPTURED ← by make_scorer
make_scorer params: threshold(0) locals: hits(1)
threshold CAPTURED, hits CAPTURED ← by the inner function
upvalues: [0] BOOST ← from parent LOCAL slot 0
<anonymous scorer> params: article(0) locals: s(1)
upvalues: [0] threshold ← from parent LOCAL slot 0
[1] hits ← from parent LOCAL slot 1
[2] BOOST ← from parent UPVALUE 0 ★ TRANSITIVE
★ is the one to study. make_scorer never mentions BOOST in its body, and it carries an
upvalue for it anyway — because the inner function needs it and capture is transitive. This is
exactly the three-level case from
the closures chapter, and your
disassembler prints it:
$ ember disassemble scorer.ember | grep -A3 'proto'
--- proto [0]: make_scorer ---
upvalues: [0] BOOST (parent local, slot 0)
--- proto [0][0]: <anonymous> ---
upvalues: [0] threshold (parent local, slot 0)
[1] hits (parent local, slot 1)
[2] BOOST (parent upvalue, index 0) ← forwarded
Layer 5: Bytecode
$ ember disassemble scorer.ember
== scorer.ember == (14 instructions, 2 constants, 1 proto)
constants:
[ 0] 1.4
[ 1] "candidates"
offs line op operands comment
0000 2 LOAD_CONST 0 ; 1.4 → BOOST = slot 0
0002 4 CLOSURE 0 ; make_scorer → slot 1
0004 15 GET_LOCAL 1 ; make_scorer
0005 | LOAD_INT 6
0006 | CALL 1 2 ; 1 arg, want 1 result → score = slot 2
0008 16 GET_LOCAL 2 ; score
0009 | GET_GLOBAL 1 ; "candidates"
0011 | LOAD_INT 1
0012 | GET_INDEX
0013 | CALL 1 0 ; 1 arg, want ALL (biased 0 = MULTRET)
0015 | RETURN 0 ; return ALL
--- proto [0]: make_scorer --- (4 instructions, 0 constants, 1 proto)
upvalues: [0] BOOST (parent local, slot 0)
0000 5 LOAD_INT 0 ; hits = slot 1
0001 6 CLOSURE 0 ; <anonymous>
0003 13 RETURN 2 ; 1 value (implicitly closes upvalues ≥ base)
--- proto [0][0]: <anonymous> --- (21 instructions, 2 constants, 0 protos)
upvalues: [0] threshold (parent local, 0) [1] hits (parent local, 1) [2] BOOST (parent upvalue, 0)
constants: [ 0] "semantic_score" [ 1] "age_hours"
0000 6 GET_LOCAL 0 ; article
0001 | GET_FIELD 0 ; "semantic_score" → s = slot 1
0003 7 GET_LOCAL 0 ; article
0004 | GET_FIELD 1 ; "age_hours"
0006 | GET_UPVAL 0 ; threshold
0007 | LT
0008 | JUMP_IF_FALSE 0018
0010 8 GET_LOCAL 1 ; s
0011 | GET_UPVAL 2 ; BOOST
0012 | MUL
0013 | SET_LOCAL 1 ; s
0014 9 GET_UPVAL 1 ; hits
0015 | LOAD_INT 1
0016 | ADD
0017 | SET_UPVAL 1 ; hits
0018 11 GET_LOCAL 1 ; s
0019 | GET_UPVAL 1 ; hits
0020 | RETURN 3 ; 2 values
Lost: every name, the tree, the nesting. BOOST is constant 0 then slot 0 then upvalue 2,
depending on who is looking.
Gained: a linear order and numbers.
Five things worth reading twice:
LOAD_CONST 0for1.4butLOAD_INT 6for6. The hybrid from the encoding chapter: small integers go inline; floats and strings go in the pool.- No
SET_LOCALafterLOAD_CONST 0.BOOSTis stack position 0 — declarations do not need an assignment, which is the thing that confuses everyone in Lab 10. CALL 1 2versusCALL 1 0. One operand differs and it is the entire multiple-return rule: the first call is not last in a list so it is truncated to one value; the second is the returned expression so it stays open.GET_UPVAL 1thenSET_UPVAL 1forhits = hits + 1. The closure mutates a captured variable. That single pair is why capture must be by variable and not by value.JUMP_IF_FALSE 0018targets the instruction after theifbody, and0018is where both paths converge with the same stack depth — which the validator checked before anything ran.
Layer 6: Validation
$ ember disassemble --validate scorer.ember
validate: 3 protos, 39 instructions
constant indices .......... ok (4 checked)
name operands are strings . ok (3 checked)
jump targets .............. ok (1 checked, max 0018 <= 0021)
slot numbers .............. ok (max 1 < max_stack 3)
upvalue indices ........... ok (max 2 < 3)
proto indices ............. ok
ends in RETURN ............ ok (3/3)
line table length ......... ok (39 == 39)
stack depth abstraction ... ok (join at 0018: both paths depth 0)
Nine rules, one pass. The last one is only tractable because Ember is a stack machine, and it is what the JVM and WebAssembly verifiers do for the same reason.
Layer 7: Execution — the Main Chunk
$ ember trace scorer.ember
dep ip op value stack (top on the right)
1 0000 LOAD_CONST 0 ; 1.4 [1.4] ← BOOST is slot 0
1 0002 CLOSURE 0 [1.4, <closure make_scorer>] ← ★ ALLOCATION 1
1 0004 GET_LOCAL 1 [1.4, <fn>, <fn>]
1 0005 LOAD_INT 6 [1.4, <fn>, <fn>, 6]
1 0006 CALL 1 2 → push frame
★ At CLOSURE 0, three things happened:
heap:
[0] Closure { proto: make_scorer, upvals: [→ upvalue#1] } ALLOCATION 1 (32 B)
[1] Upvalue::Open(stack_index 0) ALLOCATION 2 (16 B)
open upvalue list: [Open(0)] ← slot 0 is BOOST, still live on the stack
bytes_allocated: 48
The closure captured BOOST by variable: an open upvalue pointing at stack index 0. BOOST is
still on the stack, still readable as GET_LOCAL 0 by the main chunk, and the closure sees the same
cell.
Layer 8: The Call — Frames
before CALL 1 2 after
value stack value stack
┌──────────────────────────┐ ┌──────────────────────────┐
0 │ 1.4 (BOOST) │ 0 │ 1.4 │
1 │ <fn make_scorer> │ 1 │ <fn make_scorer> │
2 │ <fn make_scorer> ← callee 2 │ <fn> ← ret_to = 2 │
3 │ 6 ← arg │ 3 │ 6 ← base = 3 │ slot 0 = threshold
└──────────────────────────┘ 4 │ nil ← reserved │ slot 1 = hits
└──────────────────────────┘
frames: [ main{base:0, ip:8} ] frames: [ main{base:0, ip:8, want:2},
make_scorer{base:3, ret_to:2, ip:0} ]
The argument was not copied. 6 was pushed by the caller at index 3, and the callee's base is
3, so its slot 0 is that cell. That is what makes calls allocation-free, and it is
the calling convention
made visible.
stack.resize(base + max_stack) filled index 4 with nil so GET_LOCAL 1 cannot read past the
frame.
2 0000 LOAD_INT 0 [.., 6, 0] ← hits = slot 1 (index 4)
2 0001 CLOSURE 0 [.., 6, 0, <closure>] ← ★★ ALLOCATION 3, 4, 5
2 0003 RETURN 2 → close, truncate, pop frame
★★ At the inner CLOSURE 0, the descriptors drive three captures:
upvals[0] threshold (parent local 0) → find_or_create_open_upvalue(base+0 = 3)
NOT in the open list → ALLOCATE Upvalue::Open(3)
upvals[1] hits (parent local 1) → find_or_create_open_upvalue(base+1 = 4)
NOT in the open list → ALLOCATE Upvalue::Open(4)
upvals[2] BOOST (parent upvalue 0)→ COPY the handle from make_scorer's closure
NO allocation — the SAME box the outer closure holds
heap:
[0] Closure make_scorer [1] Upvalue::Open(0) ← BOOST
[2] Upvalue::Open(3) ← threshold ALLOCATION 3 (16 B)
[3] Upvalue::Open(4) ← hits ALLOCATION 4 (16 B)
[4] Closure <anonymous> { upvals: [→2, →3, →1] } ALLOCATION 5 (40 B)
open upvalue list (sorted DESCENDING): [Open(4), Open(3), Open(0)]
bytes_allocated: 120
Notice upvals[2]. The transitive capture cost no allocation — it copied a handle. That is
why flat closures make access O(1) and why capture is resolved at compile time.
Layer 9: Closing Upvalues
RETURN 2 in make_scorer does three things, in this order:
1. close_upvalues(from = base = 3)
open list is sorted descending, so walk from the head:
Open(4) 4 >= 3 → read stack[4] = 0 → becomes Closed(0) ← hits
Open(3) 3 >= 3 → read stack[3] = 6 → becomes Closed(6) ← threshold
Open(0) 0 < 3 → STOP (still open; BOOST is alive in main's frame)
open list: [Open(0)]
2. move 1 result to ret_to = 2, truncate the stack to 3
3. pop the frame
heap AFTER closing:
[2] Upvalue::Closed(Integer(6)) ← threshold, rescued from stack[3]
[3] Upvalue::Closed(Integer(0)) ← hits, rescued from stack[4]
[1] Upvalue::Open(0) ← BOOST, still on the stack
value stack: [1.4, <fn make_scorer>, <closure scorer>]
▲
└── the result, at ret_to = 2 → `score` is slot 2
The closures did not change. Only the upvalue objects' contents did — which is why closing requires no pointer fixups and why every holder keeps working.
Reverse the order in step 1 and 2 — truncate before closing — and the upvalues capture whatever the stack happens to contain next. No error, plausible values, and the bug appears weeks later.
Layer 10: The Second Call — Userdata and Fields
1 0008 GET_LOCAL 2 [1.4, <fn>, <scorer>, <scorer>]
1 0009 GET_GLOBAL 1 ; candidates
[.., <userdata Candidates>]
1 0011 LOAD_INT 1 [.., <userdata>, 1]
1 0012 GET_INDEX [.., <userdata ArticleRef{idx:0}>] ← ALLOCATION 6
1 0013 CALL 1 0 → push frame (want = MULTRET)
GET_INDEX on the Candidates userdata went through its metatable's __index, which is a native
function that bounds-checked 1 against articles.len() and produced an ArticleRef —
the index-handle pattern.
One allocation, not a marshaled table of four fields.
3 0000 GET_LOCAL 0 ; article [.., <ArticleRef>]
3 0001 GET_FIELD 0 ; "semantic_score"
Inside GET_FIELD, with the inline cache warm:
site 0001 cache: Mono { shape: UserData(TypeId::of::<ArticleRef>()), answer: accessor #1 }
shape_of(recv) == cached_shape → HIT
accessor #1: |r, ctx| ctx.resolve(r).semantic_score → 0.5
One tag comparison and an accessor call, instead of: metatable lookup, __index lookup, native
call, accessor-table probe, accessor call.
3 0001 GET_FIELD 0 [.., 0.5] ← s = slot 1
3 0003 GET_LOCAL 0 [.., 0.5, <ref>]
3 0004 GET_FIELD 1 ; age_hours [.., 0.5, 2.0]
3 0006 GET_UPVAL 0 ; threshold [.., 0.5, 2.0, 6] ← reads Closed(6). One indirection.
3 0007 LT [.., 0.5, true] ← 2.0 < 6 : mixed Float/Integer,
compared EXACTLY (Lab 4)
3 0008 JUMP_IF_FALSE 0018 [.., 0.5] ← not taken
3 0010 GET_LOCAL 1 ; s [.., 0.5, 0.5]
3 0011 GET_UPVAL 2 ; BOOST [.., 0.5, 0.5, 1.4] ← reads Open(0) → stack[0]. STILL OPEN.
3 0012 MUL [.., 0.5, 0.7] ← 0.5 * 1.4
3 0013 SET_LOCAL 1 ; s [.., 0.7]
3 0014 GET_UPVAL 1 ; hits [.., 0.7, 0]
3 0015 LOAD_INT 1 [.., 0.7, 0, 1]
3 0016 ADD [.., 0.7, 1] ← Integer + Integer, wrapping
3 0017 SET_UPVAL 1 ; hits [.., 0.7] ← WRITES Closed(1). Shared mutation.
3 0018 GET_LOCAL 1 [.., 0.7, 0.7]
3 0019 GET_UPVAL 1 ; hits [.., 0.7, 0.7, 1]
3 0020 RETURN 3 → 2 values
Line 0011 is the whole of Section 4 in one instruction. BOOST is read through an upvalue that
is still open, pointing at stack index 0, which is BOOST's live slot in the main chunk's frame.
threshold and hits are read through closed upvalues, on the heap, because their frame died.
Three upvalues, two states, one instruction stream — and the script cannot tell the difference.
Layer 11: GC Roots at This Moment
Stop the world at ip = 0016 in the innermost frame and enumerate:
ROOT SET 1 value stack [1.4, <fn make_scorer>, <closure scorer>, <closure scorer>,
<ArticleRef>, 0.7, 0, 1]
ROOT SET 2 call frames main → make_scorer(popped) → <anonymous>
the live frame holds Closure#4
ROOT SET 3 globals table { candidates → UserData#5, print → Native#… }
ROOT SET 4 open upvalues [Open(0)] ← BOOST
ROOT SET 5 chunk constant pools "candidates", "semantic_score", "age_hours" (interned)
ROOT SET 6 intern table WEAK — swept, not a strong root
ROOT SET 7 host handles the Engine's stash: empty
ROOT SET 8 module cache empty (no require)
ROOT SET 9 temp roots empty (no host callback in flight)
MARK from those:
Closure#4 → upvals [Closed(6), Closed(1), Open(0)] → the Closed ones' VALUES are traced;
Open(0)'s value lives on the STACK,
already root set 1
UserData#5 (Candidates) → its per-type metatable → the __index native
ArticleRef#6 → its owner UserData#5 ← the edge Lab 20 required
every EmberStr constant
SWEEP: Closure#0 (make_scorer) is STILL REACHABLE from the main chunk's slot 1.
Nothing is garbage yet.
Two edges here are the ones people forget. ArticleRef → owner (without it, Candidates could
be freed while a reference into it is live) and the per-type metatable (without it, every field
access breaks after the first collection). Both are traced because
Lab 15's four-edge checklist
said so.
$ ember run --trace-gc --stats scorer.ember
--- heap census at exit ---
Closure 2 64 B
Upvalue 3 48 B
UserData 2 88 B
EmberStr 3 72 B
total 10 272 B (peak 272 B, 0 collections)
Ten objects. A marshaled version of this program would have allocated a table plus four strings per article — which is the measurement from Lab 20, visible at n = 1.
Layer 12: The Return into Rust
3 0020 RETURN 3 → 2 results: [0.7, 1]
close_upvalues(base) — none open above base
truncate to ret_to
1 0015 RETURN 0 → MULTRET: everything above the marker → [0.7, 1]
frames empty → return to Engine::call
#![allow(unused)] fn main() { let (score, hits): (f64, i64) = engine.call("main", ())?; // └─ FromValues::from_values(&[Float(0.7), Integer(1)]) // Float(0.7) → f64 : tag matches, no conversion // Integer(1) → i64 : tag matches, no conversion assert_eq!(score, 0.7); assert_eq!(hits, 1); }
Fourteen lines of script, eight layers, and the value that comes back to Rust is a plain f64.
No handle, no lifetime, no reference into the engine — which is
the ownership boundary
doing exactly what it promised.
And when it goes wrong:
$ ember run scorer.ember # with an Article whose semantic_score is nil
scorer.ember:8:13: error: attempt to multiply a nil value
6 │ local s = article.semantic_score
7 │ if article.age_hours < threshold then
8 │ s = s * BOOST
│ ^ this is nil
stack traceback:
in function '<anonymous>' scorer.ember:16
in main chunk scorer.ember:16
in <host>
The caret is under s — the operand, from the Binary node's left child's span, threaded from
Lab 2, reported through the line table from Lab 9, with a frame stack from Lab 11 and a caller span
from Lab 7. Six labs, one caret.
The Complete Map
scorer.ember
│
[LEXER] chars → tokens + spans §1, Lab 1
│
[PARSER] tokens → AST precedence becomes shape §1, Lab 2
│
[COMPILER] names → slots literals → constants §3, Lab 10
│ structure → jumps free names → upvalue descriptors
│
[VALIDATOR] 9 rules, one pass stack depth abstraction §3, Lab 9
│
[VM] fetch, decode, execute §3, Lab 11
│ ├── value stack operands AND locals, one array
│ ├── frame stack base, ret_to, ip, want
│ ├── budget checked in the FETCH position §5, Lab 23
│ └── inline caches shape guard + accessor §7, Lab 28
│
[HEAP] handles, not pointers §4, Lab 15
│ ├── closures proto + captured upvalue handles
│ ├── upvalues Open(stack index) | Closed(value)
│ ├── userdata host objects, per-type metatables §5, Lab 20
│ └── GC 9 root sets, 4 edge classes, mark & sweep
│
[ENGINE] marshaling, capabilities, budgets §5, Lab 19
│
Rust: (f64, i64)
Your Deliverable
Produce this document for your own runtime, with your own tool output, for a script of your own choosing that exercises at least: a closure with a mutated capture, a transitive capture, a userdata field access, a conditional, multiple returns, and a return into Rust.
Every listing must come from a command you can run. Every claim about the heap must come from
--stats or --trace-gc. If a layer's output surprises you, that is the layer to go and
understand — which is the whole reason this document is the final deliverable rather than a
summary.
Validation / Self-check
- Trace
BOOSTthrough every representation: what is it at each layer? - Why does
make_scorercarry an upvalue for a variable it never mentions? - At the moment the inner closure is created, how many objects are allocated, and which capture costs nothing?
- Which upvalues are closed at
make_scorer's return, and which is not? Why? - Why must closing precede truncation? What is the symptom of the wrong order?
CALL 1 2versusCALL 1 0— what does the differing operand mean?- Name all nine root sets live at Layer 11, and the two edges most often forgotten.
- How many allocations did this program make, and how many would a marshaled version have made?
- Trace the caret in the final error backwards: which lab produced each ingredient?
Next: When Not to Embed a Language.