Lab 18: Metatables (Milestone 12)

Background

You will implement setmetatable/getmetatable, the __index/__newindex lookup path, __call, __tostring, __len, __eq/__lt/__le, and the arithmetic and concat metamethods — in both backends, with a bounded chain and metamethod calls charged to the budget.

At the end of this lab, the language is done. Everything after it is about hosting, hardening, and speed.

Why This Lab Matters

This is where a table becomes an object system. The Account example from the warm-up runs, unchanged, in your runtime.
A metamethod can run arbitrary script code inside what looks like a primitive operation. Every dispatch site becomes a safe point and a re-entrancy hazard — Section 5's central problem, arriving one section early and in a place where you control both sides.
It is the second feature (after multiple returns) most likely to break the differential test, because the fallback order is subtle and easy to implement two different ways.

Prerequisites

Labs 13–17 complete; differential tests green.
Metatables read.
Lua 5.4 Reference Manual §2.4 open — you will consult it repeatedly, and you should.

Predict First

a.deposit where a has no deposit key but its metatable's __index is Account — how many table lookups?
t.x = 1 on a table with __newindex where x does not exist. Then again where it does. Which fires the metamethod?
Two tables whose __index fields point at each other. What must the runtime do?
1 + t where t has __add — is it called? What about t + 1?
t == t where t has __eq — is the metamethod called?
setmetatable({}, {__index = function() return 1 end}) in a loop: does the instruction budget count the metamethod calls?

Step 1: The Metatable Slot and Interned Names

#![allow(unused)]
fn main() {
// Interned ONCE at Engine creation. Every failed table lookup consults __index,
// so this name is on an already-slow path — pre-interning makes the lookup a
// handle compare instead of a string hash.
pub struct MetaNames { interned: [GcRef<EmberStr>; MetaKey::COUNT] }
}

setmetatable(t, mt) and getmetatable(t) are library functions (formally Lab 21; add minimal versions now). Two rules from the manual:

setmetatable(t, nil) removes it.
getmetatable returns the metatable's __metatable field if present — Ember omits __metatable (and says why), so it always returns the metatable. Divergence, documented.

And the GC: Table::meta is an edge. Add it to trace_children now and re-run --gc-stress; this is one of the four commonly-missed edges and you have the tool that finds it.

Step 2: `__index`

Write the iterative, bounded loop from the concept chapter. Three properties to get right:

#![allow(unused)]
fn main() {
// 1. The FAST PATH is unchanged. A key that exists is one probe; nothing about
//    metatables is consulted, checked, or branched on beyond `raw != Nil`.
// 2. A miss with NO metatable returns nil. Only indexing a NON-TABLE errors.
// 3. The loop is bounded. An __index cycle is two lines of script.
const MAX_META_CHAIN: usize = 100;
}

Warning: Property 2 is two different failures with two different messages, and conflating them makes article.missing throw where it should return nil. Test both: local t = {} return t.nope → nil; local n = nil return n.x → "attempt to index a nil value".

Strings get a shared type metatable, which is how s:upper() will work in Lab 21:

#![allow(unused)]
fn main() {
fn type_metatable(&self, v: Value) -> Option<GcRef<Table>> {
    match v {
        Value::Str(_) => Some(self.string_meta),   // one table, shared by ALL strings
        Value::UserData(u) => self.heap.userdata(u).ok()?.meta,   // §5
        _ => None,
    }
}
}

Step 3: `__newindex` and Its Asymmetry

#![allow(unused)]
fn main() {
fn newindex(&mut self, mut obj: Value, key: Value, v: Value, span: Span) -> Result<()> {
    for _ in 0..MAX_META_CHAIN {
        let meta = match obj {
            Value::Table(t) => {
                // THE ASYMMETRY: __newindex fires only for keys that do NOT
                // already exist. An assignment to an existing key is always raw.
                if self.heap.table(t)?.get(key) != Value::Nil || self.heap.table(t)?.meta.is_none() {
                    return self.heap.table_mut(t)?.set(key, v);
                }
                self.heap.table(t)?.meta.unwrap()
            }
            other => match self.type_metatable(other) {
                Some(m) => m,
                None => return Err(self.rt(span, format!(
                    "attempt to index a {} value", other.type_name()))),
            },
        };
        match self.heap.table(meta)?.get(self.names.get(MetaKey::NewIndex)) {
            Value::Nil => return self.heap.table_mut(match obj { Value::Table(t) => t, _ => unreachable!() })?
                                     .set(key, v),
            f @ (Value::Closure(_) | Value::Native(_)) => {
                self.call_value(f, vec![obj, key, v], span)?;
                return Ok(());
            }
            next => obj = next,
        }
    }
    Err(self.rt(span, "'__newindex' chain too long; possible loop"))
}
}

The asymmetry is what makes the proxy pattern work, and it is worth demonstrating rather than asserting:

local real, log = {}, {}
local proxy = setmetatable({}, {
  __index    = real,
  __newindex = function(_, k, v) log[#log+1] = k; real[k] = v end,
})
proxy.a = 1     -- logged: `a` does not exist in `proxy` (it went into `real`)
proxy.a = 2     -- logged AGAIN: still not in `proxy`

If the handler had written into proxy itself, the second assignment would go raw and the log would miss it. That is the bug people ship, and the corpus case above catches it.

Step 4: Operators

#![allow(unused)]
fn main() {
fn arith(&mut self, op: BinOp, a: Value, b: Value, span: Span) -> Result<Value> {
    if let Ok(v) = arith::binary(op, a, b) { return Ok(v); }     // ← fast path FIRST, untouched
    let key = MetaKey::for_binop(op);
    if let Some(h) = self.metamethod(a, key)? { return self.call1(h, &[a, b], span); }
    if let Some(h) = self.metamethod(b, key)? { return self.call1(h, &[a, b], span); }
    Err(self.arith_type_error(op, a, b, span))                    // ← Lab 4's message, unchanged
}
}

Comparison has its own rules, and they changed between Lua 5.3 and 5.4 — check the manual rather than remembering:

#![allow(unused)]
fn main() {
fn eq(&mut self, a: Value, b: Value, span: Span) -> Result<bool> {
    if raw_eq(a, b) { return Ok(true); }                  // identity first: t == t never calls __eq
    // __eq is consulted ONLY when both are tables (or both userdata) and raw
    // equality failed. Never across different types.
    let (Value::Table(_), Value::Table(_)) = (a, b) else { return Ok(false) };
    match self.metamethod(a, MetaKey::Eq)?.or(self.metamethod(b, MetaKey::Eq)?) {
        Some(h) => Ok(self.call1(h, &[a, b], span)?.is_truthy()),
        None => Ok(false),
    }
}
}

Note: In Lua 5.3, a <= b fell back to not (b < a) when __le was absent. Lua 5.4 removed that fallback. If you implement 5.3's behavior you will disagree with lua on a case your corpus probably does not cover. Run lua -e 'local mt={__lt=function() return true end} local a=setmetatable({},mt) print(a<=a)' and see what your version does before deciding.

Step 5: The Budget, and Re-entrancy

Script metamethods are counted automatically — they run through do_call and the budget ticks in the fetch position. Native ones are not, and Section 5 will register plenty. Charge them explicitly:

#![allow(unused)]
fn main() {
fn call1(&mut self, h: Value, args: &[Value], span: Span) -> Result<Value> {
    self.budget.tick_call()?;         // metamethod dispatch is counted work
    // ...
}
}

And the re-entrancy audit — do this now, deliberately, for every dispatch site:

#![allow(unused)]
fn main() {
// WRONG: a borrow of the heap held across a call that can re-enter the VM,
// allocate, collect, and mutate the very table we are borrowing.
let t = self.heap.table(handle)?;
let h = t.meta_get(MetaKey::Index);
self.call1(h, &[obj, key], span)?;     // ← `t` is still borrowed. Also: the table
                                       //    may have been rehashed by the callee.
}

The fix is rule 1 from the section's Rust problem: copy out what you need (Value is Copy), drop the borrow, then call. The borrow checker will enforce it; the lesson is why it is right, not merely that it compiles.

The Trace

$ ember run --trace-meta -e '
local Account = {}
Account.__index = Account
function Account.new(b) return setmetatable({balance = b}, Account) end
function Account:deposit(n) self.balance = self.balance + n end
local a = Account.new(100)
a:deposit(50)
return a.balance'

index  a."deposit"
  raw get(a, "deposit")            → nil           ← miss on the instance
  a.meta = Account
  Account.__index                  → Account       (a TABLE handler → delegate)
  raw get(Account, "deposit")      → <fn>          ← found, depth 2
index  a."balance"                 → 100           ← depth 1, no metatable touched
newindex a."balance" = 150
  raw get(a, "balance")            → 100  (EXISTS) ← __newindex NOT consulted
  raw set
index  a."balance"                 → 150
150

Read the depths. A field that exists is found in one probe and never touches the metatable. A method is always a miss on the instance, so it is always at least two probes — which is why method-heavy object code in Lua is slower than field access, and why Section 7's inline caches target exactly this site.

Now the operator path:

$ ember run --trace-meta -e '
local V = {}
V.__index = V
V.__add = function(a, b) return setmetatable({x = a.x + b.x}, V) end
V.__tostring = function(v) return "V(" .. v.x .. ")" end
local p = setmetatable({x=1}, V) + setmetatable({x=2}, V)
return tostring(p)'

arith  ADD  table + table
  fast path: arith::binary → type error, fall through   ← the fast path is TRIED first
  metamethod(a, __add)   → <fn>
  call __add(a, b)        [budget -1]
    index b."x" → 2   (depth 1)
    index a."x" → 1   (depth 1)
  → table#7
tostring table#7
  metamethod(t, __tostring) → <fn>
  call __tostring(t)      [budget -1]
V(3)

fast path: … fall through is the property to preserve. 1 + 2 never reaches line two of that trace. Confirm it: run the arithmetic benchmark from Lab 8 before and after this lab and check that it did not move.

And the loop guard:

$ ember run -e 'local a, b = {}, {}
                setmetatable(a, {__index = b}) setmetatable(b, {__index = a})
                return a.missing'
<argv>:3:24: error: '__index' chain too long; possible loop

   3 │                 return a.missing
     │                        ^^^^^^^^^
$ echo $?
1

Two lines of script, and without the bound it is an infinite loop inside the host process.

Expected Output

$ diff <(ember run tests/golden/meta/account.ember) <(lua tests/golden/meta/account.lua)
$ ember run -e 'local t = setmetatable({}, {__call = function(_, x) return x*2 end}) return t(21)'
42
$ ember run -e 'local t = setmetatable({}, {__len = function() return 99 end}) return #t'
99
$ ember run -e 'local t = {} return tostring(t == t)'
true
$ ember run -e 'return (setmetatable({}, {__tostring = function() return "custom" end}))'
custom
$ cargo test --test differential
test backends_agree_on_the_whole_corpus ... ok

Debugging Steps

`a.balance` is slow, or the trace shows a metatable consultation

The fast path is not first. raw != Nil must short-circuit before anything else.

`t.nope` errors instead of returning `nil`

You conflated "missing key" with "indexing a non-table". Two branches, two messages.

`__newindex` fires once and then never again

The handler is writing into the proxy table itself. That is the user's bug — but check that your own __newindex chain terminates by writing raw, not by recursing.

`t == t` calls `__eq`

Raw equality is not checked first.

`a <= b` disagrees with `lua`

The 5.3-versus-5.4 __le fallback. Check the manual and pick deliberately.

The `__index` loop hangs

MAX_META_CHAIN missing, or the loop continues without advancing obj.

`--gc-stress` fails only in metatable tests

Table::meta is not traced. Fourth edge.

Differential test fails on operator fallback order

One backend checks b's metamethod before a's. Lua checks the left operand first. Fix both to match the manual, not each other.

Experiment

CLAIM. Method lookup through __index is measurably slower than direct field access, and the gap is the thing inline caches exist to close.

METHOD. Benchmark three loops over the same object: (a) reading a field that exists on the instance; (b) calling a method found one level up via __index; (c) calling a method found three levels up via a chain of __index tables.

PREDICTION. What is the ratio (b)/(a)? Is (c)/(b) roughly linear in chain depth?

RESULT. Record it in docs/learning/08-tables.md. This is the motivating measurement for Section 7's inline caches, and having produced it yourself means that chapter will land as a solution to a problem you have rather than a technique you read about.

Test

#![allow(unused)]
fn main() {
#[test]
fn the_account_example_from_the_warmup_runs() {
    assert_eq!(run("local A={} A.__index=A
                    function A.new(b) return setmetatable({balance=b}, A) end
                    function A:deposit(n) self.balance=self.balance+n end
                    local a=A.new(100) a:deposit(50) return a.balance"), "150");
}

#[test]
fn missing_key_is_nil_but_indexing_a_non_table_is_an_error() {
    assert_eq!(run("local t={} return tostring(t.nope)"), "nil");
    assert_eq!(err("local n=nil return n.x").kind, ErrorKind::Runtime);
    assert!(err("local n=nil return n.x").message.contains("index a nil value"));
}

#[test]
fn newindex_fires_only_for_new_keys() {
    // The proxy pattern. If __newindex fired for existing keys, `hits` would be 1.
    assert_eq!(run("local real, hits = {}, 0
                    local p = setmetatable({}, {__index=real,
                        __newindex=function(_,k,v) hits=hits+1 real[k]=v end})
                    p.a = 1 p.a = 2 p.a = 3
                    return hits"), "3");
    // And the inverse: a handler writing into the table itself fires ONCE.
    assert_eq!(run("local hits = 0
                    local p = setmetatable({}, {__newindex=function(t,k,v)
                        hits=hits+1 rawset(t,k,v) end})
                    p.a = 1 p.a = 2 p.a = 3
                    return hits"), "1");
}

#[test]
fn index_chains_are_bounded() {
    let e = err("local a,b={},{} setmetatable(a,{__index=b}) setmetatable(b,{__index=a})
                 return a.missing");
    assert_eq!(e.kind, ErrorKind::Runtime);
    assert!(e.message.contains("too long"));
}

#[test]
fn raw_equality_is_checked_before_eq() {
    assert_eq!(run("local calls=0
                    local mt={__eq=function() calls=calls+1 return true end}
                    local t=setmetatable({},mt)
                    local _ = (t == t)
                    return calls"), "0");
}

#[test]
fn arithmetic_metamethods_check_the_left_operand_first() {
    assert_eq!(run("local L=setmetatable({},{__add=function() return 'left' end})
                    local R=setmetatable({},{__add=function() return 'right' end})
                    return L + R"), "left");
}

#[test]
fn the_arithmetic_fast_path_is_untouched() {
    // A regression guard: adding metatables must not slow down 1 + 2.
    // Compare against the recorded Lab 8 baseline, with a generous margin.
    assert!(bench_ns("loop_10m") < baseline_ns("loop_10m") * 110 / 100,
            "metatable support regressed the arithmetic fast path");
}

#[test]
fn metatables_are_traced_by_the_collector() {
    with_gc_stress(|| {
        assert_eq!(run("local t=setmetatable({}, {__index=function() return 7 end})
                        collectgarbage() return t.anything"), "7");
    });
}
}

Challenge Extensions

Lua's flags cache. Lua's Table carries a bitmask of absent metamethods, so "this metatable has no __index" is one bit test instead of a hash lookup. Implement it and measure on the method-call benchmark.
rawget/rawset/rawequal/rawlen. The escape hatches that bypass metamethods. Needed by every serious __index/__newindex implementation, including the proxy test above.
A read-only table. __newindex = function() error("read-only") end plus __index at the real data. Then ask: is this a security mechanism? (It is not — a script can call setmetatable again. That is why Section 5 sandboxes with capabilities, not metatables.)
Class inheritance. Build a class(base) helper with multi-level __index chains and measure lookup cost at depth 1, 2, 4, 8. Compare with the experiment's prediction.
__close and to-be-closed variables. Lua 5.4's local x <close> = .... Needs compiler support and interacts with error unwinding. Genuinely hard; a good gauge of whether Section 4 landed.

Deliverables

setmetatable/getmetatable; Table::meta traced by the collector (verified under --gc-stress).
Metamethod names interned once at Engine creation.
__index and __newindex with table and function handlers, iterative and bounded.
The fast path is measurably unchanged (the regression test passes).
A missing key returns nil; indexing a non-table errors, with different messages.
Strings have a shared type metatable.
__call, __tostring (validated to return a string), __len, __eq, __lt, __le, and all the arithmetic and concat metamethods, with the left-operand-first order.
Metamethod dispatch is charged to the instruction budget.
No borrow of the heap is held across a metamethod call; every dispatch site audited.
The __le 5.3-vs-5.4 question decided and documented.
Omitted metamethods (__gc, __mode, __close, __metatable, bitwise) each listed in appendix/lua-differences.md with their individual reason.
The method-lookup-cost experiment recorded.
Differential tests green.
Milestone 12 complete — the language is done.

Validation / Self-check

Walk a.deposit through the lookup path and count the probes. Why is a method always at least two?
Why does __newindex fire only for new keys? Give the two proxy variants and their hit counts.
Why must the __index traversal be iterative and bounded? Give the two-line script that needs it.
Why is raw equality checked before __eq, and what breaks if it is not?
Which operand's metamethod is consulted first for a + b, and where is that specified?
What changed about __le between Lua 5.3 and 5.4, and how did you decide?
Why is every metamethod dispatch a re-entrancy hazard? Give the wrong code and the fix.
Why is a read-only table built from __newindex not a security mechanism?
Give your measured ratio of method lookup to field access, and say what Section 7 will do about it.

Next: Section 5 — The Host Boundary. The language is complete; now make it embeddable.

Open-Source Engineer & Contributor