Lab 5: Variables and Scope (Milestone 3)

Background

You will add statements to the parser and the evaluator: local, assignment, do ... end blocks, and expression statements. By the end, Ember has variables, shadowing, and globals — and you will have built an environment you already know is the wrong data structure.

Why This Lab Matters

The local x = x rule is one line of ordering and it distinguishes Lua, JavaScript, and the naive implementation. Getting it right requires understanding when a binding comes into existence, which is a question you will answer three more times in this curriculum.
The environment is the baseline for Section 3. The --trace-scope probe counts you produce here are the number that justifies compile-time slot resolution.
Globals become a Table, which looks like over-engineering today and is the foundation of Section 5's sandbox.

Prerequisites

Section 1 complete through Lab 4.
Scope and Environments read.

Predict First

local x = 1; do local x = 2 end; return x — what is returned?
local x = 1; local x = x + 1; return x — what is returned? What if the binding were created before the initializer ran?
x = 1; return x with no local — legal? Where does x live?
return undefined_name — error or nil?
local a, b = 1 — what is b?
local a, b = 1, 2, 3 — is the extra 3 an error?
a, b = b, a — does this swap? What must be true of the evaluator for it to?

Step 1: Statements in the Parser

Goal. parse_chunk returns a Block of statements, not a single expression.

#![allow(unused)]
fn main() {
pub fn parse_chunk(&mut self) -> Result<Block> {
    let block = self.block()?;
    self.expect(TokenKind::Eof)?;      // trailing junk is an error, not ignored
    Ok(block)
}

fn block(&mut self) -> Result<Block> {
    let _guard = self.enter()?;                       // depth limit applies here too
    let start = self.peek_span();
    let mut stmts = Vec::new();
    while !self.at_block_end() {
        let s = self.statement()?;
        let is_return = matches!(s, Stmt::Return { .. });
        stmts.push(s);
        // Lua 5.4 §3.3.1: `return` must be the LAST statement in a block.
        // Enforcing it in the parser makes `return x; print(y)` a syntax error
        // with a caret, rather than silently-dead code.
        if is_return { break; }
    }
    let span = start.merge(self.prev_span());
    Ok(Block { stmts, span })
}

fn at_block_end(&self) -> bool {
    matches!(self.peek(), TokenKind::Eof | TokenKind::End
                        | TokenKind::Else | TokenKind::Elseif | TokenKind::Until)
}
}

The part that matters. at_block_end lists every token that can follow a block. Miss one and the block parser runs past the end of its construct and produces a confusing error twenty lines later. This set is the FOLLOW set of block in the grammar; you are computing it by hand, which is what a hand-written parser trades for not having a generator compute it.

Step 2: `local`, Assignment, and Expression Statements

#![allow(unused)]
fn main() {
fn statement(&mut self) -> Result<Stmt> {
    match self.peek() {
        TokenKind::Local => self.local_stat(),
        TokenKind::Do    => self.do_stat(),
        TokenKind::Semi  => { let t = self.advance(); Ok(Stmt::Empty { span: t.span }) }
        _ => self.expr_stat(),
    }
}

fn local_stat(&mut self) -> Result<Stmt> {
    let start = self.expect(TokenKind::Local)?.span;
    let mut names = vec![self.name()?];
    while self.eat(TokenKind::Comma) { names.push(self.name()?); }
    let exprs = if self.eat(TokenKind::Assign) { self.exprlist()? } else { Vec::new() };
    let span = start.merge(self.prev_span());
    Ok(Stmt::Local { names, exprs, span })
}

/// `x = 1`, `a, b = b, a`, or a bare call `f(1)`. We cannot tell which with one
/// token of lookahead, so: PARSE FIRST, CLASSIFY SECOND.
fn expr_stat(&mut self) -> Result<Stmt> {
    let start = self.peek_span();
    let first = self.suffixed_expr()?;
    if self.at(&TokenKind::Assign) || self.at(&TokenKind::Comma) {
        let mut targets = vec![first];
        while self.eat(TokenKind::Comma) { targets.push(self.suffixed_expr()?); }
        self.expect(TokenKind::Assign)?;
        let exprs = self.exprlist()?;
        for t in &targets { self.check_assignable(t)?; }   // `1 = 2` must be an error
        return Ok(Stmt::Assign { targets, exprs, span: start.merge(self.prev_span()) });
    }
    // Not an assignment: it must be a CALL. `x` alone is not a statement.
    match first {
        Expr::Call { .. } | Expr::Method { .. } =>
            Ok(Stmt::ExprStat { span: first.span(), expr: first }),
        other => Err(EmberError { kind: ErrorKind::Parse,
            message: "syntax error near this expression (expected '=' or a function call)".into(),
            span: Some(other.span()), traceback: vec![] }),
    }
}
}

The parts that matter.

check_assignable rejects 1 = 2 and f() = 2 with a span. Only Name and Index are valid targets. Doing this in the parser rather than the evaluator means the error is a compile error, which is where it belongs.
The final match enforces Lua's rule that a bare expression is not a statement. x alone is a syntax error; x() is not. This is why Lua needs no expression-statement semicolons and why the ( ambiguity from the grammar chapter exists.

Step 3: The Environment

Write src/interp/env.rs from the concept chapter, plus the guard:

#![allow(unused)]
fn main() {
pub struct ScopeGuard<'a> { env: &'a mut Env }
impl Drop for ScopeGuard<'_> { fn drop(&mut self) { self.env.pop_scope(); } }

impl Env {
    pub fn scope_guard(&mut self) -> ScopeGuard<'_> {
        self.push_scope();
        ScopeGuard { env: self }
    }
}
}

Warning: ScopeGuard holds &mut Env, so while the guard is alive you cannot use self.env. That is the borrow checker correctly noticing that the guard owns the environment. The fix used in Ember is to make the guard hold a &Cell<usize> depth marker instead, and have Drop truncate scopes to that depth — the same trick as the parser's DepthGuard, for the same reason. Work out why the naive version does not compile before reading ahead; the compiler error here is a genuinely useful lesson about what RAII costs in Rust.

Step 4: Evaluating Statements

#![allow(unused)]
fn main() {
fn eval_stmt(&mut self, s: &Stmt) -> Result<Flow> {
    Ok(match s {
        Stmt::Empty { .. } => Flow::Normal,

        Stmt::Local { names, exprs, .. } => {
            // ORDER IS THE SEMANTIC. Evaluate in the CURRENT scope, then declare.
            let values = self.eval_exprlist(exprs, names.len())?;
            for (name, v) in names.iter().zip(values) {
                self.env.declare(name.clone(), v);
            }
            Flow::Normal
        }

        Stmt::Assign { targets, exprs, .. } => {
            // ALL right-hand sides are evaluated BEFORE any assignment happens,
            // which is what makes `a, b = b, a` a swap. Lua 5.4 §3.3.3.
            let values = self.eval_exprlist(exprs, targets.len())?;
            for (t, v) in targets.iter().zip(values) { self.assign(t, v)?; }
            Flow::Normal
        }

        Stmt::Do { body, .. }      => self.eval_block(body)?,
        Stmt::ExprStat { expr, .. } => { self.eval_expr(expr)?; Flow::Normal }
        // if / while / for / break / return arrive in Lab 6
    })
}

/// Adjust a list of expressions to exactly `want` values: pad with nil,
/// discard extras. Lab 17 makes the LAST expression able to expand.
fn eval_exprlist(&mut self, exprs: &[Expr], want: usize) -> Result<Vec<Value>> {
    let mut out = Vec::with_capacity(want);
    for e in exprs { out.push(self.eval_expr(e)?); }
    out.resize(want, Value::Nil);
    Ok(out)
}
}

Checkpoint question. Why does a, b = b, a swap? Which line makes it work, and what would happen if assignment happened per-pair as you went?

Step 5: Name Resolution and Globals

#![allow(unused)]
fn main() {
Expr::Name { name, span } => {
    if let Some(v) = self.env.get(name) { return Ok(v); }
    // Not a local → a global. A MISSING global is nil, not an error (Lua 5.4 §3.5).
    let key = self.intern(name);
    self.heap.table_get(self.globals, key)
}

fn assign(&mut self, target: &Expr, v: Value) -> Result<()> {
    match target {
        Expr::Name { name, .. } => {
            if self.env.set(name, v) { return Ok(()); }   // an existing local
            let key = self.intern(name);
            self.heap.table_set(self.globals, key, v)      // otherwise a global
        }
        Expr::Index { .. } => Err(rt(target.span(), "table indexing arrives in Lab 13")),
        _ => Err(internal("check_assignable should have rejected this")),
    }
}
}

Note the shape: locals are tried first, globals are the fallback, and creating a global is implicit. That is Lua, and it is the design that makes a missing local a silent bug — which is why Section 5 swaps the globals table per execution.

The Trace

$ ember ast -e 'local x = 1
do
  local x = 2
  y = x + 1
end
return x'

Block                                @0..44
├── Local                            @0..11
│   ├── names: ["x"]
│   └── exprs: [Int(1) @10..11]
├── Do                               @12..40
│   └── Block                        @16..38
│       ├── Local                    @16..27
│       │   ├── names: ["x"]
│       │   └── exprs: [Int(2) @26..27]
│       └── Assign                   @30..38
│           ├── targets: [Name("y") @30..31]
│           └── exprs: [Binary(Add) @34..38]
└── Return                           @41..49
    └── exprs: [Name("x") @48..49]

$ ember run --trace-scope -e '<the same program>'

scope push  depth=1                       scopes=[{}]
  declare x = 1                           scopes=[{x}]
scope push  depth=2                       scopes=[{x}, {}]
  declare x = 2                           scopes=[{x}, {x}]        ← shadows
  resolve x → depth 2, 1 probe  → 2
  assign  y: not a local (2 probes)  → GLOBAL y = 3
scope pop   depth=1                       scopes=[{x}]             ← inner x gone
  resolve x → depth 1, 1 probe  → 1
1

Two things to read off that trace.

resolve x found the inner x before the pop and the outer x after. That is shadowing, and it is a consequence of searching innermost-first, nothing more.
assign y cost two probes and a miss before falling through to the global table. Every global access in the tree walker pays for the entire scope chain first. Now try it nested:

$ ember run --trace-scope -e 'local x = 1 do do do do return x end end end end'
  resolve x → miss@5, miss@4, miss@3, miss@2, found@1   (5 probes, 5 string hashes)

Write that line into docs/learning/04-interpreter.md. In Lab 10 the same read compiles to GET_LOCAL 0, which is one add and one load. That is the comparison Section 3 exists to make.

Expected Output

$ ember run -e 'local x = 1; do local x = 2 end; return x'
1
$ ember run -e 'local x = 1; local x = x + 1; return x'
2
$ ember run -e 'local a, b = 1; return b'
nil
$ ember run -e 'local a, b = 1, 2, 3; return a'
1
$ ember run -e 'local a, b = 1, 2; a, b = b, a; return a'
2
$ ember run -e 'return no_such_name'
nil
$ ember run -e '1 = 2'
<argv>:1:1: error: cannot assign to this expression

   1 │ 1 = 2
     │ ^
$ ember run -e 'local x'
<argv>:1:1: error: syntax error near this expression (expected '=' or a function call)

Wait — that last one is wrong. local x with no initializer is legal Lua (x becomes nil). If your implementation rejects it, local_stat's if self.eat(Assign) branch is missing. Verify with lua -e 'local x print(x)'. This paragraph is deliberate: expected-output blocks in this curriculum are things to check, not to trust, and one of them being wrong is a cheap way to make that point.

Debugging Steps

`local x = x` gives `nil`

You declared before evaluating. Move eval_exprlist above the declare loop.

`a, b = b, a` does not swap

You are assigning inside the loop that evaluates. Evaluate the whole right-hand list first.

Variables from a block are still visible after `end`

pop_scope is not running on some exit path — almost certainly an early ?. Use the guard.

Everything resolves to a global

Env::get is searching scopes in forward order and missing, or declare is writing to scopes[0] instead of last_mut().

`return x; print(y)` parses

The if is_return { break; } in block() is missing, so a statement after return is silently accepted. Lua rejects it; so should you.

Stack overflow parsing `do do do ... end end end`

block() is missing self.enter()?. Every recursive entry point needs the guard.

Experiment

CLAIM. The cost of a local variable read in the tree walker grows with scope depth, and the cost of a global read is worse than the deepest local.

METHOD. Add a counter to Env::get recording probes. Write a generator that emits programs with n nested blocks reading an outer variable, for n in 1..20. Record probes and wall time. Then do the same for a global read at each depth.

PREDICTION. Before running: is the relationship linear? Where do globals cross over? Does string length of the variable name matter — and by how much?

RESULT. Plot it or tabulate it into docs/learning/04-interpreter.md. This table is the Section 3 motivation, in your own numbers, and you will refer back to it in Lab 10.

Test

#![allow(unused)]
fn main() {
#[test]
fn shadowing_and_the_initializer_rule() {
    // Lua 5.4 §3.5. Verified: lua -e 'local x=1 do local x=2 end print(x)'
    assert_eq!(run("local x = 1; do local x = 2 end; return x"), "1");
    // The initializer is evaluated in the ENCLOSING scope, before the new
    // binding exists. JavaScript's `let x = x` is a ReferenceError; Lua's is 2.
    assert_eq!(run("local x = 1; local x = x + 1; return x"), "2");
}

#[test]
fn assignment_evaluates_all_rhs_before_assigning() {
    // Lua 5.4 §3.3.3 — this is what makes the swap idiom work.
    assert_eq!(run("local a, b = 1, 2; a, b = b, a; return a .. ',' .. b"), "2,1");
}

#[test]
fn list_adjustment_pads_with_nil_and_discards_extras() {
    assert_eq!(run("local a, b = 1; return type_of(b)"), "nil");
    assert_eq!(run("local a, b = 1, 2, 3; return a"), "1");
}

#[test]
fn missing_global_is_nil_missing_local_does_not_exist() {
    assert_eq!(run("return no_such_name"), "nil");
    // Assigning a name with no `local` creates a global.
    assert_eq!(run("g = 5; return g"), "5");
}

#[test]
fn scopes_are_popped_on_every_exit_path() {
    // The RAII guard is the point. A block whose body errors must still pop.
    let mut interp = Interp::new();
    let _ = interp.run_str("do local hidden = 1; error_here() end");   // errors
    assert!(interp.env_depth() == 0, "scope leaked after an error");
    assert_eq!(interp.run_str("return hidden").unwrap(), "nil");
}

#[test]
fn invalid_assignment_targets_are_compile_errors() {
    for src in ["1 = 2", "\"a\" = 2", "(x) = 2"] {
        let e = compile_err(src);
        assert_eq!(e.kind, ErrorKind::Parse, "{src} must fail at PARSE time");
    }
}
}

Challenge Extensions

Strict mode. Add --strict: reading a global that was never assigned is an error, not nil. Run it over your golden corpus. How many real typos does it catch? Would you make it the default? Write the ADR either way.
Shadowing lint. Warn when a local shadows a name in an enclosing scope. Compare with luacheck's behavior. Is the warning worth the noise in real Lua code?
Environment option B. Replace the Vec<HashMap> with a single HashMap plus a save/restore undo log on scope exit. Benchmark against A. Report the speedup and the bug you introduced while writing it — there is one, and it involves shadowing.
Probe histogram. Make --trace-scope emit a histogram of probe counts over a whole program run. Run it on your largest golden program. What fraction of reads are depth-1?
_ENV. Implement Lua's real design: globals as an upvalue named _ENV, with x desugaring to _ENV.x in the parser. It cannot fully work until Lab 14 gives you upvalues — do the desugaring now and note exactly what is missing.

Deliverables

local, assignment, do ... end, and expression statements parse and evaluate.
local x = x reads the outer x; a, b = b, a swaps. Both tested with the rule named.
List adjustment pads with nil and discards extras.
Globals live in a Table, not a HashMap; missing globals read as nil.
Invalid assignment targets are parse errors with spans.
return must be the last statement in a block.
Scopes are popped via an RAII guard, with a test that errors mid-block.
--trace-scope prints push/pop/resolve with probe counts.
The probe-count experiment is recorded in docs/learning/04-interpreter.md.
docs/adr/ADR-012-globals-as-a-table.md written.

Validation / Self-check

Which single line makes local x = x read the outer x? What do JavaScript and the naive implementation do instead?
Why does a, b = b, a swap? What is the rule, and where is it in the Lua manual?
What is at_block_end computing, and what goes wrong if a token is missing from it?
Why is 1 = 2 a parse error rather than a runtime error?
Why is x alone not a statement, while x() is?
Give the probe count for reading a variable declared 5 scopes out, and what it becomes in Lab 10.
Why are globals a Table rather than a HashMap? Name the Section 5 capability it enables.
What compiler error does the naive ScopeGuard { env: &mut Env } produce, and what does that teach you about RAII in Rust?

Next: Lab 6 — Control Flow.

Open-Source Engineer & Contributor