Lab 17: Multiple Returns and Varargs (Milestone 12)

Background

You will implement Lua's list-adjustment rules in both backends: the 255 sentinel (biased), the Paren AST node, VARARG, and select.

Your differential tests will fail on the first run. That is not a setback; it is the single clearest demonstration in the curriculum of why ADR-003 exists. The tree walker returns a Vec<Value>; the VM leaves values on a shared stack. Two structurally different implementations of one fiddly rule is exactly the situation the comparison was built for.

Why This Lab Matters

• Every Lua idiom depends on it: local ok, err = pcall(f), for k, v in pairs(t), return nil, "message", f(table.unpack(args)).
• It is the one place syntax carries semantics — (f()) truncates — which forces an AST node the design otherwise refuses.
• It is a stack-layout problem. The reason it is hard in a VM and easy in a tree walker is that the VM has to express "however many there are" in a fixed-width operand.

Prerequisites

• Labs 13–16 complete.
• Multiple Returns and Varargs read.
• Warm-up Experiment 4 re-run with your predictions in hand.

Predict First

1. local a, b, c = f() where f returns two values — what is c?
2. print(f(), "z") versus print("z", f()) — how many values does f contribute in each?
3. #{f()} versus #{f(), 1} where f returns three — what are they?
4. (f()) — how many values, and what does that require of the AST?
5. sum(1, nil, 3) — what does select('#', ...) give? What does #{...} give?
6. A function returns exactly 255 values under an unbiased sentinel encoding. What goes wrong?

Step 1: Bias the Count Encoding

Before anything else, fix the encoding flaw noted in the concept chapter:

#![allow(unused)]
fn main() {
// BIASED: the operand is (count + 1), so 0 means "all of them".
//   0 → MULTRET ("however many there are")
//   1 → zero values
//   n → n-1 values
// Unbiased, 255 would mean both "all" and "exactly 255", which a generated
// program can reach. Lua uses this bias for exactly this reason.
const MULTRET: u8 = 0;
}

Update CALL, RETURN, and VARARG in the opcode reference, the disassembler (CALL 2 0 ; all results), and the validator.

Checkpoint question. With the bias, what is the maximum number of results a call site can request explicitly, and what happens if a program needs more?

Step 2: The Paren Node

#![allow(unused)]
fn main() {
pub enum Expr {
    // ...
    /// The ONLY reason parentheses survive parsing. `(f())` yields exactly one
    /// value. Nothing else about this node has meaning — do not "clean it up".
    Paren { inner: Box<Expr>, span: Span },
}
}

primary() stops discarding parentheses only when the inner expression is multi-valued:

#![allow(unused)]
fn main() {
TokenKind::LParen => {
    let inner = self.expr_bp(0)?;
    self.expect(TokenKind::RParen)?;
    if is_multi_valued(&inner) { Expr::Paren { inner: Box::new(inner), span } } else { inner }
}
}

Keeping the node only when it matters means (1 + 2) still produces the same tree it always did, so no existing test or trace changes. A semantic exception with a syntactic guard.

Step 3: compile_exprlist, One Function

Write it per the concept chapter. Then find every list context and route it through:

Warning: If any of these hand-rolls the adjustment, it will differ — and it will be the one your corpus does not cover. Grep for every place that compiles more than one expression in sequence and confirm it calls compile_exprlist. There should be exactly five call sites.

Step 4: CALL and RETURN in the VM

#![allow(unused)]
fn main() {
fn do_return(&mut self, n: u8) -> Result<Option<Vec<Value>>> {
    let frame = self.frames.pop().unwrap();
    let count = if n == MULTRET { self.stack.len() - frame.result_base } else { (n - 1) as usize };
    let results: Vec<Value> = self.stack[self.stack.len() - count..].to_vec();

    self.close_upvalues(frame.base);          // Lab 14: BEFORE the truncate
    self.stack.truncate(frame.ret_to);
    if self.frames.is_empty() { return Ok(Some(results)); }

    match frame.want {
        MULTRET => self.stack.extend_from_slice(&results),
        w => for i in 0..(w - 1) as usize {
                 self.stack.push(results.get(i).copied().unwrap_or(Value::Nil))
             },
    }
    Ok(None)
}
}

result_base is new. For a RETURN MULTRET, the VM needs to know where "the results" start — which is the stack position at the point the returning expression list began. Record it when the return statement's list starts compiling and stash it in the frame, or compute it from the frame's declared slot count. Whichever you choose, write the invariant down; it is the piece that makes return f() work.

Step 5: Varargs

#![allow(unused)]
fn main() {
pub struct CallFrame {
    // ...
    vararg_base: usize,     // first extra argument's stack index
    n_varargs: usize,       // recorded at CALL; NOT derivable later
}
}

do_call records both; VARARG n copies them up. And select:

#![allow(unused)]
fn main() {
// select('#', ...) → the COUNT, including trailing nils.
// select(n, ...)   → everything from the nth onward.
// Both are library functions (Lab 21 formalizes them); the VM only provides VARARG.
}

Checkpoint question. Why can n_varargs not be recomputed at VARARG time from nparams and the stack? (Because the stack has moved: the frame's slots were reserved above the arguments.)

Step 6: Run the Differential Test and Read the Failures

$ cargo test --test differential
test backends_agree_on_the_whole_corpus ... FAILED
  OUTPUT differs in functions/multret-in-constructor
    left:  "3"      (interp)
    right: "1"      (vm)

Work each one with the four-step procedure. Expect three to six divergences. The likely ones, in the order people hit them:

Each fix leaves a corpus case behind. That is the rule from Lab 12 and it is what turns a fixed instance into a fixed class.

The Trace

$ ember trace -e 'local function three() return 1,2,3 end
                  local a, b, c, d = three()
                  return d'

dep  ip    op                       stack after
  1  0000  CLOSURE     0            [<fn three>]
  1  0002  GET_LOCAL   0            [<fn>, <fn>]
  1  0003  CALL        1 5          → want = 5-1 = 4 results
  2  0000  LOAD_INT    1            [<fn>, <fn>, 1]
  2  0001  LOAD_INT    2            [<fn>, <fn>, 1, 2]
  2  0002  LOAD_INT    3            [<fn>, <fn>, 1, 2, 3]
  2  0003  RETURN      4            → n-1 = 3 results: [1,2,3]
                                    → truncate to ret_to=1
                                    → want=4: push 1, 2, 3, NIL   ← the pad
  1  0005  GET_LOCAL   3     ; d    [<fn>, 1, 2, 3, nil, nil]
  1  0006  RETURN      2
nil

The → want=4: push 1, 2, 3, NIL line is the adjustment, and it happens in exactly one place.

Now the four contexts side by side, which is the table to keep:

$ ember run -e 'local function f() return 1,2,3 end return select("#", f())'         --> 3
$ ember run -e 'local function f() return 1,2,3 end return select("#", (f()))'       --> 1
$ ember run -e 'local function f() return 1,2,3 end return select("#", f(), "z")'    --> 2
$ ember run -e 'local function f() return 1,2,3 end return #{f()}'                   --> 3
$ ember run -e 'local function f() return 1,2,3 end return #{f(), "z"}'              --> 2

And the disassembly that shows why:

$ ember disassemble -e 'local function f() end local t = {f(), 1}'
0004     |  CALL        1 2          ; ← want 1 result: NOT last in the list
0006     |  LOAD_INT    1
0007     |  SET_LIST    2 0

$ ember disassemble -e 'local function f() end local t = {1, f()}'
0004     |  LOAD_INT    1
0005     |  CALL        1 0          ; ← want ALL: it IS last  (biased 0 = MULTRET)
0007     |  SET_LIST    0 0          ; ← n=0 means "everything on the stack"

One operand differs — 2 versus 0 — and it is the entire multiple-return rule. Put those two listings in docs/learning/12-multret.md; they are the clearest possible statement of what the compiler decides and what the VM merely obeys.

Expected Output

$ ember run -e 'local function f() return 1,2,3 end
                local a,b,c,d = f() return tostring(a)..tostring(b)..tostring(c)..tostring(d)'
123nil
$ ember run -e 'local function f() return 1,2,3 end local x,y = f() return x..","..y'
1,2
$ ember run -e 'local function s(...) return select("#", ...) end return s(1, nil, 3)'
3
$ ember run -e 'local function s(...) return #{...} end return s(1, nil, 3)'
1
$ ember run -e 'local function f() return 1,2 end local function g(...) return f(...) end
                return select("#", g())'
2

That fourth line — #{...} giving 1 where select('#') gives 3 — is the one people call a bug. It is not: {1, nil, 3} has a hole, and # is the length of the trimmed array part. Two different questions, two different answers, and a corpus case for each.

Debugging Steps

{f()} has one element

Constructor positional fields are not using compile_exprlist, or the last field is not getting Want::All.

(f()) returns everything

Paren is missing, or is_multi_valued recurses into it. It must return false for Paren.

return f() returns one value

RETURN's count is fixed. It must be MULTRET when the returned list ends in a call.

A call in the middle of a list contributes more than one value

Your loop compiles all expressions the same way. Only exprs.last() may stay open.

select('#', ...) disagrees between backends with trailing nils

The tree walker is trimming trailing nils from its Vec<Value>. nil in a value list is not a terminator.

The stack grows across many multi-return calls

RETURN's truncate target is base instead of ret_to, so the varargs region survives.

f(table.unpack(t)) with a big t exhausts memory

Expected without a cap. Add a max-results check before pushing.

Experiment

CLAIM. Lua's design avoids an allocation per multi-return, and the saving is measurable.

METHOD. Implement a second RETURN path behind a flag that boxes the results in a table and returns one value (option B from the concept chapter). Rewrite pcall-style code to unbox. Benchmark a loop calling a two-value-returning function a million times.

PREDICTION. How many allocations does the tuple version add? What is the wall-clock ratio?

RESULT. Record it in docs/learning/12-multret.md. Then note the other side: how much simpler was the tuple version's compiler? That is what Python and JavaScript bought with their allocation, and it is a legitimate trade — the point is knowing what was traded.

Test

#![allow(unused)]
fn main() {
#[test]
fn every_warmup_experiment_4_case_matches_lua() {
    // The complete rule set, from the warm-up. Each row was verified with `lua`.
    let f = "local function f() return 1,2,3 end ";
    let cases = [
        ("local a,b,c,d = f() return tostring(d)",              "nil"),
        ("local x,y = f() return x..','..y",                    "1,2"),
        ("return select('#', f())",                             "3"),
        ("return select('#', f(), 'z')",                        "2"),
        ("return select('#', (f()))",                           "1"),
        ("return #{f()}",                                       "3"),
        ("return #{f(), 'z'}",                                  "2"),
        ("return #{'z', f()}",                                  "4"),
    ];
    for (src, want) in cases { assert_eq!(run(&format!("{f}{src}")), want, "{src}"); }
}

#[test]
fn nil_is_not_a_terminator_in_a_value_list() {
    assert_eq!(run("local function s(...) return select('#', ...) end return s(1, nil, 3)"), "3");
    assert_eq!(run("local function f() return 1, nil, 3 end return select('#', f())"), "3");
}

#[test]
fn varargs_forward_without_allocation() {
    // The case that decided Lua's design: `function f(...) return g(...) end`.
    assert_eq!(run("local function g(...) return select('#', ...) end
                    local function f(...) return g(...) end
                    return f(1,2,3,4,5)"), "5");
}

#[test]
fn parentheses_truncate_everywhere() {
    let f = "local function f() return 1,2,3 end ";
    assert_eq!(run(&format!("{f}return #{{(f())}}")), "1");
    assert_eq!(run(&format!("{f}local a,b = (f()) return tostring(b)")), "nil");
}

#[test]
fn a_call_returning_exactly_the_bias_boundary_works() {
    // With an UNBIASED sentinel this test fails: 255 results would be read as
    // MULTRET. With the bias, the boundary is clean.
    let src = format!("local function f() return {} end return select('#', f())",
                      (1..=200).map(|i| i.to_string()).collect::<Vec<_>>().join(","));
    assert_eq!(run(&src), "200");
}

#[test]
fn huge_result_lists_hit_a_limit_not_the_allocator() {
    let e = run_with_limits("local t={} for i=1,1000000 do t[i]=i end return table.unpack(t)",
                            &Limits::test_defaults()).unwrap_err();
    assert_eq!(e.kind, ErrorKind::Limit);
}
}

Challenge Extensions

1. table.unpack with a range. table.unpack(t, i, j). Then find the largest j - i your implementation survives and put a documented cap below it.
2. table.pack. Returns {n = select('#', ...), ...} — the standard idiom for capturing a vararg list including trailing nils. Once you have it, the #{...} versus select('#') confusion has an answer to point users at.
3. Multiple assignment with side effects. t[f()], t[g()] = h(), k() — what is the evaluation order? Lua's manual is deliberately loose here; Ember guarantees left-to-right. Write the corpus case that pins yours and check it against lua.
4. pcall and multiple returns. pcall(f, ...) returns true followed by all of f's results. Implement it and note how the adjustment interacts with error handling.
5. A MULTRET fuzz target. Generate random nestings of calls in list positions and assert backend agreement. This is the highest-yield fuzz target in the curriculum.

Deliverables

• The count encoding is biased; 0 means MULTRET; the disassembler and validator updated.
• Expr::Paren, kept only when the inner expression is multi-valued.
• compile_exprlist with exactly five call sites; no context hand-rolls adjustment.
• CALL/RETURN adjust in one place; result_base recorded and its invariant documented.
• VARARG, vararg_base, n_varargs, and select.
• All eight warm-up Experiment 4 cases pass and match lua.
• nil is not a terminator; select('#') and #{...} each have a corpus case.
• A max-results cap producing ErrorKind::Limit before allocation.
• Every differential divergence found is fixed and left a corpus case behind, and the first three are written up in docs/learning/12-multret.md.
• The two disassembly listings ({f(), 1} versus {1, f()}) saved in the journal.
• Differential tests green again.

Validation / Self-check

1. Give the five list contexts and the Want each passes.
2. Why must the count encoding be biased? Give the program that breaks without it.
3. Why does Expr::Paren exist, and why is it created conditionally?
4. In {1, f()} versus {f(), 1}, which operand differs in the disassembly and what does it mean?
5. Why is #{...} different from select('#', ...)? Which question does each answer?
6. Why can n_varargs not be recomputed later?
7. Which divergences did your differential test find? Which would you have written a test for?
8. What did Lua trade away by choosing open lists over tuples, and what did it buy?

Next: Lab 18 — Metatables.