Lab 24: Diagnostics (Milestone 14)

Background

You will build the diagnostic renderer, add parser error recovery, and wire the help: line. Almost none of the information is new — it has been threaded through since Lab 1. This lab is where it gets displayed.

Why This Lab Matters

• It is the payoff chapter for four labs' worth of span discipline. If the caret lands correctly, the decisions in Labs 1, 2, 4, and 9 were right.
• Error recovery is a measurable feature, not a vibe, and the measurement is "how many of the extra reported errors are real?"
• A diagnostic is the only part of a runtime most users ever read carefully.

Prerequisites

• Section 5 complete.
• Diagnostics read.

Predict First

1. return base * article.boost where boost is nil — where should the caret point, and why is that possible?
2. A file with five syntax errors. Without recovery, how many do you see? With naive recovery, how many are real?
3. Rendering a caret under a span on a line containing tabs. What goes wrong if you convert tabs to spaces?
4. A traceback with 200 frames. What should be printed?
5. os.time() in an Ember script. What is the error, and what should it say?

Step 1: The Renderer

#![allow(unused)]
fn main() {
pub struct Renderer<'a> {
    map: &'a SourceMap,
    color: bool,            // --no-color and NO_COLOR both honored
    context_before: usize,  // 2
    context_after: usize,   // 1
    max_line_width: usize,  // 120; window longer lines
}

impl Renderer<'_> {
    pub fn render(&self, err: &EmberError, src: SourceId) -> String { /* ... */ }
}
}

The four rules from the chapter, each as a test:

#![allow(unused)]
fn main() {
#[test] fn caret_aligns_under_tabs() { /* prefix copies tabs through */ }
#[test] fn long_lines_are_windowed_with_an_elision_marker() { }
#[test] fn multi_line_spans_get_a_summary_not_a_caret() { }
#[test] fn no_color_output_contains_no_ansi_escapes() { }
}

The tab test is the one that catches the common bug. The padding must copy the prefix character by character, substituting a space for everything except a tab:

#![allow(unused)]
fn main() {
let pad: String = prefix.chars().map(|c| if c == '\t' { '\t' } else { ' ' }).collect();
}

Step 2: Context Lines and Windowing

policy.ember:12:17: error: attempt to multiply a nil value

  10 │ function score(article)
  11 │   local base = article.semantic_score
  12 │   return base * article.boost
     │                 ^^^^^^^^^^^^^ this is nil
  13 │ end

The gutter width comes from the largest line number printed, so a four-digit line number does not shift the bars. Windowing for long lines:

 142 │ …ml.decode(payload).items[3].metadata.tags[0].normalized_scor…
     │                             ^^^^^^^^ this is nil

Step 3: Parser Error Recovery

Add synchronize() from the chapter, and change parse_chunk to collect errors rather than returning at the first:

#![allow(unused)]
fn main() {
pub fn parse_chunk(&mut self) -> (Block, Vec<EmberError>) {
    let mut stmts = Vec::new();
    let mut errors = Vec::new();
    while !self.at_block_end() {
        match self.statement() {
            Ok(s) => stmts.push(s),
            Err(e) => {
                errors.push(e);
                if errors.len() >= MAX_REPORTED_ERRORS { break; }
                self.synchronize();
                // A synchronize that does not CONSUME at least one token is an
                // infinite loop. Assert it.
                debug_assert!(self.pos > pos_before, "synchronize made no progress");
            }
        }
    }
    (Block { stmts, .. }, errors)
}
}

Warning: synchronize must always make progress. A recovery loop that lands on a token it also refuses to consume spins forever on a file that a user is waiting on. The debug_assert catches it; a MAX_REPORTED_ERRORS cap catches the pathological-input case.

Compilation must not proceed if there are parse errors. Report all of them, then stop — a tree containing error nodes will produce cascade errors from the compiler that are pure noise.

Step 4: The help: Line

Three sources, in increasing order of value:

#![allow(unused)]
fn main() {
// 1. Userdata field miss → list the registered fields. The builder knows them.
// 2. Missing global → Levenshtein ≤ 2 against the globals table.
// 3. A LUA-ONLY NAME → say so, and say why, and say what to do.
static LUA_ONLY: &[(&str, &str)] = &[
    ("os",     "excluded for determinism and process safety; ask your host to grant CLOCK"),
    ("io",     "excluded: no ambient filesystem. Your host can register narrow functions"),
    ("require","not installed: your host must supply a module resolver"),
    ("load",   "excluded: compiling code at run time defeats static policy review"),
    ("debug",  "excluded: it defeats every sandbox control"),
    ("coroutine", "not implemented (see docs/limitations.md)"),
    ("utf8",   "not implemented; Ember strings are bytes"),
];
}

That table turns every documented divergence into a helpful error, which means appendix/lua-differences.md earns its keep twice. Generate it from the appendix if you can, so they cannot drift.

Step 5: Traceback Rendering

Truncation, copying Lua: first 10, ..., last 11.

stack traceback:
  in function 'score'    policy.ember:12
  in function 'rank'     policy.ember:22
  in function 'apply'    policy.ember:31
  ... (183 more)
  in function 'main'     policy.ember:4
  in main chunk          policy.ember:41
  in <host>

in <host> at the bottom is worth having: it tells a reader the call originated in Rust, which answers "how did we get here?" for a policy invoked by engine.call.

The Trace

Every error kind, rendered:

$ ember run -e 'return 0x'
<argv>:1:8: error: malformed hexadecimal number

   1 │ return 0x
     │        ^^

$ ember run -e 'return 1 + * 2'
<argv>:1:12: error: expected an expression, found '*'

   1 │ return 1 + * 2
     │            ^

$ cat > /tmp/p.ember <<'EOF'
function score(article)
  local base = article.semantic_score
  return base * article.boost
end
return score({semantic_score = 1.0})
EOF
$ ember run /tmp/p.ember
/tmp/p.ember:3:17: error: attempt to multiply a nil value

   1 │ function score(article)
   2 │   local base = article.semantic_score
   3 │   return base * article.boost
     │                 ^^^^^^^^^^^^^ this is nil
   4 │ end

stack traceback:
  in function 'score'   /tmp/p.ember:5
  in main chunk         /tmp/p.ember:5

Trace the caret backwards through the labs:

Seven pieces, six labs, one message. Defer any one of them and this degrades to error: runtime error.

And recovery:

$ cat > /tmp/bad.ember <<'EOF'
local a = 1 +
local b = 2
if a then
  print(b
end
return a b
EOF
$ ember run /tmp/bad.ember
/tmp/bad.ember:1:14: error: expected an expression, found 'local'
/tmp/bad.ember:4:10: error: expected ')', found 'end'
/tmp/bad.ember:6:12: error: syntax error near this expression (expected '=' or a function call)
3 errors
$ echo $?
1

Three errors for three mistakes. Count yours: if the same file produces eight, your synchronization set is recovering mid-expression and generating cascade noise.

And the help: line:

$ ember run -e 'return os.time()'
<argv>:1:8: error: attempt to index a nil value (global 'os')

   1 │ return os.time()
     │        ^^

help: `os` exists in Lua but not in Ember: excluded for determinism and process
      safety; ask your host to grant the CLOCK capability.

$ ember run -e 'return articl.score'
<argv>:1:8: error: attempt to index a nil value (global 'articl')
help: did you mean `article`?

Expected Output

$ cargo test --test diagnostics
test caret_aligns_under_tabs ... ok
test caret_points_at_the_operand_not_the_operator ... ok
test long_lines_are_windowed ... ok
test no_color_output_contains_no_ansi_escapes ... ok
test recovery_reports_all_real_errors ... ok
test synchronize_always_makes_progress ... ok
test traceback_is_truncated ... ok
test lua_only_names_get_a_help_line ... ok

Debugging Steps

The caret is under the operator

The runtime error was constructed with the Binary node's span rather than the operand's.

The caret is offset in a file with tabs

The padding converted tabs to spaces.

Recovery reports twelve errors for four mistakes

The synchronization set includes tokens that appear mid-expression. Restrict it to statement starters and terminators.

The parser spins forever on a malformed file

synchronize landed on a token it will not consume. The debug_assert names it.

The traceback shows the callee's definition line for every frame

call_span versus the proto's span. Lab 11's bug, resurfacing.

The help: table has drifted from the appendix

Generate one from the other. Two lists of the same facts always diverge.

Experiment

CLAIM. Error recovery is worth its complexity only if most of the extra errors it reports are real.

METHOD. Take five real Ember files from your corpus. Introduce N deliberate syntax errors into each (N = 1, 3, 5). Run with and without recovery. For each reported error, classify it by hand: real (corresponds to a deliberate mistake) or cascade (an artifact of recovering in the wrong place).

PREDICTION. What real-to-cascade ratio do you expect? What ratio would make you revert recovery?

RESULT. Record the table in docs/learning/14-diagnostics.md. Then tune the synchronization set and re-measure. This is the protocol from performance engineering applied to a usability feature, which is a transferable move: usability changes can be measured too, if you define the metric first.

Test

#![allow(unused)]
fn main() {
#[test]
fn caret_points_at_the_operand_not_the_operator() {
    let out = render_error("local t = {} return 1 * t.missing");
    let caret_line = out.lines().find(|l| l.contains('^')).unwrap();
    let source_line = out.lines().find(|l| l.contains("return")).unwrap();
    let caret_col = caret_line.find('^').unwrap();
    assert_eq!(&source_line[caret_col..caret_col + 9], "t.missing");
}

#[test]
fn caret_aligns_under_tabs() {
    let out = render_error("local t = {}\n\t\treturn 1 * t.missing");
    let caret_line = out.lines().find(|l| l.contains('^')).unwrap();
    // The padding must contain the SAME tabs as the source prefix.
    assert_eq!(caret_line.matches('\t').count(), 2);
}

#[test]
fn recovery_reports_all_real_errors_and_few_cascades() {
    let src = "local a = 1 +\nlocal b = 2\nif a then\n  print(b\nend\nreturn a b\n";
    let errs = parse_collecting(src);
    assert!(errs.len() >= 3, "missed a real error");
    assert!(errs.len() <= 4, "cascade noise: {} errors for 3 mistakes", errs.len());
}

#[test]
fn synchronize_always_makes_progress() {
    // Every token kind, as the token synchronize lands on.
    for kind in every_token_kind() {
        let mut p = parser_positioned_at(kind);
        let before = p.pos();
        p.synchronize();
        assert!(p.pos() > before || p.at_eof(), "no progress at {kind:?}");
    }
}

#[test]
fn traceback_is_truncated_and_names_the_caller() {
    let e = deep_error(200);
    assert!(e.traceback.len() <= 22);
    assert!(render(&e).contains("... ("));
    assert!(render(&e).contains("in <host>"));
}

#[test]
fn lua_only_names_get_a_help_line_generated_from_the_appendix() {
    for (name, _) in LUA_ONLY {
        let out = render_error(&format!("return {name}.anything"));
        assert!(out.contains("help:"), "no help line for `{name}`");
    }
    // And the coupling: every entry must exist in the appendix.
    let documented = names_in_lua_differences_appendix();
    for (name, _) in LUA_ONLY { assert!(documented.contains(name), "{name} undocumented"); }
}

#[test]
fn no_color_output_contains_no_ansi_escapes() {
    let out = Renderer::new(&map).color(false).render(&err, src);
    assert!(!out.contains('\x1b'));
}
}

Challenge Extensions

1. Compare with ariadne. Render the same five errors with ariadne and with yours. Which is better? Would you take the dependency? Write the ADR either way — this is the moment Section 1 deferred it to.
2. Multi-line spans. Proper rendering with a bracket down the gutter. This is where ariadne earns its keep; do it once by hand to know what you would be buying.
3. Suggested fixes. expected ')' → suggest inserting one, with a machine-applicable span. Now an editor could apply it, which is the LSP protocol's CodeAction.
4. A --json diagnostic format, matching rustc's, so tooling can consume it. Then note that this makes the format an API, and version it.
5. Error deduplication. The same error at the same span reported twice is noise. Dedupe by (span, kind, message) before rendering.

Deliverables

• Renderer with context lines, gutter alignment, windowing, tab-preserving carets, and --no-color/NO_COLOR.
• The caret points at the operand; the test asserts the exact substring.
• Parser error recovery with a synchronization set, a progress assertion, and a reported-error cap.
• Compilation does not proceed when there are parse errors.
• Tracebacks truncated Lua-style, naming the caller's line, ending in in <host>.
• help: lines for userdata field misses, close-match globals, and Lua-only names — the last generated from or checked against appendix/lua-differences.md.
• The recovery experiment recorded, with the real/cascade table and a tuned synchronization set.
• docs/learning/14-diagnostics.md written.

Validation / Self-check

1. Name the seven pieces of the runtime-error diagnostic and the lab each came from.
2. Why does the caret land under the operand? Name both contributing decisions.
3. Why must the caret's padding preserve tabs?
4. What must synchronize always do, and what is the symptom of failing to?
5. What metric decides whether error recovery was worth adding? What was your ratio?
6. Why must compilation stop when there are parse errors?
7. Why should the help: table be generated from the appendix rather than maintained beside it?
8. What does in <host> at the bottom of a traceback tell a reader?

Next: Lab 25 — The CLI and REPL.