Lab 22: Modules (Milestone 13)

Background

You will build src/module.rs: the Resolver trait, require, the module cache, cycle detection with a printed chain, and a bytecode cache that goes through the validator.

require is a capability, not a core function — it does not exist in SAFE unless the host installs it.

Why This Lab Matters

ADR-010 in one sentence: the host decides what a module name means. That removes ambient filesystem authority, path traversal, and script-controlled resolution together.
A cached chunk is untrusted input, and this is the lab where the validator stops being a nice-to-have.
The capstone needs live policy reload, and this lab establishes why the right unit of reload is a whole engine.

Prerequisites

Labs 19–21 complete.
Modules and Resolution read.

Predict First

require("x") twice — same table or two? What script behavior depends on the answer?
a requires b which requires a. What are the three possible behaviors, and which do Node and Python choose?
A Precompiled chunk comes from your own compiler, cached on disk. Validate it or not?
require("../../etc/passwd") — whose problem is the ..?
invalidate_module("boosts") while a running script holds the old table. What happens to it?
Which GC root set does the module cache become?

Step 1: The Resolver Trait and Two Implementations

#![allow(unused)]
fn main() {
pub trait Resolver {
    fn resolve(&mut self, name: &str) -> Result<Module, ResolveError>;
}

pub enum Module {
    Source { name: String, src: String },
    Native(Value),
    Precompiled(Rc<Proto>),
}
}

#![allow(unused)]
fn main() {
/// The usual choice: an explicit allowlist, fixed at build or config time.
pub struct MemoryResolver { modules: HashMap<String, String> }

/// Opt-in, and its documentation is half warnings.
pub struct FsResolver { root: PathBuf }

impl Resolver for FsResolver {
    fn resolve(&mut self, name: &str) -> Result<Module, ResolveError> {
        // PATH SAFETY IS THIS FUNCTION'S RESPONSIBILITY. The engine does not
        // normalize the name, because normalization rules differ per resolver
        // and a half-normalization is worse than none.
        if name.contains('\0') || name.len() > 256 { return Err(ResolveError::BadName); }
        // Map dots to separators, reject everything else, then CANONICALIZE and
        // verify the result is still under the root. Canonicalizing is what
        // catches symlinks that escape; a textual `..` check does not.
        let rel: PathBuf = name.split('.').map(sanitize_component).collect::<Result<_, _>>()?;
        let full = self.root.join(rel).with_extension("ember");
        let real = std::fs::canonicalize(&full).map_err(|_| ResolveError::NotFound)?;
        if !real.starts_with(std::fs::canonicalize(&self.root).map_err(|_| ResolveError::NotFound)?) {
            return Err(ResolveError::Escaped);       // symlink or traversal
        }
        Ok(Module::Source { name: name.into(), src: std::fs::read_to_string(real)? })
    }
}
}

Warning: Those twenty lines are security-critical and they should read like it. Canonicalize then compare — a textual .. check is defeated by a symlink, and a starts_with on un-canonicalized paths is defeated by root/../root2. Write the tests: .., an absolute path, a symlink pointing outside, a name with a NUL, and a 10 MB name.

Step 2: `require`, the Cache, and Cycles

Write it per the concept chapter. Three things to get right:

#![allow(unused)]
fn main() {
// 1. `loading` is an ordered Vec (plus a set for O(1) membership) so the error
//    can print the CHAIN. Half the value of cycle detection is the chain.
struct Loading { order: Vec<String>, set: HashSet<String> }

// 2. An RAII guard removes the name on BOTH paths. Fifth appearance of this
//    pattern: parser depth, scope, frames, temp roots, and now `loading`.
struct LoadingGuard<'m> { modules: &'m mut Modules, name: String }
impl Drop for LoadingGuard<'_> { fn drop(&mut self) { self.modules.finish_loading(&self.name) } }

// 3. The cache stores the module's VALUE. Requiring twice returns the identical
//    table, which is what scripts rely on for shared state.
cache: HashMap<String, Value>,
}

The cache is root set 8 (or 9, depending on whether the temp-root stack landed first). Add it to enumerate_roots, bump the count constant in the Lab 15 test, and re-run the corpus under --gc-stress.

Step 3: Validate Precompiled Chunks

#![allow(unused)]
fn main() {
Module::Precompiled(p) => {
    // NEVER trust a chunk you did not just compile in this process. It may
    // have come from disk, from a shared cache, or from a different VERSION of
    // this compiler. The validator is one pass and it is the reason it exists.
    validate(&p)?;
    ctx.call_chunk(p, &[])?
}
}

And the serialized format, which is where the "different version" hazard bites:

#![allow(unused)]
fn main() {
struct ChunkHeader {
    magic:   [u8; 4],      // b"EMBC"
    format:  u16,          // bumped on ANY layout change
    opcodes: u64,          // a hash of the opcode table — catches a reordered enum
    flags:   u16,          // debug info present? endianness?
}
}

The opcode-table hash is the detail people skip. A Chunk written by version 0.3 and read by 0.4 with one opcode inserted in the middle is structurally valid and semantically garbage. Lua's lundump.c checks a version byte, a format byte, and the sizes of every type for the same reason.

Step 4: `require` as a Capability

#![allow(unused)]
fn main() {
Engine::builder()
    .capabilities(Capabilities::SAFE)
    .resolver(MemoryResolver::from([("boosts", BOOSTS), ("rules", RULES)]))   // installs `require`
    .build()
}

No resolver, no require. It appears in the capability-surface golden file when it is installed, which means adding module loading to a deployment shows up in a diff.

Step 5: Reload

Engine::invalidate_module(name) evicts a cache entry. It does not affect anyone already holding the old table — which is what makes it safe, and what makes it not the reload primitive the capstone wants.

#![allow(unused)]
fn main() {
// The pattern the capstone uses, and the one to document in docs/embedding.md.
fn reload(current: &ArcSwap<Engine>, src: &str, canary: &CanaryInput) -> Result<()> {
    let mut candidate = build_engine()?;         // fresh: fresh globals, fresh heap, fresh cache
    candidate.execute(src)?;                     // compile errors caught HERE
    let _ = candidate.call::<_, f64>("score", canary)?;   // a SMOKE TEST before promoting
    current.store(Arc::new(candidate));          // atomic swap; in-flight evaluations finish on the old one
    Ok(())
}
}

Swapping whole engines sidesteps every partial-reload hazard, and it makes validation possible: you can run the new policy on a canned input before promoting it. A policy that compiles but returns nil never reaches production.

The Trace

$ cat > /tmp/main.ember <<'EOF'
local boosts = require("boosts")
local rules  = require("rules")
return boosts.freshness(2) + rules.penalty("spam")
EOF
$ ember run --trace-modules --resolver=memory /tmp/main.ember

require "boosts"    cache MISS   loading=[boosts]
  resolve            → Source (412 bytes)
  compile            → 1 proto, 6 constants     [budget -180]
  execute chunk      → table#12
  cache INSERT boosts → table#12                loading=[]
require "rules"     cache MISS   loading=[rules]
  resolve            → Source (298 bytes)
  require "boosts"  cache HIT    → table#12     ← the SAME table, not a second load
  compile            → 1 proto, 4 constants     [budget -140]
  execute chunk      → table#19
  cache INSERT rules  → table#19                loading=[]
1.62

The cache HIT line is the singleton semantics. rules and main hold the same boosts table, so state set by one is visible to the other — which scripts rely on and which is an aliasing consequence worth stating in the docs.

Now the cycle:

$ ember run --trace-modules -e 'return require("a")'
require "a"         cache MISS   loading=[a]
  require "b"       cache MISS   loading=[a, b]
    require "a"     ← already loading
error: cyclic require: "a" → "b" → "a"

  1 │ local a = require("a")
    │           ^^^^^^^^^^^^
$ echo $?
1

The chain is the message. cycle detected would have told you a fact; "a" → "b" → "a" tells you where to look.

And the validator earning its keep:

$ ember compile /tmp/boosts.ember -o /tmp/boosts.embc
$ printf '\x99' | dd of=/tmp/boosts.embc bs=1 seek=64 conv=notrunc 2>/dev/null
$ ember run --module-cache=/tmp /tmp/main.ember
error: invalid bytecode in module 'boosts': jump target out of range at instruction 14
$ echo $?
1

Not a panic, not an out-of-bounds read, not silent nonsense. That is what the validator is for, and until this lab there was no way to demonstrate it.

Expected Output

$ cargo test --test modules
test require_twice_returns_the_same_value ... ok
test cyclic_require_reports_the_chain ... ok
test failed_require_does_not_poison_the_loading_set ... ok
test precompiled_chunks_are_validated ... ok
test opcode_table_hash_rejects_a_mismatched_chunk ... ok
test fs_resolver_rejects_traversal_and_symlink_escape ... ok
test require_is_absent_without_a_resolver ... ok

Debugging Steps

`require("x")` twice returns different tables

The cache stores the source instead of the value, or the insert happens before the chunk runs.

A failed `require` makes every later attempt report a cycle

The loading entry was not removed on the error path. RAII guard.

The cycle error says "cycle detected" with no chain

loading is a HashSet only. Keep an ordered Vec alongside.

A module gets collected while a script still uses it

The cache is not in enumerate_roots.

`FsResolver` allows `../../etc/passwd`

Textual checking instead of canonicalize-then-compare. Also test the symlink case, which textual checking cannot catch at all.

A cached chunk from an older build produces nonsense

The header has no opcode-table hash. Add it; the failure should be "refuses to load", not "runs wrong".

`require` exists in a `SAFE` engine

It is being installed unconditionally instead of with the resolver.

Experiment

CLAIM. A bytecode cache saves meaningful time for a policy loaded on every request, and the validation pass costs a small fraction of what it saves.

METHOD. Time three paths for a 500-line policy: (a) parse + compile from source; (b) load a precompiled chunk with validation; (c) load it without validation (temporarily, for the measurement only).

PREDICTION. What fraction of (a) is (b)? What fraction of (b) is validation?

RESULT. Record it in docs/learning/12-embedding.md. If validation is a small fraction — it will be, it is one linear pass — then the "we skip validation for our own chunks" argument has no performance basis, and you have the number that says so. Delete path (c) afterwards.

Test

#![allow(unused)]
fn main() {
#[test]
fn require_twice_returns_the_same_value() {
    let mut e = engine_with_modules([("m", "local t = {n = 0} return t")]);
    assert_eq!(e.eval::<bool>("local a, b = require('m'), require('m') return a == b").unwrap(),
               true);
    // And the shared-state consequence scripts rely on:
    assert_eq!(e.eval::<i64>("require('m').n = 5 return require('m').n").unwrap(), 5);
}

#[test]
fn cyclic_require_reports_the_chain() {
    let mut e = engine_with_modules([("a", "local b = require('b') return {}"),
                                     ("b", "local a = require('a') return {}")]);
    let err = e.execute("require('a')").unwrap_err();
    assert!(err.message.contains("\"a\" → \"b\" → \"a\""), "{}", err.message);
}

#[test]
fn failed_require_does_not_poison_the_loading_set() {
    let mut e = engine_with_modules([("bad", "error('nope')")]);
    assert!(e.execute("pcall(require, 'bad')").is_ok());
    // A SECOND attempt must report the module's own error, not a spurious cycle.
    let err = e.execute("require('bad')").unwrap_err();
    assert!(!err.message.contains("cyclic"), "{}", err.message);
}

#[test]
fn precompiled_chunks_are_validated() {
    let mut proto = compile_proto("return 1");
    corrupt_jump_target(&mut proto);
    let mut e = engine_with_precompiled([("m", proto)]);
    let err = e.execute("require('m')").unwrap_err();
    assert!(err.message.contains("jump target out of range"));
}

#[test]
fn opcode_table_hash_rejects_a_mismatched_chunk() {
    let mut bytes = serialize(&compile_proto("return 1"));
    bytes[8] ^= 0xff;                                  // corrupt the opcode hash
    assert!(deserialize(&bytes).is_err());
}

#[test]
fn fs_resolver_rejects_traversal_and_symlink_escape() {
    let (root, outside) = temp_tree_with_symlink_escape();
    let mut r = FsResolver::rooted(&root);
    for name in ["..\u{2e}.passwd", "a.b..c", "\u{0}evil"] {
        assert!(r.resolve(name).is_err(), "{name} resolved");
    }
    assert!(r.resolve("escape").is_err(), "a symlink escaped the root");
    assert!(r.resolve("legit").is_ok());
}

#[test]
fn require_is_absent_without_a_resolver() {
    let e = Engine::new();
    assert!(!e.enumerate_globals().contains(&"require".to_string()));
}
}

Challenge Extensions

A signed chunk cache. Add an HMAC over the serialized chunk, keyed by a host secret. Then ask what it buys over validation — the answer is provenance, which the validator explicitly does not provide, and that distinction is worth writing down.
Partial-module cycles, Node-style. Implement option B (return the partial table) behind a flag and run both on a cyclic corpus. Which produces better error messages when the cycle is a mistake?
A database resolver. Modules from a table, versioned, with the version in the cache key. This is what a real policy service does.
Module-level hot reload with generation counters. Each module gets a generation; a script holding an old table can ask require.generation("boosts"). Then decide whether that is a feature or an invitation to write fragile scripts.
ember compile and ember run --module-cache. The CLI surface for the bytecode cache, including a --strip-debug flag whose output must produce identical results and worse error messages.

Deliverables

Resolver trait; MemoryResolver and FsResolver, the latter with canonicalize-then-compare and five negative tests.
require with a value cache, cycle detection printing the chain, and an RAII guard on loading.
The module cache is a GC root set; the count constant bumped; corpus green under --gc-stress.
Precompiled modules go through validate(); a corrupted chunk is a clean error.
A serialized chunk header with magic, format version, and an opcode-table hash; a mismatch refuses to load.
require is installed only with a resolver, and appears in the capability-surface golden file.
Engine::invalidate_module, plus the whole-engine reload pattern documented in docs/embedding.md with the smoke-test step.
--trace-modules showing cache hits, misses, and the loading chain.
The bytecode-cache experiment recorded, and path (c) deleted.
docs/adr/ADR-010-host-controlled-modules.md written.

Validation / Self-check

State ADR-010 in one sentence and name the three problems it removes.
Why does the cache store the module's value rather than its source? Give the script behavior that depends on it.
Why must the loading set be an ordered Vec as well as a set?
Why canonicalize before comparing paths? What does a textual .. check fail to catch?
Why validate a chunk your own compiler produced? Name the two ways it can still be wrong.
What does the opcode-table hash catch that the format version does not?
What does invalidate_module not do, and why is that the safe behavior?
Describe the whole-engine reload pattern and the one thing it makes possible.
What does validation prove about a cached chunk, and what does signing add?

Next: Lab 23 — Limits and Sandboxing.

Open-Source Engineer & Contributor