Modules and Resolution

Three concepts: the resolver, the cache, and cycles.

This chapter produces ADR-010, and its content is one sentence: the host decides what a module name means. Everything else follows.

Concept 1: The Resolver

1. Concept

require("policy.boosts") asks the runtime for a module. The runtime does not know what that string means — it asks a host-supplied resolver, which returns source (or a pre-compiled chunk), or an error.

#![allow(unused)]
fn main() {
pub trait Resolver {
    /// Resolve a module name to source. The HOST decides what names mean.
    fn resolve(&mut self, name: &str) -> Result<Module, ResolveError>;
}

pub enum Module {
    Source { name: String, src: String },   // compiled by the engine
    Native(Value),                          // a host-built table
    Precompiled(Rc<Proto>),                 // from a cache; VALIDATED before use
}
}

2. Problem

Lua's default require searches package.path — a filesystem search path, configurable from inside the script. For an embedded runtime that is three problems in one: ambient filesystem authority, a path-traversal surface, and a resolution policy the host cannot see.

require("../../../etc/passwd")     -- what does your path search do with this?
package.path = "/tmp/?.lua"        -- the script rewrote its own search path

3. Mental model

require is a function call into host code that happens to return a table. The runtime supplies caching and cycle detection; the host supplies meaning.

4. Implementation

#![allow(unused)]
fn main() {
fn require(ctx: &mut Ctx, name: &str) -> Result<Value> {
    if let Some(&cached) = ctx.modules().cache.get(name) { return Ok(cached); }

    // Cycle detection BEFORE resolving, so a self-require is caught immediately.
    if ctx.modules().loading.contains(name) {
        return Err(ctx.error(format!(
            "cyclic require: {}", ctx.modules().loading_chain_with(name))));
    }
    ctx.modules().loading.insert(name.to_string());

    let result = (|| {
        let module = ctx.resolver().resolve(name)?;              // ← HOST CODE
        let value = match module {
            Module::Native(v) => v,
            Module::Source { name, src } => {
                let proto = ctx.compile(&src, &name)?;
                ctx.call_chunk(proto, &[])?.into_iter().next().unwrap_or(Value::Nil)
            }
            Module::Precompiled(p) => {
                validate(&p)?;                                    // ← NEVER trust a cached chunk
                ctx.call_chunk(p, &[])?.into_iter().next().unwrap_or(Value::Nil)
            }
        };
        ctx.modules().cache.insert(name.to_string(), value);
        Ok(value)
    })();

    ctx.modules().loading.remove(name);      // on BOTH paths — use a guard
    result
}
}

Four details:

1. resolve is host code called from inside the VM — so it is a re-entrancy point, and all three rules apply. It may allocate; it may even call back into the engine.
2. The loading set is removed on both paths. Another RAII guard; another instance of the pattern from the parser, the environment, and the VM's frames.
3. Precompiled chunks are validated. A bytecode cache is exactly the case the validator was built for. Skipping validation because "we wrote it ourselves" fails the moment the cache is on disk, shared, or versioned.
4. The cache stores the module's value, not its source. Requiring twice returns the identical table — which is the semantics scripts rely on for shared state, and a real aliasing consequence.

5. Alternatives

6. Decision

ADR-010: A. The engine provides caching, cycle detection, compilation, and validation. The host provides resolution — and with it, all path handling, all allowlisting, and all authorization.

Ember ships two resolvers and neither is the default:

#![allow(unused)]
fn main() {
MemoryResolver::from([("boosts", BOOSTS_SRC), ("rules", RULES_SRC)])   // the usual choice
FsResolver::rooted("/etc/policies")   // opt-in, and its docs are half warnings
}

FsResolver's documentation must state that path safety is its responsibility and describe how it achieves it — canonicalize, then verify the result is still under the root, and reject symlinks that escape. That is a security-critical twenty lines and it should read like it.

7. Tradeoffs

8. Production concerns

• A module name is untrusted input. It reaches resolve verbatim. Bound its length, reject interior NULs, and let the resolver decide about .. — the engine must not try to normalize it, because normalization rules differ per resolver and a half-normalization is worse than none.
• require must be a capability. In SAFE there is no require at all unless the host installs one. Enumerated in the capability surface.
• Module loading counts against the budget. Compiling a large module is real work; a script that requires in a loop over generated names is a CPU vector. Charge compilation to the instruction budget and cap the number of distinct modules.

9. References

• Lua 5.4 Reference Manual §6.3 (package), and package.searchers — Lua's own extension point, which is option A wearing a Lua costume.
• Node's require resolution algorithm, as an example of how complex "what does this name mean" becomes when the runtime owns it.
• Python's import system and the long history of sys.path manipulation as an attack vector.

Concept 2: The Cache

1–3. Concept, problem, mental model

require("x") twice must return the identical value, not two copies. A module is a singleton per engine, and scripts rely on that for shared state.

-- a.ember
local counters = require("counters")
counters.hits = counters.hits + 1
-- b.ember
local counters = require("counters")     -- the SAME table

4. Implementation

The cache is HashMap<String, Value> on the engine, and it is root set 8 — add it to enumerate_roots in the same commit, or a required module gets collected while a script still references it through a local.

Warning: That is the seven-root-sets list becoming eight, and this is exactly the situation the Lab 15 count test exists to catch. Bumping the constant should be a deliberate, reviewed edit.

5–8. Alternatives, decision, production concerns

Decision: A, plus B for live reload.

Live policy reload is a real capstone requirement and it is worth being precise about what B does not do: evicting "boosts" does not affect a script that already called require("boosts") and holds the table in a local. The old module keeps running until its holders are gone. That is not a bug — it is what makes reload safe under concurrency — but it means "reload" means "new evaluations get the new version", and the capstone's documentation must say so.

The cleanest reload design, and the one the capstone uses:

   new policy source
        │
        ▼
   build a NEW Engine, load, VALIDATE with a smoke evaluation
        │
        ├─ ok    → atomically swap the Arc<Engine-per-worker> pointer
        └─ error → keep the old engine, report, alert

Swapping whole engines rather than modules sidesteps every partial-reload hazard, and it makes validation possible: you can run the new policy on a canned input before promoting it. Note this in docs/embedding.md; it is the single most useful operational pattern in the section.

Concept 3: Cycles

1–3. Concept, problem, mental model

-- a.ember:  local b = require("b")
-- b.ember:  local a = require("a")     ← a is still loading

When b requires a, a's module table does not exist yet — a is mid-execution. Something must happen, and the options are: error, or return a partial module.

4–7. Implementation, alternatives, decision

Decision: A. B is what Node and Python do and it is defensible for a general-purpose module system where cycles between large modules are hard to avoid. For a policy engine, where modules are small and a cycle is a mistake, an error naming the chain is strictly more useful than a table whose fields are conditionally nil.

The error must print the chain, not just "cycle detected":

error: cyclic require: "policy.main" → "policy.boosts" → "policy.rules" → "policy.boosts"

Half the value of cycle detection is the chain. Keep loading as an ordered Vec (with a set beside it for O(1) membership) so you can print it.

8. Production concerns

• The loading set must be cleared on error paths. A failed require that leaves its name in loading makes every subsequent attempt report a spurious cycle. RAII guard, fourth appearance.
• A cycle through a native module is possible too — a host resolver that calls back into the engine and requires something. Same detection, same chain.
• Depth. require nesting consumes Rust stack through the resolver and the compiler. Count it against the call-depth limit, or a chain of 10,000 modules overflows.

9. References

• Node.js's documentation on circular dependencies, including its explicit description of the partial-object behavior — read it as a case study in documenting a sharp edge rather than removing it.
• Python's import system on circular imports, and ImportError: cannot import name X (most likely due to a circular import) — an error message that got much better over time and is worth studying.

Things to Notice

• The host decides what a name means. One sentence, and it removes ambient filesystem authority, path traversal, and script-controlled resolution in one move.
• A cached chunk is untrusted input. Validate it. This is why the validator exists.
• The module cache is a GC root set, and adding it takes the count from seven to eight — a deliberate, reviewed edit.
• Reload means "new evaluations get the new version", not "everything switches". Swap whole engines and validate before promoting.
• Cycle errors must print the chain. Half the value is in the chain.
• The RAII guard pattern appears for the fifth time — parser depth, scope, frames, temp roots, and now loading. Any push/pop straddling a ? belongs in a Drop.

Validation / Self-check

1. State ADR-010 in one sentence. What three problems does it remove?
2. Why must a Precompiled module be validated, given that your own compiler produced it?
3. Why does the cache store the module's value rather than its source? What script behavior depends on that?
4. Which root set does the module cache become, and why is bumping the count a reviewed edit?
5. Why is require a capability rather than a core function?
6. What does invalidate_module not do, and why is that the safe behavior?
7. Describe the whole-engine reload pattern and the one thing it makes possible that module-level reload does not.
8. Give the three cycle policies and say which two production languages chose the one Ember rejected.
9. Why must the loading set be an ordered Vec as well as a set?

Next: The Threat Model.