Host Objects and Userdata

Three concepts: userdata, the copy-versus-borrow decision, and capability boundaries.

This is how article.semantic_score works when Article is a Rust struct — and it is where the capstone's performance is decided, because the alternative (marshaling every candidate into a table) allocates thousands of objects per request.

Concept 1: Userdata

1. Concept

Userdata is a heap object owned by the engine whose contents are an opaque Rust value plus a metatable. Scripts can hold it, pass it, store it in tables, and index it — and cannot see inside it except through the methods the host provides.

2. Problem

#![allow(unused)]
fn main() {
struct Article { id: u64, semantic_score: f64, age_hours: f64, topic: String }
}

A script needs article.semantic_score. Three bad answers and one good one:

3. Mental model

Userdata is a box with a type tag and a metatable. The engine owns the box and the collector traces it. The script holds a handle. Field access goes through __index — the same path tables use, which is why the machinery already exists.

   Value::UserData(GcRef)
            │
   heap ────▼──────────────────────────────────────────┐
     [42] UserData {                                    │
            type_id: TypeId::of::<Article>(),           │  ← the tag: downcast safely
            data:    Box<dyn UserDataValue>,            │  ← the host's Article
            meta:    Some(GcRef<Table>),                │  ← per-TYPE, shared
          }                                             │
   ───────────────────────────────────────────────────-─┘
                          │
   Article's metatable ───▼──── __index = native fn: (ud, key) → field value

4. Implementation

#![allow(unused)]
fn main() {
pub trait UserData: 'static {
    /// Register this type's methods and fields. Called ONCE per type per engine.
    fn build(builder: &mut UserDataBuilder<Self>) where Self: Sized;
    /// Report GC edges. Default: none. Override if the type holds Ember Values.
    fn trace(&self, _grey: &mut Vec<Handle>) {}
}

impl UserData for Article {
    fn build(b: &mut UserDataBuilder<Self>) {
        b.field("id",             |a| a.id as i64);
        b.field("semantic_score", |a| a.semantic_score);
        b.field("age_hours",      |a| a.age_hours);
        b.field("topic",          |a| a.topic.as_str());
        b.method("is_fresh", |a, _ctx, _args| Ok(vec![Value::Boolean(a.age_hours < 6.0)]));
        b.meta_tostring(|a| format!("Article({})", a.id));
    }
}
}

#![allow(unused)]
fn main() {
// Engine side:
let handle = engine.create_userdata(article)?;   // MOVES the Article into the engine
engine.set_global("article", handle)?;
}

Three properties of that design:

1. The metatable is per type, not per object. One Article metatable is built once and shared by every article, exactly as strings share one. Building it per object would allocate a table per candidate — the very cost userdata exists to avoid.
2. type_id makes downcasting safe. A native method registered for Article receiving a Vector3 gets a typed error, not a transmute. This is std::any::TypeId, and it is the reason the trait requires 'static.
3. trace is not optional if the userdata holds Values. A host object storing an Ember callback is a GC edge, and the collector cannot discover it. Default to none, and say loudly what overriding it is for.

5. Alternatives

Lua's light userdata is worth knowing about precisely because Ember refuses it: it is a raw void* with no metatable and no lifetime management, and it exists for C hosts that need to pass an address through. In Rust it would be a raw pointer in a safe API, which is exactly the thing the ownership boundary forbids. Note the omission in appendix/lua-differences.md with that reason.

6. Decision

A only. No light userdata. Option C stays available and is the right answer for small, short-lived, mostly-read data — the choice is Concept 2.

7. Tradeoffs

8. Production concerns

• The engine owns it, so the host cannot mutate it. create_userdata moves the value in. A host that needs to keep mutating must either take it back out (Engine::take_userdata, which invalidates the script's handle) or store a shared handle to its own storage (Arc<Mutex<_>>, with all that implies). Say which your API supports; ambiguity here becomes aliasing bugs.
• Userdata is a capability. A host object exposing db.execute(sql) grants the script database access, whatever the standard library does. The capability audit in the threat model must enumerate host objects, not just globals.
• Dropping order at engine shutdown. When the Engine is dropped, every userdata's Drop runs. If one of them re-enters the engine — a Drop impl that calls a script callback — it re-enters a half-destroyed runtime. Ember forbids it structurally: Drop for userdata gets no Ctx. State it, because someone will want a finalizer, and that is the same argument that excluded __gc.
• trace bugs are collector bugs. A host object holding an Ember Value without tracing it produces stale handles under memory pressure — in host code, which is the worst place to debug it. Make trace prominent in the docs and test host objects under --gc-stress.

9. References

• Lua 5.4 Reference Manual §2.1 (userdata) and §4.1.5 on light versus full userdata.
• mlua's UserData trait and UserDataMethods — the closest analogue, including its handling of &self versus &mut self methods, which is a real design question Ember simplifies by making fields read-only by default.
• V8's ObjectTemplate and internal fields, for the same idea at scale.

Concept 2: Copy versus Borrow

1–3. Concept, problem, mental model

Copy the data if it is small, short-lived, and mostly read in full. Expose it as userdata if it is large, long-lived, or read selectively.

The capstone makes this concrete. A request ranks 10,000 candidate articles; the policy reads two or three fields from each.

The third row is the answer for the capstone, and it is not obvious until you count. Marshaling is O(fields exposed); userdata is O(fields read). When the policy reads 3 of 12 fields, that is a 4× difference in allocation before any ranking happens.

4. Implementation

#![allow(unused)]
fn main() {
// The capstone's shape: the candidate SET is one userdata; indexing it yields
// per-article userdata that borrow (by index) into the same host storage.
struct Candidates { articles: Vec<Article> }

impl UserData for Candidates {
    fn build(b: &mut UserDataBuilder<Self>) {
        b.meta_len(|c| c.articles.len() as i64);
        b.meta_index_int(|c, i, ctx| {          // candidates[i]
            let idx = (i - 1) as usize;
            if idx >= c.articles.len() { return Ok(Value::Nil); }
            // An ArticleRef is (candidates_handle, index) — no copy, no lifetime.
            ctx.create_userdata(ArticleRef { owner: ctx.self_handle(), index: idx })
        });
    }
}
}

ArticleRef is an index, not a pointer, and that is the whole trick. It is the same move as GcRef one layer up: a handle into storage the engine can validate, instead of an address it cannot.

5–8. Alternatives, decision, production concerns

Decision: A for individual objects, C for collections, with the host-side rule written down: the backing storage must not be mutated while a script holds indices into it. In the capstone, the candidate list is built before evaluation and dropped after, so the rule is structurally satisfied — which is the right way to satisfy a rule.

Production concerns:

• Validate the index every time. articles[idx] with an unchecked idx from a script is an out-of-bounds read. Rust will panic rather than corrupt, which is better and still a host crash. Return nil or an error.
• A userdata handle can outlive the request. A script that stashes candidates in a global keeps it alive across evaluations. If the host expected the data to be dropped, it is now leaked — and worse, the next request's script can read the previous request's candidates. Swap the globals table per evaluation (ADR-012), and the capstone does exactly that.
• Field access cost. A native call per field is more expensive than a table probe. Measure it in the capstone before optimizing; if it dominates, an inline cache on the userdata __index site is the fix, and it is the same mechanism as for tables.

Concept 3: Capability Boundaries

1–3. Concept, problem, mental model

Every method on a host object is a capability grant. The standard library is not the attack surface; the union of the standard library and every registered function and every userdata method is.

-- The standard library grants nothing dangerous. This still deletes the database.
db:execute("DROP TABLE articles")

4. Implementation

The audit that makes this real, and it belongs in the test suite rather than in a document:

#![allow(unused)]
fn main() {
#[test]
fn the_full_capability_surface_is_enumerated_and_reviewed() {
    let engine = build_production_engine();
    let mut surface: Vec<String> = engine.enumerate_globals();
    for ud in engine.registered_userdata_types() {
        surface.extend(ud.methods().map(|m| format!("{}:{}", ud.name(), m)));
    }
    surface.sort();
    // A golden file, reviewed like any other diff. Adding a capability CHANGES
    // this file, which makes the grant visible in code review instead of
    // invisible in a builder call three modules away.
    assert_eq!(surface, read_golden("tests/golden/capability-surface.txt"));
}
}

That test is the single most valuable security control in Section 5. Not because it prevents anything — it prevents nothing — but because it makes every capability grant appear in a diff.

5–8. Alternatives, decision, production concerns

Decision: A, and say clearly that A is not D. Ember's capability model defends against scripts doing what you did not grant. It does not defend against bugs in the runtime itself. A host that needs the second one needs a process boundary, and the threat model says so in those words.

Production concerns:

• Transitive reachability. A registered function returning a table the host also holds gives the script write access to it. Return copies, or return userdata with read-only accessors.
• Capabilities are per-engine, so per-tenant means per-engine. One engine with a tenant id in a global is not isolation; it is a bug waiting for a script that changes the global.
• The debug library does not exist in Ember, and that is a capability decision. Lua's debug can read locals, upvalues, and other coroutines' stacks — it defeats every sandbox built on metatables. Note it as a divergence with the reason.

9. References

• Lua's lua_setuservalue/lua_getuservalue, and the luaL_newmetatable registry pattern.
• The lua-users wiki sandboxing page — read it, then read the many follow-ups explaining why each version was escapable. It is the best available argument for option A over C.
• Capability-based security literature: the E language, Capsicum, and the general "no ambient authority" principle that this chapter is an instance of.

Things to Notice

• Userdata is a handle into engine-owned storage, and ArticleRef is a handle into host-owned storage. The same pattern at two layers, for the same reason: an index can be validated, an address cannot.
• Metatables are per type, not per object. Getting this wrong reintroduces the allocation cost userdata exists to remove.
• Marshaling is O(fields exposed); userdata is O(fields read). Count before choosing.
• trace is where a host can break the collector. Make it prominent.
• Every host method is a capability, and the capability surface is the union — not the standard library.
• The enumerated-surface golden test prevents nothing and is the best control in the section, because it moves grants into code review.
• Runtime capability control is not process isolation. Say which one you have.

Validation / Self-check

1. Give the four ways to expose a Rust struct to a script and the specific problem with each of the three Ember rejects.
2. Why is the metatable per type rather than per object? What does the alternative cost?
3. What does TypeId buy, and why does UserData: 'static follow?
4. When should a host marshal into a table instead of using userdata? State the rule as a complexity comparison.
5. Why is ArticleRef an index rather than a reference? Where have you seen that move before?
6. What must be true of the host's backing storage while a script holds indices into it, and how does the capstone satisfy it structurally?
7. Why does userdata Drop get no Ctx? Which earlier decision is that the same as?
8. What is the full capability surface, and why is a golden test of it valuable despite preventing nothing?
9. Why is Lua's light userdata absent from Ember?

Next: The Standard Library.