Value Marshaling

Three concepts: the conversion traits, conversion failure, and rooting.

Marshaling is the code every value crosses on its way in or out. It is unglamorous, it is where panics hide, and its error messages are what a host developer sees when their integration is wrong — which makes it worth more care than it usually gets.

Concept 1: ToValue and FromValue

1. Concept

Two traits. ToValue converts a Rust value into an Ember Value (which may allocate). FromValue converts an Ember Value into a Rust type (which may fail).

#![allow(unused)]
fn main() {
pub trait ToValue   { fn to_value(self, ctx: &mut Ctx<'_>) -> Result<Value>; }
pub trait FromValue: Sized { fn from_value(v: Value, ctx: &Ctx<'_>) -> Result<Self>; }
}

2. Problem

Without them, every host writes conversion code by hand at every call site, gets the integer/float distinction wrong in half of them, and panics on the rest.

3. Mental model

The asymmetry is the whole design. Going in is total and may allocate. Coming out is partial and must be fallible. Every Rust i64 is a valid Ember integer; not every Ember Value is a valid Rust i64.

4. Implementation

#![allow(unused)]
fn main() {
impl ToValue for i64  { fn to_value(self, _: &mut Ctx) -> Result<Value> { Ok(Value::Integer(self)) } }
impl ToValue for f64  { fn to_value(self, _: &mut Ctx) -> Result<Value> { Ok(Value::Float(self)) } }
impl ToValue for bool { fn to_value(self, _: &mut Ctx) -> Result<Value> { Ok(Value::Boolean(self)) } }
impl ToValue for &str {
    fn to_value(self, ctx: &mut Ctx) -> Result<Value> {
        Ok(Value::Str(ctx.intern(self.as_bytes())?))    // ALLOCATES — hence &mut Ctx
    }
}
impl<T: ToValue> ToValue for Option<T> {
    fn to_value(self, ctx: &mut Ctx) -> Result<Value> {
        match self { Some(v) => v.to_value(ctx), None => Ok(Value::Nil) }
    }
}
impl<T: ToValue> ToValue for Vec<T> {
    fn to_value(self, ctx: &mut Ctx) -> Result<Value> {
        let t = ctx.new_table()?;
        let _root = ctx.root(t);          // ← the table must be ROOTED: each element
        for (i, v) in self.into_iter().enumerate() {   //   conversion may allocate
            let v = v.to_value(ctx)?;
            ctx.table_set(t, Value::Integer(i as i64 + 1), v)?;
        }
        Ok(t)
    }
}
}

Coming out, every implementation names both types and the position:

#![allow(unused)]
fn main() {
impl FromValue for i64 {
    fn from_value(v: Value, ctx: &Ctx) -> Result<Self> {
        match v {
            Value::Integer(i) => Ok(i),
            // Lua 5.4 §3.4.3: a float converts to an integer only if it has an
            // exact integer value. 3.0 → 3; 3.5 → error, NOT truncation.
            Value::Float(f) if f.floor() == f && f >= -(2f64.powi(63)) && f < 2f64.powi(63)
                => Ok(f as i64),
            Value::Float(_) => Err(ctx.type_error("integer", "number with a fractional part")),
            other => Err(ctx.type_error("integer", other.type_name())),
        }
    }
}
}

And the argument-position wrapper, which is what makes the error message useful:

#![allow(unused)]
fn main() {
// "bad argument #2 to 'score' (integer expected, got string)"
// That is Lua's format, it names the function, the position, and both types,
// and it is the difference between a five-second fix and a bisect.
pub fn arg<T: FromValue>(ctx: &Ctx, args: &[Value], i: usize, fname: &str) -> Result<T> {
    let v = args.get(i).copied().unwrap_or(Value::Nil);
    T::from_value(v, ctx).map_err(|e| ctx.error(format!(
        "bad argument #{} to '{}' ({})", i + 1, fname, e.message)))
}
}

5. Alternatives

6. Decision

A, with B behind an optional serde feature.

serde is genuinely convenient for a policy engine — #[derive(Serialize)] struct Article and it crosses the boundary — and it is genuinely a large dependency with its own compile-time cost. Behind a feature flag both audiences are served, and the core crate keeps its empty [dependencies].

7. Tradeoffs

8. Production concerns

• Never panic. Not on non-UTF-8 bytes, not on an out-of-range integer, not on a missing field. The fuzzer in Section 6 targets exactly this module because it is where expect habitually survives review.
• The integer/float rule must match the language. 3.0 converts to 3; 3.5 does not. If marshaling truncates while the language errors, a host and a script disagree about the same value — and that disagreement will be found in production by a number that is off by one.
• Non-UTF-8 strings. Ember strings are bytes (ADR: byte strings), so FromValue for String must return an error, not a lossy conversion. Provide Vec<u8> for hosts that want the raw bytes.
• Conversion cost is not free. Marshaling a 10,000-element Vec<Article> into a table allocates 10,000 tables. For the capstone, that is exactly the wrong design — pass the candidates as userdata and let the script pull fields on demand. The next chapter is about that, and this row is why.

9. References

• mlua's IntoLua/FromLua and its Variadic/MultiValue types — option A in production.
• serde's data model, and specifically where it does not fit a dynamic language (tables that are sometimes arrays and sometimes maps).
• Lua's luaL_check* family (luaL_checkinteger, luaL_argerror) — the error-message format Ember copies, and worth reading for how much care went into the wording.

Concept 2: Conversion Failure

1–3. Concept, problem, mental model

A conversion failure is an ordinary, expected event — the script passed a string where the host wanted a number — and it must produce an error a human can act on, at the boundary, before any host logic runs on a wrong value.

4. Implementation

#![allow(unused)]
fn main() {
#[derive(Debug)]
pub enum ErrorKind { Lex, Parse, Compile, Runtime, Limit, Host, Marshal }
//                                                          ^^^^^^^ new in §5
}

Marshal is separate from Host and from Runtime on purpose, and the distinction is operational:

Four kinds because four different people need to be told. That is the test for whether an error taxonomy is real: if two kinds always route to the same place, merge them.

8. Production concerns

• Failure must happen before side effects. Convert all arguments first, then run the host logic. A function that converts argument 1, writes to a database, and then fails converting argument 2 is a bug the marshaling layer can prevent structurally.
• Do not echo unbounded script data into an error. got string is safe; got string "…3 MB…" is a log-flooding bug. Truncate, always.
• Round-tripping is not guaranteed and should not be implied. f64 → Ember → f64 is exact; Vec<T> → table → Vec<T> loses nothing only if the script did not touch it. Document what round-trips.

Concept 3: Rooting

1. Concept

A Value held in a Rust local is not a GC root. If the host allocates again while holding one, the collector may free it.

2. Problem

#![allow(unused)]
fn main() {
// WRONG. new_table may collect, and `a` is reachable from nothing.
let a = ctx.new_table()?;
let b = ctx.new_table()?;          // ← a collection here frees `a`
ctx.table_set(b, key, a)?;
}

This is the allocation hazard crossing the boundary. Inside the VM it is mostly avoided by accident, because intermediates live on the value stack, which is a root set. Host code has no such stack, which is precisely why Lua's C API makes you use one.

3. Mental model

A RootGuard is a temporary root with a lexical lifetime. While it exists, the collector knows about the value. When it drops, the value is on its own again.

4. Implementation

#![allow(unused)]
fn main() {
pub struct RootGuard<'c> { ctx: *mut TempRoots, index: usize, _p: PhantomData<&'c ()> }
impl Drop for RootGuard<'_> { fn drop(&mut self) { /* pop the temp-root stack */ } }

impl Ctx<'_> {
    #[must_use = "a RootGuard that is immediately dropped roots nothing"]
    pub fn root(&mut self, v: Value) -> RootGuard<'_> { /* push onto vm.temp_roots */ }
}
}

#[must_use] with that exact message is doing real work: ctx.root(v); with a semicolon compiles, drops the guard immediately, and roots nothing. The lint catches the most likely misuse.

5–7. Alternatives, decision, tradeoffs

Decision: A. B is stronger — it makes the mistake inexpressible — at the cost of an API where nothing returns a value directly. That tradeoff is defensible and it is why the C API looks alien; Ember chooses ergonomics plus a lint plus --gc-stress in the test suite.

8. Production concerns

• --gc-stress is the test for this. Every example and every host-function test must pass under it. A missing root is otherwise invisible until the heap is large.
• Host handles are root set 7. Engine::set_global stores into the globals table (already a root), but a value the host stashes in its own struct across calls needs a persistent registry — Lua's LUA_REGISTRYINDEX, mlua's RegistryKey. Ember exposes Engine::stash/unstash for this, and those entries are traced.
• Do not root more than you must. A registry that only grows is a leak with no cycle in it, and the collector will never help you find it. unstash must actually remove.

9. References

• Lua's LUA_REGISTRYINDEX and luaL_ref/luaL_unref — the canonical persistent-root API.
• mlua's RegistryKey and its explicit remove_registry_value, with the documentation's warning about leaks.
• V8's Local/Persistent/HandleScope — the same problem, and the reason V8 embedders write HandleScope scope(isolate); at the top of every function.

Things to Notice

• In is total, out is fallible. Every asymmetry in the traits follows from that.
• Four error kinds because four different people need to be told. If two always route the same way, they are one kind.
• Convert everything before doing anything. Structural prevention of half-applied side effects.
• A Rust local is not a GC root. V8 embedders learn this via HandleScope; Lua embedders learn it via the stack API; Ember embedders learn it via #[must_use] and --gc-stress.
• Marshaling a big collection is the wrong design. That cost is what motivates userdata, and the capstone depends on getting it right.
• serde behind a feature keeps the core dependency-free without punishing the hosts that want it.

Validation / Self-check

1. Why is ToValue total and FromValue fallible? Give a value that shows it.
2. Why does ToValue for &str need &mut Ctx when ToValue for i64 does not?
3. What are the four error kinds at the boundary, and who does each one blame?
4. What does arg::<T> add over T::from_value, and why does it matter?
5. Give Lua's integer-conversion rule for floats and say what a truncating implementation would break.
6. Why must all arguments be converted before any host logic runs?
7. What is a RootGuard, and what does #[must_use] prevent? Write the mistake it catches.
8. Why is marshaling a 10,000-element Vec the wrong design for the capstone?
9. Why does unstash have to actually remove the entry?

Next: Host Objects and Userdata.