The Rust Crate Design

This chapter defines the shape of the thing you are building: one crate, a module tree with enforced dependency rules, and the tooling that keeps it honest. Set it up now — the labs assume these paths, these command names, and these rules.

1. One Crate, Not a Workspace

ADR-001 — Single crate. Ember is one crate with modules, not a Cargo workspace. Split only when a real boundary demands it.

The temptation is to open with crates/ember-lexer, crates/ember-parser, crates/ember-vm, crates/ember-gc. Resist it. A workspace on day one buys you:

• Longer compile times (more crate boundaries, more link steps).
• A version-bumping ritual for every internal change.
• Public APIs where you wanted private ones — pub(crate) stops working across crates, so everything internal becomes pub, and now you are maintaining an accidental API surface.
• The appearance of architecture, with none of its substance.

What actually enforces architecture is a dependency rule you check, and you can check one inside a single crate. Section 5 revisits this: if the core turns out to be genuinely reusable — for wasm32, for no_std, for a separate fuzz target — that is a reason to split, and by then you will know exactly where the seam is. Splitting a crate you understand is an afternoon. Un-splitting one you designed by guesswork is a week.

The companion workspace at book/projects/ember/ implements exactly this layout.

2. The Module Tree

ember/
├── Cargo.toml
├── src/
│   ├── lib.rs           the public API surface and the crate docs
│   ├── span.rs          Span, SourceId, SourceMap, line/column resolution
│   ├── error.rs         EmberError, ErrorKind, Diagnostic rendering
│   │
│   ├── token.rs         TokenKind, Token { kind, span }
│   ├── lexer.rs         &str → Vec<Token>                              §1
│   ├── ast.rs           Expr, Stmt, Block — every node carries a Span  §1
│   ├── parser.rs        tokens → Ast: recursive descent + Pratt        §1
│   │
│   ├── value.rs         Value, the tagged union; truthiness; equality  §2
│   ├── interp/          THE REFERENCE TREE-WALKING INTERPRETER         §2
│   │   ├── mod.rs
│   │   ├── env.rs       scopes (hash maps — deliberately naive)
│   │   └── eval.rs
│   │
│   ├── bytecode.rs      Op, Chunk, constant pool, line table, disasm   §3
│   ├── compiler.rs      AST → Chunk: scopes, slots, jumps, upvalues    §3
│   ├── vm.rs            the dispatch loop, frames, the value stack     §3
│   │
│   ├── heap.rs          slot table, GcRef<T>, mark & sweep, accounting §4
│   ├── table.rs         array part + insertion-ordered hash part       §4
│   ├── closure.rs       Proto, Closure, Upvalue { Open | Closed }      §4
│   ├── strings.rs       EmberStr, interning, hashing                   §4
│   ├── meta.rs          metatables, metamethod dispatch                §4
│   │
│   ├── engine.rs        Engine — the embedding API                     §5
│   ├── userdata.rs      host objects, the UserData trait               §5
│   ├── module.rs        require(), the resolver trait, the cache       §5
│   ├── limits.rs        instruction / memory / depth budgets           §5
│   ├── stdlib/                                                          §5
│   │   ├── mod.rs       Capabilities: what a script is allowed
│   │   ├── base.rs      print, type, assert, error, pcall, tostring…
│   │   ├── math.rs
│   │   ├── string.rs
│   │   └── table.rs
│   │
│   ├── stats.rs         counters: instructions, allocs, GC runs, depth §6
│   └── bin/
│       └── ember.rs     run | repl | tokens | ast | disassemble | trace
│
├── tests/
│   ├── golden/          *.ember + *.expected — run by BOTH backends
│   ├── differential.rs  interp vs VM on every golden case
│   ├── lexer.rs  parser.rs  compiler.rs  vm.rs  gc.rs  engine.rs
│   └── property.rs      proptest: parser round-trip, table invariants
├── benches/
│   ├── dispatch.rs  tables.rs  calls.rs  strings.rs  gc.rs
├── fuzz/fuzz_targets/
│   ├── lex.rs  parse.rs  compile.rs  run.rs  bytecode_validate.rs
├── examples/
│   ├── hello.rs  embedding.rs  host_objects.rs  recommendation_policy.rs
├── scripts/
│   └── boundary-audit.sh
└── docs/
    ├── architecture.md  bytecode.md  gc.md  embedding.md  sandboxing.md
    ├── learning/        your journal — one file per subsystem
    └── adr/             your decision records

3. The Dependency Rule (Enforced, Not Aspirational)

   span ─── error
     │        │
     ▼        ▼
   token ─▶ lexer ─▶ ast ─▶ parser
                       │       │
              ┌────────┴───────┴──────────┐
              ▼                            ▼
           interp                      compiler ─▶ bytecode
              │                            │            │
              └──────────┬─────────────────┘            ▼
                         ▼                             vm
                       value ◀── heap ◀── table, closure, strings, meta
                                   ▲
                                   │
                    engine ─▶ userdata, module, limits, stdlib, stats

Three rules, and a script that checks them:

1. lexer, parser, ast, and bytecode must never import vm, heap, engine, or stdlib. The front end and the instruction encoding know nothing about execution. This is what makes the disassembler usable as a standalone tool, the parser fuzzable in isolation, and the whole front end compilable to wasm32 without a runtime.
2. value and heap must never import engine. The object model does not know it is embedded.
3. The graph is acyclic. Rust permits module cycles; they are still a design smell, and they make "which layer is this bug in?" unanswerable.

# scripts/boundary-audit.sh — run it in CI
set -euo pipefail
fail=0
check() {  # check <module> <forbidden-regex>
  if rg -n "use crate::($2)" "src/$1" >/dev/null 2>&1; then
    echo "BOUNDARY VIOLATION: src/$1 imports $2"; rg -n "use crate::($2)" "src/$1"; fail=1
  fi
}
check lexer.r*    'vm|heap|engine|stdlib|value'
check parser.rs   'vm|heap|engine|stdlib'
check ast.rs      'vm|heap|engine|stdlib'
check bytecode.rs 'vm|heap|engine|stdlib'
check value.rs    'engine|stdlib|module'
check heap.rs     'engine|stdlib|module'
exit $fail

Tip: Add this on day one, when it trivially passes. A boundary check added after the violation exists is a refactor; added before, it is a guardrail. This is the same reason you write the Span field in Lab 1.

4. Cargo.toml

[package]
name = "ember"
version = "0.1.0"
edition = "2021"
rust-version = "1.75"
description = "A small, embeddable, Lua-like scripting runtime"
license = "MIT OR Apache-2.0"
categories = ["compilers", "development-tools"]

[dependencies]
# Deliberately empty for Sections 1–4. Every dependency you add is a dependency
# a host inherits, an audit surface, and a thing you did not learn to build.

[dev-dependencies]
criterion = "0.5"
proptest  = "1"

[features]
default = []
serde   = []      # §5: marshaling host data via serde
jit     = []      # §7: pulls in cranelift; OFF by default, and that is a security posture
std-io  = []      # §5: opt-in capability. NOT default. See the threat model.

[[bench]]
name = "dispatch"
harness = false

[profile.release]
debug = true      # keep symbols: you will profile this, and a stripped flamegraph is useless

Why the empty [dependencies]. Every crate you pull in is code your host's security team inherits. For a runtime that advertises sandboxing, "what does it link against" is a question with a real answer, and none is the strongest one. You will add exactly three things across the whole curriculum, each with a written justification in an ADR: cranelift-* (behind jit, off by default), optionally serde (behind serde), and rustc-hash or similar only if Section 7's benchmarks show the default hasher is a real cost — and if you add it, you must also state what happens to hash-collision resistance, because that is a denial-of-service surface. See the threat model.

5. The Binary: Six Subcommands, One per Representation

ember run script.ember          # execute (VM backend, the default)
ember run --interp script.ember # execute with the tree walker — same output, always
ember repl                      # interactive
ember tokens script.ember       # LEXER output
ember ast script.ember          # PARSER output
ember disassemble script.ember  # COMPILER output
ember trace script.ember        # VM output: one line per instruction

This is not a nice-to-have. Each subcommand prints exactly one representation from Claim 1 of the mental model. When something is wrong, you bisect by representation rather than by guessing, and each subcommand is thirty lines because the data structure already exists.

Useful global flags, added as the relevant lab arrives:

--interp                 use the tree-walking backend (differential debugging)
--trace-gc               log every collection: before/after bytes, objects freed, duration
--stats                  print the stats block on exit
--max-instructions N     instruction budget
--max-memory BYTES       heap budget
--max-depth N            call-depth budget
--no-color               for golden tests and CI logs

6. Testing Layout

The golden corpus is shared by both backends from Lab 12 onward, which means every test you ever write becomes a differential test for free. That is the payoff for keeping the tree walker, and it is why ADR-003 exists.

7. CI

# .github/workflows/ci.yml
name: ci
on: [push, pull_request]
jobs:
  check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: dtolnay/rust-toolchain@stable
        with: { components: rustfmt, clippy }
      - run: cargo fmt --check
      - run: cargo clippy --workspace --all-targets --all-features -- -D warnings
      - run: cargo test  --workspace --all-features
      - run: cargo doc   --workspace --no-deps
      - run: ./scripts/boundary-audit.sh

-D warnings from day one. A clippy warning you have decided to keep gets an #[allow] with a comment saying why — that comment is worth more than the lint.

cargo audit and cargo deny check join in Section 6, alongside fuzzing. Adding them earlier is noise; adding them never is negligence.

8. Bootstrap It Now

cargo new --lib ember && cd ember
mkdir -p src/{interp,stdlib,bin} tests/golden benches fuzz examples scripts \
         docs/{learning,adr}
git init && git add -A && git commit -m "chore: scaffold"

# a habit worth forming: one commit per lab, with the lab number in the subject
git commit --allow-empty -m "lab-00: workspace scaffolded"

Then create the two files that exist before any language feature does — because Claim 11 says you cannot retrofit them:

#![allow(unused)]
fn main() {
// src/span.rs
/// A half-open byte range into one source file. 8 bytes, `Copy`.
/// Byte offsets, not line/column: line/column is a *presentation* concern and is
/// computed on demand from the SourceMap. Storing lines here would make every
/// node bigger and would be wrong the moment a source is edited in the REPL.
#[derive(Copy, Clone, PartialEq, Eq, Debug)]
pub struct Span { pub start: u32, pub end: u32 }

impl Span {
    pub const EMPTY: Span = Span { start: 0, end: 0 };
    /// The smallest span covering both. Used to give a binary expression the
    /// span of its whole subtree, so `a + b * c` underlines all of it.
    pub fn merge(self, other: Span) -> Span {
        Span { start: self.start.min(other.start), end: self.end.max(other.end) }
    }
}
}

#![allow(unused)]
fn main() {
// src/error.rs
#[derive(Debug)]
pub struct EmberError {
    pub kind: ErrorKind,
    pub message: String,
    pub span: Option<Span>,
    pub traceback: Vec<Frame>,   // empty until §3; the field exists now
}

#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum ErrorKind {
    Lex, Parse, Compile,      // compile time — the source is available
    Runtime,                  // run time — a traceback is available
    Limit,                    // a budget was exceeded: instructions, memory, depth
    Host,                     // a registered Rust function returned an error
}

pub type Result<T> = std::result::Result<T, EmberError>;
}

Note: ErrorKind::Limit is separate from Runtime on purpose, and it is the kind of distinction that is free now and expensive later. A host needs to tell "the script had a bug" apart from "the script hit the wall we put up" — the first is a page for the script author, the second is a metric and possibly a bigger budget. Section 5 depends on this split; declaring it in Lab 0 costs one line.

Deliverables

• ember crate created, module directories in place, first commit made.
• src/span.rs and src/error.rs written and compiling.
• scripts/boundary-audit.sh written, executable, and passing (trivially).
• CI workflow committed; cargo fmt --check and cargo clippy -- -D warnings both green.
• docs/adr/ADR-001-single-crate.md written — Context, Options, Decision, Consequences.
• docs/learning/ and docs/adr/ exist with a README.md each explaining what goes in them.

Validation / Self-check

1. Give three concrete costs of starting with a Cargo workspace, and the one condition that would justify splitting later.
2. State the three dependency rules and name a capability each one buys.
3. Why does Span store byte offsets rather than line and column?
4. Why is ErrorKind::Limit distinct from ErrorKind::Runtime? Give a host behavior that depends on the distinction.
5. Why is [dependencies] empty, and what must accompany every future addition?
6. Which subcommand of ember corresponds to which representation from Claim 1?
7. Why is debug = true set in the release profile?

Next: The Roadmap: Fifteen Milestones.