JIT Architecture

The last layer, and the one where the interesting content is not the code generation. Anyone can emit an add instruction. The engineering is in the guards, the deoptimization, and the collector's cooperation.

Read this chapter before Lab 30. The lab compiles a two-line function; this chapter is why that is enough to understand the architecture.

The Pipeline

   interpreter
        │  counts executions per function (or per loop back-edge)
        ▼
   hot?  ─────no────▶ keep interpreting
        │ yes
        ▼
   collect the TYPE FEEDBACK the interpreter already gathered  (§7's earlier chapters)
        │
        ▼
   build an IR, ASSUMING the feedback holds
        │
        ▼
   optimize (Cranelift does this)
        │
        ▼
   machine code + GUARDS + a STACK MAP
        │
        ├── guard holds ──▶ run at native speed
        └── guard FAILS  ──▶ DEOPTIMIZE: reconstruct interpreter state, resume interpreting

The dashed arrow back is the feature. Without it you cannot speculate; without speculation you cannot be fast; and reconstructing an interpreter frame from an optimized machine frame is most of the work in every production JIT.

Method JIT versus Trace JIT

The fork, and both answers ship:

Ember builds a method JIT, because a trace recorder is a second interpreter with recording instrumentation and that is a much larger project than one section can carry. Note that LuaJIT — the fastest thing in this family — is a trace compiler, and that its speed comes substantially from specializing an entire path at once, which is why it needs no inline caches.

Guards

A guard is a cheap runtime check protecting an assumption the compiler made.

   function add(a, b) return a + b end

   feedback says: both operands have been Integer, always.

   compiled:
       guard  tag(a) == Integer   ──fail──▶ deoptimize
       guard  tag(b) == Integer   ──fail──▶ deoptimize
       result = a.int + b.int          (one machine instruction)
       return result

Two compares and an add, versus the interpreter's fetch, dispatch, five-way match, and stack manipulation. That is where the win comes from — and note that the win is not "compiled code is fast", it is "we deleted the checks that the profile said were unnecessary, and kept one cheap one".

What needs a guard, in general:

Every assumption is a guard, and every guard is a place you might deoptimize. A compiled function with twenty guards is not obviously better than the interpreter; part of the art is choosing assumptions that are both likely and cheap to check.

Deoptimization

The hard part, invented for Self (Hölzle, Chambers & Ungar, 1992).

   OPTIMIZED FRAME                      INTERPRETER FRAME
   ┌────────────────────┐               ┌──────────────────────┐
   │ a in register rdi  │  ──deopt──▶   │ stack[base + 0] = a  │
   │ b in register rsi  │               │ stack[base + 1] = b  │
   │ temp in xmm0       │               │ stack[base + 2] = …  │
   │ (no frame at all,  │               │ ip = 0007            │
   │  it was inlined)   │               │ frames: 2 frames!    │
   └────────────────────┘               └──────────────────────┘

   The compiler must record HOW to do this, for EVERY guard site.
   That record is a DEOPTIMIZATION MAP.

Three things make it hard:

1. Values live in registers, not stack slots. The map says "at guard 3, a is in rdi, b is in xmm0, and the value that would be at stack slot 2 was constant-folded to 7".
2. Inlining means one optimized frame may become several interpreter frames. Deoptimizing an inlined call reconstructs the whole chain.
3. It must be correct at every guard, including guards inside optimized code the compiler moved or duplicated.

Ember's Lab 30 avoids most of this by compiling only a leaf function with no inlining, so the map is "copy these two registers to these two stack slots and set ip to 0". That is the honest scope, and it is enough to see the shape.

The Collector Must Cooperate

A JIT and a GC cannot ignore each other, and this is the part most explanations skip.

The collector needs to find every root. When a value lives in a machine register inside compiled code, the collector cannot see it. Two things are required:

Safepoints. Compiled code may only be interrupted for a collection at points the compiler chose, where it has recorded what is live. A loop with no safepoint can prevent a collection indefinitely — a real class of bug in production JITs.

Stack maps. At every safepoint, a map says which registers and stack slots hold GC references. The collector consults it to find roots inside compiled frames.

   safepoint @ 0x7f...a3
     rdi     → GcRef<Table>
     rsi     → not a reference (an integer)
     [rsp+8] → GcRef<EmberStr>

And if the collector moved objects, it would have to rewrite those registers. Ember's collector is non-moving, which removes that entire problem — a simplification that the GC chapter predicted would matter later, and this is later.

Cranelift supports stack maps for exactly this reason; Lab 30's subset avoids needing them by not allocating in compiled code, which is a legitimate scope decision as long as it is stated.

Why Cranelift

ADR-014: use Cranelift, not hand-emitted machine code.

Register allocation and instruction encoding are not the lesson. They are a well-solved problem with an excellent Rust implementation, and spending three weeks on them means never reaching deoptimization — which is the lesson.

Option D deserves a look: copy-and-patch generates machine-code templates at build time and stitches them at run time, giving most of a JIT's win with a fraction of the complexity. It is what CPython 3.13's experimental JIT does, and it is a strong candidate if you ever want a JIT without a code generator.

The Security Posture

The JIT is off by default, and that is a security decision, not a packaging one:

• It is the only part of Ember with unsafe — executing generated code requires it.
• It requires W^X-violating memory at some point (write, then make executable), which some hardened environments forbid outright.
• It enlarges the trusted computing base by a large dependency.

#![allow(unused)]
#![cfg_attr(not(feature = "jit"), forbid(unsafe_code))]
fn main() {
}

That line is the claim, and it is checkable. Every unsafe block behind the feature needs the five-part treatment from the hardening checklist.

The Honest Expectation

Ember's JIT will beat Ember's VM on a microbenchmark and will not approach LuaJIT. Say so, and say why:

• LuaJIT is a trace compiler with hand-written assembly interpreters per architecture, NaN boxing, allocation sinking, and a decade of tuning by one exceptional engineer.
• Ember's is a method JIT compiling one function shape, with no inlining and no allocation.

The objective is that you can draw the pipeline, explain every arrow, and read lj_record.c knowing what you are looking at. Beating LuaJIT was never the goal and pretending otherwise would be the one thing this curriculum has consistently refused to do.

Things to Notice

• The arrow back — deoptimization — is the feature. Everything else is code generation.
• A guard is an assumption made cheap to check. Choosing assumptions that are likely and cheap is the art.
• Method versus trace is a real fork, and the fastest implementation in this family took the other branch.
• A JIT and a GC must cooperate: safepoints and stack maps. A non-moving collector removes half the problem, which is a simplification chosen four sections earlier paying off here.
• Cranelift because register allocation is not the lesson. Naming what you are not learning is part of scoping.
• The JIT is the only unsafe in Ember, and it is off by default. A security posture, stated.
• Copy-and-patch is the underrated option, and CPython 3.13 picked it.

References

# Cranelift, in the wasmtime repository:
rg -n 'fn define_function|StackMap|safepoint' cranelift/codegen/src/
# LuaJIT's trace recorder — read after the lab, and read the comments:
rg -n 'lj_record_ins|rec_loop|snap_' src/lj_record.c src/lj_snap.c

• Hölzle, Chambers & Ungar, Debugging Optimized Code with Dynamic Deoptimization (PLDI 1992) — the paper that invented deoptimization. Short, and the single most important reference here.
• Gal, Probst & Franz, HotpathVM (VEE 2006) — trace compilation, the other branch.
• Hölzle, Chambers & Ungar, Optimizing Dynamically-Typed Object-Oriented Languages With Polymorphic Inline Caches (ECOOP 1991) — where the feedback comes from.
• Cranelift's documentation and the cranelift-jit examples in wasmtime.
• LuaJIT's lj_record.c, lj_snap.c (snapshots — LuaJIT's deopt maps), and Mike Pall's mailing-list explanations, which are more informative than most published papers.
• CPython 3.13's copy-and-patch JIT, and the Xu & Kjolstad paper it is based on.

Validation / Self-check

1. Draw the JIT pipeline from memory. Which arrow is the feature, and why?
2. What is a guard? Give four assumptions a JIT makes and the guard for each.
3. What is deoptimization, and name the three things that make it hard?
4. Give the method-versus-trace comparison, and say which LuaJIT chose and what that buys it.
5. What are safepoints and stack maps for? Which of Ember's earlier decisions removes half the problem?
6. State ADR-014's decision and the thing it deliberately declines to teach.
7. Why is the JIT off by default? Give three reasons, only one of which is packaging.
8. What is copy-and-patch, and who uses it?

Next: Lab 28 — Inline Caches.