Project 3: A GPU Renderer

2–3 weeks · ●●●○○ · touches wgpu, glyph atlases, instanced drawing, the render-model boundary

1. The Problem

Your CPU renderer is fine at 80×24 and unusable at 400×100. A full redraw blends millions of pixels on one core, and a cat of a large file drops frames.

   Measured on a typical laptop, CPU renderer, full redraw:

     80×24    ~1,900 cells    0.3-0.8 ms    fine
    200×50   ~10,000 cells    2-4 ms        noticeable
    400×100  ~40,000 cells    8-16 ms       dropping frames

The prerequisite for this project is that measurement. If you have not taken it, do Lab 12's damage experiment first. A GPU renderer written without a number to justify it is cargo-culting.

2. Why It Is Hard

Not conceptually — the architecture is well known. The difficulty is that GPU bugs are opaque.

Warning: Do this after Lab 12, never instead of it. If you write the GPU renderer first, you will be debugging shader pipelines while you still do not know where the baseline is, and you will not be able to tell which layer is wrong.

3. The Design

The architecture every GPU terminal uses:

   ┌──────────────────────────────────────────────────────────────┐
   │  ONE TEXTURE: the glyph atlas (R8 coverage, or RGBA for      │
   │  color emoji). Uploaded lazily as glyphs are rasterized.      │
   └──────────────────────────────────────────────────────────────┘
                              ▲ sampled by
   ┌──────────────────────────────────────────────────────────────┐
   │  ONE INSTANCE BUFFER: one entry per visible cell              │
   │    { cell_x, cell_y, atlas_uv, size, fg_rgba, bg_rgba, flags }│
   └──────────────────────────────────────────────────────────────┘
                              │
   ┌──────────────────────────▼───────────────────────────────────┐
   │  VERTEX SHADER: expands each instance into a quad             │
   │  FRAGMENT SHADER: samples the atlas, mixes fg/bg by coverage  │
   └──────────────────────────────────────────────────────────────┘
                              │
                      ONE DRAW CALL for the whole screen

#![allow(unused)]
fn main() {
/// One instance per cell. Keep it small: this is uploaded every frame.
/// 32 bytes × 40,000 cells = 1.25 MB per frame, which is nothing for a GPU
/// and everything for a CPU memcpy — so make it tight.
#[repr(C)]
#[derive(Copy, Clone, bytemuck::Pod, bytemuck::Zeroable)]
struct CellInstance {
    /// Grid position, expanded to a quad in the vertex shader.
    grid_pos: [u16; 2],        // 4 bytes
    /// Atlas rect, in texels.
    atlas_pos: [u16; 2],       // 4
    atlas_size: [u16; 2],      // 4
    /// Glyph offset from the cell origin (xmin, and the baseline adjustment).
    glyph_offset: [i16; 2],    // 4
    fg: [u8; 4],               // 4
    bg: [u8; 4],               // 4
    flags: u32,                // 4  (underline style, strike, cursor, ...)
}                              // = 28, padded to 32
}

// The vertex shader expands one instance into one quad. No vertex buffer is
// needed: the four corners are derived from the vertex index.
struct Uniforms {
    screen_size: vec2<f32>,
    cell_size: vec2<f32>,
    atlas_size: vec2<f32>,
};
@group(0) @binding(0) var<uniform> u: Uniforms;
@group(0) @binding(1) var atlas: texture_2d<f32>;
@group(0) @binding(2) var atlas_sampler: sampler;

@vertex
fn vs_main(@builtin(vertex_index) vi: u32, inst: CellInstance) -> VertexOutput {
    // Corner selection from the vertex index: 0=TL 1=TR 2=BL 3=BR.
    let corner = vec2<f32>(f32(vi & 1u), f32((vi >> 1u) & 1u));

    let cell_origin = vec2<f32>(inst.grid_pos) * u.cell_size;
    let px = cell_origin + vec2<f32>(inst.glyph_offset) + corner * vec2<f32>(inst.atlas_size);

    // Pixels → normalized device coordinates. NOTE the Y flip: screen Y grows
    // DOWN, NDC Y grows UP. Forgetting this renders the screen upside down,
    // which is the single most common first-run symptom.
    var out: VertexOutput;
    out.clip_pos = vec4<f32>(
        (px.x / u.screen_size.x) * 2.0 - 1.0,
        1.0 - (px.y / u.screen_size.y) * 2.0,
        0.0, 1.0);
    out.uv = (vec2<f32>(inst.atlas_pos) + corner * vec2<f32>(inst.atlas_size)) / u.atlas_size;
    out.fg = unpack4x8unorm(inst.fg);
    out.bg = unpack4x8unorm(inst.bg);
    return out;
}

@fragment
fn fs_main(in: VertexOutput) -> @location(0) vec4<f32> {
    // The atlas stores COVERAGE in R8. Mix background and foreground by it.
    let coverage = textureSample(atlas, atlas_sampler, in.uv).r;
    return mix(in.bg, in.fg, coverage);
}

The two-pass question

Backgrounds and glyphs can be drawn in one pass or two:

Recommend two passes. Descenders, accents, and box-drawing characters routinely overflow their cell, and the one-pass artifact is visible and hard to diagnose.

4. Milestones

Milestone 2 is where most of the difficulty is. Once one quad is correct — right position, right orientation, right UVs — the rest is bookkeeping.

5. The Tests

GPU output is hard to unit-test. Test what is testable, and make the untestable part comparable.

#![allow(unused)]
fn main() {
#[test]
fn instance_data_matches_the_cpu_renderer() {
    // The highest-value test: build instances from a snapshot and assert they
    // describe exactly what the CPU renderer would draw. This catches every
    // coordinate and color bug WITHOUT a GPU.
    let snap = snapshot_of(b"\x1b[31mred\x1b[0m normal");
    let instances = build_instances(&snap, &metrics, &atlas);
    assert_eq!(instances.len(), snap.visible_cells());
    assert_eq!(instances[0].fg, [255, 0, 0, 255]);
    assert_eq!(instances[0].grid_pos, [0, 0]);
    assert_eq!(instances[4].fg, DEFAULT_FG);
}

#[test]
fn cell_to_ndc_conversion_is_correct() {
    // Pure arithmetic, and the source of the upside-down-screen bug.
    let u = Uniforms { screen_size: [800.0, 600.0], cell_size: [8.0, 17.0], .. };
    assert_eq!(cell_to_ndc([0, 0], &u), [-1.0, 1.0]);       // top-left
    assert_eq!(cell_to_ndc([100, 0], &u), [1.0, 1.0]);      // top-right
    // Y is FLIPPED between screen space and NDC.
    assert!(cell_to_ndc([0, 35], &u)[1] < 0.0);
}

#[test]
fn atlas_full_triggers_eviction_not_corruption() {
    // A session with thousands of distinct CJK glyphs WILL fill the atlas.
    let mut atlas = GpuAtlas::new(256, 256);            // deliberately tiny
    for c in ('\u{4E00}'..).take(10_000) {
        atlas.get_or_insert(GlyphKey { ch: c, size_px: 16, ..Default::default() });
    }
    assert!(atlas.evictions() > 0);
    assert!(atlas.used_bytes() <= 256 * 256);
}

#[test]
#[ignore]                                    // needs a GPU; run with --ignored
fn gpu_output_matches_cpu_output() {
    // The correctness test that actually matters. Render the same snapshot both
    // ways, read back the GPU framebuffer, and compare with a tolerance for
    // rasterization differences.
    let snap = snapshot_of(GOLDEN_INPUT);
    let cpu = CpuRenderer::new(..).render_to_buffer(&snap);
    let gpu = GpuRenderer::new(..).render_to_buffer(&snap);
    let diff = max_channel_difference(&cpu, &gpu);
    assert!(diff <= 4, "CPU and GPU renderers diverge by {diff}/255");
}
}

That last test is worth the effort to set up. wgpu can render to a texture and copy it back, so it runs headless in CI on a software adapter.

6. The Measurement

This project is justified by numbers or not at all.

git clone https://github.com/alacritty/vtebench && cd vtebench && cargo build --release
./target/release/vtebench -b ./benchmarks alt-screen-random-write > /tmp/bench.raw
time (cat /tmp/bench.raw)     # in each renderer

The honest expected result: at 80×24 the GPU renderer is not measurably better, and may be slightly worse because of upload overhead. It wins at 400×100 and under floods. Report that. A project write-up that says "the GPU renderer is 40× faster at the size I actually use, and identical at the size most people use" is a better piece of engineering writing than one that only reports the win.

7. Known Traps

Tip: Keep the CPU renderer. Ship both behind a feature flag or a config option. It is your correctness oracle, your fallback for machines without a usable GPU, and the thing that makes the comparison test possible. Every serious terminal that went GPU-only later regretted it.

Deliverables

• A wgpu renderer consuming the same RenderSnapshot as the CPU one.
• A GPU glyph atlas with lazy upload and eviction.
• Instanced drawing, two passes, one draw call each.
• The CPU renderer retained and selectable.
• The instance-data test and the NDC test (no GPU required).
• The CPU/GPU comparison test, running headless in CI.
• The full measurement table at three grid sizes, including where the GPU does not win.
• No change to terminal-core or terminal-render-model — verified with git diff --stat.

That last one is the real point of the project: if the boundary is right, a whole new renderer touches exactly one crate.

Next: Project 4 — A Windows ConPTY Backend