Lab 5: The PTY Session Recorder and Replayer

Background

Everything from here on needs deterministic input. Testing a terminal parser by launching vim and eyeballing the screen is not testing; it is hoping. And debugging a rendering bug that only happens when htop redraws is impossible if you have to reproduce it by hand each time.

The fix is a recorder: capture every byte the PTY master produced, with timestamps and direction, to a file. Then replay that file into your terminal core without spawning a shell at all. Recording is a Section 1 concern (it is pure PTY plumbing); replay is what makes Sections 2–5 testable.

This is the first piece of terminal-debugger, and it is the single highest-leverage tool you will build.

Why This Lab Matters

• Golden tests are recorded streams plus expected snapshots. No recorder, no golden tests.
• Differential tests replay one stream into two parsers.
• Every bug report you write for the rest of this curriculum should include a .cast file.
• asciinema and script already do this; you are building a version whose format you control, which is why you can put parser-state annotations in it later.

Prerequisites

• Lab 3 working — you record from inside that loop.

Predict First

1. You record a session of vim, then replay it. Will the screen be identical? Name two reasons it might not be.
2. Your recording is of top. What in the byte stream makes replay non-deterministic?
3. Should you record input (keystrokes) as well as output? What breaks if you only record output?

Step 1: Choose a Format

Three candidates. Understand the trade-offs before picking.

Recommendation: implement an asciinema-v2-compatible writer, plus a small extension for resize and input. You get asciinema play for free as a sanity check, and your own replay for tests.

# asciinema v2: a JSON header line, then one JSON array per event.
{"version":2,"width":80,"height":24,"timestamp":1735689600,"env":{"TERM":"xterm-256color","SHELL":"/bin/bash"}}
[0.248, "o", "bash-5.2$ "]
[1.102, "i", "l"]
[1.103, "o", "l"]
[1.240, "i", "s"]
[1.241, "o", "s"]
[1.500, "i", "\r"]
[1.502, "o", "\r\nCargo.toml  src\r\n"]
[3.000, "r", "30x100"]

Time is seconds since the start, as a float. Data is a JSON string — which means it must be valid UTF-8. That is a real constraint; see Step 3.

Step 2: The Recorder

#![allow(unused)]
fn main() {
// crates/terminal-debugger/src/record.rs
use std::io::{self, Write};
use std::time::Instant;

pub struct Recorder<W: Write> {
    out: W,
    start: Instant,
}

#[derive(Copy, Clone, Debug, PartialEq, Eq)]
pub enum Direction { Output, Input, Resize }

impl<W: Write> Recorder<W> {
    pub fn new(mut out: W, cols: u16, rows: u16, term: &str, shell: &str,
               unix_ts: u64) -> io::Result<Self> {
        // The header is a single JSON object on line 1.
        writeln!(
            out,
            r#"{{"version":2,"width":{cols},"height":{rows},"timestamp":{unix_ts},"env":{{"TERM":"{term}","SHELL":"{shell}"}}}}"#
        )?;
        out.flush()?;
        Ok(Self { out, start: Instant::now() })
    }

    pub fn event(&mut self, dir: Direction, data: &[u8]) -> io::Result<()> {
        let t = self.start.elapsed().as_secs_f64();
        let code = match dir {
            Direction::Output => "o",
            Direction::Input => "i",
            Direction::Resize => "r",
        };
        writeln!(self.out, "[{:.6}, \"{}\", \"{}\"]", t, code, json_escape(data))?;
        // Flush every event. A recording of a crash is only useful if the bytes
        // that caused the crash actually reached the disk.
        self.out.flush()
    }
}

/// Escape bytes for a JSON string.
///
/// The hard part: the byte stream may contain INVALID UTF-8 — either genuinely
/// (a program emitting binary) or transiently (a multi-byte character split
/// across two read() calls). JSON strings must be valid UTF-8, so invalid bytes
/// are escaped as \u00XX, preserving them losslessly for the replayer, which
/// converts them back to single bytes.
fn json_escape(data: &[u8]) -> String {
    let mut s = String::with_capacity(data.len() + 8);
    let mut i = 0;
    while i < data.len() {
        let b = data[i];
        match b {
            b'"' => { s.push_str("\\\""); i += 1; }
            b'\\' => { s.push_str("\\\\"); i += 1; }
            b'\n' => { s.push_str("\\n"); i += 1; }
            b'\r' => { s.push_str("\\r"); i += 1; }
            0x08 => { s.push_str("\\b"); i += 1; }
            0x0c => { s.push_str("\\f"); i += 1; }
            b'\t' => { s.push_str("\\t"); i += 1; }
            0x00..=0x1f | 0x7f => { s.push_str(&format!("\\u{:04x}", b)); i += 1; }
            0x20..=0x7e => { s.push(b as char); i += 1; }
            _ => {
                // Possible UTF-8 multi-byte sequence. Try to decode it; if it is
                // valid, emit it as text. If not, escape each byte individually.
                match std::str::from_utf8(&data[i..]) {
                    Ok(valid) => {
                        let c = valid.chars().next().unwrap();
                        s.push(c);
                        i += c.len_utf8();
                    }
                    Err(e) if e.valid_up_to() > 0 => {
                        let valid = std::str::from_utf8(&data[i..i + e.valid_up_to()]).unwrap();
                        s.push_str(valid);
                        i += e.valid_up_to();
                    }
                    Err(_) => {
                        // Invalid or incomplete: preserve the raw byte.
                        s.push_str(&format!("\\u{:04x}", b));
                        i += 1;
                    }
                }
            }
        }
    }
    s
}
}

Warning: The invalid-UTF-8 case is not theoretical. A read() on the master will regularly split a multi-byte character across two chunks — and that split is exactly what you want to reproduce, because a parser that mishandles it is a parser with a real bug. Escaping the bytes individually, rather than replacing them with U+FFFD, is what preserves that. A recorder that "cleans up" its input is worse than useless: it hides the class of bug you most want to catch.

Wiring it into the event loop

#![allow(unused)]
fn main() {
// In the Lab 3 loop, at the two points where bytes cross the boundary:

// master → stdout
if n > 0 {
    if let Some(rec) = recorder.as_mut() {
        rec.event(Direction::Output, &buf[..n as usize])?;
    }
    to_stdout.queue(&buf[..n as usize]);
}

// stdin → master
if n > 0 {
    if let Some(rec) = recorder.as_mut() {
        rec.event(Direction::Input, &buf[..n as usize])?;
    }
    to_master.queue(&buf[..n as usize]);
}

// on SIGWINCH, after propagating:
if let Some(rec) = recorder.as_mut() {
    rec.event(Direction::Resize, format!("{}x{}", ws.ws_col, ws.ws_row).as_bytes())?;
}
}

Record at the boundary, not deeper. The recording should be exactly the bytes that crossed the PTY, so that replay is a faithful substitute for the PTY.

Step 3: The Replayer

#![allow(unused)]
fn main() {
// crates/terminal-debugger/src/replay.rs
use std::io;
use std::time::Duration;

pub struct Event {
    pub time: f64,
    pub direction: Direction,
    pub data: Vec<u8>,
}

pub struct Recording {
    pub width: u16,
    pub height: u16,
    pub events: Vec<Event>,
}

impl Recording {
    pub fn parse(text: &str) -> io::Result<Recording> {
        let mut lines = text.lines();
        let header = lines.next().ok_or_else(
            || io::Error::new(io::ErrorKind::InvalidData, "empty recording"))?;
        let (width, height) = parse_header(header)?;
        let mut events = Vec::new();
        for line in lines {
            if line.trim().is_empty() { continue; }
            events.push(parse_event(line)?);
        }
        Ok(Recording { width, height, events })
    }

    /// Feed the OUTPUT events into a sink as fast as possible.
    /// This is the mode tests use: no sleeping, fully deterministic.
    pub fn replay_output_fast(&self, sink: &mut impl FnMut(&[u8])) {
        for e in &self.events {
            if e.direction == Direction::Output {
                sink(&e.data);
            }
        }
    }

    /// Replay with the original timing, for humans watching.
    pub fn replay_realtime(&self, speed: f64, sink: &mut impl FnMut(&[u8])) {
        let mut prev = 0.0;
        for e in &self.events {
            if e.direction != Direction::Output { continue; }
            let delta = ((e.time - prev) / speed).max(0.0);
            std::thread::sleep(Duration::from_secs_f64(delta));
            prev = e.time;
            sink(&e.data);
        }
    }

    /// Replay output in CHUNKS OF ONE BYTE. This is the single most valuable
    /// test mode you have: a correct parser produces identical state whether it
    /// receives a sequence all at once or one byte at a time. Most naive parsers
    /// fail this immediately.
    pub fn replay_output_bytewise(&self, sink: &mut impl FnMut(&[u8])) {
        for e in &self.events {
            if e.direction != Direction::Output { continue; }
            for b in &e.data {
                sink(std::slice::from_ref(b));
            }
        }
    }
}
}

Tip: replay_output_bytewise is worth the whole lab on its own. Feed every golden recording through both replay_output_fast and replay_output_bytewise and assert the resulting screens are identical. Any difference is a parser that is not a real state machine — which is exactly the bug Milestone 4 exists to prevent.

Step 4: The CLI

# Record a session:
pty-runner --record session.cast

# Replay for a human, at 2x:
terminal-debugger play session.cast --speed 2

# Replay into a terminal core and print the final screen (Milestone 6):
mini-term replay session.cast --rows 24 --cols 80 --format text

# Sanity check against a reference implementation:
asciinema play session.cast

Expected Output

$ pty-runner --record demo.cast
bash-5.2$ printf '\033[31mred\033[0m and \033[1mbold\033[0m\n'
red and bold
bash-5.2$ exit

$ head -4 demo.cast
{"version":2,"width":80,"height":24,"timestamp":1767225600,"env":{"TERM":"xterm-256color","SHELL":"/bin/bash"}}
[0.031000, "o", "bash-5.2$ "]
[1.204000, "i", "p"]
[1.205000, "o", "p"]

$ grep -c '"o"' demo.cast
47

$ terminal-debugger play demo.cast --speed 4
# the session replays, colors and all, with no shell involved

Debugging Steps

Replay produces different output than the live session

The usual causes, in order of likelihood:

1. Terminal queries. The program wrote \x1b[6n (cursor position) or \x1b[c (device attributes) and waited for a reply. Live, your terminal replied. On replay, nothing does — so the recorded output diverges from that point. Fix: record the input events too, and have the replayer feed the recorded replies, or have your terminal core generate them.
2. Time-dependent programs. top and watch print the clock. The bytes differ every run. Compare screens with time fields masked, or record a deterministic program.
3. Environment. $TERM, $COLUMNS, locale. The header records them; the replay must apply them.
4. Size. If the replay terminal is a different size than the recording, wrapping differs. Always use the header's width/height.

The recording contains invalid JSON

Your escaping is incomplete. The usual culprits: an unescaped " or \, or a raw control byte. Test with python3 -c 'import json,sys; [json.loads(l) for l in sys.stdin]' < demo.cast.

Recording a long session produces an enormous file

Expected — yes for one second is megabytes. Add --max-bytes and a note in the header when truncated. Do not silently truncate.

asciinema play rejects the file

Your header is missing a required field, or a time value is not monotonically non-decreasing. Times must never go backwards.

Experiment

CLAIM. A recording that omits input events cannot faithfully replay any session in which the program queried the terminal.

METHOD.

# 1. Record a session that queries the terminal.
pty-runner --record query.cast
#    inside:  printf '\033[6n'; read -r -d R reply; echo "reply was: ${reply#*[}"
#    exit

# 2. Look at the recording:
grep -n '"i"' query.cast | head
#    You will see the terminal's REPLY arriving as an input event.

# 3. Replay with output only, then with input replay enabled. Compare.

PREDICTION. Before step 3: what does the shell print on an output-only replay? Does it hang?

RESULT. Record it. Then decide: should your terminal core answer \x1b[6n itself during replay? (The answer is yes — it is Terminal::take_replies() in the workspace design — and now you know why that method exists.)

Test

#![allow(unused)]
fn main() {
#[test]
fn round_trip_preserves_arbitrary_bytes() {
    // Recording must be lossless, including invalid UTF-8 — a split multi-byte
    // character is a real occurrence and a real source of parser bugs.
    let payloads: Vec<Vec<u8>> = vec![
        b"hello".to_vec(),
        b"\x1b[31mred\x1b[0m".to_vec(),
        vec![0x00, 0x01, 0x1b, 0x7f, 0xff],          // control + invalid UTF-8
        vec![0xf0, 0x9f],                             // an INCOMPLETE emoji
        "日本語 🙂".as_bytes().to_vec(),
        b"quote\" backslash\\ newline\n".to_vec(),
    ];
    let mut buf = Vec::new();
    {
        let mut rec = Recorder::new(&mut buf, 80, 24, "xterm-256color", "/bin/sh", 0).unwrap();
        for p in &payloads {
            rec.event(Direction::Output, p).unwrap();
        }
    }
    let text = String::from_utf8(buf).unwrap();
    let parsed = Recording::parse(&text).unwrap();
    let got: Vec<Vec<u8>> = parsed.events.iter().map(|e| e.data.clone()).collect();
    assert_eq!(got, payloads, "recording must be byte-exact");
}

#[test]
fn every_line_after_the_header_is_valid_json() {
    let mut buf = Vec::new();
    {
        let mut rec = Recorder::new(&mut buf, 80, 24, "xterm", "/bin/sh", 0).unwrap();
        rec.event(Direction::Output, &[0x00, 0x22, 0x5c, 0xff]).unwrap();
    }
    for line in String::from_utf8(buf).unwrap().lines() {
        assert!(line.starts_with('{') || line.starts_with('['), "bad line: {line}");
        // If you add a JSON dependency for tests, parse each line here.
    }
}

#[test]
fn timestamps_never_go_backwards() {
    // asciinema players reject non-monotonic times, and so should you.
    let rec = Recording::parse(SAMPLE).unwrap();
    let mut prev = f64::NEG_INFINITY;
    for e in &rec.events {
        assert!(e.time >= prev, "time went backwards: {} after {}", e.time, prev);
        prev = e.time;
    }
}
}

Challenge Extensions

1. Annotate the recording. Add a "p" (parser) event type that your debugger writes alongside output, recording parser state transitions. Now a recording is a full trace, and terminal-debugger step file.cast can walk it. This is the seed of the debugger.

2. Record termios changes. Poll tcgetattr(master) after each output burst; when it changes, emit a "t" event. Now your recording shows the exact moment vim switched the terminal to raw mode — which is the answer to a whole class of "why did input behavior change?" questions.

3. Build a corpus. Record ten sessions: ls --color, vim opening and quitting, top for two seconds, less on a long file with scrolling, htop with a mouse drag, a python3 REPL session, git log --graph --color, a CJK text file, an emoji-heavy file, and cat /dev/urandom | head -c 4096. Check them into tests/golden/cases/. These are your regression corpus for the rest of the curriculum.

4. Binary format. Implement a length-prefixed binary variant and benchmark: file size and replay throughput versus the JSON version. This is the same trade-off you will weigh for the mux protocol, so do the measurement now and keep the numbers.

5. Deterministic replay with reply injection. Make the replayer answer terminal queries from the recorded input stream, so a session containing \x1b[6n replays faithfully.

Implementation Requirements / Deliverables

• Recorder writes a valid asciinema-v2 file, verified with asciinema play if available.
• Invalid UTF-8 and split multi-byte sequences round-trip byte-exactly.
• Recording::parse + all three replay modes (fast, realtime, bytewise) implemented.
• --record wired into pty-runner, capturing output, input, and resize.
• The ten-recording corpus from challenge 3, checked in.
• All three tests pass.
• A written answer to: why must the recorder preserve invalid UTF-8 rather than sanitizing it?

Validation / Self-check

1. Why record input as well as output? Give a concrete session that breaks without it.
2. Why must the recorder preserve invalid UTF-8 byte-for-byte?
3. What is replay_output_bytewise for, and what class of bug does it catch?
4. Name three sources of non-determinism in a recorded session and how to handle each.
5. Why does the recorder flush after every event?
6. Your replay of a vim session diverges after 200 events. What is the first thing you check?
7. Why does Terminal::take_replies() exist, and how did this lab prove you need it?
8. What is the trade-off between the JSON and binary recording formats? Which numbers did you measure?

Section 1 Complete

You now have:

• A raw-mode byte inspector that decodes keys.
• A PTY layer written from raw syscalls, with every failure mode understood.
• A correct event loop with resize, signals, child reaping, and buffered non-blocking I/O.
• Seven experiments' worth of direct observation.
• A recorder and replayer that make everything after this testable.

What you do not have: any understanding of the bytes themselves. Your runner relays \x1b[31mred\x1b[0m faithfully without knowing it means "red". That is the next section.

Next: Section 2 — The Minimal Terminal Emulator.