The terminal-debugger

A terminal is a pipeline of six layers, and a bug lives in exactly one of them. Without instrumentation you find out which by guessing. With it, you find out in thirty seconds.

terminal-debugger is the tool that makes every layer inspectable. Build it early — it pays for itself the first time you use it — and extend it as you add layers.

What It Does

The brief asks for nine capabilities. Here they are, with what each is for:

The CLI

terminal-debugger hex      [--width 16] [--color] FILE|-
terminal-debugger decode   [--verbose] FILE|-
terminal-debugger split    FILE|-                    # text vs. control, side by side
terminal-debugger actions  FILE|-                    # the parsed action stream
terminal-debugger step     [--from N] [--to N] FILE  # interactive parser stepping
terminal-debugger play     [--speed N] FILE.cast
terminal-debugger replay   [--rows R] [--cols C] [--bytewise] FILE.cast
terminal-debugger snapshot [--rows R] [--cols C] [--format text|debug|json] FILE
terminal-debugger diff     [--format text|json] A B
terminal-debugger record   -- COMMAND [ARGS...]
terminal-debugger bisect   --expect FILE.snapshot FILE.cast   # find the first bad byte

1. Hex Dump with Decoding

$ printf '\033[31mred\033[0m\n' | terminal-debugger hex
00000000  1b 5b 33 31 6d 72 65 64  1b 5b 30 6d 0a           |.[31mred.[0m.|
          ^^^^^^^^^^^^^^^ CSI 31 m (SGR: foreground red)
                          ^^^^^^^^ text "red"
                                   ^^^^^^^^^^^ CSI 0 m (SGR: reset)
                                               ^^ LF

#![allow(unused)]
fn main() {
pub fn hex_dump(data: &[u8], width: usize, out: &mut impl Write) -> io::Result<()> {
    // Annotate first, so the byte view can point at semantic spans.
    let spans = SequenceDecoder::spans(data);
    for (offset, chunk) in data.chunks(width).enumerate() {
        write!(out, "{:08x}  ", offset * width)?;
        for (i, b) in chunk.iter().enumerate() {
            write!(out, "{b:02x} ")?;
            if i == width / 2 - 1 { write!(out, " ")?; }
        }
        for _ in chunk.len()..width { write!(out, "   ")?; }
        write!(out, " |")?;
        for &b in chunk {
            // Caret notation for controls; '.' for non-ASCII. NEVER print the
            // raw byte — a debugger that emits escape sequences into the
            // terminal you are debugging is worse than useless.
            match b {
                0x20..=0x7e => write!(out, "{}", b as char)?,
                _ => write!(out, ".")?,
            }
        }
        writeln!(out, "|")?;
        // Underline any decoded sequence that starts in this row.
        for span in spans.iter().filter(|s| s.overlaps(offset * width, width)) {
            writeln!(out, "{}{} {}", " ".repeat(10 + span.col_offset(offset, width) * 3),
                     "^".repeat(span.len * 3 - 1), span.description)?;
        }
    }
    Ok(())
}
}

Warning: The single most important rule for a terminal debugger: never write raw bytes to the terminal you are debugging. A hex dumper that prints \x1b[31m verbatim changes the state of the thing under investigation. Always escape, and prefer writing to a file or a second terminal.

2. Sequence Decoding

$ terminal-debugger decode session.cast | head
     0  CSI ? 1049 h    Enter alternate screen buffer (save cursor + clear)
    10  CSI ? 1 h       DECCKM: application cursor keys
    16  CSI ? 2004 h    Enable bracketed paste
    25  CSI 1 ; 24 r    DECSTBM: scroll region rows 1-24
    34  CSI 2 J         ED 2: erase entire screen (cursor unmoved)
    39  CSI H           CUP: cursor to (1,1)
    42  OSC 0 ; "vim"   Set window title and icon name
    54  CSI 1 m         SGR 1: bold
    58  text            "~"  (1 grapheme, 1 cell)

#![allow(unused)]
fn main() {
pub struct Description {
    pub offset: usize,
    pub len: usize,
    /// The canonical mnemonic: "CSI ? 1049 h".
    pub raw: String,
    /// What it MEANS, in a sentence.
    pub description: String,
    /// Whether we implement it. This is what turns the decoder into a to-do list.
    pub supported: bool,
}
}

The supported flag is the trick. terminal-debugger decode --unsupported-only gives you exactly the Milestone 14 backlog for any recording.

3. Text vs. Control, Split

$ terminal-debugger split session.cast
CONTROL                          | TEXT
---------------------------------|-------------------------------------------
CSI ?1049h                       |
CSI 2J                           |
CSI H                            |
                                 | "~"
CSI 2;1H                         |
                                 | "~"
CSI 24;1H CSI 1m                 |
                                 | "\"[No Name]\"      0,0-1     All"
CSI 0m                           |

Answers "is the program sending the text I expect, and is the positioning wrong?" in one glance. That question comes up constantly, and this view answers it immediately.

4. Parser Stepping

The highest-value feature, and the reason to build the TracingPerform wrapper early.

$ terminal-debugger step --from 0 --to 20 session.cast
  #  off  byte  state before      → state after       action
  0    0  0x1b  Ground            → Escape            (clear)
  1    1  0x5b  Escape            → CsiEntry          —
  2    2  0x3f  CsiEntry          → CsiParam          private='?'
  3    3  0x31  CsiParam          → CsiParam          param=1
  4    4  0x30  CsiParam          → CsiParam          param=10
  5    5  0x34  CsiParam          → CsiParam          param=104
  6    6  0x39  CsiParam          → CsiParam          param=1049
  7    7  0x68  CsiParam          → Ground            CSI_DISPATCH ?[1049] 'h'
                                                      → alt screen ON, cursor saved
  8    8  0x1b  Ground            → Escape            (clear)
...
[n]ext  [p]rev  [c]ontinue  [s]creen  [q]uit  [/]search-for-action

#![allow(unused)]
fn main() {
/// Wraps any Perform, logging every call before forwarding. This is
/// --debug-actions, and it is how you debug every screen bug for the rest of
/// the project. Build it in the same session as the parser, not later.
pub struct TracingPerform<'a, P: Perform, W: Write> {
    inner: &'a mut P,
    out: W,
    step: usize,
}
}

The interactive mode ([s]creen after any step) is what turns this from a log into a debugger: you can watch the screen evolve one byte at a time.

5. Snapshots and Diffing

$ terminal-debugger snapshot --rows 5 --cols 20 --format debug session.cast
=== TERMINAL 5x20 ===
cursor:  row=2 col=7 pending_wrap=false visible=true shape=Block
modes:   AUTO_WRAP | CURSOR_VISIBLE | ALT_SCREEN | BRACKETED_PASTE | APP_CURSOR_KEYS
region:  0..4
title:   "vim"
buffer:  ALTERNATE
scrollback: 0 lines (primary: 143 lines)
--- screen ---
 0 |~
 1 |~
 2 |~      █
 3 |~
 4 |"[No Name]"  0,0-1
--- non-default styles ---
 (4,0)-(4,18) fg=Default bg=Indexed(7) INVERSE
--- damage ---
rows: 2, 4

$ terminal-debugger diff before.snapshot after.snapshot
--- before.snapshot
+++ after.snapshot
@@ cursor @@
-cursor: row=0 col=5
+cursor: row=1 col=0
@@ row 0 @@
-hello
+hello world
@@ row 1 @@
-
+more text
@@ modes @@
+ALT_SCREEN
@@ styles @@
+(0,6)-(0,10) fg=Indexed(1)

Snapshot diffing is the fastest way to answer "what did this sequence do?" Snapshot, feed one sequence, snapshot, diff. It replaces a great deal of guessing.

6. Bisect: Find the First Bad Byte

Forty lines that turn a two-hour session into two minutes.

#![allow(unused)]
fn main() {
/// Binary-search a byte stream for the first prefix whose snapshot diverges
/// from the expected one. Assumes divergence is monotonic — once wrong, it
/// stays wrong — which is true for the vast majority of terminal bugs.
pub fn bisect(bytes: &[u8], rows: usize, cols: usize, expected: &str) -> Option<usize> {
    let ok = |n: usize| {
        let mut t = Terminal::new(rows, cols, Default::default());
        t.advance(&bytes[..n]);
        expected.starts_with(t.snapshot_text().trim_end())
    };
    if ok(bytes.len()) { return None; }          // nothing diverges
    let (mut lo, mut hi) = (0usize, bytes.len());
    while lo + 1 < hi {
        let mid = (lo + hi) / 2;
        if ok(mid) { lo = mid; } else { hi = mid; }
    }
    Some(hi)
}
}

$ terminal-debugger bisect --expect expected.snapshot session.cast
First divergence at byte 4,127 of 89,204.

Context (bytes 4110-4145):
  1b 5b 33 38 3b 32 3b 32 35 35 3b 30 3b 30 6d 48   |.[38;2;255;0;0mH|
     ^^ CSI 38;2;255;0;0 m  (SGR: truecolor foreground)

Screen BEFORE byte 4127:  row 3 = "Loading"
Screen AFTER  byte 4127:  row 3 = "Loading" (expected: "Loading" with fg=Rgb(255,0,0))

Likely cause: SGR 38 semicolon-form parameter consumption.

7. Recording and Replay

Covered in Lab 5. The debugger is where the replay half lives.

terminal-debugger record -- vim -u NONE       # capture
terminal-debugger play session.cast --speed 4 # watch it back
terminal-debugger replay session.cast --bytewise --format debug
#                                     ^^^^^^^^^ the parser-correctness mode

The Debug Channel Design

Every layer gets a switchable channel:

#![allow(unused)]
fn main() {
bitflags! {
    pub struct DebugChannels: u32 {
        const BYTES   = 1 << 0;   // raw hex, both directions, with timestamps
        const UTF8    = 1 << 1;   // decode events, including errors
        const PARSER  = 1 << 2;   // state transitions
        const ACTIONS = 1 << 3;   // the parsed action stream
        const CURSOR  = 1 << 4;   // every movement, with its cause
        const DAMAGE  = 1 << 5;   // dirty rows per frame
        const MODES   = 1 << 6;   // every set/reset, by DEC number and name
        const PROC    = 1 << 7;   // spawns, signals, exits
        const RENDER  = 1 << 8;   // frame times, atlas hit rate
        const PROTO   = 1 << 9;   // mux protocol messages
    }
}
}

MINI_TERM_DEBUG=parser,actions,cursor MINI_TERM_DEBUG_LOG=/tmp/t.log cargo run

The four rules (from the teaching method):

1. Never to the terminal being debugged. File, or fd 2 when it is redirected, or a separate socket.
2. Switchable at runtime, not by recompiling.
3. Cheap when off — an if on a bitflag, not a formatted string that is discarded.
4. Replayable — anything recorded can be replayed without a shell.

#![allow(unused)]
fn main() {
// ✓ Cheap when off: nothing is formatted unless the channel is enabled.
macro_rules! dbg_channel {
    ($self:expr, $ch:expr, $($arg:tt)*) => {
        if $self.debug.contains($ch) { $self.debug_log(format_args!($($arg)*)); }
    };
}
dbg_channel!(self, DebugChannels::CURSOR, "({},{}) → ({},{}) cause={}",
             old.row, old.col, new.row, new.col, cause);
}

The Debugging Workflow

The five-question order from the teaching method, with the tool for each:

   1. WHICH LAYER?
        terminal-debugger hex        → are the bytes right?
        terminal-debugger actions    → is the parse right?
        terminal-debugger snapshot   → is the state right?
      The first one that is wrong is your layer.

   2. WHICH SIDE OF THE KERNEL?
        stty -a -F /dev/pts/N        → line-discipline configuration
      Compare against a working terminal.

   3. WHICH PROCESS?
        ps -o pid,pgid,sid,tpgid,stat,tty,comm

   4. WHICH DIRECTION?
        MINI_TERM_DEBUG=bytes        → both directions are logged separately

   5. MY BUG OR A MISSING FEATURE?
        terminal-debugger decode --unsupported-only session.cast
      If it lists something, that is your answer.

A worked example. "Colors are wrong in ls but right in vim."

# 1. Which layer? Record and inspect the bytes.
mini-term run --record ls.cast -- ls --color=always
terminal-debugger hex ls.cast | head -20
#    → the bytes contain CSI 0;38;5;33m. Bytes are fine; the layer is downstream.

# 2. Is the parse right?
terminal-debugger actions ls.cast | head
#    → CSI [0, 38, 5, 33] 'm'.  Correct parse.

# 3. Is the state right?
terminal-debugger snapshot --format debug ls.cast | grep 'styles'
#    → (0,0) fg=Indexed(5)     ← WRONG. Should be Indexed(33).

# 4. Found it: SGR 38's semicolon-form parameter consumption is off by one —
#    it read params[i+1] as the color instead of params[i+2].
#    vim uses SGR 1;34 (the simple form), which is why it looked fine.

Four commands, one bug, no guessing.

Experiment

CLAIM. Instrumentation converts debugging from search into lookup.

METHOD. Introduce a deliberate bug and find it twice — once without the debugger, once with.

# 1. Break something subtle: make ED 2 also home the cursor (a real bug that
#    many implementations have shipped).
#      2 => { erase_all(); self.cursor = Cursor::default(); }   // WRONG

# 2. Find it WITHOUT the debugger. Time yourself. You are allowed println!.

# 3. Revert your println!s, and find it WITH:
terminal-debugger snapshot --format debug before.cast > /tmp/a
#    ...feed CSI 2J...
terminal-debugger snapshot --format debug after.cast > /tmp/b
terminal-debugger diff /tmp/a /tmp/b
#    → "@@ cursor @@  -row=3 col=7  +row=0 col=0"

PREDICTION. Before starting: how long will each take? Write both estimates.

RESULT. Record the two times. The ratio is your return on building this tool, and it is usually somewhere between five and twenty.

Test

#![allow(unused)]
fn main() {
#[test]
fn hex_dump_never_emits_raw_control_bytes() {
    // A debugger that writes escape sequences into the terminal you are
    // debugging changes the thing under investigation.
    let mut out = Vec::new();
    hex_dump(b"\x1b[31mred\x1b[0m\x07", 16, &mut out).unwrap();
    assert!(!out.contains(&0x1b), "the dump must not contain ESC");
    assert!(!out.contains(&0x07), "the dump must not contain BEL");
}

#[test]
fn decoder_identifies_common_sequences() {
    let cases: &[(&[u8], &str)] = &[
        (b"\x1b[31m", "SGR"),
        (b"\x1b[?1049h", "alternate screen"),
        (b"\x1b[2J", "erase"),
        (b"\x1b]0;title\x07", "title"),
        (b"\x1b[6n", "cursor position report"),
    ];
    for (bytes, needle) in cases {
        let d = SequenceDecoder::describe(bytes);
        assert!(d[0].description.to_lowercase().contains(needle),
                "{:?} → {}", bytes, d[0].description);
    }
}

#[test]
fn decoder_flags_unsupported_sequences() {
    // This flag turns the decoder into a Milestone 14 to-do list.
    let d = SequenceDecoder::describe(b"\x1b[?9999h");
    assert!(!d[0].supported);
}

#[test]
fn snapshot_diff_reports_cursor_state_and_content_separately() {
    let a = snapshot_of(b"hello");
    let b = snapshot_of(b"hello\x1b[5;5H");
    let d = snapshot_diff(&a, &b);
    assert!(d.contains("cursor"));
    assert!(!d.contains("row 0"), "content did not change; do not report it");
}

#[test]
fn bisect_finds_the_first_divergent_byte() {
    let good = b"hello world";
    let bad  = b"hello WORLD";
    let expected = snapshot_text_of(good);
    let n = bisect(bad, 3, 20, &expected).unwrap();
    assert_eq!(n, 7, "the first difference is at index 6; bisect reports the prefix length");
}

#[test]
fn debug_channels_are_free_when_disabled() {
    // Cheap-when-off: no formatting, no allocation. Measured, not assumed.
    let mut t = Terminal::new(24, 80, Default::default());
    t.set_debug(DebugChannels::empty());
    let start = std::time::Instant::now();
    for _ in 0..10_000 { t.advance(b"\x1b[31mx\x1b[0m"); }
    let off = start.elapsed();
    t.set_debug(DebugChannels::all());
    t.set_debug_sink(Box::new(std::io::sink()));
    let start = std::time::Instant::now();
    for _ in 0..10_000 { t.advance(b"\x1b[31mx\x1b[0m"); }
    let on = start.elapsed();
    assert!(off * 3 < on, "debug-off ({off:?}) should be much faster than on ({on:?})");
}
}

Challenge Extensions

1. A TUI for step mode using your own terminal, with the screen on one side and the byte stream on the other. Dogfooding at its most direct.
2. Live attach: connect the debugger to a running terminal over a Unix socket and stream channels in real time.
3. A flamegraph of sequence frequency across your corpus: which sequences dominate real workloads? The answer tells you what to optimize.
4. --explain: given a sequence, print its full specification, its ECMA-48/DEC origin, which programs use it, and your implementation status.
5. Automatic minimization: given a failing recording, shrink it to the smallest byte sequence that still reproduces the failure — like creduce for terminals.
6. A protocol channel for the mux, showing framed messages with timing, so client/server issues get the same treatment.

Validation / Self-check

1. Name the nine capabilities and one bug each finds.
2. What is the single most important rule for a terminal debugger's output?
3. What does the supported flag turn the decoder into?
4. Why is split (text vs. control) useful? What question does it answer instantly?
5. How does snapshot diffing answer "what did this sequence do?"
6. Explain the bisect algorithm and the assumption it makes.
7. State the four instrumentation design rules and the bug from breaking each.
8. Give the five debugging questions and the tool for each.
9. Walk through the ls color bug: four commands, and what each ruled out.
10. What was your with-versus-without debugging time ratio?

Next: Linux, macOS & ConPTY.