Project 1: Sixel and the Kitty Graphics Protocol

3–4 weeks · ●●●●○ · touches DCS/APC parsing, image decoding, the cell/pixel boundary, rendering

1. The Problem

Your terminal displays text. It cannot display an image. Programs like timg, chafa, viu, img2sixel, matplotlib's sixel backend, and ranger's previewer all want to put pixels in a cell grid, and today your terminal silently discards their output.

Two competing protocols exist, and supporting either is a real feature.

2. Why It Is Hard

The difficulty is not decoding — it is that images break the cell abstraction your entire architecture is built on.

That last row is the design crux, and getting it right is the project.

3. The Design

The boundary decision

   ✗ WRONG: terminal-core stores decoded RGBA pixels.
     It now depends on an image decoder, cannot compile to wasm32 without one,
     and a headless mux server allocates megabytes per pane for pictures nobody
     is looking at.

   ✓ RIGHT: terminal-core stores an IMAGE PLACEMENT — an id, a grid anchor, a
     cell footprint, and the *undecoded* payload (or a handle to it). The
     RENDERER decodes and draws. The core knows where an image is, never what
     it looks like.

#![allow(unused)]
fn main() {
/// Lives in terminal-core. Note what is absent: no pixels, no decoder, no RGBA.
pub struct ImagePlacement {
    pub id: ImageId,
    /// Anchored in ABSOLUTE (scrollback-inclusive) coordinates so it scrolls
    /// with the text for free — the same trick as selections.
    pub anchor: AbsolutePoint,
    pub cells_wide: u16,
    pub cells_high: u16,
    /// Pixel dimensions as declared by the protocol; the renderer scales.
    pub pixel_width: u32,
    pub pixel_height: u32,
    pub z_index: i32,
}

/// The undecoded payload, in a side table the renderer consumes.
pub struct ImageStore {
    images: HashMap<ImageId, EncodedImage>,
    /// Bounded. An unbounded image store is a trivial memory-exhaustion vector.
    total_bytes: usize,
    limit: usize,
}
}

The two protocols

Recommendation: implement sixel first. It is self-contained, has no external dependency (the decoder is ~200 lines), and forces you to solve the placement problem. Kitty's protocol is easier to parse but needs a PNG decoder and has a much larger command surface.

Sixel, decoded

   Each printable character 0x3F..0x7E encodes SIX VERTICAL PIXELS:
       value = byte - 0x3F        (0..63)
       bit 0 = top pixel ... bit 5 = bottom pixel

   Control characters within the payload:
       #<n>            select color n
       #<n>;<t>;<a>;<b>;<c>   define color n (t=2 → RGB, values 0-100)
       !<n><char>      repeat <char> n times (RLE)
       $               carriage return: back to the left of this six-pixel band
       -               newline: advance to the next six-pixel band
       "<a>;<b>;<w>;<h>   raster attributes: aspect ratio and size

   So a sixel image is drawn in horizontal BANDS SIX PIXELS TALL, left to right,
   band by band. That is the whole format.

#![allow(unused)]
fn main() {
/// Streaming sixel decoder. Fed via the parser's put() so a multi-megabyte
/// image never buffers whole.
pub struct SixelDecoder {
    palette: [Rgb; 256],
    current_color: u8,
    /// Position within the image, in PIXELS.
    x: u32,
    band_y: u32,
    /// The decoded image, grown band by band. Bounded by `max_pixels`.
    pixels: Vec<u8>,
    width: u32,
    height: u32,
    max_pixels: u64,
    repeat: u32,
}
}

4. Milestones

Milestone 1 alone is a legitimate, shippable improvement: a terminal that cleanly ignores images is better than one that vomits sixel data onto the screen.

5. The Tests

#![allow(unused)]
fn main() {
#[test]
fn sixel_payload_is_streamed_not_buffered() {
    // A 10 MB sixel must not accumulate in the parser. The DCS hook/put/unhook
    // interface exists for exactly this.
    let mut t = Terminal::new(24, 80, cfg());
    let mut input = b"\x1bPq".to_vec();
    input.extend(std::iter::repeat(b'~').take(10_000_000));
    input.extend_from_slice(b"\x1b\\");
    t.advance(&input);
    assert!(t.parser_buffered_bytes() < 100_000);
}

#[test]
fn image_store_is_bounded() {
    let mut t = Terminal::new(24, 80, TerminalConfig { image_limit: 1 << 20, ..cfg() });
    for _ in 0..100 { t.advance(&one_megabyte_sixel()); }
    assert!(t.image_store().total_bytes() <= 1 << 20);
}

#[test]
fn images_scroll_with_the_text() {
    // Absolute anchoring means this is free. If it is not free, the anchor is
    // in screen coordinates and will drift.
    let mut t = Terminal::new(10, 40, cfg());
    t.advance(&sixel_10x10());
    let before = t.images()[0].anchor;
    for _ in 0..20 { t.advance(b"\n"); }
    assert_eq!(t.images()[0].anchor, before, "the anchor is absolute, so unchanged");
    assert!(t.images()[0].screen_row(&t).is_none(), "but it is now off-screen");
}

#[test]
fn erase_display_removes_images() {
    let mut t = Terminal::new(10, 40, cfg());
    t.advance(&sixel_10x10());
    assert_eq!(t.images().len(), 1);
    t.advance(b"\x1b[2J");
    assert_eq!(t.images().len(), 0);
}

#[test]
fn malformed_sixel_never_panics() {
    // Fuzz-adjacent: images are attacker-controlled input.
    for case in malformed_sixel_corpus() {
        let mut t = Terminal::new(24, 80, cfg());
        t.advance(&case);
    }
}

#[test]
fn terminal_core_still_builds_for_wasm() {
    // The boundary check that matters: adding images must NOT pull an image
    // decoder into the core.
    // CI: cargo build --target wasm32-unknown-unknown -p terminal-core
}
}

That last one is the real test of the project. If it fails, the placement/pixel boundary is wrong.

6. The Measurement

Report all four, including the ones where you lose. "Our decoder is 4× slower than libsixel because it is a straightforward scanline implementation with no SIMD" is a good sentence.

7. Known Traps

Tip: Milestone 1 (parse and discard) takes an afternoon and immediately improves your terminal. Do it first, ship it, then decide whether to continue. That is the correct way to attack every project in this portfolio.

Deliverables

• A streaming sixel decoder, bounded, that never panics on malformed input.
• ImagePlacement in terminal-core with no pixel data and no decoder dependency.
• Rendering in the GUI, correctly positioned and scaled.
• Scroll, erase, and alt-screen interactions correct.
• cargo build --target wasm32-unknown-unknown -p terminal-core still passes.
• The four measurements.
• A written note on what you did not implement (transparency? animation? the kitty protocol?).

Next: Project 2 — Reflow on Resize