Lab 9: UTF-8, Wide Characters, and Combining Marks

Background

Your terminal handles ASCII. This lab makes it handle the other 99% of Unicode: multi-byte decoding that survives arbitrary chunking, double-width CJK characters with their spacer cells, combining marks that absorb into the previous cell, and emoji sequences that are seven codepoints and one grapheme.

This is where "works for me" implementations break, because the author's test data was English.

Why This Lab Matters

• A terminal that mishandles width corrupts every program's cursor arithmetic. The failure is not cosmetic — a TUI drawing a box around CJK text will draw the box in the wrong place, forever.
• The invariants you enforce here (spacer pairing, cursor bounds) are the ones a property test can actually check, which makes them the ones that stay correct.

Prerequisites

• Lab 7 complete.
• UTF-8 and Graphemes read.

Predict First

1. printf '日本語\n' in an 80-column terminal. How many columns does the cursor advance?
2. Feed F0 9F 99 (three of the four bytes of 🙂) then stop. What should the screen show?
3. printf 'é' — how many cells?
4. Writing 日 when the cursor is at column 79 of an 80-column grid. What happens?

Step 1: The Incremental Decoder

Write it by hand. It is 60 lines and you will understand every terminal UTF-8 bug afterwards.

#![allow(unused)]
fn main() {
// crates/terminal-protocol/src/utf8.rs

/// Incremental UTF-8 decoder. Must survive being fed one byte at a time,
/// because a read() on a PTY master splits multi-byte characters routinely.
#[derive(Default)]
pub struct Utf8Decoder {
    buf: [u8; 4],
    len: u8,
    needed: u8,
}

pub enum DecodeResult {
    Char(char),
    /// More bytes required. Emit NOTHING — emitting U+FFFD here is the classic
    /// "emoji sometimes shows as two replacement characters" bug.
    Incomplete,
    /// Invalid. Emit one U+FFFD for the maximal subpart. When `resume_with` is
    /// Some, that byte may START a new sequence and must be reprocessed
    /// (the Unicode "best practice" substitution rule).
    Invalid { resume_with: Option<u8> },
}

impl Utf8Decoder {
    pub fn feed(&mut self, byte: u8) -> DecodeResult {
        if self.needed == 0 {
            return match byte {
                0x00..=0x7f => DecodeResult::Char(byte as char),
                // 0x80-0xBF as a LEAD byte is a stray continuation byte.
                0x80..=0xbf => DecodeResult::Invalid { resume_with: None },
                // 0xC0/0xC1 would always be overlong encodings. Reject them:
                // overlongs are how filters get bypassed.
                0xc0..=0xc1 => DecodeResult::Invalid { resume_with: None },
                0xc2..=0xdf => { self.start(byte, 2) }
                0xe0..=0xef => { self.start(byte, 3) }
                0xf0..=0xf4 => { self.start(byte, 4) }
                // 0xF5-0xFF would encode > U+10FFFF.
                0xf5..=0xff => DecodeResult::Invalid { resume_with: None },
            };
        }
        if !(0x80..=0xbf).contains(&byte) {
            // A non-continuation byte inside a sequence: the maximal subpart
            // ends here. Emit U+FFFD and REPROCESS this byte.
            self.reset();
            return DecodeResult::Invalid { resume_with: Some(byte) };
        }
        self.buf[self.len as usize] = byte;
        self.len += 1;
        if self.len < self.needed { return DecodeResult::Incomplete; }

        let s = &self.buf[..self.len as usize];
        let result = match std::str::from_utf8(s) {
            // std::str::from_utf8 already rejects overlongs and surrogates,
            // which is exactly the validation we want.
            Ok(valid) => DecodeResult::Char(valid.chars().next().unwrap()),
            Err(_) => DecodeResult::Invalid { resume_with: None },
        };
        self.reset();
        result
    }
}
}

Step 2: Width and the Print Path

#![allow(unused)]
fn main() {
use unicode_width::UnicodeWidthChar;

/// This terminal's width policy, stated explicitly:
///   • East Asian Wide and Fullwidth  → 2
///   • Combining marks, ZWJ, variation selectors → 0
///   • East Asian AMBIGUOUS → 1 (the Latin interpretation)
///   • Controls → 0 and never printed
/// Programs use their own wcwidth; where it disagrees, their cursor arithmetic
/// breaks. There is no fix, only a documented choice.
pub fn cell_width(c: char) -> u8 {
    match c.width() { Some(w) => w as u8, None => 0 }
}
}

The print path branches three ways — width 0, width 1, width 2 — and each has an edge case. See Lab 7's print_char for the full function; this lab adds the width-0 branch:

#![allow(unused)]
fn main() {
fn absorb_combining(&mut self, c: char) {
    // A combining mark joins the cell to the LEFT of the cursor, unless a
    // pending wrap means the cursor is still logically on the last written cell.
    let (row, col) = if self.cursor.pending_wrap {
        (self.cursor.row, self.cursor.col)          // the cell we just wrote
    } else if self.cursor.col > 0 {
        (self.cursor.row, self.cursor.col - 1)
    } else {
        // No previous cell: the standard says the mark applies to a space.
        // Silently dropping it makes text that begins with a combining mark
        // invisible — a real bug in several terminals.
        self.print_char(' ');
        (self.cursor.row, self.cursor.col - 1)
    };

    // Bound the cluster: an adversarial stream of 10,000 combining marks on one
    // cell must not allocate unboundedly.
    const MAX_COMBINING: usize = 8;
    self.screen.push_combining(row, col, c, MAX_COMBINING);
    self.damage.mark(row);

    // Variation Selector 16 promotes the base character to emoji presentation,
    // which changes its width from 1 to 2 RETROACTIVELY.
    if c == '\u{FE0F}' { self.promote_to_wide(row, col); }
}
}

Warning: promote_to_wide is genuinely awkward — the cell was already written at width 1, and the following cell may already contain a character. The pragmatic policy: promote only if the next cell is blank; otherwise leave the width at 1. Document it. This is exactly the ambiguity mode 2027 exists to remove.

Step 3: The Spacer Invariant, Enforced

Every operation that touches cells must preserve it. Centralize the enforcement:

#![allow(unused)]
fn main() {
impl Line {
    /// The ONLY way to clear a cell. Handles wide/spacer pairing so no caller
    /// can create an orphan.
    pub fn clear_cell(&mut self, col: usize, blank: Cell) {
        if self.cells[col].flags.contains(CellFlags::WIDE) && col + 1 < self.cells.len() {
            self.cells[col + 1] = blank;
        }
        if self.cells[col].flags.contains(CellFlags::SPACER) && col > 0 {
            self.cells[col - 1] = blank;
        }
        self.cells[col] = blank;
    }
}
}

Then audit: erase_in_line, erase_in_display, ech, dch, ich, scroll, resize, and the print path all go through clear_cell or write_cell_preserving_invariant. If any writes cells[i] = x directly, that is the bug you will spend a day on.

Expected Output

$ mini-term run --rows 3 --cols 20 --format debug -- printf '日本語\n'
--- screen ---
 0 | 日本語
--- cells ---
 (0,0) '日' WIDE
 (0,1) ''   SPACER
 (0,2) '本' WIDE
 (0,3) ''   SPACER
 (0,4) '語' WIDE
 (0,5) ''   SPACER
cursor: row=1 col=0

$ mini-term run --rows 3 --cols 20 --format debug -- printf 'e\xcc\x81\n'
 (0,0) 'é' (base 'e' + U+0301)
cursor: row=1 col=0    ← the combining mark did NOT advance the cursor

Debugging Steps

Emoji occasionally render as two �

The decoder emits U+FFFD on Incomplete. Buffer instead.

CJK text overlaps or leaves gaps

Spacer invariant violated. Run the property test; find the operation that writes cells directly.

The cursor is one column off after CJK

Your cell_width disagrees with the program's wcwidth. Check ambiguous-width handling first.

A box drawn around CJK text is misaligned

Same cause, one layer up. Compare against xterm with the same input to see whose width is "standard."

A stream of combining marks eats all memory

No cluster bound. Cap at 8.

Experiment

CLAIM. Terminals disagree about width, and the disagreement is measurable with CSI 6n.

METHOD. Write a small script that prints a test string, queries the cursor, and reports the width the terminal actually used:

width_of() {
  printf '\033[H'          # home
  printf '%s' "$1"
  printf '\033[6n'
  read -r -d R pos
  echo "${1} → ${pos##*;} columns"
}
width_of '日本'
width_of 'é'          # precomposed
width_of $'é'   # decomposed
width_of '👍'
width_of $'\U0001F44D\U0001F3FD'    # thumbs up + skin tone
width_of $'\U0001F468‍\U0001F469‍\U0001F467'   # family
width_of '°'
width_of '┌'

Run it in: your terminal, xterm, tmux inside your terminal, and your own emulator. Tabulate.

PREDICTION. Which of the eight will all four agree on? Which will differ most?

Test

#![allow(unused)]
fn main() {
#[test]
fn incomplete_sequences_buffer_across_chunks() {
    let mut t = Terminal::new(3, 10);
    t.advance(&[0xF0, 0x9F]);
    assert_eq!(t.snapshot_text().trim(), "", "nothing yet — do not emit U+FFFD");
    t.advance(&[0x99, 0x82]);
    assert_eq!(t.screen().row(0).cell(0).grapheme(), "🙂");
}

#[test]
fn invalid_byte_inside_a_sequence_resumes_correctly() {
    // F0 9F then 'A': the maximal subpart is invalid, and 'A' must be printed
    // rather than swallowed.
    let mut t = Terminal::new(3, 10);
    t.advance(&[0xF0, 0x9F, b'A']);
    assert_eq!(t.screen().row(0).cell(0).grapheme(), "\u{FFFD}");
    assert_eq!(t.screen().row(0).cell(1).grapheme(), "A");
}

#[test]
fn overlong_encodings_are_rejected() {
    // C0 80 is an overlong NUL. Accepting overlongs is a security bug.
    let mut t = Terminal::new(3, 10);
    t.advance(&[0xC0, 0x80]);
    assert_eq!(t.screen().row(0).cell(0).grapheme(), "\u{FFFD}");
}

#[test]
fn surrogates_are_rejected() {
    // ED A0 80 encodes U+D800, which is not valid UTF-8.
    let mut t = Terminal::new(3, 10);
    t.advance(&[0xED, 0xA0, 0x80]);
    assert_eq!(t.screen().row(0).cell(0).grapheme(), "\u{FFFD}");
}

#[test]
fn combining_cluster_is_bounded() {
    let mut t = Terminal::new(3, 10);
    let mut input = "a".to_string();
    for _ in 0..10_000 { input.push('\u{0301}'); }
    t.advance(input.as_bytes());
    assert!(t.screen().row(0).cell(0).grapheme().chars().count() <= 9);
}

#[test]
fn wide_char_wraps_rather_than_splitting() {
    let mut t = Terminal::new(3, 5);
    t.advance(b"abcd");
    t.advance("日".as_bytes());
    assert_eq!(t.screen().row(0).cell(4).grapheme(), " ");
    assert_eq!(t.screen().row(1).cell(0).grapheme(), "日");
}

#[test]
fn spacer_invariant_holds_under_random_operations() {
    let mut t = Terminal::new(10, 20);
    for op in random_unicode_ops(20_000) { t.advance(&op); }
    assert_spacer_invariant(&t);
}
}

Challenge Extensions

1. Implement mode 2027 and demonstrate the family-emoji difference with and without it.
2. Make the ambiguous width configurable, with a test that shows box-drawing corruption in the wrong setting.
3. Handle ZWJ sequences: after a ZWJ, the next emoji absorbs into the same cell.
4. Handle regional indicator pairs (flags) as single width-2 graphemes.
5. Build the width-comparison tool from the experiment as a real binary and publish the table for five terminals.
6. Fuzz the decoder against std::str::from_utf8 on complete slices: for any input, your incremental decode must produce the same characters as the batch decode.

Deliverables

• A hand-written incremental UTF-8 decoder rejecting overlongs, surrogates, and out-of-range codepoints.
• Incomplete sequences buffered, never replaced with U+FFFD prematurely.
• The Unicode maximal-subpart resume rule implemented.
• Wide characters with correct spacer pairing, enforced through one function.
• Combining marks absorbing, bounded, with the leading-mark case handled.
• The spacer property test passing over 20,000 random operations.
• A documented width policy, including the ambiguous-width choice.
• The cross-terminal width comparison table from the experiment.

Validation / Self-check

1. Why must an incomplete sequence at a chunk boundary be buffered rather than replaced?
2. What is the maximal-subpart rule, and what does resume_with implement?
3. Why are overlong encodings rejected? What is the security concern?
4. State the spacer invariant and name the single function that must enforce it.
5. What happens to a width-2 character at the last column? Name both valid policies.
6. Why does a combining mark not advance the cursor? What if there is no previous cell?
7. What does U+FE0F do to the width of the preceding character, and why is that awkward?
8. Why must combining clusters be bounded?
9. Your terminal and tmux disagree about °. What is the consequence, and is there a fix?
10. What does mode 2027 solve that the terminal cannot solve alone?

Next: Lab 10 — Scroll Regions and the Alternate Screen.