UTF-8, Widths, and Grapheme Clusters

A terminal is a grid of fixed-width cells, and Unicode is not fixed-width. Reconciling those two facts is one of the genuinely hard parts of a terminal emulator, and it is where "works for me" implementations break on the first CJK filename or family emoji.

Three separate problems, often confused:

1. Decoding — bytes to codepoints.
2. Width — how many cells a codepoint occupies (0, 1, or 2).
3. Clustering — which codepoints belong in the same cell.

Problem 1: Decoding, and Where It Sits

The layering rule

   ✗ WRONG                              ✓ RIGHT
   bytes                                bytes
     │                                    │
     ▼                                    ▼
   UTF-8 decoder                        VT state machine
     │  (chars)                           │
     ▼                                    ├─ Ground state? → UTF-8 decoder → print(char)
   VT state machine                       └─ any other state? → raw bytes
     │
     ▼
   actions

Decoding in front of the parser is wrong, and the reason is precise: escape sequences are defined over bytes, not characters. Put the decoder first and:

• 0x9b (a valid UTF-8 continuation byte) gets decoded as part of a character, or gets replaced with U+FFFD and thereby destroys a legitimate multi-byte character.
• An OSC 52 base64 payload or an OSC 8 URI containing arbitrary bytes is mangled.
• A DCS sixel payload — pure binary — is destroyed.
• An incomplete multi-byte character at the end of a read() forces you to buffer across the parser boundary, which is exactly the coupling you were trying to avoid.

The decoder lives inside the Ground state. Only bytes that would be printed go through it.

The decoder itself

You need an incremental decoder that can be fed one byte at a time and holds partial state, because a read() splits multi-byte characters routinely.

#![allow(unused)]
fn main() {
/// Incremental UTF-8 decoder. Holds partial state across calls, because a read()
/// on a PTY master splits multi-byte characters all the time.
#[derive(Default)]
pub struct Utf8Decoder {
    /// Bytes of the sequence accumulated so far.
    buf: [u8; 4],
    len: u8,
    /// Total bytes expected for the current sequence (0 = not in one).
    needed: u8,
}

pub enum DecodeResult {
    /// A complete character.
    Char(char),
    /// More bytes needed; nothing to emit yet.
    Incomplete,
    /// Invalid sequence. Emit U+FFFD and resynchronize.
    /// `resume_with` is Some(byte) when the offending byte may START a new
    /// sequence and must be reprocessed — the standard "maximal subpart" rule.
    Invalid { resume_with: Option<u8> },
}
}

The rules that matter, per the Unicode standard's "best practice for U+FFFD substitution":

Warning: That last row is the one people get wrong, and it produces the classic bug: "emoji sometimes appears as two replacement characters." The cause is emitting U+FFFD when a read() ends mid-character instead of buffering the partial sequence. Your bytewise replay test catches it immediately — which is precisely why that test mode exists.

#![allow(unused)]
fn main() {
#[test]
fn split_multibyte_character_is_buffered_not_replaced() {
    // "🙂" is F0 9F 99 82. Feeding it in two chunks must produce ONE char,
    // not two U+FFFD. This is the single most common UTF-8 bug in terminals.
    let mut d = Utf8Decoder::default();
    assert!(matches!(d.feed(0xF0), DecodeResult::Incomplete));
    assert!(matches!(d.feed(0x9F), DecodeResult::Incomplete));
    assert!(matches!(d.feed(0x99), DecodeResult::Incomplete));
    assert!(matches!(d.feed(0x82), DecodeResult::Char('\u{1F642}')));
}
}

Tip: Rust's std::str::from_utf8 is not directly usable here because it works on complete slices. core::str::Utf8Error::valid_up_to() plus error_len() gives you enough to build the incremental version, and writing it yourself is a 60-line exercise worth doing once. The utf8parse crate (used by vte) is the production answer.

Problem 2: Width

Every codepoint occupies 0, 1, or 2 cells. The classification comes from the Unicode East Asian Width property plus a handful of category rules.

The ambiguous-width problem

  U+00B0 DEGREE SIGN  °
    In a Latin context:      1 cell
    In a CJK context:        2 cells (legacy CJK fonts made it fullwidth)

  There is NO correct answer. Every terminal makes a choice:
    - xterm:  configurable (`-cjk_width`)
    - most modern terminals: 1 cell (Latin interpretation)
    - some CJK-locale setups: 2 cells

The consequence is real: if your terminal says 1 and the program (via its own wcwidth) says 2, the program's cursor arithmetic is wrong and its display corrupts. This is why tmux inside a terminal inside ssh sometimes garbles box-drawing characters — three layers, potentially three different width tables.

Decision for this curriculum: ambiguous = 1. Document it. Make it configurable later.

Implementation

#![allow(unused)]
fn main() {
use unicode_width::UnicodeWidthChar;

/// Cell width of a character, as this terminal defines it.
///
/// Policy decisions, documented deliberately:
///   • East Asian Ambiguous → 1 (the Latin interpretation)
///   • Controls → 0 here; they never reach the grid as text
///   • unicode-width returns None for controls, Some(0/1/2) otherwise
fn cell_width(c: char) -> u8 {
    match c.width() {
        Some(w) => w as u8,     // 0, 1, or 2
        None => 0,              // a control character; the caller must not print it
    }
}
}

Warning: unicode-width implements the Unicode rules, which differ subtly from the wcwidth in your system's libc, which differs between glibc versions, which differs from what a given program computed. There is no single truth. What matters is that you are consistent and that you can explain your choice. Differential testing against a real terminal will surface the differences; log them rather than chasing them all.

Wide characters in the grid: the spacer invariant

A width-2 character occupies two cells. The second cell is a spacer — it holds no character of its own and exists so that column arithmetic works.

   Writing "日本" at column 0 in an 8-column grid:

   col:    0      1      2      3      4      5      6      7
        ┌──────┬──────┬──────┬──────┬──────┬──────┬──────┬──────┐
        │  日  │spacer│  本  │spacer│      │      │      │      │
        └──────┴──────┴──────┴──────┴──────┴──────┴──────┴──────┘
   cursor after writing: col 4

The invariant, which you must enforce and property-test:

A wide cell is always immediately followed by exactly one spacer cell, and a spacer cell is always immediately preceded by a wide cell.

Every operation must preserve it:

#![allow(unused)]
fn main() {
#[test]
fn overwriting_half_a_wide_char_clears_both_cells() {
    // Writing 'X' over the first half of 日 must not leave an orphan spacer,
    // which would render as a phantom blank and break column arithmetic.
    let mut t = Terminal::new(3, 10);
    t.advance("日本".as_bytes());
    t.advance(b"\x1b[1;1H");     // cursor home
    t.advance(b"X");
    let row = t.screen().row(0);
    assert_eq!(row.cell(0).grapheme(), "X");
    assert_eq!(row.cell(1).grapheme(), " ", "the orphaned spacer must become a blank");
    assert!(!row.cell(1).is_spacer(), "no orphan spacers may survive");
}
}

Problem 3: Grapheme Clusters

A "character" as a user perceives it may be several codepoints:

   é           U+00E9                          1 codepoint,  1 cell   (precomposed)
   é           U+0065 U+0301  (e + ◌́)          2 codepoints, 1 cell   (decomposed)
   👍🏽          U+1F44D U+1F3FD                  2 codepoints, 2 cells  (+ skin tone)
   👨‍👩‍👧‍👦     U+1F468 ZWJ U+1F469 ZWJ U+1F467
               ZWJ U+1F466                      7 codepoints, 2 cells  (family, ZWJ)
   🇯🇵          U+1F1EF U+1F1F5                  2 codepoints, 2 cells  (flag = 2 regional indicators)
   ❤️          U+2764 U+FE0F                    2 codepoints, ? cells  (VS16 → emoji presentation)

The terminal's approach: absorption

A terminal does not run a full grapheme-segmentation algorithm on a buffer — it receives characters one at a time and must decide, for each, "does this join the previous cell or start a new one?"

#![allow(unused)]
fn main() {
fn print(&mut self, c: char) {
    let w = cell_width(c);

    if w == 0 {
        // Zero-width: a combining mark, ZWJ, or variation selector.
        // It ABSORBS into the previous cell rather than occupying its own.
        if let Some(prev) = self.previous_written_cell_mut() {
            prev.push_combining(c);
            return;
        }
        // No previous cell (start of line, or after an erase): the standard says
        // the mark applies to a space. Insert one and absorb into it.
        // Silently dropping it here is a real bug — text that starts with a
        // combining mark becomes invisible.
        self.write_cell(' ');
        self.previous_written_cell_mut().unwrap().push_combining(c);
        return;
    }

    // Width 1 or 2: a new cell.
    self.write_grapheme(c, w);
}
}

Where it gets hard:

Note: There is no fully correct answer here — the terminal protocol has no way for a program to say "this is one grapheme." The mode 2027 proposal (grapheme clustering) exists exactly to fix this: a program sets CSI ?2027h to say "I will send graphemes and I compute widths the way you do." It is supported by Ghostty, contour, and others, and it is the right long-term answer. Support it if you can; at minimum, know why it exists.

Recommended scope for this curriculum: implement combining-mark absorption and variation selectors. Handle ZWJ sequences by absorbing after a ZWJ. Treat regional indicator pairs as a special case if you have the appetite. Document exactly what you support — and, more importantly, what you do not.

The Cell Representation Problem

#![allow(unused)]
fn main() {
pub struct Cell {
    grapheme: String,   // ← the naïve version. A heap allocation PER CELL.
    style: Style,
    width: CellWidth,
}
}

An 80×24 grid is 1,920 Strings = 1,920 allocations, and clear() re-allocates all of them. At 400×100 with scrollback it is catastrophic. Real terminals use one of:

#![allow(unused)]
fn main() {
/// One screen cell. Deliberately Copy and small: no heap allocation on the
/// common path, so clearing a screen is a memset rather than 1,920 frees.
#[derive(Copy, Clone, PartialEq, Eq)]
pub struct Cell {
    /// The base character. ' ' for an empty cell, '\0' for a wide-char spacer.
    ch: char,                    // 4 bytes
    /// Packed attributes + colors.
    style: Style,                // 8-12 bytes
    /// Flags: IS_WIDE, IS_SPACER, HAS_COMBINING, WRAPPED.
    flags: CellFlags,            // 1 byte
    /// Index into the grid's combining-character table, when HAS_COMBINING is set.
    combining: u16,              // 2 bytes  (0 = none)
}
}

Measure before you optimize, but know where you are going. A benchmark that clears and refills a 400×100 grid 1,000 times will tell you immediately which representation you have.

Experiment

CLAIM. Terminals disagree about width, and the disagreement is directly visible.

METHOD.

# 1. Wide characters and column arithmetic.
printf '日本語\n'
printf '12345678\n'      # compare alignment: 日本語 should span 6 columns

# 2. Combining marks — the same visual result from different byte sequences.
printf 'é vs é\n'          # e+combining acute  vs  precomposed é
printf 'é́́ stacked\n'

# 3. Emoji and ZWJ.
printf '\U0001F44D \U0001F44D\U0001F3FD\n'          # thumbs up, and with skin tone
printf '\U0001F468‍\U0001F469‍\U0001F467\n' # family via ZWJ
printf '\U0001F1EF\U0001F1F5\n'                      # flag

# 4. The ambiguous-width problem.
printf '°±α\n'
printf '123\n'           # do they align? depends on your terminal's policy

# 5. Box drawing after CJK — the classic corruption case.
printf '┌────┐\n│日本│\n└────┘\n'

# 6. Cursor arithmetic. Write CJK then query the cursor position:
printf '日本\033[6n'; read -r -d R pos; echo "cursor: ${pos#*[}"
#    Expect column 5 (1-based) if 日本 is 4 cells wide.

Run all six in: your terminal, xterm, tmux inside your terminal, and (once it exists) your own emulator. Tabulate the differences.

PREDICTION. Before running: how many columns does 日本語 occupy? How many cells does the family emoji occupy? Where is the cursor after printf '日本'?

Test

#![allow(unused)]
fn main() {
#[test]
fn wide_characters_occupy_two_cells_with_a_spacer() {
    let mut t = Terminal::new(3, 10);
    t.advance("日本".as_bytes());
    let row = t.screen().row(0);
    assert_eq!(row.cell(0).grapheme(), "日");
    assert!(row.cell(0).is_wide());
    assert!(row.cell(1).is_spacer(), "cell 1 must be the spacer for 日");
    assert_eq!(row.cell(2).grapheme(), "本");
    assert!(row.cell(3).is_spacer());
    assert_eq!(t.cursor().col, 4, "two wide chars advance the cursor by four");
}

#[test]
fn combining_marks_absorb_into_the_previous_cell() {
    // "e" + U+0301 must be ONE cell, not two.
    let mut t = Terminal::new(3, 10);
    t.advance("e\u{0301}".as_bytes());
    assert_eq!(t.screen().row(0).cell(0).grapheme(), "e\u{0301}");
    assert_eq!(t.cursor().col, 1, "a combining mark must not advance the cursor");
}

#[test]
fn a_leading_combining_mark_gets_a_base_space() {
    // A combining mark with no preceding cell applies to a space. Dropping it
    // silently makes such text invisible.
    let mut t = Terminal::new(3, 10);
    t.advance("\u{0301}".as_bytes());
    assert_eq!(t.screen().row(0).cell(0).grapheme(), " \u{0301}");
}

#[test]
fn wide_char_at_the_last_column_wraps() {
    // A width-2 character cannot be split. In a 5-column grid with the cursor at
    // column 4, it must wrap to the next line, leaving column 4 blank.
    let mut t = Terminal::new(3, 5);
    t.advance(b"abcd");            // cursor at col 4
    t.advance("日".as_bytes());
    assert_eq!(t.screen().row(0).cell(4).grapheme(), " ", "last cell stays blank");
    assert_eq!(t.screen().row(1).cell(0).grapheme(), "日");
    assert!(t.screen().row(1).cell(1).is_spacer());
}

#[test]
fn no_orphan_spacers_survive_any_operation() {
    // Property test: after an arbitrary sequence of writes, erases, and scrolls,
    // every spacer is preceded by a wide cell and every wide cell is followed by
    // a spacer. This invariant is what makes column arithmetic sound.
    let mut t = Terminal::new(10, 20);
    for op in random_ops(5_000) { t.advance(&op); }
    for row in t.screen().rows() {
        for (i, cell) in row.cells().enumerate() {
            if cell.is_spacer() {
                assert!(i > 0 && row.cell(i - 1).is_wide(),
                        "orphan spacer at column {i}");
            }
            if cell.is_wide() {
                assert!(i + 1 < row.len() && row.cell(i + 1).is_spacer(),
                        "wide cell without a spacer at column {i}");
            }
        }
    }
}
}

Challenge Extensions

1. Write the incremental UTF-8 decoder by hand, without str::from_utf8, following the Unicode "maximal subpart" U+FFFD rule. Test it against the standard's conformance examples for invalid sequences.

2. Implement mode 2027 (grapheme clustering). When set, use unicode-segmentation for real cluster boundaries and report widths accordingly. Compare the family-emoji behavior with and without.

3. Make the ambiguous width configurable and write a test that shows box-drawing corruption in the wrong setting. This is how you prove the setting matters rather than asserting it.

4. Benchmark cell representations. Implement String, char + overflow, and inline-[u8; 8]. Benchmark clear-and-refill on a 400×100 grid. Report allocation counts and time. Let the numbers pick.

5. Build a width-comparison tool that prints a test string, queries the cursor with CSI 6n, and reports what the terminal actually believed the width was. Run it against five terminals and publish the table. This is genuinely useful to the ecosystem.

Validation / Self-check

1. Why must the UTF-8 decoder live inside the parser's Ground state rather than in front of it? Give two concrete corruptions caused by the wrong order.
2. What must a decoder do with an incomplete sequence at the end of a read(), and what is the bug if it does the other thing?
3. Name the three width classes and give an example of each.
4. What is East Asian Ambiguous width, why is there no correct answer, and what did you choose?
5. State the spacer invariant, and list four operations that can violate it.
6. What happens when a width-2 character is written at the last column? Name two valid policies.
7. Why does a combining mark not advance the cursor, and what must happen if one arrives with no preceding cell?
8. Why is Cell { grapheme: String } a performance problem? Give the allocation count for an 80×24 clear.
9. What does mode 2027 solve, and why can the terminal not solve it alone?
10. A user reports that box-drawing characters garble after CJK text inside tmux over ssh. Name the three layers involved and the likely disagreement.

Next: The Screen Model.