The Client/Server Protocol

The client and server are separate processes talking over a Unix domain socket. This chapter designs that protocol: framing, the message catalog, versioning, backpressure, and the JSON-versus-binary trade-off.

Start with framed JSON. It is debuggable with cat, extensible without ceremony, and fast enough. Move to binary only with a measurement in hand — and the measurement is in the challenges.

Transport: The Unix Domain Socket

#![allow(unused)]
fn main() {
/// Socket path, with the security properties spelled out.
fn socket_path() -> PathBuf {
    // A PER-USER directory, not a shared /tmp path. A world-writable socket
    // would let any local user attach to your shells — which is a shell, on
    // your account, with your credentials. This is the single most important
    // security decision in the whole section.
    let uid = unsafe { libc::getuid() };
    let dir = std::env::var("XDG_RUNTIME_DIR")
        .map(PathBuf::from)
        .unwrap_or_else(|_| PathBuf::from(format!("/tmp/mini-mux-{uid}")));
    dir.join("default")
}

fn create_socket_dir(dir: &Path) -> io::Result<()> {
    std::fs::create_dir_all(dir)?;
    // 0700: owner only. Check it even if we just created it — an attacker may
    // have pre-created the directory with looser permissions.
    std::fs::set_permissions(dir, std::fs::Permissions::from_mode(0o700))?;
    let meta = std::fs::metadata(dir)?;
    if meta.uid() != unsafe { libc::getuid() } {
        return Err(io::Error::new(io::ErrorKind::PermissionDenied,
                                  "socket directory is owned by another user"));
    }
    Ok(())
}
}

Why a Unix socket rather than a TCP socket or a named pipe:

SCM_RIGHTS is worth knowing about even if you do not use it: it lets the server hand a PTY master fd directly to the client, so the client reads the PTY without the server relaying bytes. That is a real design option, and it is what some multiplexers do for performance. It also breaks the "server owns the state" model, so it is a trade-off, not an upgrade.

Stale socket handling:

#![allow(unused)]
fn main() {
fn bind_or_takeover(path: &Path) -> io::Result<UnixListener> {
    match UnixListener::bind(path) {
        Ok(l) => Ok(l),
        Err(e) if e.kind() == io::ErrorKind::AddrInUse => {
            // The file exists. Is a server actually alive behind it?
            match UnixStream::connect(path) {
                Ok(_) => Err(io::Error::new(io::ErrorKind::AddrInUse, "server already running")),
                // ECONNREFUSED: the socket file is a corpse from a crashed server.
                Err(_) => { std::fs::remove_file(path)?; UnixListener::bind(path) }
            }
        }
        Err(e) => Err(e),
    }
}
}

Framing

A stream socket has no message boundaries. You must impose them.

   ┌────────────┬──────────────────────────────────┐
   │ length: u32│  payload (JSON, `length` bytes)  │
   │ big-endian │                                  │
   └────────────┴──────────────────────────────────┘

   Rules:
     • length is the PAYLOAD length, not including the 4-byte header.
     • MAX_FRAME = 16 MB. A frame claiming 4 GB is a hostile or corrupt peer:
       close the connection rather than allocating.
     • Read exactly 4 bytes, then exactly `length` bytes. Never assume one
       read() gives you a whole frame — this is the same split-input problem
       as the parser, one layer up.

#![allow(unused)]
fn main() {
const MAX_FRAME: u32 = 16 * 1024 * 1024;

/// Incremental frame decoder. Fed arbitrary chunks, yields whole frames.
/// Exactly the same shape as the VT parser: partial state across calls.
pub struct FrameDecoder {
    buf: Vec<u8>,
    need: Option<usize>,
}

impl FrameDecoder {
    pub fn feed(&mut self, bytes: &[u8]) -> Result<Vec<Vec<u8>>, ProtocolError> {
        self.buf.extend_from_slice(bytes);
        let mut out = Vec::new();
        loop {
            let need = match self.need {
                Some(n) => n,
                None => {
                    if self.buf.len() < 4 { break; }
                    let n = u32::from_be_bytes(self.buf[..4].try_into().unwrap());
                    if n > MAX_FRAME { return Err(ProtocolError::FrameTooLarge(n)); }
                    self.buf.drain(..4);
                    self.need = Some(n as usize);
                    n as usize
                }
            };
            if self.buf.len() < need { break; }
            out.push(self.buf.drain(..need).collect());
            self.need = None;
        }
        Ok(out)
    }
}
}

The Message Catalog

Two enums: what the client asks, and what the server reports.

#![allow(unused)]
fn main() {
#[derive(Serialize, Deserialize, Debug, PartialEq)]
#[serde(tag = "type")]
pub enum Request {
    // ── Connection lifecycle ────────────────────────────────────────────
    /// Always the first message. Carries the protocol version so a mismatched
    /// client gets a clear error instead of confusing behavior.
    Hello { protocol_version: u32, client_name: String, term: String,
            size: Size, capabilities: Vec<String> },
    Attach { session: Option<String> },      // None = the most recent
    Detach,
    Goodbye,

    // ── Input ───────────────────────────────────────────────────────────
    /// Raw bytes destined for the focused pane's PTY. The client has ALREADY
    /// stripped prefix commands; anything here is for the program.
    Input { bytes: ByteBuf },
    /// A paste, kept separate so the SERVER can apply bracketed-paste rules
    /// using the target pane's modes — which the client does not know.
    Paste { text: String },

    // ── Sizing ──────────────────────────────────────────────────────────
    ClientResize { size: Size },

    // ── Session / window / pane management ──────────────────────────────
    NewSession { name: Option<String>, command: Option<Vec<String>> },
    NewWindow { session: String, name: Option<String> },
    SplitPane { pane: PaneId, direction: SplitDirection, ratio: f32 },
    ClosePane { pane: PaneId },
    FocusPane { pane: PaneId },
    FocusDirection { direction: Direction },
    ResizePane { pane: PaneId, delta: i32, direction: Direction },
    SelectWindow { session: String, index: usize },
    RenameSession { session: String, name: String },
    ListSessions,

    // ── Copy mode / scrollback ──────────────────────────────────────────
    ScrollPane { pane: PaneId, lines: i32 },
    RequestScrollback { pane: PaneId, from: u64, count: usize },
}

#[derive(Serialize, Deserialize, Debug, PartialEq)]
#[serde(tag = "type")]
pub enum Event {
    /// The reply to Hello. If versions are incompatible, the server says so
    /// HERE and closes, rather than failing mysteriously three messages later.
    Welcome { protocol_version: u32, server_version: String, sessions: Vec<SessionInfo> },
    ProtocolMismatch { server_version: u32, client_version: u32 },

    /// A full repaint of a pane, from STATE. Sent on attach, on resize, and
    /// whenever a client is too far behind to update incrementally.
    PaneFull { pane: PaneId, size: Size, screen: ScreenSnapshot,
               cursor: CursorInfo, modes: ModeFlags },
    /// An incremental update: only the rows that changed.
    PaneRows { pane: PaneId, rows: Vec<(usize, RowSnapshot)>, cursor: CursorInfo },
    /// The pane's terminal modes changed (alt screen, mouse, cursor visibility).
    /// The client needs these to encode input and to render the cursor.
    PaneModes { pane: PaneId, modes: ModeFlags },
    PaneTitle { pane: PaneId, title: String },
    PaneBell { pane: PaneId },
    PaneExited { pane: PaneId, status: i32 },

    LayoutChanged { session: String, window: usize, layout: LayoutSnapshot },
    FocusChanged { pane: PaneId },
    SessionList { sessions: Vec<SessionInfo> },

    /// The server is asking the client to do something only it can do:
    /// set the OS clipboard, or set the outer terminal's title.
    SetClipboard { selection: String, text: String },
    SetTitle { title: String },

    Error { message: String },
    Detached { reason: String },
}
}

Why Paste is separate from Input

The client does not know whether the target pane has bracketed paste enabled — that is the pane's Terminal state, which lives in the server. Sending the raw text and letting the server apply encode_paste with the correct modes is the only way to get it right. This is a small design decision that prevents a real class of bug.

Why PaneModes exists

The client must encode arrow keys, and that depends on DECCKM — a pane-side mode. The client therefore needs a subset of the pane's mode flags mirrored to it. Ship only what the client actually needs: APP_CURSOR_KEYS, BRACKETED_PASTE, the mouse modes, CURSOR_VISIBLE, and ALT_SCREEN.

Screen Updates: Full vs. Incremental

#![allow(unused)]
fn main() {
#[derive(Serialize, Deserialize)]
pub struct ScreenSnapshot {
    pub rows: Vec<RowSnapshot>,
}

#[derive(Serialize, Deserialize)]
pub struct RowSnapshot {
    /// Runs of identically-styled cells. Far more compact than per-cell data,
    /// and it matches what a renderer wants anyway.
    pub runs: Vec<StyledRun>,
    pub wrapped: bool,
}
}

The policy:

That fourth rule is the backpressure valve, and it is why full repaint must be cheap: it is the recovery path, not just the attach path.

Note: Sending screen state rather than raw escape sequences is the design that makes multi-client work. Two clients of different sizes, or with different capabilities, each get a rendering appropriate to them. tmux does the opposite — it re-emits escape sequences into the client's terminal — which is why tmux must know the client's terminfo and why it re-implements so much. Sending state is the cleaner design; sending sequences is the compatible one. Know that you chose.

JSON First, Binary Later

Start with JSON. The reasons are pedagogical and practical:

Then measure. The honest numbers to collect:

   Benchmark: `yes` in one pane, one attached client, 60 updates/second.
     JSON:   bytes/sec on the socket, CPU% in the server, CPU% in the client
     Binary: the same three

   Decide with those numbers. A terminal's update rate is bounded by the frame
   rate, not by the producer — so JSON is very often fast enough, and the
   engineer who says "obviously binary" without measuring is guessing.

When binary is justified, the design:

   ┌────────┬────────┬──────────────────────────────┐
   │ len u32│ tag u8 │  payload (bincode/postcard)  │
   └────────┴────────┴──────────────────────────────┘

   • A tag byte, so a decoder can skip an unknown message type instead of
     desynchronizing — the single most important property.
   • Little-endian fixed-width fields; varints for lengths.
   • Cells packed: (char u32, style u32, flags u8) or run-length encoded.
   • Keep the JSON codec behind a feature flag FOREVER, for debugging.

Versioning

#![allow(unused)]
fn main() {
pub const PROTOCOL_VERSION: u32 = 1;

// Rules:
//   • Bump on any BREAKING change.
//   • Adding an optional field with #[serde(default)] is NOT breaking.
//   • Adding a new enum variant IS breaking for a decoder that errors on
//     unknown variants — so decode unknown variants as `Unknown` and ignore
//     them. That one decision buys you forward compatibility for free.
//   • Hello/Welcome exchange versions FIRST, before anything else.
}

#![allow(unused)]
fn main() {
#[derive(Deserialize)]
#[serde(tag = "type")]
enum Request {
    Input { bytes: ByteBuf },
    // ...
    /// Forward compatibility: an unknown message from a newer client is ignored
    /// rather than fatal. Without this, every new message type breaks old peers.
    #[serde(other)]
    Unknown,
}
}

Backpressure and Client Buffers

#![allow(unused)]
fn main() {
struct ClientConn {
    stream: UnixStream,
    decoder: FrameDecoder,
    /// Bounded. A slow or stalled client must not make the server allocate
    /// without limit — that is a trivial local denial of service.
    out: VecDeque<u8>,
    attached: Option<SessionId>,
    size: Size,
    /// Set when we dropped updates; the next send must be a PaneFull.
    needs_full_repaint: bool,
}

const MAX_CLIENT_BUFFER: usize = 4 * 1024 * 1024;

impl ClientConn {
    fn queue(&mut self, frame: Vec<u8>) {
        if self.out.len() + frame.len() > MAX_CLIENT_BUFFER {
            // Drop the pending queue and mark for a full repaint. Sending a
            // fresh snapshot is both smaller and MORE CORRECT than a backlog of
            // stale increments.
            self.out.clear();
            self.needs_full_repaint = true;
            return;
        }
        self.out.extend(&(frame.len() as u32).to_be_bytes());
        self.out.extend(frame);
    }
}
}

Writes to clients must be non-blocking, with POLLOUT interest managed dynamically — exactly the pattern from Lab 3. A blocking write to one stalled client would freeze every pane in every session.

Experiment

CLAIM. A framed JSON protocol is directly inspectable, and that is worth real money in debugging.

METHOD.

# 1. Start the server and attach a client.
mini-mux server & sleep 0.5
mini-mux attach &

# 2. Watch the traffic. socat can sit in the middle:
mv /tmp/mini-mux-$(id -u)/default /tmp/mini-mux-$(id -u)/real
socat -v UNIX-LISTEN:/tmp/mini-mux-$(id -u)/default,fork \
         UNIX-CONNECT:/tmp/mini-mux-$(id -u)/real 2>&1 | tee /tmp/proto.log

# 3. Or add a --log-protocol flag to your server and read it:
mini-mux server --log-protocol /tmp/proto.jsonl
jq -c 'select(.type=="PaneRows") | {pane, rows: (.rows|length)}' /tmp/proto.jsonl | head

# 4. Measure. Run `yes` in a pane for 10 seconds:
wc -c /tmp/proto.jsonl
jq -s 'length' /tmp/proto.jsonl

PREDICTION. Before measuring: how many bytes per second does one yes pane generate on the socket at 60 updates/second in an 80×24 pane? Compare with the raw PTY output rate. Which is larger, and why?

RESULT. Record both numbers. The comparison — screen updates versus raw bytes — is the argument for state-based updates in one measurement.

Test

#![allow(unused)]
fn main() {
#[test]
fn frames_survive_arbitrary_chunking() {
    // The same split-input property as the VT parser, one layer up. A socket
    // read() gives you whatever the kernel has, not whole messages.
    let msgs = vec![req_hello(), req_input(b"ls\n"), req_detach()];
    let encoded: Vec<u8> = msgs.iter().flat_map(encode_frame).collect();
    for chunk in [1usize, 3, 7, 64, 4096] {
        let mut dec = FrameDecoder::default();
        let mut got = Vec::new();
        for c in encoded.chunks(chunk) {
            got.extend(dec.feed(c).unwrap().into_iter().map(decode_frame));
        }
        assert_eq!(got, msgs, "chunk size {chunk}");
    }
}

#[test]
fn oversized_frame_is_rejected_not_allocated() {
    let mut dec = FrameDecoder::default();
    let hostile = u32::MAX.to_be_bytes();
    assert!(matches!(dec.feed(&hostile), Err(ProtocolError::FrameTooLarge(_))));
}

#[test]
fn unknown_message_types_are_ignored_not_fatal() {
    // Forward compatibility: a newer client must not break an older server.
    let json = br#"{"type":"SomeFutureThing","x":1}"#;
    let decoded: Request = serde_json::from_slice(json).unwrap();
    assert_eq!(decoded, Request::Unknown);
}

#[test]
fn all_messages_round_trip() {
    for msg in every_request_variant() {
        assert_eq!(decode_frame(&encode_frame(&msg)), msg);
    }
    for ev in every_event_variant() {
        assert_eq!(decode_event(&encode_event(&ev)), ev);
    }
}

#[test]
fn protocol_mismatch_is_reported_clearly() {
    let server = MuxServer::spawn_daemon().unwrap();
    let resp = raw_hello(&server.socket_path(), PROTOCOL_VERSION + 100);
    assert!(matches!(resp, Event::ProtocolMismatch { .. }),
            "a version mismatch must be diagnosed, not silently misbehave");
}

#[test]
fn slow_client_gets_dropped_updates_and_a_full_repaint() {
    let mut conn = ClientConn::test();
    for _ in 0..100_000 { conn.queue(vec![0u8; 1024]); }
    assert!(conn.out.len() <= MAX_CLIENT_BUFFER);
    assert!(conn.needs_full_repaint);
}

#[test]
fn socket_directory_permissions_are_enforced() {
    let dir = tempdir();
    std::fs::set_permissions(&dir, Permissions::from_mode(0o777)).unwrap();
    // Binding must either fix the permissions or refuse — never proceed with a
    // world-writable socket directory.
    let r = create_socket_dir(dir.path());
    let mode = std::fs::metadata(dir.path()).unwrap().permissions().mode() & 0o777;
    assert!(r.is_err() || mode == 0o700);
}
}

Challenge Extensions

1. Implement a binary codec behind a feature flag and benchmark against JSON: socket bytes/sec, server CPU, client CPU, under a yes flood. Publish the numbers and say which you would ship.
2. Pass a PTY master fd over SCM_RIGHTS so the client reads the PTY directly. Then explain what you lose (state authority, multi-client, detach correctness) and decide whether it is worth it.
3. Add SO_PEERCRED verification so the server refuses connections from other UIDs even if the filesystem permissions were wrong.
4. A tmux-style control mode (-CC): a line-oriented text protocol for programmatic control, so an editor can embed your mux.
5. Protocol fuzzing: feed random bytes to the server's socket and assert it never panics, never allocates unboundedly, and always closes bad connections.
6. Compression for PaneFull messages, measured. Screen snapshots have very high redundancy.

Validation / Self-check

1. Why a Unix domain socket rather than TCP? Name three properties.
2. What does SCM_RIGHTS allow, and what would you lose by using it?
3. Why does a stream socket need framing, and what is the maximum-frame rule for?
4. Why is Paste a separate message from Input?
5. Why does the client need PaneModes? Which flags, specifically?
6. When do you send PaneFull rather than PaneRows? Name all four cases.
7. Why does sending screen state rather than escape sequences make multi-client easier? What does tmux do instead, and what does that cost it?
8. What makes a protocol change breaking, and which two decisions buy forward compatibility for free?
9. Why must writes to clients be non-blocking with a bounded buffer?
10. What did your JSON-vs-binary measurement show, and what would you ship?
11. What are the socket directory's required permissions and ownership check, and what is the attack?

Next: Panes, Windows, and Layout.