Blocking, Non-Blocking, and I/O Multiplexing

Your terminal must watch several things at once: your keyboard, the PTY master, a signal wake-up channel, and later a Unix socket and several more PTYs. This chapter covers blocking vs. non-blocking I/O and the multiplexing mechanisms — select, poll, epoll, kqueue, and async runtimes — with the six-part treatment.

Concept 1: Blocking vs. Non-Blocking I/O

1. What problem it solves

A blocking read() is the simplest possible interface: ask for data, get data. It fails as soon as you have two sources. Blocked on the keyboard, you cannot notice the shell's output; blocked on the shell, you cannot notice a keystroke. Non-blocking I/O turns "wait for data" into "tell me if there is data," which lets one thread serve many fds.

2. Where it exists

Kernel. O_NONBLOCK is a status flag on the open file description, not on the fd.

   BLOCKING                          NON-BLOCKING
   read(fd, buf, n)                  read(fd, buf, n)
     data available? → copy, return    data available? → copy, return n
     none?           → SLEEP           none?           → return -1, errno = EAGAIN

3. Who owns or interacts with it

Your program sets it. Note the sharing hazard: because the flag lives on the open file description, setting O_NONBLOCK on an fd affects every duped copy — including copies a child inherited.

Warning: Never set O_NONBLOCK on the PTY slave before exec. The child (and everything it runs) will get EAGAIN from what it believes is a blocking terminal. Most programs do not handle that and will spin, error out, or lose input. Set O_NONBLOCK on the master only, in your own process, after the fork.

4. Syscalls and errors

fcntl(fd, F_GETFL)                     → current flags
fcntl(fd, F_SETFL, flags | O_NONBLOCK) → make it non-blocking
open(..., O_NONBLOCK)                  → from the start

The errors you must handle, and what each means:

Note: EAGAIN and EWOULDBLOCK are the same value on Linux and macOS, but POSIX permits them to differ. In Rust, io::ErrorKind::WouldBlock covers both — use it rather than comparing raw errno.

Partial reads and writes

This is the bug that hides for weeks. write() returning fewer bytes than you asked for is not an error — it is normal, especially on a non-blocking fd with a nearly-full buffer.

#![allow(unused)]
fn main() {
/// Write ALL of `buf`, handling short writes, EINTR, and EAGAIN.
/// Returns how many bytes were written; the caller must retain the remainder
/// and retry when the fd becomes writable.
fn write_some(fd: i32, buf: &[u8]) -> std::io::Result<usize> {
    let mut written = 0;
    while written < buf.len() {
        let n = unsafe {
            libc::write(fd,
                        buf[written..].as_ptr() as *const libc::c_void,
                        buf.len() - written)
        };
        if n > 0 {
            written += n as usize;
            continue;
        }
        let err = std::io::Error::last_os_error();
        match err.kind() {
            std::io::ErrorKind::Interrupted => continue,        // EINTR: retry
            std::io::ErrorKind::WouldBlock => break,            // EAGAIN: try again later
            _ => return Err(err),
        }
    }
    Ok(written)
}
}

The consequence for your design: every fd you write to needs an output buffer and an interest in writability, not just readability. A terminal that assumes write always completes will drop keystrokes the first time you paste 200 KB into a slow program.

   ┌──────────────────────────────────────────────────────────┐
   │  For each fd you write to:                               │
   │     pending: VecDeque<u8>                                │
   │     if !pending.is_empty():  register interest in WRITE  │
   │     on writable: write_some(); drain what was written    │
   │     if pending becomes empty: DEregister WRITE interest  │
   └──────────────────────────────────────────────────────────┘

That last line matters: if you leave write-interest registered on an idle, always-writable fd, poll returns immediately forever and you burn a core. This is the single most common event-loop bug in existence.

5. Experiment

# Watch a blocking write stall. Terminal A, inside your Lab 2 runner:
yes
# Now stop reading the master (comment out that poll branch and rerun).
# Terminal B (Linux):
cat /proc/<child-pid>/status | grep State      # S (sleeping)
cat /proc/<child-pid>/wchan                    # a tty write wait

And directly, in Rust or Python:

import os, fcntl, pty, time
pid, fd = pty.fork()
if pid == 0:
    os.execvp("bash", ["bash", "--norc"])
fcntl.fcntl(fd, fcntl.F_SETFL, os.O_NONBLOCK)
try:
    print(os.read(fd, 1024))     # likely raises BlockingIOError immediately
except BlockingIOError as e:
    print("EAGAIN — no data yet:", e)

Predict first: with O_NONBLOCK set and nothing typed, does that read return b"", raise, or block? What is the difference between "returns 0 bytes" and "raises EAGAIN", and why does it matter enormously?

6. Failure mode

Concept 2: Multiplexing — poll, select, epoll, kqueue

1. What problem it solves

"Sleep until any of these fds is ready." Without it you either burn CPU polling or need one thread per fd.

2. Where it exists

Kernel, all of them. They differ in interface and in how they scale with the number of fds.

3. Who owns or interacts with it

Your event loop. Exactly one loop per thread; everything else is a callback into it.

4. The four mechanisms

 select(nfds, &readfds, &writefds, &exceptfds, &timeout)
   • POSIX, everywhere, since forever
   • fd_set is a BITMAP limited to FD_SETSIZE (1024 on Linux) — an fd of 1025 is UB
   • O(n) scan; you rebuild the sets on every call
   • Use it only if you must be portable to something ancient. Not in this curriculum.

 poll(&[pollfd], nfds, timeout_ms)
   • POSIX, everywhere, no FD_SETSIZE limit
   • You pass an array; the kernel fills in `revents`
   • O(n) per call, but n is small for a terminal
   • ← THIS is what you use in Lab 3. Simple, portable, sufficient.

 epoll (Linux only)
   • epoll_create1 / epoll_ctl(ADD|MOD|DEL) / epoll_wait
   • The interest set lives in the KERNEL; you do not resend it every call
   • O(ready), not O(watched) — scales to tens of thousands of fds
   • Level-triggered (default) vs. edge-triggered (EPOLLET): with ET you MUST drain
     until EAGAIN or you will never be woken again for that data

 kqueue (macOS / BSD)
   • kqueue() / kevent()
   • Same idea as epoll, plus it can watch signals (EVFILT_SIGNAL), timers (EVFILT_TIMER),
     processes (EVFILT_PROC — including child exit!), and files
   • EV_EOF flag reports the peer closing

Tip: For a terminal with 3–10 fds, poll is not merely adequate — it is correct. epoll's advantage appears in the thousands. Use poll in Section 1, and reach for epoll/kqueue (or mio, which picks for you) in Section 4 when you have one fd per pane plus one per client.

The poll loop, concretely

#![allow(unused)]
fn main() {
use std::io;

const POLL_STDIN: usize = 0;
const POLL_MASTER: usize = 1;
const POLL_SIGNAL: usize = 2;

fn run(stdin: i32, master: i32, sigpipe: i32) -> io::Result<()> {
    let mut fds = [
        libc::pollfd { fd: stdin,   events: libc::POLLIN, revents: 0 },
        libc::pollfd { fd: master,  events: libc::POLLIN, revents: 0 },
        libc::pollfd { fd: sigpipe, events: libc::POLLIN, revents: 0 },
    ];

    loop {
        // -1 = wait indefinitely. Use a timeout only if you have periodic work.
        // SAFETY: fds is a valid array of len 3; poll writes revents.
        let n = unsafe { libc::poll(fds.as_mut_ptr(), fds.len() as libc::nfds_t, -1) };
        if n < 0 {
            let e = io::Error::last_os_error();
            if e.kind() == io::ErrorKind::Interrupted { continue; }  // EINTR
            return Err(e);
        }

        // POLLIN  — readable
        // POLLHUP — the peer hung up. NOTE: you may get POLLIN|POLLHUP together, with
        //           data still buffered. Read first; only treat HUP as EOF once read
        //           returns 0 or EIO. Treating HUP as immediate EOF loses the last output.
        // POLLERR — an error condition; read/write will tell you what
        // POLLNVAL— the fd is not open. A bug in YOUR bookkeeping. Panic in debug.

        if fds[POLL_SIGNAL].revents & libc::POLLIN != 0 { drain_signals()?; }
        if fds[POLL_STDIN].revents & (libc::POLLIN | libc::POLLHUP) != 0 { pump_stdin_to_master()?; }
        if fds[POLL_MASTER].revents & (libc::POLLIN | libc::POLLHUP) != 0 { pump_master_to_stdout()?; }
        if fds[POLL_MASTER].revents & (libc::POLLERR | libc::POLLNVAL) != 0 { break; }
    }
    Ok(())
}
}

The comment about POLLHUP is the load-bearing part. On a PTY master, when the child exits you typically get POLLIN | POLLHUP with data still in the buffer. A loop that treats POLLHUP as "exit now" drops the child's last output — the intermittently-missing-last-line bug from the signals chapter.

Level-triggered vs. edge-triggered

 LEVEL-TRIGGERED (poll, select, epoll default)
   "Tell me whenever there is data."
   Data arrives → wake. You read 10 of 100 bytes → next poll wakes again immediately.
   Forgiving. Slightly more syscalls. USE THIS.

 EDGE-TRIGGERED (EPOLLET, EV_CLEAR)
   "Tell me when the state CHANGES."
   Data arrives → wake ONCE. You read 10 of 100 bytes → no further wake-up until
   MORE data arrives. If none ever does, the 90 bytes sit there forever and your
   terminal appears to hang, then unblocks when the user types.
   Requires: loop reading until EAGAIN, every time. Fewer syscalls. Easy to get wrong.

Warning: Edge-triggered mode is the standard way to write a hanging terminal. If you use it, the rule is absolute: read until EAGAIN, always, no early exit. Not "read until you have enough." Not "read one buffer and render." Until EAGAIN.

5. Experiment

# Watch poll wake up. Linux:
strace -e trace=poll,ppoll,read,write -p <your runner's pid>
# Type a key → poll returns 1, read(0) returns 1 byte, write(master) 1 byte,
#              poll returns 1, read(master) returns the echo, write(1).
# FOUR syscalls per keystroke. That is the round trip, made visible.

# macOS:
sudo dtruss -f -p <pid> 2>&1 | grep -E 'kevent|read|write'

This experiment is the single best way to see the echo round trip described in the mental model. Do it.

6. Failure mode

Concept 3: The Rust Abstractions, and What They Wrap

First the Unix API, then the crate — the rule for this section. Here is what each crate is doing underneath, so you can choose deliberately.

The honest recommendation for this curriculum:

 Lab 1–2   libc only.            You are learning the syscalls.
 Lab 3     libc, then port to nix/rustix + signal-hook and DIFF the two.
 Section 4 mio (or plain poll — one fd per pane is still a small number).
 Async     Only if you want to learn async. It buys a terminal very little:
           the workload is a handful of fds, not ten thousand connections.

Note: The "diff the two" step in Lab 3 is the exercise, not a formality. Write the libc version, get it working, then rewrite with nix, and read the two side by side. Every line that disappeared is a piece of unsafety the crate is handling for you, and you should be able to name which one.

Threads vs. one event loop

You will be tempted to spawn a thread per fd. For a terminal, resist:

The one place a thread genuinely helps is rendering: a render thread consuming RenderSnapshots while the event loop keeps draining the PTY. That is a Section 3 topic, and it works precisely because the boundary between the two is a value, not shared mutable state.

Validation / Self-check

1. Where does O_NONBLOCK live — the fd, the open file description, or the object? Why does that matter after fork?
2. What is the difference between read() returning 0 and read() returning EAGAIN?
3. Why is a short write() not an error, and what data structure must your design have because of it?
4. What is the bug that results from leaving write-interest registered on an idle fd?
5. Compare select, poll, epoll, and kqueue on: platform, scaling, where the interest set lives, and whether they can watch signals.
6. Why does this curriculum use poll in Section 1 rather than epoll?
7. Explain level-triggered vs. edge-triggered, and state the absolute rule for edge-triggered.
8. On a PTY master, why must you not treat POLLHUP as immediate EOF?
9. Name the four syscalls involved in one round trip of a single keystroke, in order.
10. For each of nix, rustix, signal-hook, mio, and tokio: what OS functionality does it wrap, and at what point in this curriculum is it appropriate?
11. Why should the render loop not live inside the "master is readable" branch?

Next: Lab 1 — The Raw Keyboard Byte Inspector.