Lab 2: The PTY Shell Runner (Milestone 2)

Background

In Lab 1 you borrowed someone else's terminal. Now you make your own.

This lab writes the code that every terminal emulator, every ssh server, script, expect, and tmux contains: allocate a PTY pair, fork, configure the child's session and controlling terminal, dup2 the slave onto its standard streams, and exec a shell. Seven syscalls, in a specific order, each with a specific failure mode.

You will write it with libc only. No portable-pty, no forkpty. forkpty() would do the whole job in one line — and hide exactly the four things this lab exists to teach.

Why This Lab Matters

This is Learning Priority #1: understand exactly what happens when a terminal starts a shell.
Every "job control does not work" bug in every terminal-adjacent tool is one of the four mistakes in this lab.
The terminal-pty crate you write here is used unchanged by the GUI in Section 3 and by the mux server in Section 4.

Prerequisites

Lab 1 complete; you have a working RawMode guard to reuse.
TTY and PTY, File Descriptors, fork & exec, and Sessions, Process Groups & the Controlling Terminal read.

Predict First

Write these down before you start:

If you forget setsid(), what is the first thing you will notice?
If you call TIOCSCTTY before setsid(), what error do you get?
If the parent forgets to close(slave), what happens when the shell exits?
If you never call TIOCSWINSZ, what does stty size print inside the child?
If you forget to set TERM, what does vim do?

The Target Architecture

 PARENT (your program)                                CHILD (bash)
 ─────────────────────                                ────────────
  fd 0 ─── your real terminal (RAW mode)
  fd 1 ─── your real terminal
  master ── PTY master  ◀──────────┐
                                    │
      relay loop:                   │  ═══ KERNEL ═══
        read(0)  → write(master) ───┤    PTY pair
        read(master) → write(1) ◀───┤    + line discipline
                                    │
                                    └─▶ /dev/pts/N ─── fd 0,1,2 of bash
                                        (bash's controlling terminal)

Step-by-Step Tasks

Step 1: Create the crate

cd mini-terminal
cargo new --lib crates/terminal-pty --name terminal-pty
cargo new --bin crates/pty-runner  --name pty-runner

# crates/terminal-pty/Cargo.toml
[dependencies]
libc = "0.2"

# crates/pty-runner/Cargo.toml
[dependencies]
libc = "0.2"
terminal-pty = { path = "../terminal-pty" }

Step 2: Allocate the PTY pair

crates/terminal-pty/src/lib.rs:

#![allow(unused)]
fn main() {
use std::ffi::{CStr, CString, OsStr};
use std::io;
use std::os::fd::{AsRawFd, BorrowedFd, FromRawFd, OwnedFd, RawFd};

/// A PTY master paired with a child process running on the slave.
pub struct Pty {
    master: OwnedFd,
    child: libc::pid_t,
    exited: Option<i32>,
}

/// Rows/columns/pixels, exactly `struct winsize`.
#[derive(Copy, Clone, Debug, PartialEq, Eq)]
pub struct PtySize {
    pub rows: u16,
    pub cols: u16,
    pub pixel_width: u16,
    pub pixel_height: u16,
}

impl PtySize {
    pub fn new(rows: u16, cols: u16) -> Self {
        Self { rows, cols, pixel_width: 0, pixel_height: 0 }
    }
    fn to_winsize(self) -> libc::winsize {
        libc::winsize {
            ws_row: self.rows,
            ws_col: self.cols,
            ws_xpixel: self.pixel_width,
            ws_ypixel: self.pixel_height,
        }
    }
}

/// Open a new PTY pair. Returns (master, slave_path).
///
/// This is the POSIX sequence, spelled out. `openpty()` does all of it; we do it
/// by hand because each step has a distinct failure mode you should be able to name.
fn open_pty_pair() -> io::Result<(OwnedFd, CString)> {
    // 1. Open the PTY multiplexer. The kernel allocates a fresh pair and hands
    //    back the master. O_NOCTTY: do not let this become our controlling terminal.
    //    SAFETY: posix_openpt takes only flags.
    let master_raw: RawFd = unsafe { libc::posix_openpt(libc::O_RDWR | libc::O_NOCTTY) };
    if master_raw < 0 {
        return Err(io::Error::last_os_error());
    }
    // SAFETY: posix_openpt returned a fresh, owned fd.
    let master = unsafe { OwnedFd::from_raw_fd(master_raw) };

    // 2. Fix ownership/permissions on the slave device.
    //    A no-op on modern Linux with devpts; still required for portability.
    // SAFETY: master is a valid PTY master fd.
    if unsafe { libc::grantpt(master.as_raw_fd()) } != 0 {
        return Err(io::Error::last_os_error());
    }

    // 3. Clear the slave's lock so it can be opened. Skip this and open() fails EIO.
    // SAFETY: as above.
    if unsafe { libc::unlockpt(master.as_raw_fd()) } != 0 {
        return Err(io::Error::last_os_error());
    }

    // 4. Get the slave's path. ptsname() returns a pointer to a STATIC buffer —
    //    not thread-safe. We copy it immediately, which makes the race window
    //    as small as it can be without ptsname_r (unavailable everywhere).
    // SAFETY: master is a valid PTY master; the returned pointer is valid until
    // the next ptsname() call on this thread, and we copy before returning.
    let name_ptr = unsafe { libc::ptsname(master.as_raw_fd()) };
    if name_ptr.is_null() {
        return Err(io::Error::last_os_error());
    }
    // SAFETY: name_ptr is a valid NUL-terminated C string.
    let slave_path = unsafe { CStr::from_ptr(name_ptr) }.to_owned();

    Ok((master, slave_path))
}
}

Failure modes, one per step:

Step	Skip it and you get
`posix_openpt`	Nothing works
`grantpt`	Permission denied opening the slave on older/unusual systems
`unlockpt`	`open(slave)` fails with `EIO` — a genuinely baffling error the first time
copy `ptsname` result	A second PTY allocation in another thread overwrites your path

Step 3: Spawn the child

This is the heart of the lab. Read every comment.

#![allow(unused)]
fn main() {
pub struct PtyConfig {
    pub program: CString,
    pub args: Vec<CString>,
    pub env: Vec<CString>,        // "KEY=VALUE" strings
    pub size: PtySize,
}

impl Pty {
    pub fn spawn(cfg: &PtyConfig) -> io::Result<Pty> {
        let (master, slave_path) = open_pty_pair()?;

        // Set the size BEFORE forking. If the child's first TIOCGWINSZ returns 0x0,
        // programs like `top` render nothing and some cache the bad value forever.
        let ws = cfg.size.to_winsize();
        // SAFETY: TIOCSWINSZ reads a struct winsize through the pointer.
        if unsafe { libc::ioctl(master.as_raw_fd(), libc::TIOCSWINSZ, &ws) } != 0 {
            return Err(io::Error::last_os_error());
        }

        // Open the slave in the PARENT, before fork. Doing it here means the child
        // does not have to reopen it (one less failure point in the fragile
        // between-fork-and-exec window), and it keeps `slave_path` out of that window.
        // SAFETY: slave_path is a valid NUL-terminated path from ptsname.
        let slave_raw = unsafe {
            libc::open(slave_path.as_ptr(), libc::O_RDWR | libc::O_NOCTTY)
        };
        if slave_raw < 0 {
            return Err(io::Error::last_os_error());
        }

        // SAFETY: fork() in a program that is single-threaded at this point.
        // In a multi-threaded program (your GUI later) everything between fork and
        // exec must be async-signal-safe — no allocation, no locks.
        let pid = unsafe { libc::fork() };
        match pid {
            -1 => {
                unsafe { libc::close(slave_raw) };
                Err(io::Error::last_os_error())
            }
            0 => {
                // ═══════════════════ CHILD ═══════════════════
                // From here to execve, ONLY async-signal-safe calls.
                // Any error path must _exit(), never exit() and never panic.
                unsafe {
                    child_setup_and_exec(slave_raw, master.as_raw_fd(), cfg);
                }
                // child_setup_and_exec never returns.
                unreachable!()
            }
            child => {
                // ═══════════════════ PARENT ═══════════════════
                // MANDATORY: close our copy of the slave. While ANY slave fd is
                // open anywhere, the kernel considers the terminal "connected", so
                // read(master) will never report EIO/EOF when the child exits and
                // your event loop will hang forever. This is the #1 PTY bug.
                // SAFETY: slave_raw is our own open fd.
                unsafe { libc::close(slave_raw) };

                Ok(Pty { master, child, exited: None })
            }
        }
    }
}

/// Runs in the child between fork and exec. Never returns.
///
/// # Safety
/// Must only be called in a freshly forked child. Only async-signal-safe calls
/// are permitted: no allocation, no locks, no Rust std I/O.
unsafe fn child_setup_and_exec(slave: RawFd, master: RawFd, cfg: &PtyConfig) -> ! {
    // 1. New session. The child becomes session leader and process group leader,
    //    and DROPS any controlling terminal inherited from the parent.
    //    Fails with EPERM if we were already a process group leader — which a
    //    freshly forked child never is, so a failure here means something is
    //    structurally wrong and we must not continue.
    if libc::setsid() < 0 {
        libc::_exit(126);
    }

    // 2. Make the slave our controlling terminal. REQUIRES step 1 to have happened:
    //    the caller must be a session leader with no controlling terminal.
    //    Without this: no job control, ^C does nothing, `vim` misbehaves,
    //    and bash prints "cannot set terminal process group".
    #[cfg(any(target_os = "linux", target_os = "android"))]
    let sctty = libc::ioctl(slave, libc::TIOCSCTTY as libc::c_ulong, 0);
    #[cfg(not(any(target_os = "linux", target_os = "android")))]
    let sctty = libc::ioctl(slave, libc::TIOCSCTTY, 0);
    if sctty < 0 {
        libc::_exit(126);
    }

    // 3. The slave becomes the child's standard streams.
    if libc::dup2(slave, libc::STDIN_FILENO) < 0
        || libc::dup2(slave, libc::STDOUT_FILENO) < 0
        || libc::dup2(slave, libc::STDERR_FILENO) < 0
    {
        libc::_exit(126);
    }

    // 4. Drop the redundant descriptors. Guard against slave being 0/1/2 —
    //    which happens when the standard fds were closed before us.
    if slave > libc::STDERR_FILENO {
        libc::close(slave);
    }
    // 5. The child must NOT hold the master end.
    libc::close(master);

    // 6. Clear the inherited signal MASK. It survives execve, and a shell that
    //    starts with SIGCHLD blocked has silently broken job control.
    let mut set: libc::sigset_t = std::mem::zeroed();
    libc::sigemptyset(&mut set);
    libc::sigprocmask(libc::SIG_SETMASK, &set, std::ptr::null_mut());

    // 7. Reset dispositions we (or our parent) may have set to SIG_IGN.
    //    Handlers do not survive execve, but SIG_IGN DOES.
    for sig in [libc::SIGINT, libc::SIGQUIT, libc::SIGTSTP,
                libc::SIGTTIN, libc::SIGTTOU, libc::SIGCHLD, libc::SIGPIPE] {
        libc::signal(sig, libc::SIG_DFL);
    }

    // 8. exec. Build the argv/envp arrays as NULL-terminated pointer arrays.
    //    NOTE: these Vecs were allocated in the PARENT before fork, so no
    //    allocation happens here — which is what makes this async-signal-safe.
    let mut argv: Vec<*const libc::c_char> =
        cfg.args.iter().map(|a| a.as_ptr()).collect();
    argv.push(std::ptr::null());
    let mut envp: Vec<*const libc::c_char> =
        cfg.env.iter().map(|e| e.as_ptr()).collect();
    envp.push(std::ptr::null());

    libc::execve(cfg.program.as_ptr(), argv.as_ptr(), envp.as_ptr());

    // execve only returns on failure.
    libc::_exit(127);
}
}

Warning: The argv/envp Vec::collect() in step 8 does allocate, which violates the async-signal-safety rule. It is written this way for readability. Fix it as your first challenge: build the pointer arrays in the parent, before fork, and pass them in. In a single-threaded program at fork time you will get away with the allocation; the moment your GUI forks from a multi-threaded process, you will not, and the failure is a rare hang that is essentially undebuggable. Do the fix.

Step 4: The naïve relay loop

crates/pty-runner/src/main.rs. This version is deliberately incomplete — no resize, no signals, no poll. Getting it working and then feeling what it lacks is the point; Lab 3 fixes it.

use std::io::{self, Read, Write};
use std::os::fd::AsRawFd;
use terminal_pty::{Pty, PtyConfig, PtySize};

fn main() -> io::Result<()> {
    // Reuse the RawMode guard from Lab 1 — copy the module across.
    let _raw = raw_mode::RawMode::enable(0)?;

    let shell = std::env::var("SHELL").unwrap_or_else(|_| "/bin/bash".into());
    let cfg = PtyConfig {
        program: std::ffi::CString::new(shell.clone()).unwrap(),
        args: vec![std::ffi::CString::new(shell).unwrap()],
        env: build_env(),
        // Start with the OUTER terminal's real size, not a guess.
        size: current_size(0).unwrap_or(PtySize::new(24, 80)),
    };

    let pty = Pty::spawn(&cfg)?;
    let master = pty.master_fd();

    // Naïve: two blocking reads cannot both be waited on, so this version uses a
    // thread for one direction. Lab 3 replaces this with a single poll() loop —
    // and you should be able to say why that is better before you get there.
    let master_for_thread = master;
    std::thread::spawn(move || {
        let mut buf = [0u8; 4096];
        loop {
            // SAFETY: master_for_thread is a valid open fd for the process lifetime.
            let n = unsafe {
                libc::read(master_for_thread, buf.as_mut_ptr() as *mut _, buf.len())
            };
            if n <= 0 { break; }   // 0 = EOF (macOS); <0 = EIO (Linux) when the child exits
            let _ = io::stdout().write_all(&buf[..n as usize]);
            let _ = io::stdout().flush();
        }
        // The child is gone. Restore and exit hard, because the main thread is
        // blocked in read(0) and cannot be woken. This is the flaw Lab 3 fixes.
        std::process::exit(0);
    });

    let mut stdin = io::stdin();
    let mut buf = [0u8; 1024];
    loop {
        let n = match stdin.read(&mut buf) {
            Ok(0) => break,
            Ok(n) => n,
            Err(e) if e.kind() == io::ErrorKind::Interrupted => continue,
            Err(e) => return Err(e),
        };
        // NOTE: this ignores short writes. Lab 3 fixes that too.
        // SAFETY: master is valid; buf[..n] is initialized.
        unsafe { libc::write(master, buf.as_ptr() as *const _, n) };
    }
    Ok(())
}

fn build_env() -> Vec<std::ffi::CString> {
    let mut env: Vec<std::ffi::CString> = std::env::vars()
        // Stale LINES/COLUMNS override the ioctl in some programs. Remove them.
        .filter(|(k, _)| k != "LINES" && k != "COLUMNS" && k != "TERM")
        .map(|(k, v)| std::ffi::CString::new(format!("{k}={v}")).unwrap())
        .collect();
    // Without TERM, vim says "terminal too dumb" and nothing has colors.
    env.push(std::ffi::CString::new("TERM=xterm-256color").unwrap());
    env
}

Step 5: Run it

cargo run -p pty-runner

You should get a shell. Now verify it is a real one.

Expected Output

$ tty
/dev/pts/2                      ← your OUTER terminal

$ cargo run -p pty-runner
bash-5.2$ tty
/dev/pts/9                      ← the INNER PTY. Different. This is the proof.

bash-5.2$ ps -o pid,ppid,pgid,sid,tpgid,stat,tty,comm
    PID    PPID    PGID     SID   TPGID STAT TT       COMMAND
  50123   50100   50123   50123   50130 Ss   pts/9    bash
  50130   50123   50130   50123   50130 R+   pts/9    ps
#                                  ▲
#   TPGID == ps's PGID → `ps` is the foreground process group. Job control works.
#   bash: PID == PGID == SID → it is both session leader and group leader. setsid worked.

bash-5.2$ stty size
24 80                           ← TIOCSWINSZ worked

bash-5.2$ echo $TERM
xterm-256color

bash-5.2$ vim                   ← opens, alt screen works, :q quits
bash-5.2$ top                   ← renders and updates, q quits
bash-5.2$ exit

There must be no message saying bash: no job control in this shell and no cannot set terminal process group.

Debugging Steps

`bash: cannot set terminal process group (-1): Inappropriate ioctl for device`

The child has no controlling terminal. Check, in order:

Did setsid() succeed? (Check its return; do not assume.)
Did TIOCSCTTY succeed?
Are they in that order?

`bash: no job control in this shell`

Same cause. bash tests for a controlling terminal at startup.

The program hangs after the shell exits

The parent still has the slave fd open. close(slave_raw) in the parent, right after fork.

`open(slave)` fails with `EIO`

You skipped unlockpt.

`stty size` reports `0 0`

You did not call TIOCSWINSZ, or you called it after fork. Set it on the master before forking.

`vim` says "terminal too dumb" / no colors

TERM is unset or wrong in the child's environment.

Everything is doubled: `llss` when you type `ls`

Your outer terminal is still echoing and the inner line discipline is echoing. Enable raw mode on fd 0 in the parent.

Ctrl+C kills your runner instead of the inner job

Your outer terminal still has ISIG set. Same fix: raw mode on fd 0.

Output is a staircase

You are relaying bytes correctly; your outer terminal has OPOST cleared by cfmakeraw. That is correct — the inner PTY's line discipline already turned \n into \r\n on the way out, so what you relay is already CRLF. If you see a staircase, check that you are not adding processing of your own.

Linux: an error message on exit mentioning "Input/output error"

read(master) returned EIO because the child exited. On Linux that is the EOF condition, not an error. Handle it as a clean shutdown.

Experiment

CLAIM. Removing TIOCSCTTY breaks job control while leaving basic I/O intact — proving that a controlling terminal is about signals and process groups, not about moving bytes.

METHOD. Comment out the TIOCSCTTY block. Rebuild. Run. Then inside the child shell:

tty                # still works — the slave is still fd 0/1/2
echo hello         # still works — bytes still flow
ps -o pid,tpgid,tty,comm    # look at TPGID and TT
sleep 100          # then press ^C

PREDICTION. Write down, before running: (a) does tty still print a device? (b) does ^C interrupt sleep? (c) what does the TT column show? (d) does vim still open?

RESULT. Record it. Then restore the code and confirm the difference.

Test

PTY code needs integration tests, because the behavior is the OS interaction.

#![allow(unused)]
fn main() {
// crates/terminal-pty/tests/spawn.rs
use std::ffi::CString;
use std::io::Read;
use terminal_pty::{Pty, PtyConfig, PtySize};

fn cfg(cmd: &str, args: &[&str], size: PtySize) -> PtyConfig {
    PtyConfig {
        program: CString::new(cmd).unwrap(),
        args: std::iter::once(cmd).chain(args.iter().copied())
            .map(|s| CString::new(s).unwrap()).collect(),
        env: vec![CString::new("TERM=xterm-256color").unwrap(),
                  CString::new("PATH=/usr/bin:/bin").unwrap()],
        size,
    }
}

#[test]
fn echo_produces_crlf_because_of_onlcr() {
    // The child writes "hi\n". The line discipline's ONLCR turns it into "hi\r\n"
    // on the way to the master. That transformation is the whole point of the test:
    // a pipe would give us "hi\n".
    let mut pty = Pty::spawn(&cfg("/bin/echo", &["hi"], PtySize::new(24, 80))).unwrap();
    let mut out = String::new();
    pty.read_to_string_until_eof(&mut out).unwrap();
    assert!(out.contains("hi\r\n"), "expected CRLF from ONLCR, got {out:?}");
}

#[test]
fn child_sees_the_size_we_set() {
    // `stty size` inside the child reads TIOCGWINSZ. If our TIOCSWINSZ before fork
    // worked, it reports exactly what we asked for.
    let mut pty = Pty::spawn(&cfg("/bin/sh", &["-c", "stty size"], PtySize::new(30, 100))).unwrap();
    let mut out = String::new();
    pty.read_to_string_until_eof(&mut out).unwrap();
    assert!(out.contains("30 100"), "expected '30 100', got {out:?}");
}

#[test]
fn child_is_a_session_leader_with_a_controlling_terminal() {
    // If setsid()+TIOCSCTTY worked, `ps` reports PID == SID for the shell, and the
    // TT column is a pts device rather than "?".
    let mut pty = Pty::spawn(&cfg(
        "/bin/sh", &["-c", "ps -o pid,sid,tty -p $$"], PtySize::new(24, 80))).unwrap();
    let mut out = String::new();
    pty.read_to_string_until_eof(&mut out).unwrap();
    assert!(!out.contains(" ? "), "child has no controlling terminal: {out:?}");
}

#[test]
fn isatty_is_true_in_the_child() {
    // The single most important difference from a pipe.
    let mut pty = Pty::spawn(&cfg(
        "/bin/sh", &["-c", "test -t 0 && echo YES || echo NO"], PtySize::new(24, 80))).unwrap();
    let mut out = String::new();
    pty.read_to_string_until_eof(&mut out).unwrap();
    assert!(out.contains("YES"), "isatty(0) must be true on a PTY: {out:?}");
}
}

You will need a read_to_string_until_eof helper that loops until read returns 0 (macOS) or EIO (Linux). Writing that helper is the lesson about the platform difference.

cargo test -p terminal-pty

Challenge Extensions

Fix the async-signal-safety hole. Build the argv/envp pointer arrays in the parent, before fork, so nothing between fork and exec allocates. Verify by reading the code and naming every call in that window.
Compare with forkpty. Write a second spawn implementation using forkpty() and diff the two files. Every line that disappeared corresponds to something forkpty does for you — enumerate them. Now you know exactly what the convenience function hides.
Compare with portable-pty. Add it as a dev-dependency, write a third implementation, and diff again. Note which of your failure modes it makes impossible and which it merely makes less likely.
Add packet mode. ioctl(master, TIOCPKT, &1) makes every read(master) return a status byte first, reporting flush events and termios changes. Print those events. This is how rlogin-style relays propagate ^C flush semantics.
Spawn something other than a shell. Pty::spawn a Python REPL, then vim, then less on a large file. Note which of them immediately reconfigure the termios of the slave — watch it with stty -a -F /dev/pts/N from another window while they run.

Implementation Requirements / Deliverables

terminal-pty compiles and spawns a shell using only libc.
A written, ordered list of every syscall in the spawn path, each with (a) its purpose and (b) the symptom when it is omitted or misordered.
All four integration tests pass on your platform.
tty, ps -o pid,pgid,sid,tpgid,tty, stty size, echo $TERM, vim, and top all behave correctly inside the runner.
No bash: no job control in this shell message.
The TIOCSCTTY removal experiment performed, with prediction and result recorded.
At least challenge 1 (async-signal-safety) and challenge 2 (forkpty diff) completed.

Validation / Self-check

List the seven syscalls of the spawn path in order, with one sentence each.
Why must setsid() precede TIOCSCTTY? What error results from the other order?
Why must the parent close the slave fd? What exact symptom appears if it does not?
Why must the child close the master fd?
Why is _exit(126) used in the child's error paths instead of panic! or exit?
Name three things that must happen between fork and exec that forkpty() hides.
Which two things survive execve that you must explicitly reset, and what breaks if you do not?
Why do you unset LINES and COLUMNS in the child's environment?
What does read(master) return after the child exits — on Linux, and on macOS?
Your relay works, but the inner vim renders at 80×24 no matter how you resize. Which lab fixes that, and which two ioctls are involved?

Next: Lab 3 — The PTY Event Loop, where the naïve relay becomes a correct one.

Open-Source Engineer & Contributor