File Descriptors, fork, exec, and Spawn

Two concepts, six parts each: file descriptors, and fork/exec/spawn. Together they are how a shell ends up with its standard streams on a PTY slave — which is the single most important mechanical fact in this whole curriculum.

Concept 1: File Descriptors

1. What problem it solves

A process needs to refer to kernel objects — files, sockets, pipes, terminals — without being able to touch them. A file descriptor is a small non-negative integer that indexes into a per-process table. The integer is meaningless outside the process; the kernel resolves it to an actual object.

The design consequence that matters here: every I/O interface is the same interface. read() and write() work identically on a file, a pipe, a socket, and a terminal. That uniformity is why a shell written in 1975 works inside a GPU-accelerated terminal emulator written in 2026.

2. Where it exists in the OS

Kernel, in three layers. Understanding all three explains dup2, fork inheritance, and why closing one fd does not always tear down the object.

  PROCESS A                          KERNEL
  ┌───────────────────┐
  │ fd table          │       ┌─────────────────────────┐
  │  0 ──────────────────────▶│ open file description   │
  │  1 ──────────────────────▶│  • file offset          │───┐
  │  2 ──────────────────────▶│  • status flags         │   │
  │  3 ─────────────┐         │    (O_NONBLOCK, O_APPEND)│  │
  └─────────────────│─────────└─────────────────────────┘   │
                    │                                       ▼
  PROCESS B         │         ┌─────────────────────────┐  ┌──────────────┐
  ┌───────────────┐ │         │ another open file descr.│─▶│ inode /      │
  │  0 ───────────┼─┘         └─────────────────────────┘  │ device /     │
  │  1 ───────────────────────▶ (shared after fork)         │ tty struct  │
  └───────────────┘                                         └──────────────┘
      fd table          open file descriptions           the actual object
      (per process)     (shared by dup/fork)             (shared by open)

Warning: O_NONBLOCK lives on the open file description, not the fd. Setting it on the master fd affects every fd that shares that description — including any the child inherited. This is a real source of confusion when a child mysteriously gets EAGAIN from a "blocking" fd.

3. Who owns or interacts with it

Every process. Three fds are conventional, not magic:

The kernel gives no special meaning to 0/1/2. execve does not reset them. That is the entire trick of shell redirection and of PTY spawning: you rearrange the child's fd table between fork and exec, and the new program inherits the arrangement without knowing.

4. Bytes and syscalls

open(path, flags)        → lowest unused fd
dup(oldfd)               → lowest unused fd, same open file description
dup2(oldfd, newfd)       → newfd refers to oldfd's description; closes newfd first if open;
                           newfd's FD_CLOEXEC is CLEARED (important!)
dup3(oldfd, newfd, flags)→ Linux; lets you set O_CLOEXEC atomically
close(fd)                → drops this reference; the object dies when the last one goes
fcntl(fd, F_GETFL/F_SETFL, ...)  → status flags (O_NONBLOCK)
fcntl(fd, F_GETFD/F_SETFD, ...)  → the FD_CLOEXEC flag (per-fd, not per-description)

The redirection idiom, which is also the PTY idiom:

dup2(slave, STDIN_FILENO);    // fd 0 now refers to the PTY slave
dup2(slave, STDOUT_FILENO);   // fd 1 too
dup2(slave, STDERR_FILENO);   // fd 2 too
if (slave > STDERR_FILENO) close(slave);   // the original fd is now redundant

Note: The if (slave > 2) guard matters. If slave happened to be fd 0, 1, or 2 (possible when the standard fds were closed), closing it unconditionally would close the very fd you just duplicated onto. Real code has this bug; it manifests as "works normally, breaks when run from a daemon."

In Rust, std::os::fd gives you OwnedFd and BorrowedFd, which make the ownership rules compile-time:

#![allow(unused)]
fn main() {
use std::os::fd::{AsRawFd, FromRawFd, OwnedFd};

// An OwnedFd closes on drop. A BorrowedFd cannot outlive its owner.
// Prefer these over raw i32 as soon as you leave the libc layer —
// "which function closes this fd" is the most common source of double-close bugs.
}

5. Experiment

# What are your shell's fds actually pointing at?
ls -l /proc/$$/fd            # Linux
lsof -p $$ | head            # macOS or Linux

# Watch redirection rearrange the table:
ls -l /proc/self/fd          # 0,1,2 → your tty
ls -l /proc/self/fd > /tmp/x ; cat /tmp/x    # 1 → the file, 0 and 2 still the tty

# Prove that dup2 shares the offset (the open file description):
exec 3>/tmp/dup-demo
exec 4>&3            # fd 4 dups fd 3 — SAME description, SAME offset
echo -n "AAA" >&3
echo -n "BBB" >&4
exec 3>&- 4>&-
cat /tmp/dup-demo    # "AAABBB" — appended, not overwritten, because the offset is shared

Predict first: what does /tmp/dup-demo contain if you replace exec 4>&3 with exec 4>/tmp/dup-demo (a second independent open)? Write it down, then try it.

6. Failure mode

Tip: Set FD_CLOEXEC on every fd you create in a program that spawns children, and clear it only on the specific fds you intend to pass. In Rust, std::fs::File and most of std already set O_CLOEXEC; raw libc::open does not unless you pass O_CLOEXEC.

Concept 2: fork, exec, and Spawn

1. What problem it solves

You need to start a new program with a specific environment: specific fds, a specific session, a specific process group, a specific controlling terminal. Unix splits this into two operations — fork (make a copy of me) and execve (replace my program image) — precisely so that there is a window between them in which you can configure the new process.

That window is where the entire PTY setup happens. This is not an accident of history; it is the design.

  parent
    │
    ├── fork() ──────────────┐
    │                        │  the CHILD, still running the parent's code
    │                        │  ┌──────────────────────────────────────┐
    │                        ├─▶│  THE WINDOW                          │
    │                        │  │   setsid()                            │
    │                        │  │   open slave; ioctl(TIOCSCTTY)        │
    │                        │  │   dup2 ×3; close extras               │
    │                        │  │   setenv("TERM", ...)                 │
    │                        │  │   signal dispositions reset to default│
    │                        │  └──────────────┬───────────────────────┘
    │                        │                 │
    │                        └── execve("/bin/bash") ──▶ bash starts,
    │                                                     inherits everything above
    ▼
  parent continues: close slave, poll master

2. Where it exists in the OS

Kernel. fork creates a new process with a copy-on-write address space. execve discards the address space and loads a new program image.

3. Who owns or interacts with it

The parent (your emulator) and the child (the shell). What crosses the boundary:

Warning: Two of those rows cause real bugs in terminal code. (1) The signal mask survives execve. If your emulator blocks SIGCHLD and forgets to unblock it in the child, the shell starts with SIGCHLD blocked and its job control breaks in ways that take a day to find. Always reset the mask to empty in the child. (2) Only the forking thread survives fork. In a multi-threaded program (which a GUI terminal is), the child may hold a mutex that no thread will ever unlock. That is why the code between fork and exec must be async-signal-safe: no malloc, no locks, no println!.

4. Bytes and syscalls

fork()            → 0 in the child, child's pid in the parent, -1 on error
execve(path, argv, envp)  → never returns on success
_exit(code)       → in the child if exec fails.  NOT exit() — that would run atexit handlers
                     and flush the parent's buffers a second time
waitpid(pid, &status, 0 | WNOHANG | WUNTRACED)  → reap; get exit/stop status
posix_spawn(...)  → fork+exec in one call, with a "file actions" list; safe in threaded programs
vfork()           → historical optimization; do not use

In Rust, the idiomatic version uses std::process::Command with a pre-exec hook:

#![allow(unused)]
fn main() {
use std::os::unix::process::CommandExt;
use std::process::{Command, Stdio};

let mut cmd = Command::new(shell);
cmd.env("TERM", "xterm-256color");
// stdio() takes ownership of fds that become the child's 0/1/2.
cmd.stdin(unsafe { Stdio::from_raw_fd(slave_dup0) })
   .stdout(unsafe { Stdio::from_raw_fd(slave_dup1) })
   .stderr(unsafe { Stdio::from_raw_fd(slave_dup2) });

unsafe {
    // SAFETY: this closure runs in the child between fork and exec.
    // It must be async-signal-safe: no allocation, no locks, no Rust I/O.
    // Every call below is a direct libc call with no allocation.
    cmd.pre_exec(move || {
        if libc::setsid() < 0 { return Err(std::io::Error::last_os_error()); }
        if libc::ioctl(slave_raw, libc::TIOCSCTTY as _, 0) < 0 {
            return Err(std::io::Error::last_os_error());
        }
        // Reset the signal mask — it survives execve and would break the shell's job control.
        let mut set: libc::sigset_t = std::mem::zeroed();
        libc::sigemptyset(&mut set);
        libc::sigprocmask(libc::SIG_SETMASK, &set, std::ptr::null_mut());
        Ok(())
    });
}
let child = cmd.spawn()?;
}

Line-by-line, the parts that matter:

• pre_exec is unsafe for a reason. Its closure runs in a forked child of a possibly multi-threaded program. The rule is the same as for a signal handler: only async-signal-safe calls. println! allocates and locks; it is forbidden here. If you need to report an error, return Err — Command transmits it to the parent over an internal CLOEXEC pipe.
• setsid() before TIOCSCTTY — non-negotiable; see the sessions chapter.
• Stdio::from_raw_fd takes ownership. Do not also close those fds yourself; that is a double-close.
• Resetting the signal mask is the step almost everyone forgets.

Note on posix_spawn: Rust's Command uses posix_spawn when it can, and falls back to fork+exec when you use pre_exec (because posix_spawn has no arbitrary-code hook). That fallback is exactly what you want here — but be aware it means adding pre_exec changes the spawn mechanism, and therefore its performance characteristics in a threaded program.

5. Experiment

# 1. Watch inheritance across fork+exec. Run this and inspect from another shell:
bash -c 'exec 9</etc/hostname; sleep 60' &
ls -l /proc/$!/fd        # Linux: fd 9 is there — inherited across exec (no FD_CLOEXEC)

# 2. Prove FD_CLOEXEC works:
bash -c 'exec 9</etc/hostname; exec sleep 60' &     # still inherited
# now compare with a program that sets FD_CLOEXEC — your Rust code, with and without O_CLOEXEC.

# 3. Prove the signal mask survives exec:
#    Write a tiny program that blocks SIGINT, then execs `bash`.
#    Inside that bash, press Ctrl+C.  Predict what happens before you try it.

Predict first: in experiment 3, does Ctrl+C interrupt the shell? Does it interrupt a sleep 100 started by that shell? Write both predictions before running.

6. Failure mode

The Complete Child Setup, Annotated

This is the checklist. Lab 2 implements it; this is the reference you check your implementation against.

IN THE CHILD, between fork() and execve():

 1. setsid()
       → new session, new process group, child is session leader
       → drops any inherited controlling terminal
       → FAILS with EPERM if the child is already a process group leader

 2. open(slave_path, O_RDWR)            (or use the inherited slave fd)
       → on some systems this alone acquires the controlling terminal;
         do not rely on it — be explicit in step 3

 3. ioctl(slave, TIOCSCTTY, 0)
       → THIS session's controlling terminal is now the PTY slave
       → requires: caller is a session leader with no controlling terminal

 4. dup2(slave, 0); dup2(slave, 1); dup2(slave, 2)
       → the new program's standard streams

 5. if (slave > 2) close(slave)
       → drop the redundant fd

 6. close(master)
       → the child must NOT hold the master end

 7. sigprocmask(SIG_SETMASK, &empty, NULL)
       → clear the inherited signal mask (it survives execve)

 8. reset dispositions of any signal you set to SIG_IGN
       → SIG_IGN survives execve; handlers do not

 9. setenv("TERM", "xterm-256color")
    unsetenv("COLUMNS"); unsetenv("LINES")     ← stale values confuse programs

10. execve(shell, argv, envp)
       → if this returns, it failed: _exit(127)

And in the parent, immediately after fork():

 1. close(slave)          ← MANDATORY. Without it you never see the child's exit on the master.
 2. keep master
 3. record child pid for waitpid()

Validation / Self-check

1. Name the three layers a file descriptor resolves through, and say which layer O_NONBLOCK lives on.
2. What exactly does dup2(a, b) do to b's FD_CLOEXEC flag, and why does that matter?
3. Which of these survive execve: signal handlers, the signal mask, SIG_IGN dispositions, fds without FD_CLOEXEC, the environment?
4. Why must the code between fork and exec be async-signal-safe? Give the concrete deadlock.
5. Why _exit() and not exit() in a child whose execve failed?
6. Why must the parent close the slave fd? Describe the exact hang that results if it does not.
7. In the redirection idiom, why is the close(slave) guarded by slave > 2?
8. What does posix_spawn give you that fork+exec does not, and why does Rust's Command stop using it when you add pre_exec?
9. You spawn a shell and Ctrl+C does nothing. List, in order, the four things you would check.

Next: Sessions, Process Groups & the Controlling Terminal.