Durability and Filesystems

Three concepts in the six-part treatment: writeback, the durability contract, and journalling and crash consistency.

This is the part of storage that is genuinely hard, and it is hard for a reason worth stating up front: every layer between your program and the platter is allowed to reorder and delay, and durability is built by constraining that reordering at exactly the right points and nowhere else.

Concept 1: Writeback

1. What problem it solves

If write() waited for the device, every program would run at disk speed. So it does not: the data is copied into the page cache, the folio is marked dirty, and the syscall returns.

Something has to write it out eventually. That something is writeback, and its scheduling is a control problem: too lazy and a crash loses a lot and the eventual flush is a latency spike; too eager and you lose the batching and re-write pages that were about to change again.

2. Where it exists in the kernel

rg -n "balance_dirty_pages\b" -A 40 mm/page-writeback.c | head -50
rg -n "wb_workfn|writeback_inodes_wb" fs/fs-writeback.c | head
rg -n "struct backing_dev_info \{" -A 25 include/linux/backing-dev-defs.h
ls /sys/kernel/tracing/events/writeback/
ps -eo pid,comm | grep -E 'kworker.*flush|writeback'

3. The control loop

   write() ──▶ folio marked DIRTY, accounted in NR_FILE_DIRTY
                    │
                    ├── vm.dirty_background_ratio (default ~10%)
                    │     crossed → wake the per-bdi writeback worker.
                    │     ASYNCHRONOUS. Your process notices nothing.
                    │
                    ├── vm.dirty_ratio (default ~20%)
                    │     crossed → balance_dirty_pages() THROTTLES the
                    │     WRITING TASK: it sleeps until enough is written.
                    │     This is where "the machine froze while copying a
                    │     big file" comes from.
                    │
                    └── vm.dirty_expire_centisecs (default 30 s)
                          old dirty pages get written even if the machine
                          is idle and no threshold was crossed.

cat /proc/sys/vm/dirty_ratio /proc/sys/vm/dirty_background_ratio
cat /proc/sys/vm/dirty_expire_centisecs /proc/sys/vm/dirty_writeback_centisecs
grep -E 'Dirty|Writeback' /proc/meminfo

Note the two distinct states: Dirty is "modified, not yet submitted"; Writeback is "submitted, in flight". A folio moves from one to the other and is clean only when the device confirms.

Writeback is per-backing-device (bdi), not global, so a slow USB stick cannot throttle writes to your NVMe — a bug that took years to fix properly.

4. Experiment

CLAIM. The dirty thresholds are real, and you can watch a process get throttled.

METHOD.

grep -E 'Dirty|Writeback' /proc/meminfo

# Write more than dirty_background_ratio but less than dirty_ratio:
dd if=/dev/zero of=/tmp/w1 bs=1M count=512 2>&1 | tail -1
grep -E 'Dirty|Writeback' /proc/meminfo      # immediately after

# Watch it drain on its own:
for i in 1 2 3 4 5; do sleep 2; grep -E '^Dirty' /proc/meminfo; done

# Now exceed dirty_ratio and watch the throttle:
sudo bpftrace -e 'kprobe:balance_dirty_pages { @[comm] = count(); }' &
dd if=/dev/zero of=/tmp/w2 bs=1M count=8192 2>&1 | tail -1
kill %1

PREDICT FIRST: the first dd reports a throughput far above your disk's real speed. Why? And what happens to the second dd's reported throughput as it crosses dirty_ratio?

Then make the throttle unmistakable:

sudo sysctl vm.dirty_ratio=5 vm.dirty_background_ratio=2
dd if=/dev/zero of=/tmp/w3 bs=1M count=4096 2>&1 | tail -1
sudo sysctl vm.dirty_ratio=20 vm.dirty_background_ratio=10

5. Failure mode

Concept 2: The Durability Contract

1. What problem it solves

"Is my data safe?" has a precise answer at each layer, and almost every data-loss bug in application software is a misunderstanding of which layer was reached.

2. Where it exists in the kernel

rg -n "SYSCALL_DEFINE1\(fsync|do_fsync|vfs_fsync_range" fs/sync.c | head
rg -n "REQ_PREFLUSH|REQ_FUA" block/blk-flush.c include/linux/blk_types.h | head
rg -n "blkdev_issue_flush" block/ | head

3. The ladder

   write() returns
     └── the data is in the PAGE CACHE.
         Survives: the process crashing, the process being killed.
         Does NOT survive: kernel panic, power loss.

   fsync(fd) returns
     └── the filesystem has written this file's DATA and the METADATA
         needed to find it, AND issued a cache flush to the device.
         Survives: power loss.
         Does NOT cover: the DIRECTORY ENTRY, if the file is new.

   fsync(dirfd) returns
     └── the directory entry is durable too.
         THIS IS THE STEP EVERYONE FORGETS. A newly created file that was
         fsynced can still vanish entirely after a crash, because the name
         pointing at it was never made durable.

   fdatasync(fd)
     └── data plus only the metadata REQUIRED to read it back (i.e. size,
         but not mtime). Cheaper than fsync. Usually what you want.

   O_SYNC / O_DSYNC
     └── every write behaves as if followed by fsync/fdatasync.

   REQ_FUA on the bio
     └── the device must place THIS write on stable media before completing.
   REQ_PREFLUSH
     └── flush everything already in the device's volatile cache first.

The safe pattern for "create a file atomically", which is what every database and every text editor does:

fd = open("file.tmp", O_WRONLY | O_CREAT | O_TRUNC, 0644);
write(fd, data, len);
fsync(fd);                    /* 1. the DATA is durable      */
close(fd);
rename("file.tmp", "file");   /* 2. atomic replace           */
dirfd = open(".", O_RDONLY | O_DIRECTORY);
fsync(dirfd);                 /* 3. the NAME is durable      */
close(dirfd);

Miss step 3 and, after a crash, you can have neither the old file nor the new one.

Warning: fsync() can fail, and what to do about it is genuinely subtle. On some filesystems a failed fsync() marks the error and the next fsync() returns success — even though the data was lost. The error is reported once, to whoever asks first. Retrying fsync() after a failure is not a recovery strategy; the correct response is to treat the data as lost and rewrite it from a known-good source.

rg -n "errseq_t|filemap_check_errors" include/linux/errseq.h fs/ | head
$EDITOR Documentation/filesystems/errseq.rst 2>/dev/null || find Documentation -name 'errseq*'

4. Experiment

CLAIM. Each rung of the ladder costs measurably more, and the numbers explain why applications batch.

METHOD.

cat > /tmp/dur.c <<'EOF'
#define _GNU_SOURCE
#include <fcntl.h>
#include <stdio.h>
#include <string.h>
#include <time.h>
#include <unistd.h>
static double now(void){ struct timespec t; clock_gettime(CLOCK_MONOTONIC,&t);
                         return t.tv_sec + t.tv_nsec/1e9; }
static void run(const char *tag, int flags, int do_fsync, int do_fdatasync) {
    char buf[4096]; memset(buf, 'x', sizeof buf);
    int fd = open("/tmp/durtest", O_WRONLY|O_CREAT|O_TRUNC|flags, 0644);
    double t0 = now();
    for (int i = 0; i < 1000; i++) {
        if (write(fd, buf, sizeof buf) != sizeof buf) break;
        if (do_fsync) fsync(fd);
        if (do_fdatasync) fdatasync(fd);
    }
    printf("%-28s %8.1f us/write\n", tag, (now()-t0)*1e6/1000);
    close(fd);
}
int main(void){
    run("write only",            0,        0, 0);
    run("write + fdatasync",     0,        0, 1);
    run("write + fsync",         0,        1, 0);
    run("O_DSYNC",               O_DSYNC,  0, 0);
    run("O_SYNC",                O_SYNC,   0, 0);
    return 0;
}
EOF
gcc -O2 -o /tmp/dur /tmp/dur.c && /tmp/dur

PREDICT FIRST: rank the five, and predict the ratio between the fastest and the slowest. Most people underestimate it by an order of magnitude.

Then watch the flushes reach the device:

sudo bpftrace -e '
  tracepoint:block:block_rq_issue /args.rwbs =~ /F/ { @flushes = count(); }
  tracepoint:block:block_rq_issue { @all = count(); }' &
/tmp/dur; kill %1

5. Failure mode

Concept 3: Journalling and Crash Consistency

1. What problem it solves

A single logical operation — "append 4 KB to this file" — touches several independent on-disk structures: the data block, the block allocation bitmap, the inode's size and block pointers, and possibly the directory. A crash between any two of them leaves the filesystem inconsistent, and some inconsistencies are worse than data loss: a block marked free while an inode still points at it gets reallocated, and now two files share it.

A journal makes a group of changes atomic with respect to a crash.

2. Where it exists in the kernel

ls fs/jbd2/                      # the journalling layer ext4 uses
rg -n "jbd2_journal_start|jbd2_journal_stop|jbd2_journal_get_write_access" fs/ext4/ | head
ls fs/xfs/ | grep -i log         # XFS has its own
$EDITOR Documentation/filesystems/ext4/journal.rst 2>/dev/null || ls Documentation/filesystems/ext4/

3. How it works

   1. Write the intended changes to the JOURNAL, a contiguous on-disk area.
   2. FLUSH, so the journal entry is durable.
   3. Write a COMMIT RECORD.
   4. FLUSH again.
   5. Only now write the changes to their real locations ("checkpointing").
   6. Eventually, free the journal space.

   AFTER A CRASH:
     - a transaction with no commit record  → discard it. It never happened.
     - a transaction WITH a commit record   → REPLAY it. Idempotent, so
                                               replaying twice is harmless.

   The cost: metadata is written TWICE. The benefit: mount is a journal
   replay taking seconds, not a full fsck taking hours.

The modes, using ext4's names — and the middle one is the default almost everywhere for good reason:

mount | grep -E 'ext4|xfs|btrfs'
sudo tune2fs -l /dev/<dev> 2>/dev/null | grep -iE 'journal|features'
sudo dumpe2fs -h /dev/<dev> 2>/dev/null | grep -i journal

Copy-on-write filesystems (btrfs, ZFS) solve the same problem differently: never overwrite in place, write the new version elsewhere, then atomically update one pointer. No journal, but the same underlying requirement — the pointer update must not be reordered before the data it points to.

4. Experiment

CLAIM. Journalling costs write amplification you can measure, and the mode changes it.

METHOD. In the guest, with a throwaway loop device:

dd if=/dev/zero of=/tmp/fsimg bs=1M count=512 status=none
LOOP=$(sudo losetup -f --show /tmp/fsimg)
sudo mkfs.ext4 -q "$LOOP"
mkdir -p /mnt/t

for mode in ordered writeback journal; do
  sudo mount -o "data=$mode" "$LOOP" /mnt/t 2>/dev/null || { echo "$mode: not supported"; continue; }
  before=$(awk -v d="$(basename $LOOP)" '$3==d {print $10}' /proc/diskstats)
  sudo dd if=/dev/zero of=/mnt/t/f bs=4k count=20000 conv=fsync 2>&1 | tail -1
  sync
  after=$(awk -v d="$(basename $LOOP)" '$3==d {print $10}' /proc/diskstats)
  echo "  data=$mode: sectors written = $((after - before)) (payload = $((20000*8)))"
  sudo umount /mnt/t
done
sudo losetup -d "$LOOP"

PREDICT FIRST: you wrote 80 MB of payload. How many sectors does each mode actually write? The ratio is the write amplification, and data=journal should be visibly different.

Then watch a journal commit:

sudo mount "$LOOP" /mnt/t
sudo bpftrace -e '
  tracepoint:jbd2:jbd2_start_commit  { @commits = count(); }
  tracepoint:jbd2:jbd2_run_stats     { @runs = count(); }' &
sudo dd if=/dev/zero of=/mnt/t/f bs=4k count=1000 conv=fsync 2>/dev/null
kill %1

5. Failure mode

Validation / Self-check

1. Distinguish Dirty and Writeback in /proc/meminfo. What moves a folio between them?
2. What do dirty_background_ratio and dirty_ratio each do? Which one stalls your process?
3. Why is writeback per-bdi rather than global?
4. Give the durability ladder from write() to REQ_FUA, and say what each rung survives.
5. Why must you fsync() the directory after creating a file? What is the failure without it?
6. Write the atomic-replace pattern from memory, with all three durability steps.
7. What is the difference between fsync and fdatasync, and which should an append-only log use?
8. fsync() returned an error. Why is retrying it not a recovery strategy?
9. Explain journalling in six steps, and say what happens at mount for a transaction with and without a commit record.
10. Compare data=journal, data=ordered, and data=writeback. Which has a security implication and why?
11. How does a copy-on-write filesystem achieve the same guarantee without a journal? What ordering requirement remains?
12. Your benchmark shows 2 GB/s writes to a device rated at 500 MB/s. What did you actually measure, and what one dd flag fixes it?

Next: Lab 12 — Trace a Write — one write(), from the syscall to the device and back.