Disk full at 3am: finding it fast

A full disk takes services down in confusing ways: databases go read-only, logs stop, sessions fail to write. Here is the order that finds it fastest.

Which filesystem, and is it really space

df -h
df -i        # inodes - a filesystem can be full with space remaining

Check inodes early. Millions of tiny files exhaust the inode table while df -h shows free space, and every write fails with “no space left on device” while the disk looks half empty.

Where the space went

du -xh --max-depth=1 /var 2>/dev/null | sort -h | tail
find /var -xdev -type f -size +500M -exec ls -lh {} \; 2>/dev/null

-x matters: it stops du descending into other filesystems and mixing results.

When df and du disagree

This is the case that wastes the most time. df reports the disk full, du accounts for far less. The cause is a deleted file still held open by a process — the space is not released until the descriptor closes.

lsof +L1 2>/dev/null | sort -k7 -n | tail
lsof -nP | grep '(deleted)' | awk '{print $1, $2, $7}' | sort -k3 -n | tail

Classic cause: someone deleted a large log while the process was writing to it. Restarting that process releases the space instantly. If a restart is not acceptable, the file can be truncated through its descriptor:

: > /proc/<pid>/fd/<fd>

Immediate relief

journalctl --vacuum-size=200M
apt-get clean
find /tmp -type f -atime +7 -delete

Afterwards

Buying space treats the symptom. The real fix is rotation that works, retention that is bounded, and an alert at 80% rather than at 100%. The difference between those two thresholds is the difference between a scheduled task and an outage.