$ df -P /dev/sda3 Filesystem 1024-blocks Used Available Capacity Mounted on /dev/sda3 8256952 7007368 830160 90% /var $ df -i /dev/sda3 Filesystem Inodes IUsed IFree IUse% Mounted on /dev/sda3 524288 21548 502740 5% /var
So far, so bad. The disk is full and we can’t blame it on inodes. Since it’s /var, it’s probably just some log files…
$ du -sh /var 1.9G /var
…or not?
One of the features of Unix filesystems (compared to Windows) is that “deleting” a file does not necessary delete it. The file will be unlinked (so it doesn’t appear in a file system anymore), but still remains usable. Open file handles to this file are tracked, and the file will only be deleted when the last handle is closed (this is why Windows has the annoying “Cannot delete file, it’s open somewhere [and I won’t tell you where]” and Linux doesn’t). df obviously has to include unlinked files, but du doesn’t. For that, we have lsof:
When +L is followed by a number, only files having a link count less than that number will be listed. (No number may follow -L.) A specification of the form “+L1” will select open files that have been unlinked. A specification of the form “+aL1 ” will select unlinked open files on the specified file system.
$ lsof +L1 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NLINK NODE NAME apache2 5102 www-data 2w REG 8,3 627670 0 450637 /var/log/apache2/error.log.1 (deleted) apache2 5102 www-data 7w REG 8,3 2030629 0 450922 /var/log/apache2/other_vhosts_access.log.1 (deleted) apache2 5102 www-data 8w REG 8,3 2030629 0 450922 /var/log/apache2/other_vhosts_access.log.1 (deleted) …and fifty other big log files…
It actually were log files to blame for this – Apache holds a handle to its closed log files after logrotate rotated (and deleted them). An Apache restart after logrotate runs fixes this particular problem.
(Thanks to ServerFault for pointing to lsof +L1!)