Finding the du – df difference.
Long time system admins will have run into this from time to time: a file system appears to be full when you look at it with df but not when you use du to check it out.
Today, this problem appeared dramatically. Our monitoring system reported that the root file system one of our servers, cloudhost07, was at 96% utilization. I logged onto that server and used df and du to start looking for the culprit:
[root@cloudhost07 /]# du -hsx / 1.3G / [root@cloudhost07 /]# df -h / Filesystem Size Used Avail Use% Mounted on /dev/vg/lv_root 7.7G 7.0G 354M 96% /
The two bold numbers should be about the same. Large differences like the one we see here mean that a file has been deleted from the filesystem but some process still has it open. This happens from time to time and it’s not usually a problem. In our case, though, it’s happened because a verbose process is writing log entries to a log file that hasn’t been properly rotated.
We “rotate” log files to keep them from growing without bound and filling a file system. There are a few ways to get this done: (1) the process can do it internally, (2) we humans can do it manually from time to time or (3) we can use an independent program like logrotate to do it for us. Humans are not good at repetition on the scale and at the accuracy needed to manage computers. So we usually go with item (1) or (3). Either way, the way to do the rotation is:
- check the current log file: does it need rotation? Yes? Proceed:
- rename the current log file (mv daily.log daily.log.0)
- create the new log file (touch daily.log; chmod/chown daily.log)
- signal the process to close the filehandle to it’s current log file and open one to the new log file, thereby change the current to old and the new to current.
At step four, the process starts writing into the new log file, completing the rotation processes. There’s more to it, but that’s the gist.
Imagine what happens if you forget to do step four. Or can’t for some reason. The process keeps writing into the current/old file and the log rotation process never sees the new/current file get bigger. And the file grows without bound, invisibly.