an old sysadmin once told me "on every drive, write 20gb to a file as ballast to delete in an emergency if you run out of disk" and in my old age, i am passing it on to you
Absolute galaxy brain move. Incredible. I stand in awe. This person was the only true sysadmin, the rest of us were just pretenders to the throne.
Also, another gem to keep in your back pocket in case you need it...
Sometimes, when you really run out of disc space, there isn't enough disc space to delete the ballast file (Try not to think about it too hard.):
rm: can not remove Ballast: No space left on device
If this happens, feed it to the disc space bird:
:> Ballast
The disc space bird will eat all the contents of the file, leaving it at 0 bytes, and you with enough disc space to delete things again!
we are not going to do this
but it's pretty cool
especially the disc space bird. that's hilarious
if you're using ext4 or another Linux-first filesystem, you actually have more disk space than the tools tell you about
there's an extra percent or so that is hidden from the free-space reports and can only be accessed by root. stuff running as non-root users is told "can't do that, no free space" before it's actually true.
of course if all your daemons run as root, then the logs or whatever that are filling your disk to the breaking point will also fill the extra space. good distros will run each daemon as its own user because that's the modern way, but you may still run into trouble with your own custom services if you didn't set up a user for them
this means that if you are ever trying to free up disk space and everything is jammed so tight that it can't do that, sometimes you can get it unjammed by doing stuff as root
