It’s quite easy for a disk to have a large number of inodes used even if the disk is not very full.
You will get “no space left on device” errors despite having substantial free space on device. This is due to you might no longer have free inodes. df -h (check free space) / df -i (to check free inodes).
An inode is allocated to a file so, if you have gazillions of files, all 1 byte each, you’ll run out of inodes long before you run out of disk.
It’s also possible that deleting files will not reduce the inode count if the files have multiple hard links. As I said, inodes belong to the file, not the directory entry. If a file has two directory entries linked to it, deleting one will not free the inode.
Additionally, you can delete a directory entry but, if a running process still has the file open, the inode won’t be freed.
My initial advice would be to delete all the files you can, then reboot the box to ensure no processes are left holding the files open.
If you do that and you still have a problem, let us know.
By the way, if you’re looking for the directories that contain lots of files, this script may help:
# count_em - count files in all subdirectories under current directory.
echo 'echo $(ls -a "$1" | wc -l) $1' >/tmp/count_em_$$
chmod 700 /tmp/count_em_$$
find . -mount -type d -print0 | xargs -0 -n1 /tmp/count_em_$$ | sort -n
rm -f /tmp/count_em_$$
Try to find if this is an inodes problem with:
[crayon-5b4e79033e3ce420385286 ]df -ih
Try to find root folders with large inodes count:
[crayon-5b4e79033e3d0655459547 ]for i in /*; do echo $i; find $i |wc -l; done
Try to find specific folders:
[crayon-5b4e79033e3d2645934643 ]for i in /src/*; do echo $i; find $i |wc -l; done
If this is linux headers, try to remove oldest with:
[crayon-5b4e79033e3d4720827623 ]sudo apt-get autoremove linux-headers-3.13.0-24
Personally I moved them to a mounted folder (because for me last command failed) and installed the latest with:
[crayon-5b4e79033e3d7012241181 ]sudo apt-get autoremove -f
This solved my problem.