Yesterday one of my development servers decided it was going to do some very strange things.Wordpress and other websites stopped working properly, I got session errors when trying to usePHPMyAdmin, I couldn't upload files through web forms (the server complained there was no temporary directory). So I logged in to try and work out what was going on. The temporary directory was there and had the correct permissions, however if I tried to create a file in it I was told:
$ touch /tmp/testfile
Unable to create file /tmp/testfile: No space left on device
So I must have run out of disk space, which is odd as I had loads last time I checked.
$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 15G 8.5G 6.5G 57% /
devtmpfs 299M 112K 299M 1% /dev
none 308M 0 308M 0% /dev/shm
none 308M 64K 308M 1% /var/run
none 308M 0 308M 0% /var/lock
none 308M 0 308M 0% /lib/init/rw
/dev/sdc1 40G 6.4G 32G 17% /home
Oh, I have plenty of disk space! What the hell is going on then? As my server is an Amazon EC2instance my first thoughts were there was a problem with the block storage. So I spent an hour or so trying to find any clues in their forums and got nowhere.
After another few hours of scouring the internet for people having similar problems and finding nothing at all I was about to give up. As a last ditch attempt to find the solution I checked myMunin stats for the server and immediately I noticed that the inode graphs for one of the mounted disks had been rising steadily over the last few weeks and had just reached 100%!!!
$ df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sda1 983040 983040 0 100% /
devtmpfs 76490 1957 74533 3% /dev
none 78747 1 78746 1% /dev/shm
none 78747 34 78713 1% /var/run
none 78747 2 78745 1% /var/lock
none 78747 1 78746 1% /lib/init/rw
/dev/sdc1 2621440 13238 2608202 1% /home
So then, where are all these files? There must be thousands of them to be using up 100% of just under a million.
To count all the files in a directory and all it's subdirectories:
$ for i in /*; do echo $i; find $i | wc -l; done
Then you can narrow down your search by replacing the /* for any directory that has an unusually large number of files in. For me it was /var
$ for i in /var/*; do echo $i; find $i | wc -l; done
Eventually I narrowed it down to the reports being held by the Squid Proxy server report generator sarg so a simple fix was to clear out all the old reports and stop sarg from auto generating reports every day.
$ rm -rf /var/log/sarg/*
And thats it! Server fixed and back up and running without any problems. All I have to do is remember to keep an eye on any autogenerated logs and reports and make sure that old ones are actually being deleted!