Our secondary FPBX v17 distro has run out of storage and we’ve lost web access to it.
First time round we had a storage problem was on our earlier FPBX 16 platform, and it was because _cache directory had grown too large.
In that earlier topic bksales commented : “This and runaway logs when modules break or licenses expire are usually the cause of storage issues.”
This time it seems to be to do with exactly what bksales said; but again I’m not sure what I can delete.
Looking at commands from the earlier topic I’ll paste some output below, and if someone could suggest what I I have to do, and what I might then update to stop it happening again I’d be very grateful.
root@freepbxp:/var/log# du -h -d3 | sort -rh | head -n30
root@freepbxs:/var/log# du -h -d3 | sort -rh | head -n30
4.4G .
2.9G ./journal/1926ac7812174f11810274e2076c0d18
2.9G ./journal
1.4G ./asterisk
16M ./installer
The journal folder is busy with lots of files like:
root@freepbxs:/var/log/journal/1926ac7812174f11810274e2076c0d18# ls
system@97044270e9794cc299c884878a569317-00000000006981ce-0006489ba4be0241.journal
system@97044270e9794cc299c884878a569317-00000000006a8088-000648c3c550eaae.journal
system@97044270e9794cc299c884878a569317-00000000006b80b0-000648ebad476e8e.journal
system@97044270e9794cc299c884878a569317-00000000006c7f77-00064913b6e09621.journal
system@97044270e9794cc299c884878a569317-00000000006d7e4e-0006493bced7e06a.journal
system@97044270e9794cc299c884878a569317-00000000006e7d83-00064963a66bccf0.journal
system@97044270e9794cc299c884878a569317-00000000006f7889-0006498dda494080.jour…
Certainly finding it hard to figure out what to delete.
Hard restart of the VM seems to have allowed it to boot actually, so haven’t rolled back just yet.
Has reduced storage use by 8.5GB by the look of it because the tmp folder is gone now root@freepbxs:/home/freepbx17# sudo du -h -d4 /boot /etc /home /opt /root /tmp /usr /var | sort -rh | head -n30
11G /var
6.1G /usr
4.4G /var/log
3.7G /usr/lib
3.0G /var/lib
2.9G /var/log/journal/1926ac7812174f11810274e2076c0d18
2.9G /var/log/journal
And I can now get into the WFE again.
This is great, and all because of a reboot!
So what might I need to do to control other file sizes like logs, or should I take the win here, update what needs updating and leave well enough alone?
As a rule of thumb on logs, if it’s got a date in the name and that’s not recent, it’s usually safe to delete. Snapshot first though, don’t just take my word for it.
Use something like WinSCP and click around the log directories. Sort by size and get rid of the big stuff. You might find some go a long way back. If they are all recent, deal with the issue.
I’d also check your scheduled backups are working. A failing backup in /tmp that doesn’t clean up after itself could easily explain where the 8.5GB came from.
