I run my FreePBX v16 on a DigitalOcean droplet. Yesterday the system crashed and I realized it was because I still had the droplet on the default 25GB of disk space. The droplet runs XFS on CentOS 7 so I was able to follow a simple guide using an XFS utility to expand FreePBX’s partition all the way to 120GB of space.
Today, to my surprise, I got an email saying I was almost out of storage again… Checked the system and saw I was using 107GB on the file system already. I went and deleted all the daily log files in /var/log and /var/log/asterisk which got me all the way down to under 60GB. However, the Storage widget on the Dashboard still shows a ton of used space under ‘Other’…
My question that I haven’t been able to find an answer to yet is: What is categorized under ‘Other’ storage and where can I go to clear up this used space?? This is a bit ridiculous that a still fresh installation of FreePBX hogs up this much storage this quickly, all I’ve done with the system so far is setup-type tasks in SysAdmin module and wiring up my SIP settings.
The only troubleshooting step I’ve taken so far to mitigate this moving forward is turning off a few logging verbosity options in the Advanced Settings.
If you had that much logging, you probably had a ton of SIP traffic hitting your system and generating logs and CDRs. Check Asterisk SIP Settings; default value for Allow SIP Guest is YES and it will benefit you to set that to NO which will stop SIP scanners from hitting the default dialplan and generating logging and call detail records.
So I would guess that a portion of that consumed 60GB is in the mysql asteriskcdrdb database.
Interesting, is setting Allow SIP Guest to NO a popular best practice? Will that have any impact on any local customers trying to call me in the future? Or does this option not affect inbound calls in any way?
.
That’s an awesome command! Thanks for showing me. My result:
So is F2B running up a huge logging bill a common thing FPBX admins deal with or am I a special case? This makes me feel like my cloud Droplet that runs my FreePBX system is just getting absolutely hammered by bots…
Well, Fail2ban looks at connections and if you accept anonymous and guest access to UDP/5060 expect exactly what you are getting. (Yes it is a common thing, but nobody has to do that , many just do for some reason )
Dunno why I didn’t open the file until after my last reply BUT this is what has spammed the entire log file:
2024-04-09 03:24:02,661 fail2ban.filter.datedetector[2396]: DEBUG Matched time template Year-Month-Day Hour:Minute:Second[,subsecond]
2024-04-09 03:24:02,661 fail2ban.filter.datedetector[2396]: DEBUG Got time using template Year-Month-Day Hour:Minute:Second[,subsecond]
I was only able to load 7 million lines before I didn’t wanna wait anymore… What in the world is this supposed to mean? I haven’t touched any of the out-of-the-box date formatting anywhere in FreePBX yet…
That is because your datetime filter doesn’t match that that the log file is providing, What version of F2B are you using ? Did you add a Jail ? Did you modify anything otherwise ?
Well you might remember us being acquainted on my post from the other day regarding my F2B dying. The last time I messed with F2B was a reinstall with yum install fail2ban-fpbx.noarch and then @billsimon’s Getting Started steps for the SysAdmin module. I 110% have not touched a single thing regarding datetime anywhere!
I’m also fully updated:
# yum update
Loaded plugins: fastestmirror, versionlock
Repository sng-sng7php74 is listed more than once in the configuration
Loading mirror speeds from cached hostfile
sng-base | 3.6 kB 00:00:00
sng-epel | 2.9 kB 00:00:00
sng-extras | 2.9 kB 00:00:00
sng-pkgs | 3.4 kB 00:00:00
sng-sng7php74 | 3.4 kB 00:00:00
sng-updates | 2.9 kB 00:00:00
No packages marked for update
# fwconsole ma upgradeall
No repos specified, using: [standard,extended,commercial] from last GUI settings
Up to date.
Updating Hooks...Done
Not super sure how to verify my Fail2Ban version but I’m presuming based on the above output it should be up-to-date. But yeah otherwise I definitely haven’t messed with anything else. I’m just trying to have a cheap business phone number for myself, nothing crazy! I’m trying to keep this simple but some of these bizarre issues I’m running into are making it a bit difficult!!
The Distro is notorious for supplying woefully out of date Fail2Ban, fail2ban-client version will get your current version , the latest stable version is 1.0, but your problem is anomalous, only funky jails need the datetime format messed with and only funky logfiles are not parseable
@dicko So which of these packages do I actually need for FreePBX to function as expected? Moreover, you mentioned 1.0 is latest stable, and I see that on the GitHub as well, is this not recommended by Sangoma or something? The below repo is only sporting a max version of 0.11!
# yum list fail2ban*
Loaded plugins: fastestmirror, versionlock
Repository sng-sng7php74 is listed more than once in the configuration
Loading mirror speeds from cached hostfile
Installed Packages
fail2ban-fpbx.noarch 0.8.14-78.sng7 @sng-pkgs
Available Packages
fail2ban.noarch 0.11.1-9.el7.2 sng-epel
fail2ban-all.noarch 0.11.1-9.el7.2 sng-epel
fail2ban-firewalld.noarch 0.11.1-9.el7.2 sng-epel
fail2ban-hostsdeny.noarch 0.11.1-9.el7.2 sng-epel
fail2ban-mail.noarch 0.11.1-9.el7.2 sng-epel
fail2ban-selinux.noarch 0.11.1-9.el7.2 sng-epel
fail2ban-sendmail.noarch 0.11.1-9.el7.2 sng-epel
fail2ban-server.noarch 0.11.1-9.el7.2 sng-epel
fail2ban-shorewall.noarch 0.11.1-9.el7.2 sng-epel
fail2ban-systemd.noarch 0.11.1-9.el7.2 sng-epel
fail2ban-tests.noarch 0.11.1-9.el7.2 sng-epel
Hold up. Don’t replace distro packaging. It will break dependencies.
From your other thread, it seems you made some changes to the fail2ban filters before getting things working again by updating/instaling sysadmin. So it would be a good idea to retrace your steps.
Other systems are not experiencing this. Something’s broken on yours.
I haven’t made any changes at all to the F2B filters, I don’t even know why I would do that or what I would change. I’m not an expert on F2B, I’ve been under the impression that you just deploy it and let it do its thing.
I literally just deployed your 1-click droplet, started setting up the basic PBX config, due to my issues in the other post I uninstalled F2B then reinstalled using the distro package. I did your module fix we discussed. That makes up the totality of my interaction with this system’s F2B.
So if I broke something I have no idea how. I feel like the steps I’ve taken on my FreePBX instance the past week are pretty typical of an average user! I’m also unsure how to fix this if something’s broken?
Last night I deleted the 80GB Fail2Ban log, woke up this morning to another 38GB one…
)