Excessive Filesystem Storage Space Usage on a new FreePBX Install

I run my FreePBX v16 on a DigitalOcean droplet. Yesterday the system crashed and I realized it was because I still had the droplet on the default 25GB of disk space. The droplet runs XFS on CentOS 7 so I was able to follow a simple guide using an XFS utility to expand FreePBX’s partition all the way to 120GB of space.

Today, to my surprise, I got an email saying I was almost out of storage again… Checked the system and saw I was using 107GB on the file system already. I went and deleted all the daily log files in /var/log and /var/log/asterisk which got me all the way down to under 60GB. However, the Storage widget on the Dashboard still shows a ton of used space under ‘Other’…

My question that I haven’t been able to find an answer to yet is: What is categorized under ‘Other’ storage and where can I go to clear up this used space?? This is a bit ridiculous that a still fresh installation of FreePBX hogs up this much storage this quickly, all I’ve done with the system so far is setup-type tasks in SysAdmin module and wiring up my SIP settings.

The only troubleshooting step I’ve taken so far to mitigate this moving forward is turning off a few logging verbosity options in the Advanced Settings.

yum install -y ncdu

and use that to analyze your server to see which directories/files are using the most space.

If you had that much logging, you probably had a ton of SIP traffic hitting your system and generating logs and CDRs. Check Asterisk SIP Settings; default value for Allow SIP Guest is YES and it will benefit you to set that to NO which will stop SIP scanners from hitting the default dialplan and generating logging and call detail records.

So I would guess that a portion of that consumed 60GB is in the mysql asteriskcdrdb database.

See what this yields…

du -sh /var/lib/mysql/*

The largetst directory is shown on the penultimate line of

du -hx --max-depth=1 /|sort -h

Interesting, is setting Allow SIP Guest to NO a popular best practice? Will that have any impact on any local customers trying to call me in the future? Or does this option not affect inbound calls in any way?


That’s an awesome command! Thanks for showing me. My result:

du -hx --max-depth=1 /|sort -h
0       /media
0       /mnt
0       /opt
0       /srv
44K     /root
236K    /tftpboot
13M     /tmp
64M     /etc
75M     /boot
549M    /home
2.4G    /usr
85G     /var
90G     /

Of course, I kept digging:

du -hx --max-depth=1 /var|sort -h
8.0K    /var/db
528K    /var/spool
348M    /var/cache
661M    /var/lib
1012M   /var/www
83G     /var/log
85G     /var

And found that Fail2Ban is the culprit here…

ls -lah /var/log/fail*
-rw------- 1 root root 77G Apr  9 23:08 /var/log/fail2ban.log

So is F2B running up a huge logging bill a common thing FPBX admins deal with or am I a special case? This makes me feel like my cloud Droplet that runs my FreePBX system is just getting absolutely hammered by bots… :grimacing:

Well, Fail2ban looks at connections and if you accept anonymous and guest access to UDP/5060 expect exactly what you are getting. (Yes it is a common thing, but nobody has to do that , many just do for some reason :slight_smile: )

1 Like

Dunno why I didn’t open the file until after my last reply BUT this is what has spammed the entire log file:

2024-04-09 03:24:02,661 fail2ban.filter.datedetector[2396]: DEBUG   Matched time template Year-Month-Day Hour:Minute:Second[,subsecond]
2024-04-09 03:24:02,661 fail2ban.filter.datedetector[2396]: DEBUG   Got time using template Year-Month-Day Hour:Minute:Second[,subsecond]

I was only able to load 7 million lines before I didn’t wanna wait anymore… What in the world is this supposed to mean? I haven’t touched any of the out-of-the-box date formatting anywhere in FreePBX yet…

That is because your datetime filter doesn’t match that that the log file is providing, What version of F2B are you using ? Did you add a Jail ? Did you modify anything otherwise ?

Well you might remember us being acquainted on my post from the other day regarding my F2B dying. The last time I messed with F2B was a reinstall with yum install fail2ban-fpbx.noarch and then @billsimon’s Getting Started steps for the SysAdmin module. I 110% have not touched a single thing regarding datetime anywhere!

I’m also fully updated:

# yum update
Loaded plugins: fastestmirror, versionlock
Repository sng-sng7php74 is listed more than once in the configuration
Loading mirror speeds from cached hostfile
sng-base                                                                                                                             | 3.6 kB  00:00:00
sng-epel                                                                                                                             | 2.9 kB  00:00:00
sng-extras                                                                                                                           | 2.9 kB  00:00:00
sng-pkgs                                                                                                                             | 3.4 kB  00:00:00
sng-sng7php74                                                                                                                        | 3.4 kB  00:00:00
sng-updates                                                                                                                          | 2.9 kB  00:00:00
No packages marked for update

# fwconsole ma upgradeall
No repos specified, using: [standard,extended,commercial] from last GUI settings

Up to date.
Updating Hooks...Done

Not super sure how to verify my Fail2Ban version but I’m presuming based on the above output it should be up-to-date. But yeah otherwise I definitely haven’t messed with anything else. I’m just trying to have a cheap business phone number for myself, nothing crazy! I’m trying to keep this simple but some of these bizarre issues I’m running into are making it a bit difficult!! :face_with_head_bandage:

The Distro is notorious for supplying woefully out of date Fail2Ban, fail2ban-client version will get your current version , the latest stable version is 1.0, but your problem is anomalous, only funky jails need the datetime format messed with and only funky logfiles are not parseable

Ahh jeez, I have v0.8.14, gonna try and update it…

@dicko So which of these packages do I actually need for FreePBX to function as expected? Moreover, you mentioned 1.0 is latest stable, and I see that on the GitHub as well, is this not recommended by Sangoma or something? The below repo is only sporting a max version of 0.11!

# yum list fail2ban*
Loaded plugins: fastestmirror, versionlock
Repository sng-sng7php74 is listed more than once in the configuration
Loading mirror speeds from cached hostfile
Installed Packages
fail2ban-fpbx.noarch                                                           0.8.14-78.sng7                                                      @sng-pkgs
Available Packages
fail2ban.noarch                                                                0.11.1-9.el7.2                                                      sng-epel
fail2ban-all.noarch                                                            0.11.1-9.el7.2                                                      sng-epel
fail2ban-firewalld.noarch                                                      0.11.1-9.el7.2                                                      sng-epel
fail2ban-hostsdeny.noarch                                                      0.11.1-9.el7.2                                                      sng-epel
fail2ban-mail.noarch                                                           0.11.1-9.el7.2                                                      sng-epel
fail2ban-selinux.noarch                                                        0.11.1-9.el7.2                                                      sng-epel
fail2ban-sendmail.noarch                                                       0.11.1-9.el7.2                                                      sng-epel
fail2ban-server.noarch                                                         0.11.1-9.el7.2                                                      sng-epel
fail2ban-shorewall.noarch                                                      0.11.1-9.el7.2                                                      sng-epel
fail2ban-systemd.noarch                                                        0.11.1-9.el7.2                                                      sng-epel
fail2ban-tests.noarch                                                          0.11.1-9.el7.2                                                      sng-epel

I can’t speak for Sangoma but 0.11 is plenty good

Hold up. Don’t replace distro packaging. It will break dependencies.

From your other thread, it seems you made some changes to the fail2ban filters before getting things working again by updating/instaling sysadmin. So it would be a good idea to retrace your steps.

Other systems are not experiencing this. Something’s broken on yours.

I haven’t made any changes at all to the F2B filters, I don’t even know why I would do that or what I would change. I’m not an expert on F2B, I’ve been under the impression that you just deploy it and let it do its thing.

I literally just deployed your 1-click droplet, started setting up the basic PBX config, due to my issues in the other post I uninstalled F2B then reinstalled using the distro package. I did your module fix we discussed. That makes up the totality of my interaction with this system’s F2B.

So if I broke something I have no idea how. I feel like the steps I’ve taken on my FreePBX instance the past week are pretty typical of an average user! I’m also unsure how to fix this if something’s broken?

Last night I deleted the 80GB Fail2Ban log, woke up this morning to another 38GB one…

Just to bring this topic full-circle even if it’s quite an anomaly… @billsimon and I collaborated on this further, his remarks:

And I finally don’t have insanely huge Fail2Ban logs anymore. It’s also good to ensure your /etc/fail2ban/fail2ban.conf has a loglevel = 3 line.

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.