Hello FreePBX experts, hope everyone is doing great.
I installed 2 separate brand new FreePBX16 instances from a 1 click droplet in digitalocean, and both of them seem to have this issue with fail2ban producing so much unnecessary log that it ends up taking up most of the disk space. One of these instances has 200+ extensions and thus generates even more log and eventually it stops working because of no disk space.
This is the log it produces. Hundreds and hundreds of logs like this.
First question. What are these logs anyway? I tried switching fail2ban.conf to log level 1, so it only produces error logs, but it still keeps logging these authentication messages.
Why do they keep coming and why is it trying to authenticate these extensions constantly?
How do I stop it from trying to authenticate already authenticated extensions?
When I check var/log/asterisk/fail2ban files, it seems like logrotate doesnât delete the files that are too old. Why doesnât the logrorate work out of the box for such an established PBX like FreePBX16? Itâs been an issue with multiple FreePBX16 brand new instances.
These are not fail2ban logs, they are the Asterisk logs that fail2ban uses as input.
Asterisk will challenge for authentication for all attempts to start calls by endpoints that use authentication, and for all out of dialogue requests from them, such as re-registrations. Endpoints can also choose to volunteer authentication.
Note that authentication on REGISTER only authenticates the REGISTER. Especially with UDP, failing to challenge every session starting INVITE would lay you open to toll fraud, as, particularly for chan_pjsip, only the user name is, otherwise, needed in order to make a call in your from-internal context, which would normally be allowed to make chargeable calls.
Iâm unable to answer the logrotate part of the question.
Thanks for you response. Would you happen know why there are so many of these challenges? This PBX has 200+ extensions with Adtrans as clients registered to those extensions. There are so many challenges in the log that they take gigabytes of storage in the matter of days. I doubt that the residents are making that many calls. If itâs just one challenge per call, there is no way there are thousands of calls are made.
Is that something that the Adtrans are constantly trying to re-authenticate or something like that?
There is no way a simple text can take up so much storage.
You may have a short re-registration period configured, or the phones may be forcing one. You could use sngrep to find out what traffic you are getting.
(Short re-registrations are often used when there is a NAT router that is relying on dynamically created port forwarding rules, rather than having all necessary port forwarding pre-configured.)
Our Adtrans registration time out is configured to 5 minutes. Is that considered short? If it re-registers every 5 minutes, you think itâs supposed to create so much log that 20GB of space is not enough for them withing only a few weeks?
Also is there an option to reduce this excessive logging? If something is successful I donât need to see it. I only need to see if there are unsuccessful attempts. I checked in Settings > Asterisk Log File Settings and the Security are all turn off.
My question is, why would these challenges take up so much space. Whatever the amount of extensions there is, FreePBX should be able to rotate the logs and not let it take up so much space. Both brand new installations of FreePBX16 are doing that.
I just donât understand why would this be an issue on a brand new FreePBX right out of the box? Why would it not work properly on such an established PBX platform and I need to dig into log files as soon as I install it.
Here is the logrotate file. It looks the same as FreePBX15 to me. But I never had these issues on another PBX.
/var/spool/mail/asterisk {
daily
missingok
rotate 7
notifempty
sharedscripts
create 0660 asterisk mail
su asterisk mail
}
/var/log/asterisk/messages
/var/log/asterisk/event_log
/var/log/asterisk/full
/var/log/asterisk/dtmf
/var/log/asterisk/fail2ban {
daily
missingok
rotate 7
notifempty
sharedscripts
create 0640 asterisk asterisk
su asterisk asterisk
postrotate
/usr/sbin/asterisk -rx âlogger reloadâ > /dev/null 2> /dev/null
endscript
} #This comment is to fix rpm file replacing #Config file built on Fri Sep 3 10:23:55 UTC 2021
And here are the fail2ban log files being generated. How can I just keep 3 files rotated and no more than 3? I donât need so much unnecessary log. Also how do I reduce fail2ban to only notify me if there is an issue or unsuccessful login attempt. I donât need successful login attempts to be logged. Iâd rather use the CPU power for something else.
SIP doesnât have logins. These are authentications, and they apply to individual requests. A successful authentication is more important to know about that a failed one, if it turns out to have come from an attacker.
You didnât post your fail2ban conf file and you seem to have two ârotatesâ configured but chosing just one, either renaming to date or just the standard by number and changing rotate from 7 to 3 should fix it.
You donât say how you installed FreePBX but each recipe usually provided a generic rotation of logs it generates, you would have to complain to the relevant distributor.
This is my fail2ban.conf file. I changed loglevel from 3 to 1 a while ago but all these challenge messages never went away.
Fail2Ban main configuration file
Comments: use â#â for comment lines and â;â (following a space) for inline comments
Changes: in most of the cases you should not modify this
file, but provide customizations in fail2ban.local file, e.g.:
[Definition]
loglevel = 1
[Definition]
Option: loglevel
Notes.: Set the log level output.
1 = ERROR
2 = WARN
3 = INFO
4 = DEBUG
Values: [ NUM ] Default: 1
loglevel = 1
Even if I change logrotate from 7 to 3, some of the fail2ban individual files are like 3GB and more. It might still take up most of the disk space on the server. I need to reduce the amount of logging coming from fail2ban and there doesnât seem to be a way to do that. No one has ever told me that so far.
Also, if my logrotate is set to 7 by defail, why are there so many fail2ban files left with long numbers at the end like fail2ban-20230407. Why canât it keep the log to exactly 7 files, or 3 files if I set it to 3?
Iâm seriously considering turning fail2ban off. There were multiple posts like mine @dicko and you seemed to answer to each one with a different solution and non of those helped recude fail2ban logging. My fail2ban is eating up all of my disk space and it is getting redicolous. Out of the box FreePBX16 installed from a distro on a digitalocean cloud server. Nothing out of ordinary, yet FreePBX is failing to fix this.
The configuration you provided is for fail2ban output logging, but the log sample you provided is the Asterisk security log, which is the input to fail2ban. That is controlled by /etc/asterisk/logger.conf (which I think is owned by FreePBX in FreePBX systems), and, as far as I know, it is all or nothing.
Your directory shows 7 of these files, which is consistent with the logrotate configuration you provided. I presume the files with dates in their names are the outputs from fail2ban, which is what your log level setting will control. I would say the logrotates were doing what was asked of them.
A typical messages is 388 bytes (I did my rough calculations based on 200. 200 extensions with 5 minute registrations, would come to about 22.5MB a day (I think I got the number of 5 minutes a day wrong, before as well as the message length). However the actual registration interval could be as little as half the maximum, so we are at about 45MB a day. We are still out by about two orders of magnitude, so I think you need to look at exactly what traffic you are getting. Something may be registering far too often. The security log files will tell you which extension tried to authenticate, but you will probably need to use sngrep to really understand what is going on.
Demangled version of your file follows:
# Fail2Ban main configuration file
#
# Comments: use '#' for comment lines and ';' (following a space) for inline comments
#
# Changes: in most of the cases you should not modify this
# file, but provide customizations in fail2ban.local file, e.g.:
#
# [Definition]
# loglevel = 1
#
[Definition]
# Option: loglevel
# Notes.: Set the log level output.
# 1 = ERROR
# 2 = WARN
# 3 = INFO
# 4 = DEBUG
# Values: [ NUM ] Default: 1
#
loglevel = 1
As has been stated here in this thread already, the file /var/log/asterisk/fail2ban is generated by Asterisk, disabling fail2ban will have zero effect on this file. This file is verbose by design, pretty much any event that asterisk deems related to security is logged here. When enabled, fail2ban will watch this file for spurious activity and react accordingly.
I expect that the reason these files are so large is due to having your Asterisk services exposed to untrusted traffic. You should address this before you start worrying about logfile minutiae.
On the FreePBX distro, logrotate is configured by default to rotate daily and purge after 7 days. However, the log volume in your case is such that Asterisk is also rotating the files, thatâs where the .0, .1 and .2 files are coming from. Youâll want to remove those manually and in Settings â Logfile Settings set Log Rotation to none to disable the Asterisk rotations so that logrotate will take care of it.
No, my FreePBX is closed down and all these challenges come from the authorized extensions and all of them are showing successful authentication.
Still noone has answered why an out of the box PBX would have this issue and how to fix it.
The Adtrans that are registered to all these extensions are set to registration expire in 5 minutes. But even if they re-register every 5 minutes, why would there be so much unnecessary log that it takes up so much space.
A âbrand new FreePBX right out of the boxâ listens to connections to UDP/5060 . The remote address and local address can be spoofed but do a reality check on them anyway that they match where the accountID is expected to be , because of the above , many recommend not using that port or protocol, as they are invariably less noisy.
sngrep is an effective way to see incoming connections
I just logged into that FeePBX instance and it was about to overfill with the fail2ban logs again. Without erasing anything manually, I just ran this command to rotate the logs, and it deleted all the GBs of those logs.
logrotate -vf /etc/logrotate.d/asterisk
Could that mean my logrotate is not rotating those logs and not deleting them?
How can I troubleshoot logrotate and what it does in the system and find out if logrotate is the issue.
â