We are currently running FreePBX 2.9.0.14 with Asterisk 1.4.29.1. Yesterday we experienced a really strange issue that was repeated today. Our phone suddenly became unable to make a call both internally and externally. We could occasionally make a call in, but the audio was very choppy and intermittent. Other times the call went directly to voicemail. We were not experiencing any problems with our internet. I conducted many successful speedtests. Our router was not showing any signs or logs of an attack. Navigating to the Freepbx system webpage was very slow. The FreePBX system status frequently showed Web Server Timeouts.
It seems that our PBX suddenly became very unresponsive and slow. I checked the network connections to our PBX and there were no issues with the connection. Looking at the PBX status, the CPU load & memory were all very low. We have network probes using PRTG to monitor our network and the only issue that I could view was the PBX http web page ping being exceptionally long.
Can you please help me pinpoint the cause of these issues? I tried restarting the PBX numerous times. Like yesterday, the problem all the sudden cleared up without any signs of anything changing. Any help or locations for logs would be appreciated.
Well just randomly stabbing in the dark you’re running out of resources. Judging by your versions you are not a huge fan of updating things so you could be compromised.
So we can all understand the urgency of your problem, You installed a system that was based on code from 2010-01-14 (asterisk) and a FreePBX from sometime in 2011. On this system you have since performed zero maintenance or updates, correct?
I am pretty sure you are experiencing a DOS (denial of service) attack, check your /var/log/asterisk/*.log files to check.
Perhaps you should update to supported versions which would cover you from the well known and published and ultimately fixed insecurities of both asterisk and FreePBX in the last three years. Do that first and then someone can perhaps help.
I’ve kinda inherited this system from my boss. I’m not that familiar with Linux, so I didn’t want to “fix what wasn’t broken.” Now that we are having an issue I’m trying to isolate the problem. If upgrading our installation might very well fix the issue then I’ll put in the hours to bring me up to speed.
Whenever we have experienced previous DOS or outside attacks, we have been able to detect it via our router. Since our initial DOS attacks we have modified our rules so that the PBX is only accepting communications from the addresses of our trunks. I don’t believe it is a DOS attack, but i’ll check my log files.
May I suggest hiring someone who is. Though I am a big advocate of people learning to use FreePBX and Linux, the time to learn is NOT during business hours on a failing system.
That said you may want to Setup a (replacement) system with current (up to date) software that you can drop in place. This system can limp along while you do that.
Depending on resources you may wish to do the above as a weekend project. I would go pickup another hard drive so your current install is safe. Build the new system on a new drive and drop it in place.
The time to start a weekend project is Friday night by the way. I have seen many folks attempt this on a Sunday afternoon and not have phones Monday.
Sorry but there is no way out, you WILL have to be familiar with linux (many tutorials on line for that) and you will need to accept the fact that your system is very old, un-maintained and apparently broken, further to that the guys who are likely attacking you are VERY much cleverer than you or any of your current diagnostic abilities/filters. I suggest you have your boss get his check-book out rusty though it might be, no-one here will do it for him (or you ) for free.
Thank you for your comments. We restricted our internal traffic access for the PBX in case some internal source was bombarding our PBX. We haven’t experienced any issues with our PBX since.
I am going to attempt a weekend project to re-build our PBX setup a new installation of the latest FreePBX distro. For now the issue seems under control.
Just to wrap it all up. We had an interesting past couple of days. Our PBX had ended up dying. I was able to bring up a new installation of our setup on Free PBX v2.11.0.32 & Asterisk v 1.8.30. Along with our new installation we’re having a couple of problems, however they are different from the ones listed above.
I will start a new post as this one can be closed.