Hello, we are a mid sized company with around 60 Freepbx servers hosted on Vultr.
I have a handful of severs that appear to be crashing every few days. When the issue occurs, you are able to touch the dashboard of the GUI but unable to navigate further. Fwconsole restarts do nothing. Once you restart the server on vultr it becomes operational again.
Each time this has happened I have checked the asterisk logs. And what I am finding on the days that a server crashes is:
Each of these servers have a taskprocessor warning, the thing is though, these servers are from 3-10 extensions each. Nothing crazy no 5 hop ring groups or huge queues. One server is just a ring all 3 user hunt group, no IVR.
I’ve read through a couple of threads in regards to taskprocessor issue but 1 of the threads was in regards to a larger phone server and the other did not appear to have a lot of information.
What exactly happens when one of these systems “crashes” as you say? Does the box itself become unresponsive and require a reboot, or is it only the FreePBX GUI that stops responding? Are phones losing registration? Do the trunks drop off?
You are able to SSH into the server, when we try fwconsole restart we just receive asterisk unable to startup/shutdown errors over and over until the Vultr server is hard restarted.
Calls go into dead air for around 60 seconds followed by a busy signal hangup. Extensions also become unregistered.
I thinking about setting up freepbx server restart cronjobs on these in the mornings to see if it clears out the taskprocessor before it possibly reaches the state requiring a hard server restart.
I’m still unsure if the taskprocessor issue is connected but it is curious that it appears in logs for each of the servers before the server “crashes”.
With only 5/60 servers affected, I’d be tempted to isolate one over the weekend by temporarily disabling trunks or blocking external SIP sources to see if the problem disappears. If it does, you’ve established that some external traffic is involved rather than assuming the pjsip/distributor warning is the root cause.
I just came to say I experienced my first “crash” of a FreePBX system this morning. I ran v16 for a few years and we just moved to v17 this summer. I never experienced any issues with v16, but today I attempted to apply settings after making a change to an extension’s account code and all phones dropped offline. This is a VM in ProxMox and the qemu guest agent stopped responding as well. The web interface continued to work and I was able to gracefully reboot the machine via System Admin and everything seems to be back to normal. I’m not sure where to look to determine the cause, but it was a first for me!
You need to be more specific on what a crash is. A real “crash” in Asterisk is going to through specific type errors. You’ve not supplied any real pertinent information about the server such as what kind of resources it has, RAM, CPU, etc.
This is not a crash it’s a warning it means the taskprocessor has more than 500 requests in the queue. This could be caused by many things including a misconfigured setup, network issues or the system having resource issues.
So you need to be more specific about what is actually happening here. Because right now you’re not showing anything that indicates a crash.
I have experienced system unresponsive as well. This just happened to us now. I was going to add another DID to the system and when I click on the “Add Inbound Route” button it spins and spins without bringing up the screen to add it. Eventually clients get disconnected from the PBX and we see the trunks aren’t connected to the upstream provider anymore. The only fix at this point is to restart the asterisk. This is probably about the 3rd time since we’ve been running on FreePBX 17 for that last 7 months that it’s happened. At the time it happens CPU and memory are fine. I would love some helpful advice on troubleshooting this issue.
Inbound / outbound calls could not be placed or received. When trying to access the GUI I could land on the dashboard but I would not be able to navigate through the GUI as it would just spin for example when you tried to select asterisk info or system admin settings.
The only thing that would resolve this issue is hard restart on the physical server and then a fwconsole restart.
I had this happen on multiple Freepbx 17 servers, and each time I found taskprocessor errors. After I ran core module updates it stopped happening.
Again, you aren’t providing any context in this. A service hanging or something else happening could be the cause. Without looking at logs or any other activity on the system you actually don’t know what is causing this problem. A hard restart resets everything including clearly memory cache, etc.
You haven’t provided any server specs or what this system does overall. I have installed FreePBX v17 on numerous systems and haven’t had this problem.
I’ve also had task processor warnings, it happens on systems that get busy or don’t have enough resources to deal with the load.
These are sub 10 user servers with no more than 2 active call sessions at a time.
Each time they went down I went through the weeks worth of full asterisk logs. I understand that the task processor error would be a symptom and not the cause (if it was related to the issue), it was the only thing in the logs that jumped out to me.
There’s part of the problem. You should be using at least 2MB of RAM. But again, without knowing exactly what is being done on the server you might need more CPU as well.