FreePBX locked up, GUI unresponsive, comes back up on it's own?

I had a customer last week tell me their system stopped responding, showed a red error in the Asterisk server status area, and would not reload. Logs from that day have quite a few of these:

Unable to connect to manager localhost:5038 (110): Connection timed out

Don’t see them on any other days and it’d make sense that that would cause things to break. I’d rather they don’t go through it again; we are trying to move them to the distro, but for the time being they are stuck on an old version because of some special requirements that means I can’t upgrade them easily. FPBX version is 2.9.0.7, asterisk 1.8.6.

If you’ve seen this happen, or have any ideas why it would happen spontaneously, let me know, I’d appreciate it.

Bad hardware? full disk? the list goes on, what was the exact “red error”?, what happened when they tried to reload? what service was trying connect to 5038 on the local network? what is their current distro?

Without any of that necessary information, I would suggest to you that your client’s system is indeed broken, just as you diagnosed, and will probably remain so unto you do your due diligence as the vendor.

(all the mind readers left the forum last month)

Sorry, that error was from /var/log/httpd/error.log

The “red error” was as I said, under server status in FreePBX. Menu options were greyed out. The server is being hosted on a virtual machine, been running for ~18mo with no issues. No other VMs on this cluster have any issues. Reloading asterisk did not help.

netstat -an | grep 5038

tcp 0 0 0.0.0.0:5038 0.0.0.0:* LISTEN
tcp 0 1 10.100.151.151:43202 10.96.1.115:5038 SYN_SENT
tcp 0 1 10.100.151.151:56467 10.100.52.207:5038 SYN_SENT
tcp 0 1 10.100.151.151:56463 10.100.52.207:5038 SYN_SENT

We are running FOP2 on this machine, stopping it did not improve things. It’s happening again, right now. I am unable to open the FreePBX administrator interface. I’ll update this if I find out more.

That means that FreePBX can’t connect to the Asterisk server as whatever user and password you have in /etc/asterisk/amportal.conf. I would suspect runaway memory or processor usage by asterisk, I advise you to install sysstat and configure it to run in the future. For now a quick

top

might help.

To check for users of 5038 connections

for i in netstat -ant|grep 5038|awk '{print $4}'|cut -d":" -f 2|grep -v 5038;do lsof -i :$i;done

It was an issue with name resolution. Localhost was resolving to IPs other than the local address because the server is not part of the AD domain. Anyway, I’ve resolved this.