PBX took a dump, all inbound calls stopped working

Really strange set of issues. Nothing has been changed on the system was chugging away for at least a week. I did add in some more misc extensions but probably unrelated. At 5 today, people reported calls were no longer coming in. At that same time all the trunks went offline as well.

As of right now, it seems everything is working other than really strange behavior on the inbound call. It will come into the system from the outside, but it looks like one call spawns many calls in the system. So many that of course it’s unusable. I had only found that after changing the destination from my IVR to a single extension.

I am at a complete loss as to where to look to troubleshoot this one. I really don’t want to spend all night rebuilding from scratch, as I have done a fair amount of customization to get it to this point. Any ideas?

Got a copy of a log file where the problem starts?

Well I would have whatever is in /full, sure. The system is in a VM, I have shut it down and started rebuilding a new one from scratch. I had tried to recover from a backup and it actually got worse. I took a current backup from the slightly less broken current version. I plan to snapshot the base install, restore the backup into a fresh install and see if it was some core piece that got corrupted and not a config. If that doesn’t work, well I’ll just have to bite the bullet. I looked at the logs from the time it failed and honestly I didn’t see anything that jumped out. I’ll post them up and see if more trained eyes can see anything.

Interesting. When you say there was a lot of spawning of calls, it kind of sounds like to me that, if its not some DOS attack on your PBX, that you have a loop. Maybe extensions that are forwarded to each other? Call goes to the first extension, which is forwarded to the second one, then back to the first, ping pongs back and forth?

Perhaps you had a final destination for a call that sent the call back to the beginning of your dialplan, looping over and over?

The logs would definitely assist on the troubleshooting of your problem.
Sounds like things got ugly for you trying to fix it though. That is super frustrating, I feel your pain.

Yeah it wasn’t any kind of config. It was all working fine, and then just went haywire. Something of note, as I wake up from my 4:30am end time, lol… At almost exactly the same time, my primary DNS server shut off abruptly. I didn’t notice it really, as my secondary was serving up requests just fine. At least I THOUGHT it was. at about 2am I was trying to activate the new build of FreePBX, and it kept complaining. I can’t recall the exact message but Google said it was likely due to DNS issues. It was then I realized that my main DC for DNS was down. After trying in vain to set google DNS on the PBX itself manually, I made the drive in to the office and powered back up the DNS server, and the request to activate went in without a hitch. Very strange, I wouldn’t think it could be related but man it’s too close to ignore. It shut down at 4:52p, and my users complained about the outage from 5p onward. Trunks were up and down (DNS would need to resolve those) It did not however magically fix the original server, I still had to rebuild.

It doesn’t logically explain the call activity, maybe I don’t have all the pieces. Maybe it’s nothing, heh. I had assumed when I recovered from a full backup that it would create the system as it was. Trunks, extensions, etc. I had selected ‘full backup’ in the backup regimen. It recovered tftp folder, and the /asterisk folder, and something else I believe. It saved some time, but I’ll have to do some more research on that front.