Running FreePBX 22.214.171.124 and Asterisk 13.17.0
Recently used the “warm spare” method to move to a new server (new VM on KVM/proxmox)
The server has about 120 remote extension, and had no real problems before.
I posted about this crash issue yesterday here, but my hypothesis was off: Media_index.c: Failed to stat
Today we had a ton of users call and say their phones weren’t working. Funny thing is, they show OK with IP address in peers list when running “sip show peers” in cli.
So we did a fwconsole restart, and things started working again.
This crash has happened three times this week already. Sunday morning, yesterday morning, and today.
I starting digging through the logs, and these are the errors that may or may not be the cause. I’m hoping someone can give me some insight. Here are some error examples:
These ones show all over the logs, way before, way after, and right around the crash time:
[2017-08-22 09:30:59] ERROR[C-00000028] pbx_functions.c: Function PJSIP_HEADER not registered
These ones yesterday were fairly close to before the crash, but there were none today before the crash:
[2017-08-22 09:38:08] ERROR netsock2.c: getaddrinfo("2605:e000:6045:3a00:20b:82ff:feac:c151:13312", "(null)", ...): Name or service not known [2017-08-22 09:38:08] WARNING chan_sip.c: Could not resolve socket address for '2605:e000:6045:3a00:20b:82ff:feac:c151:13312'
These existing on all three instances:
Line 33144: [2017-08-20 10:34:38] ERROR chan_sip.c: Serious Network Trouble; __sip_xmit returns error for pkt data
And finally, these ones look the like the most likely culprit, but didn’t show on Yesterday’s crash (just today and Sunday’s):
*note that the difference between Today’s and Sunday’s, vs Yesterday’s, is that the former showed all endpoints “OK”, though they truly weren’t, the latter showed only about half of them
[2017-08-23 13:42:49] ERROR astobj2.c: FRACK!, Failed assertion bad magic number 0x0 for object 0x3de7430 (0)
So all that to say, I hope someone can help us find the root cause of all this.
Again, this server was a fresh v13 FreePBX server that we just “warm spare” copied to from an existing server. The existing was running on an ESXi host, fully updated to …66-21. We fully updated the fresh VM to 66-21 as well before running the backup/restore. The new server is a VM on KVM/proxmox.