Okay. Anyone I can get some paid support from on an issue like this?
Sangoma? We’ve always had paid support. have you never tried it?
Did you ever contact Asterisk with your crash?
Did you try to remove your trunks?
I had same issue before, and there was problem with my sip trunks. something like misconfiguration.
Im going to move this back to our original issue because you are mixing up a FRACK! which is a memory leak which is your crash with “Serious Network Trouble” which is not related to the FRACK.
Looking through your previous replies on this I don’t believe you ever attempted to get the crash reports and send them to Asterisk.
This is a memory leak in Asterisk. As such it casues a crash. Even if you are doing something wrong Asterisk should not leak memory and therefor you should really go to the Asterisk issues tracker and see if they have any issues related to yours.
Which error did you have? The Serious Network Trouble, or FRACK?
Well, we’ve created them from scratch (new servers) and are having these issues still. That said, the IPs Ive noticed in the logs that related to the serious network trouble errors are not trunk IPs, but endpoints. Could be both though, just haven’t dug that deep to confirm/deny.
Here are our trunks, anveo, anveo 911, and then a FXO we have setup. Everything else are extensions:
ANVEO IN EXAMPLE
ANVEO OUT EXAMPLE
Hi Andrew. The only reason it’s come up is that there seems to be a relation. But obviously you know much more accurately. I will consider the two separate issues going forward.
I posted on the Asterisk community forums, and the issue tracker here: https://issues.asterisk.org/jira/browse/ASTERISK-27321
Where we’re currently at is that my backtrace isn’t showing any “symbolic information”. I tried installing debuginfo, but when I run the gdb command after I did that, I get these warnings, and the same “not very helpful” backtrace:
warning: the debug information found in "/usr/lib/debug//usr/lib64/asterisk/modules/format_jpeg.so.debug" does not match "/usr/lib64/asterisk/modules/format_jpeg.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug/usr/lib64/asterisk/modules/format_jpeg.so.debug" does not match "/usr/lib64/asterisk/modules/format_jpeg.so" (CRC mismatch).
Corey on the issue tracker said, “This looks like it’s saying that the asterisk binary package does not match the asterisk debuginfo package. My guess is that this is a packaging bug with FreePBX. The debuginfo packages are extracted from the binary packages, and it’s only usable if the CRC of each file matches what is expected.”
Can you provide any assistance with helping me generate a more useful backtrace? I believe I’ve followed the instructions here correctly: https://wiki.freepbx.org/display/SUP/Providing+Great+Debug
Asterisk 13.7.0 is old so you should first upgrade to the latest Asterisk 13. What you did was install debuginfo for a newer version of Asterisk without upgrading your asterisk. After you fix this your older dumps will no longer be valid.
Example of when this happens (Notice that 13.17.0 no longer matches debuginfo of 13.17.1):
[root@Andrew13 ~]# yum list asterisk13* Loaded plugins: fastestmirror, kmod Loading mirror speeds from cached hostfile Installed Packages asterisk13.x86_64 13.17.0-2.shmz65.1.176 @pbx asterisk13-addons.x86_64 13.17.0-2.shmz65.1.176 @pbx asterisk13-addons-bluetooth.x86_64 13.17.0-2.shmz65.1.176 @pbx asterisk13-addons-core.x86_64 13.17.0-2.shmz65.1.176 @pbx asterisk13-addons-mysql.x86_64 13.17.0-2.shmz65.1.176 @pbx asterisk13-addons-ooh323.x86_64 13.17.0-2.shmz65.1.176 @pbx asterisk13-core.x86_64 13.17.0-2.shmz65.1.176 @pbx asterisk13-curl.x86_64 13.17.0-2.shmz65.1.176 @pbx asterisk13-dahdi.x86_64 13.17.0-2.shmz65.1.176 @pbx asterisk13-debuginfo.x86_64 13.17.1-1.shmz65.1.179 @pbx
Thank you. I will update Asterisk on one VM and monitor results.
I’ve got a fairly significant number of production PBX’s running V12 and V13 under Proxmox 4.x on Dell hardware (multiple PBX’s per box) and have never had a single issue with this setup.
Last week however I deployed a new i3 NUC with Proxmox5 and a brand new FreePBX13 distro install and have seen this crash over the weekend but Proxmox was still running fine. Considering how long it’s been since I’ve seen a FreePBX system crash this has me pretty alarmed.
I’ve yet to see anything in the logs so far to indicate what could have caused this or what went wrong but I’ll certainly be going over things today looking for any clues.
Generally something to consider about frack errors is that freepbx can’t cause them as it’s a memory issue with asterisk and the system. So they should always be reported to asterisk.
ERROR[C-00003919] astobj2.c: Excessive refcount 100000 reached on ao2 object 0x142f648
So this morning, we had hundreds of fracks. I noticed in the logs, that it happend right after one of our users unplugged there phone. The next thing noticable in the log is Serious Network Trouble errors, referring to the IP of the router that the phone was on (no longer connected, and no other endpoints on that IP), and then right after, the first of hundreds of FRACKs.
Here’s part of the log. The first line is when the phone was unplugged.
[2017-10-18 06:32:31] NOTICE chan_sip.c: Peer '235' is now UNREACHABLE! Last qualify: 23 [2017-10-18 06:32:31] VERBOSE chan_sip.c: Extension Changed 235[ext-local] new state Unavailable for Notify User 201 [2017-10-18 06:53:14] WARNING chan_sip.c: sip_xmit of 0x7f6784903580 (len 759) to 188.8.131.52:59972 returned -2: Interrupted system call [2017-10-18 06:53:14] ERROR chan_sip.c: Serious Network Trouble; __sip_xmit returns error for pkt data [2017-10-18 06:53:15] WARNING chan_sip.c: sip_xmit of 0x7f678421a740 (len 760) to 184.108.40.206:59972 returned -2: No such file or directory [2017-10-18 06:53:15] ERROR chan_sip.c: Serious Network Trouble; __sip_xmit returns error for pkt data [2017-10-18 06:53:16] WARNING chan_sip.c: sip_xmit of 0x7f678417cbc0 (len 760) to 220.127.116.11:59972 returned -2: No such file or directory [2017-10-18 06:53:16] ERROR chan_sip.c: Serious Network Trouble; __sip_xmit returns error for pkt data [2017-10-18 06:53:25] ERROR astobj2.c: FRACK!, Failed assertion bad magic number 0x0 for object 0x22c55a0 (0) [2017-10-18 06:53:26] VERBOSE logger.c: Got 21 backtrace records [2017-10-18 06:53:26] VERBOSE logger.c: #0: [0x603fb2] /usr/sbin/asterisk(__ast_assert_failed+0x88) [0x603fb2] [2017-10-18 06:53:26] VERBOSE logger.c: #1: [0x45d11a] /usr/sbin/asterisk() [0x45d11a] [2017-10-18 06:53:26] VERBOSE logger.c: #2: [0x45d147] /usr/sbin/asterisk() [0x45d147] [2017-10-18 06:53:26] VERBOSE logger.c: #3: [0x45e4f2] /usr/sbin/asterisk() [0x45e4f2] [2017-10-18 06:53:26] VERBOSE logger.c: #4: [0x45e729] /usr/sbin/asterisk(__ao2_link+0x43) [0x45e729] [2017-10-18 06:53:26] VERBOSE logger.c: #5: [0x45eb9c] /usr/sbin/asterisk() [0x45eb9c] [2017-10-18 06:53:26] VERBOSE logger.c: #6: [0x45ee3f] /usr/sbin/asterisk(__ao2_callback+0x5f) [0x45ee3f] [2017-10-18 06:53:26] VERBOSE logger.c: #7: [0x7f672298a04c] /usr/lib64/asterisk/modules/chan_sip.so(+0x6d04c) [0x7f672298a04c] [2017-10-18 06:53:26] VERBOSE logger.c: #8: [0x7f6722989d8e] /usr/lib64/asterisk/modules/chan_sip.so(+0x6cd8e) [0x7f6722989d8e] [2017-10-18 06:53:26] VERBOSE logger.c: #9: [0x4da4cc] /usr/sbin/asterisk(ast_cli_command_full+0x274) [0x4da4cc] [2017-10-18 06:53:26] VERBOSE logger.c: #10: [0x54d66e] /usr/sbin/asterisk() [0x54d66e] [2017-10-18 06:53:26] VERBOSE logger.c: #11: [0x5534fe] /usr/sbin/asterisk() [0x5534fe] [2017-10-18 06:53:26] VERBOSE logger.c: #12: [0x553e34] /usr/sbin/asterisk() [0x553e34] [2017-10-18 06:53:26] VERBOSE logger.c: #13: [0x5542ff] /usr/sbin/asterisk() [0x5542ff] [2017-10-18 06:53:26] VERBOSE logger.c: #14: [0x5ed787] /usr/sbin/asterisk() [0x5ed787] [2017-10-18 06:53:26] VERBOSE logger.c: #15: [0x600bb4] /usr/sbin/asterisk() [0x600bb4]
After talking with Digium Re: BETTER_BACKTRACES we are going to include the option in our build of Asterisk but only on the SNG 7 distro. We might include it on the SNG 6 distro but that is more time consuming
Okay thanks. ETA?
Which system are you on
We’re on 6.
This is now live in 6 and 7 for all versions of Asterisk except 11
Thank you! Does this mean I need to reinstall? Or if I run the current upgrade script it will be activated?
yum update will get you the latest Asterisk version.