Okay. Anyone I can get some paid support from on an issue like this?
Sangoma? We’ve always had paid support. have you never tried it?
Did you ever contact Asterisk with your crash?
Did you try to remove your trunks?
I had same issue before, and there was problem with my sip trunks. something like misconfiguration.
1 Like
Im going to move this back to our original issue because you are mixing up a FRACK! which is a memory leak which is your crash with “Serious Network Trouble” which is not related to the FRACK.
Looking through your previous replies on this I don’t believe you ever attempted to get the crash reports and send them to Asterisk.
This is a memory leak in Asterisk. As such it casues a crash. Even if you are doing something wrong Asterisk should not leak memory and therefor you should really go to the Asterisk issues tracker and see if they have any issues related to yours.
Which error did you have? The Serious Network Trouble, or FRACK?
Well, we’ve created them from scratch (new servers) and are having these issues still. That said, the IPs Ive noticed in the logs that related to the serious network trouble errors are not trunk IPs, but endpoints. Could be both though, just haven’t dug that deep to confirm/deny.
Here are our trunks, anveo, anveo 911, and then a FXO we have setup. Everything else are extensions:
E911 EXAMPLE
username=xxxxxxxxxx
type=peer
secret=xxxxxxxxxx
port=5010
insecure=port,invite
host=sip.anveo.com
dtmfmode=rfc2833
disallow=all
context=from-anveo
allow=ulaw&alaw&g729
ANVEO IN EXAMPLE
host=50.22.101.14
type=friend
insecure=port,invite
context=from-trunk
canreinvite=yes
qualify=no
disallow=all
allow=ulaw&g729
ANVEO OUT EXAMPLE
type=peer
host=sbc.anveo.com
port=5060
insecure=port,invite
canreinvite=yes
disallow=all
allow=ulaw
FXO EXAMPLE
username=8880
type=friend
qualifyfreq=25
qualify=yes
nat=yes
host=dynamic
dtmfmode=auto
context=from-pstn
canreinvite=no
secret=xxxxxxxxxxxx
transport=tcp
Hi Andrew. The only reason it’s come up is that there seems to be a relation. But obviously you know much more accurately. I will consider the two separate issues going forward.
I posted on the Asterisk community forums, and the issue tracker here: https://issues.asterisk.org/jira/browse/ASTERISK-27321
Where we’re currently at is that my backtrace isn’t showing any “symbolic information”. I tried installing debuginfo, but when I run the gdb command after I did that, I get these warnings, and the same “not very helpful” backtrace:
warning: the debug information found in "/usr/lib/debug//usr/lib64/asterisk/modules/format_jpeg.so.debug" does not match "/usr/lib64/asterisk/modules/format_jpeg.so" (CRC mismatch).
warning: the debug information found in "/usr/lib/debug/usr/lib64/asterisk/modules/format_jpeg.so.debug" does not match "/usr/lib64/asterisk/modules/format_jpeg.so" (CRC mismatch).
Corey on the issue tracker said, “This looks like it’s saying that the asterisk binary package does not match the asterisk debuginfo package. My guess is that this is a packaging bug with FreePBX. The debuginfo packages are extracted from the binary packages, and it’s only usable if the CRC of each file matches what is expected.”
Can you provide any assistance with helping me generate a more useful backtrace? I believe I’ve followed the instructions here correctly: https://wiki.freepbx.org/display/SUP/Providing+Great+Debug
Asterisk 13.7.0 is old so you should first upgrade to the latest Asterisk 13. What you did was install debuginfo for a newer version of Asterisk without upgrading your asterisk. After you fix this your older dumps will no longer be valid.
Example of when this happens (Notice that 13.17.0 no longer matches debuginfo of 13.17.1):
[root@Andrew13 ~]# yum list asterisk13*
Loaded plugins: fastestmirror, kmod
Loading mirror speeds from cached hostfile
Installed Packages
asterisk13.x86_64 13.17.0-2.shmz65.1.176 @pbx
asterisk13-addons.x86_64 13.17.0-2.shmz65.1.176 @pbx
asterisk13-addons-bluetooth.x86_64 13.17.0-2.shmz65.1.176 @pbx
asterisk13-addons-core.x86_64 13.17.0-2.shmz65.1.176 @pbx
asterisk13-addons-mysql.x86_64 13.17.0-2.shmz65.1.176 @pbx
asterisk13-addons-ooh323.x86_64 13.17.0-2.shmz65.1.176 @pbx
asterisk13-core.x86_64 13.17.0-2.shmz65.1.176 @pbx
asterisk13-curl.x86_64 13.17.0-2.shmz65.1.176 @pbx
asterisk13-dahdi.x86_64 13.17.0-2.shmz65.1.176 @pbx
asterisk13-debuginfo.x86_64 13.17.1-1.shmz65.1.179 @pbxThank you. I will update Asterisk on one VM and monitor results.
I’ve got a fairly significant number of production PBX’s running V12 and V13 under Proxmox 4.x on Dell hardware (multiple PBX’s per box) and have never had a single issue with this setup.
Last week however I deployed a new i3 NUC with Proxmox5 and a brand new FreePBX13 distro install and have seen this crash over the weekend but Proxmox was still running fine. Considering how long it’s been since I’ve seen a FreePBX system crash this has me pretty alarmed.
I’ve yet to see anything in the logs so far to indicate what could have caused this or what went wrong but I’ll certainly be going over things today looking for any clues.
Generally something to consider about frack errors is that freepbx can’t cause them as it’s a memory issue with asterisk and the system. So they should always be reported to asterisk.
So this morning, we had hundreds of fracks. I noticed in the logs, that it happend right after one of our users unplugged there phone. The next thing noticable in the log is Serious Network Trouble errors, referring to the IP of the router that the phone was on (no longer connected, and no other endpoints on that IP), and then right after, the first of hundreds of FRACKs.
Here’s part of the log. The first line is when the phone was unplugged.
[2017-10-18 06:32:31] NOTICE[2636] chan_sip.c: Peer '235' is now UNREACHABLE! Last qualify: 23
[2017-10-18 06:32:31] VERBOSE[2559] chan_sip.c: Extension Changed 235[ext-local] new state Unavailable for Notify User 201
[2017-10-18 06:53:14] WARNING[2636] chan_sip.c: sip_xmit of 0x7f6784903580 (len 759) to 76.168.37.93:59972 returned -2: Interrupted system call
[2017-10-18 06:53:14] ERROR[2636] chan_sip.c: Serious Network Trouble; __sip_xmit returns error for pkt data
[2017-10-18 06:53:15] WARNING[2636] chan_sip.c: sip_xmit of 0x7f678421a740 (len 760) to 76.168.37.93:59972 returned -2: No such file or directory
[2017-10-18 06:53:15] ERROR[2636] chan_sip.c: Serious Network Trouble; __sip_xmit returns error for pkt data
[2017-10-18 06:53:16] WARNING[2636] chan_sip.c: sip_xmit of 0x7f678417cbc0 (len 760) to 76.168.37.93:59972 returned -2: No such file or directory
[2017-10-18 06:53:16] ERROR[2636] chan_sip.c: Serious Network Trouble; __sip_xmit returns error for pkt data
[2017-10-18 06:53:25] ERROR[7153] astobj2.c: FRACK!, Failed assertion bad magic number 0x0 for object 0x22c55a0 (0)
[2017-10-18 06:53:26] VERBOSE[7153] logger.c: Got 21 backtrace records
[2017-10-18 06:53:26] VERBOSE[7153] logger.c: #0: [0x603fb2] /usr/sbin/asterisk(__ast_assert_failed+0x88) [0x603fb2]
[2017-10-18 06:53:26] VERBOSE[7153] logger.c: #1: [0x45d11a] /usr/sbin/asterisk() [0x45d11a]
[2017-10-18 06:53:26] VERBOSE[7153] logger.c: #2: [0x45d147] /usr/sbin/asterisk() [0x45d147]
[2017-10-18 06:53:26] VERBOSE[7153] logger.c: #3: [0x45e4f2] /usr/sbin/asterisk() [0x45e4f2]
[2017-10-18 06:53:26] VERBOSE[7153] logger.c: #4: [0x45e729] /usr/sbin/asterisk(__ao2_link+0x43) [0x45e729]
[2017-10-18 06:53:26] VERBOSE[7153] logger.c: #5: [0x45eb9c] /usr/sbin/asterisk() [0x45eb9c]
[2017-10-18 06:53:26] VERBOSE[7153] logger.c: #6: [0x45ee3f] /usr/sbin/asterisk(__ao2_callback+0x5f) [0x45ee3f]
[2017-10-18 06:53:26] VERBOSE[7153] logger.c: #7: [0x7f672298a04c] /usr/lib64/asterisk/modules/chan_sip.so(+0x6d04c) [0x7f672298a04c]
[2017-10-18 06:53:26] VERBOSE[7153] logger.c: #8: [0x7f6722989d8e] /usr/lib64/asterisk/modules/chan_sip.so(+0x6cd8e) [0x7f6722989d8e]
[2017-10-18 06:53:26] VERBOSE[7153] logger.c: #9: [0x4da4cc] /usr/sbin/asterisk(ast_cli_command_full+0x274) [0x4da4cc]
[2017-10-18 06:53:26] VERBOSE[7153] logger.c: #10: [0x54d66e] /usr/sbin/asterisk() [0x54d66e]
[2017-10-18 06:53:26] VERBOSE[7153] logger.c: #11: [0x5534fe] /usr/sbin/asterisk() [0x5534fe]
[2017-10-18 06:53:26] VERBOSE[7153] logger.c: #12: [0x553e34] /usr/sbin/asterisk() [0x553e34]
[2017-10-18 06:53:26] VERBOSE[7153] logger.c: #13: [0x5542ff] /usr/sbin/asterisk() [0x5542ff]
[2017-10-18 06:53:26] VERBOSE[7153] logger.c: #14: [0x5ed787] /usr/sbin/asterisk() [0x5ed787]
[2017-10-18 06:53:26] VERBOSE[7153] logger.c: #15: [0x600bb4] /usr/sbin/asterisk() [0x600bb4]After talking with Digium Re: BETTER_BACKTRACES we are going to include the option in our build of Asterisk but only on the SNG 7 distro. We might include it on the SNG 6 distro but that is more time consuming
Okay thanks. ETA?
Which system are you on
We’re on 6.
This is now live in 6 and 7 for all versions of Asterisk except 11
Thank you! Does this mean I need to reinstall? Or if I run the current upgrade script it will be activated?
Running yum update will get you the latest Asterisk version.