Hi Andrew. The only reason it’s come up is that there seems to be a relation. But obviously you know much more accurately. I will consider the two separate issues going forward.
I posted on the Asterisk community forums, and the issue tracker here: https://issues.asterisk.org/jira/browse/ASTERISK-27321
Where we’re currently at is that my backtrace isn’t showing any “symbolic information”. I tried installing debuginfo, but when I run the gdb command after I did that, I get these warnings, and the same “not very helpful” backtrace:
warning: the debug information found in "/usr/lib/debug//usr/lib64/asterisk/modules/format_jpeg.so.debug" does not match "/usr/lib64/asterisk/modules/format_jpeg.so" (CRC mismatch).
warning: the debug information found in "/usr/lib/debug/usr/lib64/asterisk/modules/format_jpeg.so.debug" does not match "/usr/lib64/asterisk/modules/format_jpeg.so" (CRC mismatch).
Corey on the issue tracker said, “This looks like it’s saying that the asterisk binary package does not match the asterisk debuginfo package. My guess is that this is a packaging bug with FreePBX. The debuginfo packages are extracted from the binary packages, and it’s only usable if the CRC of each file matches what is expected.”
Can you provide any assistance with helping me generate a more useful backtrace? I believe I’ve followed the instructions here correctly: https://wiki.freepbx.org/display/SUP/Providing+Great+Debug
Asterisk 13.7.0 is old so you should first upgrade to the latest Asterisk 13. What you did was install debuginfo for a newer version of Asterisk without upgrading your asterisk. After you fix this your older dumps will no longer be valid.
Example of when this happens (Notice that 13.17.0 no longer matches debuginfo of 13.17.1):
[root@Andrew13 ~]# yum list asterisk13*
Loaded plugins: fastestmirror, kmod
Loading mirror speeds from cached hostfile
Installed Packages
asterisk13.x86_64 13.17.0-2.shmz65.1.176 @pbx
asterisk13-addons.x86_64 13.17.0-2.shmz65.1.176 @pbx
asterisk13-addons-bluetooth.x86_64 13.17.0-2.shmz65.1.176 @pbx
asterisk13-addons-core.x86_64 13.17.0-2.shmz65.1.176 @pbx
asterisk13-addons-mysql.x86_64 13.17.0-2.shmz65.1.176 @pbx
asterisk13-addons-ooh323.x86_64 13.17.0-2.shmz65.1.176 @pbx
asterisk13-core.x86_64 13.17.0-2.shmz65.1.176 @pbx
asterisk13-curl.x86_64 13.17.0-2.shmz65.1.176 @pbx
asterisk13-dahdi.x86_64 13.17.0-2.shmz65.1.176 @pbx
asterisk13-debuginfo.x86_64 13.17.1-1.shmz65.1.179 @pbxThank you. I will update Asterisk on one VM and monitor results.
I’ve got a fairly significant number of production PBX’s running V12 and V13 under Proxmox 4.x on Dell hardware (multiple PBX’s per box) and have never had a single issue with this setup.
Last week however I deployed a new i3 NUC with Proxmox5 and a brand new FreePBX13 distro install and have seen this crash over the weekend but Proxmox was still running fine. Considering how long it’s been since I’ve seen a FreePBX system crash this has me pretty alarmed.
I’ve yet to see anything in the logs so far to indicate what could have caused this or what went wrong but I’ll certainly be going over things today looking for any clues.
Generally something to consider about frack errors is that freepbx can’t cause them as it’s a memory issue with asterisk and the system. So they should always be reported to asterisk.
So this morning, we had hundreds of fracks. I noticed in the logs, that it happend right after one of our users unplugged there phone. The next thing noticable in the log is Serious Network Trouble errors, referring to the IP of the router that the phone was on (no longer connected, and no other endpoints on that IP), and then right after, the first of hundreds of FRACKs.
Here’s part of the log. The first line is when the phone was unplugged.
[2017-10-18 06:32:31] NOTICE[2636] chan_sip.c: Peer '235' is now UNREACHABLE! Last qualify: 23
[2017-10-18 06:32:31] VERBOSE[2559] chan_sip.c: Extension Changed 235[ext-local] new state Unavailable for Notify User 201
[2017-10-18 06:53:14] WARNING[2636] chan_sip.c: sip_xmit of 0x7f6784903580 (len 759) to 76.168.37.93:59972 returned -2: Interrupted system call
[2017-10-18 06:53:14] ERROR[2636] chan_sip.c: Serious Network Trouble; __sip_xmit returns error for pkt data
[2017-10-18 06:53:15] WARNING[2636] chan_sip.c: sip_xmit of 0x7f678421a740 (len 760) to 76.168.37.93:59972 returned -2: No such file or directory
[2017-10-18 06:53:15] ERROR[2636] chan_sip.c: Serious Network Trouble; __sip_xmit returns error for pkt data
[2017-10-18 06:53:16] WARNING[2636] chan_sip.c: sip_xmit of 0x7f678417cbc0 (len 760) to 76.168.37.93:59972 returned -2: No such file or directory
[2017-10-18 06:53:16] ERROR[2636] chan_sip.c: Serious Network Trouble; __sip_xmit returns error for pkt data
[2017-10-18 06:53:25] ERROR[7153] astobj2.c: FRACK!, Failed assertion bad magic number 0x0 for object 0x22c55a0 (0)
[2017-10-18 06:53:26] VERBOSE[7153] logger.c: Got 21 backtrace records
[2017-10-18 06:53:26] VERBOSE[7153] logger.c: #0: [0x603fb2] /usr/sbin/asterisk(__ast_assert_failed+0x88) [0x603fb2]
[2017-10-18 06:53:26] VERBOSE[7153] logger.c: #1: [0x45d11a] /usr/sbin/asterisk() [0x45d11a]
[2017-10-18 06:53:26] VERBOSE[7153] logger.c: #2: [0x45d147] /usr/sbin/asterisk() [0x45d147]
[2017-10-18 06:53:26] VERBOSE[7153] logger.c: #3: [0x45e4f2] /usr/sbin/asterisk() [0x45e4f2]
[2017-10-18 06:53:26] VERBOSE[7153] logger.c: #4: [0x45e729] /usr/sbin/asterisk(__ao2_link+0x43) [0x45e729]
[2017-10-18 06:53:26] VERBOSE[7153] logger.c: #5: [0x45eb9c] /usr/sbin/asterisk() [0x45eb9c]
[2017-10-18 06:53:26] VERBOSE[7153] logger.c: #6: [0x45ee3f] /usr/sbin/asterisk(__ao2_callback+0x5f) [0x45ee3f]
[2017-10-18 06:53:26] VERBOSE[7153] logger.c: #7: [0x7f672298a04c] /usr/lib64/asterisk/modules/chan_sip.so(+0x6d04c) [0x7f672298a04c]
[2017-10-18 06:53:26] VERBOSE[7153] logger.c: #8: [0x7f6722989d8e] /usr/lib64/asterisk/modules/chan_sip.so(+0x6cd8e) [0x7f6722989d8e]
[2017-10-18 06:53:26] VERBOSE[7153] logger.c: #9: [0x4da4cc] /usr/sbin/asterisk(ast_cli_command_full+0x274) [0x4da4cc]
[2017-10-18 06:53:26] VERBOSE[7153] logger.c: #10: [0x54d66e] /usr/sbin/asterisk() [0x54d66e]
[2017-10-18 06:53:26] VERBOSE[7153] logger.c: #11: [0x5534fe] /usr/sbin/asterisk() [0x5534fe]
[2017-10-18 06:53:26] VERBOSE[7153] logger.c: #12: [0x553e34] /usr/sbin/asterisk() [0x553e34]
[2017-10-18 06:53:26] VERBOSE[7153] logger.c: #13: [0x5542ff] /usr/sbin/asterisk() [0x5542ff]
[2017-10-18 06:53:26] VERBOSE[7153] logger.c: #14: [0x5ed787] /usr/sbin/asterisk() [0x5ed787]
[2017-10-18 06:53:26] VERBOSE[7153] logger.c: #15: [0x600bb4] /usr/sbin/asterisk() [0x600bb4]After talking with Digium Re: BETTER_BACKTRACES we are going to include the option in our build of Asterisk but only on the SNG 7 distro. We might include it on the SNG 6 distro but that is more time consuming
Okay thanks. ETA?
Which system are you on
We’re on 6.
This is now live in 6 and 7 for all versions of Asterisk except 11
Thank you! Does this mean I need to reinstall? Or if I run the current upgrade script it will be activated?
Running yum update will get you the latest Asterisk version.
Thanks!
So we have iSymphony on some of our servers. I get this error when trying the yum update. Doesn’t look like the asterisk update happens as a result. Any help appreciated.
Transaction Check Error:
file /opt/isymphony3/server/conf/main.xml from install of iSymphonyServerV3-fpbx-3.5.1.24-1.noarch conflicts with file from package iSymphonyServerV3-3.4.2.75-1.noarch
file /opt/isymphony3/server/config.sh from install of iSymphonyServerV3-fpbx-3.5.1.24-1.noarch conflicts with file from package iSymphonyServerV3-3.4.2.75-1.noarch
file /opt/isymphony3/server/iSymphonyServer.jar from install of iSymphonyServerV3-fpbx-3.5.1.24-1.noarch conflicts with file from package iSymphonyServerV3-3.4.2.75-1.noarch
file /opt/isymphony3/server/install.sh from install of iSymphonyServerV3-fpbx-3.5.1.24-1.noarch conflicts with file from package iSymphonyServerV3-3.4.2.75-1.noarch
file /opt/isymphony3/server/modules/asterisk/module.xml from install of iSymphonyServerV3-fpbx-3.5.1.24-1.noarch conflicts with file from package iSymphonyServerV3-3.4.2.75-1.noarch
file /opt/isymphony3/server/modules/chat/module.xml from install of iSymphonyServerV3-fpbx-3.5.1.24-1.noarch conflicts with file from package iSymphonyServerV3-3.4.2.75-1.noarch
file /opt/isymphony3/server/modules/contact/module.xml from install of iSymphonyServerV3-fpbx-3.5.1.24-1.noarch conflicts with file from package iSymphonyServerV3-3.4.2.75-1.noarchJust upgrade asterisk like you did earlier in this thread.
Just to update my issue - I tracked this down to FreePBX13 running on a gen 7 i3 which of course isn’t supported by CentOS 6.6
I had assumed Proxmox would isolate me from this issue running FreePBX as a VM but clearly it does not.
You can choose the virtual CPU per VM in ProxMox
What CPU type did you choose in proxmox? “Default (kvm64)”? Perhaps CentOS 6.6 doesn’t support my physical processor either. I will look into this.