Removal of Asterisk 13.26.0-1 and Asterisk 15.7.2-1

Hello!

Quick update for you all, we’ve removed Asterisk 13.26.0-1 and 15.7.2-1 from our SNG7 repositories today after getting numerous complaints about Asterisk instability. We want to assure you that we do test all Asterisk instances but sometimes issues like these will slip through our own QA channel.

To downgrade your Asterisk 13 simply run yum downgrade asterisk13* this will place you on Asterisk 13.22

That said we think we have identified the issue and have released 13.26-3 and 15.7.2-3 to our testing repository. If you want to check this version of Asterisk simply run yum install sangoma-devel && yum upgrade and you’ll be placed on the version of Asterisk that we believe fixes this issue. If this does not work for you feel free to just run yum downgrade asterisk13* to be placed on a working version

Note This only affects the FreePBX Distro

9 Likes

Hello Andy,

Thank you for doing this: Making an obvious post about it, and immediate resolution on how to repair. One of the challenges I had when experiencing our failures was the proper search terms – asterisk crash wasn’t the best description to search against.

Best,
Christian

1 Like

Sorry - that made me laugh. Searching for “Asterisk Crash” is a little like searching for “the internet”.

6 Likes

That is quite alright. The computer itself didn’t lock up, or stop responding to pings. Linux was fully useful. The logs for me just stopped – /var/log/asterisk/full just stopped recording items, so I figured it was the asterisk process that died. FreePBX web management worked fine in the areas I looked, but SIPSTATION wouldn’t report any status details. Yet in the asterisk console (in linux), sip show connections revealed active SIP connections and links. My OpenNMS server wasn’t catching any traps, or detecting any issues. Then again, I haven’t deeply integrated OpenNMS into Asterisk yet, but SNMP, postfix, httpd, and all the other friends were actively working.

I was quite stuck on trying to construct a useful search string.

Our phones in the store have been going offline ever since the install of these upgrades. I didn’t know what the issue was and had to keep rebooting the computer. It was cutting our customers off. There was a message in the log about some task reaching its limit. I’ve been wasting hours on this. I hope this back-out fixes it.

FYI We identified this issue and it is resolved. It was resolved 3 days ago as previously stated

That said we think we have identified the issue and have released 13.26-3 and 15.7.2-3 to our testing repository.

1 Like

We are still experiencing this issue on a variety of customer servers and VPS’s.
We have now switched to Asterisk 13 certified on four problematic systems to see if this resolves the issue.
yum install sangoma-devel
yum upgrade asterisk-version-switch
asterisk-version-switch

Yes, and you had what Asterisk versions? Why Asterisk 13 certified? Why not Asterisk 16. It is current and LTS. Lots of opaque logic here.

You complain that something broken, which was fixed, is still broken. Yet you provide zero information

Hi Jared:
I’m sorry that my prior post didn’t satisfy your curiosity.
We had recently rolled out a new customer with a Cloud-Switch delivering calls to three office PBX’s with a unified call plan.
After updated the Cloud-VPS together with the three premises PBX’s to the latest releases, we were suffering from random Asterisk lockup.
This update included Asterisk 13.26-2 which as you know, was the LTS Version
This new LTS release, was causing systems that had previously been working brilliantly to become unresponsive.

We use a Custom Destination with alarms to notify us of any incoming call failures on inbound routes to our customer prem PBX’s so we were notified at the first undelivered call.

SAR showed that the systems were being starved of processor resources to the point they because un-responsive.
02:00:02 PM all 28.77 0.00 30.89 0.03 0.00 40.31
02:10:01 PM all 28.74 0.00 30.92 0.03 0.00 40.31
02:20:01 PM all 30.96 0.00 31.15 0.03 0.00 37.87
02:30:02 PM all 33.65 0.00 31.49 0.03 0.00 34.83
02:40:01 PM all 29.02 0.00 30.92 0.03 0.00 40.04
02:50:02 PM all 30.17 0.00 31.00 0.03 0.00 38.80
03:00:02 PM all 31.10 0.00 31.22 0.03 0.00 37.66
03:10:01 PM all 30.17 0.00 31.18 0.03 0.00 38.62
03:20:01 PM all 29.85 0.00 30.80 0.04 0.00 39.31
03:30:01 PM all 29.05 0.00 30.93 0.03 0.00 40.00
03:40:01 PM all 33.82 0.00 31.39 0.03 0.00 34.77
03:50:02 PM all 29.47 0.00 30.93 0.03 0.00 39.57
04:00:01 PM all 35.43 0.00 30.43 0.03 0.00 34.12
04:10:01 PM all 37.80 0.00 30.36 0.02 0.00 31.82
04:20:13 PM all 24.38 0.00 70.66 0.00 0.00 4.95
04:30:06 PM all 22.07 0.00 77.93 0.00 0.00 0.00

Our monitoring system would then have to execute a remote call to reboot the PBX.

04:36:20 PM LINUX RESTART

Rebooting appeared to be the only way to “break” the Asterisk lock up.
Everything was tried, fwconsole restart…… no joy
Killall -9 would not kill all of the cascaded running instances…


Commands to run for Diagnostics when Asterisk crashes again.
• ps auxffww > /root/process.txt 2>&1
• iostat > /root/iostat.txt 2>&1
• lsof > /root/lsof.txt 2>&1
• dmesg > /root/dmesg.txt 2>&1
• netstat -anp > /root/netstat-anp.txt 2>&1
Then run a packet capture:
• touch /root/asterisk-manager.pcap && tshark -i eth0 -f ‘tcp port 5038’ -w /root/asterisk-manager.pcap
Cancel the packet capture after it’s got at least 100 packets If after 2 minutes, it hasn’t captured any packets, please make note of that.
Ctrl-C should stop the packet capture
Then run the following:
tar jcvf /root/asterisk-crash-date -I.tar.bz2 /root/.txt /root/.pcap && rm -f /root/.txt /root/.pcap


We ran comprehensive diagnostics in an attempt to understand the problem but it kept coming back to a bug in the actual Asterisk LTS release.

I know Andrew recommended running the yum downgrade asterisk13*
However, I was enamoured by Andrew’s post of https://www.freepbx.org/sangoma-7-6-distro-ga/
Sangoma 7.6 Distro GA
Posted on May 10, 2019 by Andrew Nagy
“One more thing: Have you always longed for the day where you could install Asterisk Certified on Sangoma 7 and utilize it with FreePBX?
Well, today’s your day! If you’d like to test out Asterisk Certified on your Sangoma 7 FreePBX system you’ll just need to run the following commands (note this is only in the testing repo):”
yum install sangoma-devel
yum upgrade asterisk-version-switch
asterisk-version-switch
Then select Asterisk 13 Certified
So now we are happily running the Certified Asterisk 13.21.0 with happy customers and a happier support team.

Don.

@tm1000 Is the issue present in 13.23? Or did you just go down to 13.22 to be safe?

This issue is only present in the distro rpm:

asterisk13.x86_64            13.26.0-1.sng7                         sng7-testing

Any version of 13.26 you download today will be newer than that, and does not have this issue. No other versions of 13 were affected.

Hi - I’ve been rolled back to asterisk 13.22.0.1.sng7 ever since this issue killed our PBX last year, and I’ve been holding those RPM’s back from upgrading ever since, as it was causing multiple complete lockup’s of the PBX and loss of incoming phone calls several times a day, in a 24/7 call center. If I allow yum to update asterisk to 13.29.2-1.sng7, are we sure this problem is gone? Thanks!

As stated throughout this thread, only rpm versions 13.26.0-1 and 15.7.2-1 are affected.

1 Like