ERROR[5383][C-00003919] astobj2.c: Excessive refcount 100000 reached on ao2 object 0x142f648

[mtaneja712 (Mohit Taneja)]

Hi

We are recently installed Freepbx distro “FreePBX 13.0.192.16” running “Asterisk 13.17.0” on a x86_64 running Linux. From last 2 weeks the CPU suddenly goes to 100% utilization everyday between 9:50 AM EDT to 10:30 AM EDT.
In the full logs we saw below error during the high CPU utilization. Once the error stop appearing the CPU agin goes to normal
[2017-12-21 10:03:01] ERROR[5383][C-00003919] astobj2.c: Excessive refcount 100000 reached on ao2 object 0x142f648
[2017-12-21 10:03:01] ERROR[5383][C-00003919] astobj2.c: FRACK!, Failed assertion Excessive refcount 100000 reached on ao2 object 0x142f648 (0)
[2017-12-21 10:03:01] VERBOSE[6096][C-00003957] logger.c: Got 28 backtrace records
[2017-12-21 10:03:01] VERBOSE[6096][C-00003957] logger.c: #0: [0x60605e] /usr/sbin/asterisk(__ast_assert_failed+0x88) [0x60605e]
[2017-12-21 10:03:01] VERBOSE[6096][C-00003957] logger.c: #1: [0x45e15b] /usr/sbin/asterisk() [0x45e15b]
[2017-12-21 10:03:01] VERBOSE[6096][C-00003957] logger.c: #2: [0x45e4d0] /usr/sbin/asterisk(__ao2_ref+0x31) [0x45e4d0]
[2017-12-21 10:03:01] VERBOSE[6096][C-00003957] logger.c: #3: [0x524ad9] /usr/sbin/asterisk(ast_frdup+0x26c) [0x524ad9]
[2017-12-21 10:03:01] VERBOSE[6096][C-00003957] logger.c: #4: [0x5c563b] /usr/sbin/asterisk(ast_slinfactory_feed+0x298) [0x5c563b]
[2017-12-21 10:03:01] VERBOSE[6096][C-00003957] logger.c: #5: [0x463d57] /usr/sbin/asterisk(ast_audiohook_write_frame+0x328) [0x463d57]
[2017-12-21 10:03:01] VERBOSE[6096][C-00003957] logger.c: #6: [0x466821] /usr/sbin/asterisk() [0x466821]
[2017-12-21 10:03:01] VERBOSE[6096][C-00003957] logger.c: #7: [0x4670fc] /usr/sbin/asterisk(ast_audiohook_write_list+0x3d) [0x4670fc]
[2017-12-21 10:03:01] VERBOSE[6096][C-00003957] logger.c: #8: [0x4b9293] /usr/sbin/asterisk() [0x4b9293]
[2017-12-21 10:03:01] VERBOSE[6096][C-00003957] logger.c: #9: [0x4b95ea] /usr/sbin/asterisk(ast_read+0x1d) [0x4b95ea]
[2017-12-21 10:03:01] VERBOSE[6096][C-00003957] logger.c: #10: [0x4867f3] /usr/sbin/asterisk() [0x4867f3]
[2017-12-21 10:03:01] VERBOSE[6096][C-00003957] logger.c: #11: [0x486cec] /usr/sbin/asterisk() [0x486cec]
[2017-12-21 10:03:01] VERBOSE[6096][C-00003957] logger.c: #12: [0x4873c2] /usr/sbin/asterisk(bridge_channel_internal_join+0x51a) [0x4873c2]
[2017-12-21 10:03:01] VERBOSE[6096][C-00003957] logger.c: #13: [0x46d990] /usr/sbin/asterisk(ast_bridge_join+0x26c) [0x46d990]
[2017-12-21 10:03:01] VERBOSE[6096][C-00003957] logger.c: #14: [0x50e220] /usr/sbin/asterisk(ast_bridge_call_with_flags+0x1b1) [0x50e220]
[2017-12-21 10:03:01] VERBOSE[6096][C-00003957] logger.c: #15: [0x7f5470c4c39e] /usr/lib64/asterisk/modules/app_queue.so(+0x2039e) [0x7f5470c4c39e]
[2017-12-21 10:03:01] VERBOSE[6096][C-00003957] logger.c: #16: [0x7f5470c50606] /usr/lib64/asterisk/modules/app_queue.so(+0x24606) [0x7f5470c50606]
[2017-12-21 10:03:01] VERBOSE[6096][C-00003957] logger.c: #17: [0x588994] /usr/sbin/asterisk(pbx_exec+0x135) [0x588994]
[2017-12-21 10:03:01] VERBOSE[6096][C-00003957] logger.c: #18: [0x5753e3] /usr/sbin/asterisk() [0x5753e3]
[2017-12-21 10:03:01] VERBOSE[6096][C-00003957] logger.c: #19: [0x578a89] /usr/sbin/asterisk(ast_spawn_extension+0x65) [0x578a89]
[2017-12-21 10:03:01] VERBOSE[6096][C-00003957] logger.c: #20: [0x579877] /usr/sbin/asterisk() [0x579877]
[2017-12-21 10:03:01] VERBOSE[6096][C-00003957] logger.c: #21: [0x57b27c] /usr/sbin/asterisk() [0x57b27c]
[2017-12-21 10:03:01] VERBOSE[6096][C-00003957] logger.c: #22: [0x602c60] /usr/sbin/asterisk() [0x602c60]
[2017-12-21 10:03:01] VERBOSE[5383][C-00003919] logger.c: Got 32 backtrace records
[2017-12-21 10:03:01] VERBOSE[5383][C-00003919] logger.c: #0: [0x60605e] /usr/sbin/asterisk(__ast_assert_failed+0x88) [0x60605e]
[2017-12-21 10:03:01] VERBOSE[5383][C-00003919] logger.c: #1: [0x45e15b] /usr/sbin/asterisk() [0x45e15b]
[2017-12-21 10:03:01] VERBOSE[5383][C-00003919] logger.c: #2: [0x45e4d0] /usr/sbin/asterisk(__ao2_ref+0x31) [0x45e4d0]
[2017-12-21 10:03:01] VERBOSE[5383][C-00003919] logger.c: #3: [0x524ad9] /usr/sbin/asterisk(ast_frdup+0x26c) [0x524ad9]
[2017-12-21 10:03:01] VERBOSE[5383][C-00003919] logger.c: #4: [0x5c563b] /usr/sbin/asterisk(ast_slinfactory_feed+0x298) [0x5c563b]
[2017-12-21 10:03:01] VERBOSE[5383][C-00003919] logger.c: #5: [0x463d57] /usr/sbin/asterisk(ast_audiohook_write_frame+0x328) [0x463d57]
[2017-12-21 10:03:01] VERBOSE[5383][C-00003919] logger.c: #6: [0x466821] /usr/sbin/asterisk() [0x466821]
[2017-12-21 10:03:01] VERBOSE[5383][C-00003919] logger.c: #7: [0x4670fc] /usr/sbin/asterisk(ast_audiohook_write_list+0x3d) [0x4670fc]
[2017-12-21 10:03:01] VERBOSE[5383][C-00003919] logger.c: #8: [0x4b9293] /usr/sbin/asterisk() [0x4b9293]
[2017-12-21 10:03:01] VERBOSE[5383][C-00003919] logger.c: #9: [0x4b95ea] /usr/sbin/asterisk(ast_read+0x1d) [0x4b95ea]
[2017-12-21 10:03:01] VERBOSE[5383][C-00003919] logger.c: #10: [0x4867f3] /usr/sbin/asterisk() [0x4867f3]
[2017-12-21 10:03:01] VERBOSE[5383][C-00003919] logger.c: #11: [0x486cec] /usr/sbin/asterisk() [0x486cec]
[2017-12-21 10:03:01] VERBOSE[5383][C-00003919] logger.c: #12: [0x4873c2] /usr/sbin/asterisk(bridge_channel_internal_join+0x51a) [0x4873c2]
[2017-12-21 10:03:01] VERBOSE[5383][C-00003919] logger.c: #13: [0x46d990] /usr/sbin/asterisk(ast_bridge_join+0x26c) [0x46d990]
[2017-12-21 10:03:01] VERBOSE[5383][C-00003919] logger.c: #14: [0x50e220] /usr/sbin/asterisk(ast_bridge_call_with_flags+0x1b1) [0x50e220]
[2017-12-21 10:03:01] VERBOSE[5383][C-00003919] logger.c: #15: [0x50e2fb] /usr/sbin/asterisk(ast_bridge_call+0x31) [0x50e2fb]
[2017-12-21 10:03:01] VERBOSE[5383][C-00003919] logger.c: #16: [0x7f54adf6b035] /usr/lib64/asterisk/modules/app_dial.so(+0x10035) [0x7f54adf6b035]
[2017-12-21 10:03:01] VERBOSE[5383][C-00003919] logger.c: #17: [0x7f54adf6b3e7] /usr/lib64/asterisk/modules/app_dial.so(+0x103e7) [0x7f54adf6b3e7]
[2017-12-21 10:03:01] VERBOSE[5383][C-00003919] logger.c: #18: [0x588994] /usr/sbin/asterisk(pbx_exec+0x135) [0x588994]
[2017-12-21 10:03:01] VERBOSE[5383][C-00003919] logger.c: #19: [0x5753e3] /usr/sbin/asterisk() [0x5753e3]
[2017-12-21 10:03:01] VERBOSE[5383][C-00003919] logger.c: #20: [0x578a89] /usr/sbin/asterisk(ast_spawn_extension+0x65) [0x578a89]
[2017-12-21 10:03:01] VERBOSE[5383][C-00003919] logger.c: #21: [0x7f5461d5025f] /usr/lib64/asterisk/modules/app_macro.so(+0x325f) [0x7f5461d5025f]
[2017-12-21 10:03:01] VERBOSE[5383][C-00003919] logger.c: #22: [0x7f5461d5150e] /usr/lib64/asterisk/modules/app_macro.so(+0x450e) [0x7f5461d5150e]
[2017-12-21 10:03:01] VERBOSE[5383][C-00003919] logger.c: #23: [0x588994] /usr/sbin/asterisk(pbx_exec+0x135) [0x588994]
[2017-12-21 10:03:01] VERBOSE[5383][C-00003919] logger.c: #24: [0x5753e3] /usr/sbin/asterisk() [0x5753e3]
[2017-12-21 10:03:01] VERBOSE[5383][C-00003919] logger.c: #25: [0x578a89] /usr/sbin/asterisk(ast_spawn_extension+0x65) [0x578a89]
[2017-12-21 10:03:01] VERBOSE[5383][C-00003919] logger.c: #26: [0x579877] /usr/sbin/asterisk() [0x579877]

We are using this server in our production call center and would appreciate immediate assistance.

Thanks,
Mohit taneja

[mtaneja712 (Mohit Taneja)]

I have disabled CDR Reports and Call Recording Report modules but the problem is still happening

[root@is20-astrsk-001 ~]# fwconsole ma list
No repos specified, using: [standard] from last GUI settings

±--------------------±------------±---------±-----------+
| Module | Version | Status | License |
±--------------------±------------±---------±-----------+
| accountcodepreserve | 13.0.2 | Enabled | GPLv2 |
| announcement | 13.0.7.1 | Enabled | GPLv3+ |
| areminder | 13.0.10.6 | Enabled | Commercial |
| arimanager | 13.0.4 | Enabled | GPLv3+ |
| asterisk-cli | 13.0.4 | Enabled | GPLv3+ |
| asteriskinfo | 13.0.7.1 | Enabled | GPLv3+ |
| backup | 13.0.27.1 | Enabled | GPLv3+ |
| blacklist | 13.0.14.3 | Enabled | GPLv3+ |
| broadcast | 13.0.12.7 | Enabled | Commercial |
| builtin | | Enabled | |
| bulkhandler | 13.0.14.4 | Enabled | GPLv3+ |
| callback | 13.0.5.2 | Enabled | GPLv3+ |
| callerid | 13.0.8.4 | Enabled | Commercial |
| callforward | 13.0.4.1 | Enabled | AGPLv3+ |
| calllimit | 13.0.5.5 | Enabled | Commercial |
| callrecording | 13.0.11.4 | Enabled | AGPLv3+ |
| callwaiting | 13.0.4.1 | Enabled | GPLv3+ |
| campon | 13.0.4.1 | Enabled | GPLv3+ |
| cdr | 13.0.30.6 | Disabled | GPLv3+ |
| cel | 13.0.26.2 | Enabled | GPLv3+ |
| certman | 13.0.36.10 | Enabled | AGPLv3+ |
| cidlookup | 13.0.12.1 | Enabled | GPLv3+ |
| conferences | 13.0.23.7 | Enabled | GPLv3+ |
| conferencespro | 13.0.27.7 | Enabled | Commercial |
| configedit | 13.0.7 | Enabled | AGPLv3+ |
| contactmanager | 13.0.42.8 | Enabled | GPLv3+ |
| core | 13.0.120.10 | Enabled | GPLv3+ |
| cos | 13.0.11.4 | Enabled | Commercial |
| customappsreg | 13.0.5.1 | Enabled | GPLv3+ |
| cxpanel | 13.0.4.3 | Enabled | GPLv3 |
| dahdiconfig | 13.0.33.12 | Enabled | GPLv3+ |
| dashboard | 13.0.25.1 | Enabled | AGPLv3+ |
| daynight | 13.0.14 | Enabled | GPLv3+ |
| dictate | 13.0.5 | Enabled | GPLv3+ |
| digium_phones | 13.0.7.1 | Enabled | GPLv2 |
| directory | 13.0.19.3 | Enabled | GPLv3+ |
| disa | 13.0.6.1 | Enabled | AGPLv3+ |
| donotdisturb | 13.0.3 | Enabled | GPLv3+ |
| endpoint | 13.0.109 | Enabled | Commercial |
| extensionroutes | 13.0.10.4 | Enabled | Commercial |
| fax | 13.0.40.3 | Enabled | GPLv3+ |
| faxpro | 13.0.38.7 | Enabled | Commercial |
| featurecodeadmin | 13.0.6.4 | Enabled | GPLv3+ |
| findmefollow | 13.0.38.10 | Enabled | GPLv3+ |
| firewall | 13.0.45.3 | Enabled | AGPLv3+ |
| framework | 13.0.192.16 | Enabled | GPLv2+ |
| fw_langpacks | 12.0.7 | Enabled | GPLv3+ |
| hotelwakeup | 13.0.17.1 | Enabled | GPLv2 |
| iaxsettings | 13.0.6.3 | Enabled | AGPLv3 |
| infoservices | 13.0.1.1 | Enabled | GPLv2+ |
| irc | 2.11.0.7 | Enabled | GPLv3+ |
| ivr | 13.0.25 | Enabled | GPLv3+ |
| languages | 13.0.6 | Enabled | GPLv3+ |
| logfiles | 13.0.10.3 | Enabled | GPLv3+ |
| manager | 13.0.2.5 | Enabled | GPLv2+ |
| miscapps | 13.0.3.1 | Enabled | GPLv3+ |
| miscdests | 13.0.5 | Enabled | GPLv3+ |
| music | 13.0.22.3 | Enabled | GPLv3+ |
| outroutemsg | 13.0.2.1 | Enabled | GPLv3+ |
| paging | 13.0.26.2 | Enabled | GPLv3+ |
| pagingpro | 13.0.19.6 | Enabled | Commercial |
| parking | 13.0.19.6 | Enabled | GPLv3+ |
| parkpro | 13.0.30.10 | Enabled | Commercial |
| pbdirectory | 2.11.0.5 | Enabled | GPLv3+ |
| phonebook | 13.0.5.7 | Enabled | GPLv3+ |
| phpinfo | 13.0.2 | Enabled | GPLv2+ |
| pinsets | 13.0.8 | Enabled | GPLv3+ |
| pinsetspro | 13.0.9.3 | Enabled | Commercial |
| pm2 | 13.0.3.18 | Enabled | AGPLv3+ |
| presencestate | 13.0.7 | Enabled | GPLv3+ |
| printextensions | 13.0.3.1 | Enabled | GPLv3+ |
| queueprio | 13.0.2 | Enabled | GPLv3+ |
| queues | 13.0.34.1 | Enabled | GPLv2+ |
| qxact_reports | 13.0.15.3 | Enabled | Commercial |
| recording_report | 13.0.24.6 | Disabled | Commercial |
| recordings | 13.0.30.11 | Enabled | GPLv3+ |
| restapi | 13.0.21.1 | Enabled | AGPLv3 |
| restapps | 13.0.88.5 | Enabled | Commercial |
| ringgroups | 13.0.23.1 | Enabled | GPLv3+ |
| rmsadmin | 13.0.14.1 | Enabled | Commercial |
| sangomacrm | 13.0.2.1 | Disabled | Commercial |
| setcid | 13.0.6.2 | Enabled | GPLv3+ |
| sipsettings | 13.0.24.8 | Enabled | AGPLv3+ |
| sipstation | 13.0.13.20 | Enabled | Commercial |
| sms | 13.0.11.3 | Enabled | Commercial |
| sng_mcu | 13.0.5 | Enabled | Commercial |
| soundlang | 13.0.24.5 | Enabled | GPLv3+ |
| speeddial | 2.11.0.4 | Enabled | GPLv3+ |
| superfecta | 13.0.4.1 | Enabled | GPLv2+ |
| sysadmin | 13.0.74.5 | Enabled | Commercial |
| timeconditions | 13.0.34.5 | Enabled | GPLv3+ |
| tts | 13.0.10 | Enabled | GPLv3+ |
| ttsengines | 13.0.7.1 | Enabled | AGPLv3 |
| ucp | 13.0.42.2 | Enabled | AGPLv3+ |
| ucpnode | 13.0.34.9 | Enabled | Commercial |
| userman | 13.0.76.27 | Enabled | AGPLv3+ |
| versionupgrade | 13.0.1.1 | Enabled | Commercial |
| vmblast | 13.0.8 | Enabled | GPLv3+ |
| vmnotify | 13.0.18.3 | Enabled | Commercial |
| voicemail | 13.0.54.9 | Enabled | GPLv3+ |
| voicemail_report | 13.0.13.2 | Enabled | Commercial |
| vqplus | 13.0.26.10 | Enabled | Commercial |
| weakpasswords | 13.0.2 | Enabled | GPLv3+ |
| webcallback | 13.0.11.2 | Enabled | Commercial |
| webrtc | 13.0.32.8 | Enabled | GPLv3+ |
| xmpp | 13.0.17.3 | Enabled | AGPLv3 |
| zulu | 13.0.53.2 | Enabled | Commercial |
±--------------------±------------±---------±-----------+

[mtaneja712 (Mohit Taneja)]

Can someone please help me here ? It’s impacting our business.

[james (James Zhu)]

@mtaneja712

astobj2

it looks locked by some tasks. how many users are running?

[Marbled (Nicolas Riendeau)]

Hi!

Please take a look at this thread:

Consistent Asterisk/FreePBX Crash Issue

Running FreePBX 13.0.192.16 and Asterisk 13.17.0 Recently used the “warm spare” method to move to a new server (new VM on KVM/proxmox) The server has about 120 remote extension, and had no real problems before. I posted about this crash issue yesterday here, but my hypothesis was off: Media_index.c: Failed to stat Today we had a ton of users call and say their phones weren’t working. Funny thing is, they show OK with IP address in peers list when running “sip show peers” in cli. So we did a…

You are getting FRACK errors…

As Andrew said in this thread

Consistent Asterisk/FreePBX Crash Issue

Generally something to consider about frack errors is that freepbx can’t cause them as it’s a memory issue with asterisk and the system. So they should always be reported to asterisk.

they should be reported to Asterisk.

That said, did you update your packages? You are not running the latest Asterisk 13 and the one you are running, 13.17 is known have problems according to dicko’s post:

Consistent Asterisk/FreePBX Crash Issue

Again , there IS an acknowledged problem by Digium with 13.17 , just don’t use it and you won’t have to chase rabbits down the wrong hole.

Good luck and have a nice day!

Nick

[stevensedory]

@mtaneja712

Can someone please help me here ? It’s impacting our business.

Run the upgrade scrips. Follow process here: Sangoma Documentation

Once completed, they will bring your asterisk up to 13.17.2 or higher. Since getting on 13.17.2, we haven’t really had any of the serious issues we previously had on 13.17.0 and some earlier releases.

[mtaneja712 (Mohit Taneja)]

180 concurrent agents logged in , maximum 40 concurrent calls.
We are only using freepbx for queueing , no device is registered on it
It’s a VM with 12 GB memory , 50 GB HDD and 4 VCPU

Why it’s locked?

[mtaneja712 (Mohit Taneja)]

I am thinking of upgrading it to 13.18 instead of 13.17.2 . Would it be advisable ?

[stevensedory]

All I can vouch for is that I’m on 13.17.2 on several servers that were previously having memory leak issues (on older versions), and now we’re not. We have a similar load as you described above.