ERROR[5383][C-00003919] astobj2.c: Excessive refcount 100000 reached on ao2 object 0x142f648

Hi

We are recently installed Freepbx distro “FreePBX 13.0.192.16” running “Asterisk 13.17.0” on a x86_64 running Linux. From last 2 weeks the CPU suddenly goes to 100% utilization everyday between 9:50 AM EDT to 10:30 AM EDT.
In the full logs we saw below error during the high CPU utilization. Once the error stop appearing the CPU agin goes to normal
[2017-12-21 10:03:01] ERROR[5383][C-00003919] astobj2.c: Excessive refcount 100000 reached on ao2 object 0x142f648
[2017-12-21 10:03:01] ERROR[5383][C-00003919] astobj2.c: FRACK!, Failed assertion Excessive refcount 100000 reached on ao2 object 0x142f648 (0)
[2017-12-21 10:03:01] VERBOSE[6096][C-00003957] logger.c: Got 28 backtrace records
[2017-12-21 10:03:01] VERBOSE[6096][C-00003957] logger.c: #0: [0x60605e] /usr/sbin/asterisk(__ast_assert_failed+0x88) [0x60605e]
[2017-12-21 10:03:01] VERBOSE[6096][C-00003957] logger.c: #1: [0x45e15b] /usr/sbin/asterisk() [0x45e15b]
[2017-12-21 10:03:01] VERBOSE[6096][C-00003957] logger.c: #2: [0x45e4d0] /usr/sbin/asterisk(__ao2_ref+0x31) [0x45e4d0]
[2017-12-21 10:03:01] VERBOSE[6096][C-00003957] logger.c: #3: [0x524ad9] /usr/sbin/asterisk(ast_frdup+0x26c) [0x524ad9]
[2017-12-21 10:03:01] VERBOSE[6096][C-00003957] logger.c: #4: [0x5c563b] /usr/sbin/asterisk(ast_slinfactory_feed+0x298) [0x5c563b]
[2017-12-21 10:03:01] VERBOSE[6096][C-00003957] logger.c: #5: [0x463d57] /usr/sbin/asterisk(ast_audiohook_write_frame+0x328) [0x463d57]
[2017-12-21 10:03:01] VERBOSE[6096][C-00003957] logger.c: #6: [0x466821] /usr/sbin/asterisk() [0x466821]
[2017-12-21 10:03:01] VERBOSE[6096][C-00003957] logger.c: #7: [0x4670fc] /usr/sbin/asterisk(ast_audiohook_write_list+0x3d) [0x4670fc]
[2017-12-21 10:03:01] VERBOSE[6096][C-00003957] logger.c: #8: [0x4b9293] /usr/sbin/asterisk() [0x4b9293]
[2017-12-21 10:03:01] VERBOSE[6096][C-00003957] logger.c: #9: [0x4b95ea] /usr/sbin/asterisk(ast_read+0x1d) [0x4b95ea]
[2017-12-21 10:03:01] VERBOSE[6096][C-00003957] logger.c: #10: [0x4867f3] /usr/sbin/asterisk() [0x4867f3]
[2017-12-21 10:03:01] VERBOSE[6096][C-00003957] logger.c: #11: [0x486cec] /usr/sbin/asterisk() [0x486cec]
[2017-12-21 10:03:01] VERBOSE[6096][C-00003957] logger.c: #12: [0x4873c2] /usr/sbin/asterisk(bridge_channel_internal_join+0x51a) [0x4873c2]
[2017-12-21 10:03:01] VERBOSE[6096][C-00003957] logger.c: #13: [0x46d990] /usr/sbin/asterisk(ast_bridge_join+0x26c) [0x46d990]
[2017-12-21 10:03:01] VERBOSE[6096][C-00003957] logger.c: #14: [0x50e220] /usr/sbin/asterisk(ast_bridge_call_with_flags+0x1b1) [0x50e220]
[2017-12-21 10:03:01] VERBOSE[6096][C-00003957] logger.c: #15: [0x7f5470c4c39e] /usr/lib64/asterisk/modules/app_queue.so(+0x2039e) [0x7f5470c4c39e]
[2017-12-21 10:03:01] VERBOSE[6096][C-00003957] logger.c: #16: [0x7f5470c50606] /usr/lib64/asterisk/modules/app_queue.so(+0x24606) [0x7f5470c50606]
[2017-12-21 10:03:01] VERBOSE[6096][C-00003957] logger.c: #17: [0x588994] /usr/sbin/asterisk(pbx_exec+0x135) [0x588994]
[2017-12-21 10:03:01] VERBOSE[6096][C-00003957] logger.c: #18: [0x5753e3] /usr/sbin/asterisk() [0x5753e3]
[2017-12-21 10:03:01] VERBOSE[6096][C-00003957] logger.c: #19: [0x578a89] /usr/sbin/asterisk(ast_spawn_extension+0x65) [0x578a89]
[2017-12-21 10:03:01] VERBOSE[6096][C-00003957] logger.c: #20: [0x579877] /usr/sbin/asterisk() [0x579877]
[2017-12-21 10:03:01] VERBOSE[6096][C-00003957] logger.c: #21: [0x57b27c] /usr/sbin/asterisk() [0x57b27c]
[2017-12-21 10:03:01] VERBOSE[6096][C-00003957] logger.c: #22: [0x602c60] /usr/sbin/asterisk() [0x602c60]
[2017-12-21 10:03:01] VERBOSE[5383][C-00003919] logger.c: Got 32 backtrace records
[2017-12-21 10:03:01] VERBOSE[5383][C-00003919] logger.c: #0: [0x60605e] /usr/sbin/asterisk(__ast_assert_failed+0x88) [0x60605e]
[2017-12-21 10:03:01] VERBOSE[5383][C-00003919] logger.c: #1: [0x45e15b] /usr/sbin/asterisk() [0x45e15b]
[2017-12-21 10:03:01] VERBOSE[5383][C-00003919] logger.c: #2: [0x45e4d0] /usr/sbin/asterisk(__ao2_ref+0x31) [0x45e4d0]
[2017-12-21 10:03:01] VERBOSE[5383][C-00003919] logger.c: #3: [0x524ad9] /usr/sbin/asterisk(ast_frdup+0x26c) [0x524ad9]
[2017-12-21 10:03:01] VERBOSE[5383][C-00003919] logger.c: #4: [0x5c563b] /usr/sbin/asterisk(ast_slinfactory_feed+0x298) [0x5c563b]
[2017-12-21 10:03:01] VERBOSE[5383][C-00003919] logger.c: #5: [0x463d57] /usr/sbin/asterisk(ast_audiohook_write_frame+0x328) [0x463d57]
[2017-12-21 10:03:01] VERBOSE[5383][C-00003919] logger.c: #6: [0x466821] /usr/sbin/asterisk() [0x466821]
[2017-12-21 10:03:01] VERBOSE[5383][C-00003919] logger.c: #7: [0x4670fc] /usr/sbin/asterisk(ast_audiohook_write_list+0x3d) [0x4670fc]
[2017-12-21 10:03:01] VERBOSE[5383][C-00003919] logger.c: #8: [0x4b9293] /usr/sbin/asterisk() [0x4b9293]
[2017-12-21 10:03:01] VERBOSE[5383][C-00003919] logger.c: #9: [0x4b95ea] /usr/sbin/asterisk(ast_read+0x1d) [0x4b95ea]
[2017-12-21 10:03:01] VERBOSE[5383][C-00003919] logger.c: #10: [0x4867f3] /usr/sbin/asterisk() [0x4867f3]
[2017-12-21 10:03:01] VERBOSE[5383][C-00003919] logger.c: #11: [0x486cec] /usr/sbin/asterisk() [0x486cec]
[2017-12-21 10:03:01] VERBOSE[5383][C-00003919] logger.c: #12: [0x4873c2] /usr/sbin/asterisk(bridge_channel_internal_join+0x51a) [0x4873c2]
[2017-12-21 10:03:01] VERBOSE[5383][C-00003919] logger.c: #13: [0x46d990] /usr/sbin/asterisk(ast_bridge_join+0x26c) [0x46d990]
[2017-12-21 10:03:01] VERBOSE[5383][C-00003919] logger.c: #14: [0x50e220] /usr/sbin/asterisk(ast_bridge_call_with_flags+0x1b1) [0x50e220]
[2017-12-21 10:03:01] VERBOSE[5383][C-00003919] logger.c: #15: [0x50e2fb] /usr/sbin/asterisk(ast_bridge_call+0x31) [0x50e2fb]
[2017-12-21 10:03:01] VERBOSE[5383][C-00003919] logger.c: #16: [0x7f54adf6b035] /usr/lib64/asterisk/modules/app_dial.so(+0x10035) [0x7f54adf6b035]
[2017-12-21 10:03:01] VERBOSE[5383][C-00003919] logger.c: #17: [0x7f54adf6b3e7] /usr/lib64/asterisk/modules/app_dial.so(+0x103e7) [0x7f54adf6b3e7]
[2017-12-21 10:03:01] VERBOSE[5383][C-00003919] logger.c: #18: [0x588994] /usr/sbin/asterisk(pbx_exec+0x135) [0x588994]
[2017-12-21 10:03:01] VERBOSE[5383][C-00003919] logger.c: #19: [0x5753e3] /usr/sbin/asterisk() [0x5753e3]
[2017-12-21 10:03:01] VERBOSE[5383][C-00003919] logger.c: #20: [0x578a89] /usr/sbin/asterisk(ast_spawn_extension+0x65) [0x578a89]
[2017-12-21 10:03:01] VERBOSE[5383][C-00003919] logger.c: #21: [0x7f5461d5025f] /usr/lib64/asterisk/modules/app_macro.so(+0x325f) [0x7f5461d5025f]
[2017-12-21 10:03:01] VERBOSE[5383][C-00003919] logger.c: #22: [0x7f5461d5150e] /usr/lib64/asterisk/modules/app_macro.so(+0x450e) [0x7f5461d5150e]
[2017-12-21 10:03:01] VERBOSE[5383][C-00003919] logger.c: #23: [0x588994] /usr/sbin/asterisk(pbx_exec+0x135) [0x588994]
[2017-12-21 10:03:01] VERBOSE[5383][C-00003919] logger.c: #24: [0x5753e3] /usr/sbin/asterisk() [0x5753e3]
[2017-12-21 10:03:01] VERBOSE[5383][C-00003919] logger.c: #25: [0x578a89] /usr/sbin/asterisk(ast_spawn_extension+0x65) [0x578a89]
[2017-12-21 10:03:01] VERBOSE[5383][C-00003919] logger.c: #26: [0x579877] /usr/sbin/asterisk() [0x579877]

We are using this server in our production call center and would appreciate immediate assistance.

Thanks,
Mohit taneja

I have disabled CDR Reports and Call Recording Report modules but the problem is still happening

[[email protected] ~]# fwconsole ma list
No repos specified, using: [standard] from last GUI settings

±--------------------±------------±---------±-----------+
| Module | Version | Status | License |
±--------------------±------------±---------±-----------+
| accountcodepreserve | 13.0.2 | Enabled | GPLv2 |
| announcement | 13.0.7.1 | Enabled | GPLv3+ |
| areminder | 13.0.10.6 | Enabled | Commercial |
| arimanager | 13.0.4 | Enabled | GPLv3+ |
| asterisk-cli | 13.0.4 | Enabled | GPLv3+ |
| asteriskinfo | 13.0.7.1 | Enabled | GPLv3+ |
| backup | 13.0.27.1 | Enabled | GPLv3+ |
| blacklist | 13.0.14.3 | Enabled | GPLv3+ |
| broadcast | 13.0.12.7 | Enabled | Commercial |
| builtin | | Enabled | |
| bulkhandler | 13.0.14.4 | Enabled | GPLv3+ |
| callback | 13.0.5.2 | Enabled | GPLv3+ |
| callerid | 13.0.8.4 | Enabled | Commercial |
| callforward | 13.0.4.1 | Enabled | AGPLv3+ |
| calllimit | 13.0.5.5 | Enabled | Commercial |
| callrecording | 13.0.11.4 | Enabled | AGPLv3+ |
| callwaiting | 13.0.4.1 | Enabled | GPLv3+ |
| campon | 13.0.4.1 | Enabled | GPLv3+ |
| cdr | 13.0.30.6 | Disabled | GPLv3+ |
| cel | 13.0.26.2 | Enabled | GPLv3+ |
| certman | 13.0.36.10 | Enabled | AGPLv3+ |
| cidlookup | 13.0.12.1 | Enabled | GPLv3+ |
| conferences | 13.0.23.7 | Enabled | GPLv3+ |
| conferencespro | 13.0.27.7 | Enabled | Commercial |
| configedit | 13.0.7 | Enabled | AGPLv3+ |
| contactmanager | 13.0.42.8 | Enabled | GPLv3+ |
| core | 13.0.120.10 | Enabled | GPLv3+ |
| cos | 13.0.11.4 | Enabled | Commercial |
| customappsreg | 13.0.5.1 | Enabled | GPLv3+ |
| cxpanel | 13.0.4.3 | Enabled | GPLv3 |
| dahdiconfig | 13.0.33.12 | Enabled | GPLv3+ |
| dashboard | 13.0.25.1 | Enabled | AGPLv3+ |
| daynight | 13.0.14 | Enabled | GPLv3+ |
| dictate | 13.0.5 | Enabled | GPLv3+ |
| digium_phones | 13.0.7.1 | Enabled | GPLv2 |
| directory | 13.0.19.3 | Enabled | GPLv3+ |
| disa | 13.0.6.1 | Enabled | AGPLv3+ |
| donotdisturb | 13.0.3 | Enabled | GPLv3+ |
| endpoint | 13.0.109 | Enabled | Commercial |
| extensionroutes | 13.0.10.4 | Enabled | Commercial |
| fax | 13.0.40.3 | Enabled | GPLv3+ |
| faxpro | 13.0.38.7 | Enabled | Commercial |
| featurecodeadmin | 13.0.6.4 | Enabled | GPLv3+ |
| findmefollow | 13.0.38.10 | Enabled | GPLv3+ |
| firewall | 13.0.45.3 | Enabled | AGPLv3+ |
| framework | 13.0.192.16 | Enabled | GPLv2+ |
| fw_langpacks | 12.0.7 | Enabled | GPLv3+ |
| hotelwakeup | 13.0.17.1 | Enabled | GPLv2 |
| iaxsettings | 13.0.6.3 | Enabled | AGPLv3 |
| infoservices | 13.0.1.1 | Enabled | GPLv2+ |
| irc | 2.11.0.7 | Enabled | GPLv3+ |
| ivr | 13.0.25 | Enabled | GPLv3+ |
| languages | 13.0.6 | Enabled | GPLv3+ |
| logfiles | 13.0.10.3 | Enabled | GPLv3+ |
| manager | 13.0.2.5 | Enabled | GPLv2+ |
| miscapps | 13.0.3.1 | Enabled | GPLv3+ |
| miscdests | 13.0.5 | Enabled | GPLv3+ |
| music | 13.0.22.3 | Enabled | GPLv3+ |
| outroutemsg | 13.0.2.1 | Enabled | GPLv3+ |
| paging | 13.0.26.2 | Enabled | GPLv3+ |
| pagingpro | 13.0.19.6 | Enabled | Commercial |
| parking | 13.0.19.6 | Enabled | GPLv3+ |
| parkpro | 13.0.30.10 | Enabled | Commercial |
| pbdirectory | 2.11.0.5 | Enabled | GPLv3+ |
| phonebook | 13.0.5.7 | Enabled | GPLv3+ |
| phpinfo | 13.0.2 | Enabled | GPLv2+ |
| pinsets | 13.0.8 | Enabled | GPLv3+ |
| pinsetspro | 13.0.9.3 | Enabled | Commercial |
| pm2 | 13.0.3.18 | Enabled | AGPLv3+ |
| presencestate | 13.0.7 | Enabled | GPLv3+ |
| printextensions | 13.0.3.1 | Enabled | GPLv3+ |
| queueprio | 13.0.2 | Enabled | GPLv3+ |
| queues | 13.0.34.1 | Enabled | GPLv2+ |
| qxact_reports | 13.0.15.3 | Enabled | Commercial |
| recording_report | 13.0.24.6 | Disabled | Commercial |
| recordings | 13.0.30.11 | Enabled | GPLv3+ |
| restapi | 13.0.21.1 | Enabled | AGPLv3 |
| restapps | 13.0.88.5 | Enabled | Commercial |
| ringgroups | 13.0.23.1 | Enabled | GPLv3+ |
| rmsadmin | 13.0.14.1 | Enabled | Commercial |
| sangomacrm | 13.0.2.1 | Disabled | Commercial |
| setcid | 13.0.6.2 | Enabled | GPLv3+ |
| sipsettings | 13.0.24.8 | Enabled | AGPLv3+ |
| sipstation | 13.0.13.20 | Enabled | Commercial |
| sms | 13.0.11.3 | Enabled | Commercial |
| sng_mcu | 13.0.5 | Enabled | Commercial |
| soundlang | 13.0.24.5 | Enabled | GPLv3+ |
| speeddial | 2.11.0.4 | Enabled | GPLv3+ |
| superfecta | 13.0.4.1 | Enabled | GPLv2+ |
| sysadmin | 13.0.74.5 | Enabled | Commercial |
| timeconditions | 13.0.34.5 | Enabled | GPLv3+ |
| tts | 13.0.10 | Enabled | GPLv3+ |
| ttsengines | 13.0.7.1 | Enabled | AGPLv3 |
| ucp | 13.0.42.2 | Enabled | AGPLv3+ |
| ucpnode | 13.0.34.9 | Enabled | Commercial |
| userman | 13.0.76.27 | Enabled | AGPLv3+ |
| versionupgrade | 13.0.1.1 | Enabled | Commercial |
| vmblast | 13.0.8 | Enabled | GPLv3+ |
| vmnotify | 13.0.18.3 | Enabled | Commercial |
| voicemail | 13.0.54.9 | Enabled | GPLv3+ |
| voicemail_report | 13.0.13.2 | Enabled | Commercial |
| vqplus | 13.0.26.10 | Enabled | Commercial |
| weakpasswords | 13.0.2 | Enabled | GPLv3+ |
| webcallback | 13.0.11.2 | Enabled | Commercial |
| webrtc | 13.0.32.8 | Enabled | GPLv3+ |
| xmpp | 13.0.17.3 | Enabled | AGPLv3 |
| zulu | 13.0.53.2 | Enabled | Commercial |
±--------------------±------------±---------±-----------+

Can someone please help me here ? It’s impacting our business.

it looks locked by some tasks. how many users are running?

Hi!

Please take a look at this thread:

You are getting FRACK errors…

As Andrew said in this thread

they should be reported to Asterisk.

That said, did you update your packages? You are not running the latest Asterisk 13 and the one you are running, 13.17 is known have problems according to dicko’s post:

Good luck and have a nice day!

Nick

Run the upgrade scrips. Follow process here: https://wiki.freepbx.org/display/PPS/FreePBX-Distro-10.13.66

Once completed, they will bring your asterisk up to 13.17.2 or higher. Since getting on 13.17.2, we haven’t really had any of the serious issues we previously had on 13.17.0 and some earlier releases.

180 concurrent agents logged in , maximum 40 concurrent calls.
We are only using freepbx for queueing , no device is registered on it
It’s a VM with 12 GB memory , 50 GB HDD and 4 VCPU

Why it’s locked?

I am thinking of upgrading it to 13.18 instead of 13.17.2 . Would it be advisable ?

All I can vouch for is that I’m on 13.17.2 on several servers that were previously having memory leak issues (on older versions), and now we’re not. We have a similar load as you described above.

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.