Asterisk 13.8.2 Reloading Bug

Hi all,
i have a Bug with Asterisk 13.8.2

I can reload asterisk exactly for 3 times in a row most times.
Then asterisk gets stuck and all phones go offline.

I dont know what to do anymore.
I already opened an issue but Sangoma cannot help me.

http://issues.freepbx.org/browse/FREEPBX-12057

I reloaded asterisk 4 times in a row directly
I think there is a problem with reading the database.
Maybe some cron can block it?
service asterisk reload (worked)
service asterisk reload (worked)
service asterisk reload (worked)
service asterisk reload (stucking)
[2016-04-28 22:54:53] VERBOSE[15682] pbx.c: Added extension ‘510’ priority 1 to ext-local
[2016-04-28 22:54:53] VERBOSE[15682] pbx.c: Added extension ‘510’ priority 2 to ext-local
[2016-04-28 22:54:53] ERROR[11503] pbx.c: Dangerous function DB read blocked
[2016-04-28 22:54:53] VERBOSE[11503] pbx.c: Added extension ‘27105’ priority -1 to timeconditions-toggles
[2016-04-28 22:54:53] VERBOSE[15682] pbx.c: Added extension ‘510’ priority 3 to ext-local
[2016-04-28 22:54:53] VERBOSE[15682] pbx.c: Added extension ‘510’ priority 4 to ext-local
here it stops…
if i reload again now then this log appears
[2016-04-28 22:59:22] VERBOSE[16875] loader.c: The previous reload command didn’t finish yet
[2016-04-28 22:59:22] VERBOSE[11469] asterisk.c: Remote UNIX connection
[2016-04-28 22:59:22] VERBOSE[16875] asterisk.c: Remote UNIX connection disconnected
[2016-04-28 22:59:24] VERBOSE[11469] asterisk.c: Remote UNIX connection
[2016-04-28 22:59:24] VERBOSE[17031] loader.c: The previous reload command didn’t finish yet
[2016-04-28 22:59:24] VERBOSE[17031] asterisk.c: Remote UNIX connection disconnected
[2016-04-28 22:59:53] VERBOSE[11469] asterisk.c: Remote UNIX connection
[2016-04-28 22:59:53] VERBOSE[17289] loader.c: The previous reload command didn’t finish yet
[2016-04-28 22:59:53] VERBOSE[17289] asterisk.c: Remote UNIX connection disconnected

As I said in the ticket, it’s obviously an Asterisk problem, not a FreePBX problem.

If you reload slowly does it still fail?

I notice you didn’t upgrade asterisk, does it still happen with 13.8.2?

Does it only happen with ‘service asterisk reload’? Can you duplicate it with asterisk -rx "core reload"

With asterisk -rx “core reload” the same problem occours.

I also noticed that the log stops everytime anywhere and not here as described before
[2016-04-28 22:54:53] ERROR[11503] pbx.c: Dangerous function DB read blocked

The log stops anywhere, all on the same timestamp


[2016-04-28 23:40:10] VERBOSE[8249] pbx.c: Added extension ‘76’ priority 1 to park-return-routing
[2016-04-28 23:40:10] VERBOSE[8249] pbx.c: Added extension ‘77’ priority 1 to park-return-routing
[2016-04-28 23:40:10] VERBOSE[8249] pbx.c: Added extension ‘78’ priority 1 to park-return-routing
[2016-04-28 23:40:10] VERBOSE[8249] pbx.c: Registered extension context ‘park-hints’; registrar: pbx_config
[2016-04-28 23:40:10] VERBOSE[8249] pbx.c: Including context ‘park-hints-custom’ in context ‘park-hints’
[2016-04-28 23:40:10] VERBOSE[8249] pbx.c: Including context ‘parkedcalls’ in context ‘park-hints’
[2016-04-28 23:40:10] VERBOSE[8249] pbx.c: Added extension ‘71’ priority 1 to park-hints
[2016-04-28 23:40:10] VERBOSE[8249] pbx.c: Added extension ‘71’ priority -1 to park-hints
[2016-04-28 23:40:10] VERBOSE[8249] pbx.c: Added extension ‘72’ priority 1 to park-hints
[2016-04-28 23:40:10] VERBOSE[8249] pbx.c: Added extension ‘72’ priority -1 to park-hints
[2016-04-28 23:40:10] VERBOSE[8249] pbx.c: Added extension ‘73’ priority 1 to park-hints
[2016-04-28 23:40:10] VERBOSE[8249] pbx.c: Added extension ‘73’ priority -1 to park-hints
[2016-04-28 23:40:10] VERBOSE[8249] pbx.c: Added extension ‘74’ priority 1 to park-hints
[2016-04-28 23:40:10] VERBOSE[8249] pbx.c: Added extension ‘74’ priority -1 to park-hints
[2016-04-28 23:40:10] VERBOSE[8249] pbx.c: Added extension ‘75’ priority 1 to park-hints
stuck…

Notice 2 is that reload works first time, after 2 times here the reload stucked.

Upgrade to 13.8.2 is should be only done with an official Upgrade script on our production system i think?
Am i right?

^^^

Also, ‘yum update’ is always fine on FreePBX Distro machines.

http://downloads.asterisk.org/pub/security/AST-2013-007.html

not a bug, a feature

[2016-04-28 22:54:53] ERROR[11503] pbx.c: Dangerous function DB read blocked

is generated by someone setting live_dangerously=no

Doesnt matter if i reload slowly. It stucks everytime anotherwhere with no error in the log at the end.

After changing some Queue settings in FreePBX and apply, the reload doesnt work and i also have no error. The log stops at the same second with hundreds of entrys like this
[2016-04-28 23:40:10] VERBOSE[8249] pbx.c: Added extension ‘77’ priority 1 to park-return-routing
[2016-04-28 23:40:10] VERBOSE[8249] pbx.c: Added extension ‘78’ priority 1 to park-return-routing
.
.
.

Upgraded to 13.8.2

fwconsole reload (OK)
1 minute later
fwconsole reload (OK)
1 minute later (i still have not changed anything)
fwconsole reload (freepbx says OK but all phones and trunks are down now)

sometimes after the stuck reload i have to restart httpd to get back to the freepbx web admin

Sounds like you need to rebuild the whole server

oh no… i dont hope you mean make a fresh install of the distro and configure the whole server new?

Sounds like a lot of work :frowning:

I have seen this issue on only two** systems, both initially running Asterisk 13.7.2 and upgrading to 13.9 did not fix. As in the first post, a newly booted system will allow you to do a core reload 3 or 4 times, then the next will lock things up. Some notes:

  • Asterisk does not crash, there is no core dump.
  • CPU and Mem usage are not an issue.
  • If you use the ‘Apply Config’ button the GUI stays unresponsive
  • ssh session(s) seem unaffected.
  • It makes no difference if there is voice traffic or not, of if systems are under load or not.
  • Both systems with this issue had 'core show taskprocessors" showing 300+ entries and I observed asterisk on one system crash with 500 task processors, though I don’t know if that is related or not. High taskprocessor count seems to be normal for Asterisk 13, all the Asterisk 11 systems I’ve observed stay at single digits.
  • Restarting the asterisk service does not fix, rebooting the PBX gives you 3-4 reloads then the cycle starts again.

Changing Asterisk to 11.21.2 did fix the issue. I have been unable to reproduce this issue on any of my test systems, so I am speculating that there is some specific config that triggers this with Asterisk 13.x

**edit - another system known bringing total to 3

I removed all digium modules and upgraded to 13.9.1

My asterisk dont crash anymore.

Works fine now!

1 Like