Asterisk 16 won't start after reboot using latest Distro ISO

ben_itse · March 23, 2021, 8:57pm

@dicko @billsimon - further update: I read on another post back from 2016 to do a yum downgrade asterisk##*. I did that for asterisk16, which brought down the version to Asterisk 16.15.0 from 16.15.1. I rebooted and asterisk came up as expected. Tested on two PBXs. Thoughts?

billsimon · March 23, 2021, 9:18pm

https://downloads.asterisk.org/pub/telephony/asterisk/releases/ChangeLog-16.15.1

The changes from 16.15.0 to 16.15.1 look pretty small to me. I don’t see how those changes would cause a crash. But that’s an interesting finding.

dicko · March 23, 2021, 9:21pm

Interesting to see what a return to 16.15.1 would results with . .

sorvani · March 23, 2021, 11:05pm

A couple things need to be very carefully clarified here.

Is Asterisk actually crashed.
The status of Asterisk within the FreePBX GUI.

Big point of note #2 can show the flame even if Asterisk is running. I see this very regularly after a reboot. Have for years.

Asterisk is running, and calls are being processed. But the GUI does recognize it. It does require a fwconsole restart in order for everything to get started and controlled by the right processes.

So instead of telling us that Asterisk is crashed and only showing a log of Asterisk starting up, why not clarify what exactly is happening?

ben_itse · March 29, 2021, 1:16pm

Did a yum upgrade asterisk16* which brought back Asterisk 16.15.1 and asterisk is not loading back at boot.

ben_itse · March 29, 2021, 1:18pm

@sorvani - I have clarified. Asterisk is not running. It gets a PID, but it is not running as evidenced by no trunks and extensions are registering or online. Also, it does not show up on the pm2 list as previously stated. Thanks for your input.

billsimon · March 29, 2021, 1:20pm

Two more-appropriate ways to determine whether asterisk is running:

ps ax | grep asterisk

and

asterisk -r

ben_itse · March 29, 2021, 1:48pm

@billsimon - I get this when running ps aux | grep asterisk:

asterisk  1260  0.2  3.2 746288 61496 ?        S    09:33   0:01 /usr/sbin/httpd -DFOREGROUND
asterisk  1273  0.0  1.3 730756 25100 ?        S    09:33   0:00 /usr/sbin/httpd -DFOREGROUND
asterisk  1447  0.1  2.9 745956 55716 ?        S    09:33   0:00 /usr/sbin/httpd -DFOREGROUND
asterisk  1630  0.0  0.4 198096  8584 ?        Ss   09:33   0:00 /usr/bin/python /usr/local/bin/pnp_server
asterisk  2151  0.0  2.3 738112 44288 ?        S    09:34   0:00 /usr/sbin/httpd -DFOREGROUND
asterisk  2158  0.0  1.3 730748 24520 ?        S    09:34   0:00 /usr/sbin/httpd -DFOREGROUND
asterisk  2229  0.2  1.1 730660 22308 ?        S    09:34   0:01 /usr/sbin/httpd -DFOREGROUND
asterisk  2231  0.0  2.5 744328 48684 ?        S    09:34   0:00 /usr/sbin/httpd -DFOREGROUND
asterisk  2232  0.1  2.9 746540 56152 ?        S    09:34   0:00 /usr/sbin/httpd -DFOREGROUND
root      2393  0.0  0.0 115404   796 ?        S    09:34   0:00 /bin/sh /usr/sbin/safe_asterisk -U asterisk -G asteris
asterisk  2410  1.1  3.2 1737296 60508 ?       Sl   09:34   0:06 /usr/sbin/asterisk -f -U asterisk -G asterisk -vvvg -c
asterisk  2698  0.2  1.6 920252 30876 ?        Ssl  09:34   0:01 PM2 v4.5.0: God Daemon (/home/asterisk/.pm2)
asterisk  2783  0.1  1.4 979096 26568 ?        Ssl  09:34   0:00 node /var/www/html/admin/modules/core/node/fastagi-server.js
asterisk  2894  0.0  2.2 497900 42064 ?        Ss   09:34   0:00 php /var/www/html/admin/modules/restapps/restapps.php
asterisk  3030  0.2  3.0 1076624 57304 ?       Ssl  09:34   0:01 node /var/www/html/admin/modules/ucp/node/index.js
asterisk  3278  1.0  6.0 1268024 114536 ?      Ssl  09:34   0:05 letschat
asterisk  3863  0.0  1.8 739128 34476 ?        S    09:35   0:00 /usr/sbin/httpd -DFOREGROUND
asterisk  4249  0.0  1.1 730644 21412 ?        S    09:36   0:00 /usr/sbin/httpd -DFOREGROUND
root      6253  0.0  0.0 112808   968 pts/0    R+   09:43   0:00 grep --color=auto asterisk

And running asterisk -r tells me I am “Connected to Asterisk 16.15.1 currently running on c2cpbx (pid = 2410)” - but that seems to be a false positive… as I mentioned, it does get a PID at boot, but if it was running, why is it not registering any peer? Upon a fwconsole restart peers (trunks, extensions) show up as “online” on the asterisk info report page. Otherwise, they show up as offline and no calls flow in and out.

billsimon · March 29, 2021, 2:04pm

It’s not a “false positive” - it is Asterisk running; you have connected to it and seen it in the process list.

What you need to determine is where in the loading process does it stop loading modules. If FreePBX can’t connect and your phones can’t connect, then it is not loading the manager module or the SIP driver.

Please share the contents of /etc/asterisk/modules.conf here.

ben_itse · March 29, 2021, 2:38pm

;--------------------------------------------------------------------------------;
;          Do NOT edit this file as it is auto-generated by FreePBX.             ;
;--------------------------------------------------------------------------------;
; For information on adding additional paramaters to this file, please visit the ;
; FreePBX.org wiki page, or ask on IRC. This file was created by the new FreePBX ;
; BMO - Big Module Object. Any similarity in naming with BMO from Adventure Time ;
; is totally deliberate.                                                         ;
;--------------------------------------------------------------------------------;

[modules]
autoload=yes
preload = pbx_config.so
preload = chan_local.so
preload = func_db.so
preload = res_odbc.so
preload = res_config_odbc.so
preload = cdr_adaptive_odbc.so
noload = chan_woomera.so
noload = pbx_gtkconsole.so
noload = pbx_kdeconsole.so
noload = app_intercom.so
noload = chan_modem.so
noload = chan_modem_bestdata.so
noload = chan_modem_i4l.so
noload = app_trunkisavail.so
noload = chan_alsa.so
noload = chan_oss.so
noload = app_directory_odbcstorage.so
noload = app_voicemail_odbcstorage.so
noload = chan_modem_aopen.so
noload = cdr_radius.so
noload = cel_radius.so
noload = cdr_mysql.so
noload = res_phoneprov.so
noload = res_config_ldap.so
noload = res_config_sqlite3.so
noload = res_clialiases.so
noload = chan_mgcp.so
noload = cdr_custom.so
noload = app_minivm.so
noload = cel_custom.so
noload = chan_sip.so
load = format_wav.so
load = format_pcm.so
load = format_mp3.so
load = res_musiconhold.so

billsimon · March 29, 2021, 2:50pm

No hints there; another way to get a hint would be to stop FreePBX (fwconsole stop) and then run Asterisk in console mode with debug: asterisk -cd

Look for errors other than what you have already reported here.

Exit console-mode asterisk with Ctrl-C.

ben_itse · March 29, 2021, 3:00pm

Looks like the same errors:

[2021-03-29 10:57:11] WARNING[18192]: loader.c:2381 load_modules: Some non-required modules failed to load.
[2021-03-29 10:57:11] ERROR[18192]: loader.c:2396 load_modules: Error loading module 'chan_local.so': /usr/lib64/asterisk/modules/chan_local.so: cannot open shared object file: No such file or directory
[2021-03-29 10:57:11] ERROR[18192]: loader.c:2396 load_modules: Error loading module 'res_pjsip_phoneprov_provider.so', missing dependency: res_phoneprov
[2021-03-29 10:57:11] ERROR[18192]: loader.c:2396 load_modules: res_config_sqlite declined to load.
[2021-03-29 10:57:11] ERROR[18192]: loader.c:2396 load_modules: res_timing_dahdi declined to load.
[2021-03-29 10:57:11] ERROR[18192]: loader.c:2396 load_modules: res_stun_monitor declined to load.
[2021-03-29 10:57:11] ERROR[18192]: loader.c:2396 load_modules: res_xmpp declined to load.
[2021-03-29 10:57:11] ERROR[18192]: loader.c:2396 load_modules: Declined modules which depend on res_xmpp: chan_motif
[2021-03-29 10:57:11] ERROR[18192]: loader.c:2396 load_modules: chan_console declined to load.
[2021-03-29 10:57:11] ERROR[18192]: loader.c:2396 load_modules: chan_skinny declined to load.
[2021-03-29 10:57:11] ERROR[18192]: loader.c:2396 load_modules: chan_mobile declined to load.
[2021-03-29 10:57:11] ERROR[18192]: loader.c:2396 load_modules: chan_ooh323 declined to load.
[2021-03-29 10:57:11] ERROR[18192]: loader.c:2396 load_modules: res_hep declined to load.
[2021-03-29 10:57:11] ERROR[18192]: loader.c:2396 load_modules: Declined modules which depend on res_hep: res_hep_rtcp, res_hep_pjsip
[2021-03-29 10:57:11] ERROR[18192]: loader.c:2396 load_modules: app_agent_pool declined to load.
[2021-03-29 10:57:11] ERROR[18192]: loader.c:2396 load_modules: res_calendar declined to load.
[2021-03-29 10:57:11] ERROR[18192]: loader.c:2396 load_modules: Declined modules which depend on res_calendar: res_calendar_caldav, res_calendar_ews, res_calendar_exchange, res_calendar_icalendar
[2021-03-29 10:57:11] ERROR[18192]: loader.c:2396 load_modules: cdr_csv declined to load.
[2021-03-29 10:57:11] ERROR[18192]: loader.c:2396 load_modules: cdr_manager declined to load.
[2021-03-29 10:57:11] ERROR[18192]: loader.c:2396 load_modules: cdr_odbc declined to load.
[2021-03-29 10:57:11] ERROR[18192]: loader.c:2396 load_modules: app_alarmreceiver declined to load.
[2021-03-29 10:57:11] ERROR[18192]: loader.c:2396 load_modules: app_festival declined to load.
[2021-03-29 10:57:11] ERROR[18192]: loader.c:2396 load_modules: app_followme declined to load.
[2021-03-29 10:57:11] ERROR[18192]: loader.c:2396 load_modules: chan_phone declined to load.
[2021-03-29 10:57:11] ERROR[18192]: loader.c:2396 load_modules: chan_unistim declined to load.
[2021-03-29 10:57:11] ERROR[18192]: loader.c:2396 load_modules: pbx_ael declined to load.
[2021-03-29 10:57:11] ERROR[18192]: loader.c:2396 load_modules: pbx_dundi declined to load.
[2021-03-29 10:57:11] ERROR[18192]: loader.c:2396 load_modules: pbx_lua declined to load.
[2021-03-29 10:57:11] ERROR[18192]: loader.c:2396 load_modules: func_odbc declined to load.
[2021-03-29 10:57:11] ERROR[18192]: loader.c:2396 load_modules: res_pktccops declined to load.
Asterisk Ready.

Any other ideas? I feel like we are narrowing this down.

billsimon · March 29, 2021, 3:03pm

I don’t think those two should matter but you could try changing the /etc/asterisk/modules.conf file (don’t worry about the warning at the top; you can edit it).

Get rid of: preload = chan_local.so (this module doesn’t exist in current versions of Asterisk)

Set:

noload = res_pjsip_phoneprov_provider.so

ben_itse · March 29, 2021, 4:10pm

Did those changes to the file, but the issue persisted; reverted changes; rebooted.

However, that got me thinking. We do change the “SIP channel driver” setting from ‘both’ to only use ‘chan_pjsip’.

I switched that back to ‘both’, applied config, and asterisk showed as recently started in the dashboard (yellow triangle) and peers registered - so, I left it as ‘both’, rebooted, and asterisk started as expected!

So, I switched that back to “chan_pjsip”, rebooted, and asterisk is doing the same behavior as before. So, looks like we have our culprit. Any ideas on where to go from here??

billsimon · March 29, 2021, 4:18pm

Adding noload = chan_sip.so in the modules file would ensure Asterisk does not try to load it.

ben_itse · March 29, 2021, 4:20pm

So, I should try to add that and reboot to see if it loads pjsip? Is there a command that would ensure that Asterisk loads pjsip?

billsimon · March 29, 2021, 4:20pm

All modules that exist are attempted to be loaded automatically. The noload prevents it.

billsimon · March 29, 2021, 5:14pm

Just to make sure I’m not missing something: you have your trunks and extensions all configured as PJSIP before setting FreePBX to chan_pjsip only, right?

ben_itse · March 29, 2021, 6:48pm

Yes sir… it is one of the first things we do when configuring a PBX from scratch: Change the driver to only use PJSIP. No extensions or trunks are created until that is done and we only select PJSIP.

ben_itse · March 29, 2021, 8:10pm

Just verified that this line gets added automatically to the config when I select pjsip only.