In my previous post (My [awful] upgrade from 13 to 14) I documented that after upgrading my system from Freepbx 13 to SNG7 that the asterisk sysv service was set to DISABLED and my server was unusable because the asterisk process never started. A FreePBX developer said that it’s supposed to be that way because another process is supposed to start asterisk. Well, tonight I just updated my system (FreePBX modules and yum updates) and the exact same thing happened. The sysv service got set back to DISABLED and my server was unusable. The web interface said that it couldn’t connect to asterisk and, at the console, I verified that the asterisk service was not running. I had to set the sysv service back to ENABLED and manually start the service to get everything back to normal. Here is a screenshot to prove at after the reboot asterisk isn’t running. I would love some actual help with this since, apparently, the way my server is acting isn’t how FreePBX is supposed to act.
There is more to this story than you are telling us. Out of the (literally) thousands of installations world wide, you appear to be the only person reporting this.
The “other program” that is supposed to be starting your system is a call to fwconsole, which is used to establish the run-time environment and trouble check the system before Asterisk starts. The fact that Asterisk is not starting is a symptom of a larger problem. You shouldn’t be using the SysV start (“service asterisk start”) system to be running the system.
I don’t know what else to tell you. There is something about your startup environment that is keeping the system from running, but telling us over and over again that it isn’t running, isn’t going to help us help you troubleshoot this. We need more.
As a suggestion, try adding “fwconsole start” (or restart) to your /etc/rc.local. This might help you moving. Another place to check would be to wait for the system to get up and running, then log in as “root” and try running “fwconsole start” and see what the debug logs (in /var/log/) tell you.
Where can I start looking? What, upon system start, is supposed to issue “fwconsole start” so that I can investigate why that’s not happening? I don’t have details because I don’t know how freepbx was reworked to not rely on sysv services.
The startup scripts (including whatever the rc.local equivalent is) should be starting the system with fwconsole. With that said, it’s possible that the system is actually issuing the correct fwconsole command, and it’s failing.
The real conundrum is going to be if the system tried to start, fails (automatically) then starts OK with a manual start.
if you find that is the way the problem is happening, you need to file an Issues ticket that documents your system config as well as the startup steps that both fail and succeed.
So I found that freepbx.service is the systemd script that calls fwconsole. Checking its status I found that the process exited.
[root@voip2 ~]# systemctl status freepbx.service -l
â freepbx.service - FreePBX VoIP Server
Loaded: loaded (/usr/lib/systemd/system/freepbx.service; enabled; vendor preset: disabled)
Active: active (exited) since Tue 2018-06-26 23:11:37 EDT; 13h ago
Process: 1264 ExecStart=/usr/sbin/fwconsole start (code=exited, status=0/SUCCESS)
Main PID: 1264 (code=exited, status=0/SUCCESS)
CGroup: /system.slice/freepbx.service
Jun 26 23:11:36 voip2 runuser[5101]: pam_unix(runuser:session): session opened for user asterisk by (uid=0)
Jun 26 23:11:36 voip2 runuser[5101]: pam_unix(runuser:session): session closed for user asterisk
Jun 26 23:11:36 voip2 runuser[5111]: pam_unix(runuser:session): session opened for user asterisk by (uid=0)
Jun 26 23:11:36 voip2 runuser[5111]: pam_unix(runuser:session): session closed for user asterisk
Jun 26 23:11:36 voip2 runuser[5121]: pam_unix(runuser:session): session opened for user asterisk by (uid=0)
Jun 26 23:11:37 voip2 fwconsole[1264]: [>---------------------------] < 1 sec
Jun 26 23:11:37 voip2 fwconsole[1264]: Started Chat Server. PID is 5079
Jun 26 23:11:37 voip2 fwconsole[1264]: Running Asterisk post from Zulu module
Jun 26 23:11:37 voip2 fwconsole[1264]: PJSIP Transports for WS and WSS must be enabled in Asterisk SIP Settings under the Chan PJSIP Settings tab. Then restart Asterisk
Jun 26 23:11:37 voip2 systemd[1]: Started FreePBX VoIP Server.
Looking for the PID it says it started with, 1264, it’s not running.
My system is working at the moment, however, because I started the asterisk systemd service (which didn’t start everything, as you mentioned it wouldn’t) then reloaded the config from the web interface, which kicked off everything else that was needed.
So now I just have to figure out why freepbx.service dies… Any hints on which log I should look in to find these answers? Nothing in the default “messages” log shows errors; it looks like it worked fine although it died.
Jun 26 23:11:37 voip2 systemd: Started FreePBX VoIP Server.
Jun 26 23:11:37 voip2 systemd: Reached target Multi-User System.
Jun 26 23:11:37 voip2 systemd: Starting Multi-User System.
Jun 26 23:11:37 voip2 systemd: Starting Update UTMP about System Runlevel Changes…
Jun 26 23:11:37 voip2 systemd: Started Update UTMP about System Runlevel Changes.
Jun 26 23:11:37 voip2 systemd: Startup finished in 693ms (kernel) + 1.297s (initrd) + 1min 38.309s (userspace) = 1min 40.300s.
You mean sysv… But I get your point. I’ll disable the sysv service again and reboot tonight, then try manually running fwconsole and see what happens. More to come…
What are those other services, specifically? Can I see them in a “ps” after executing fwconsole to verify that fwconsole did its job?
Perfect. I’ll reboot tonight, check for asterisk and run the pm2 list to verify that nothing is running, then manually run fwconsole to start everything and see what happens.
If you are ‘off the reservation’ as you appear to be, then I suggest you research a bit how the various flavors of Linux ‘startup’ applications individually work
So after Grub(2) there are various flavors. mostly SysV, upstart. runit and systemd
You mention SysV and that was the mainstay for years, however more modern ‘linui’ have moved on, there was for a while upstart and even runit, but basically most all now use systemd.
Now, Asterisk source comes with templates for many of these and following a standard Asterisk installation including
make config
is a step that you probably initally used to generate your SysV startup script in /etc/init.d, similarly, that would now setup a systemd asterisk.service file .
FreePBX is not just astererisk however and relies on two key services , mysql (technically flavor independent) and a webserver (technically flavor independent) and the raw install script, you also probably used originally , has parameters you can set as to the constraints needed.
Basically Astererisk needs to be run as user:asterisk group:asterisk, this is easy to set with any particular OS or startup method, BUT . . . .
It also needs to start after your Webserver and mysql are running, also both easy to modifiy the startup script that asterisk would have done to force that condition.
BUT . . . . .
It is much easier to have a wrapper script, originally called ‘amportal’ now ‘fwconsole’ that ‘curates’ the basic order of startup of services with thecorrect permissions, and enables further ‘add-ons’ to be similarly managed.
Now although not necessarily a ‘clean’ solution, that script is found at the end of /etc/rc.local
You can certainly start FreePBX cleanly not using that script but unless you are fully aware of the how to similarly manage permission and order of prerequisites etc. I suggest you just trust /etc/rc.local/ to call fwconsole for you and further use the same script when you need to “change shi*” outside the Gooey. So also delete any other attempt to startup asterisk otherwise.
So I disabled the sysv asterisk service as recommended and restarted. Asterisk didn’t start.
[root@voip2 ~]# chkconfig
Note: This output shows SysV services only and does not include native
systemd services. SysV configuration data might be overridden by native
systemd configuration.
If you want to list systemd services use ‘systemctl list-unit-files’.
To see services enabled on particular target use
‘systemctl list-dependencies [target]’.
[root@voip2 ~]# fwconsole pm2 --list
±-------------±----±-------±---------±-------±----±----+
| Process Name | PID | Status | Restarts | Uptime | CPU | Mem |
±-------------±----±-------±---------±-------±----±----+
[root@voip2 ~]#
I started it manually with fwconsole and it came up fine.
[root@voip2 ~]# fwconsole start
Running FreePBX startup…
Taking too long? Customize the chown command, See Sangoma Documentation
Setting Permissions…
Setting base permissions…Done
Setting specific permissions…
81277 [============================]
Finished setting permissions
Running Asterisk pre from Dahdiconfig module
Wanrouter: No valid Sangoma Hardware found, if you have no Sangoma cards this is OK
DAHDi: Already started
Running Asterisk pre from Firewall module
Running Asterisk pre from Pms module
Running Asterisk pre from Sysadmin module
Running Sysadmin Hooks
Restarting fail2ban
fail2ban Restarted
Updating License Information for 21793354
Checking Vpn server
Starting Asterisk…
[============================] < 1 sec
Asterisk Started
Running Asterisk post from Dahdiconfig module
Running Asterisk post from Endpoint module
Running Asterisk post from Pagingpro module
Running Asterisk post from Restapps module
RestApps Server has already been running on PID 3489 for 19 seconds
Running Asterisk post from Ucp module
UCP Node Server has already been running on PID 4617 for 14 seconds
Running Asterisk post from Vqplus module
RestApps is not licensed.
Running Asterisk post from Xmpp module
Chat Server has already been running on PID 5138 for 10 seconds
Running Asterisk post from Zulu module
Enabling the Asterisk ARI Server
Increasing HTTP Session Limit
Changing Websocket Mode to PJSIP
Reloading PBX to update new settings…Done
Enabling PJSIP Websocket server
Websocket changed, updating webrtc clients…Done
This product is not licensed
I went to /var/log/asterisk and saw which logs were recently written to and saw freepbx.log, so I poked around looking for the time I rebooted the server. There were a million lines like this one:
[2018-Jun-27 22:05:02] [WARNING] (libraries/modulefunctions.legacy.php:7) - Depreciated Function module_getinfo detected in /var/www/html/admin/modules/cxpanel/functions.inc.php on line 76
If you’ve never used it, cxpanel is the internal name for iSymphony, which I have installed. No one uses it, so I disabled it via the sysv service (the only way it’s configured to start):
[root@voip2 asterisk]# chkconfig iSymphonyServerV3 off
I rebooted again and once I was able to login I watched the startup of the systemd service freepbx.service with
systemctl status freepbx.service -l
Eventually I saw that it started correctly, my phone came online, and the web interface worked again. No idea what iSymphony was doing, but disabling it made my server work normally again.
Eventually I have to make sure that no one actually uses it first. I just kind of assumed it wasn’t being used. I should probably be certain before uninstalling…
