Firewall no longer starts on reboot


(Jean Sebastien Carle) #1

For at least a week now (since I noticed), more than 50% of our 120+ FreePBX machines fail to start the firewall after a reboot. The only time consuming fix so far has been to logon to the web UI of each server, disable the firewall and enable it again in order to get the service started. This is happening both on FreePBX 14 and FreePBX 15, both of which are fully up to date (both in terms of yum and modules).

This is what the firewall shows after reboot (even several hours or days after boot):

# iptables --list
Chain INPUT (policy ACCEPT)
target     prot opt source               destination
fail2ban-recidive  all  --  anywhere             anywhere
fail2ban-BadBots  tcp  --  anywhere             anywhere             multiport dports http,https
fail2ban-FTP  tcp  --  anywhere             anywhere             multiport dports ftp
fail2ban-apache-auth  all  --  anywhere             anywhere
fail2ban-SSH  tcp  --  anywhere             anywhere             multiport dports ssh
fail2ban-SIP  all  --  anywhere             anywhere
fail2ban-SIP  all  --  anywhere             anywhere

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Chain fail2ban-BadBots (1 references)
target     prot opt source               destination
RETURN     all  --  anywhere             anywhere

Chain fail2ban-FTP (1 references)
target     prot opt source               destination
RETURN     all  --  anywhere             anywhere

Chain fail2ban-SIP (2 references)
target     prot opt source               destination
RETURN     all  --  anywhere             anywhere
RETURN     all  --  anywhere             anywhere

Chain fail2ban-SSH (1 references)
target     prot opt source               destination
RETURN     all  --  anywhere             anywhere

Chain fail2ban-apache-auth (1 references)
target     prot opt source               destination
RETURN     all  --  anywhere             anywhere

Chain fail2ban-recidive (1 references)
target     prot opt source               destination
RETURN     all  --  anywhere             anywhere

(Lorne Gaetz) #2

We have a report of delayed start of voipfirewalld on boot, but no reports (of which I’m aware) of failure to start at all.


(Jean Sebastien Carle) #3

I just found this in firewall.log:

PHP Warning:  mkdir(): No such file or directory in phar:///var/www/html/admin/modules/firewall/hooks/voipfirewalld/lock.php on line 22
PHP Fatal error:  Uncaught exception 'Exception' with message 'Can't create /var/run/asterisk/firewall directory' in phar:///var/www/html/admin/modules/firewall/hooks/voipfirewalld/lock.php:25
Stack trace:
#0 phar:///var/www/html/admin/modules/firewall/hooks/voipfirewalld/firewall.php(14): FreePBX\modules\Firewall\Lock::canLock('firewall')
#1 /var/www/html/admin/modules/firewall/hooks/voipfirewalld(3): include('phar:///var/www...')
#2 {main}
  thrown in phar:///var/www/html/admin/modules/firewall/hooks/voipfirewalld/lock.php on line 25
PHP Warning:  mkdir(): No such file or directory in phar:///var/www/html/admin/modules/firewall/hooks/voipfirewalld/lock.php on line 22
PHP Fatal error:  Uncaught exception 'Exception' with message 'Can't create /var/run/asterisk/firewall directory' in phar:///var/www/html/admin/modules/firewall/hooks/voipfirewalld/lock.php:25
Stack trace:
#0 phar:///var/www/html/admin/modules/firewall/hooks/voipfirewalld/firewall.php(14): FreePBX\modules\Firewall\Lock::canLock('firewall')
#1 /var/www/html/admin/modules/firewall/hooks/voipfirewalld(3): include('phar:///var/www...')
#2 {main}
  thrown in phar:///var/www/html/admin/modules/firewall/hooks/voipfirewalld/lock.php on line 25

I will check other systems to see if I find similar errors.


(Jean Sebastien Carle) #4

On a second system, the error is a bit different but similar:

Unparseable output from getservices - ["Exception: Asterisk is not connected in file \/var\/www\/html\/admin\/libraries\/php-asmanager.php on line 242","Stack trace:","  1. Exception->() \/var\/www\/html\/admin\/libraries\/php-asmanager.php:242","  2. AGI_AsteriskManager->send_request() \/var\/www\/html\/admin\/modules\/firewall\/Smart.class.php:447","  3. FreePBX\\modules\\Firewall\\Smart->getPjsipContacts() \/var\/www\/html\/admin\/modules\/firewall\/Smart.class.php:437","  4. FreePBX\\modules\\Firewall\\Smart->getRegistrations() \/var\/www\/html\/admin\/modules\/firewall\/Smart.class.php:69","  5. FreePBX\\modules\\Firewall\\Smart->getAllPorts() \/var\/www\/html\/admin\/modules\/firewall\/Firewall.class.php:1110","  6. FreePBX\\modules\\Firewall->getSmartPorts() \/var\/www\/html\/admin\/modules\/firewall\/bin\/getservices:22"] - returned 1
PHP Warning:  Error while sending QUERY packet. PID=4377 in phar:///var/www/html/admin/modules/firewall/hooks/voipfirewalld/firewall.php on line 378
Unable to connect to Database, sleeping 2 seconds and retrying. (1)
Unable to connect to Database, sleeping 2 seconds and retrying. (2)
PHP Warning:  mkdir(): No such file or directory in phar:///var/www/html/admin/modules/firewall/hooks/voipfirewalld/lock.php on line 22
PHP Fatal error:  Uncaught exception 'Exception' with message 'Can't create /var/run/asterisk/firewall directory' in phar:///var/www/html/admin/modules/firewall/hooks/voipfirewalld/lock.php:25
Stack trace:
#0 phar:///var/www/html/admin/modules/firewall/hooks/voipfirewalld/firewall.php(14): FreePBX\modules\Firewall\Lock::canLock('firewall')
#1 /var/www/html/admin/modules/firewall/hooks/voipfirewalld(3): include('phar:///var/www...')
#2 {main}
  thrown in phar:///var/www/html/admin/modules/firewall/hooks/voipfirewalld/lock.php on line 25

(Jean Sebastien Carle) #5

Third system has a slighty different error, but same ending:

Unparseable output from getservices - ["Exception: Asterisk is not connected in file \/var\/www\/html\/admin\/libraries\/php-asmanager.php on line 248","Stack trace:","  1. Exception->() \/var\/www\/html\/admin\/libraries\/php-asmanager.php:248","  2. AGI_AsteriskManager->send_request() \/var\/www\/html\/admin\/modules\/firewall\/Smart.class.php:447","  3. FreePBX\\modules\\Firewall\\Smart->getPjsipContacts() \/var\/www\/html\/admin\/modules\/firewall\/Smart.class.php:437","  4. FreePBX\\modules\\Firewall\\Smart->getRegistrations() \/var\/www\/html\/admin\/modules\/firewall\/Smart.class.php:69","  5. FreePBX\\modules\\Firewall\\Smart->getAllPorts() \/var\/www\/html\/admin\/modules\/firewall\/Firewall.class.php:1477","  6. FreePBX\\modules\\Firewall->getSmartPorts() \/var\/www\/html\/admin\/modules\/firewall\/bin\/getservices:22"] - returned 1
PHP Warning:  mkdir(): No such file or directory in phar:///var/www/html/admin/modules/firewall/hooks/voipfirewalld/lock.php on line 22
PHP Fatal error:  Uncaught exception 'Exception' with message 'Can't create /var/run/asterisk/firewall directory' in phar:///var/www/html/admin/modules/firewall/hooks/voipfirewalld/lock.php:25
Stack trace:
#0 phar:///var/www/html/admin/modules/firewall/hooks/voipfirewalld/firewall.php(14): FreePBX\modules\Firewall\Lock::canLock('firewall')
#1 /var/www/html/admin/modules/firewall/hooks/voipfirewalld(3): include('phar:///var/www...')
#2 {main}
  thrown in phar:///var/www/html/admin/modules/firewall/hooks/voipfirewalld/lock.php on line 25

(Jean Sebastien Carle) #6

The common denominator seems to be this: 'Can't create /var/run/asterisk/firewall directory'

I suppose this can be fixed with fwconsole chown, however is there a way to replicate the disable / enable from the GUI as a command-line? Some sort of firewall restart command?


(Lorne Gaetz) #7
fwconsole firewall start|stop|restart

(Lucas Ryan) #8

Hello,
Has there been any update on this? I am running into the same thing on both a FPBX13 and FPBX15 server. fwconsole firewall start does not seem to help. My “solution” is to back down the firewall module to the previous one.


(Jared Busch) #9

@lgaetz so my system was updated earlier this morning. (see the thread about colors)…

After seeing this again, I rebooted via sysadmin. Firewall is not running. It should not be in delayed startup.

So I waited…

Nope still not started.

and OMFG the crap hitting my system.

Side note, Asterisk showing not running in the GUI after the reboot…

One sudo fwconsole restart later, both issues are fixed… The firewall is running and Asterisk is showing online in the GUI.


(Lorne Gaetz) #10

I can’t repro this @sorvani. My 15 system running firewall 15.0.6.32 starts within 3 minutes on boot, but we have reports of start being delayed for as long as 5 minutes. What version of Firewall? I can’t think of anything in Firewall that might affect how/when Asterisk starts.


(Jared Busch) #11

I dumped my entire firewall.log to pastebin.
https://pastebin.freepbx.org/view/72083e0a#L650
1603908456 < about the time I rebooted

1603908456: wait_response returned false. Restarting monitoring thread.
PHP Warning:  Error while sending QUERY packet. PID=27971 in phar:///var/www/html/admin/modules/firewall/hooks/voipfirewalld/firewall.php on line 378
Unable to connect to Database, sleeping 2 seconds and retrying. (1)
PHP Warning:  mkdir(): No such file or directory in phar:///var/www/html/admin/modules/firewall/hooks/voipfirewalld/lock.php on line 22
PHP Fatal error:  Uncaught exception 'Exception' with message 'Can't create /var/run/asterisk/firewall directory' in phar:///var/www/html/admin/modules/firewall/hooks/voipfirewalld/lock.php:25
Stack trace:
#0 phar:///var/www/html/admin/modules/firewall/hooks/voipfirewalld/firewall.php(14): FreePBX\modules\Firewall\Lock::canLock('firewall')
#1 /var/www/html/admin/modules/firewall/hooks/voipfirewalld(3): include('phar:///var/www...')
#2 {main}
  thrown in phar:///var/www/html/admin/modules/firewall/hooks/voipfirewalld/lock.php on line 25
Starting firewall.
1603908913: Wall: 'Firewall service now starting.
 
' returned 0

#12

https://issues.freepbx.org/browse/FREEPBX-21923


(Luke C) #13

I will be spinning up a new install next week, hopfully it will be stable by then.


(Jared Busch) #14

I cannot reproduce it at will, but I can see it happen enough to know there is something weird…

I just rebooted my PBX again, and this time everything came up. There were no updates or changes all day.


(Lorne Gaetz) #15

New version published yesterday. Others are reporting success with versions 15.0.6.34 or 13.0.60.18