How to monitor FreePBX System Firewall/IPTables

[davids]

Hi,
I am trying to monitor a problematic firewall on a fully up to date FreePBX 15.0.16.72 virtual machine but unsure how to do so.
I am using a CheckMK agent to monitor lots of items including Systemd Service Summary

Every few days we start getting hundreds of Fail2Ban emails and when we log into the PBX we see the System Firewall is disabled. CheckMK [Systemd Service Summary] still shows the same information as when the firewall was enabled.
OK - 108 services in total, 15 disabled services

I have read other forum ports about checking the firewall and iptables service so I have started trying to run manual commands to see what results I get.

On a fresh reboot of FreePBX, it shows the System Firewall and Firewall Config both have green ticks in the dashboard, however if I run these commands I get the results below them

service firewalld status
Active: inactive (dead)

service iptables status
Active: inactive (dead)

service fail2ban status
Active: active (running)

service freepbx status
Active: active (exited)

Q1 - How can firewalld and iptables be inactive (dead) if the firewall is working correctly?

fwconsole firewall stop - the firewall stops and the pbx is accessable

fwconsole firewall start - the firewall starts and the pbx is secured again

service firewalld status - Active: inactive (dead)
service iptables status - Active: inactive (dead)

If I disable and enable the firewall via the GUI then we get -
service firewalld status - Active: inactive (dead)
service iptables status - Active: active (exited)

if I run service firewalld stop
and then service firewalld start
Redirecting to /bin/systemctl start firewalld.service
we get
Firewall Rules corrupted! Restarting in 5 seconds
More information available in /tmp/firewall.log

But finally we have
service firewalld status - Active: active (running)

When I look at the firewall log I can see
'Firewall Rules corrupted! Restarting in 5 seconds
No fpbx-rtp in ipv6

Could this be why the firewall keeps disabling itself every few days? Sorry im a bit stuck.
I have lots of other FreePBXs and PBXacts running with no issues and I may even just rebuild this one, but I would like to learn how to fix it rather than just rebuild.
Any help would be greatly appreciated,
Sorry this is a bit rushed, I can add more detail if needed in the next few days.
Thx
Dave

[jerrm]

firewalld never be running under a distro install. First thing I would do is systemctl mask firewalld and reboot. Then see where things are.

[davids]

Hi Jerrm
When I checked this PBX this morning the System Firewall in FreePBX Dashboard was showing a red X.

Looks like at 2.30am the firewall shutdown. I ran the command you gave and it has linked the firewalld service to /dev/null
I shall reboot the PBX after hours this evening.
Currently the firewalld service is

service firewalld status
Redirecting to /bin/systemctl status firewalld.service
● firewalld.service
Loaded: masked (/dev/null; bad)
Active: inactive (dead)

I have only started playing with firewalld in the last few days to find a way to monitor when the system firewall stops working. Every few days the firewall changes to disabled for some reason. Only one of about 40 pbx’s having this issue so would love to find the fix and how to monitor rather than rebuild.

If the issue has anything to do with this error I can confirm the eth0 has ipv6 enabled and it has an IPv6 address.
'Firewall Rules corrupted! Restarting in 5 seconds
No fpbx-rtp in ipv6

inet6 fe80::215:5dff:xxxxx:xxxx/64 scope link
valid_lft forever preferred_lft forever
(I have changed the last two sets for characters to x’s for security)

Thanks
Dave

[davids]

After the changes yesterday I rebooted the pbx last night.

This morning the firewall was disabled

before I manually enabled the firewall via the GUI, the firewall log showed this

OUT >>> Redirecting to /bin/systemctl restart ip6tables.service
OUT >>> Redirecting to /bin/systemctl restart iptables.service
OUT >>> Redirecting to /bin/systemctl restart ip6tables.service
OUT >>> Redirecting to /bin/systemctl restart iptables.service
OUT >>> Redirecting to /bin/systemctl restart ip6tables.service
OUT >>> Redirecting to /bin/systemctl restart iptables.service
OUT >>> Redirecting to /bin/systemctl restart ip6tables.service
OUT >>> Redirecting to /bin/systemctl restart iptables.service
OUT >>> Redirecting to /bin/systemctl restart ip6tables.service
OUT >>> Redirecting to /bin/systemctl restart iptables.service
OUT >>> [2020-08-10 23:15:28] - LetsEncrypt filter: Enabled
OUT >>> [2020-08-10 23:15:27] - /sbin/iptables -w5 -W10000 -t nat -A masq-output -o eth0 -j MARK --set-xmark 0x2/0x2
OUT >>> [2020-08-10 23:15:27] - /sbin/iptables -w5 -W10000 -A fpbxinterfaces -i eth0 -j zone-external

with the firewall showing disabled on the FreePBX dashboard I ran the following itables status commands via CLI

service iptables status
Redirecting to /bin/systemctl status iptables.service
● iptables.service - IPv4 firewall with iptables
   Loaded: loaded (/usr/lib/systemd/system/iptables.service; disabled; vendor preset: disabled)
   Active: active (exited) since Tue 2020-08-11 06:30:19 UTC; 4min 55s ago
  Process: 126620 ExecStop=/usr/libexec/iptables/iptables.init stop (code=exited, status=0/SUCCESS)
  Process: 126632 ExecStart=/usr/libexec/iptables/iptables.init start (code=exited, status=0/SUCCESS)
 Main PID: 126632 (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/iptables.service

Aug 11 06:30:19 pbxname.domain.co.uk systemd[1]: Stopped IPv4 firewall with iptables.
Aug 11 06:30:19 pbxname.domain.co.uk systemd[1]: Starting IPv4 firewall with ipta....
Aug 11 06:30:19 pbxname.domain.co.uk systemd[1]: Started IPv4 firewall with iptables.
Hint: Some lines were ellipsized, use -l to show in full.

and this command
service ip6tables status
Redirecting to /bin/systemctl status ip6tables.service
● ip6tables.service - IPv6 firewall with ip6tables
Loaded: loaded (/usr/lib/systemd/system/ip6tables.service; disabled; vendor preset: disabled)
Active: active (exited) since Tue 2020-08-11 06:30:19 UTC; 5min ago
Process: 126653 ExecStop=/usr/libexec/iptables/ip6tables.init stop (code=exited, status=0/SUCCESS)
Process: 126665 ExecStart=/usr/libexec/iptables/ip6tables.init start (code=exited, status=0/SUCCESS)
Main PID: 126665 (code=exited, status=0/SUCCESS)
CGroup: /system.slice/ip6tables.service

Aug 11 06:30:19 pbxname.domain.co.uk systemd[1]: Stopped IPv6 firewall with ip6ta....
Aug 11 06:30:19 pbxname.domain.co.uk systemd[1]: Starting IPv6 firewall with ip6t....
Aug 11 06:30:19 pbxname.domain.co.uk systemd[1]: Started IPv6 firewall with ip6ta....
Hint: Some lines were ellipsized, use -l to show in full.

when I enable the firewall via GUI we get this in logs

OUT >>> [2020-08-11 06:36:50] - /sbin/ip6tables -w5 -W10000 -N fpbxfirewall
OUT >>> [2020-08-11 06:36:50] - /sbin/iptables -w5 -W10000 -N fpbxfirewall
OUT >>> ' returned 0
OUT >>>
OUT >>> [2020-08-11 06:36:49] - Wall: 'Firewall service now starting.
OUT >>> Starting firewall.
OUT >>> Redirecting to /bin/systemctl restart ip6tables.service
OUT >>> Redirecting to /bin/systemctl restart iptables.service

Do we know what happens at 03:48:03 to show this log?
full log
[2020-08-10 23:15:01] VERBOSE[7240] asterisk.c: Asterisk Ready.
[2020-08-11 03:48:03] VERBOSE[7322] asterisk.c: Remote UNIX connection

I’m also still looking to see how I can monitor the firewall if it disables and the iptables service still show as Active
Thanks
David

[davids]

I believe I have fixed the issue where the firewall was disabling itself every 24/48 hrs.

In Firewall - Advanced Settings I had enabled
LetsEncrypt Rules
Allow full Internet zone access to the Let’s Encrypt acme-challenge folder on port 80.

Now that this is disabled the firewall is staying enabled. 4 days so far

Lets Encrypt is allowed to the internet via the Firewall Services tab, perhaps they were conflicting having both enabled.

Still unsure how to monitor the firewall via an RMM package but at least its solid again.