Firewall causing random disconnects, even when disabled?

FreePBX Security
1

Moved our FreePBX to Vultr in March when employees began working remotely due to covid. It’s worked well until last week when–depending on which WAN connection you were on–some phones randomly disconnected & the GUI wouldn’t load. It effected one WAN, then 2, then back to 1.

To keep this brief, I found that disabling/re-enabling the firewall module or doing fwconsole firewall disable immediately restored connections. I turned the FW back on & things would be fine for a while (hours) then break again. So I disabled the FW but I’m still getting disconnects. (In fact, the GUI stops responding after several minutes now.) Doing fwconsole firewall disable still immediately restores connections even though the FW is already off.

I don’t see anything unusual in the logs.

Any thoughts on this would be greatly appreciated.

(Running FreePBX 14.0.13.28/Asterisk 13.32.0 with no updates applied in the past month. Been using Vultr’s firewall the entire time. Disabling Vultr’s FW has no effect on this issue.)

2

Vultr provides a ‘console’ from where to diagnose such problems

3

@dicko, how do you think I’m able to use fwconsole since I can’t connect to the server :wink: (But sincerely, I have learned a great deal from your many contributions on this site. Many thanks for sharing & continuing to share your expertise!)

4

Are the WAN’s behind NAT? If all the endpoints look like one IP to the PBX, then one misbehaving client can block the entire location.

5

Because if you go to your vultr login , for each of your machines there is a “view console” button, that gives you when clicked a ‘shell’ on your machine, this shell is effectively a TTY so you will be able to run fwconsole or anything you want from there

6

Yes, excellent point @jerrm. And in fact, I removed one extension around the time this weirdness started. That phone was trying to connect & failing bc it was removed.
But I added our 3 WANs to trusted zones, & now, the firewall is off. So how can connections be failing when it’s off? Weird.

7

Sorry, yes, my comment was to say that I’m already using the Vultr console to issue these commands (since I can’t always connect to FreePBX). But thanks for the input.

8

Existing connections dropping points to fail2ban. Unfortunately that can’t be disabled in the distro (the GUI has a “stop” button, but it will eventually restart and come back).

If the WAN IPs are static, make sure they are whitelisted under sysadmin->intrusion detection.

If they aren’t static, then adjust the fail2ban settings to effectively neuter it.

1 Like
9

In general, iptables rules exposing IP addresses are shown with

iptables -L -n

locate the culprit and the ‘Chain’ would be identified, It might be Fail2Ban or it might be "The Firewall’

From a TTY you can

iptables -F

from an ssh session

iptables -P INPUT ACCEPT
iptables -P FORWARD ACCEPT
iptables -P OUTPUT ACCEPT
iptables -t nat -F
iptables -t mangle -F
iptables -F
iptables -X

is safer

1 Like
10

More correct to say it can’t be disabled in the GUI. The following should be enough to disable from the console:

systemctl stop fail2ban
systemctl disable fail2ban
systemctl mask fail2ban

But still a possibility a system update might bring it back.

1 Like
11

Likely fail2ban if existing active connections are dropping.

Fail2ban happens before the state rule, all “The Firewall” FreePBX rules are after.

12

there is a fail2ban-client that can ban and unban whitelist ips, query, stop and start any jail.

14

Quick summary for newbs like me (& pls correct whatever I get wrong!).

I was under the mistaken impression that FreePBX:

  1. has a single firewall controlled in Connectivity > Firewall.
  2. adding our WAN IPs in the Trusted Zone would permit us access.
  3. “Disabling Firewall” turns off the firewall.

Turns out:

  1. The firewall module is but one feature of the firewall (iptables). Another important firewall setting is located (as @jerrm pointed out) in Admin > System Admin > Intrusion Detection: Whitelist.
  2. Zone settings only apply to the Responsive Firewall feature, not the entire firewall.
  3. “Disabling Firewall” does not disable the entire firewall but instead only disables the Responsive Firewall feature.

In my case, I thought the firewall was off…it wasn’t. Some condition was causing our WAN to be rejected by fail2ban. (This was determined by (as @dicko instructed) issuing iptables -L -n. This showed the firewall was indeed still running & our WAN IP was being rejected. Note: You may also be able to see this in the gui at Admin > System Admin > Intrusion Detection: “IP’s that are currently banned.”)

It also turns out that by entering fwconsole firewall commands I was actually just clearing out the fail2ban list. While this temporarily reestablished connections, the real solution was adding our WAN IPs into the Intrusion Detection: Whitelist.

Huge thanks to @dicko and @jerrm. TIL.

15

Hard to comment on FreePBX’ “intrusion detection” it is in fact a very ‘long in the tooth’ version of Fail2Ban with a very restricted ability to control itself as it only does a very old “Asterisk” jail

If you care to get a way better version then go to their site and get

https://github.com/fail2ban/fail2ban/archive/0.10.tar.gz

RTFM and you can protect all your other open services like SSH and MAIL HTTPS with it’s pre-built ‘jails’ , Sorry no Gooey here

1 Like
16

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.