Sangoma Firewall banning "work from home" folks with Sangoma phones

Thank you. I did some of that but not all, I’ll double check my work tomorrow and report back.

We have had to do the same.
At the moment, that is the only way we can confirm that the remote phones will not be blocked by the Firewall and ID.

This is not fail2ban, this is the FreePBX firewall. I have encountered the same issue with many clients. Seems like any disconnect/reconnect is causing them to get blocked by the FreePBX firewall.

This is with pjsip endpoints and Responsive Firewall enabled for pjsip. Since there is no real firewall logging, this is difficult to troubleshoot.

Adding their IP to the firewall prevents the issue, but these users have dynamic IPs which is what the Responsive Firewall function is supposed to accommodate. I wonder could it be something like BLF subscriptions prior to registration being interpreted as failed logins or something like that…

2 Likes

Setup a DDNS client on the remote side, whitelist it in the firewall and you can now turn off the responsive firewall.

That’s not practical on scale

1 Like

Dynamic DNS may be a good workaround for some users, but not others. In many cases we are going to be asking people to put a DDNS client on their personal devices because their work device uses a VPN and would therefore report the wrong IP address.

I suspect the firewall is being indiscriminate about the kind of traffic it uses to rate limit and block users, and not taking into account whether or not the user has successfully authenticated in the past. These devices are not failing to authenticate, I suspect they are just sending some miscellaneous traffic at the server during the time between when the phone goes offline and when it re-registers.

This is not, it seems, a new issue, here’s a few similar threads:

Hopefully it’s something that can be improved in the firewall in the near future.

1 Like

No, it likely CAN’T be improved not easily due to a fundamental networking issue. Here is the most likely problem

When the phone is running at your employee site it’s traffic is going out on to the Internet through an address translator in their router. That translator has a set of “connection track timeouts” That is, when the phone registers in, it setups up a translation entry in the memory of the user’s router at their house.

If the TCP registration sends keepalives periodically, then the user’s router will keep the TCP connection slot open. If it does not then eventually the router will expire the connection slot, and when the phone then tries to send something it will discover it’s unregistered (and maybe try to register again) I suspect the excessive registration attempts (when FreePBX thinks the phone is already registered) are triggering the firewall. If someone wants to plug a Linux system into a monitoring port on their ethernet switch their FreePBX system is on, then setup packet filters and run a trace under Wireshark they can probably get a more exact description of what is happening.

Different models of phones handle this differently. I have experimented with a number of different models and some of them you can set a configuration that will turn on keepalives and some you can’t. Some router models will pay attention to keepalives and some won’t.

And then there are UDP connection tracking timeouts. Yes, UDP is connectionliess but a “slot” has to be setup in the router’s memory to handle the translation and that can expire also.

Here are some “fixes”/hacks you can try:

For the Netgear R7000 put the inside interface into the DMZ zone. (the end user must have a R7000 obviously) This is mentioned here:

For dd-wrt the timeouts can be modified in the web interface as discussed here:

https://wiki.dd-wrt.com/wiki/index.php/Router_Slowdown

There are 2 more discussions on this that explain the pros and cons of fiddling with these settings and why different router firmware has set what it has set in dd-wrt and one in Linksys’es forum:

https://svn.dd-wrt.com/ticket/947
https://svn.dd-wrt.com/ticket/3559

What I have done to fix this is the following:

  1. Assign an available private subnet consistent with the office (and with all other remote worker subnets) for the user
  2. Turn off the translator in the user’s cable modem, or DSL modem.
  3. Install a router capable of running dd-wrt (or reflash the users router if they already own one capable of running dd-wrt)
  4. Setup an OpenVPN connection from the user’s router to an Untangle server at the main office (the dd-wrt router is setup as an OpenVPN client the Untangle box is setup as an OpenVPN server)
  5. Configure the DHCP server in the users router to pass the address of a TFTP server located at the office.
  6. Configure dynamic dns on the dd-wrt router
  7. Setup an access rule in the OpenVPN server to only permit the remotes access to the FreePBX server and TFTP server
  8. Renumber the users home network to fit in to the office subnet (renumber to 192.168.101.x etc.)

This solution has no software licensing costs, it does NOT require that VoIP be passed through ANY translators, and I can reprogram the phones easily by changing the configuration files in the TFTP server. It is rock solid stable, does not require a DDNS client to be run on a PC, (if you want to do that you can do it on the dd-wrt ) and I use a free dydns server on the Internet which is predefined in modern dd-wrt configurations.

Routers already loaded with dd-wrt can be purchased from dd-wrt or you can download the firmware and reflash your own devices.

Untangle is also free and runs fine on any old Core i3 with 8gb ram you have lying around (just install another ethernet card in it)

And 99 times out of 100 your dd-wrt router is a superior wifi access point for the end user over their cable company’s POS wifi-enabled cable modem.

A UDP timeout value that is set too short would cause the phone to go unreachable by the PBX but should not cause the user’s public IP to get blocked by the firewall.

If the phone can stay connected without issue when the user’s public IP is added to the firewall’s Trusted zone, it’s hard to blame the user’s network.

1 Like

UDP timeouts have previously long ago been increased to mitigate those related issues. I get the other guys thought on the ports from nat changing but it’s hard to say. I’m going to try provisioning via open VPN client to see results.

My understanding is that IPs from which devices successfully register; the PUBLIC IP, not the internal LAN IP, are supposed to be whitelisted for some period in the Responsive Firewall. This is what seems to be not working properly, as if that was happening, re-registrations wouldn’t be a problem.

We have this issue this AM at a client with one remote phone connected via router to router VPN. The remote handset has been in place almost a year without issue, and they just informed us that it’s disconnecting. It was blocked in the Responsive Firewall.

Sangoma folks, I hope that this can get bumped up the list; it’s something that has suddenly become critical with so many people working remotely.

When you say UDP timeouts have been increased to mitigate those issues the problem is that FreePBX does not have control over the UDP timeouts on the intermediate translator routers. There is really nothing that can be done outside of that other than the freepbx server and the phone being configured to just hammer the heck out of each other sending vast quantities of keepalives. But that also assumes that the router or routers in between aren’t doing something stupid. My testing has shown that maintaining a VoIP phone registration through every translating router I have tested with including Cisco’s enterprise code, is a tremendous carpshoot.

I myself run a softphone on my laptop and use it from coffee shops, etc. when I am on the road so I am no stranger to this. Sometimes I can leave the softphone going for an hour or two. Other times I have to close and reopen the softphone program to make a call every time. You are welcome to try to solve this problem with theory but I think you are wasting your time and for the amount of trouble you will go through tuning that prefect setup, a VPN could have been setup with a lot less trouble.

If you read my post above, you’ll see that the Responsive Firewall is also blocking remote IPs connected via site to site VPNs (no NAT or PAT), so going to the trouble of setting up VPNs won’t fix the root issue.

How could that be? Every VPN gear I have ever worked with uses a subnet of private IP addresses for the remotes that is routable and known to the VPN concentrator, and never changes. So if you cut 192.168.100.0/24 for your VPN server to use to assign remotes, that would be added into the list of private subnets in the FreePBX configuration and exempted from the firewall.

Surely you don’t allow the firewall to block your private subnets? Because if you do I think you don’t understand the purpose of the firewall. The firewall exists for when people port forward outside IP addresses to an inside IP address on your FreePBX system. It detects when malevolent people on the outside are repeatedly trying to guess authentication credentials. If you have a malevolent player on the INSIDE that is trying to guess credentials then your security is already compromised and the FreePBX firewall just gives a false sense of security.

You are aware that FreePBX has a spot in the configuration specifically for you to list all your private inside subnets, aren’t you? Perhaps you are not?

I’m not going to spend time explaining site to site VPNs. DHCP is not involved, though. It’s straight routing.

And the point is that we should not have to whitelist IPs like this; the responsive firewall should NOT be blocking them. And it is. Across a half-dozen installs right now.

Silly question, but for your sites connected via VPN, do you have the INTERFACE they connect on trusted in the Sangoma Firewall? You still have to do that regardless if they’re VPN, public IP, etc., or whitelist the VPN subnets

Not until today, and it had been fine for months like that; with the Responsive Firewall not getting in the way.

Today I found the IP of the remote phone blocked. I unblocked and whitelisted the remote subnet.

So it will work, and this isn’t how it’s supposed to work, and it’s an enormous problem for one-off phones connecting over the internet from locations with dynamic public IPs. Every day we’re having to update IPs in the whitelists.

Are you able to post the relevant fail2ban logs? I’m curious if it’s the same trigger that’s blocking these ips each time.

The firewall is intended to protect from dynamic endpoints, from phones logging in from untrusted networks where there is no consistency in the IP assignments. It is useless if the FreePBX server NEVER has to allow for outside incoming SIP from the Internet. If you are an admin of a FreePBX system that doesn’t have to expose it to incoming VoIP calls from the outside you WOULDN’T. You would just turn off the firewall completely since it’s unnecessary and make sure there’s no port forwarding to it.

You have a network where all your phones are coming in from known subnets that you assigned. You therefore have a solution in front of you that would take 10 minutes to implement and you would be done with it and you could go on to other things - and that solution is add all your trusted subnets to the list of trusted subnets so the firewall ignores them.

I don’t completely understand your complaint. On one post you say all your phones are coming in from VPNs. In another you say you also have problems from phones coming in from dynamically assigned public IP addresses.

As I said, there are problems with running extensions over the Internet not inside a VPN that are more than just the firewall acting up. The firewall is a poor substitute, a last ditch attempt to make a horrible situation better. Unless your remote phones are all encrypting their calls, your VoIP going over the public internet can easily be evesdropped there’s some fun videos on YouTube that show how to use Cain and Able to do it. Lots of phones out there have very poor password control they won’t allow passwords longer than 8 characters, and many admins use poor passwords anyway. So why fight against use of a VPN? You can, for free, setup an OpenVPN server, run OpenVPN clients on your single dynamic hosts and softphones on those machines. That solves the firewall issue as well as protects your calls as well as fixes any timeout issues.

I have news for you we got another 10 months to go before a vaccine is going to be widely available. Until then you are going to have a lot of folks at home. If you are going to have your security with it’s back door hanging wide open for a year, your gonna get gunned. You are acting like this business is “just a few months and we can go back to normal” and this remote from home stuff is a bandaid you are looking forward to tearing off.

Grow the eff up and start taking this seriously.

I am the OP, I think my question has become lost in the arguing by others. My question was not about VPN, but about dynamic IP endpoints floating around behind NAT endpoints, on the internet getting blocked by the firewall when there is no abnormal behavior. My thread was kind of hijacked

3 Likes

We support 20+ FreePBX installs. Some have no external endpoints. Some are dynamic only. Some are branch office with site to site VPNs: It’s every combination.

And my point is that the Responsive Firewall is blocking IPs that it shouldn’t. These aren’t all natted devices connecting over the internet, devices connecting over site to site VPNs via routing are affected, as are devices that have been working fine for months in their current config.

@thx2000 I will see if I can retrieve the fail2ban logs. I’m not sure where they are, and will hunt.

1 Like