Best on-prem solution for handling international staff

General Help
1

Hi all,

I’ve made a few server moves in the US, and currently have about 7 years experience with FreePBX/Asterisk, plus another 2 years with Cisco.

I’ve moved our FreePBX server from Chicago/Upcloud to NYC/DigitalOcean, and while it performs a bit better, it still just has so many mystery connection issues with our staff in Ecuador, as well as me when I’m in Europe.

By mystery, I mean lost connections within SIP clients (we mainly use Bria and some Poly phones), horrendous latency, and tons of dropped packets. My background and intuition immediately tell me this isn’t worth more tinkering. It’s simply that we are trying to connect from Tenerife, Spain, to a server in NYC for voice purposes. The latency will be poor and packets will be dropped. Latency here to our server is currently about 140ms; in NYC it’s closer to 10ms and performs great.

Again, not looking to troubleshoot…just asking for hosting suggestions here. I do use Gcloud personally for my VPN and it does perform reasonably most everywhere I am. I’ve thought about switching to that. Their “premium” plan apparently uses exit nodes all over the world connected by fiber, thus ensuring good performance.

If it absolutely comes down to it, I can run a load balancer with a few international servers, but I don’t think this will be a reasonable solution. It sounds more like a week of messing around and not getting anywhere.

Any other ideas? Appreciate it.

2

What are ‘mystery connections issues’ ?
sngrep ?

3

Gladly - see below. I’m honestly not familiar with this tool but using just common sense, I am wary of these INVITE and REGISTER tags that do not correlate with our extensions at all. It seems like an awful lot of traffic for just a few staff.

By ‘mystery,’ I’m mostly expressing my frustration about the latency, dropped packets, low quality calls, etc. It has become so embarrassing that calling customers yields such a choppy call, we just default to Google Meet. Even our weekly meeting used to be on a conference line, but everyone (esp outside of the US) kept cutting out. Basically, the further from our NYC server, the worse the call quality becomes. When I am in NYC it is top notch. In Europe, it is horrendous

Thanks for the help and surely ask me if you need more details. Traveling tomorrow if I don’t reply!

image
image2350×1284 249 KB

4

As ever , you are under attack. . . .

It is always painful to watch but as long as you use UDP/5060 for your transport (and everyone else who does can take notice) , it just won’t go away, despite the claims of many that that whole concept is “Theatrical” and your firewall will protect you, it is them who will have to come up with their ‘dutch boy thumb defense’ that is actually effective as you needlessly continue to leave that open.

5

Although most of the SIP traffic is malicious, see Whois Lookup Captcha , the Msgs column shows 1 for these packets (there was no reply), showing that FreePBX Firewall successfully blocked them; few resources were consumed. (sngrep sees inbound traffic before the firewall and outbound traffic after it.) Even if there were enough to seriously degrade system performance, this would also affect calling in NYC, so IMO your problem is unrelated to undesired traffic.

I have no idea how Tenerife gets its internet connectivity, but would not be surprised if their fiber ran via Western Sahara rather than Spain. Also, the local connection may be overloaded, you may be on crappy Wi-Fi, etc. Although your observation of good quality with Google Meet indicates decent connectivity, I believe that Meet’s use of compression codecs, forward error correction and better jitter buffering is allowing it to work well under impaired conditions.

I have a FreePBX on Vultr in Los Angeles, with extensions in US, Paris and Bangkok. The foreign phones do not have any issues with voice quality or dropped calls. Of course, latency is much worse, but when calling the US that is unavoidable. If we had much domestic traffic on the foreign extensions, we’d set up branch PBXes in Western Europe and Southeast Asia.

IMO you need to find where the trouble lies. If it’s in your LAN or with your ISP, quality will be poor no matter where you host the PBX. Even an on-site PBX wouldn’t help, because the trunk connections would have poor quality.

I would start by running simultaneous pings in 4 windows, to the local router, ISP first gateway, 8.8.8.8, and 209.97.155.54 . (Do this at a time when you observe voice quality issues.) Compare packet loss and jitter.

Other things to try, if they are easy, include setting up one extension to use Opus codec (with Asterisk transcoding), and configuring a phone to access a trunking provider directly (outbound only), bypassing the PBX.

6

Fellas, I appreciate your replies so much. Again as I have some travel I have to do and some poking around on our server, give me a bit of time. Nothing personal, and thank you.

7

As long as you accept traffic from

whois -h whois.cymru.com " -f -v 78.138.127.110"

You will see that. They are a German provider often used by Palestinians, but they also use AWS Microsoft and DO and every other provider because they can.

Pinging won’t help because ICMP is not related in any way to VOIP and any packet loss should not be alarming as ICMP will always be preferentially dropped, even under light load.

However, more pertinently, not listening on UDP/5060 will be +99.9% effective though, and works also +99.9% for every other guy trying to steal calls from you.

I see your server is on DO, your name server with cloudflare, no problem there, now take the next step.

I as yet still can’t figure out why so many folks still don’t understand this, it’s as dumb as using 22 for ssh with root passwords allowed.

8

I am reasonably certain that the malicious traffic, while it should certainly be mitigated, is not responsible for the choppy voice experienced by the OP.

Though I agree that ICMP-based tests are highly imperfect and a bad result can be very misleading, a good result does indicate that the trouble is past the point tested, so the test is a lot better than nothing.

If the OP wants a better test, I suggest setting up a FreePBX instance at Vultr in Madrid, and configuring one line key on his Poly to register there, leaving the original line registering to NYC. He can then easily compare voice quality on the two systems.

9

That these incoming packets are numerous but unidirectional is just noise, the network impact would be negligible, as such consider it a red herring and move on.

Between two hosts, use iperf to test performance by protocol and/or port perhaps. it can report loss and latency and using -l2checks is often useful for identifying oopsies on the link where RTP traffic is sporadic.

Using TCP is often a better transport under such circumstances, it can’t fix bad networks but it can identify them more easily.

1 Like
10

I think you have a point here. I’m now in Lisbon Portugal, and the call quality is vastly improved.

We do have Opus configured, but I recently found out Flowroute doesn’t support it. If the compression would help, I wouldn’t mind trying another trunk provider. However, porting numbers I suspect will lead to serious downtime.

Lastly, how did you do this? If I can solve NYC and Ecuador, I’d feel a lot better.
I could also take the typical businessman shortcut and move to a server in Miami or Central America, with hopes to land in between. NYC will no longer be needed I suspect in the next year due to staff relocations.

11

Is this as simple as blocking them? Or (I suspect) much more complex.

Absolutely. If I change to a random port, will I need to help staff with hard/deskphones reconfigure? Bria is easy enough as it’s a simple app and I can guide them with screenshots.

12

Blocking them or even 78.138.126.0/23 is pretty well util, THEY won’t get through but their ilk will. I suggest you find out exactly how to configure your endpoints first, usually either a separate field or by using your.url.org:<random port chosen>

choosing an unused port above 20000 , and take the opportunity right now to use a domainname and not an IP address if you currently are. As you make changes , think about moving up to TLS , this is mostly safe on the default port 5061, at least I have never seen an attack against it on my IP (it should be obvious as to why not)

1 Like
13

Would most endpoints switch to the TLS port automatically? We have some staff in other continents, so changing those could be tricky. I can easily do it on my next visit to them though.

I’ll try blocking that in the meantime.

Thanks for your help.

14

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.