VOIP.MS DDOS 7 Day outage dump them? is this normal in the industry


(Lorne Gaetz) #62

Thread locked after multiple flagged posts. Feel free to flag this post or PM me if you disagree.

edit - Thread unlocked.


(Lorne Gaetz) opened #63

(Canada) #64

@BlazeStudios, Absolutely correct. Even if you’re dropping traffic on an edge device interface(s), the goal is to throw so much traffic at the security engine, whether that rule be geo denial, ip denial or snort/layer 7 behaviour rules that the engine and its resources such as memory or cpu on that security device or cluster security device gets overwhelmed slowing down the processing of regular accepted traffic as well on the interface(s) because it’s working so hard to drop that denied traffic based on the defined rules. Eventually resource needed outweighs actual system resources. And services and traffic grind to a halt, as the edge device struggles to do its job under the weight of the attack. They’re consecutively hammering ALL ports not caring what is there, the goal is to create as much disruption and traffic as possible. The port or service doesn’t matter at that point, they’re aiming to overwhelm the system and its function effecting all servers and systems behind the security wall. They want to choke off the pipe(s).


(Ted Mittelstaedt) #65

I’ll chime in on this as well - Tom is perfectly correct. It makes zero difference whether you listen on 5060 or something else other than using something other than 5060 breaks a lot of older stuff like phones and router code.

There’s only 2 ways to protect against DDoS. First is to be like me - so small with services (webserving mainly) that are so common that there’s no money there to get - an attacker can take me offline if they want but I don’t derive enough income from those services to -ever- justify paying a ransom to get back online. An attacker can see my allocations and know this immediately and they won’t waste time since there’s nothing there to get. Unless that is they are one of those stupid criminals like the stupid criminal that buys $500 worth of oxy-acetylene cutting torches to break into a garbage dumpster or a porta-pottie. I have the equivalent of quarter inch thick steel locks on my little stuff so they would need it which is why the Dumpsters are also made out of the same such steel - it won’t deter a criminal with the right gear to break into one but the right gear is more expensive than paying the tipping fee to dump your trash properly.

The second method is to be so big and rich and wealthy that you can throw unlimited amounts of bandwidth at the problem thus reducing the attacker to the equivalent of a buzzing mosquito annoyance.

This is a characteristic of a mature market, folks. VoIP is no longer a hobby market you either go big or stay home. There is no more room for someone wanting to grow from small tiny low-value company with a handful of customers to a midsize company with has a fair number of customers. This has happened with software which is why Apple’s MacOS is permanently a small low value product compared to Windows and why nothing else has come along the same as Windows. It’s why there was no room for 7-up or Royal Crown cola just Coke and Pepsi. And so on.

voipms’s choice is pretty clear, if they want to stick around, they need to raise prices to afford the unlimited bandwidth and go big. Or they need to do nothing and allow customer after customer to stop using them and then shrink back down to just a “have server will travel” going concern. There’s no more room in this business for a midlevel concern that tries to get the best of a small operator and the best of a larger operator.


#66

Intermittent outages today some pops offline


(United States) #67

Yep, at the beginning of pandemic WFH in 2020, I did open up UDP and TCP port 5060 along with TLS on 5061 (actually I spun up another Asterisk server to handle those, didn’t open up main PBX to Internet and had trunks between that one and main PBX) to accommodate phone stations at people’s homes and softphones on mobile devices (softphones on laptops, etc already were over VPN). UDP:5060 was getting hammered the first day (but service was still working). TCP:5060 was less attacked (we setup most WFH to use TCP for NAT friendlyness), but TLS:5061 was nice and quiet and malicious stuff that did try to connect didn’t even get to auth phase, so less resources were sucked away. Quickly rolled out configs for all TLS connections, so shutdown the others…
I have found that the OpenVPN feature of Yealink phones works very well, so were now slowly replacing the 15 year old Polycom 650s sent with people for WFH (that we were going to e-waste anyway, luckily we had not done that yet at start of pandemic) with yealink (mostly T53W and T54Ws so they support wifi) connecting via OpenVPN.
Side note, a vast majority of WFH workers preferred a IP phone at home over any softphone or mobile solution.


#68

It’s not that listening on 5060 is the killer, it is that listening on any “well known sip port” (which would include in my experience 5000-5999 and mostly UDP, but there are a few outliers (a conntrack port flood rule for these ) will enable these guys (who are obviously cleverer than the guys they exploit) to ‘fingerprint’ you as maybe a "FreePBX user

https://wiki.freepbx.org/display/PPS/Ports+used+on+your+PBX would be a fingerprint for FReePBX
https://freeswitch.org/confluence/display/FREESWITCH/Firewall for FreeSwotch

The historical list of penetrations against such targets is hard to deny and none are based on primarily SIP penetrations, just the second derivative of http perhaps sloppy 5038 or 3306 :wink:

You won’t see the attacks immediately but soon after, often other services you might have open will be probed first , and not by the original Ip, (as I say , they are undeniably cleverer than the victims they penetrate) but by the C&C infrastructure that is similar to the one that brings down unprotected services like DNS voip.ms was using a few days ago.

If you have a well deployed DDoS protection in place for your DNS, then you are probably ahead of voip.ms game. If you are not then please think about this. if on the other hand you think you are ‘fine’ then maybe you are.

This is my considered opinion, and one I have formed after watching this for a dozen years or more. (and loosing lots of money by not knowing yet )


#69

Yep, not hard to do, demonstrably highly effective and only pisses of a few ‘deniers’ if you do it.
Call it a pragmatic remedy maybe :slight_smile:


#70

Please tell us which of your phones/routers don’t support SIP connections over other than UDP:5060 ?


(Ted Mittelstaedt) #71

I’ve posted about this before but here goes again

Originally I exposed 5060 to the Internet and I use a Cisco 2811 running IOS 12.4 (and I mean the REAL IOS not Apples despicable theft of the name)

Cisco IOS has this “thing” in it’s NAT code where if you port forward udp5060 from the outside to the inside PBX you don’t have to then port forward all of the higher UDP sip ports. The router sees the incoming SIP from the extensions on the Internet and will dynamically open port forwards for the higher udp ports in the upper range for the voip connection.

It’s really slick and before the script kiddies ruined exposed 5060 for people who don’t want to put an OC3 on it, it allowed me to plug in a physical phoneset anywhere on any remote customer’s network that was statically configured to login to my PBX. I tried it with several but the Polycom vx series was my preferred portable extension.

When my logs started overflowing with trash from the kiddies I attempted to move it from 5060 to something else. That broke the ‘automagic’ port forward for the higher ports. So there is that incompatibility. Although today I don’t use it since it’s hopeless to expose that port to the Internet anymore.

And then there’s the other issue which is I don’t buy SIP trunks. I use bona-fied POTS trunks that feed into a SIP gateway. I happen to use another Cisco router for that gateway purpose. Anyway that does not work with pjsip. It works fine with chansip. So I have my FreePBX system configured with chansip listing on 5160 and the Cisco gateway using that while pjsip listens on 5060.

Originally I tried it the other way - chansip on 5060 pjsip on 5160 - but had some phones not work entirely correctly. It wasn’t the Polycoms, those had no problem. I’m pretty sure the Cisco 7940’s didn’t like it. For a while I was testing a whole range of different makes and models of phones as well as single pots to sip gateways (used to make a pushbutton POTS phone work as a sip phone) and some of those had weird issues. I still have a box of those phones in the basement.

Note that getting SIP working over VPNs has not been a picnic either. I’ve posted about that. Today my configuration is stable solid and works but interestingly, the Cisco 7940’s going over the VPN work fine with UDP5060 while I had to configure the Polycom’s to use TCP5060 because they would drop registration over the VPN using UDP5060. This is an OpenVPN vpn.


#72

I have ported out numbers to Telnyx in under 1 day, they are a very large tear one carrier licensed in 25 countries, with 24/7 live chat and phone support for free, I asked them if they have DDoS protection on their VoIP infrastructure the CEO David Casem came up on the chat and told me they have a robust Network scrubbing infrastructure against DDoS and they have recently stress-tested it and they have faced DDOS attacks and no impact on their users, they also have their own private fiber backbone not relying on the public internet reducing latency and better voice quality and a great API, for sip trunks, but I’m also going to set up flowroute as a backup and have all the companies toll free DIDs by telnyx and the local numbers by flowroute so there is inbound redundancy because even the biggest carriers are not immuned to an outage, and it’s especially important to have inbound redundancy because you’re totally in the hands of the carrier


(Itzik) #73

IIRC, they are using SubSpace, but I may be wrong.


#75

I don’t understand why this keeps coming up.

If you register with your service provider, you shouldn’t need to open or forward any ports. Any Iptables based firewall with related/established enabled will automatically accept replies on Port 5060 when you register from that port. And nearly every consumer and most commercial routers (iptables based, for example), will do the same.

But, even if you don’t register and you do need to open that port up, you should only open it from IP addresses associated with your provider. For nearly every ITSP, there are only a small number of IP addresses that signaling (Port 5060 traffic) will come from. Any good provider will give you a list of addresses to whitelist should that be necessary (and again - I’ve never found it necessary).

Similiarly, media ports (10k-20k) never need to be opened. Related/established rules on your firewall (and your router) will open them when you initiate a call.

If you need to accept connections from anywhere, such as if you have remote users who can be anywhere on the planet, they should connect to your server using a VPN, and you should only open Port 5060 to those that come in over the VPN connection.


#76

To answer the very first question that prompted this thread:

This is not normal in the industry, but it does happen.

Back in 2012, Callcentric experienced a DDOS attack that kept them down for about two weeks. Unlike VOIP.ms, Callcentric uses one domain name for all registrations, and so there was no way to work around the problem by choosing a server that was unaffected. Everyone just had to wait until CC got it under control before phone service could be restored.

VOIP.ms’s decentralized architecture, with many different servers to choose from, actually makes things a bit better than the CC attack. There are voip.ms servers that have worked fine during the entirety of the DDOS attack. As a result, it was relatively easy to work around this DDOS attack and keep the phones working. That wasn’t an option during the CC attack.


(David55) #77

I believe some providers have started using encryption, and this will only work when the SDP isn’t encrypted.

Also a lot of people report problems with application level gateways on routers, and the general advice is to turn them off and handle everything by lower level configuration.


#78

Restricting port forwarding to an IP range does not require DPI it is just on the IP layer I’ve been doing that forever never had a SIP issue, if your router makes problems use pfSense or just use the FreePBX firewall which works really good with restricting IPs and I don’t use the responsive feature


#80

A smart person learns from his mistakes, but a truly wise person learns from the mistakes of others, that was 2012 why didn’t they have any DDoS protection infrastructure on the POPs the tier one SIP providers have DDoS protection not only on their website but also on their POPs, that’s why I’ll never look at any of the smaller providers, and the 7-Day outage doesn’t happen with large tech companies with all the emphasis on redundancy and disaster recovery even for the utility company after power lines get knocked down it doesn’t take them that long


#81

I agree 100%.


#82

I use encrypted Trunks and do not forward any ports, either on my router or on IPTables.

My comments about port forwarding are unrelated to SIP ALG, which I agree should be turned off.


#83

The safer way is to not forward ports from your router at all and to configure IPTables on your FreePBX machine yourself using standard rules that block all incoming traffic unless it is related or established or specifically allowed local devices. Here’s a sample:

*filter
:INPUT DROP [0:0]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [34:2985]

# Allow Localhost
-A INPUT -i lo -j ACCEPT

# Allow related and established
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT

# Allow Ping
-A INPUT -p icmp --icmp-type echo-request -j ACCEPT

# Allow everything from LAN - Currently disabled in favor of specific rules below
#-A INPUT -s 192.168.1.0/24 -j ACCEPT

# Allow SSH/SCP from LAN
-A INPUT -s 192.168.1.0/24 -p tcp --dport 22 -j ACCEPT

# Allow Local Phones
-A INPUT -s 192.168.1.0/24 -p udp --dport 5060 -j ACCEPT
-A INPUT -s 192.168.1.0/24 -p udp --dport 10000:20000 -j ACCEPT

# Allow local web-access
-A INPUT -s 192.168.1.1/24 -p tcp --dport 80 -j ACCEPT

# Allow local TFTP Access
-A INPUT -s 192.168.1.0/24 -p udp --dport 69 -j ACCEPT

COMMIT

Note that an even safer way is to allow local access specifically to each individual IP address. Regardless, the above configuration will block any connections unless the PBX first reaches out to the destination IP address (such as by a registration or qualify packet, or initiating a call). Use of qualify packets can keep the connection open.