PSA: Bandwidth is experiencing outages

And the attacks resumed as the business day got going. I assume, because the attackers are smart enough not to waste resources when the U.S. is not on the phone it seems.

Wonder if they received a ransom demand like VoIP.MS did? If it was the initial 1 BTC demand then Iā€™d bet impacted customers could all willingly dollar up for that.

If bandwidth is struggling to fight these latest DDoS attacks, then I doubt that any provider can actually fight it. Wow.

Whatever it was/is, itā€™s a large enough outage to get customers upset.

2 Likes

Microsoft Teams uses BW as their ULC, so Iā€™m also getting notifications about this. Not to mention how that Microsoft 365 accounts that employ MFA are having sporadic issues getting the SMS notifications. This is definitely a beast thatā€™s rearing its head in terms of knock-on effects.

Sure. Just like back in the day when a truck took out a pole or someone dug up cables. People were upset that their service was impacted. Expected but large carriers having outages aint new.

Expected but large carriers having outages aint new.

True. But thinking back over my past 25+ years, I recall true major outages taking upwards of maybe one full day to recover. Back around 1997 in Florida there was a major fiber cut to a backbone link that affected pretty much the entire central part of the state. Repair teams fixed it by the following day. Then I recall issues with core DNS servers that resulted in lots of host connectivity issues for pretty much most of the US. Again, problem resolved by the following day.

While this DDoS exploit isnā€™t equivalent to a truly absolute outage, it is impactful. And itā€™s been taking place off-and-on since this past Saturday. Going on day three now. I canā€™t say I remember a major provider experiencing something prolonged as that very often. Maybe thatā€™s just my failing memory?

This really might not be a memory issue since what is happening is pretty new. The reports have shown an increase in DDoS/Ransomware attacks since 2020 and they are getting worse and worse. VoIP networks havenā€™t been a real target of DDoS attacks like this in the past but in the last four weeks weā€™ve seen it happen multiple times. Donā€™t expect this to go away, expect this to be a possible start of a whole new sector being attacked like this.

Agreed. Though much like companies in other tech spaces, these carriers and ITSPs will likely adopt better DDoS mitigation efforts quickly.

And side by side Voip.ms was in a far worse position this far into it. Bandwidth is fairing much better.

Bandwidth CEO;

1 Like

Here we go again:

https://status.bandwidth.com/incidents/krmlg4nd8vgn

I mean Verizon ā†’ Bandwidth having an issues really isnā€™t ā€œhere we go againā€. Carriers having issues switching calls between each other is quite common and could in no way be related to anything that has happened in the last 4 days.

Alternate Hot Take: This is a Verizon issue. Totally possible.

While I donā€™t want to overreact, there seems to be concerted efforts by some bad actor(s) to disrupt communications. VZWā€™s own 4G/LTE service was disrupted for over a full day. This should be a bigger news story, since communications are a key part of our infrastructure. And as has been pointed out, while this isnā€™t a hard down scenarios necessarily it could be a harbinger of things to come down the road.

1 Like

Well do you know of any providers or carriers that donā€™t use Bandwidth or Verizon? Iā€™m just askingā€¦for everyone else that will want to jump ship off Verizon now.

3 Likes

No argument here. Just passing along what is going on. Only time professionally Iā€™ve recently jumped ship is when one provider stopped supporting POTS, that we needed for UL insurance alarm line requirements, and a small VPS provider when our middleware services were totally down for a week. :grinning:

There really is no iron-clad solution when it comes to these recent scenarios. Other than patience and understanding. Of course you could build something like this and settle for old 2G cell phones. But still needing PSTN/Internet accessā€¦lol.

This one is really becoming a pet peeve of mine. POTS = Plain Ole Telephone Service. It was how to designate the type of service being delivered in a time where everything was copper. Example ā€œWhat type of service does the customer have?ā€ Possible answers: POTS, ISDN, PRI, T1. It denoted the level of service and even interconnect handoff.

POTS is a level of service so back in the start of VoIP, as the VoIP provider you pushed that you had POTS/traditional features on your service. Letā€™s not forget too, POTS basically means you have a simple connection to the PSTN. Features like Call Waiting, Voicemail, 3-Way Calling, CallerID Name (even Call Waiting with Caller Name) have always been add-on services that eventually got packaged into things.

Copper is being discontinued, all the major ILECs are retiring it. There is going to be a point where you cannot get these services (and in some markets you canā€™t now) so people need to adjust to this.

There are numerous carriers/providers that provide POTS services but it is being delivered over VoIP because POTS just means ā€œconnection to PSTN with featuresā€.

1 Like

Agreed on all counts. The copper part is really where it hits our 66-block on prem. Further upstream itā€™s VoIP regardless. Our sites each have vaults with $15MM+ of jewelry and watches stored in there, so our insurance company requires us to have three layers of alarm coverage (Internet, copper, and cellular radio). Currently we are paying AT&T around $60/line for copper at each site for this expressed purpose. Which still makes me sick.

When AT&T finally totally drops that last hop to copper at new sites then I can just use an Adtran to break out of the FreePBX so itā€™s copper where the inspectors can see at least.

1 Like

Yes, basic copper based lines have gone from being $10/month to $60/month. When something like that goes back to its original market cost (high) it generally means they are trying to get rid of it and make it unappealing.

Mitel does that with their old FXS/PRI cards and they want to push more IP based systems. A new PRI card for your Mitel can be almost $5K before the installerā€™s time is billed for putting it in.

Well, now itā€™s outbound as well

https://status.bandwidth.com/incidents/d8vft1kpxdmk