PSA: Bandwidth is experiencing outages

…as well as interconnects besides just VZW. I received the Office 365 admin notifications regarding Microsoft Teams PSTN calls as well, since presumably they are using BW. Our experience here is spotty but the majority of calls are okay. I’d say maybe 10% of them aren’t.

This e-mail just came in. From SIP.US, my SIP service provider who uses BW as their ULC. Sounds like even the resellers are porting away from BW due to the length of this disruption.

SIP.US Logo
Hi Greg,

First, we apologize for any service impact you may be experiencing. As you may be aware, one of the US’s largest underlying carriers (ULC) has been under a Distributed Denial-of-Service (DDOS) attack this week. This attack is similar but larger in scope than the attack on VoIP.MS that started on September 16th. Since this is a ULC we can easily route around them on outbound calls, but this is not the case for inbound calls. The ULCs serve as the “carrier of record” for the DIDs or telephone numbers used by our customers. The current design of the US telephone network means that all calls destined to a particular telephone number have to be handled by the carrier of record. The only way to change this is to port or move the number to a new carrier of record and for landlines, this process can take 5 to 7 days under normal circumstances.

SIP.US is investigating all options to address intermittent inbound call issues caused by the DDOS on the ULC, but please know this is affecting large and small customers the likes of which including Microsoft Teams, Google voice, RingCentral, etc… to just name a few

We have seen significant improvement on the ULC that is being attacked, and we have confidence they will get their network restored to 100% capacity as quickly as possible.

However, with that said SIP.US has been working with other ULCs to arrange porting of DIDs to alternative ULCs. As you can imagine there are a number of carriers and customers attempting mass ports and this is causing congestion in the porting process. We will continue to work to have inbound calls delivered via alternative ULCs.

If you are experiencing service issues, please contact support at [email protected]. This way our team will know you have been impacted and can communicate directly back to you once updates are available.

Regards,

TBH, for some reason, bandwidth did not release yet the size of the attack. There is no way anyone can prepare/simulate for this without knowing the attackers’ capabilities. So porting away from BW to a carrier that has no detailed information of what’s happening is kind of ironic.

2 Likes

On Wednesday, the Federal Bureau of Investigation said it was aware of the cyber attack against Bandwidth.

“The FBI is aware of reports of a DDoS attack involving Bandwidth,” Shelley Lynch, a spokesperson for the FBI’s Charlotte Division, said in an email. “We are monitoring the situation and have been in contact with the company.”

Source

1 Like

I am curious which ULC they will go with. Verizon? They had issues the last couple days too. CenturyLink/Level 3? Not only do their outages impact their own users they have broken the Internet in 2020. Perhaps ATT? They are know to be outage free all the time.

During this issue Ive had one day where calls were problematic. Some API stuff was down but calls worked. I had customers complain but not threats of people jumping ship if the intermittent outage wasn’t solved.

Oh while I was typing this a customer called about phone issues. They have Straight Talk, which uses Verizon (with others) they have been having spotty cell service today.

Seems like bandwidth went from “monitoring” to “major outage”. @BlazeStudios, do you see any issues on your network now?

As I write this, my DIDs (Bandwidth via Anveo Direct) are working fine.
Incoming SMS also OK.

1 Like

It is after hours on the East Coast so most of the disruptions will have slowed or stopped. Whoever is financing the botnet is only paying for it to run during East Coast work hours for the most part.

4 Likes

A couple complaints but nothing consistent. Issues are intermittent.

2 Likes

In my observations over the years, yes they might currently seem below the trench line, and that makes sense. But they are opportunistic, so if the night-watch are snoozing at 3AM on Sunday . . .

(pretty sure these guys are not short on ‘sponsorship’ )

As of this afternoon Bandwidth is now routing through Cloudflare Magic Transit for DDOS mitigation as you can see here. https://bgp.tools/prefix/67.231.12.0/24 and https://bgp.tools/prefix/216.82.224.0/24

This is the same service voip.ms used to mitigate the DDOS. Prior to that it would appear they were using Arbor DDOS Mitigation service through the NTT Fiber but it did not seem to be helping.

1 Like

Is that Eastern or Pacific?

There are ‘easts’ in both europe and far, which one do you put your money on?

Looks like about 5-6 hours ago. https://stat.ripe.net/ui2013/67.231.12.0#tabId=routing

Just a minute ago, they were showing MAJOR OUTAGE in red on all the Voice Services - now it shows DEGRADED PERFORMANCE in yellow.

Are they actually getting it fixed?

Man, I would hate to be working there…

Thanks, I was more just making a vague reference to End of Days in which the priest said hell on earth would be unleashed at midnight and the response was basically “Eastern or Pacific” (or a rough variation) pointing out that one would give them an extra three hours to prepare.

1 Like

Success, or just knocking off for the night?

At least Cloudflare is getting calls during these DDoS attacks. :grin:

I wonder who their VoIP provider is??

(post withdrawn by author, will be automatically deleted in 24 hours unless flagged)

Hey @ITconsultant, I suggest you please remove this post, as a lot of people in the VoIP community speculate that these attackers have chosen their targets based off of online forums.

There’s no need to advertise who everyone’s provider is…