PSA: Bandwidth is experiencing outages

Bandwidth is the upstream carrier for many providers, such as SIP Station, BulkVS, VoIPms, CIP and many more.

See status updates: https://status.bandwidth.com/

1 Like

(Not a big believer in co-incidents, but time will out . . .)

Coincidentally voip.ms now naming their ā€˜Upstream Providerā€™ after a week of no workee :wink:

https://twitter.com/voipms

(Time to seriously Caveate Emptor ?)

3 Likes

Andā€¦ Today again.

ā€¦ today also.

Thatā€™s what I was thinking! With so many smaller providers SoL I wondered if it was indeed an issue with a bigger fish ā€œupstream.ā€

Been having call issues today since 9:12AM EST and Bandwidth.com is indeed the downed carrier unfortunately.

Looking at all of the updates on their status page, it seems as if one service after another is lumped into the service disruption. There must be some scrambling engineers over there Iā€™m sure.

Makes you wonder if its more than just an ā€œoutageā€

<Vladimir exits the chat>

voip.ms was hit with ddos attack a couple of days agoā€¦ Iā€™d guess itā€™s related.

You canā€™t make this stuff up! Here is what the Bandwidth web bot just said to me:

Yeah!

4 Likes

And then thereā€™s this. Just received the alert notification. VZW nationwide. Iā€™m a ham radio operator, so hit me up at KE8KUJ. :joy:

I wonder if itā€™s related - a different DDOS against Verizon - but only their 4G? My 5G is working here in Albuquerque.

Around 2:00PM MST Outbound started working again - around 6:00PM MST Inbound came back and it has been up since.

Wow, that sucked!

The most recent instances of significant downtime (at least for my providers) have been due to major fiber cuts, propagated BPG routing issues, and DDoS. I would love to be a fly on the wall to see what the hell happened over at BW this time.

After almost 12 hours of downtime yesterday, apparently BW got dinged againā€¦

Vitelity was down last week because of ā€œA loop between two edge routers in the Denver POPā€. They literally plugged a router in wrong and created a network loop :man_facepalming:

Vitelity Status - Vitelity Outage Notification // Parent Case 00933822 // Multiple Issues

Here was the response I got when I asked my SIP.US support folks about the impact of the BW issues.

Vincent Aukskalnis (SIP.US Support) 
Sep 28, 2021, 10:31 AM EDT 

Over the weekend one of the largest US underlying carriers (ULC) network was impacted by a 
Distributed Denial of Service (DDoS) cyber attack causing a disruption in services. This attack is 
similar to the attack on Canadian carrier VoIP.MS that started on September 16th and two UK 
providers earlier in the month. The US based ULC service is not down, but the flood of malicious 
traffic is blocking legitimate traffic into their network. SIP.US/SIPTRUNK.COM uses this ULC as the 
carrier of record from many of our telephone numbers.  We are in communication with the ULC, as 
our upstream carrier, but at the time of this email an estimated time of resolution is unknown. 
Services may remain intermittent until the attack can be mitigated. We are sorry for the pain this has 
caused you and your customers. We will provide more updates as they become available.

Thanks,
Vinny

I just want to clarify something here. Many people over the last 24 hours have stated that ā€œBandwidth is downā€. As a Bandwidth customer I can tell you that 100% Bandwidth was not down. Did I have issues with calls yesterday? Yes, I did. Was it every call? No, it was not. It was an intermittent service issue.

There is a different between what happened at VoIP.ms and what is happening at Bandwidth. In the case of the former, 90%+ of their network was taken out. With the latter, it was more a case of things being up and working but there were issues.

The biggest issue I saw was due to latency. Calls would be a bit choppy and when incoming calls didnā€™t complete or had one way audio is was due to this latency and Bandwidth not sending back ACKā€™s to the 200 OK responses to the incoming INVITE. This would allow the call to be answered, the callee be able to send audio but no incoming audio and then the call would disconnect due to no incoming RTP flow from Bandwidth.

This was not, however, every call and I would say I had about 85% success during the worst time of the incident yesterday.

3 Likes

Good point. It certainly wasnā€™t/isnā€™t a true ā€œhard down.ā€ For us it was the opposite. Where roughly 75% of the calls were too choppy to stay on the line for. At least based on the user trouble reports. Since looking at the local call stats showed MoS being fine.