PSA: Bandwidth is experiencing outages


(Itzik) #1

Bandwidth is the upstream carrier for many providers, such as SIP Station, BulkVS, VoIPms, CIP and many more.

See status updates: https://status.bandwidth.com/


#2

(Not a big believer in co-incidents, but time will out . . .)

Coincidentally voip.ms now naming their ‘Upstream Provider’ after a week of no workee :wink:

https://twitter.com/voipms

(Time to seriously Caveate Emptor ?)


(Itzik) #3

And… Today again.


(Adolfo) #4

… today also.


(Greg Kujawa) #5

That’s what I was thinking! With so many smaller providers SoL I wondered if it was indeed an issue with a bigger fish “upstream.”


(Jeremy Lizzotte) #6

Been having call issues today since 9:12AM EST and Bandwidth.com is indeed the downed carrier unfortunately.


(Greg Kujawa) #7

Looking at all of the updates on their status page, it seems as if one service after another is lumped into the service disruption. There must be some scrambling engineers over there I’m sure.


(Luke C) #8

Makes you wonder if its more than just an “outage”


(Greg Kujawa) #9

<Vladimir exits the chat>


(Jake G) #10

voip.ms was hit with ddos attack a couple of days ago… I’d guess it’s related.


(Greg Snover) #11

You can’t make this stuff up! Here is what the Bandwidth web bot just said to me:

Yeah!


(Greg Kujawa) #12

And then there’s this. Just received the alert notification. VZW nationwide. I’m a ham radio operator, so hit me up at KE8KUJ. :joy:


(Greg Snover) #13

I wonder if it’s related - a different DDOS against Verizon - but only their 4G? My 5G is working here in Albuquerque.


(Greg Snover) #14

Around 2:00PM MST Outbound started working again - around 6:00PM MST Inbound came back and it has been up since.

Wow, that sucked!


(Greg Kujawa) #15

The most recent instances of significant downtime (at least for my providers) have been due to major fiber cuts, propagated BPG routing issues, and DDoS. I would love to be a fly on the wall to see what the hell happened over at BW this time.


(Greg Kujawa) #16

After almost 12 hours of downtime yesterday, apparently BW got dinged again…


(Andrew) #17

Vitelity was down last week because of “A loop between two edge routers in the Denver POP”. They literally plugged a router in wrong and created a network loop :man_facepalming:

Vitelity Status - Vitelity Outage Notification // Parent Case 00933822 // Multiple Issues


(Greg Kujawa) #18

Here was the response I got when I asked my SIP.US support folks about the impact of the BW issues.

Vincent Aukskalnis (SIP.US Support) 
Sep 28, 2021, 10:31 AM EDT 

Over the weekend one of the largest US underlying carriers (ULC) network was impacted by a 
Distributed Denial of Service (DDoS) cyber attack causing a disruption in services. This attack is 
similar to the attack on Canadian carrier VoIP.MS that started on September 16th and two UK 
providers earlier in the month. The US based ULC service is not down, but the flood of malicious 
traffic is blocking legitimate traffic into their network. SIP.US/SIPTRUNK.COM uses this ULC as the 
carrier of record from many of our telephone numbers.  We are in communication with the ULC, as 
our upstream carrier, but at the time of this email an estimated time of resolution is unknown. 
Services may remain intermittent until the attack can be mitigated. We are sorry for the pain this has 
caused you and your customers. We will provide more updates as they become available.

Thanks,
Vinny

(Tom Ray) #19

I just want to clarify something here. Many people over the last 24 hours have stated that “Bandwidth is down”. As a Bandwidth customer I can tell you that 100% Bandwidth was not down. Did I have issues with calls yesterday? Yes, I did. Was it every call? No, it was not. It was an intermittent service issue.

There is a different between what happened at VoIP.ms and what is happening at Bandwidth. In the case of the former, 90%+ of their network was taken out. With the latter, it was more a case of things being up and working but there were issues.

The biggest issue I saw was due to latency. Calls would be a bit choppy and when incoming calls didn’t complete or had one way audio is was due to this latency and Bandwidth not sending back ACK’s to the 200 OK responses to the incoming INVITE. This would allow the call to be answered, the callee be able to send audio but no incoming audio and then the call would disconnect due to no incoming RTP flow from Bandwidth.

This was not, however, every call and I would say I had about 85% success during the worst time of the incident yesterday.


(Greg Kujawa) #20

Good point. It certainly wasn’t/isn’t a true “hard down.” For us it was the opposite. Where roughly 75% of the calls were too choppy to stay on the line for. At least based on the user trouble reports. Since looking at the local call stats showed MoS being fine.