Anybody else experiencing issues:
Appears to be so --> https://www.isitdownrightnow.com/portal.vitelity.net.html
They are having another major service outage. They are calling it “Degraded performance” but all of our inbound through them right now is hard down.
Vitelity is not experiencing DDoS. They have had major internal infrastructure issues recently. I have a previously scheduled call with a regional VP later today and will provide updates here.
Is Vitelity part of the DDoS attack (or part of sip.us - since there’s a response here from sip.us)? It’s been fine all week and haven’t had issues until just today. Unless they just got targeted today?
Thanks, @adell4444. Figured it was another unscheduled outage and it’s getting old real quick.
It’s not DDoS. Just read their last 2 post mortems on status.vitelity.com
Really looking forward to hearing any updates you have in the near future after your meeting with the regional VP. We use Vitelity for about 90% of our end users - the other 10% use VoIP.ms. The last month or so has been extremely detrimental to our business with inbound/outbound call issues week to week with Vitelity and then VoIP.ms being down for close to a week after the DDoS attacks.
My operations manager wants me to look into other options for SIP trunk/voip services with all the issues we’ve had recently with Vitelity/VoIP.ms
Well your outbound I at least suggest routing elsewhere for now. For inbound, you’ll need to port those numbers out if you want them spread out over different carriers.
Call with RVP and Inteliquent COO is postponed until next week since they cannot pull a technical resource for the call due to today’s issues…
This is going to be a tough one in the US with STIR/SHAKEN. You end up giving your users a B attestion score which can cause some carriers (mobile for sure) to mark it unverified or even block it.
There’s gotta be a better way to handle it besides providing any backup outbound providers a list of your DIDs and settling for a B score, right? It seems like that hasn’t really been thought through.
Actually it was thought through very well. STIR/SHAKEN is an Operating Carrier level exchange. You, as a wholesaler, are not an Operating Carrier therefore you must go through one that will sign calls on your behalf. So that means you have to follow their rules and the rules are if the callerid you present to them isn’t theirs, they mark it as a B. You can get around this by becoming an Operating Carrier and then sign the calls as valid off your network.
Things are changing and VoIP now has all sorts of eyes on it due to IP becoming the primary transportation of things. During the Kari’s Law discovery it was found that VoIP providers were woefully negligent in having 911 locations even registered for their users. I know this is true, I’ve dealt with a few over the years and they all cite the same reason “Why pay for it when it is barely used? It’s a waste of money”. Wrong.
More along the lines of STIR/SHAKEN there was a point where VoIP providers, even registered as a CLEC, still could not be their own number authority and had to get DIDs from an existing tier 1 carrier. They loosened that up a few years back and there is now correlating data that shows allowing VoIP providers to be a number authority is in line with the uptick in the amount of SPAM/robo calls that are being generated now. To the point, the FCC may reconsider allowing VoIP providers to be number authorities. Which will suck and force us to user Tier 1’s for STIR/SHAKEN compliance.
I am convinced this is how they are going to get control of things. They are going to hammer down on the Tier 1 players to make it trickle down to their downstream’s who will also have to trickle it down to theirs if they want to keep calls flowing. There are too many low tier VoIP providers who just connect FreePBX (or other FOSS systems) to places like VoIP Innovations or Vitelity and “let carrier deal with things” in regards to telecom compliance. However, they present themselves to customers as a provider that can give them voice services and all sorts of features with the service.
I fully believe the time of people playing at being a provider while passing the buck to their upstream’s and their platforms is having the sun set on it. You want to be a voice provider, you have to start acting like a voice provider.
(and I’m not directing this at anyone. “You” is being used in a very broad sense.)
So the biggest concern here is carriers marking calls with a B attestation as unverified. Regardless of our thoughts of ITSPs and “voice” providers (I agree with your points on that), the bottom line is that your outbound traffic is at risk of being delegitimized unless you are willing to stick with a single provider for class A attestation, or sign the calls yourself.
Correct which is a way to ensure that providers are playing ball with the same rules. Being a provider shouldn’t be as easy as signing up for a platform in 5 minutes.
But if I have 5 terminating providers and they have rules about CPS and call limits, I can then spread my calls over them to avoid these limits. That is a valid thing to do but it also means that bad actors can do the same thing (which is what they do now). Most of the time the callerid being used is spoofed so no way it belongs to any of the terminating carriers or it belongs to another carrier not being used for termination.
Is this a perfect solution as is? No and I’m sure it will be improved upon. However, the more and more IP (VoIP) becomes prominent the more and more robo/spam calls are increasing. Part of that is due to being able to throw calls out a dozen networks at once.
The other problem is that mobile carriers have different rules to play by than landline/VoIP carriers. They are allowed to block both incoming and outgoing calls over their network without consent of the user. Landline/VoIP, however, has to terminate the call. We can’t block it because the destination is $0.10 to call but mobile carriers can.
So copper/SS7 based carriers don’t do STIR/SHAKEN while IP based has no choice. For the most part the B level score isn’t going to matter to landline/VoIP providers, they won’t really take action on it. They will leave that to the user or provide a service the user must opt for (even if auto optin is done). The SS7 carriers won’t care. The mobile carriers are the only ones that will be the concern.
I mean haven’t you also noticed that mobile carriers have been prepending a + in front of any number presented? This includes valid US numbers in 10 digit formats make a call from the 313 area code look like +3135551212 and will even display it as “Netherlands” for caller name. I had users complaining about that for months, I had to start presenting 11 digit callerid for them to get call backs from mobile phones because trying to call back from call history on the cell phone attempted a international call to +3135551212 instead of calling the US.
In regard to Vitelity, I had a call with them today which included the Director of their Voice Engineering and their NOC director. Basically:
- They have been working towards upgrading their infrastructure to be in line with Inteliquent, and on the same network as theirs. This is largely the cause, or indirectly related to the cause of their recent issues in the last few months.
- They have had to roll back their outbound platform migration, but are saying that they have roughly 20% of their customer base on the new platform running without issue. However, being on this new platform would not have avoided the service issues they had on 9/22 and 9/30-10/1. The issues they had on 9/14 were not experienced by anyone on the new platform.
- They are hoping to have their infrastructure upgraded by “Thanksgiving” is what they said. Worst case is Q1 2022. They did not inspire confidence when speaking about possible issues between now and then.
- The issue from 9/30-10/1 seems to be something different, but they were not able to tell me anything other than it being an “Exterior Network Event”. This in particular sounds different than the previous issues since they really could not provide me with any information, though I am only speculating.
I wouldn’t be surprised if they have more issues in the next few months related to their infrastructure. I hope this information is useful to anyone reading.
This is interesting. Toward the end of last month there seems to have been multiple players who were impacted by either BGP routing issues or DDoS attacks. Not saying this was the case with these guys, but at least a half dozen service providers I use sent out generic explanations as to what was going on around that timeframe. Look at the Facebook issues yesterday. They supposedly botched their BGP router configs and that results in a ~4 hour outage.
Other folks who have also apparently ‘botched’ recently
Best advice , “just don’t Botch ;-)”
We can go on all day speculating and going off-topic about facebook
I will post back here once a post-mortem becomes available.
With respect, the call you had seems to me so far to just reflect ‘their speculation’, no?