Hello. We upgraded our old phone system last year to FreePBX. The system runs good for a couple days and then incoming and outgoing calls start to get choppy. I’ve contacted xfinity, who has our internet service, and they see a lot of dropped packets from the modem. Our current service is 100mbps. When I restart the modem, the issue goes away for a couple days and then progressively gets worse. I’ve recently had the modem replaced and am still experiencing the same issues. The phones are on their own vlan. I’m thinking that there may be some network configuration issue. We are using Meraki devices on the network and have 20 phones at our main office. I’ve been in contact with the installer of the system and Xfinity. Xfinity has be restart the modem and the VOIP installer claims its an Xfinity issue. I’m thinking that it’s a network configuration issue. I’m new to this service. I would appreciate any help. Thank you.
What services does the modem provide? DHCP? Routing? If you have a typical modem that ONLY transfers data from/to the provider, then I find it difficult to follow your leap that anything other than the modem is at fault here.
The modem provides DHCP routing. It’s basically a router/modem. I’ve put the gateway into bridge mode and that shut everything down. We have 13 static IP addresses attached to our Xfinity service. I had someone tell me that putting the gateway in bridge mode would cause issues with the static IP addresses. Not sure if that’s true or not. Also, I’ve connected an ATA device on our network to connect a fax machine. I keep getting “poor line quality” and failed faxes. Not sure if that’s connected or not.
Contract a consultant or someone that knows what they are doing. You have basic network design issues to resolve here.
Thanks.
There are several options here.
My first choice would be to troubleshoot bridge mode. Connect a computer directly to the modem and determine the static or DHCP configuration required. Use a machine with nothing important on it, as you will be directly on a public IP with no firewall.
Next, if the modem’s UI has an option to reboot it, rig a script to programmatically restart the modem every day at e.g. 3 AM. If you can’t do it from the UI, consider a device such as https://www.amazon.com/Ubiquiti-Networks-MPOWER-MINI-1-Port-Power/dp/B00HXT8IGO/ . This can also be set up to trigger a reboot if the internet connection goes down.
Also, look at the modem stats and see if the signal levels, SNR and error counts are reasonable.
If all else fails, do some real troubleshooting and find out what is going wrong.
They have 13 IP addresses. What are you going to do about all the (assumed) services you break by doing something like this?
Does that also include still having dropped packets?
Yes. Packets are still dropping.
I’m going to get in touch with the company that installed the VOIP system. They have a separate gateway attached to the Comcast router. So, there are two devices connected to the Comcast router.
I connected a laptop directly to the Comcast router. I’m getting an IP address supplied by the router. Then our Meraki MX device is also providing an address. IMO, I would think that we are running a double NAT. I’d like to visit one of our remote sites and bridge the gateway on that Comcast device and see what happens. I’ll let you know what I find.
What is the brand, make model of the modem, Comcast has 3 of them they use, this is important.
Xfinity is the home service, you have Comcast Business.
The modem should NOT be running anything other than acting as a router. It cannot be a bridge. NAT should be off. You should have a real router behind the Comcast device that acts as a NAT device. You should not be using any of the 10.x numbers from the comcast modem.
Poor signal quality on the copper line can sometimes cause this. But as I said without the make/model/type of the modem, we have nothing.
99.9999% chance this is the Comcast service. By accusing the installer of a configuration issue you are just creating an enemy of who your strongest supporter would be and almost guaranteeing this will never work properly. The installer knows more about this than you do yet you claim you know more than he does when you say that it must be a problem with the phones. Uh huh. And I’ll bet that on top of this you were the ones who insisted on using Comcast.
And are they dropping regardless of what router/edge device being used to test from? As you pointed out the voice system has its own gateway attached to the Comcast device and then you have yours for the data.
If you are seeing packet loss no matter which gateway you test through then you are probably looking at an issue with the Comcast connection itself. The only real way to test this (and this is exactly what a tech would do onsite from Comcast) is they would bypass the entire local network and connect directly to the modem to do tests. If the packet loss still exists, it is them. If the packet loss no longer exists then it’s not them and something on the network.
This is nothing more but inference. You are inferring the OP is accusing the installer when all the OP said was they would get in touch with the VoIP installer. Which is actually the right move to do since the installer also installed the voice gateway/router that sits in front of it and connects to the Comcast modem. So if there are connectivity issues over that network, that is 100% on them to support it.
Tom did you even read the OP:
He didn’t “all the OP said was they would get in touch with the VoIP installer” Oh sure, they got in touch with the installer - by telling the installer it must be their gear that’s broken. The OP even says that this is what they did. Re-read the post.
“He’s thinking” yet he says he has no experience. So what “he’s thinking” means nothing - by definition, by his own admission. It’s like some dude saying “my car drives real funny, I don’t know anything about cars, but it’s the engine”
If the installer selected Comcast or the installer agreed to use an existing Comcast setup, then he should be telling the installer “you chose the connection, you fix it, here’s our Comcast account number and Comcast’s 800 number, have fun” He shouldn’t be in the middle of it particularly after admitting he doesn’t know the tech.
You are correct that this is 100% the installer to support. As you say - 100%. Meaning ALL OF IT. Meaning 0% of the OP needs to be involved.
I’ve seen this kind of thing before. People like the OP get involved because they have a stake. The OP may have insisted Comcast be used, the OP may have insisted other components of the network be left undisturbed. They may have hamstrung the installer and browbeat them into using an existing setup the installer didn’t want to use. And now it’s blown up. So the OP is going to be darn sure the installer gets blamed because they don’t want it coming out that the installer recommended against using Comcast and they overrode the recommendation.
Lastly, it’s clear you aren’t familiar with Comcast’s copper service. For a static IP subnet to work Comcast REQUIRES YOU to use THIER cable modem and it MUST BE CONFIGURED to be in routing mode. This isn’t like a T1 handoff or fiber handoff where the customer can use whatever they like. This isn’t a dynamically assigned IP setup where you can use ANY docsis 3 cable modem in bridged mode and your own gear behind it. He CANNOT “substitute any router/edge device to test from” as you suggest.
As sorvani said - contact someone that knows what they are doing. If the VoIP installer said “Comcast will work fine” then this screwup is on him and the OP can justifiably sit on the installer until the installer screams. But I’m betting there was some hesitancy from the installer, or the installer had no experience with Comcast, or there is some other reason the OP feels compelled to get in the middle of this mess and defend Comcast.
I asked earlier if there was packet loss on the Internet connection. There is. There is the Comcast modem and two routers (or at least one for the voice). My questions now are, when doing tests over the data network is there packet loss? When doing a test over the voice network, is there packet loss? If the answer is yes to both, then it is most likely a Comcast issue. You can confirm that by passing the entire network and testing directly on Comcast.
However, if there is no packet loss over the data network and only over the voice network then there is a configuration issue somewhere in that part of the network that is causing this issue. So until the actual cause and source of the issue is discovered then all parties must be involved. Someone from the data side and the voice side as these are managed by two different parties. There needs to be some working together to determine where the issue is. Obviously, I’m in agreement with not throwing blame around until troubleshooting is done.
Actually, I am. I would say I have less than 6 locations where I have to put the Comcast router in DMZ mode because the location is small and is using the router portion for their data side and I need my edge device to have a static IP and not have a firewall blocking anything. That’s generally where they either have dynamic IPs or a single static IP.
In cases where there are /29 or more being used, the Comcast modem ends up in bridge mode and my router is where all Layer3 is handled. Generally Comcast just gives you a straight subnet and you can set any of the usable IPs as the WAN IP. In some cases they may have the WAN on a /30 and then route the “Public LAN” of /29 or /28 IPs over that /30. Either way, the modem can be bridged and your own router can deal with the routing and assignment of those IPs on your network.
OK then Comcast is doing different things in different areas. In the area I’m in, the Comcast cable modem is required because the modem sources a routing protocol that is sent towards Comcast that advertises the subnet. Static IPs are /30s and work the same way - you cannot remove the Comcast router. The cable modem CANNOT be configured as a bridge because it would then not make this advertisement. Unfortunately the OP does not appear to know enough to know if he is in one of these areas like mine or one of the areas like yours where you can replace the Comcast cable modem routing functionality with your own router (which, obviously, is the very best case since the Comcast-supplied cable modems are garbage)
Now, note in areas like mine that the Comcast routers user-interface has a “bridge” button that is for the dumb users who don’t understand networking. This button merely turns off the iptables inside the modem. The modem is still acting as a router and still is handing you off the IP addressing and is still routing between your static IP subnet and the rest of Comcast’s network.
Only in areas that you could replace the comcast modem with your own docsis 3 cable modem that is bridged can you configure your own router to handle the static subnet. That is the litmus test here. If you cannot do this then your Comcast devices ARE NOT in bridged mode no matter what their interface says and no matter what you think. Your response has a bit of hand-waving (generally?!?) so I’m not 100% sure you are right but knowing Comcast I would not put it past them to have different networking in different areas. Your response doesn’t quite jive with what’s on Comcast’s own forums (where this issue has been discussed) but no matter - you and I are speculating because the OP has not supplied the router model or what type of service he’s on.
Now as for the packet loss thing. For starters Comcast often says “there’s packet loss” when there really isn’t any at all. It’s like one of their things their first level support people say to get the complainer to go away. Secondly, packet loss on VoIP works a bit differently. Most counters that people have in networking gear that “measure packet loss” do it by looking at TCP retransmits. If there’s a lot of TCP retransmits then they assume there’s a lot of packet loss. But VoIP is heavily dependent on UDP so by the time you notice “packet loss” in TCP the UDP has completely gone to hell in a handbasket.
I don’t see how Comcast can actually measure or see “packet loss” because they don’t have control over EITHER end. All they can do is look at the cable modem statistics and if they see a lot of noise on the line, or a terrible signal to noise ratio, then they can infer there’s packet loss because they know that the noise or SNR is trashing most of the packets. At the layer 1 level on a Cable line there are no packets. There is just a stream of 1’s and 0’s. bad SNR will trash a lot of those 1’s and 0’s and flip them to 0’s and 1’s thus any higher layer that relies on them will get trashed. If that is happening, say 5% of the time then most smaller UDP packets will make it through unscathed so a voice signal may sound OK and the TCP setup and signaling will just retransmit and you won’t notice anything. If it’s happening 80% of the time then likely the call won’t even setup at all. None of this sounds like the OP’s original “progressively gets worse” explanation. And rebooting the cable modem isn’t going to fix bad SNR or line errors so once more, I don’t trust that packet loss explanation.
Another place where there’s often problems that get blamed on “packet loss” is trouble inside the router itself. For example take the case of a NAT device that is not expiring connection table entries rapidly enough for the amount of ram that it has. As time passes the ram inside the router gets used up, and when it runs out it’s no longer able to create translation entries, then packets going through it start getting dropped randomly. So, you reboot the device - measure “packet loss” right after rebooting - think everything is fine - then 3 days later you measure “packet loss” see tons of it, and conclude the line is the problem when the reality is the router has run out of scratch ram. You reboot the router, packet loss goes away, and instead of blaming the device you think the magic fairies on the line saw the router rebooting and immediately fixed the impedance mismatch or area in the line that a squirrel ate through, or whatever else, which just confirms your erroneous idea that it’s the line.
I’ve seen this more times than you would think. You very likely had the same issue before you got your VOIP system, but didn’t notice it because dropped packets don’t really impact ordinary internet usage the way that they impact VOIP traffic. If you’re just surfing the web or watching a youtube video, the packets will just retry until they get through. You might not notice the delay at all. With VOIP, however, a dropped packet necessarily means lost audio.
This problem is often caused by RF interference at your router/modem. Look very carefully in the area near your router and your modem for any device that might be transmitting on a radio frequency band. This includes wifi access points, cellular extenders, mobile telephones, cordless phone base stations, cordless phone handsets, baby monitors, etc. If there are any such devices within 10 feet of your router or your modem, either move them or turn them off.
Obviously, if your router/modem includes its own Wifi access point, it should be shielded against RF on the frequencies that it uses. However, it will not be shielded from interference caused by transmissions on other frequencies. And if your router/modem does not include a Wifi access point, it is very likely not shielded at all.
If that doesn’t solve the issue, then look for other unintentional transmitters. Any electronic device is capable of emitting very low power RF as an incident to its operation. That’s why your FM radio might not work if you place it next to your computer. Move the router/modem to a place where it is at least 5 feet away from any other electronic device (including other computers, servers, speakers, etc). If that solves the problem, then you can figure out which of the devices near it are the problem by moving them closer, or just keep your router/modem away from everything else in the office.
I’ve seem this issue countless times because people tend to aggregate all of their communications and networking equipment in one place.
Ted,
The only numbers that I see on the modem are: CBR-T. I’ve had Comcast come out already and check the connection from the pole to the building. They confirmed once before that there was packet loss. At that time, they installed the replacement modem/router. Everything looked good then.
I’ve got a Meraki MX84 connected behind the router. That is acting as a NAT device. I’m not using any of the 10.x number from the Comcast device.
When the installer implemented the system, they installed a gateway that connects directly to the Comcast modem/router. That is bypassing our Meraki device. IMO, I think that the installer should have been able to configure our gateway to work with their VOIP service. I’m not really pointing the finger at them. The installer initially told us that the phone system would work with our existing infrastructure. Now they are recommending that we install an additional internet service line exclusively for the phone system.
Did I insist on using Comcast? Nope. I really dislike their service. We don’t have any other options in this area though. So, I’m stuck with them.
This makes sense. I’d have to configure another cable modem to test it.
“The installer initially told us that the phone system would work with our existing infrastructure.”
Do you have that in writing? If you don’t - they will claim they didn’t say it. If you do - then keep that in your vest pocket as it’s going to be worth some money.
Your installer didn’t buy or recommend the Meraki gear and using it means that they are going to be blamed for any configuration problems in it. That’s why they want to use their own gateway and now their own Internet connection.
Well, either they are right and your Meraki gear is the source of the trouble or they are wrong and Comcast is the source of the trouble. But it is absolutely the wrong thing at this point to pressure the installer to use the Meraki router. They sold you a product - it’s not working - you gotta do what you need to do to keep the responsibility on them to fix it. The more you get involved the easier it is for them to shift responsibility on to you for fixing the problem.
Your next steps with the VoIP installer need to be done VERY carefully. If they are making an open ended statement like “you got to get a separate Internet connection” then they may be trying to get you to go on record as selecting the Internet provider. That lets them off the hook. Call their bluff. Tell your VoIP provider “we have no problems getting a separate Internet connection - now, YOU tell us WHAT to get and from WHO. And you are going to GUARANTEE this is going to fix the problem, right? Let’s have that guarantee ON PAPER”
If you really have no other options than Comcast then your installer is going to ultimately be forced to tell you to get another Comcast circuit. Then when it comes in and the system fails again - you can go back to the installer and say “we did what you said, we went with who you said to go with we are using your gear and it’s not working. And we spent money to do all this. So as we see it not only do you still owe us fixing the system, now you also owe us all the money we wasted on a solution that didn’t work”
If by some miracle you do get a separate Comcast circuit and it works - well then you can then figure out what is different with the Comcast circuit the Meraki gear is on, fix that, then move the voice circuit back on to your main Comcast circuit at a later time.
I run my own VoIP over Comcast. There is NOTHING inside of the Comcast network that is inherently bad for VoIP. I can jack my laptop with it’s softphone into a Frontier or ATT or CentruyLink or my tethered cell phone and use it as an extension on my PBX no problem. As long as whatever remote network I am use isn’t VoIP hostile, nothing in the Comcast network is.
HOWEVER, the Comcast end user gear is something else entirely. Where I am they have 3 modems. One is a Netgear, one is a SMC the other is a Cisco. The netgear has known problems with ALG and VoIP - IF that is you use it in NAT mode and you try running SIP though it. The SMC has known issues with IPv6 and will crash if it’s enabled. The Cisco will not do IPv6 DHCP-PD and a host of other bugs other people have documented but it’s not known as VoIP hostile. HOWEVER the Cisco also has an internal wifi access point and it CANNOT be turned off unless you contact tech support at Comcast and have them do it. The Netegar also has a slow CPU and cannot run at the highest copper speeds, only the Cisco can keep up with the highest Comcast service levels. Also none of Comcast’s gear has a lot of CPU overhead and will fall down if anything on your public subnet is being attacked. For example does the VoIP gateway have an open NTP server by any chance? An attacker on the Internet can use an unsecured ntp server on a gateway for an NTP amplified attack. Your cable modem may very well be rated for thousands of packets per second - assuming the packets are 1k in size - it may roll over and die if those packets are all 64byte ntp reflection packets…
And, that last mile - the copper drop from the cable head to you - is a very tricky businss. Remember you are on a shared copper cable. Some bozo up the street could have a Comcast cable modem that is missing an attenuator and be trashing everyone on your segment. You absolutely need a very good Comcast employee to fix this if it’s the problem. Some of those Comcast installers are old-guard Telco people who understand copper. Others are young guys who pretend they know what they are doing but don’t.
Now it seems you have 2 remaining tests to do:
When voice services goes to heck does it recover if you only reboot the gateway that the installer installed and NOT reboot the comcast gear?
When voice services go to heck if you unplug your Meraki for an hour or so, so that the only thing on there is the voice service, does phone service return to normal?