How does one investigate call quality issues? I have several (about 8) PBX systems running on CyberLynk VPS servers. I have upgraded them to more CPU, more RAM, SSD Hard disks, and upgraded my customer internet connections to 100mbps, 150Mbps, and one even to 1Gbps to no avail. I continue to have various call quality issues on all the systems with different phones, analog, digital, and have been working on it for more than 4 years. It just never seems to get any better. The only common denominators are FreePBX, Telnyx, and Cyberlink. I need to figure out which component is causing the issue and fix or replace it.
I am looking for help in determining the issue and fixing it. If there is a way to get these to all work together then great, if not I will replace any of the components needed to resolve. Looking for comments on anyone who uses CyberLynk, AWS, Telnet. I have 200 phones on 8 PBX’s and want to expand to 5000-10000 phones but need a robust environment that works reliable and clear.
Telnyx has a TLS connection to AWS so I am thinking that if I setup an AWS server this weekend I can see if its the CyberLynk environment is an issue. I asked them to create a TLS to Telnyx several years ago and they were not interested. My gut instinct tells me the issue is the network between CyberLynk and Telnyx. They are literally in the same city but have ping times much higher then me 500 miles away (i.e. I can pink Cyberlynk, and I can pink Telnyx both much lower pink time then pinging telnyx from my servers at cyberlynk.
Suggestions? Is there an app I can install on my PBX to show me the cll quality and where the issue is?
Does your PBX have similar quality issues? If so, let’s track them down on your system.
If not, is your system essentially the same as what you sell? If so, do you have any idea of what the difference may be? If your system is different, why?
Approximately what percentage of calls are bad? About how many calls per day for a typical customer? Are incoming calls affected? Outgoing? Calls between extensions? Does the problem affect what the PBX user hears, what the remote party hears, or both?
Describe the impairments (choppy voice, dropouts lasting several seconds, echo, bad latency, noise, distortion, ‘watery’ sound, inadequate or excessive volume, etc.)
Does the trouble typically start while a call has been in progress for a while? If so, does the call return to normal after a few seconds, or does it remain bad (forcing the parties to hang up)? If so, is quality almost always better when the call is re-established?
You mention both Telnyx and Telnet? Was the latter just a typo, or do you also use TelNet Worldwide? If so and you have both trunks on the same system, how do they compare?
I will take all the suggestions I can get. I have been dealing with this for a while but I feel like its been getting worse. On a typical call I will hear the occasional chirp, stutter occasionally but the remote user asks people to repeat what they said often as it was compromised quality. I cant really say what % of calls are affected but I believe almost every call more than a few minutes has some audio quality issues but again, mostly for the remote user. I have to do some testing but I believe there is no difference even when its calls between extensions (on the same PBX).which I believe would eliminate Telnyx the SIP provider as an issue but I am note really sure at this point. I would love to install something that would let me produce a report of all calls and their quality?
I have had several types of complaints. Typically on my side (the VoIP phone) I will hear the occasional drop out or choppy voice but when it happens its very brief. I have a 1Gbps uplink and my Ping time to the PBX is 22/31/24 (L/H/A for 100 pings of 1100 bytes). The ping from the PBX to Telnyx is similar (about 24ms on average). Many times when my customers complain they say it sounds like they are talking under water. I have called from one extension to another in my office and put the handset under the TV and watched TV for 1/2 an hour listening to the show over the phone and didnt notice any issue at all (but thats a one way conversation). Most customers complain that thie side isnt bad (ocasional chirp/droupout) but that their clients complain that they didnt hear something and to repeat it so I assume thats more significant droupouts or choppy. So its affecting the remote users a lot more.
it allows you to take any easily found call apart down all the way to the individual SIP/SDP packet and that at your leisure using tools such as sngrep, wireshark , audacity or whatever, (TLS a little shaky though)
Are these symmetrical rates or is the uplink much lower, as would be typical of a cable (DOCSIS) connection?
This is why I ask the previous question. A unidirectional choppiness could mean that throughput is constrained in one direction only. In this case your customers might be saturating their uplink while there is plenty of capacity left on the downlink. If so, QOS/shaping will help.
I agree with @billsimon that saturated uplink is likely.
To confirm or refute, capture all traffic at the PBX, wait for a report of a bad call, find the captured call and analyze the RTP with Wireshark, looking for lost packets and excessive jitter. See
You mentioned TLS and if all traffic is encrypted, it makes troubleshooting more difficult. In this case, I would recommend also turning pjsip logger on. You can then see unencrypted SIP for the bad call with accurate timestamps in the Asterisk log, so you can find the RTP in Wireshark. Although the RTP is also encrypted (you can’t listen to it), you can still analyze it for loss and jitter.
Depending on what router you have in place, might check there for bandwidth utilization at the time of a call with poor quality. Might be able to confirm link saturation very quickly.
I disagree with both @RealRuler2112 and @dicko on this. For example, assuming 10 Mbps upload speed, if a non-VoIP device sends one-megabit bursts once per second, each burst will take 100 ms to send and a VoIP packet after that in queue will be delayed by up to 100 ms, enough to cause choppy voice. However, the bandwidth utilization (even if reported every second) will be only 10% and it would appear everything is fine.
IMO the first step should be to determine whether the traffic arriving at the PBX has excessive packet loss or jitter.
all services I was talking about are symmetrical, fiber internet like FiOS. Even when the office is busy they are not using anywhere near the rated speeds. I can do a speed test anytime and still get near full speed. I did some testing and from my phone to the PBX is 22-31ms average is 24. From the pbx to telnyx is 21-26 average 24ms (both tests I was using a 1000 byte packet). In doing more diagnostics, so far I think it happens even when I call from 1 extension to the other which technically should eliminate telnyx from the equations although I am not 100% certain that its not worse when its to an external number.
I am not using TLS (yet) but have a desire. I am trying to get CyberLync to create a TLS link to Telnyx to minimize any impact between these 2 components.
most of the results were pretty benign but I did notice some rxdrops. Keep in mind the busiest part of the day was only 101kbps (between midnight and 2am I do backups to a remote system so it was a lot busier for CPU, disk, and network) but are the rxdrops an indication of an issue? I dont like how often they pop up here considering how little traffic I am using.
I agree but in my office, I have 1gbps Internet (fiber) and during the day my PC is literally the only one doing anything on the internet (small home office) and I have had an issue when it was pretty much idle (not accounting for any background tasks) but I am absolutely sure I am not utilizing much bandwidth when I am on a call.
I am just an average user…but I always use the freePBX server and (if possible) the phones on-site…in the same local network.
If your server is at Amazon, you could check voicemail recordings, if the audio is bad too. Or you could do a forwarding from your server at AWS to your cell phone. Do you hear artefacts too? If not, your problem might be located somewhere between your phones and your server. If yes, your problem might be your SIP-Trunk provider. You could add a SIP-trunk to a different provider and check if you get the same issues.
EDIT: I once had similar quality issues and I checked everything…it finally turned out that the cause was my SIP-trunk provider. Unfortunately, this was the last one I thought of…
rxdrops (lack of reply) could be an indication of congestion outside your network, if there is a router between your pbx and the world, do you have access to it’s logging or other low level services ?
on the PBX side no, on the client side yes. How bad are the rxdrops in the above log? Is there anything I can run on my PC to test the connection to the PBX?
Not easily, sipsak comes to mind, but as mentioned saving everything with pcapsipdump lets you analyse credible events in great detail at your liesure. I would turn off TLS first, a) to eliminate that as a source and b) because pcapsipdump’s -k option probably won’t be able to decrypt current default certifications so extracting media would not be possible.
If you don’t have encryption on, you could get a network tap & capture everything just before your modem, then analyze the captures with wireshark to determine where the problem is. (A cheap 5 port managed switch is perfect for this - just set up port mirroring & connect it inline, running the mirrored port to a laptop running tshark or tcpdump - the GUI becomes unstable when doing long-term captures.) By doing this, you can see the statistics about the call & also listen to the RTP streams.
This would at least tell you if the audio is leaving your network and coming back in clean. I’ve never dealt with a PBX in the cloud before; a network tap was exactly what I needed to solve call quality issues here as it proved the audio was going out was good - our SIP provider’s upstream carrier had an issue in our case.
