Choppy audio in every one second interval

Hi, this may be too broad issue to ask in this forum, but I am currently experiencing choppy voice issue with my PBX. Issue is more common when call from outside is involved, (call travelling through SIP trunk)

As for testing purpose, I created inbound route to echo test (*43) then I call from my cellular to the DID, I notice my looped back voice gets chopped in every (almost) one second interval. Strictly speaking, for the very beginning of the call until 3 seconds or so my voice loops back fine, but after that every almost one second interval, my voice was disrupted as if silent was inserted in between.

Difficulty is that this ssue is not happening all the time, it happens randomly.
I assume it may relate to the network congestion or some sort, but I am curious to know what could be the root cause of one second interval disruption, it is almost like a heartbeat…

Any advice / comment would be appreciated. Thank you! :slight_smile:
Environment:

  • Asterisk: 13.6.0
  • FreePBX: 13.0.192.18
  • Linux (CentOS7): 3.10.0-693.2.2.el7.x86_64

What are you running the PBX on? Enough CPU and RAM?

Hi,

It is on CentOS7, it is under VMWare’s VM environment. 4GB memory is allocated, in average 650MB is in use and steady. CPU usage is very low, by monitoring with top command. (It is a small office setup with around 30 extensions, and call volume is very low, perhaps 6 to 7 concurrent call at a peak)

from asterisk cli

rtp set debug on (or host . . . or ip . . .)

would show poorly connected rtp streams

Hi Dicko,

Thanks for the advice! I tried “rtp set debug on” then all of sudden tons of debug information showed up, I want to narrow down the amount of information so that /var/log/asterisk/full file won’t explode. :sunglasses:

Is there way to monitor call session on particular DID? In this case, I am curious to take the debug information on the call received on my test DID. (destination is sent to echo test, *43)

If you debug rtp then you will get t loglines every 80 ms. they are timestamped. if the get/put lines are not perfectly interleaved, then you are starting to see the root of your problem, usually network congestion or bad qos.

Thank you Dicko, FYI, I ended up running tcpdump on PBX side for easier analysis with Wireshark,

tcpdump -i ens32 -n -p -vvv -s0 udp -w dump.pcap

then it became obvious as in below. (Gray is from my cellular to PBX, and Blue is from PBX to my cellular)

Approximately until 28 seconds, “echo test” announcement was played from PBX, and this voice prompt was heard clearly, and from 31 seconds, and after I made continuous “ahhhhhh” and “eeeeeeee” sound for the test, and as you can see my voice was “chopped off” in periodical manner (slightly less than 1 second interval).

I have never seen a pattern like that before, can we eliminate this machine being virtualized and that you are not using a cable modem?

Hi Dicko,
PBX is in a datacenter, we use fiber connection for the Internet, I doubt the issue on ISP side.
Removing from VM environment may take a while because our setup virtualized most of servers.
Still looking for what could be the cause of saw-tooth like wave form, I understand my voice is chopped off before getting into PBX, and wondering what could cause such periodical disconnect. Something must be at the router or firewall? I need to dig into a little more…

Then you are left with the virtualization, as I said before, do you see the rtp debug showing delayed packets or are they just being dropped?. Delayed packets point to a kernel problem , dropped packets to a network problem (which might also be your virualization) . Please explain what you mean by “sawtooth” . . .

Hi Dicko,

I think the voice packet has been “chopped” before reaching to PBX. (Sorry, I should rather said “comb-like” wave form than sawtooth, I am basically meant to say the gray part of the waveform it should be continuous)

Below is the statistic of RTP, no lost packet and jitter value is negligible.

So I guess some where before the voice reached to PBX, voice is periodically discarded, I am looking into network for some hint. Thanks for rtp debug advice though! :wink:

Again, you need to look at the raw packets,be careful with trusting wireshark, it will reassemble out of order packets unless you tell it not to. So please just show us your raw rtp packets as is before you go any further, (sawtooth and comb neither make sense here and wireshark is not the right tool either)

Hi Dicko,
I stripped rtp debug info from /var/log/asterisk/full, should I be pasting the text file here or is there any tool to analyze those “Got / Sent RTP packet from /to” message? Apologize, I am novice in RTP analysis… (very much humbled)

Just look at the stream, unless it is got/sent/got/sent then no proplem if there are more than a couple of got’s or sent’s in a row you have a problem.

Hi Dicko,
It is very clean got/sent/got/sent pattern during call with incremental sequence numbers.

Yet your experience shows otherwise, I still suspect your virtualization.

Hi there,

as we are facing the exact same issue, I am wondering, if you have been able to solve it?
And if so: How did you manage it?

Greetings,

Manuel

Hi Manuel,

In my case, issue was obvious with smartphone devices. after I received complaint from my user, I tested several devices (iPhone and Android), and I was able to reproduce the issue regardless iphone or android when I talk louder and continuously to the smartphone or using it in speakerphone mode.

It seemed to me very simple voice suppression feature was kicking in on smartphone side and once started, it was cutting off my voice approximately every 1 second. PBX was basically receiving my “chopped” voice, PBX wasn’t the suspect of the issue.

Perhaps this is because the enclosure of smartphone nowadays is very thin and tight, acoustic leakage from speaker to mic is hard to eliminate, perhaps mic gain is forcefully muted to prevent howling condition. Tricky part is the user won’t notice the issue, but the far-end parties, especially during conference call.

I basically told the user not to use speakerphone mode or use Bluetooth headset with good acoustic echo canceler/suppressor feature when the user is on conference call since far-end parties may hear choppy voice of the user otherwise.

Interesting thing to note was that when I made a call from Google Hangouts Dialer, “choppiness” characteristic was difficult to reproduce. Perhaps because of the protocol difference, I am not certain, but I am not experiencing the issue with regular desk phone (those business phone’s enclosure are normally well designed to eliminate acoustic leakage) so another recommendation is to use desk phone with handset (not speakerphone) during conference call. :wink:

Regards,