Dahdi_test timing _way_ off on vmware esxi 5.5, severe audio chop

Running FreePBX distro, 6.12.65 as a virtual machine.

I’m having a heck of a time with the audio on this machine. Around the beginning of the year or so (can’t pin it down due to lack of activity around the holidays), we started getting some serious ‘chop’ in our audio, especially in conferences (though it is present in every call, it’s almost impossible to use a conference room due to the amount of audio chop).

I investigated our UDP traffic, and had our network guys do extensive testing on the network and internet connection to make sure we weren’t just dropping packets due to a bad piece of network hardware. Finally did a dahdi_test, and found this:

Opened pseudo dahdi interface, measuring accuracy...
99.997% 99.913% 99.809% 99.997% 99.945% 99.750% 99.809% 99.907%
99.901% 99.939% 99.927% 99.907% 99.900% 99.858% 97.374% 99.947%
99.995% 99.903% 99.656% 99.546% 99.995% 99.998% ^C
--- Results after 22 passes ---
Best: 99.998% -- Worst: 97.374% -- Average: 99.771525%
Cummulative Accuracy (not per pass): 99.867

Pretty bad. I installed a brand new VM with similar settings and a fresh copy of freepbx, latest version. Same thing. I put it on a different host within our organization, same thing. If I go to another, similar host at a different company, they have a perfect dahdi_test.

So it’s probably something in our VM configuration, I concluded. Me and our ESX guy tried increasing the priority given to the VM by setting the 'Latency Sensitivity" to “High” and reserving 2 full cores and 3gb memory, dedicated them to the PBX. It made absolutely no difference; in fact, I got worse results occasionally.

So we decided to get a Sangoma USB VoiceTime device; plugged it in and attached to the VM. I had to download and compile an older version of Dahdi (2.5 or so) in order to get the voicetime drivers to compile and install/work. I had to contact support to make sure the device was installed and working. They verified that it was.

Installing the VoiceTime device helped a little bit, but the same symptoms are still there and I have no idea what to try next. Help!

Oh, also, I lied; the dahdi_test results above were done about 10 minutes ago with the voicetime device attached. Unattached, it’s pretty similar, but actually worse.

I could really use some suggestions. Conducting some tests today to see if how the VM behaves on different hosts; perhaps we can narrow something down.

We loaded the PBX on a new ESXi host; it is, in fact, the only VM running on that hardware. I’ve added similar high sensitivity options for this VM. I get better results, but still get things outside of range (less than 99.98%); some as low as 99.7%. I’ve gotten as low as 93% on my other VM, though (which sounds like garbage).

I installed a brand new, fresh PBX from distro on our problem host environment; right out of the box with nothing customized or running, we got audio chop.

Now is when I get really confused, because I did some tests in another environment, on another FreePBX distro VM, and the dahdi_test results were very similar (low of 99.6, cumulative avg of around 99.98). However, there is no chop in their audio whatsoever.

Did I mention this occurs on fully local connections? Two phones, 20 ft away from the pbx, making an internal call without our provider (sipstation) in the mix, and there will be audio chop. Not as much as a conference call, but chop none-the-less.

I’m really at the point where I’m about to give up and put the PBX on hardware. This has been going on for a month, and my customer is very unhappy. Even if I do so, I want to figure this out! Other people run their FreePBX on a VM (including me) with no issue. Some of those other people are my clients; I really don’t want to start issuing hardware PBXs if this happens to my clients.

Are there any other causes for this kind of choppiness, outside of network packet loss? Is there anything I haven’t tried to optimize this dahdi timing?

It’s not the dahdi_dummy, that is really not needed anymore if you use app_confbridge and not app_meetme for conferences, the former uses kernel based timing, but . . .

It is almost certainly an admixture of your OS, hardware and your choice of virtualization technology, as the cpu/network passed through to the VM and the transparency of that VM to/from the real world is your problem.

I would try using KVM (qemu) on a linux machine with VT-X/AMD-V enabled CPU and decent hardware, ensure the virtualization technology is enabled in the bios.

You could quickly test the whole shebang with:-

https://www.proxmox.com/en/downloads

OS/GUI/KVM all setup for you (4 FreePBI and a Windoze of choice easily on a NUC i5, save a client, spend 500 bucks :wink: )

I have just recently run into this same issue for the first time after many years of deploying freepbx on vmware ESXi Dell Servers. ESXi 5.5U2 on a Dell poweredge T410 and a T430. If I vmotion the PBX to T410 server, it works fine for the most part. Put it on the T430 and I have audio issues. So, I tried a different NIC thinking this is a broadcom / esxi driver issue, so switching to Intel NICs in the T430 made no difference - same issues. So, I am stuck running the VM on an old server for now. Faxing still doesn’t work AT ALL through freepbx on that server, T.38 or ulaw, so this is still not ideal. I am not sure sure what else to do, but the calls get recorded with bad audio also, so it isn’t a LAN or WAN issue with the SIP provider. It is a PBX / Server issue . I have not tried upgrading to ESXi 5.5U3 or 6.x yet because I am not convinced this will fix it either. … And Dicko - changing hypervisors is not an option in an environment where there is already an investment in Vmware and Veeam, also, I already know vmware esxi is more than capable of successfully and flawlessly handling freepbx, it just seems certain environments or hardware combos give us grief.