Intermittent brief audio dropouts

FreePBX: 13.0.190.19
Asterisk: 13.11.2

Server is running as a VM on ESXi 5.5 host.

We seem to be experiencing random, brief (2-5s) drops in audio, on both internal and external calls. I can’t reliably reproduce the problem, wondering what direction to head in terms of troubleshooting this issue.

Will log files on the server show anything useful, or am I better off trying to get a packet capture of a call where the issue occurs? Or is there another avenue of troubleshooting I should look into?

2 Likes

Same issues here

We have had the same. Reducing the fail2ban time and system load somewhat reduced the problem.
Ping or latency peaked at the same time as the dropout is happening and always exactly on the whole minute (Cronjob??). Not sure yet what is causing this.
Tested it on Esxi 5.5, 6 and 6.5. Al the same.
Older Freepbx Distro 12 is not showing this issues on the same ESXi servers.
Read also my previous post please. Audio dropouts on vm systeem with >25 Distro vm's

Are you running multiple FreePBXDistro’s on the same ESXi servers?

I have two ESX environments where I have seen similar as well. How many sockets and cores do you assign to your VM’s? VMware does behave funny when you assign multiple CPU to a machine which can have a negative effect on performance. It’s something to do with waiting for ESX to free up the number of processors you assigned to the VM before allowing a VM to use it. If something changed with FreePBX 13 that requires more cpu horsepower such as cronjobs then this could be the issue. It’s better to have one 1 Ghz processor than two 500Mhz processors.

I say this like I have solved my issue, but I have not been able to figure this out either. With the new firewall module in FreePBX 13 I was hoping fail2ban could go away because when I do a freepbx reload, Fail2Ban makes my box very busy for about 10 seconds afterwards.

This problem affects all my FreePBX13 deployments on ESX, but only a few people complain about noticing it. Likely only the busier systems have users that notice the audio drop.

Don’t have much (read: any) experience with ESX and FPBX… BUT I do use HyperV / FPBX, and have use ESX in the past with other things.

Do you have the proper drivers installed for ESX. For HyperV, the lastest versions of Centos have them built in, but before one had to install “Linux Intergration Services” to get it running properly. I believe ESX has a similiar situation. It may be that it is EMULATING an HD instead of accessing the actual hardware through proper software, which is much quicker.

Good luck

We’re having this exact issue also.

I’ve found that I’m seeing a bunch of entries in log files like:
[2017-03-17 08:30:12] NOTICE[13967] chan_sip.c: Peer '228' is now Lagged. (2071ms / 2000ms)

2071ms seems pretty excessive for a LAN link, and I suspect there is some sort of latency or packet loss on my internal network. Pinging from the PBX server console, I see this latency to edge switches, but not to the core.

VMWare guest is 2 vCPUs and 4gb ram. But I see very little load on our server, CPU is 97% idle.

I want to share some of our experiences till now, with this specific problem.

We have used the trial-version of ‘pingplotter pro’ windows version to ping each vps at every 0,5ms.
You can put several FreePBX instances below eachother to see the graphical latency history.
From external or from internal network we noticed jitter/latency peaks at every whole minute exactly.

After that we looked at cronjobs, and also we have reduced the fail2ban Bantime and Findtime.
Something is overloading at some times. There is no packet-loss, but a sudden latency change is causing these audio-dropouts.
In the Asterisk log-settings of FreePBX 13> by default the debug option is on, which creates very big log files. I can imagine that fail2ban needs more resources to process these log files.

Increasing memory and CPU-vcpu’s didn’t resolve the issue.

Now we are looking at SolusVM to check if we can reproduce this issue.

Hello,

You need to setup 8 CPUs and 8 GB of RAM at least to get normal behavior from the Asterisk server.
Furthermore, if you do not need the iSymphony system turn the service off.

Most important thing: Do not share the resources of the virtual pbx with the other machines.

Thank you,

Daniel Friedman
Trixton LTD.

8 cpu and 8 Gb of ram is not making sense.
1cpu and 768Mb ram would be more than enough. We also tried to dedicate cpu and ram to a pbx, but it was making no difference at all.
Disabling iSymphony is a good idea, but not enough to tackle this problem. I think it is a combination of problems, which causes this audio dropouts.

8 posts were split to a new topic: Inappropriate comments

I believe i am also “more experienced” :wink:

depending on your virtualization, I use KVM and have made several hundred vm’s over many years starting with vmware ( bad) then Xen but happy now, one core and 512 M of memory is good for about 20 phones, if you add 10 fax modems which need ghostscript, or you are transcoding you might need to double up.

I would have to agree with @4allbusiness though 8 cpu’s and 8g of ram is ridiculously over provisioning, maybe @danielf should explain why he thinks that and what virtualization he is using. If he did that we might be able to edumicate him and save him a whole sh#t load of money.

6 Likes

Hello friends :wave:

I am taking this time on a Sunday morning to remind everyone this is a place of civil discourse. This thread got hijacked and that may make the OP sad. We do not want sad people because sad people are not happy people. I have moved the extra chatter to a small abyss to never be heard from again. Let us please stay on topic and make sure we are being constructive to fixing or helping the OP.

Happy sunday yall :slight_smile:

3 Likes

If I may chime in here as ‘the virtualization guy’, using 8 CPUs is (almost always) wrong The more CPUs you assign to a VM, the less chance it has to run. Even though your host may have 12 CPUs, every other VM running on that host must have completed its work enough to free up 8 cores.

This is getting into reasonably technical territory here, but a good rule of thumb is to not use more than 2 CPUs for a VM unless you’re extremely sure you know what you’re doing.

VMware actually have a stat you can monitor on the ESX host which will show you how long a VM is waiting for available cycles, but I can’t remember what it is offhand.

A symptom of this is that the host will appear to be almost idle, but the responsiveness of the guest is inexplicably slow.

5 Likes

Hello @xrobau,

Yes, I agree with you. But, the systems I use for Virtualization contains at least 64 CPUs and 64 Gb of RAM so I do not have problems with performance. Less than that, It becomes not suitable to run real time applications like RTP (audio and video). And I am sure with what I am doing.

Now, the problem starts with the Freepbx Distro (which I like and use it a lot with my system optimization and customization) consumes too much resources from the virtual machine (Java, Asterisk, HTTP, PHP, Mysql, recordings, transcoding ,etc.).
If you wish to get good and reliable results from your Freepbx virtual machine do not compromise with your resources. Assigning 8 CPUs and 8 Gb of RAM (not shared) will give you good results when running on virtual machines.

If you are using public clouds (amazon, azure, digital ocean etc.) make sure that you are using SSD hard disks to get even better results.

If you choose to compromise on your resources I would go on 4 CPUs and 4 Gb of RAM (no recordings, no transcoding). Less would get terrible results.

Furthermore, you have to make sure that your infrastructure is dedicated:

  1. separated internet lines
  2. separated switches
  3. separated infrastrucrute
  4. QOS

If you are connecting your phones to your network with a separated VLAN (I never recommend that to my clients) make sure that you configure the QOS in the switches (Vlan priority).

If you are using NFS (or any other ip sharing protocol) for your recordings, use another physical ethernet port on your server and connect it to your storage network segment (a private network with a separated switch).

If your Freepbx will contain 20 extensions and less, I suggest to install the Freepbx on a small computer (Intel NUC or something similar). The results will be better than your virtual machine with the same configuration.

So, for conclusion, a voip network is very complicated and you have to consider with a lot of factors (including virtual machines). I believe that if you start your voip network design with the infrastructure, you will be able to grow and expand better without too much hassle.

Thank you,

Daniel Friedman
Trixton LTD.

That’s not a standard setup, which is why everyone was disagreeing with you. A ‘standard’ host will have maybe 24 cores and 256 or 512GB of RAM. Your design is the opposite of what most best practices are, which is why there’s so much confusion.

I never EVER recommend public clouds. Their network infrastructure is set up for high throughput, not low latency that VoIP requires.

Basically, your advice is correct if you’re using an unusual hosting setup, and are deliberately over-spec’ing your VMs. That’s fine.

But that’s not what most people do, and if you’re going to give advice that relates to your unusual setup, you need to clarify that ‘this is what I do, which is not correct for most other people’.

Edit: If I may segue slightly into nerding, having 4, or 8 CPUs in a virtualization host can often be counter-productive.

Delays have historically stayed about the same orders of magnitude, and this has always been a good rule of thumb:

  • It’s 1000 times slower to get data from Disk than from Disk Cache.
  • It’s 1000 times slower to get data from disk cache than it is from memory.
  • It’s 1000 times slower to get data from memory than it is from CPU cache.
  • it’s 1000 times slower to get data from L1 cache than it is from L3 cache.

Adding multiple CPUs adds a new an exciting level of delays (and synchronization issues) by having core 12 of CPU 0 needing to read and write to a spot of memory, and then suddenly core 7 of cpu 4 needs to read and write to that chunk of memory, too. Even worse, that RAM may be physically attached to CPU 3.

In both of those cases, the memory traffic has to be routed through the CPU cross connect bus, adding additional latency. It gets extremely technical, and if you read up on NUMA you’ll learn some history about how this became so complex so quickly.

(I actually worked for Sequent Computer Systems at the time, who were pioneers in NUMA, which is how I happen to know so much about this).

This is why most ‘high powered’ virtualization hosts only have 2 CPUs, to avoid that bottleneck totally.

2 Likes

Hello @xrobau,

Yes, sometimes you do not need to follow the stream as anyone else suggests. I get fantastic results with my Freepbx virtual machines (private and public) and I am running hundreds of extensions on every server based upon my deep learning and testing over the last 10 years. Which I mentioned in my previous replies to this post.

I do appreciate your vast experience in the matter, but I think differently from you.

If people choose not to accept my solutions (which works great on any virtual system) that is fine by me. I am sure that other people will read my last post and they will find it useful in their voip network design.

Thank you,

Daniel Friedman
Trixton LTD.

To get back “on subject”, if you have intermittent audio (no matter who the poster) , please post these answers. . .

the first question would be "what are you using to virtualize?"
the second, "how is the server dimensioned ?"
The third, “what else is running on the host machine (server) ?”

I experienced this issue a long time ago right after installing VMWare Tools on the FreePBX Distro VM. I cured the problem by disabling the balloon file.

software - VMware esxi 5.5 with vsphere essentials

host - Intel® Xeon® CPU E5506 @ 2.13GHz x 2, so 8 logical CPUs, 48Gb RAM, storage is all HP SANs, RAID5, 8 disks per tray. There are 10 other VMs running on this host, however the host total load is well within limits as many of the VMs are fairly small and undemanding.

guest - 2 vCPUs, 4Gb RAM