depending on your virtualization, I use KVM and have made several
hundred vm’s over many years starting with vmware ( bad) then Xen but
happy now
then read
Then understand that that is not the way to go, a linux 2.? kernel won’t cut the mustard and moving it in effect to a proprietary “user-space” would be like using M$ stuff, Linux has over the years significantly improved in it’s virtualisation , your cpu’s are well set to leverage those kernel enhancements . it has both vt-x (essential) and vt-d (good if you need bus pass-through )
I use very similar hardware and I would expect about 24 or more FreePBi with one or two cpus and 1 to 2 g memory each, (true , it will be busy so do your backups out of business hours ) -anything more will be wasteful. BUT you really will benefit from linux kernel 4 , with this you can use ( as I do ) kvm optimise always , so far to 2017.
In my case I find the ProxMox GUI (HA and fail-over included for free if you have two or more) an excellent platform, even your most junior techs will “get it”
YMMV but it has worked for me flawlessly for 4 years If you doubt it, get a NUC i5 with 16G memory, installl ProxMox4 (takes 20 minutes apart from the RTFM bit) and you will easily be able to run 2xFreePBI,a kamailio proxy, a windows desktop for Quickbooks and my usb weather station, a Plex server, and still have a usable desktop on the core machine (no virtualisation ) , in my case lxde. Then try firing up your problematic VM , KVM/qemu speaks vmware disks ;-), there will be no “brief audio dropouts”
We are also experiencing random, brief (2-5s) drops in audio, on both internal and external calls. We are currently using FreePBX 13.0.190.19 with Core i5 / 8GB Memory / 500GB Sata harddisk, connected to Layer 3 Switch Port with 1G throughput.
All scenarios are reviewed at Firewall, Switch and replaced network card on System, redeployed FreePBX and moved to similar machine with same specification of different brand, but still we face same issues. L3 switch support team have also confirmed that all VLAN configurations are correct and no issues.
Since my godaddy and PBX hostname are same, hence thought this may be conflicting. My hostname was /etc/sysconfig/network - “NNN.domain.com” and after your suggestion I thought this could cause issue, hence I have modified now to NNN.domain.local.Will this solve the issue? Let me observe today and keep you posted. Thanks and appreciate your time with this.
I don’t find any difference even after updating in host file. Still intermittent brief audio drops for 2-5 seconds. This is frustrating for all users and clients.
Maybe something?
I see in the memory usage, that httpd is using more and more memory. ‘# service httpd restart’ will drop the memory usage directly.
Maybe by growing the memory, the vmtools ballooning (swap uasge) is causing this dropouts.
Disable vmware balooning is achieved by uninstalling the vmware-tools “vmware-uninstall-tools.pl” from CLI. update on vmtools: This does not change the problem, vmtools aswell open-vmtools and no vmtools do not have any effect on audio dropouts.
Afterwards check the memory usage with 'free’command, if it grows again.
I am curious if this tip will help you out. Please let me know if it resolves the dropouts.
I am still investigating the problem every week. Trying KVM instead of eSXI, etc. Hope to pinpoint the cause soon.
I am regularly seeing issues with lag which I suspect may be related to these audio dropouts. I’ve tested pinging from server console these end station IP addresses and I do see similar responses, which makes me think it’s not an issue with the freePBX software side of things.
[2017-04-28 09:00:26] NOTICE[12532]: chan_sip.c:24457 handle_response_peerpoke: Peer '245' is now Lagged. (3007ms / 2000ms)
[2017-04-28 09:00:27] NOTICE[12532]: chan_sip.c:24457 handle_response_peerpoke: Peer '307' is now Lagged. (3008ms / 2000ms)
[2017-04-28 09:00:27] NOTICE[12532]: chan_sip.c:24457 handle_response_peerpoke: Peer '308' is now Lagged. (3006ms / 2000ms)
[2017-04-28 09:00:28] NOTICE[12532]: chan_sip.c:29976 sip_poke_noanswer: Peer '306' is now UNREACHABLE! Last qualify: 8
[2017-04-28 09:00:28] NOTICE[12532]: chan_sip.c:29976 sip_poke_noanswer: Peer '303' is now UNREACHABLE! Last qualify: 8
[2017-04-28 09:00:28] NOTICE[12532]: chan_sip.c:29976 sip_poke_noanswer: Peer '301' is now UNREACHABLE! Last qualify: 7
[2017-04-28 09:00:28] NOTICE[12532]: chan_sip.c:29976 sip_poke_noanswer: Peer '250' is now UNREACHABLE! Last qualify: 7
[2017-04-28 09:00:28] NOTICE[12532]: chan_sip.c:29976 sip_poke_noanswer: Peer '251' is now UNREACHABLE! Last qualify: 8
[2017-04-28 09:00:28] NOTICE[12532]: chan_sip.c:29976 sip_poke_noanswer: Peer '304' is now UNREACHABLE! Last qualify: 77
Detailed info fetched. What is causing CPU overload at every minute for 3 seconds long?
Using Fpbx Distro 10.13.66-19 with asterisk 13.
We used “top -d 0.1” command to look what is happening every whole minute, and recorded the screen for analyse.
When the audio drops at every whole minute, the top command shows us several command which overloads all the cpu cores.
I will attach 3 screenshots, with 3 secons of top command.
What is causing the overload in cpu?
user ‘asterisk’ with command ‘asterisk’
user ‘asterisk’ with several commands ‘php’
This lasts for 2 or 3 seconds, afterwards, al the audio is good again for at least 55seconds.
The screenshots will show it all.
can I disable this asterisk user php commands to test it?
I believe that you need to raise your memory to at least 4 GB and add another CPU to your machine. After you will add the resources you can compare again with the numbers that you have shown us.
Thank you for the answer. Already did that. Added 4cpu and8Gb ram. Still the same results.
I should notice, that in this case, the pbx is running on a esxi server with 20pcs pbxs. All the pbxs run something on the whole minute, which is causing the overload.
Older pbx distro 5.x. are not showing this load at the whole minute.
If I can find what is running, than I can do something about is.
Also tried KVM and other hypervisors, but the PBX itself is running something which causes this.
I will continu the tests.
There are a limited number of things that run “on the minute” and almost all of those run out of the Crontab. The PBX itself is ‘real time’ and doesn’t really operate on a batch process. Most of the jobs inside the PBX are event-driven.
This sounds like a problem with an external process that’s chatting with Asterisk and jamming you up. Check your crontabs for both root and asterisk. Start “commenting out” the jobs that run every minute in there and see if you can narrow down the problem to one of the maintenance scripts.
According to your screenshots you do not have 8 GB of memory and 4 CPUs. If you added it already, can you share with us the updated screenshots?
If you are using the ESXi platform, check your i/o status, because every hit on the mysql database is spiking your system and probably you will need to optimize it. If you see that your CPUs are reaching the 75%, you will suffer from poor quality and you will have to optimize that as well. Check that you are using the VMnet driver for your network interfaces. And the most important thing to my opinion, make sure that you are not sharing this pbx resources with the other virtual machines on your system.
Thank for your great input.
I tested the nwe 14rc distro, and also php running at the whole minute.
I already disabled all asterisk and root cronjobs en restarted the cron service. Even remove the cronfiles in the /var/spool/cron directory. Nothing changes and the problem still excists.
You can test it for yourself, I think every new Distro is showing this result.
top -d 0.05 -u asterisk
and wait till the whole minute passes in the time upper left top. You will see a quick pass of multiple php executions.
Something is talking to asterisk, but how can I find out what is talking?
It sleeps for a random amount of time before starting. That’s why it took 19 seconds to run, but only used 0.2seconds of CPU time 8)
It sleeps for a random amount explicitly to avoid the issue of people running a bunch of FreePBX machines on the same host, and having the stats collector run at the same time.
Thank very much for this great answers!
I noticed this random 30sec switch in the scheduler.php file earlier, that is perfect, but only in the scheduler.php. Other modules not.
But other things do not run random and will create a cpu boottleneck. escpecialy the comercial disabled modules like pagingpro, queuestats and qxact_reports are killing the cpu on exacly the whole minute. When removed or uninstalled, the problem is gone or far less than before. I also think that there has to be a service priority difference. All processess run at the same priority, ik wil test a higher priority for asterisk, and less for php executions. For now I have solved the audio dropouts at the whole minute.
If you doubt it, get a NUC i5 with 16G memory, installl ProxMox4 (takes 20 minutes apart from the RTFM bit) and you will easily be able to run 2xFreePBI,a kamailio proxy, a windows desktop for Quickbooks and my usb weather station, a Plex server, and still have a usable desktop on the core machine (no virtualisation ) , in my case lxde. Then try firing up your problematic VM , KVM/qemu speaks vmware disks ;-), there will be no “brief audio dropouts”
