Audio dropouts on vm systeem with >25 Distro vm's

Distro Discussion & Help
1

Hello FreePBX and Users,

We are seeing very short audio dropouts at every whole minute and whole hour on systems where we haven 25 or more VM FreePBX-Distro’s. Version, memory or other changes like x32 or x64, does not show any difference.
For our feeling, older upgraded Distro’s from 5.12.x show this problem lesser than the newest 6.13.x or 10.13.x versions.

After investigating for day’s we can conclude this is coming from within the FreePBX vm’s

Searching for cronjobs every minute and hour, but not sure what job, or combination, can create this issue.

The network latency shows spikes at the whole minute, probably cause by cronjobs, which are all executed at the exact same time.

Now we looked in the file ‘/var/www/html/admin/modules/dashboard/scheduler.php’
Which contains the next rule:

// Sleep to fix crazy issues with large VM hosting providers
sleep(mt_rand(1,30));

Does anybody know which scheduled job takes the most time or system resources which can cause this latency problems?

Any help is welcome. Many thanks in advance.

2

Found some extra info.
I used ‘top’ commando with 0.1sec refresh. Than you can see that PHP is taking almost all CPU resources at every whole minute.
What could that be?, and how can I restrict php CPU usage?
See screenshot.
Also Fail2ban is growing in CPU usage.

sshot-111.png758×418 32.1 KB

3

You should be able to find the jobs by running

ps fauxwww

There are also a few cronjobs which run all the time under asterisk’s crontab, you could comment these out by using crontab

crontab -e -u asterisk

These are the jobs I’ve seen and commented out simply because I do not need them:



*/1 * * * * /var/lib/asterisk/bin/pagepro-scheduler.php 2>&1 >/dev/null

* * * * * [ -x /var/lib/asterisk/bin/schedtc.php ] && /var/lib/asterisk/bin/schedtc.php

*/5 * * * * [ -e /var/www/html/admin/modules/qxact_reports/import_queue_data.php ] && php /var/www/html/admin/modules/qxact_reports/import_queue_data.php

As always make sure you do not need any jobs you are considering not running. I’ve also disabled fail2ban but ONLY because I have my pbx in a vlan which does not allow ssh or web access from the outside network. Also when running top if you have multiple cores hit the number 1 on your keyboard to list all the cores, if you are utilizing 100% of a single core that should be ok. I had similar issues but they stopped when removing the jobs and stopping fail2ban, however this box is a fairly small build, so I’m not sure if the situation is comparable.

1 Like
4

Hello FreePBX,

We have changed our Virtualization software from Esxi to KVM. Now we see exactly the same problem when we host more than 15> FreePBX distros on one server.
It has something to do with the execution of some cron tasks or system stats, which the pbx fetches every whole minute.
This is a big problem for Fpbx hosters.
I have made a youtube movie to show what happens in the host CPU when the minute passes 00 seconds.
This produces a overload, which cause audio dropouts.
See: https://youtu.be/2Bwdjm8eOaU

Can you point me a way where I can adjust some files for testing to do this randomly, instead of exactly when passing a minute?

My suspecion is somewhere in system stats or Dashboard stats or something like that,
/var/www/html/admin/modules/dashboard/scheduler.php

I think it can be a simple solution, but any help is appreciated.

5

I think the proble can be solved by adding a random time in schedtc.php in the cronjob.
I wil test it in the upcoming time.

when looking in
crontab -e -u asterisk
We will see the line
* * * * * [ -x /var/lib/asterisk/bin/schedtc.php ] && /var/lib/asterisk/bin/schedtc.php
This can be changed in something like
* * * * * /bin/sleep/usr/bin/expr $RANDOM % 30;/[ -x /var/lib/asterisk/bin/schedtc.php ] && /var/lib/asterisk/bin/schedtc.php
Which will execute schedtc.php randomly between 0~30seconds
test it with:
# time /bin/sleep/usr/bin/expr $RANDOM % 30;/var/lib/asterisk/bin/schedtc.php

Or we can add a random time in the schedtc.php file, but I don’t know how yet. This is also written in the dashboard module scheduler.php with the content:
//sleep to fix crazy issues with large VM hosting providers sleep(mt_rand(1,30));

info took from: http://zcentric.com/2011/06/03/crontab-with-a-random-sleep/

I am curious what you may think of this.
Many thanks in advance.

6

I found a way to add a random time in the
/var/www/html/admin/modules/timeconditions/bin/schedtc.php
just add a line after //include bootstrap with:
sleep (rand (0,30));

I will test it on a host with >20 VPSs on KVM and let you know the results.

1 Like