Wow, your fwconsole reloads take a very long time.
How many extensions do you have on your system?
All this points to a php culprit. A wild guess is that is being called from crond , I would in one session
tailf /var/log/cron
and in another, run whatever diagnostic you are comfortable with. If they correlate , then “next step”
It’s more the massive call flow rather than the number of extensions. But we have ~ 1900 extensions 1160 of which are being used. While the call flow is being processed I don’t hear any sound issues until it finishes and asterisk switches over to the new dial plan but thats just momentary, really we expect that and its not that bad. Then for a few minutes after load spikes and call quality goes to shit AFTER the reload is done.
Had a slightly smaller number before moving from Freepbx 13 -> 14 and asterisk was handling the load like a champ and using lower spec hardware.
So you mean your call quality goes to shit for a few minutes after each reload?
Once a minute our cron jobs run and it does cause some spike. We have looked into that and it by itself doesn’t seem to cause any issues so we haven’t been really worried about that. They cause a 30-50% spike in the CPU but we will jump from 50% usage to 200% usage inbetween the cron jobs running.
We only seem to start having any call quality issues once the CPU approaches 250% or above thanks to our reniceing of it, if you haven’t reniced asterisk I would highly suggest it.
Yes. After the Fwconsole reload is finished that is when it goes to shit, it lasts a couple minutes. Any issues while the reload is running have been minor since we reniced asterisk. The moment the reload is finished I hear a small noise but I find that to be expected when switching to a new dialplan.
Edit: I just noticed I hijacked someone else’s thread instead of posting in my own. If a mod wants to here is where this discussion should have been at: Freepbx & Asterisk using far more resources on the Freepbx 14 distro
The point is, WHAT is being run and is it critical ( I doubt that it is probably just for the FreePBX gui) if not critical relegate that cron job to run out of hours only or just on Sunday morning,
Here is what is being ran every minute:
Mar 30 16:55:01 voicelync3 crond[1207]: (asterisk) RELOAD (/var/spool/cron/asterisk)
Mar 30 16:55:01 voicelync3 CROND[23538]: (asterisk) CMD ([ -x /var/lib/asterisk/bin/schedtc.php ] && /var/lib/asterisk/bin/schedtc.php)
Mar 30 16:55:01 voicelync3 CROND[23539]: (asterisk) CMD (/var/lib/asterisk/bin/pagepro-scheduler.php 2>&1 >/dev/null)
Mar 30 16:55:01 voicelync3 CROND[23540]: (asterisk) CMD ([ -x /var/www/html/admin/modules/dashboard/scheduler.php ] && /var/www/html/admin/modules/dashboard/scheduler.php > /dev/null 2>&1)
Mar 30 16:55:01 voicelync3 CROND[23541]: (asterisk) CMD (/usr/sbin/fwconsole calendar --sync 2>&1 > /dev/null)
Mar 30 16:55:01 voicelync3 crontab[23572]: (asterisk) LIST (asterisk)
Mar 30 16:55:01 voicelync3 crontab[23574]: (asterisk) LIST (asterisk)
Mar 30 16:55:01 voicelync3 crontab[23577]: (asterisk) LIST (asterisk)
Mar 30 16:55:03 voicelync3 crontab[23602]: (asterisk) LIST (asterisk)
Mar 30 16:55:03 voicelync3 crontab[23604]: (asterisk) LIST (asterisk)
Mar 30 16:55:03 voicelync3 crontab[23605]: (asterisk) REPLACE (asterisk)
Mar 30 16:55:03 voicelync3 crontab[23607]: (asterisk) LIST (asterisk)
Mar 30 16:55:03 voicelync3 crontab[23609]: (asterisk) LIST (asterisk)
Mar 30 16:55:03 voicelync3 crontab[23610]: (asterisk) REPLACE (asterisk)
Mar 30 16:55:03 voicelync3 crontab[23612]: (asterisk) LIST (asterisk)
We disabled Time Conditions, Pagepro, Dashboard,and Calendar and call quality was still horrible right after a reload.
crontab -e -u asterisk
comment out the stupid ones. (time conditions not so stupid)
I have no idea where these are coming from:
Mar 30 16:55:01 voicelync3 crontab[23572]: (asterisk) LIST (asterisk)
Mar 30 16:55:01 voicelync3 crontab[23574]: (asterisk) LIST (asterisk)
Mar 30 16:55:01 voicelync3 crontab[23577]: (asterisk) LIST (asterisk)
Mar 30 16:55:03 voicelync3 crontab[23602]: (asterisk) LIST (asterisk)
Mar 30 16:55:03 voicelync3 crontab[23604]: (asterisk) LIST (asterisk)
Mar 30 16:55:03 voicelync3 crontab[23605]: (asterisk) REPLACE (asterisk)
Mar 30 16:55:03 voicelync3 crontab[23607]: (asterisk) LIST (asterisk)
Mar 30 16:55:03 voicelync3 crontab[23609]: (asterisk) LIST (asterisk)
Mar 30 16:55:03 voicelync3 crontab[23610]: (asterisk) REPLACE (asterisk)
Mar 30 16:55:03 voicelync3 crontab[23612]: (asterisk) LIST (asterisk)
Everything else we use we have commented out (disabled) and tested without an impact on call quality after a reload. Causes a few less spikes in the CPU but does nothing for the major 200% spikes which seem to be caused by ???.
We can have 1 person in total making a call and it still messes up after the reload is finished.
the only thing you have commented out (pagepro-schedular) is later still there, so you changed nothing
will show you what and when the first five fields are interpreted as. correlate them with your spikes
I know it is not commented out right now. We spent a day looking into Cron as being a source of our issues, testing, and retesting. We found that yes it caused increase loads when the tasks ran but the increases never resulted in call quality problems even when we spammed the cron jobs over and over and over to try and break it.
Disabling all of them did not help after the reload so we accepted that we just have harmless spikes from cron jobs every minute.
reloading 1900 extensions in FreePBX will never be quick, I believe that was improved in newer versions (what are you using ?), I suggest you refrain from reloading during working hours
During the Fwconsole reload there are very little if any call quality issues and it seems to reload quickly, well my idea of quick might be slow to you. After Fwconsole reload is completed then that is when we get our call quality issues. Load will start rising and keep rising for a few minutes during which call quality gets really bad, in the reload video. Once it stops rising the call quality issues stop and everything is normal.
Its pretty strange and has us baffled. This never happened with Freepbx 13 and we cannot revert. It seems like something changed with Freepbx 14 because our asterisk version was and still is 13.
Bug report?
I think that cries for a bug report or even a commercial support ticket if needed.
Was it you that also saw painful GUI slowness with a large number of extensions?
I think Rob said that that would be much improved with PHP7 in FreePBX 16 (or later?)
Guys after the reolosd finishes from.the GUI in the background all the hints are resubsribing and rebuilding. With 1900 extensions without lots of CPU power you will see problems. Even with beefy CPU you can see problems. It’s why reloads have to be done after hours.
3 Likes
No. It doesn’t. If you can’t figure out the issue in this thread what makes you think opening a bug report would help. We would immediately close that as “Cannot reproduce”.
Commercial support is different. Don’t open a bug just because you have an issue you can’t figure out. Bug reports are for reproducible issues.
Not really. Furthermore nothing in this thread has conclusively stated it’s php causing the issues. What Tony lewis said is accurate.
I’ve been quietly watching this thread. I only have a couple of quickies:
- 1900 extensions is a lot of extensions for a single instance of anything. I have the magic number “800” in my head as the most that will work reliably. It could be from discussions here, or I might have dreamed it, but almost 2000 extensions on a single instance is a LOT.
- We’ve heard lots of reports in the past with lots smaller systems that hint processing takes a considerable amount of CPU horsepower. Even if all that’s happening is all of the 1900 extensions setting up their hints, that’s still something like 5500 hints to process.
3 Likes
Not the reload but the GUI slowness that others were experiencing with a large number of extensions is what I read somewhere will be much improved with PHP 7, or did I dream that?