FreePBX | Register | Issues | Wiki | Portal | Support

Freepbx & Asterisk using far more resources on the Freepbx 14 distro

asterisk
freepbx
Tags: #<Tag:0x00007fdfa851ee58> #<Tag:0x00007fdfa851e778>

(ADTopkek) #1

Has anyone else noticed that the Freepbx 14 Distro uses far more CPU than the Freepbx 13 Distro used to use? We upgraded from Distro 13->14 and were having issues so we wiped everything and reinstalled a fresh 14 then restored from a backup. Memory usage seems about the same. Using asterisk 13 (LTS) like we used when we were on the 13 distro .

While the system is actively being used the load averages are just higher than they used to be on Freepbx 13 and the cpu spikes a lot more. When reloading on 14 the load average goes to a 10-12 when it used to be 6-9 on 13. We haven’t added much since the upgrade so it shouldn’t have changed this much from us doing things. This is a pretty active server with 50 calls being norm in the middle of the workday.

Has anyone else experienced this or found out what is going on?


(Andrew Nagy) #2

(ADTopkek) #3

I wasn’t referring to the GUI, the GUI is quite a bit faster in 14. Just watching htop while not touching the GUI at all with the UCP off. Doing a reload from fwconsole reload causes the load average to spike far more than it used to on 13.

Performing reloads is starting to cause a lot of worry due to the spikes.


(Andrew Nagy) #4

The cross referenced post still applies.


(ADTopkek) #5

So we found something that was causing CPU spikes on a schedule. It also seems that it is having a consistent error over and over and over and over:

Bad File Mode

Checking the /etc/cron.d/ folder we had this:
Selection_018

It had this in the file:

# This runs the update check every night at midnight. It sleeps for up to an hour
# as a random delay.
#
# This file was installed by the 'sangoma-pbx' package.
#
0 0 * * * root [ -e /etc/profile.d/z001-updates.sh ] && /etc/profile.d/z001-updates.sh update

Isn’t only the root user suppose to be in the etc/cron.d/ directory?


(Andrew Nagy) #7

Change the ownership of the file if you think that is what’s causing your systemwide issues (I doubt it is)


(Rob Thomas) #8

Whoops, yes, that’s a bug. That REALLY shouldn’t be owned by the asterisk user! Thanks very much for picking that up!


(Rob Thomas) #9

This is now fixed in sangoma-pbx-1802-2, just run ‘yum update’ and it’ll pick it up. (You may need to ‘yum clean metadata’ first, if it doesn’t pick it up as available). Thanks!


(ADTopkek) #10

^ :smiley: found another security bug, thanks for fixing that.

We found a way to make asterisk run better during normal operations and through reloads, although reload still cause issues at times. We used HTOP to decrease the nice setting to -10 on the main asterisk thread and that made it run MUCH better and stopped some of the call quality issues plus bought us time to look into this more.

It seems that the more calls asterisk has processed without restarting the worse it’s performance gets. If we let it get to around 60,000 calls processed we will hear small sound issues and around 100,000 we can notice sound issues on customers that should have no sound issues. Upon an fwconsole restart everything sounds great again.

Reloads also follow a similar pattern and progressively use more resources the more calls that have been processed but it has an odd little pattern that I am not sure what to make of:

We will apply changes or fwconsole reload.
I will watch asterisk be fed the new dial plan.
Load jumps to around 7-10 in HTOP
We then get informed that the reload is done.
Load then spikes to 10-24 on a 12 core server. It progressively rises and hits a very high load average then drops.

I would expect the largest spike to be the asterisk dialplan switch over but a large spike continues on after the dialplan switch over happens. What happens after fwconsole reload says it has completed?


(Andrew Nagy) #11

Rest Apps is restarted.


(ADTopkek) #12

Does anything else happen? I stopped restapps from PM2 and reloaded.

While Fwconsole reload was running load average maxed at 9.46 then once Fwconsole was done it shot up quickly shooting up and went to 17 then held there for about a minute. 36 calls active 89425 calls processed.

Call quality went to shit even when it was at 9.46. Have 12 cores and none of them were maxed according to htop but the main asterisk thread was at 200% cpu despite the CPU cores not reflecting that individually.


High CPU Usage for an idle box
(Tony Lewis) #13

Asterisk will only basically use 1 thread. So having more than 1 core doesn’t help the asterisk side of things.


(ADTopkek) #14

So the main asterisk thread will only use 1 thread. Am I correct in thinking that the child threads of the main thread can then use whatever core the kernal assigns it?


#15

Hi,
I have just tried to transition from FreePBX 13 to FreePBX 14.
This would not do an upgrade in place so I did a clean install and followed the migration steps, which worked for the most part, a few tweeks and a missing trunk, but it worked.

Problem is that I have excessive load in general, let alone when something is happening… Initially I thought it was just re-configuring as a background process so I muddled with it until it is still the same after 24 hours.
About this time I loose web GUI connections as it is all running so slow.
Rebooting the server is not making things any better, a good 6 mins to get asterisk running, and more before I can get a GUI connection.

My setup is nothing like what you would expect to be causing load. This is just running in my home for personal and home office lines.
I have a total of five extensions and four trunks.
Most of the time the server is sitting doing nothing and I have a load ranging from 1.2 up to over 30.0 on a 5 min average.
When it takes off everything grinds, I am able to login to the console after about 45 seconds and then it is slow to respond when in the shell.
When the load picks up, top shows several php scripts running.

I moved back to the FreePBX13 that was running and the load at idle remains between 0.02 and 0.06, most of that is from the top application itself.

The environment is a ProxMox VM. Both have the same config:
two cores on one CPU socket
2Gb RAM
20GB storage.

With FreePBX13 running, my ProxMox host is running at a 5min load average of 0.5 - 0.8

The FreePBX14 server is the latest ISO image and when it installed I performed an OS update.
I have removed most modules I will not use or have no license for and the load did not change.

Is there a fix I am missing? Not much came up under searching for excessive load and the like.

Can anyone offer a solution?

Thanks

Paul Budden


(Andrew Nagy) #16

Nope. There is no solution.


(ADTopkek) #17

I’ve done micro-improvements here and there that have lessened the load (Removing old stuff, condensing other stuff, making sure recordings and music on hold have proper file extensions, switching extensions over to PJSIP) but no real solution.

After cleaning up a lot of stuff I have noticed that most of the normal non-reload spikes come from new inbound calls or calls that are being hung up. When you reload the spikes come from BLF/Hint updates.


(Avayax) #18

Have you found an answer to that question?


(ADTopkek) #19

Nope.


(Andrew Nagy) #20

Yes but the code section has to be coded for multi-threading. EG PJSIP is multi-threaded. chan_sip is not. Other parts of asterisk are other parts are not.


(ADTopkek) #21

That would explain why we are showing performance gains with PJSIP. We have tons of mostly idle cores.