FreePBX | Register | Issues | Wiki | Portal | Support

Freepbx & Asterisk using far more resources on the Freepbx 14 distro

freepbx
asterisk
Tags: #<Tag:0x00007fd2ef6f9a68> #<Tag:0x00007fd2ef6f94a0>

(ADTopkek) #1

Has anyone else noticed that the Freepbx 14 Distro uses far more CPU than the Freepbx 13 Distro used to use? We upgraded from Distro 13->14 and were having issues so we wiped everything and reinstalled a fresh 14 then restored from a backup. Memory usage seems about the same. Using asterisk 13 (LTS) like we used when we were on the 13 distro .

While the system is actively being used the load averages are just higher than they used to be on Freepbx 13 and the cpu spikes a lot more. When reloading on 14 the load average goes to a 10-12 when it used to be 6-9 on 13. We haven’t added much since the upgrade so it shouldn’t have changed this much from us doing things. This is a pretty active server with 50 calls being norm in the middle of the workday.

Has anyone else experienced this or found out what is going on?


(Andrew Nagy) #2

(ADTopkek) #3

I wasn’t referring to the GUI, the GUI is quite a bit faster in 14. Just watching htop while not touching the GUI at all with the UCP off. Doing a reload from fwconsole reload causes the load average to spike far more than it used to on 13.

Performing reloads is starting to cause a lot of worry due to the spikes.


(Andrew Nagy) #4

The cross referenced post still applies.


(ADTopkek) #5

So we found something that was causing CPU spikes on a schedule. It also seems that it is having a consistent error over and over and over and over:

Bad File Mode

Checking the /etc/cron.d/ folder we had this:
Selection_018

It had this in the file:

# This runs the update check every night at midnight. It sleeps for up to an hour
# as a random delay.
#
# This file was installed by the 'sangoma-pbx' package.
#
0 0 * * * root [ -e /etc/profile.d/z001-updates.sh ] && /etc/profile.d/z001-updates.sh update

Isn’t only the root user suppose to be in the etc/cron.d/ directory?


(Andrew Nagy) #7

Change the ownership of the file if you think that is what’s causing your systemwide issues (I doubt it is)


(Rob Thomas) #8

Whoops, yes, that’s a bug. That REALLY shouldn’t be owned by the asterisk user! Thanks very much for picking that up!


(Rob Thomas) #9

This is now fixed in sangoma-pbx-1802-2, just run ‘yum update’ and it’ll pick it up. (You may need to ‘yum clean metadata’ first, if it doesn’t pick it up as available). Thanks!


(ADTopkek) #10

^ :smiley: found another security bug, thanks for fixing that.

We found a way to make asterisk run better during normal operations and through reloads, although reload still cause issues at times. We used HTOP to decrease the nice setting to -10 on the main asterisk thread and that made it run MUCH better and stopped some of the call quality issues plus bought us time to look into this more.

It seems that the more calls asterisk has processed without restarting the worse it’s performance gets. If we let it get to around 60,000 calls processed we will hear small sound issues and around 100,000 we can notice sound issues on customers that should have no sound issues. Upon an fwconsole restart everything sounds great again.

Reloads also follow a similar pattern and progressively use more resources the more calls that have been processed but it has an odd little pattern that I am not sure what to make of:

We will apply changes or fwconsole reload.
I will watch asterisk be fed the new dial plan.
Load jumps to around 7-10 in HTOP
We then get informed that the reload is done.
Load then spikes to 10-24 on a 12 core server. It progressively rises and hits a very high load average then drops.

I would expect the largest spike to be the asterisk dialplan switch over but a large spike continues on after the dialplan switch over happens. What happens after fwconsole reload says it has completed?


(Andrew Nagy) #11

Rest Apps is restarted.


(ADTopkek) #12

Does anything else happen? I stopped restapps from PM2 and reloaded.

While Fwconsole reload was running load average maxed at 9.46 then once Fwconsole was done it shot up quickly shooting up and went to 17 then held there for about a minute. 36 calls active 89425 calls processed.

Call quality went to shit even when it was at 9.46. Have 12 cores and none of them were maxed according to htop but the main asterisk thread was at 200% cpu despite the CPU cores not reflecting that individually.


High CPU Usage for an idle box
(Tony Lewis) #13

Asterisk will only basically use 1 thread. So having more than 1 core doesn’t help the asterisk side of things.


(ADTopkek) #14

So the main asterisk thread will only use 1 thread. Am I correct in thinking that the child threads of the main thread can then use whatever core the kernal assigns it?