Problem with high CPU

Hi guys,

We have noticed that the high CPU destroing poor FreePBX. I mean load average sometimes is beyond 3.

Also, we found that the CPU goes crazy when there are lots of active calls. Like there were 50 active calls, which is impossible because in this shift we have like 15 agents.

This is the output:

Channel Context Extension Prio State Application Data CallerID Duration Accountcode PeerAccount BridgeID
Local/[email protected] zulu-mobile-phone-pu 905010 1 Ring Stasis zulu-mobile-call-push-pro 0258111495 00:00:25
Local/[email protected] zulu-mobile-phone-pu 905010 1 Down AppDial (Outgoing Line) 905010 00:00:25
Local/[email protected] from-queue 123 1 Ringing AppQueue (Outgoing Line) 0110120422 00:00:26
Local/[email protected] macro-dial-one s 63 Ring Dial Local/[email protected],t 0258111495 00:00:26
Local/[email protected] zulu-mobile-phone-pu 905007 1 Down AppDial (Outgoing Line) 905007 00:00:25
Local/[email protected] zulu-mobile-phone-pu 905007 1 Ring Stasis zulu-mobile-call-push-pro 0258111495 00:00:25
Local/[email protected] macro-dial-one s 63 Ring Dial Local/[email protected],t 0258111495 00:00:26
Local/[email protected] macro-dial-one s 63 Ring Dial Local/[email protected],t 0258111495 00:00:26
Local/[email protected] from-queue 123 1 Ringing AppQueue (Outgoing Line) 0110120422 00:00:26
Local/[email protected] from-queue 123 1 Ringing AppQueue (Outgoing Line) 0110120422 00:00:26
Local/[email protected] zulu-mobile-phone-pu 905004 1 Ring Stasis zulu-mobile-call-push-pro 0258111495 00:00:25
Local/[email protected] zulu-mobile-phone-pu 905004 1 Down AppDial (Outgoing Line) 905004 00:00:25
Local/[email protected] zulu-desktop-phone 905012 7 Ring Dial PJSIP/905012/sip:l6m631hn 0258111495 00:00:25
Local/[email protected] zulu-desktop-phone 905012 1 Ringing AppDial (Outgoing Line) 905012 00:00:25
Local/[email protected] zulu-mobile-phone-pu 905016 1 Down AppDial (Outgoing Line) 905016 00:00:25
Local/[email protected] zulu-mobile-phone-pu 905016 1 Ring Stasis zulu-mobile-call-push-pro 0258111495 00:00:25
Local/[email protected] zulu-call 905013 3 Ring Dial Local/[email protected] 0258111495 00:00:25
Local/[email protected] zulu-call 5013 1 Ringing AppDial (Outgoing Line) 5013 00:00:25
Local/[email protected] macro-dial-one s 63 Ring Dial Local/[email protected],t 0258111495 00:00:26
Local/[email protected] from-queue 123 1 Ringing AppQueue (Outgoing Line) 0110120422 00:00:26
Local/[email protected] from-queue 123 1 Ringing AppQueue (Outgoing Line) 0110120422 00:00:26
Local/[email protected] zulu-call 905000 3 Ring Dial Local/[email protected] 0258111495 00:00:25
Local/[email protected] macro-dial-one s 63 Ring Dial Local/[email protected],t 0258111495 00:00:26
Local/[email protected] zulu-call 5000 1 Ringing AppDial (Outgoing Line) 5000 00:00:25
PJSIP/902201-000006f zulu-desktop-phone 902201 1 Up AppDial (Outgoing Line) 2201 00:00:08 92902420-6db9-4cd1-9
Local/[email protected] from-queue 123 1 Ringing AppQueue (Outgoing Line) 0110120422 00:00:26
Local/[email protected] macro-dial-one s 63 Ring Dial Local/[email protected],t 0258111495 00:00:26
PJSIP/905012-000006f zulu-from-internal 905012 1 Ringing AppDial (Outgoing Line) 5012 00:00:21
SIP/IN_0110120422-00 ext-queues 123 53 Up Queue 123,tr, 0258111495 00:00:45
Local/[email protected] macro-dial-one s 63 Ring Dial Local/[email protected],t 0258111495 00:00:26
Local/[email protected] from-queue 123 1 Ringing AppQueue (Outgoing Line) 0110120422 00:00:26
PJSIP/902009-000006f macro-dialout-trunk s 35 Up Dial SIP/out_0240043532_cloudi +390240043532 00:00:37 fe0ccfe8-176b-4cac-b
Local/[email protected] zulu-call 5017 1 Ringing AppDial (Outgoing Line) 5017 00:00:25
Local/[email protected] zulu-call 905017 3 Ring Dial Local/[email protected] 0258111495 00:00:25
Local/[email protected] zulu-call 5014 1 Ringing AppDial (Outgoing Line) 5014 00:00:25
Local/[email protected] zulu-call 905014 3 Ring Dial Local/[email protected] 0258111495 00:00:25
Local/[email protected] zulu-call 905008 3 Ring Dial Local/[email protected] 0258111495 00:00:25
Local/[email protected] zulu-mobile-phone-pu 905013 1 Ring Stasis zulu-mobile-call-push-pro 0258111495 00:00:25
Local/[email protected] zulu-call 5008 1 Ringing AppDial (Outgoing Line) 5008 00:00:25
Local/[email protected] zulu-mobile-phone-pu 905013 1 Down AppDial (Outgoing Line) 905013 00:00:25
Local/[email protected] zulu-call 5003 1 Ringing AppDial (Outgoing Line) 5003 00:00:25
Local/[email protected] zulu-call 905003 3 Ring Dial Local/[email protected] 0258111495 00:00:25
Local/[email protected] zulu-mobile-phone-pu 905008 1 Down AppDial (Outgoing Line) 905008 00:00:25
Local/[email protected] zulu-mobile-phone-pu 905008 1 Ring Stasis zulu-mobile-call-push-pro 0258111495 00:00:25
Local/[email protected] zulu-call 5006 1 Ringing AppDial (Outgoing Line) 5006 00:00:25
Local/[email protected] zulu-call 905006 3 Ring Dial Local/[email protected] 0258111495 00:00:25
Local/[email protected] zulu-mobile-phone-pu 905011 1 Ring Stasis zulu-mobile-call-push-pro 0258111495 00:00:25
Local/[email protected] zulu-mobile-phone-pu 905011 1 Down AppDial (Outgoing Line) 905011 00:00:25
Local/[email protected] from-queue 123 1 Ringing AppQueue (Outgoing Line) 0110120422 00:00:26
Local/[email protected] zulu-call 5012 1 Ringing AppDial (Outgoing Line) 5012 00:00:25
Local/[email protected] macro-dial-one s 63 Ring Dial Local/[email protected],t 0258111495 00:00:26
Local/[email protected] from-queue 123 1 Ringing AppQueue (Outgoing Line) 0110120422 00:00:26
Local/[email protected] zulu-call 905012 3 Ring Dial Local/[email protected] 0258111495 00:00:25
Local/[email protected] macro-dial-one s 63 Ring Dial Local/[email protected],t 0258111495 00:00:26
Local/[email protected] from-queue 123 1 Ringing AppQueue (Outgoing Line) 0110120422 00:00:26
Local/[email protected] macro-dial-one s 63 Ring Dial Local/[email protected],t 0258111495 00:00:26
Local/[email protected] zulu-call 905005 3 Ring Dial Local/[email protected] 0258111495 00:00:25
Local/[email protected] zulu-call 5005 1 Ringing AppDial (Outgoing Line) 5005 00:00:25
SIP/out_0240043532_c from-trunk-sip-out_0 1 Up AppDial (Outgoing Line) 00390124425058 00:00:37 fe0ccfe8-176b-4cac-b
Local/[email protected] zulu-call 905016 3 Ring Dial Local/[email protected] 0258111495 00:00:25
Local/[email protected] zulu-call 5016 1 Ringing AppDial (Outgoing Line) 5016 00:00:25
Local/[email protected] zulu-call 5007 1 Ringing AppDial (Outgoing Line) 5007 00:00:25
Local/[email protected] zulu-call 5015 1 Ringing AppDial (Outgoing Line) 5015 00:00:25
Local/[email protected] zulu-call 905007 3 Ring Dial Local/[email protected] 0258111495 00:00:25
Local/[email protected] zulu-call 905015 3 Ring Dial Local/[email protected] 0258111495 00:00:25
Local/[email protected] zulu-mobile-phone-pu 905000 1 Ring Stasis zulu-mobile-call-push-pro 0258111495 00:00:25
Local/[email protected] zulu-mobile-phone-pu 905000 1 Down AppDial (Outgoing Line) 905000 00:00:25
Local/[email protected] from-queue 123 1 Ringing AppQueue (Outgoing Line) 0110120422 00:00:26
Local/[email protected] macro-dial-one s 63 Ring Dial Local/[email protected],t 0258111495 00:00:26
SIP/100819-0000055b ext-queues 304 50 Up Queue 304,t, +34638540231 00:00:24 2124f3e7-71b4-4222-a
Local/[email protected] macro-dial-one s 63 Ring Dial Local/[email protected],t 0258111495 00:00:26
Local/[email protected] from-queue 123 1 Ringing AppQueue (Outgoing Line) 0110120422 00:00:26
Local/[email protected] from-queue 123 1 Ringing AppQueue (Outgoing Line) 0110120422 00:00:26
Local/[email protected] macro-dial-one s 63 Ring Dial Local/[email protected],t 0258111495 00:00:26
Local/[email protected] zulu-mobile-phone-pu 905015 1 Ring Stasis zulu-mobile-call-push-pro 0258111495 00:00:25
Local/[email protected] zulu-mobile-phone-pu 905015 1 Down AppDial (Outgoing Line) 905015 00:00:25
Local/[email protected] zulu-mobile-phone-pu 905005 1 Ring Stasis zulu-mobile-call-push-pro 0258111495 00:00:25
Local/[email protected] zulu-mobile-phone-pu 905005 1 Down AppDial (Outgoing Line) 905005 00:00:25
Local/[email protected] from-queue 123 1 Ringing AppQueue (Outgoing Line) 0110120422 00:00:26
Local/[email protected] macro-dial-one s 63 Ring Dial Local/[email protected],t 0258111495 00:00:26
Local/[email protected] zulu-mobile-phone-pu 905014 1 Ring Stasis zulu-mobile-call-push-pro 0258111495 00:00:25
Local/[email protected] zulu-mobile-phone-pu 905014 1 Down AppDial (Outgoing Line) 905014 00:00:25
Local/[email protected] zulu-mobile-phone-pu 905003 1 Ring Stasis zulu-mobile-call-push-pro 0258111495 00:00:25
Local/[email protected] zulu-mobile-phone-pu 905003 1 Down AppDial (Outgoing Line) 905003 00:00:25
Local/[email protected] zulu-mobile-phone-pu 905006 1 Down AppDial (Outgoing Line) 905006 00:00:25
Local/[email protected] zulu-mobile-phone-pu 905006 1 Ring Stasis zulu-mobile-call-push-pro 0258111495 00:00:25
Local/[email protected] zulu-call 905011 3 Ring Dial Local/[email protected] 0258111495 00:00:25
Local/[email protected] zulu-call 5011 1 Ringing AppDial (Outgoing Line) 5011 00:00:25
Local/[email protected] zulu-call 5004 1 Ringing AppDial (Outgoing Line) 5004 00:00:25
Local/[email protected] zulu-call 905004 3 Ring Dial Local/[email protected] 0258111495 00:00:25
Local/[email protected] from-queue 123 1 Ringing AppQueue (Outgoing Line) 0110120422 00:00:26
Local/[email protected] macro-dial-one s 63 Ring Dial Local/[email protected],t 0258111495 00:00:26
Local/[email protected] zulu-call 905010 3 Ring Dial Local/[email protected] 0258111495 00:00:25
Local/[email protected] zulu-call 5010 1 Ringing AppDial (Outgoing Line) 5010 00:00:25
Local/[email protected] macro-dial-one s 63 Up Dial Local/[email protected],12 +34638540231 00:00:10 92902420-6db9-4cd1-9
Local/[email protected] from-queue 304 1 Up AppQueue (Outgoing Line) 840219 00:00:10 2124f3e7-71b4-4222-a
Local/[email protected] zulu-mobile-phone-pu 905017 1 Down AppDial (Outgoing Line) 905017 00:00:25
Local/[email protected] zulu-mobile-phone-pu 905017 1 Ring Stasis zulu-mobile-call-push-pro 0258111495 00:00:25
Local/[email protected] zulu-mobile-phone-pu 905012 1 Ring Stasis zulu-mobile-call-push-pro 0258111495 00:00:25
Local/[email protected] zulu-mobile-phone-pu 905012 1 Down AppDial (Outgoing Line) 905012 00:00:25
100 active channels
50 active calls

I can’t figure out why he’s behaving like this.

I have Zulu on my mobile also and when I logged in and make some calls it’s regular, there is no “zulu-mobile-call-push-pro”. Does anyone know what that means (zulu-mobile-call-push-pro)?


  • Asterisk 16.29.1

  • [[email protected] ~]# ps -eo pcpu,pid,user,args | sort -k1 -r -n | head -10
    9.3 1427 mysql /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --log-error=/var/log/mariadb/mariadb.log --pid-file=/var/run/mariadb/mariadb.pid --socket=/var/lib/mysql/mysql.sock
    2.6 3751 asterisk /usr/sbin/asterisk -f -U asterisk -G asterisk -vvvg -c
    2.4 7442 asterisk /usr/sbin/httpd -DFOREGROUND
    1.9 7716 asterisk /usr/sbin/httpd -DFOREGROUND
    1.8 7725 asterisk /usr/sbin/httpd -DFOREGROUND
    1.8 7125 asterisk /usr/sbin/httpd -DFOREGROUND
    1.8 5513 asterisk node /var/www/html/admin/modules/zulu/node/index.js
    1.6 9756 asterisk /usr/sbin/httpd -DFOREGROUND
    1.4 8779 asterisk /usr/sbin/httpd -DFOREGROUND
    1.3 19071 root htop

There are no rules, just a sudden boom over 50 calls, that lasts for a while, and calms down in a few minutes.

Thanks in advance for the help :slight_smile:

Also, we noticed that when we call some queues with lots of agents like we have a queue with 18 agents, then active calls skyrocket to 60+, and the CPU goes crazy the more phone ringing.
In all queues, we have static agents configured, when someone calls the queue it will call all agents plus some have Zulu mobile installed, but why this high number of active calls?

How many CPU cores do you have? If the 1 to 6 are cores, your load averages indicate that you are not using all of them.

It looks like this is output from htop, and the top six bars are CPU core usage, in which case the machine was very lightly loaded when this was captured. I’m not sure why the load average is rather high for the CPU usage. Looking at performance - High load average, low CPU usage - why? - Server Fault it could be that the system is disk bound, rather than CPU bound. Maybe there are other resource waits that are included in the load average.

A call waiting in a queue is still using resources, possibly quite a lot if you don’t have music on hold and announcement files in the codec actually being used, and one of those codecs is a vocoder one.

You might want to try “ps -L ax” and check for any stuck D states, indicating kernel or driver bugs.

I believe that you will get a better 40000 foot view with sngrep. Concurrent calls displayed in realtime, with the ability to ‘drill down’

Do you have FastAGI enabled?

Hi @PitzKey yes, it’s enabled.

We noticed that the problem with a high CPU is the queue with a bunch of users. In some queues, we have more than 20 users, and when a call hits the queue we saw lots of active calls, and the CPU goes really high. I mean we can not use ring groups because of the functionality. The only thing that we did to reduce active calls and therefore also decreased CPU, we set “Skip busy agent to Yes + (ringinuse=no)”.
With 6 CPU cores, this is crazy behavior for the queue. We updated all modules and system recently and after a few days it start.

Please provide evidence of high CPU. Your screen shot shows under 3% loading.

it was all 6 cores on 100% for a few seconds, load average was like 12 at some point, but as I said when we put skip busy agent to Yes + (ringinuse=no) now it’s much much better.