Problem with high CPU

Fule_slippers · January 17, 2023, 2:29pm

Hi guys,

We have noticed that the high CPU destroing poor FreePBX. I mean load average sometimes is beyond 3.

Also, we found that the CPU goes crazy when there are lots of active calls. Like there were 50 active calls, which is impossible because in this shift we have like 15 agents.

This is the output:

Channel Context Extension Prio State Application Data CallerID Duration Accountcode PeerAccount BridgeID
Local/905010@zulu-mo zulu-mobile-phone-pu 905010 1 Ring Stasis zulu-mobile-call-push-pro 0258111495 00:00:25
Local/905010@zulu-mo zulu-mobile-phone-pu 905010 1 Down AppDial (Outgoing Line) 905010 00:00:25
Local/5004@from-queu from-queue 123 1 Ringing AppQueue (Outgoing Line) 0110120422 00:00:26
Local/5004@from-queu macro-dial-one s 63 Ring Dial Local/905004@zulu-call,t 0258111495 00:00:26
Local/905007@zulu-mo zulu-mobile-phone-pu 905007 1 Down AppDial (Outgoing Line) 905007 00:00:25
Local/905007@zulu-mo zulu-mobile-phone-pu 905007 1 Ring Stasis zulu-mobile-call-push-pro 0258111495 00:00:25
Local/5008@from-queu macro-dial-one s 63 Ring Dial Local/905008@zulu-call,t 0258111495 00:00:26
Local/5011@from-queu macro-dial-one s 63 Ring Dial Local/905011@zulu-call,t 0258111495 00:00:26
Local/5008@from-queu from-queue 123 1 Ringing AppQueue (Outgoing Line) 0110120422 00:00:26
Local/5011@from-queu from-queue 123 1 Ringing AppQueue (Outgoing Line) 0110120422 00:00:26
Local/905004@zulu-mo zulu-mobile-phone-pu 905004 1 Ring Stasis zulu-mobile-call-push-pro 0258111495 00:00:25
Local/905004@zulu-mo zulu-mobile-phone-pu 905004 1 Down AppDial (Outgoing Line) 905004 00:00:25
Local/905012@zulu-de zulu-desktop-phone 905012 7 Ring Dial PJSIP/905012/sip:l6m631hn 0258111495 00:00:25
Local/905012@zulu-de zulu-desktop-phone 905012 1 Ringing AppDial (Outgoing Line) 905012 00:00:25
Local/905016@zulu-mo zulu-mobile-phone-pu 905016 1 Down AppDial (Outgoing Line) 905016 00:00:25
Local/905016@zulu-mo zulu-mobile-phone-pu 905016 1 Ring Stasis zulu-mobile-call-push-pro 0258111495 00:00:25
Local/905013@zulu-ca zulu-call 905013 3 Ring Dial Local/905013@zulu-desktop 0258111495 00:00:25
Local/905013@zulu-ca zulu-call 5013 1 Ringing AppDial (Outgoing Line) 5013 00:00:25
Local/5005@from-queu macro-dial-one s 63 Ring Dial Local/905005@zulu-call,t 0258111495 00:00:26
Local/5013@from-queu from-queue 123 1 Ringing AppQueue (Outgoing Line) 0110120422 00:00:26
Local/5005@from-queu from-queue 123 1 Ringing AppQueue (Outgoing Line) 0110120422 00:00:26
Local/905000@zulu-ca zulu-call 905000 3 Ring Dial Local/905000@zulu-desktop 0258111495 00:00:25
Local/5013@from-queu macro-dial-one s 63 Ring Dial Local/905013@zulu-call,t 0258111495 00:00:26
Local/905000@zulu-ca zulu-call 5000 1 Ringing AppDial (Outgoing Line) 5000 00:00:25
PJSIP/902201-000006f zulu-desktop-phone 902201 1 Up AppDial (Outgoing Line) 2201 00:00:08 92902420-6db9-4cd1-9
Local/5006@from-queu from-queue 123 1 Ringing AppQueue (Outgoing Line) 0110120422 00:00:26
Local/5006@from-queu macro-dial-one s 63 Ring Dial Local/905006@zulu-call,t 0258111495 00:00:26
PJSIP/905012-000006f zulu-from-internal 905012 1 Ringing AppDial (Outgoing Line) 5012 00:00:21
SIP/IN_0110120422-00 ext-queues 123 53 Up Queue 123,tr, 0258111495 00:00:45
Local/5017@from-queu macro-dial-one s 63 Ring Dial Local/905017@zulu-call,t 0258111495 00:00:26
Local/5017@from-queu from-queue 123 1 Ringing AppQueue (Outgoing Line) 0110120422 00:00:26
PJSIP/902009-000006f macro-dialout-trunk s 35 Up Dial SIP/out_0240043532_cloudi +390240043532 00:00:37 fe0ccfe8-176b-4cac-b
Local/905017@zulu-ca zulu-call 5017 1 Ringing AppDial (Outgoing Line) 5017 00:00:25
Local/905017@zulu-ca zulu-call 905017 3 Ring Dial Local/905017@zulu-desktop 0258111495 00:00:25
Local/905014@zulu-ca zulu-call 5014 1 Ringing AppDial (Outgoing Line) 5014 00:00:25
Local/905014@zulu-ca zulu-call 905014 3 Ring Dial Local/905014@zulu-desktop 0258111495 00:00:25
Local/905008@zulu-ca zulu-call 905008 3 Ring Dial Local/905008@zulu-desktop 0258111495 00:00:25
Local/905013@zulu-mo zulu-mobile-phone-pu 905013 1 Ring Stasis zulu-mobile-call-push-pro 0258111495 00:00:25
Local/905008@zulu-ca zulu-call 5008 1 Ringing AppDial (Outgoing Line) 5008 00:00:25
Local/905013@zulu-mo zulu-mobile-phone-pu 905013 1 Down AppDial (Outgoing Line) 905013 00:00:25
Local/905003@zulu-ca zulu-call 5003 1 Ringing AppDial (Outgoing Line) 5003 00:00:25
Local/905003@zulu-ca zulu-call 905003 3 Ring Dial Local/905003@zulu-desktop 0258111495 00:00:25
Local/905008@zulu-mo zulu-mobile-phone-pu 905008 1 Down AppDial (Outgoing Line) 905008 00:00:25
Local/905008@zulu-mo zulu-mobile-phone-pu 905008 1 Ring Stasis zulu-mobile-call-push-pro 0258111495 00:00:25
Local/905006@zulu-ca zulu-call 5006 1 Ringing AppDial (Outgoing Line) 5006 00:00:25
Local/905006@zulu-ca zulu-call 905006 3 Ring Dial Local/905006@zulu-desktop 0258111495 00:00:25
Local/905011@zulu-mo zulu-mobile-phone-pu 905011 1 Ring Stasis zulu-mobile-call-push-pro 0258111495 00:00:25
Local/905011@zulu-mo zulu-mobile-phone-pu 905011 1 Down AppDial (Outgoing Line) 905011 00:00:25
Local/5010@from-queu from-queue 123 1 Ringing AppQueue (Outgoing Line) 0110120422 00:00:26
Local/905012@zulu-ca zulu-call 5012 1 Ringing AppDial (Outgoing Line) 5012 00:00:25
Local/5010@from-queu macro-dial-one s 63 Ring Dial Local/905010@zulu-call,t 0258111495 00:00:26
Local/5007@from-queu from-queue 123 1 Ringing AppQueue (Outgoing Line) 0110120422 00:00:26
Local/905012@zulu-ca zulu-call 905012 3 Ring Dial Local/905012@zulu-desktop 0258111495 00:00:25
Local/5007@from-queu macro-dial-one s 63 Ring Dial Local/905007@zulu-call,t 0258111495 00:00:26
Local/5003@from-queu from-queue 123 1 Ringing AppQueue (Outgoing Line) 0110120422 00:00:26
Local/5003@from-queu macro-dial-one s 63 Ring Dial Local/905003@zulu-call,t 0258111495 00:00:26
Local/905005@zulu-ca zulu-call 905005 3 Ring Dial Local/905005@zulu-desktop 0258111495 00:00:25
Local/905005@zulu-ca zulu-call 5005 1 Ringing AppDial (Outgoing Line) 5005 00:00:25
SIP/out_0240043532_c from-trunk-sip-out_0 1 Up AppDial (Outgoing Line) 00390124425058 00:00:37 fe0ccfe8-176b-4cac-b
Local/905016@zulu-ca zulu-call 905016 3 Ring Dial Local/905016@zulu-desktop 0258111495 00:00:25
Local/905016@zulu-ca zulu-call 5016 1 Ringing AppDial (Outgoing Line) 5016 00:00:25
Local/905007@zulu-ca zulu-call 5007 1 Ringing AppDial (Outgoing Line) 5007 00:00:25
Local/905015@zulu-ca zulu-call 5015 1 Ringing AppDial (Outgoing Line) 5015 00:00:25
Local/905007@zulu-ca zulu-call 905007 3 Ring Dial Local/905007@zulu-desktop 0258111495 00:00:25
Local/905015@zulu-ca zulu-call 905015 3 Ring Dial Local/905015@zulu-desktop 0258111495 00:00:25
Local/905000@zulu-mo zulu-mobile-phone-pu 905000 1 Ring Stasis zulu-mobile-call-push-pro 0258111495 00:00:25
Local/905000@zulu-mo zulu-mobile-phone-pu 905000 1 Down AppDial (Outgoing Line) 905000 00:00:25
Local/5012@from-queu from-queue 123 1 Ringing AppQueue (Outgoing Line) 0110120422 00:00:26
Local/5012@from-queu macro-dial-one s 63 Ring Dial Local/905012@zulu-call,t 0258111495 00:00:26
SIP/100819-0000055b ext-queues 304 50 Up Queue 304,t, +34638540231 00:00:24 2124f3e7-71b4-4222-a
Local/5014@from-queu macro-dial-one s 63 Ring Dial Local/905014@zulu-call,t 0258111495 00:00:26
Local/5014@from-queu from-queue 123 1 Ringing AppQueue (Outgoing Line) 0110120422 00:00:26
Local/5015@from-queu from-queue 123 1 Ringing AppQueue (Outgoing Line) 0110120422 00:00:26
Local/5015@from-queu macro-dial-one s 63 Ring Dial Local/905015@zulu-call,t 0258111495 00:00:26
Local/905015@zulu-mo zulu-mobile-phone-pu 905015 1 Ring Stasis zulu-mobile-call-push-pro 0258111495 00:00:25
Local/905015@zulu-mo zulu-mobile-phone-pu 905015 1 Down AppDial (Outgoing Line) 905015 00:00:25
Local/905005@zulu-mo zulu-mobile-phone-pu 905005 1 Ring Stasis zulu-mobile-call-push-pro 0258111495 00:00:25
Local/905005@zulu-mo zulu-mobile-phone-pu 905005 1 Down AppDial (Outgoing Line) 905005 00:00:25
Local/5000@from-queu from-queue 123 1 Ringing AppQueue (Outgoing Line) 0110120422 00:00:26
Local/5000@from-queu macro-dial-one s 63 Ring Dial Local/905000@zulu-call,t 0258111495 00:00:26
Local/905014@zulu-mo zulu-mobile-phone-pu 905014 1 Ring Stasis zulu-mobile-call-push-pro 0258111495 00:00:25
Local/905014@zulu-mo zulu-mobile-phone-pu 905014 1 Down AppDial (Outgoing Line) 905014 00:00:25
Local/905003@zulu-mo zulu-mobile-phone-pu 905003 1 Ring Stasis zulu-mobile-call-push-pro 0258111495 00:00:25
Local/905003@zulu-mo zulu-mobile-phone-pu 905003 1 Down AppDial (Outgoing Line) 905003 00:00:25
Local/905006@zulu-mo zulu-mobile-phone-pu 905006 1 Down AppDial (Outgoing Line) 905006 00:00:25
Local/905006@zulu-mo zulu-mobile-phone-pu 905006 1 Ring Stasis zulu-mobile-call-push-pro 0258111495 00:00:25
Local/905011@zulu-ca zulu-call 905011 3 Ring Dial Local/905011@zulu-desktop 0258111495 00:00:25
Local/905011@zulu-ca zulu-call 5011 1 Ringing AppDial (Outgoing Line) 5011 00:00:25
Local/905004@zulu-ca zulu-call 5004 1 Ringing AppDial (Outgoing Line) 5004 00:00:25
Local/905004@zulu-ca zulu-call 905004 3 Ring Dial Local/905004@zulu-desktop 0258111495 00:00:25
Local/5016@from-queu from-queue 123 1 Ringing AppQueue (Outgoing Line) 0110120422 00:00:26
Local/5016@from-queu macro-dial-one s 63 Ring Dial Local/905016@zulu-call,t 0258111495 00:00:26
Local/905010@zulu-ca zulu-call 905010 3 Ring Dial Local/905010@zulu-desktop 0258111495 00:00:25
Local/905010@zulu-ca zulu-call 5010 1 Ringing AppDial (Outgoing Line) 5010 00:00:25
Local/2201@from-queu macro-dial-one s 63 Up Dial Local/902201@zulu-call,12 +34638540231 00:00:10 92902420-6db9-4cd1-9
Local/2201@from-queu from-queue 304 1 Up AppQueue (Outgoing Line) 840219 00:00:10 2124f3e7-71b4-4222-a
Local/905017@zulu-mo zulu-mobile-phone-pu 905017 1 Down AppDial (Outgoing Line) 905017 00:00:25
Local/905017@zulu-mo zulu-mobile-phone-pu 905017 1 Ring Stasis zulu-mobile-call-push-pro 0258111495 00:00:25
Local/905012@zulu-mo zulu-mobile-phone-pu 905012 1 Ring Stasis zulu-mobile-call-push-pro 0258111495 00:00:25
Local/905012@zulu-mo zulu-mobile-phone-pu 905012 1 Down AppDial (Outgoing Line) 905012 00:00:25
100 active channels
50 active calls

I can’t figure out why he’s behaving like this.

I have Zulu on my mobile also and when I logged in and make some calls it’s regular, there is no “zulu-mobile-call-push-pro”. Does anyone know what that means (zulu-mobile-call-push-pro)?

Asterisk 16.29.1
[root@voip ~]# ps -eo pcpu,pid,user,args | sort -k1 -r -n | head -10
9.3 1427 mysql /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --log-error=/var/log/mariadb/mariadb.log --pid-file=/var/run/mariadb/mariadb.pid --socket=/var/lib/mysql/mysql.sock
2.6 3751 asterisk /usr/sbin/asterisk -f -U asterisk -G asterisk -vvvg -c
2.4 7442 asterisk /usr/sbin/httpd -DFOREGROUND
1.9 7716 asterisk /usr/sbin/httpd -DFOREGROUND
1.8 7725 asterisk /usr/sbin/httpd -DFOREGROUND
1.8 7125 asterisk /usr/sbin/httpd -DFOREGROUND
1.8 5513 asterisk node /var/www/html/admin/modules/zulu/node/index.js
1.6 9756 asterisk /usr/sbin/httpd -DFOREGROUND
1.4 8779 asterisk /usr/sbin/httpd -DFOREGROUND
1.3 19071 root htop

There are no rules, just a sudden boom over 50 calls, that lasts for a while, and calms down in a few minutes.

Thanks in advance for the help

Fule_slippers · January 17, 2023, 3:01pm

Also, we noticed that when we call some queues with lots of agents like we have a queue with 18 agents, then active calls skyrocket to 60+, and the CPU goes crazy the more phone ringing.
In all queues, we have static agents configured, when someone calls the queue it will call all agents plus some have Zulu mobile installed, but why this high number of active calls?

david55 · January 17, 2023, 3:13pm

How many CPU cores do you have? If the 1 to 6 are cores, your load averages indicate that you are not using all of them.

It looks like this is output from htop, and the top six bars are CPU core usage, in which case the machine was very lightly loaded when this was captured. I’m not sure why the load average is rather high for the CPU usage. Looking at performance - High load average, low CPU usage - why? - Server Fault it could be that the system is disk bound, rather than CPU bound. Maybe there are other resource waits that are included in the load average.

A call waiting in a queue is still using resources, possibly quite a lot if you don’t have music on hold and announcement files in the codec actually being used, and one of those codecs is a vocoder one.

You might want to try “ps -L ax” and check for any stuck D states, indicating kernel or driver bugs.

dicko · January 17, 2023, 3:15pm

I believe that you will get a better 40000 foot view with sngrep. Concurrent calls displayed in realtime, with the ability to ‘drill down’

PitzKey · January 17, 2023, 5:57pm

Do you have FastAGI enabled?

Fule_slippers · January 18, 2023, 9:07am

Hi @PitzKey yes, it’s enabled.

We noticed that the problem with a high CPU is the queue with a bunch of users. In some queues, we have more than 20 users, and when a call hits the queue we saw lots of active calls, and the CPU goes really high. I mean we can not use ring groups because of the functionality. The only thing that we did to reduce active calls and therefore also decreased CPU, we set “Skip busy agent to Yes + (ringinuse=no)”.
With 6 CPU cores, this is crazy behavior for the queue. We updated all modules and system recently and after a few days it start.

david55 · January 18, 2023, 2:10pm

Please provide evidence of high CPU. Your screen shot shows under 3% loading.

Fule_slippers · January 18, 2023, 6:29pm

it was all 6 cores on 100% for a few seconds, load average was like 12 at some point, but as I said when we put skip busy agent to Yes + (ringinuse=no) now it’s much much better.

system · February 17, 2023, 6:30pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.