High CPU (mysql and httpd)

Sterlingmd17 · May 3, 2022, 7:41pm

Hello, I have noticed recently as I increase my extension count in my system my CPU has steadily increased. Now I am hitting 100% cpu throughout the day quite a bit causing call quality issues.
We have it virtualized in vmware:
2 cores with reserved cpu and high latency sensitivity setting to reduce waiting for resources
8gb memory
100gb hard drive
The hard drive space teeters around 65% to 70% full.

I do get around 40-50 concurrent calls often.
It doesn’t seem like asterisk taking up the cpu. It seems more like httpd and mysql are the culprits.

Here is one example of me catching it with mysql and httpd going crazy.

I do run clearlyIP devices and I see httpd logs talking alot about clearlyip contact manager but it doesn’t seem to corelate with the high cpu when I watch the logs and CPU usage.

Any tips?

comtech · May 3, 2022, 7:49pm

if you ‘service httpd restart’ does the high usage persist? For mySQL see what is being written during the spikes and backtrack from there.

dicko · May 3, 2022, 8:17pm

mysql -e "show status where variable_name = 'threads_connected'"
mysql -e 'show processlist'

Sterlingmd17 · May 3, 2022, 8:40pm

Thank you Dicko

**
sudo mysql -e "show status where variable_name = 'threads_connected'"
+-------------------+-------+
| Variable_name     | Value |
+-------------------+-------+
| Threads_connected | 30    |
+-------------------+-------+
sudo mysql -e 'show processlist'
+-------+-------------+-----------------+---------------+---------+-------+-------+------------------+----------+
| Id    | User        | Host            | db            | Command | Time  | State | Info             | Progress |
+-------+-------------+-----------------+---------------+---------+-------+-------+------------------+----------+
|    19 | freepbxuser | localhost       | asterisk      | Sleep   |     4 |       | NULL             |    0.000 |
|    33 | freepbxuser | localhost       | asterisk      | Sleep   |    29 |       | NULL             |    0.000 |
|    52 | freepbxuser | localhost:34804 | asterisk      | Sleep   | 12732 |       | NULL             |    0.000 |
|   232 | freepbxuser | localhost       | asteriskcdrdb | Sleep   |     3 |       | NULL             |    0.000 |
|   241 | freepbxuser | localhost       | asterisk      | Sleep   |   205 |       | NULL             |    0.000 |
|  2464 | freepbxuser | localhost       | asteriskcdrdb | Sleep   |     0 |       | NULL             |    0.000 |
| 24256 | freepbxuser | localhost       | asterisk      | Sleep   |   239 |       | NULL             |    0.000 |
| 24489 | freepbxuser | localhost       | asterisk      | Sleep   |     6 |       | NULL             |    0.000 |
| 24490 | freepbxuser | localhost       | asterisk      | Sleep   |     4 |       | NULL             |    0.000 |
| 24491 | freepbxuser | localhost       | asterisk      | Sleep   |     4 |       | NULL             |    0.000 |
| 24492 | freepbxuser | localhost       | asterisk      | Sleep   |     3 |       | NULL             |    0.000 |
| 24494 | freepbxuser | localhost       | asterisk      | Sleep   |     2 |       | NULL             |    0.000 |
| 24495 | freepbxuser | localhost       | asterisk      | Sleep   |     2 |       | NULL             |    0.000 |
| 24497 | freepbxuser | localhost       | asterisk      | Sleep   |     0 |       | NULL             |    0.000 |
| 24498 | freepbxuser | localhost       | asterisk      | Sleep   |     0 |       | NULL             |    0.000 |
| 24499 | freepbxuser | localhost       | asterisk      | Sleep   |     0 |       | NULL             |    0.000 |
| 24501 | freepbxuser | localhost       | asterisk      | Sleep   |     0 |       | NULL             |    0.000 |
| 24502 | freepbxuser | localhost       | asterisk      | Sleep   |     1 |       | NULL             |    0.000 |
| 24503 | freepbxuser | localhost       | asterisk      | Sleep   |     1 |       | NULL             |    0.000 |
| 24504 | freepbxuser | localhost       | asterisk      | Sleep   |     0 |       | NULL             |    0.000 |
| 24509 | root        | localhost       | NULL          | Query   |     0 | NULL  | show processlist |    0.000
**

@Comtech the service httpd restart didn’t resolve anything. I should have mentioned I have since done a reboot with no resolution either. I did also recently upgrade to v16 from v15 but there’s not been a real difference.

Sterlingmd17 · May 3, 2022, 8:48pm

That “threads_connected” variable seems to settle back down to 8 most of the time. I think I happened to catch it at 30.

dicko · May 3, 2022, 9:01pm

How active are your UCP users?

Sterlingmd17 · May 3, 2022, 9:07pm

I suspect not very active only few logins over past few days. Its definetely not heavily used in my environment.

sudo tail /var/log/asterisk/freepbx_security.log
[2022-04-28 13:53:08] [freepbx_security.NOTICE]: Authentication successful for tghy.fax.1.businessoffice from 10.XXXXX [] []
[2022-04-28 21:11:53] [freepbx_security.NOTICE]: Authentication successful for rtkfax from 10XXXX [] []
[2022-04-29 09:37:31] [freepbx_security.NOTICE]: Authentication successful for 10530 from 10.XXXX [] []
[2022-04-29 11:51:29] [freepbx_security.NOTICE]: Authentication successful for tghy.fax.1.businessoffice from 10.XXX [] []
[2022-04-29 13:37:07] [freepbx_security.NOTICE]: Authentication successful for 10530 from 10.XXXX [] []
[2022-05-01 14:43:15] [freepbx_security.NOTICE]: Authentication failure for [email protected] from 172.XXX [] []
[2022-05-02 10:22:37] [freepbx_security.NOTICE]: Authentication successful for 10672 from 10.18.x.x [] []
[2022-05-02 10:56:54] [freepbx_security.NOTICE]: Authentication successful for rtkfax from 10.15.x.x [] []
[2022-05-02 11:12:56] [freepbx_security.NOTICE]: Authentication successful for 10530 from 10.22.x.x [] []
[2022-05-02 12:20:29] [freepbx_security.NOTICE]: Authentication successful for 10530 from 10.22.x.x [] []

Sterlingmd17 · May 3, 2022, 11:36pm

Would a lot of ringall groups be especially taxing on CPU? or was that fixed with the fastagi implementation?

Sterlingmd17 · May 4, 2022, 12:39am

Would 3128 hints from ‘core show hints’ be alot?

jfinstrom · May 4, 2022, 2:26am

Ringing a bunch of phones will be a lot
Hints will depend more on the subscribers than the actual hints. If you have a bunch of phones subscribed it will use up quite a bit.

adell4444 · May 4, 2022, 3:07pm

One of our customers has 20 people in a ring group and everyone has a BLF for everyone, which my math tells me is generating 400 hint changes every time a call comes in and then every time that same call stops ringing. This spikes CPU on the server, but only very briefly and would not cause extended CPU usage.

Also this would appear as asterisk taking the CPU, not httpd or mysql

Sterlingmd17 · May 4, 2022, 11:50pm

I did discover the source of the issue and was able to resolve it. The source was my clearlyIP phones in my environment. For some reason every call that came in the phones would reach out to FreePBX looking for a contact image, this would result in a mysql query for a photo causing apache to spawn a ‘httpd’ process for every extension involved being called. I had clearlyip support help out and we simply removed my server FQDN from the contact image lookup URL parameter on the phones and it stopped the lookups from happening. Now the server is happy as a clam.

Thank you for all your advice though my friends. I appreciate it!

system · May 11, 2022, 11:51pm

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.