CPU usage 90%+ - URGENT HELP Pls

Hi,

I have a dedicated Quad Intel® Xeon™ CPU 2.40GHz with 1GB RAM.

I have Asterisk + FreePBX installed (FOP is stopped to prevent resource utilization).

FreePBX version: 2.5.1.0

System Uptime: 2 days, 2 hours, 28 minutes
Asterisk Uptime: 24 minutes

My server’s CPU load is peaking 100% frequently (every 2 minutes) on account of resource utilization by “Asterisk”.

See logs below:

top - 11:40:06 up 2 days, 2:25, 1 user, load average: 4.19, 3.50, 3.35
Tasks: 106 total, 5 running, 101 sleeping, 0 stopped, 0 zombie
Cpu(s): 62.9%us, 31.0%sy, 0.0%ni, 2.8%id, 1.1%wa, 0.3%hi, 1.8%si, 0.0%st
Mem: 1035184k total, 968312k used, 66872k free, 29088k buffers
Swap: 2040244k total, 84k used, 2040160k free, 765420k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
30510 asterisk 15 0 38744 16m 5796 S 103 1.6 8:50.32 asterisk
29835 asterisk 17 0 21820 6288 3992 R 4 0.6 0:00.11 dialparties.agi
2691 asterisk 15 0 36680 14m 4264 S 3 1.5 0:14.32 httpd
29837 asterisk 18 0 21180 5696 4100 R 2 0.6 0:00.05 dialparties.agi
29840 asterisk 17 0 20024 4088 3104 R 1 0.4 0:00.03 dialparties.agi
29656 root 15 0 2192 1016 796 R 1 0.1 0:00.03 top
412 root 10 -5 0 0 0 S 0 0.0 0:55.58 kjournald
2393 root 21 0 10396 1160 904 S 0 0.1 0:00.38 automount
1 root 15 0 2060 664 572 S 0 0.1 0:01.20 init
2 root RT -5 0 0 0 S 0 0.0 0:01.35 migration/0

I have 5 users logged into the queue (via EyeBeam) answering calls and about 16 users waiting in the queue.

I cannot see any errors in /var/log/asterisk/full

Pls advice what could be the problem areas and possible resolution.

If you need any specific information, please do let me know.

I await some urgent help pls.

Thx
Vai

based on what you have provided you don’t have a problem.

When dealing with multiple core CPU’s it’s nice to post the use of each core from top instead of the combined (press the 1 key to toggle it) as it will show the load of the highest core, not the real total load.

You also need to provide more info about what you are doing. You runnign a GUI on this box? recording all calls? codec’s in use as Codec conversion can load down a CPU if incoming is one and phones are another. You have Music on hold? using mp3’s? As those require decoding and recoding…

If you are using the FreePBX system status page as the reporting of it hitting 100% then know it lies sometimes… If it queries the box at the same time it fires off a php process (for like displaying the page), it can make the CPU hit 100% for about 1/1000th of second and if the timing is right it will make it look bad.

I have a single dual core box with 2 gig ram, a T1 card and a tdm 400 card, 38 users, and almost always at least one meetme running at any given time and we idle at 1.3% to 3% based on top.

Hi Fskrotzki,

Thx for your detailed reply.

The additional info you have requested is as below:

You runnign a GUI on this box?
YES. FREEPBX WEB BASED GUI FOR MANAGEMENT

recording all calls?
NO RECORDING OF ANY CALLS

codec’s in use
G729 FROM DIGIUM (5 CHANNEL LICENSE)
EYEBEAM IS ALSO USING G729

Music on hold? using mp3’s?
MOH = YES
MP3 = I WILL NEED TO CHECK.
IF MOH IS MP3, I WILL CONVERT IT INTO WAV

I am sure my load is less than yours and i have a quad xeon, with 1 GB ram. I fear something is wrong with my server. it has no reason to peak to 80%+

pls help what else can i do to reduce server loads.

need MOH, so cannot eliminate it.
need g729 so cannot eliminate it.

what else to look for?

Thx to Fskrotzki & everyone

What I meant by GUI is, do you have x loaded and started on the server? If so that is VERY bad.

If you are using the web itnerface from another system there is nothing wrong with that and that’s not a GUI on the server, unless you are usign the browser directly on the server.

X takes up a huge amount of memory and CPU power to run so if it is that could be a part of the load issue.

Past that I’m sure others will chime in and say that they are running on old PIII cpu’s without much issue and that should be true.

You don’t say what OS you are using and those details…

This a hand built system or from a ISO? Is the box being used for anything else? you don’t have it running in a VM do you?

Hi,

Check if you have installed fail2ban and if you are connected directly to internet.
Maybe you’re receiving a hacking attack via SSH and your secure log can tell you about.
If the secure log is too big, fail2ban will raise your cpu to 100% every time it process the log.

If this is not your scenario, you need to explain about your distro and network configuration.

Hope this help you.

Hi,

Below is some more info as requested by fskrotzki & marlonbaez.

  • MOH is a wav file. Not MP3.
  • GUI (Graphical mode) is not insalled.
  • Its a hand built install. Not from ISO.
  • /var/log/secure is not too big. Its about 18MB.
  • I have emptied the log file to be on safe side.
  • OS is CentOS release 5.2 (Final)
  • Its is a dedicated server provided by theplanet.com
  • No VM is installed. theplanet.com gave me a preinstalled CentOS 5.2 on which we installed Asterisk, FreePBX etc.

Since past 1 week, I am facing a new problem.
In between calls, suddenly the there is a silence (we cannot hear the caller and the callers cannot hear us) for about 3-4 seconds and then the call gets back to normal. This is happening almost 1-2 times per call on almost 70% of the calls.
Not sure of this is related to CPU load spikes or its another issue.

Please do let me know what else should I check.

I Need to resolve this issue asap.
Forum, please continue to help me.

Thx
Vai