Asterisk 1.8.7.0 uses 100% CPU on CentOS

Hi everybody,

since this morning, my Asterisk server is using the whole CPU. After restarting the daemon, it takes 5-10 minutes to get back to the high cpu load.

I’ve tried disabling the console in the safe_asterisk script, deactivating the aix library in the modules.conf, clearing my cdr records and debugging the asterisk threads without success.
The asterisk server is slowing down all other processes, first the httpd daemon and after half an hour nearly everythign else.

System:

CentOS release 5.7 (Final) 64 Bit Asterisk 1.8.7.0 (also tested 1.8.7.1, same problem) FreePBX 2.9.0.2 (core version) Intel(R) Xeon(R) CPU E31275 @ 3.40GHz (4 cores with HT) MySQLd 5.5.16 16GB RAM

top-Log:

top - 12:17:02 up 2:32, 2 users, load average: 18.40, 17.30, 12.36 Tasks: 210 total, 2 running, 208 sleeping, 0 stopped, 0 zombie Cpu(s): 94.8%us, 2.8%sy, 0.0%ni, 0.5%id, 0.0%wa, 0.1%hi, 1.8%si, 0.0%st Mem: 16345680k total, 4717264k used, 11628416k free, 51204k buffers Swap: 2097144k total, 0k used, 2097144k free, 4016340k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 14822 asterisk 15 0 839m 142m 7856 S 791.1 0.9 107:56.79 asterisk

Asterisk Threads:

F S UID PID PPID LWP C NLWP PRI NI ADDR SZ WCHAN RSS PSR STIME TTY TIME CMD 4 - asterisk 14822 14820 - 99 360 - - - 221393 - 174492 - 11:51 ? 02:47:00 /usr/sbin/asterisk -f -U asterisk -G asterisk -vvvg 4 S asterisk - - 14822 0 - 75 0 - - pipe_w - 3 11:51 - 00:00:00 - 5 S asterisk - - 14824 0 - 75 0 - - inotif - 6 11:51 - 00:00:00 - 1 S asterisk - - 14825 0 - 75 0 - - futex_ - 4 11:51 - 00:00:00 - 1 S asterisk - - 14826 0 - 75 0 - - - - 0 11:51 - 00:00:00 - [...] 1 R asterisk - - 15067 60 - 76 0 - - - - 3 11:52 - 00:19:19 - 1 R asterisk - - 15096 61 - 76 0 - - - - 1 11:53 - 00:19:14 - [...]

There are currently 100 connections running a “music on hold”-app which made no problems for a month. But now, there are some threads getting a high cpu peek, seems to be a deadlock in my opinion.

When I “pstack” these threads, they are just a little different from the other threads working with nearly no cpu load.

Normal thread:

Thread 408 (Thread 0x40194940 (LWP 14825)): #0 0x00000031b480aee9 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000528a13 in tps_processing_function () #2 0x0000000000534c6c in dummy_start () #3 0x00000031b480673d in start_thread () from /lib64/libpthread.so.0 #4 0x00000031b40d44bd in clone () from /lib64/libc.so.6

High load thread:

Thread 371 (Thread 0x40813940 (LWP 15096)): #0 0x00000000004649f9 in __ast_queue_frame () #1 0x0000000000476893 in ast_safe_sleep_conditional () #2 0x00002aaab0a5ab9d in play_moh_exec () from /usr/lib/asterisk/modules/res_musiconhold.so #3 0x00000000004e9fff in pbx_exec () #4 0x00000000004f87eb in pbx_extension_helper () #5 0x00000000004fb6bb in __ast_pbx_run () #6 0x00000000004fce8b in pbx_thread () #7 0x0000000000534c6c in dummy_start () #8 0x00000031b480673d in start_thread () from /lib64/libpthread.so.0 #9 0x00000031b40d44bd in clone () from /lib64/libc.so.6

(Nearly) any comment is welcome :slight_smile:

Thanks in advance,
ping

Hi everybody,

I’ve found a solution.
The current asterisk 1.8.7 binary release is not fully compatible to the new linux kernel 2.6.18-274.7.1.e15 image.

Simply build the new version yourself.
Source: http://www.asterisk.org/downloads
Build instructions: http://www.freepbx.org/forum/freepbx/installation/asterisk-1-8-installation

1.8.7.0 also has a well-known security bug. You should be using 1.8.7.1.