Tracking down 100% CPU usage

Distro Discussion & Help
1

We are running FreePBX 12.0.51 on SHMZ release 6.5 and have been running up against asterisk spiking to 100% and staying there until I restart asterisk.

I tracked down the thread:

F S UID        PID  PPID   LWP  C NLWP PRI  NI ADDR SZ WCHAN    RSS PSR STIME TTY          TIME CMD
4 - asterisk 11647 11644     - 17   39   -   - - 18714 -      26700   - 07:32 pts/1    00:32:22 /usr/sbin/asterisk -f -U asterisk -G asterisk -g -vvvg -c
4 S asterisk     -     - 11647  0    -  80   0 -     - -          -   0 07:32 -        00:00:00 -
1 S asterisk     -     - 11648  0    -  80   0 -     - -          -   0 07:32 -        00:00:00 -
1 S asterisk     -     - 11649  0    -  80   0 -     - -          -   0 07:32 -        00:00:00 -
1 S asterisk     -     - 11650  0    -  80   0 -     - -          -   0 07:32 -        00:00:00 -
1 S asterisk     -     - 11651  0    -  80   0 -     - -          -   0 07:32 -        00:00:00 -
1 S asterisk     -     - 11652  0    -  80   0 -     - -          -   0 07:32 -        00:00:00 -
1 S asterisk     -     - 11653  0    -  80   0 -     - -          -   0 07:32 -        00:00:00 -
1 S asterisk     -     - 11654  0    -  80   0 -     - -          -   0 07:32 -        00:00:00 -
1 S asterisk     -     - 11655  0    -  80   0 -     - -          -   0 07:32 -        00:00:00 -
1 S asterisk     -     - 11656  0    -  80   0 -     - -          -   0 07:32 -        00:00:00 -
1 S asterisk     -     - 11657  0    -  80   0 -     - -          -   0 07:32 -        00:00:00 -
1 S asterisk     -     - 11658  0    -  80   0 -     - -          -   0 07:32 -        00:00:00 -
1 R asterisk     -     - 11659 16    -  80   0 -     - -          -   0 07:32 -        00:31:06 -
1 S asterisk     -     - 11662  0    -  80   0 -     - -          -   0 07:32 -        00:00:00 -
1 S asterisk     -     - 11663  0    -  80   0 -     - -          -   0 07:32 -        00:00:00 -
1 S asterisk     -     - 11664  0    -  80   0 -     - -          -   0 07:32 -        00:00:00 -
1 S asterisk     -     - 11665  0    -  80   0 -     - -          -   0 07:32 -        00:00:00 -
1 S asterisk     -     - 11666  0    -  80   0 -     - -          -   0 07:32 -        00:00:00 -
1 S asterisk     -     - 11667  0    -  80   0 -     - -          -   0 07:32 -        00:00:00 -
1 S asterisk     -     - 11668  0    -  80   0 -     - -          -   0 07:32 -        00:00:00 -
1 S asterisk     -     - 11669  0    -  80   0 -     - -          -   0 07:32 -        00:00:00 -
1 S asterisk     -     - 11670  0    -  80   0 -     - -          -   0 07:32 -        00:00:00 -
1 S asterisk     -     - 11671  0    -  80   0 -     - -          -   0 07:32 -        00:00:00 -
1 S asterisk     -     - 11672  0    -  80   0 -     - -          -   0 07:32 -        00:00:00 -
1 S asterisk     -     - 11673  0    -  80   0 -     - -          -   0 07:32 -        00:00:00 -
1 S asterisk     -     - 11674  0    -  80   0 -     - -          -   0 07:32 -        00:00:00 -
1 S asterisk     -     - 11675  0    -  80   0 -     - -          -   0 07:32 -        00:00:38 -
1 S asterisk     -     - 11676  0    -  80   0 -     - -          -   0 07:32 -        00:00:06 -
1 S asterisk     -     - 11677  0    -  80   0 -     - -          -   0 07:32 -        00:00:00 -
1 S asterisk     -     - 11678  0    -  80   0 -     - -          -   0 07:32 -        00:00:00 -
1 S asterisk     -     - 11679  0    -  80   0 -     - -          -   0 07:32 -        00:00:00 -
1 S asterisk     -     - 11680  0    -  80   0 -     - -          -   0 07:32 -        00:00:00 -
1 S asterisk     -     - 11681  0    -  80   0 -     - -          -   0 07:32 -        00:00:00 -
1 S asterisk     -     - 11682  0    -  80   0 -     - -          -   0 07:32 -        00:00:00 -
1 S asterisk     -     - 11683  0    -  80   0 -     - -          -   0 07:32 -        00:00:00 -
1 S asterisk     -     - 11684  0    -  80   0 -     - -          -   0 07:32 -        00:00:00 -
1 S asterisk     -     - 11685  0    -  80   0 -     - -          -   0 07:32 -        00:00:00 -
1 S asterisk     -     - 14013  0    -  80   0 -     - -          -   0 09:01 -        00:00:00 -
1 S asterisk     -     - 16488  1    -  80   0 -     - -          -   0 10:36 -        00:00:00 -

And dumped the thread:

Thread 28 (Thread 0xb72c3b70 (LWP 11659)):
#0  0x00ad2416 in __kernel_vsyscall ()
#1  0x00398f56 in gettimeofday () from /lib/libc.so.6
#2  0x0819a271 in ast_tvnow ()
#3  0x08182565 in ast_sched_runq ()
#4  0x08181059 in ?? ()
#5  0x0819b6bd in ?? ()
#6  0x00539b39 in start_thread () from /lib/libpthread.so.0
#7  0x003e9c2e in clone () from /lib/libc.so.6

And again in another dump:

Thread 27 (Thread 0xb72c3b70 (LWP 11659)):
#0  0x00ad2416 in __kernel_vsyscall ()
#1  0x0053d6d3 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2  0x003f98b4 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libc.so.6
#3  0x0811df2f in __ast_cond_timedwait ()
#4  0x0818100b in ?? ()
#5  0x0819b6bd in ?? ()
#6  0x00539b39 in start_thread () from /lib/libpthread.so.0
#7  0x003e9c2e in clone () from /lib/libc.so.6

But I cant see to find anything from those.

If anyone has any insight I would appreciate it.

2

Another update, I have this inking that it has to do with the timezone setting on the server.

I went into the System Admin and checked the Time Zone setting. Once I verified the setting I hit submit just for reassurance, and BAM the whole server crashes. We use FreePBX hosting through Cyberlynk.net

3

It seems an amportal restart(restarting asterisk itself doesnt, for long) takes care of it, but it pops up again from time to time

4

have you verified that the date/time are correct and that php.ini that the proper time zone set?