Asterisk crashing -100% CPU

I have been trying to troubleshoot a problem with the AlertInfo on recalled parked calls and found a big problem.

When I park a call, and wait for it to time out and ring back, as soon as it does this asterisk seems to get stuck in some sort of loop that maxes out the processor until I force a reboot. Does it every time.

This is a fully up to date PIAF install running only what it came with.

Here is what the cli has to say:

Code:
stack
– Executing [[email protected]:6] Set(“SIP/2065690119-088788c8”, “__BLKVM_BASE=600”) in new stack
– Executing [[email protected]:7] Set(“SIP/2065690119-088788c8”, “DB(BLKVM/600/SIP/2065690119-088788c8)=TRUE”) in new stack
– Executing [[email protected]:8] Set(“SIP/2065690119-088788c8”, “RRNODEST=”) in new stack
– Executing [[email protected]:9] Set(“SIP/2065690119-088788c8”, “__NODEST=600”) in new stack
– Executing [[email protected]:10] Set(“SIP/2065690119-088788c8”, “__ALERT_INFO=External”) in new stack
– Executing [600[email protected]:11] Set(“SIP/2065690119-088788c8”, “RecordMethod=Group”) in new stack
– Executing [[email protected]:12] Macro(“SIP/2065690119-088788c8”, “record-enable|101-102-103-104-106-100-110|Group”) in new stack
– Executing [[email protected]:1] GotoIf(“SIP/2065690119-088788c8”, “1?check”) in new stack
– Goto (macro-record-enable,s,4)
– Executing [[email protected]:13] Set(“SIP/2065690119-088788c8”, “RingGroupMethod=ringall”) in new stack
– Executing [[email protected]:14] Macro(“SIP/2065690119-088788c8”, “dial|60|trT|101-102-103-104-106-100-110”) in new stack
– Executing [[email protected]:1] GotoIf(“SIP/2065690119-088788c8”, “0?dial”) in new stack
– Executing [[email protected]:15] Set(“SIP/2065690119-088788c8”, “RingGroupMethod=”) in new stack
– Executing [[email protected]:16] GotoIf(“SIP/2065690119-088788c8”, “0?nodest”) in new stack
– Executing [[email protected]:17] Set(“SIP/2065690119-088788c8”, “__NODEST=”) in new stack
– Executing [[email protected]:18] DBdel(“SIP/2065690119-088788c8”, “BLKVM/600/SIP/2065690119-088788c8”) in new stack
– DBdel: family=BLKVM, key=600/SIP/2065690119-088788c8
– Executing [[email protected]:19] Goto(“SIP/2065690119-088788c8”, “ext-group|600|1”) in new stack
– Goto (ext-group,600,1)
– Executing [[email protected]:1] Macro(“SIP/2065690119-088788c8”, “user-callerid|”) in new stack
– Executing [[email protected]:1] Set(“SIP/2065690119-088788c8”, “AMPUSER=2065690119”) in new stack
– Executing [[email protected]:2] GotoIf(“SIP/2065690119-088788c8”, “0?skipdb”) in new stack
– Executing [[email protected]:3] GotoIf(“SIP/2065690119-088788c8”, “0?skipov”) in new stack
– Executing [[email protected]:4] Set(“SIP/2065690119-088788c8”, “__NODEST=”) in new stack
– Executing [[email protected]:5] Set(“SIP/2065690119-088788c8”, “__BLKVM_OVERRIDE=BLKVM/600/SIP/2065690119-088788c8”) in new stack

Please see http://freepbx.org/forum/freepbx/installation/so-you-have-a-problem-and-want-help

To start we don’t know when you installed it which versions of things you picked.

Here are additional details:

This is a PBX in a Flash distro with updated source and all Free PBX modules updated.

Cent os 5 - 2.6.18-8.1.15.el5

Software:

Basic
Builtin setup Enabled
Core setup 2.5.1.3 Enabled
Feature Code Admin setup 2.5.0.3 Enabled
FreePBX ARI Framework setup 2.5.2.rc1 Enabled
FreePBX FOP Framework setup 2.5.0.1 Enabled
FreePBX Framework setup 2.5.1.0 Enabled
FreePBX Localization Updates setup 2.5.1.1 Enabled
System Dashboard tool 2.5.0.5 Enabled
Voicemail setup 2.5.1.5 Enabled
CID & Number Management
Phonebook Directory tool 2.5.0 Enabled
Speed Dial Functions module 2.5.0 Enabled
Games
Zork (Read this link before downloading) module 0.2.0 Enabled
Inbound Call Control
Announcements setup 2.5.1.7 Enabled
Blacklist setup 2.5.0.3 Enabled
Caller ID Lookup setup 2.5.0.5 Enabled
Day Night Mode setup 2.5.0.10 Enabled
Follow Me setup 2.5.1.7 Enabled
IVR setup 2.5.20.5 Enabled
Queue Priorities setup 2.5.0.4 Enabled
Queues setup 2.5.4.5 Enabled
Ring Groups setup 2.5.1.6 Enabled
Time Conditions setup 2.5.0.8 Enabled
Internal Options & Configuration
Call Forward setup 2.5.0.1 Enabled
Call Waiting setup 2.5.0 Enabled
Callback setup 2.5.0.2 Enabled
Conferences setup 2.5.1.6 Enabled
DISA setup 2.5.1.6 Enabled
Dictation setup 2.5.0.2 Enabled
Do-Not-Disturb (DND) setup 2.5.0.5 Enabled
Info Services setup 2.5.0.1 Enabled
Languages setup 2.5.0.7 Enabled
Misc Applications setup 2.5.0.3 Enabled
Misc Destinations setup 2.5.0.2 Enabled
Music on Hold setup 2.5.1.2 Enabled
PIN Sets setup 2.5.0.1 Enabled
Paging and Intercom setup 2.5.0.6 Enabled
Parking Lot setup 2.5.1.3 Enabled
Recordings setup 3.3.8.8 Enabled
VoiceMail Blasting setup 2.5.0.4 Enabled
Maintenance
A2Billing Admin tool Not Installed (Locally available)
ConfigEdit tool 1.0 Enabled
Sys Info tool phpsysinfo 2.5.3 Enabled
phpMyAdmin tool 2.11.0 Enabled
Support
Asterisk Logfiles tool 2.5.0 Enabled
Online Support tool 2.5.0 Enabled
System Administration
Asterisk API tool 2.5.0.1 Enabled
Asterisk CLI tool 2.5.0.2 Enabled
Asterisk Info tool 2.5.0.1 Enabled
Backup & Restore tool 2.5.1.5 Enabled
Custom Applications tool 2.5.0.4 Enabled
DUNDi Lookup Registry tool 2.5.0 Enabled
Java SSH tool 2.5.0 Enabled
PHP Info tool 2.4.0 Enabled
PHPAGI Config tool 2.5.0.2 Enabled
Phonebook tool 2.5.0.2 Enabled
Third Party Addon
AsteriDex tool 4.0.0 Enabled
Customer DB tool 2.5.0.3 Enabled
Gabcast tool 2.5.0.1 Enabled
Inventory tool 2.5.0.1 Enabled
Print Extensions tool 2.5.0.3 Enabled

Hardware info:

Processors 1
Model Pentium III (Coppermine)
CPU Speed 731.50 Mhz
Cache Size 256.00 KB
System Bogomips 1464.13
PCI Devices - Ethernet controller: Digital Equipment Corporation DECchip 21041 [Tulip Pass 3]

  • FireWire
  • Host bridge: VIA Technologies, Inc. VT82C686 [Apollo Super ACPI]
  • Host bridge: VIA Technologies, Inc. VT8605 [ProSavage PM133]
  • IDE interface: VIA Technologies, Inc. VT82C586A/B/VT82C686/A/B/VT823x/A/C PIPC Bus Master IDE
  • ISA bridge: VIA Technologies, Inc. VT82C686 [Apollo Super South]
  • Input device controller: Creative Labs SB Live! Game Port
  • Multimedia audio controller: Creative Labs SB Live! EMU10k1
  • Multimedia audio controller: VIA Technologies, Inc. VT82C686 AC97 Audio Controller
  • PCI bridge: VIA Technologies, Inc. VT8605 [PM133 AGP]
  • USB Controller: VIA Technologies, Inc. USB 2.0
  • (4x) USB Controller: VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1 Controller
  • VGA compatible controller: nVidia Corporation NV15 [GeForce2 GTS/Pro]

IDE Devices - hdc: ASUS CD-S400/A

  • hda: FUJITSU MPF3204AT (Capacity: 19.09 GB)

SCSI Devices none
USB Devices none

Please let me know if any further information is required.

how much memory do you have installed? The processor is not the fastest thing and if the memory is low that could be a problem with the PHP processes running and it swapping to disk killing the box.

you using asterisk 1.4 or 1.6 branch? It’s also possible that you have found something if it is the 1.6 branch.

Looking at the snippet you posted from the call trace, it looks like it is looping. e.g. it started at [email protected]:6 and went up to [email protected]:19, and then back to [email protected]:1. I’m guessing if you had more of a trace, you’d see it repeating this infinitely, which is why asterisk is railing the CPU?

I beleive there is 1G ram in the machine. Asterisk 1.4.22. Yes, it does repeat like this infinitely.

I have a pbxiaf that does something very similar, but i haven’t been able to figure out what triggers it. I will check and see if letting someone time out in PARK triggers it. My CLI seems to talk about VMX function which I don’t see in yours - but Ill go and recheck.

System Details:
FreePBX 2.5.1.0
Running Asterisk Version : Asterisk 1.4.21.2
Asterisk Source Version : 1.4.21.2
Zaptel Source Version : 1.4.12.1
Libpri Source Version : 1.4.7
Addons Source Version : 1.4.7

All Modules up to date.

-- Goto (macro-get-vmcontext,s,300) -- Executing [[email protected]:300] NoOp("SIP/1347422825-b7907c68", "") in new stack -- Executing [[email protected]:2] Set("SIP/1347422825-b7907c68", "VMGAIN=""") in new stack -- Executing [[email protected]:3] GotoIf("SIP/1347422825-b7907c68", "1?vmx|1") in new stack -- Goto (macro-vm,vmx,1) -- Executing [[email protected]:1] GotoIf("SIP/1347422825-b7907c68", "0?s-NOANSWER|1") in new stack -- Executing [[email protected]:2] Set("SIP/1347422825-b7907c68", "MODE=unavail") in new stack -- Executing [[email protected]:3] GotoIf("SIP/1347422825-b7907c68", "1?notdirect") in new stack -- Goto (macro-vm,vmx,5) -- Executing [[email protected]:5] NoOp("SIP/1347422825-b7907c68", "Checking if ext 500 is enabled: ") in new stack -- Executing [[email protected]:6] GotoIf("SIP/1347422825-b7907c68", "1?s-NOANSWER|1") in new stack -- Executing [[email protected]:3] GotoIf("SIP/1347422825-b7907c68", "1?vmx|1") in new stack -- Goto (macro-vm,vmx,1) -- Executing [[email protected]:1] GotoIf("SIP/1347422825-b7907c68", "0?s-NOANSWER|1") in new stack -- Executing [[email protected]:2] Set("SIP/1347422825-b7907c68", "MODE=unavail") in new stack -- Executing [[email protected]:3] GotoIf("SIP/1347422825-b7907c68", "1?notdirect") in new stack -- Goto (macro-vm,vmx,5) -- Executing [[email protected]:5] NoOp("SIP/1347422825-b7907c68", "Checking if ext 500 is enabled: ") in new stack -- Executing [[email protected]:6] GotoIf("SIP/1347422825-b7907c68", "1?s-NOANSWER|1") in new stack -- Goto (macro-vm,s-NOANSWER,1) -- Executing [[email protected]:1] Macro("SIP/1347422825-b7907c68", "get-vmcontext|500") in new stack -- Executing [[email protected]:1] Set("SIP/1347422825-b7907c68", "VMCONTEXT=default") in new stack -- Executing [[email protected]:2] GotoIf("SIP/1347422825-b7907c68", "0?200:300") in new stack -- Goto (macro-get-vmcontext,s,300) -- Executing [[email protected]:300] NoOp("SIP/1347422825-b7907c68", "") in new stack -- Executing [[email protected]:2] Set("SIP/1347422825-b7907c68", "VMGAIN=""") in new stack -- Executing [[email protected]:3] GotoIf("SIP/1347422825-b7907c68", "1?vmx|1") in new stack -- Goto (macro-vm,vmx,1) -- Executing [[email protected]:1] GotoIf("SIP/1347422825-b7907c68", "0?s-NOANSWER|1") in new stack -- Executing [[email protected]:2] Set("SIP/1347422825-b7907c68", "MODE=unavail") in new stack -- Executing [[email protected]:3] GotoIf("SIP/1347422825-b7907c68", "1?notdirect") in new stack -- Goto (macro-vm,vmx,5) -- Executing [[email protected]:5] NoOp("SIP/1347422825-b7907c68", "Checking if ext 500 is enabled: ") in new stack -- Executing [[email protected]:6] GotoIf("SIP/1347422825-b7907c68", "1?s-NOANSWER|1") in new stack -- Goto (macro-vm,s-NOANSWER,1) -- Executing [[email protected]:1] Macro("SIP/1347422825-b7907c68", "get-vmcontext|500") in new stack -- Executing [[email protected]:1] Set("SIP/1347422825-b7907c68", "VMCONTEXT=default") in new stack -- Executing [[email protected]:2] GotoIf("SIP/1347422825-b7907c68", "0?200:300") in new stack -- Goto (macro-get-vmcontext,s,300) I have no idea whats triggering it. PARK was not involved. It was one lady calling in, and interacting with the voice mail of one of the extensions (500). VMX is NOT turned on, and there are NO follow me settings. The extension is the destination for two inbound routes. Its a fairly straightforward config.

With PBXIAF default logging settings, this will fill up the entire hard disk with log entries in about 12 hours. The PBX largly continues to function when in this status, but its slow to respond to ssh and web connections (for obvious reasons), if it responds at all. A reboot is required to clear the condition.

ErikU, can you call any of the extensions: 101-102-103-104-106-100-110?
To me it looks like you call a Ring Group (600) that has a list of extensions: 101-102-103-104-106-100-110, as no extension is available to call it goes to the fail over destination, and that is set to Ring Group 600.
And you have created your self a nice loop!

tshif,

Please provide the log before the – Goto (macro-get-vmcontext,s,300) statement.
We need that part to see how you end up in in the macro-vm context.

Be nice if freepbx could somehow detect a configuration like this, but I doubt it would be easy :frowning:

It looks like I’m headed in the right direction here…

I made sure that all the extension in the ring group were available. I was able to dial out through my provider back in to my PBX… answer… park the call… and have it ring back ok (however it still doesn’t send the proper alert info on the time out ring back).

I discovered a hitch though… If I park the call from the phone that I called out on, I get the looping error after it rings back. I placed the call from 101 and answered on 106:

– Goto (ext-group,600,1)
– Executing [[email protected]:1] Macro(“SIP/2065690119-b7800468”, “user-callerid|”) in new stack
– Executing [[email protected]:1] Set(“SIP/2065690119-b7800468”, “AMPUSER=2065690119”) in new stack
– Executing [[email protected]:2] GotoIf(“SIP/2065690119-b7800468”, “0?skipdb”) in new stack
– Executing [[email protected]:3] GotoIf(“SIP/2065690119-b7800468”, “0?skipov”) in new stack
– Executing [[email protected]:4] Set(“SIP/2065690119-b7800468”, “__NODEST=”) in new stack
– Executing [[email protected]:5] Set(“SIP/2065690119-b7800468”, “__BLKVM_OVERRIDE=BLKVM/600/SIP/2065690119-b7800468”) in new stack
– Executing [[email protected]:6] Set(“SIP/2065690119-b7800468”, “__BLKVM_BASE=600”) in new stack
– Executing [[email protected]:7] Set(“SIP/2065690119-b7800468”, “DB(BLKVM/600/SIP/2065690119-b7800468)=TRUE”) in new stack
– Executing [[email protected]:8] Set(“SIP/2065690119-b7800468”, “RRNODEST=”) in new stack
– Executing [[email protected]:9] Set(“SIP/2065690119-b7800468”, “__NODEST=600”) in new stack
– Executing [[email protected]:10] Set(“SIP/2065690119-b7800468”, “__ALERT_INFO=External”) in new stack
– Executing [[email protected]:11] Set(“SIP/2065690119-b7800468”, “RecordMethod=Group”) in new stack
– Executing [[email protected]:12] Macro(“SIP/2065690119-b7800468”, “record-enable|101-106-110|Group”) in new stack
– Executing [[email protected]:1] GotoIf(“SIP/2065690119-b7800468”, “1?check”) in new stack
– Goto (macro-record-enable,s,4)
– Executing [[email protected]:13] Set(“SIP/2065690119-b7800468”, “RingGroupMethod=ringall”) in new stack
– Executing [[email protected]:14] Macro(“SIP/2065690119-b7800468”, “dial|60|trT|101-106-110”) in new stack
– Executing [[email protected]:1] GotoIf(“SIP/2065690119-b7800468”, “0?dial”) in new stack
– Executing [[email protected]:15] Set(“SIP/2065690119-b7800468”, “RingGroupMethod=”) in new stack
– Executing [[email protected]:16] GotoIf(“SIP/2065690119-b7800468”, “0?nodest”) in new stack
– Executing [[email protected]:17] Set(“SIP/2065690119-b7800468”, “__NODEST=”) in new stack
– Executing [[email protected]:18] DBdel(“SIP/2065690119-b7800468”, “BLKVM/600/SIP/2065690119-b7800468”) in new stack
– DBdel: family=BLKVM, key=600/SIP/2065690119-b7800468
– Executing [[email protected]:19] Goto(“SIP/2065690119-b7800468”, “ext-group|600|1”) in new stack
– Goto (ext-group,600,1)
– Executing [[email protected]:1] Macro(“SIP/2065690119-b7800468”, “user-callerid|”) in new stack
– Executing [[email protected]:1] Set(“SIP/2065690119-b7800468”, “AMPUSER=2065690119”) in new stack
– Executing [[email protected]:2] GotoIf(“SIP/2065690119-b7800468”, “0?skipdb”) in new stack
– Executing [[email protected]:3] GotoIf(“SIP/2065690119-b7800468”, “0?skipov”) in new stack
– Executing [[email protected]:4] Set(“SIP/2065690119-b7800468”, “__NODEST=”) in new stack
– Executing [[email protected]:5] Set(“SIP/2065690119-b7800468”, “__BLKVM_OVERRIDE=BLKVM/600/SIP/2065690119-b7800468”) in new stack
– Executing [[email protected]:6] Set(“SIP/2065690119-b7800468”, “__BLKVM_BASE=600”) in new stack
– Executing [[email protected]:7] Set(“SIP/2065690119-b7800468”, “DB(BLKVM/600/SIP/2065690119-b7800468)=TRUE”) in new stack

Ill have to try to catch it at onset - perhaps turing loggin all the way up will help. Ill work to capture that info tonight if possible. THANKS for responding.