All Calls dropping simultaneously

jhayes · February 2, 2016, 6:31pm

We are currently running the asterisk 13 with PBX Firmware 10.13.66-5. Intermittently through the day all calls will drop regardless of direction. I have attempted to assess the local network, even though our remote offices calls drop at the same time. I have run an MTR for days at a time to monitor the reliability of the 100/100 connection we have. To me, it seems that asterisks stops functioning and needs to reboot. I believe this due to all in progress calls are not written to the cdr logs and any associated recordings need to be pulled manually from the monitor log.

When doing a tcp dump (tcpdump -w - -p -n -s 0 udp > /tmp/capture-asterisk.pcap) I am noticing on average a 0.00002% packet drop by the kernel.

When checking the asterisk log the only common factor seems to be an Error [numeric value] astobj2.c followed by, what I believe is, the dial plan reloading.

[2016-02-02 10:11:05] VERBOSE[11051][C-00000b2d] pbx.c: Goto (macro-dial-one,dstring,12)
[2016-02-02 10:11:05] VERBOSE[11052][C-00000b2d] pbx.c: Goto (macro-dial-one,s,12)
[2016-02-02 10:11:05] VERBOSE[11051][C-00000b2d] pbx.c: Executing [dstring@macro-dial-one:12] GotoIf("Local/7691@from-queue-000005b2;2", "0?skipset") in new stack
[2016-02-02 10:11:05] VERBOSE[11049][C-00000b2d] pbx.c: Executing [s@macro-user-callerid:33] Set("Local/71438@from-queue-000005b0;2", "CHANNEL(language)=en") in new stack
[2016-02-02 10:11:05] VERBOSE[11053][C-00000b2d] pbx.c: Executing [s@macro-dial-one:17] GotoIf("Local/7932@from-queue-000005b4;2", "0?continue") in new stack
[2016-02-02 10:11:05] ERROR[106470] astobj2.c: bad magic number for object 0x7ffd0c004668. Object is likely destroyed.
[2016-02-02 10:11:09] Asterisk 13.5.0 built by mockbuild @ jenkins2.schmoozecom.net on a x86_64 running Linux on 2015-08-10 14:03:53 UTC
[2016-02-02 10:11:09] VERBOSE[11068] manager.c: Manager registered action DBGet
[2016-02-02 10:11:09] VERBOSE[11068] manager.c: Manager registered action DBPut
[2016-02-02 10:11:09] VERBOSE[11068] manager.c: Manager registered action DBDel
[2016-02-02 10:11:09] VERBOSE[11068] manager.c: Manager registered action DBDelTree
[2016-02-02 10:11:09] VERBOSE[11068] message.c: Message handler 'dialplan' registered.
[2016-02-02 10:11:09] VERBOSE[11068] pbx.c: Registered custom function 'MESSAGE'
[2016-02-02 10:11:09] VERBOSE[11068] pbx.c: Registered custom function 'MESSAGE_DATA'
[2016-02-02 10:11:09] VERBOSE[11068] pbx.c: Registered application 'MessageSend'
[2016-02-02 10:11:09] VERBOSE[11068] manager.c: Manager registered action MessageSend
[2016-02-02 10:11:09] VERBOSE[11068] manager.c: Manager registered action DataGet
[2016-02-02 10:11:09] VERBOSE[11068] channel.c: Registered channel type 'Surrogate' (Surrogate channel used to pull channel from an application)
[2016-02-02 10:11:09] VERBOSE[11068] manager.c: Manager registered action BridgeTechnologyList
[2016-02-02 10:11:09] VERBOSE[11068] manager.c: Manager registered action BridgeTechnologySuspend
[2016-02-02 10:11:09] VERBOSE[11068] manager.c: Manager registered action BridgeTechnologyUnsuspend
[2016-02-02 10:11:09] VERBOSE[11068] loader.c: Asterisk Dynamic Loader Starting:
[2016-02-02 10:11:09] VERBOSE[11068] config.c: Parsing '/etc/asterisk/modules.conf': Found
[2016-02-02 10:11:09] NOTICE[11068] loader.c: 2 modules will be loaded.
[2016-02-02 10:11:10] WARNING[11068] loader.c: Error loading module 'chan_local.so': /usr/lib64/asterisk/modules/chan_local.so: cannot open shared object file: No such file or directory
[2016-02-02 10:11:10] WARNING[11068] loader.c: Error loading module 'chan_local.so': /usr/lib64/asterisk/modules/chan_local.so: cannot open shared object file: No such file or directory
[2016-02-02 10:11:10] WARNING[11068] loader.c: Module 'chan_local.so' could not be loaded.
[2016-02-02 10:11:10] VERBOSE[11068] loader.c: Loading pbx_config.so.
[2016-02-02 10:11:10] VERBOSE[11068] manager.c: Manager registered action DialplanExtensionAdd
[2016-02-02 10:11:10] VERBOSE[11068] manager.c: Manager registered action DialplanExtensionRemove
[2016-02-02 10:11:10] VERBOSE[11068] config.c: Parsing '/etc/asterisk/extensions.conf': Found
[2016-02-02 10:11:10] VERBOSE[11068] config.c: Parsing '/etc/asterisk/extensions_override_freepbx.conf': Found
[2016-02-02 10:11:10] VERBOSE[11068] config.c: Parsing '/etc/asterisk/extensions_additional.conf': Found
[2016-02-02 10:11:10] VERBOSE[11068] config.c: Parsing '/etc/asterisk/globals_custom.conf': Found
[2016-02-02 10:11:10] VERBOSE[11068] config.c: Parsing '/etc/asterisk/extensions_custom.conf': Found
[2016-02-02 10:11:10] VERBOSE[11068] config.c: Parsing '/etc/asterisk/extensions_custom_asternic_outbound_freepbx.conf': Found
[2016-02-02 10:11:10] VERBOSE[11068] pbx.c: Setting global variable 'CFDEVSTATE' to 'TRUE'
[2016-02-02 10:11:10] VERBOSE[11068] pbx.c: Setting global variable 'MIXMON_BEEP' to ''
[2016-02-02 10:11:10] VERBOSE[11068] pbx.c: Setting global variable 'DNDDEVSTATE' to 'TRUE'
[2016-02-02 10:11:10] VERBOSE[11068] pbx.c: Setting global variable 'FMDEVSTATE' to 'TRUE'
[2016-02-02 10:11:10] VERBOSE[11068] pbx.c: Setting global variable 'QUEDEVSTATE' to 'TRUE'
[2016-02-02 10:11:10] VERBOSE[11068] pbx.c: Setting global variable 'QUEUETOGGLE' to '*45'
[2016-02-02 10:11:10] VERBOSE[11068] pbx.c: Setting global variable 'QUEUEPAUSETOGGLE' to '*46'
[2016-02-02 10:11:10] VERBOSE[11068] pbx.c: Setting global variable 'DYNAMIC_FEATURES' to 'apprecord'
[2016-02-02 10:11:10] VERBOSE[11068] pbx.c: Setting global variable 'INTERCOMCODE' to '*80'

I have also done a backtrace but it was to large to upload in its entirety. If someone could let me know what I am suppose to be looking for I will upload the pertinent content.

jhayes · February 5, 2016, 2:18pm

Not sure if this pertains the the above issue but I came in today to an error in my open ssh terminal. The following errors where in the window.

[root@sip asterisk]# /usr/sbin/safe_asterisk: line 164: 61009 Segmentation fault      (core dumped) nice -n $PRIORITY "${ASTSBINDIR}/asterisk" -f ${CLIARGS} ${ASTARGS} > /dev/${TTY} 2>&1 < /dev/${TTY}
/usr/sbin/safe_asterisk: line 164: 70310 Segmentation fault      (core dumped) nice -n $PRIORITY "${ASTSBINDIR}/asterisk" -f ${CLIARGS} ${ASTARGS} > /dev/${TTY} 2>&1 < /dev/${TTY}
/usr/sbin/safe_asterisk: line 164: 42123 Segmentation fault      (core dumped) nice -n $PRIORITY "${ASTSBINDIR}/asterisk" -f ${CLIARGS} ${ASTARGS} > /dev/${TTY} 2>&1 < /dev/${TTY}
/usr/sbin/safe_asterisk: line 164: 75481 Segmentation fault      (core dumped) nice -n $PRIORITY "${ASTSBINDIR}/asterisk" -f ${CLIARGS} ${ASTARGS} > /dev/${TTY} 2>&1 < /dev/${TTY}
/usr/sbin/safe_asterisk: line 164: 101952 Segmentation fault      (core dumped) nice -n $PRIORITY "${ASTSBINDIR}/asterisk" -f ${CLIARGS} ${ASTARGS} > /dev/${TTY} 2>&1 < /dev/${TTY}

jhayes · February 15, 2016, 2:18pm

Does anyone have any thoughts or suggestions in troubleshooting this issue?

xrobau · February 16, 2016, 6:10am

Asterisk is crashing - have you updated? (yum update)

jhayes · February 18, 2016, 2:18pm

I attempted a yum update and this made it to where no extension could authenticate against the database. Due to time constraints I rolled the system back to the previous day’s backup without much troubleshooting on the reason for lack of authentication.