System Going Down Mid-Day - Trying to track root cause

General Help
1

Have a client with an on-prem PBXAct system and twice in the last 7 days it suddenly stopped passing calls for about 10 minutes. It behaved similar to if I were to push the red apply config button, but we didn’t press it.

In today’s instance, I screenshot the graph of memory and CPU usage to show how CPU spiked and memory flatlined at the timestamp 09:14:09.

Can anyone make sense of the logs during this time to help me determine root cause. Can include more of the logs if needed. Thanks!

lagwana freepbx memory
lagwana freepbx cpu

[2022-08-04 09:13:31] VERBOSE[26296][C-00000659] pbx_builtins.c: Goto (ivr-3,s,8)
[2022-08-04 09:13:31] VERBOSE[26296][C-00000659] pbx.c: Executing [s@ivr-3:8] Set("PJSIP/GentleSpeak-00000f0e", "IVR_MSG=custom/IVR_Instructions_-_updated_1") in new stack
[2022-08-04 09:13:31] VERBOSE[26296][C-00000659] pbx.c: Executing [s@ivr-3:9] Set("PJSIP/GentleSpeak-00000f0e", "TIMEOUT(digit)=3") in new stack
[2022-08-04 09:13:31] VERBOSE[26296][C-00000659] func_timeout.c: Digit timeout set to 3.000
[2022-08-04 09:13:31] VERBOSE[26296][C-00000659] pbx.c: Executing [s@ivr-3:10] ExecIf("PJSIP/GentleSpeak-00000f0e", "1?Background(custom/IVR_Instructions_-_updated_1)") in new stack
[2022-08-04 09:13:31] VERBOSE[26296][C-00000659] file.c: <PJSIP/GentleSpeak-00000f0e> Playing 'custom/IVR_Instructions_-_updated_1.ulaw' (language 'en')
[2022-08-04 09:13:35] NOTICE[5937] res_pjsip_exten_state.c: Endpoint '1047' state subscription failed: Extension '*992*15*1047*1' does not exist in context 'from-internal' or has no associated hint
[2022-08-04 09:13:36] NOTICE[26212] res_pjsip_exten_state.c: Endpoint '1047' state subscription failed: Extension '*992*11*1047*1' does not exist in context 'from-internal' or has no associated hint
[2022-08-04 09:13:36] NOTICE[4039] res_pjsip_exten_state.c: Endpoint '1047' state subscription failed: Extension '*992*12*1047*1' does not exist in context 'from-internal' or has no associated hint
[2022-08-04 09:14:08] NOTICE[12801] res_pjsip_exten_state.c: Endpoint '1017' state subscription failed: Extension '*992*15*1017*1' does not exist in context 'from-internal' or has no associated hint
[2022-08-04 09:14:09] NOTICE[19149] res_pjsip_exten_state.c: Endpoint '1017' state subscription failed: Extension '*992*11*1017*1' does not exist in context 'from-internal' or has no associated hint
[2022-08-04 09:14:09] NOTICE[3885] res_pjsip_exten_state.c: Endpoint '1017' state subscription failed: Extension '*992*12*1017*1' does not exist in context 'from-internal' or has no associated hint
[2022-08-04 09:14:12] VERBOSE[26296][C-00000659] pbx.c: Executing [2@ivr-3:1] Set("PJSIP/GentleSpeak-00000f0e", "__ivrreturn=0") in new stack
[2022-08-04 09:14:12] VERBOSE[26296][C-00000659] pbx.c: Executing [2@ivr-3:2] Goto("PJSIP/GentleSpeak-00000f0e", "ext-group,102,1") in new stack
[2022-08-04 09:14:12] VERBOSE[26296][C-00000659] pbx_builtins.c: Goto (ext-group,102,1)
[2022-08-04 09:14:12] VERBOSE[26296][C-00000659] pbx.c: Executing [102@ext-group:1] GotoIf("PJSIP/GentleSpeak-00000f0e", "0?cid") in new stack
[2022-08-04 09:14:12] VERBOSE[26296][C-00000659] pbx.c: Executing [102@ext-group:2] PlayTones("PJSIP/GentleSpeak-00000f0e", "ring") in new stack
[2022-08-04 09:14:12] VERBOSE[26296][C-00000659] pbx.c: Executing [102@ext-group:3] Progress("PJSIP/GentleSpeak-00000f0e", "") in new stack
[2022-08-04 09:14:12] VERBOSE[26296][C-00000659] pbx.c: Executing [102@ext-group:4] Macro("PJSIP/GentleSpeak-00000f0e", "user-callerid,") in new stack
[2022-08-04 09:14:12] VERBOSE[26296][C-00000659] pbx.c: Executing [s@macro-user-callerid:1] Set("PJSIP/GentleSpeak-00000f0e", "TOUCH_MONITOR=1659618806.16656") in new stack
[2022-08-04 09:14:12] VERBOSE[26296][C-00000659] pbx.c: Executing [s@macro-user-callerid:2] Set("PJSIP/GentleSpeak-00000f0e", "CHANCONTEXT=") in new stack
[2022-08-04 09:14:12] VERBOSE[26296][C-00000659] pbx.c: Executing [s@macro-user-callerid:3] Set("PJSIP/GentleSpeak-00000f0e", "CHANCONTEXT=") in new stack
[2022-08-04 09:14:12] VERBOSE[26296][C-00000659] pbx.c: Executing [s@macro-user-callerid:4] Set("PJSIP/GentleSpeak-00000f0e", "CHANEXTENCONTEXT=GentleSpeak-00000f0e") in new stack
[2022-08-04 09:14:12] VERBOSE[26296][C-00000659] pbx.c: Executing [s@macro-user-callerid:5] Set("PJSIP/GentleSpeak-00000f0e", "CHANEXTEN=GentleSpeak-00000f0e") in new stack
[2022-08-04 09:14:12] VERBOSE[26296][C-00000659] pbx.c: Executing [s@macro-user-callerid:6] Set("PJSIP/GentleSpeak-00000f0e", "CALLERID(number)=13176053022") in new stack
[2022-08-04 09:14:12] VERBOSE[26296][C-00000659] pbx.c: Executing [s@macro-user-callerid:7] Set("PJSIP/GentleSpeak-00000f0e", "AMPUSER=13176053022") in new stack
[2022-08-04 09:14:12] VERBOSE[26296][C-00000659] pbx.c: Executing [s@macro-user-callerid:8] Set("PJSIP/GentleSpeak-00000f0e", "HOTDESCKCHAN=GentleSpeak-00000f0e") in new stack
[2022-08-04 09:14:12] VERBOSE[26296][C-00000659] pbx.c: Executing [s@macro-user-callerid:9] Set("PJSIP/GentleSpeak-00000f0e", "HOTDESKEXTEN=GentleSpeak") in new stack
[2022-08-04 09:14:12] VERBOSE[26296][C-00000659] pbx.c: Executing [s@macro-user-callerid:10] Set("PJSIP/GentleSpeak-00000f0e", "HOTDESKCALL=0") in new stack
[2022-08-04 09:14:12] VERBOSE[26296][C-00000659] pbx.c: Executing [s@macro-user-callerid:11] ExecIf("PJSIP/GentleSpeak-00000f0e", "0?Set(HOTDESKCALL=1)") in new stack
[2022-08-04 09:14:12] VERBOSE[26296][C-00000659] pbx.c: Executing [s@macro-user-callerid:12] ExecIf("PJSIP/GentleSpeak-00000f0e", "0?Set(CALLERID(name)=)") in new stack
[2022-08-04 09:14:12] VERBOSE[26296][C-00000659] pbx.c: Executing [s@macro-user-callerid:13] GotoIf("PJSIP/GentleSpeak-00000f0e", "0?report") in new stack
[2022-08-04 09:14:12] VERBOSE[26296][C-00000659] pbx.c: Executing [s@macro-user-callerid:14] ExecIf("PJSIP/GentleSpeak-00000f0e", "1?Set(REALCALLERIDNUM=13176053022)") in new stack
[2022-08-04 09:14:12] VERBOSE[26296][C-00000659] pbx.c: Executing [s@macro-user-callerid:15] Set("PJSIP/GentleSpeak-00000f0e", "AMPUSER=") in new stack
[2022-08-04 09:14:12] VERBOSE[26296][C-00000659] pbx.c: Executing [s@macro-user-callerid:16] GotoIf("PJSIP/GentleSpeak-00000f0e", "0?limit") in new stack
[2022-08-04 09:14:12] VERBOSE[26296][C-00000659] pbx.c: Executing [s@macro-user-callerid:17] Set("PJSIP/GentleSpeak-00000f0e", "AMPUSERCIDNAME=") in new stack
[2022-08-04 09:14:12] VERBOSE[26296][C-00000659] pbx.c: Executing [s@macro-user-callerid:18] ExecIf("PJSIP/GentleSpeak-00000f0e", "0?Set(__CIDMASQUERADING=TRUE)") in new stack
[2022-08-04 09:14:12] VERBOSE[26296][C-00000659] pbx.c: Executing [s@macro-user-callerid:19] GotoIf("PJSIP/GentleSpeak-00000f0e", "1?report") in new stack
[2022-08-04 09:14:12] VERBOSE[26296][C-00000659] pbx_builtins.c: Goto (macro-user-callerid,s,28)
[2022-08-04 09:14:12] VERBOSE[26296][C-00000659] pbx.c: Executing [s@macro-user-callerid:28] NoOp("PJSIP/GentleSpeak-00000f0e", "Macro Depth is 1") in new stack
[2022-08-04 09:14:12] VERBOSE[26296][C-00000659] pbx.c: Executing [s@macro-user-callerid:29] GotoIf("PJSIP/GentleSpeak-00000f0e", "1?report2:macroerror") in new stack
[2022-08-04 09:14:12] VERBOSE[26296][C-00000659] pbx_builtins.c: Goto (macro-user-callerid,s,30)
[2022-08-04 09:14:12] VERBOSE[26296][C-00000659] pbx.c: Executing [s@macro-user-callerid:30] GotoIf("PJSIP/GentleSpeak-00000f0e", "0?continue") in new stack
[2022-08-04 09:14:12] VERBOSE[26296][C-00000659] pbx.c: Executing [s@macro-user-callerid:31] ExecIf("PJSIP/GentleSpeak-00000f0e", "1?Set(__CALLEE_```
2

look in /var/log/messages (/var/log/syslog) at around the same time.

(Install sysstat for future convenience)

3

Do I need ssh or console access to view those?

4

You do

5

Do you, by any chance, have an ISP that messes with your IP address at that time? This is not uncommon on consumer and SOHO type accounts, although I don’t know how many would do it at midday.

6

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.