System Going Down Mid-Day - Trying to track root cause

Have a client with an on-prem PBXAct system and twice in the last 7 days it suddenly stopped passing calls for about 10 minutes. It behaved similar to if I were to push the red apply config button, but we didn’t press it.

In today’s instance, I screenshot the graph of memory and CPU usage to show how CPU spiked and memory flatlined at the timestamp 09:14:09.

Can anyone make sense of the logs during this time to help me determine root cause. Can include more of the logs if needed. Thanks!

lagwana freepbx memory
lagwana freepbx cpu

[2022-08-04 09:13:31] VERBOSE[26296][C-00000659] pbx_builtins.c: Goto (ivr-3,s,8)
[2022-08-04 09:13:31] VERBOSE[26296][C-00000659] pbx.c: Executing [[email protected]:8] Set("PJSIP/GentleSpeak-00000f0e", "IVR_MSG=custom/IVR_Instructions_-_updated_1") in new stack
[2022-08-04 09:13:31] VERBOSE[26296][C-00000659] pbx.c: Executing [[email protected]:9] Set("PJSIP/GentleSpeak-00000f0e", "TIMEOUT(digit)=3") in new stack
[2022-08-04 09:13:31] VERBOSE[26296][C-00000659] func_timeout.c: Digit timeout set to 3.000
[2022-08-04 09:13:31] VERBOSE[26296][C-00000659] pbx.c: Executing [[email protected]:10] ExecIf("PJSIP/GentleSpeak-00000f0e", "1?Background(custom/IVR_Instructions_-_updated_1)") in new stack
[2022-08-04 09:13:31] VERBOSE[26296][C-00000659] file.c: <PJSIP/GentleSpeak-00000f0e> Playing 'custom/IVR_Instructions_-_updated_1.ulaw' (language 'en')
[2022-08-04 09:13:35] NOTICE[5937] res_pjsip_exten_state.c: Endpoint '1047' state subscription failed: Extension '*992*15*1047*1' does not exist in context 'from-internal' or has no associated hint
[2022-08-04 09:13:36] NOTICE[26212] res_pjsip_exten_state.c: Endpoint '1047' state subscription failed: Extension '*992*11*1047*1' does not exist in context 'from-internal' or has no associated hint
[2022-08-04 09:13:36] NOTICE[4039] res_pjsip_exten_state.c: Endpoint '1047' state subscription failed: Extension '*992*12*1047*1' does not exist in context 'from-internal' or has no associated hint
[2022-08-04 09:14:08] NOTICE[12801] res_pjsip_exten_state.c: Endpoint '1017' state subscription failed: Extension '*992*15*1017*1' does not exist in context 'from-internal' or has no associated hint
[2022-08-04 09:14:09] NOTICE[19149] res_pjsip_exten_state.c: Endpoint '1017' state subscription failed: Extension '*992*11*1017*1' does not exist in context 'from-internal' or has no associated hint
[2022-08-04 09:14:09] NOTICE[3885] res_pjsip_exten_state.c: Endpoint '1017' state subscription failed: Extension '*992*12*1017*1' does not exist in context 'from-internal' or has no associated hint
[2022-08-04 09:14:12] VERBOSE[26296][C-00000659] pbx.c: Executing [[email protected]:1] Set("PJSIP/GentleSpeak-00000f0e", "__ivrreturn=0") in new stack
[2022-08-04 09:14:12] VERBOSE[26296][C-00000659] pbx.c: Executing [[email protected]:2] Goto("PJSIP/GentleSpeak-00000f0e", "ext-group,102,1") in new stack
[2022-08-04 09:14:12] VERBOSE[26296][C-00000659] pbx_builtins.c: Goto (ext-group,102,1)
[2022-08-04 09:14:12] VERBOSE[26296][C-00000659] pbx.c: Executing [[email protected]:1] GotoIf("PJSIP/GentleSpeak-00000f0e", "0?cid") in new stack
[2022-08-04 09:14:12] VERBOSE[26296][C-00000659] pbx.c: Executing [[email protected]:2] PlayTones("PJSIP/GentleSpeak-00000f0e", "ring") in new stack
[2022-08-04 09:14:12] VERBOSE[26296][C-00000659] pbx.c: Executing [[email protected]:3] Progress("PJSIP/GentleSpeak-00000f0e", "") in new stack
[2022-08-04 09:14:12] VERBOSE[26296][C-00000659] pbx.c: Executing [[email protected]:4] Macro("PJSIP/GentleSpeak-00000f0e", "user-callerid,") in new stack
[2022-08-04 09:14:12] VERBOSE[26296][C-00000659] pbx.c: Executing [[email protected]:1] Set("PJSIP/GentleSpeak-00000f0e", "TOUCH_MONITOR=1659618806.16656") in new stack
[2022-08-04 09:14:12] VERBOSE[26296][C-00000659] pbx.c: Executing [[email protected]:2] Set("PJSIP/GentleSpeak-00000f0e", "CHANCONTEXT=") in new stack
[2022-08-04 09:14:12] VERBOSE[26296][C-00000659] pbx.c: Executing [[email protected]:3] Set("PJSIP/GentleSpeak-00000f0e", "CHANCONTEXT=") in new stack
[2022-08-04 09:14:12] VERBOSE[26296][C-00000659] pbx.c: Executing [[email protected]:4] Set("PJSIP/GentleSpeak-00000f0e", "CHANEXTENCONTEXT=GentleSpeak-00000f0e") in new stack
[2022-08-04 09:14:12] VERBOSE[26296][C-00000659] pbx.c: Executing [[email protected]:5] Set("PJSIP/GentleSpeak-00000f0e", "CHANEXTEN=GentleSpeak-00000f0e") in new stack
[2022-08-04 09:14:12] VERBOSE[26296][C-00000659] pbx.c: Executing [[email protected]:6] Set("PJSIP/GentleSpeak-00000f0e", "CALLERID(number)=13176053022") in new stack
[2022-08-04 09:14:12] VERBOSE[26296][C-00000659] pbx.c: Executing [[email protected]:7] Set("PJSIP/GentleSpeak-00000f0e", "AMPUSER=13176053022") in new stack
[2022-08-04 09:14:12] VERBOSE[26296][C-00000659] pbx.c: Executing [[email protected]:8] Set("PJSIP/GentleSpeak-00000f0e", "HOTDESCKCHAN=GentleSpeak-00000f0e") in new stack
[2022-08-04 09:14:12] VERBOSE[26296][C-00000659] pbx.c: Executing [[email protected]:9] Set("PJSIP/GentleSpeak-00000f0e", "HOTDESKEXTEN=GentleSpeak") in new stack
[2022-08-04 09:14:12] VERBOSE[26296][C-00000659] pbx.c: Executing [[email protected]:10] Set("PJSIP/GentleSpeak-00000f0e", "HOTDESKCALL=0") in new stack
[2022-08-04 09:14:12] VERBOSE[26296][C-00000659] pbx.c: Executing [[email protected]:11] ExecIf("PJSIP/GentleSpeak-00000f0e", "0?Set(HOTDESKCALL=1)") in new stack
[2022-08-04 09:14:12] VERBOSE[26296][C-00000659] pbx.c: Executing [[email protected]:12] ExecIf("PJSIP/GentleSpeak-00000f0e", "0?Set(CALLERID(name)=)") in new stack
[2022-08-04 09:14:12] VERBOSE[26296][C-00000659] pbx.c: Executing [[email protected]:13] GotoIf("PJSIP/GentleSpeak-00000f0e", "0?report") in new stack
[2022-08-04 09:14:12] VERBOSE[26296][C-00000659] pbx.c: Executing [[email protected]:14] ExecIf("PJSIP/GentleSpeak-00000f0e", "1?Set(REALCALLERIDNUM=13176053022)") in new stack
[2022-08-04 09:14:12] VERBOSE[26296][C-00000659] pbx.c: Executing [[email protected]:15] Set("PJSIP/GentleSpeak-00000f0e", "AMPUSER=") in new stack
[2022-08-04 09:14:12] VERBOSE[26296][C-00000659] pbx.c: Executing [[email protected]:16] GotoIf("PJSIP/GentleSpeak-00000f0e", "0?limit") in new stack
[2022-08-04 09:14:12] VERBOSE[26296][C-00000659] pbx.c: Executing [[email protected]:17] Set("PJSIP/GentleSpeak-00000f0e", "AMPUSERCIDNAME=") in new stack
[2022-08-04 09:14:12] VERBOSE[26296][C-00000659] pbx.c: Executing [[email protected]:18] ExecIf("PJSIP/GentleSpeak-00000f0e", "0?Set(__CIDMASQUERADING=TRUE)") in new stack
[2022-08-04 09:14:12] VERBOSE[26296][C-00000659] pbx.c: Executing [[email protected]:19] GotoIf("PJSIP/GentleSpeak-00000f0e", "1?report") in new stack
[2022-08-04 09:14:12] VERBOSE[26296][C-00000659] pbx_builtins.c: Goto (macro-user-callerid,s,28)
[2022-08-04 09:14:12] VERBOSE[26296][C-00000659] pbx.c: Executing [[email protected]:28] NoOp("PJSIP/GentleSpeak-00000f0e", "Macro Depth is 1") in new stack
[2022-08-04 09:14:12] VERBOSE[26296][C-00000659] pbx.c: Executing [[email protected]:29] GotoIf("PJSIP/GentleSpeak-00000f0e", "1?report2:macroerror") in new stack
[2022-08-04 09:14:12] VERBOSE[26296][C-00000659] pbx_builtins.c: Goto (macro-user-callerid,s,30)
[2022-08-04 09:14:12] VERBOSE[26296][C-00000659] pbx.c: Executing [[email protected]:30] GotoIf("PJSIP/GentleSpeak-00000f0e", "0?continue") in new stack
[2022-08-04 09:14:12] VERBOSE[26296][C-00000659] pbx.c: Executing [[email protected]:31] ExecIf("PJSIP/GentleSpeak-00000f0e", "1?Set(__CALLEE_```

look in /var/log/messages (/var/log/syslog) at around the same time.

(Install sysstat for future convenience)

Do I need ssh or console access to view those?

You do

Do you, by any chance, have an ISP that messes with your IP address at that time? This is not uncommon on consumer and SOHO type accounts, although I don’t know how many would do it at midday.