Server / Asterisk rebooting

Hi this is my first post.

I have a freepbx install FreePBX 2.7.0.5. So far I have a couple of issues.
this system is running in production mode for a client of mine.
first issue, i have installed Digium TE121e (with echo cancel) card in T1 mode. I was experiencing echo issues and could not figure out what the problem was, so i removed the echo daughter board and rescanned the using Dahdi and that seem to fix that problem. 3 weeks later the echo comes back, this is rectified by rebooting the server, but i am afraid the echo will reappear again not sure why this is happening if someone could help me out.

second problem even stranger. so one day i want to see the uptime and noticed its really short like 5 days. so i issue the last reboot command and find out that the server has been rebooting without me having any idea! (total of 12 reboots) now i am not a Linux expert so i asked a friend to take a look and he says its not a kernel panic so he thinks its something else that is causing the reboot. he says its like the power is being pulled out. but i have verified that there are no power outrages during this time.

i look though the asterisk logs and this is what i find that corresponds to the reboot times. last reboot shows 17:23 so i post just before and after. something seems to actually reboot the machine. now im not 100% certain its asterisk.

the server is a dell T410 with TE121e pci-e card 4GB

[Oct 31 17:13:28] VERBOSE[23364] pbx.c: – Executing [[email protected]:1] Macro(“SIP/116-00000174”, “hangupcall,”) in new stack
[Oct 31 17:13:28] VERBOSE[23364] pbx.c: – Executing [[email protected]:1] GotoIf(“SIP/116-00000174”, “1?skiprg”) in new stack
[Oct 31 17:13:28] VERBOSE[23364] pbx.c: – Goto (macro-hangupcall,s,4)
[Oct 31 17:13:28] VERBOSE[23364] pbx.c: – Executing [[email protected]:4] GotoIf(“SIP/116-00000174”, “1?skipblkvm”) in new stack
[Oct 31 17:13:28] VERBOSE[23364] pbx.c: – Goto (macro-hangupcall,s,7)
[Oct 31 17:13:28] VERBOSE[23364] pbx.c: – Executing [[email protected]:7] GotoIf(“SIP/116-00000174”, “1?theend”) in new stack
[Oct 31 17:13:28] VERBOSE[23364] pbx.c: – Goto (macro-hangupcall,s,9)
[Oct 31 17:13:28] VERBOSE[23364] pbx.c: – Executing [[email protected]:9] Hangup(“SIP/116-00000174”, “”) in new stack
[Oct 31 17:13:28] VERBOSE[23364] app_macro.c: == Spawn extension (macro-hangupcall, s, 9) exited non-zero on ‘SIP/116-00000174’ in macro ‘hangupcall’
[Oct 31 17:13:28] VERBOSE[23364] chan_dahdi.c: – Hungup ‘DAHDI/1-1’
[Oct 31 17:13:28] VERBOSE[23364] app_macro.c: == Spawn extension (macro-dialout-trunk, s, 19) exited non-zero on ‘SIP/116-00000174’ in macro ‘dialout-trunk’
[Oct 31 17:13:28] VERBOSE[23364] pbx.c: == Spawn extension (from-internal, 94169163544, 5) exited non-zero on ‘SIP/116-00000174’
[Oct 31 17:13:28] VERBOSE[3268] chan_sip.c: == Extension Changed 116[ext-local] new state Idle for Notify User 116
[Oct 31 17:13:29] NOTICE[3291] chan_sip.c: Failed to authenticate device “113” sip:[email protected];tag=BF8C7080-A3CC2B7 for SUBSCRIBE
[Oct 31 17:13:32] VERBOSE[23361] asterisk.c: – Remote UNIX connection disconnected
[Oct 31 17:13:34] VERBOSE[3431] asterisk.c: – Remote UNIX connection disconnected
[Oct 31 17:13:36] NOTICE[3291] chan_sip.c: Failed to authenticate device “113” sip:[email protected];tag=39CAB71A-2FDA9A11 for SUBSCRIBE
[Oct 31 17:13:37] VERBOSE[16117] asterisk.c: – Remote UNIX connection disconnected
[Oct 31 17:13:38] NOTICE[3291] chan_sip.c: Failed to authenticate device “113” sip:[email protected];tag=D6C324BD-E4B8154 for SUBSCRIBE
[Oct 31 17:13:45] NOTICE[3291] chan_sip.c: Failed to authenticate device “113” sip:[email protected];tag=9D884634-575BA9EB for SUBSCRIBE
[Oct 31 17:13:47] VERBOSE[3296] asterisk.c: Beginning asterisk shutdown…
[Oct 31 17:13:47] VERBOSE[27742] app.c: – User hung up
[Oct 31 17:13:47] WARNING[27742] channel.c: Channel allocation failed: Refusing due to active shutdown
[Oct 31 17:13:47] WARNING[27742] app_voicemail.c: Cannot allocate the channel for variables substitution
[Oct 31 17:13:47] WARNING[27742] channel.c: Channel allocation failed: Refusing due to active shutdown
[Oct 31 17:13:47] WARNING[27742] app_voicemail.c: Cannot allocate the channel for variables substitution
[Oct 31 17:13:47] WARNING[3291] chan_sip.c: sip_xmit of 0x1de74be0 (len 433) to 192.168.0.90:5060 returned -2: Network is unreachable
[Oct 31 17:13:47] WARNING[3291] chan_sip.c: Network error on retransmit in dialog [email protected]
[Oct 31 17:13:47] WARNING[3291] chan_sip.c: Transmit error :: Cancelling transmission on Call ID [email protected]
[Oct 31 17:13:47] VERBOSE[27742] app_macro.c: == Spawn extension (macro-vm, s-NOANSWER, 2) exited non-zero on ‘SIP/333-00000047’ in macro ‘vm’
[Oct 31 17:13:47] VERBOSE[27742] app_macro.c: == Spawn extension (macro-exten-vm, s, 18) exited non-zero on ‘SIP/333-00000047’ in macro ‘exten-vm’
[Oct 31 17:13:47] VERBOSE[27742] pbx.c: == Spawn extension (ext-local, 444, 1) exited non-zero on ‘SIP/333-00000047’
[Oct 31 17:13:47] WARNING[27742] chan_sip.c: sip_xmit of 0x1de74be0 (len 426) to 192.168.20.1:5060 returned -2: Network is unreachable
[Oct 31 17:13:47] ERROR[27742] chan_sip.c: Serious Network Trouble; __sip_xmit returns error for pkt data
[Oct 31 17:13:47] ERROR[27742] cdr_addon_mysql.c: Unknown connection error: (2002) Can’t connect to local MySQL server through socket ‘/var/lib/mysql/mysql.sock’ (2)
[Oct 31 17:13:47] ERROR[27742] cdr_addon_mysql.c: Cannot connect to database server localhost: (2002) Can’t connect to local MySQL server through socket ‘/var/lib/mysql/mysql.sock’ (2)
[Oct 31 17:13:47] WARNING[3260] acl.c: Cannot connect
[Oct 31 17:13:47] WARNING[3260] chan_sip.c: sip_xmit of 0x2aaaac062530 (len 543) to 192.168.0.117:5060 returned -2: Network is unreachable
[Oct 31 17:13:47] ERROR[3260] chan_sip.c: Serious Network Trouble; __sip_xmit returns error for pkt data
[Oct 31 17:13:47] VERBOSE[3296] asterisk.c: Executing last minute cleanups
[Oct 31 17:13:47] VERBOSE[3296] res_musiconhold.c: == Destroying musiconhold processes
[Oct 31 17:13:47] VERBOSE[3296] asterisk.c: Asterisk cleanly ending (0).
[Oct 31 17:24:00] VERBOSE[3269] logger.c: Asterisk Event Logger Started /var/log/asterisk/event_log
[Oct 31 17:24:00] VERBOSE[3269] config.c: == Parsing ‘/etc/asterisk/asterisk.conf’: [Oct 31 17:24:00] VERBOSE[3269] config.c: == Found
[Oct 31 17:24:00] VERBOSE[3269] loader.c: Asterisk Dynamic Loader Starting:
[Oct 31 17:24:00] VERBOSE[3269] config.c: == Parsing ‘/etc/asterisk/modules.conf’: [Oct 31 17:24:00] VERBOSE[3269] config.c: == Found
[Oct 31 17:24:00] NOTICE[3269] loader.c: 2 modules will be loaded.
[Oct 31 17:24:00] VERBOSE[3269] config.c: == Parsing ‘/etc/asterisk/extensions.conf’: [Oct 31 17:24:00] VERBOSE[3269] config.c: == Found
[Oct 31 17:24:00] VERBOSE[3269] config.c: == Parsing ‘/etc/asterisk/extensions_override_freepbx.conf’: [Oct 31 17:24:00] VERBOSE[3269] config.c: == Found
[Oct 31 17:24:00] VERBOSE[3269] config.c: == Parsing ‘/etc/asterisk/extensions_additional.conf’: [Oct 31 17:24:00] VERBOSE[3269] config.c: == Found
[Oct 31 17:24:00] VERBOSE[3269] config.c: == Parsing ‘/etc/asterisk/globals_custom.conf’: [Oct 31 17:24:00] VERBOSE[3269] config.c: == Found
[Oct 31 17:24:00] VERBOSE[3269] config.c: == Parsing ‘/etc/asterisk/extensions_custom.conf’: [Oct 31 17:24:00] VERBOSE[3269] config.c: == Found
[Oct 31 17:24:00] VERBOSE[3269] pbx.c: == Setting global variable ‘CFDEVSTATE’ to ‘TRUE’

By the way, I can see that it’s not a power failure because Asterisk is actually shutting down from the looks of your log files. “Asterisk Cleanly Ended” It seems as if there is a software error or cron job that is causing the reboot.

Thanks,
Perry
SIP Global Phone, LLC

Can you see any Cron jobs running at the time of the reboots? I would look in Webmin for the easiest way to find them.
Can you take the T-1 hardware out long enough to see if the reboots stop?
Do you happen to see any pattern with the reboot times?
Do you know what the processor temperature is? It’s a long shot but I once had a machine doing this (Not Asterisk) and it turned out to be a processor overheat. Normally, those would shut the machine down outright so that’s why I say it’s a long shot.

Thanks,
Perry
SIP Global Phone, LLC