My FreePBX server rebooted itself without my input

FreePBX System Administration
1

This morning for no reason I can tell, my Freepbx server rebooted without my input. Why would it do that? Where should I look to find out what caused it? btw I just updated my Freepbx from 15 to version 16.0.40.3 a couple days ago.

Here is a sample of the logs right as the reboot was happening:

– end of packet.
[2023-06-12 09:51:26] VERBOSE[3477] asterisk.c: Remote UNIX connection
[2023-06-12 09:51:26] VERBOSE[11335] asterisk.c: Waiting for inactivity to perform halt…
[2023-06-12 09:51:26] VERBOSE[11335] loader.c: Unloading res_manager_presencestate.so
[2023-06-12 09:51:26] VERBOSE[11335] manager.c: Manager unregistered action PresenceStateList
[2023-06-12 09:51:26] VERBOSE[11335] loader.c: Unloading res_manager_devicestate.so
[2023-06-12 09:51:26] VERBOSE[11335] manager.c: Manager unregistered action DeviceStateList
[2023-06-12 09:51:26] VERBOSE[11335] loader.c: Unloading app_queue.so
[2023-06-12 09:51:26] VERBOSE[11335] manager.c: Manager unregistered action QueueStatus
[2023-06-12 09:51:26] VERBOSE[11335] manager.c: Manager unregistered action QueueRule
[2023-06-12 09:51:26] VERBOSE[11335] manager.c: Manager unregistered action QueueSummary
[2023-06-12 09:51:26] VERBOSE[11335] manager.c: Manager unregistered action QueueAdd
[2023-06-12 09:51:26] VERBOSE[11335] manager.c: Manager unregistered action QueueRemove
[2023-06-12 09:51:26] VERBOSE[11335] manager.c: Manager unregistered action QueuePause
[2023-06-12 09:51:26] VERBOSE[11335] manager.c: Manager unregistered action QueueLog
[2023-06-12 09:51:26] VERBOSE[11335] manager.c: Manager unregistered action QueuePenalty
[2023-06-12 09:51:26] VERBOSE[11335] manager.c: Manager unregistered action QueueReload

2

That’s a controlled shutdown.

Also, if the server machine rebooted, the OS level log files will be needed.

3

look in /var/log/messages (perhaps /var/log/syslog) at about that time

4

this is what is in messages:
Jun 12 09:51:16 freepbx systemd: Stopping Session 18624 of user asterisk.

Jun 12 09:51:16 freepbx systemd: Stopped Stop Read-Ahead Data Collection 10s After Completed Startup.

Jun 12 09:51:16 freepbx systemd: Stopping Availability of block devices…

Jun 12 09:51:16 freepbx systemd: Stopping RPC bind service…

Jun 12 09:51:16 freepbx systemd: Stopping Session 18621 of user asterisk.

Jun 12 09:51:16 freepbx systemd: Stopping LSB: DAHDI kernel modules…

Jun 12 09:51:16 freepbx systemd: Closed LVM2 poll daemon socket.

Jun 12 09:51:16 freepbx systemd: Stopped target RPC Port Mapper.

Jun 12 09:51:16 freepbx chronyd[733]: chronyd exiting

Jun 12 09:51:16 freepbx systemd: Stopping Fail2Ban Service…

Jun 12 09:51:16 freepbx systemd: Stopped target Multi-User System.

Jun 12 09:51:16 freepbx systemd: Stopping Dynamic System Tuning Daemon…

Jun 12 09:51:16 freepbx systemd: Stopped Resets System Activity Logs.

Jun 12 09:51:16 freepbx smartd[706]: smartd received signal 15: Terminated

Jun 12 09:51:16 freepbx systemd: Stopping SYSV: Flash Operator Panel…

Jun 12 09:51:16 freepbx smartd[706]: smartd is exiting (exit status 0)

Jun 12 09:51:16 freepbx systemd: Stopping Initialize hardware monitoring sensors…

Jun 12 09:51:16 freepbx modprobe-r: No sensors with loadable kernel modules configured.

Jun 12 09:51:16 freepbx modprobe-r: Please, run ‘sensors-detect’ as root in order to search for avail$

Jun 12 09:51:16 freepbx dnsmasq[1027]: exiting on receipt of SIGTERM

Jun 12 09:51:16 freepbx systemd: Stopping Postfix Mail Transport Agent…

Jun 12 09:51:16 freepbx systemd: Stopping NTP client/server…

Jun 12 09:51:16 freepbx systemd: Stopping Software RAID monitoring and management…

Jun 12 09:51:16 freepbx systemd: Stopping Self Monitoring and Reporting Technology (SMART) Daemon…

Jun 12 09:51:16 freepbx systemd: Stopping DNS caching server…

Jun 12 09:51:16 freepbx systemd: Stopping Inotify System Scheduler…

Jun 12 09:51:16 freepbx systemd: Stopping Xinetd A Powerful Replacement For Inetd…

Jun 12 09:51:16 freepbx systemd: Stopping Command Scheduler…

Jun 12 09:51:16 freepbx systemd: Stopping Job spooling tools…

Jun 12 09:51:16 freepbx xinetd[5780]: unexpected signal: 18 (Continued) in signal pipe

Jun 12 09:51:16 freepbx systemd: Stopping Sangoma PnP Service…

Jun 12 09:51:16 freepbx systemd: Stopping FreePBX VoIP Server…

5

go back a minute or two, that is an orderly shutdown in progress after asterisk is stopped

7

this is going back to the start of the day in messages:

Jun 12 06:52:01 freepbx auditd[681]: Audit daemon rotating log files

Jun 12 07:14:55 freepbx dhclient[958]: DHCPREQUEST on eth0 to 10.0.1.1 port 67 (xid=0x4268c073)

Jun 12 07:14:55 freepbx dhclient[958]: DHCPACK from 10.0.1.1 (xid=0x4268c073)

Jun 12 07:14:57 freepbx dhclient[958]: bound to 10.0.1.20 – renewal in 33613 seconds.

Jun 12 09:51:16 freepbx systemd: Stopping Session 18624 of user asterisk.

Jun 12 09:51:16 freepbx systemd: Stopped Stop Read-Ahead Data Collection 10s After Completed Startup.

Jun 12 09:51:16 freepbx systemd: Stopping Availability of block devices…

Jun 12 09:51:16 freepbx systemd: Stopping RPC bind service…

8

Is this a cloud server? The data center operator could have initiated an ACPI shutdown.

Is it a local physical server? Connected to UPS perhaps? Do you have a cat nearby? :slight_smile:

9

Its a local physical server. Not connected to a UPS, cat is too fat to jump up that high.

10

Have you noticed any ectoplasm near the box ?

11

There’s some dark slime around the enclosure. Its very suspicious.

12

Who you gonna call ?

13

Ghostbusters!

I found this in the cron file around the time of the reboot, is this useful:

Jun 12 09:51:01 freepbx crond[1048]: (asterisk) RELOAD (/var/spool/cron/asterisk)

Jun 12 09:51:01 freepbx CROND[10472]: (asterisk) CMD ([ -e /usr/sbin/fwconsole ] && sleep $((RANDOM%30)) && /usr/sbin/fwconsole queuestats --syncall >> /var/log/asterisk/queuestats$

Jun 12 09:51:01 freepbx CROND[10473]: (asterisk) CMD ([ -e /usr/sbin/fwconsole ] && sleep $((RANDOM%30)) && /usr/sbin/fwconsole job --run --quiet 2>&1 > /dev/null)

Jun 12 09:51:16 freepbx incrond[21971]: (system::sysadmin) CMD (/usr/bin/sysadmin_manager sysadmin.reboot)

Jun 12 09:51:16 freepbx sysadmin-hook[10500]: sysadmin hook started - [“/usr/bin/sysadmin_manager”,“sysadmin.reboot”]

Jun 12 09:51:16 freepbx sysadmin-hook[10500]: Security check passed. Running '/var/www/html/admin/modules/sysadmin/hooks/reboot ’

Jun 12 09:51:16 freepbx crond[1048]: (CRON) INFO (Shutting down)

Jun 12 09:51:16 freepbx incrond[21971]: stopping service

Jun 12 09:52:11 freepbx incrond[707]: starting service (version 0.5.10, built on Apr 23 2019 17:47:01)

Jun 12 09:52:11 freepbx incrond[711]: loading system tables

Jun 12 09:52:11 freepbx incrond[711]: loading table legacy

Jun 12 09:52:11 freepbx incrond[711]: loading table local

Jun 12 09:52:11 freepbx incrond[711]: loading table sysadmin

Jun 12 09:52:11 freepbx incrond[711]: loading user tables

Jun 12 09:52:11 freepbx incrond[711]: loading table for user root

Jun 12 09:52:11 freepbx incrond[711]: ready to process filesystem events

Jun 12 09:52:17 freepbx crond[1051]: (CRON) INFO (RANDOM_DELAY will be scaled with factor 91% if used.)

Jun 12 09:52:17 freepbx crond[1051]: (CRON) INFO (running with inotify support)

Jun 12 09:52:21 freepbx incrond[711]: (system::sysadmin) CMD (/usr/bin/sysadmin_manager )

Jun 12 09:52:21 freepbx incrond[711]: (system::sysadmin) CMD (/usr/bin/sysadmin_manager firewall.stopfirewall)

Jun 12 09:52:21 freepbx incrond[711]: (system::sysadmin) CMD (/usr/bin/sysadmin_manager )

Jun 12 09:52:21 freepbx incrond[711]: (system::sysadmin) CMD (/usr/bin/sysadmin_manager firewall.stopfirewall)

Jun 12 09:52:21 freepbx sysadmin-hook[1839]: Need a hook param, not supplied.

Jun 12 09:52:21 freepbx sysadmin-hook[1842]: sysadmin hook started - [“/usr/bin/sysadmin_manager”,“firewall.stopfirewall”]

Jun 12 09:52:21 freepbx sysadmin-hook[1842]: Security check passed. Running '/var/www/html/admin/modules/firewall/hooks/stopfirewall ’

Jun 12 09:52:21 freepbx sysadmin-hook[1846]: sysadmin hook started - [“/usr/bin/sysadmin_manager”,“firewall.stopfirewall”]

Jun 12 09:52:21 freepbx sysadmin-hook[1846]: File ‘/var/spool/asterisk/incron/firewall.stopfirewall’ didn’t exist. That’s unpossible (Are you running incron commands manually?)

Jun 12 09:52:21 freepbx sysadmin-hook[1845]: Need a hook param, not supplied.

14

image
That would have come from clicking this Reboot button in the Sys Admin module.

15

And typed ‘reboot’

16

I swear to all of you, I did not hit REBOOT.

17

Seems like the logs are saying somebody did. You might be able to investigate further by looking at /var/log/httpd/acccess around the same time the reboot happened.

That might give you the IP of the device that accessed the web server around that time.

18

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.