System suddenly restarted, stuck in startup/reboot loop

Hi, our phone server suddenly started rebooting itself yesterday. I hooked up a monitor and see that it can’t get past the blue loading bar with SHMZ 6.4 in blue letters in the lower right. The blue bar shows loading progress, and when it gets to the end the system just reboots again. It does the same thing whether I allow SHMZ to load or if I intervene and quickly select CentOS. Side note, if I hit Ctrl-S then I can load the setup menu; there’s not really anything there I can change, but it stops the reboot loop from happening. The reboot specifically happens during load. A couple times it’s gotten past the blue bar and shown some boot text, but then reboots again no matter what.

Does this sound like a software or hardware issue? I was thinking the hardware is kinda old and considered cloning the hard drive to a nice new one, in case it was hard drive failure. But if it is a software problem I might need a little help.

Any way I can bypass or troubleshoot this? I’m not super familiar with linux but I can find my way around. Any help would be great. Thank you!

You can see exactly when it is restarting by pressing the ESC key while the bar is moving.

By any chance is the FreePBX server connected to a UPS? Maybe the UPS software is not detecting the connection to the UPS and it is issuing a reboot, which would not be the default behavior, as it should issue a shutdown, but it is a possibility.

It could also be a kernel panic or crash.

2 Likes

Awesome, thank you. I was able to see the moment it restarts:


A problem occurred while reading the config file /etc/prosody/prosody.cfg.lua
Error: Error loading included /etc/asterisk/prosody_additional.conf: /etc/asterisk/prosody_additional.conf: No such file or directory

Then a red [FAILED] right before “Starting fail2ban” and it reboots.

I’m currently looking up the Prosody stuff, but am kind of flying blind.

It is, in fact plugged into a UPS. I did notice that during startup it said OK next to UPS detection, but I will also try plugging it straight into an outlet.

edit the kernel boot params

remove rhgb and --quiet iirc then watch the failure

it should give you a better idea of the true failure
check this out as well
https://www.thegeekdiary.com/centos-rhel-6-how-to-change-the-verbosity-of-debug-logs-during-booting/

Pull the disk and run diagnostics with CrysalDiskInfo or a vendor tool specific to your storage. Smells like a failed HDD or just rotten luck.

I’m a “VMware guy” so all my installs use VMware ESXi free on USB as the base OS and I simply stack a FreePBX VM on top of that. Primary SSD/HDD datastore holds the PBX and it’s backed up via ghettoVCB every week or daily depending on client to a secondary HDD datastore. Just something to think about if you need to redo this for better/quicker recovery. Nice part about this setup is if the primary storage fails completely you can just add the backed up PBX as a VM and start it up while you figure out your primary storage situation.

Do you have a FreePBX backup? tar.gz

1 Like

I’ll second @waldrondigital - this sounds like a drive going away.

If you can pull is into a test machine and run a scan on it, I’d do that before doing anything else, unless you can get it to boot. If you can boot it, back it up onto a thumb drive or CD and use that are a restore to put it on a new hard drive.

Can I run a scan on a Windows machine? Or will that cause problems?

I could drop the hard drive into an external USB enclosure and run disk scans on my Windows 10 PC, but I don’t have any other Linux system I could put it into.

Thanks so much guys!

Windows doesn’t include the tools needed to scan the filesystem that is used by the Linux OS.

Well I opened up the system and it’s an old IDE drive, connected with a big flat 40-pin IDE ribbon cable.

Even IF I am able to scan or clone this drive, it’s going to be hard to find a new IDE drive to go inside this thing. I’d hate to put a used drive in there.

In the meantime I still need to find a way to scan it without a separate linux machine to hook it up to.

You would likely have success (or failure :wink: ) with

http://www.system-rescue-cd.org/

1 Like

I feel like I was throwing IDE cables and drives in the trash over a decade ago, this doesn’t look like hardware worth saving. Can you not just start over and restore from a backup?

If you can’t find an ide drive, you could go with a pci to sata card if the BIOS of your server allows booting from pci cards or an ide to sata adapter.

Unfortunately starting over isn’t in the cards for us. This phone system was purchased for our company about 5 years ago by someone who no longer works here. He bought it on Craigslist and it included all the Polycom phones and this server unit which had been pre-configured by the seller. After he left a few years ago, I — as the remaining “I.T.-ish Guy” — had to go in and familiarize myself with it (basically self-taught and googled a lot to figure out how to use FreePBX). I don’t know Linux well either, but can get by with instruction. So I can find my way around and am tech-savvy enough to figure some of this out… but unfortunately am in no way knowledgeable enough to set up a new system or start over.

I’m honestly not sure if there is a backup unless it was pre-configured by one of those guys. I think right now my best (only) course of action, given my skill set, is to try to repair the existing drive and see if I can make a clone of it. I can get an IDE ISB enclosure. Amazon does have some IDE drives; or maybe I can use something like this so I can use a SATA or SSD?

But I’m sure making a duplicate of the hard drive won’t matter if it has errors on it. Downloading that System Rescue disc now, if that will let me boot from the USB in the front of the server box.

That adapter should work.

@cwaz13
How many extensions do you have?
How do use the phone systems? calling? faxing? anything else?

We have 9 extensions. It’s for a small business. Calls only; we don’t fax through the system. Not a huge call volume, but the phone is ringing pretty consistently all day, so several years ago they upgraded from just having a receptionist picking up to this system so we could have an IVR and all have our own extensions.

Our office manager is getting quotes on a new system or something like a Hosted FreePBX solution (freepbxhosting-dot-com) … but we’re not sure what’s in the cards financially right now. We also don’t want to lose everything from our current system… voicemails, etc.

Would you consider building the FreePBX on the cloud (for good or while you troubleshooting your exciting system)? It will take you about a day to finish it up. The first year will not cost anything, the second year will be very affordable.

[How-to] Install & Secure FreePBX Distro (with commercial modules) on Google Compute Engine

You should be able to do it yourself. I am not an IT either.

1 Like

Thank you for the suggestion. Would that be compatible with all of our existing Polycom phones?

Again, with me not knowing exactly how these were set up when they bought them, the one thing I have deduced is that each physical phone is pre-configured (kind of hard-coded, for lack of better term?) with an extension, etc.

1 Like

On the phones you need to change two things: The Server 1 (Address) and Authentication (Domain & Password)

On the FreePBX you can either create a new system / extensions that mimic your old one or if things work well and you have backup from the old system, you can backup and restore.

As for the voicemails hope someone here can help with that.

If you have access to USB ports

https://clonezilla.org/liveusb.php

And two 64gb thumb drives, you would likely end up with at least an image of a failing drive.

At best a bootable USB stick to boot a $50 rescue machine from Goodwill. You won’t have to worry about the nature of the hard drives, clonezilla will sort it out almost always correctly.

There will need to be some reconfig of networking on the new machine and any commercial modules will need attention