System suddenly restarted, stuck in startup/reboot loop


#21

I currently have these items ready to purchase, which I’m hoping could result in the same thing (an image of the failing drive).

My thought was:

  1. use Ultimate Boot CD to load Linux on any of our Windows PC’s,
  2. put the failing IDE drive in that USB enclosure,
  3. connect both the USB enclosure and the new WD Gold SATA drive to the Windows PC running boot CD’s linux environment,
  4. use Clonezilla or Parted Magic, which come included on Ultimate Boot CD 5.3.8,
  5. assuming I can get a clone of the failing drive onto the new gold drive, use that IDE-SATA 40-pin adapter to hook up the new gold drive back into the phone server.

OR, possibly, instead of that IDE adapter, I wonder if I can drop the new gold drive into a SATA-USB enclosure and set THAT as the boot device, and boot from the removable enclosure…


#22

Crap, just realized that with the lack of space inside that case, there’s no way I can fit a new 3.5" SATA drive plus the IDE adapter.

I’d need either an actual IDE drive, or that WD Gold SATA drive in a USB enclosure plugged into the front of the server case, or a 2.5" or SSD with the IDE adapter (not sure about compatibility with an SSD, though, so that may be out).


#23

I was able to download a backup!

I went to turn on the server to try the USB and this time it actually booted all the way up. I ran over to my PC and logged in to the admin, and ran a full backup.

I think/hope I did it correctly… I just dragged Full Backup, CDRs, System Audio and Voice Mail over, ran the backup to local storage, and in the restore menu I downloaded a 47 MB tar.gz file… does this sound about right?


#24

While I’m logged in, I was able to check my version:

Asterisk 11.7.0 built by root @ jenkins-el6-32.schmoozecom[dot]net on a i686 running Linux on 2013-12-18 22:22:09 UTC

PBX Firmware: 4.211.64-2
PBX Service Pack: 1.0.0.0

Anything else I can or should do right now while I’m logged in? I’m trying to gather as much info, back up, and salvage as much as possible while the system is up, before the hard drive reboots itself again. So anything I’m able to do while I have access that will make restoring a backup or cloning this drive or whatever I have to do, would be awesome. :grinning:


(Lorne Gaetz) #25

Your only practical use for the backup tarball you just made would be to restore to new 15 install, I doubt you would be able to install a new 2.11 system today at all.

The only thing not included in the backup tarball is call recordings (if any) located in /var/spool/asterisk/monitor. If those are important, you should take steps to get a copy off the system.


#26

Thanks, we did not have call recording set up, so no worries there.

I found this section of legacy downloads on the freepbx site:
https:// downloads. freepbxdistro. org /ISO/

FreePBX-4.211.64-9-i386-Full-1385934267.iso

But if I can create a clone of the failing drive, perhaps I won’t need this at all?

I read that “Once the download is complete, you will have a full backup of your FreePBX server that can be restored on a FreePBX server with the same major release version.” So it sounds like I wouldn’t be able to restore my version into a new build anyway, which probably rules out moving seamlessly to a cloud-based solution with our backup data intact…

Big picture, we are okay moving to a new server (updated FreePBX, cloud-based, whatever) but I’d hate to lose everything we have.


(Lorne Gaetz) #27

I haven’t tried to install from a legacy ISO that old, but none of the repos are online so i would be surprised if it results in a usable install, but you have only your time to lose by trying.

The backup/restore module in 15 has a brand new feature that allows you to restore backup tarballs made on earlier versions. You will find thousands of instances in the wiki and here in the forum saying you must match the restore version to the backup, but that’s no longer true in 15.


#28

Ide to sata adapter should be compatible with any hard drive.

You could clone the disk now and then try to do the inline updates, using the scripts to move up one version at a time.


(Dave Burgess) #29

How big is the hard drive?

Whatever you do - do not reboot.

My suggestion would be to get a USB drive (or USB->ATA Drive cradle) and connect it. The machine will recognize the new drive without restarting anything.

Once there, you can log into the console as ‘root’. From there, you have lots of choices.

  • Use ‘dd’ to copy the failing drive onto another drive.
  • Just ‘cpio’ the drive onto another drive.
  • Create a “tar” file of the entire drive and save it off onto another drive.

With the files saved, you grab a Goodwill machine and drop a hard drive in. Boot from a rescue disk, or install a new FreePBX image onto the drive. You can then use the old drive as a ‘forensic’ source to create a new installation, or just drop the hard drive into the new machine.

Note that there are still a few “IT Guy” things, you’ll need to know. For example, your current hard drive has a ‘moniker’ like /dev/sda. If you mount a new hard drive (through a USB interface) and it gets added as ‘/dev/sdb’, you can use something like ‘dd if=/dev/sda of=/dev/sdb bsize=512m’ to literally clone the old drive onto a new drive. This would copy everything - boot sectors, files, everything.

Another option would be to create a bootable drive (install a Legacy ISO onto a new drive) and mount it using “mount /dev/sdb /mnt”. After that, you could use something like “tar -cvzf / --exclude /mnt | tar -xvzf - -C /mnt”. This will copy all of the files from the old hard drive to the new one without duplicating the files you are copying onto /mnt.

Rule 1 - do not power that machine down. After that, everything is doable.


(John Jarrett) #30

IF the hard drive fails again while in use, one thing that has saved my bacon more than a couple times is take the hard drive out and place it in a refrigerator or freezer for awhile. I have been able to boot the drive in about 50% of my hdd failures after letting it get a good cooling.


#31

Thanks so much for the replies and advice!

It’s a 40GB hard drive, but only a few GB are actually in use.

If I run disk filesystem (df) then it shows the following:

I’m sorry I don’t fully understand what command(s) to use to back up the hard drive, as it appears to have a couple partitions?

And if I insert a USB flash drive in a USB port, the screen shows:

# sd 3:0:0:0: [sdc] Assuming drive cache: write through
sd 3:0:0:0: [sdc] Assuming drive cache: write through
sd 3:0:0:0: [sdc] Assuming drive cache: write through

Given that info, what would be the exact “dd” command I need to run to copy the hard drive to the USB drive?

Also, since it’s a 40GB hard drive, do I need a 64GB USB drive? Or since only 3 or 4 GB are in use, can I use, say, an 8GB or 16GB thumb drive I have sitting around, and it only copies the “used” part of the drive?

Final question: you mentioned “This would copy everything - boot sectors, files, everything.” In theory this sounds like what I want — make a 100% duplicate drive, plug the new copied one in and everything is good to go as it was, except on a spiffy new drive. But IS this what I want? Or will that also copy over whatever was failing on the old one? Or if it’s a mechanical/hardware problem (the physical drive going out) does that not affect the data/file system/boot sectors/etc that would be cloned?


#32

Go into single user mode

init 1

Before yo do anything

did if=/dev/sda of=/dev/sdc block sync noerror

Wait about n hours. Then fix-up /dev/sdc to boot and chkdsk all the partitions on that copied drive the freezer thing is a tried and tested method over 70 years of electronics. All this is done for you with clonezilla live


(Moussa) #33

@cwaz13 the good thing happening so far is you have a lot of options and support :+1:.

If I were in your place (setting FreePBX for calling with 10 extensions), I would build new system FreePBX 15 (with the help from here, Google, wiki documentation, and YouTube) and move things over (not necessary cloning, but you can if you really trust your old system). I would keep the old system on the side until I am confident that I have everything set up the way I want. The reason by doing so, I will know what is inside my system (safety) and how it is working and it will be up to date. Building 10 extensions with IVR should not take much of time.

You will still have access to the original system in case you need access to old files / reports.


#34

Thanks all. Unfortunately the server/hard drive did not stay up long enough for me to clone before it started its reboot loop again. :disappointed_relieved:

So I dropped it into an IDE-USB enclosure and plugged it into a system running Ultimate Boot CD. I wasn’t able to use the block sync noerror commands, but dd if=/dev/sda of=/dev/sdc did (in theory) work to give me a copy of the drive.

My first attempt to plug it in, using a new SSD with that SATA adapter, did not work:

I was hoping this might just be a problem using an SSD with the SATA adapter, so now I’m cloning this SSD back to another “new” IDE drive.


#35

If I try to clone it to an image, it says that one of the partitions is broken:


#36

So I was able to clone it to the other IDE drive, and was hoping to run fsck to try to scan/fix any drive errors. But then it says there’s a “bad magic number in the super-block” and "the superblock could not be read or does not describe a correct ext2 filesystem."


(Dave Burgess) #37

I hate to say it, but now might be the time to just start a new implementation and hand-jam your stuff. Ten phones and a single SIP provider should take less than an hour.


(Luke C) #38

I am with Dave, be a lot less headache to start migrating. If you are not comfortable, hiring a consultant is always in the cards. If it were my business, I would consider it a good investment getting to a more current version.


#39

Man, I can’t buy a break. After a few more attempts at hard drive repair and no luck with what appears to be a damaged/corrupted partition or superblock, I figured I’d go ahead and install FreePBX 15 with Asterisk 16 on a separate hard drive. Install to the drive went fine, but when I dropped it into our old server, I was greeted with:

This kernel requires an x86-64 CPU, but only detected an i686 CPU.
This processor is unsupported in CentOS 7.

Is there any workaround for this so we can plug everything back into our server? I think I can figure out setting this all up — new IVR, extensions, etc are annoying to redo, but doable — but it pretty much hinged on being able to install a new hard drive and be on our way without needing a whole new server.

EDIT: I’m taking a wild guess that the FreePBX 15 hard drive I just put in there is completely useless to me, and I need to start over and install:

32-Bit-FPBX

from https://www.freepbx.org/downloads?

Unless there is some way to make the newer version that I installed (STABLE SNG7-PBX-64bit-1910 - Release Date: October 2019 FreePBX 15 • Linux 7.6 • Asterisk 13 or 16) work on this i686 system?


(Lorne Gaetz) #40

Sheesh, is this thing even Y2K ready? This server’s days as a productive member of your office are over.