I was able to successfully boot from either drive, as well as replace a drive and rebuild the RAID. I realize this isn’t really a function/feature of FreePBX, as it’s really the OS, so I thought I’d share what I did in case anyone else needed help:
First, make both drives bootable (as explained in the above link) or while your system is up and running. This would be the most important thing you can do BEFORE you have a failure!
1.Make sure you’re root.
2.Input the command “/sbin/grub”, hit Enter
3.Enter the following commands in the GRUB shell:
grub> device (hd0) /dev/sda
grub> device (hd1) /dev/sdb
grub> root (hd0,0)
grub> setup (hd0)
grub> root (hd1,0)
grub> setup (hd1)
grub> quit
Once this is complete, both your drives are bootable should one fail (or you unplug one, etc.).
Now, if there is a failure on the primary boot drive, you’ll still get a GRUB error, but you can easily change the hard drive it’s trying to boot from between hd1 or hd0 (use the ‘e’ option to edit from the GRUB menu).
Now, to replace the drive and re-build the mirror (swap sda/sdb depending on what drive failed):
1.Create the exact same partitioning as on /dev/sda. We can do this with one simple command:
sfdisk -d /dev/sda | sfdisk /dev/sdb
2.Next, add the new partitions to the active RAID:
mdadm --manage /dev/md0 --add /dev/sdb1
mdadm --manage /dev/md1 --add /dev/sdb2
mdadm --manage /dev/md2 --add /dev/sdb3
3.Verify RAID is syncronized using:
cat /proc/mdstat
You can also use these commands to see the RAID status of each:
mdadm --detail /dev/md0
mdadm --detail /dev/md1
mdadm --detail /dev/md2
For example, during re-building of md2 for a new drive it will show:
State : clean, degraded, recovering
Active Devices : 1
Working Devices : 2
Failed Devices : 0
Spare Devices : 1
Rebuild Status : 45% complete
UUID : ffc9b989:beb4db11:2b5b4783:ab923a03
Events : 0.1578
Number Major Minor RaidDevice State
2 8 3 0 spare rebuilding /dev/sda3
1 8 19 1 active sync /dev/sdb3