Hello, I take full backups of my FreePBX 14.0.13.40 server every night. Bare Metal and File lever. As well, I dump the database to the FreePBX server, with the built-in tool IE:
Admin–>Backup and Restore
Outside of this, I also wanted to create a hot-spare. So I used VMConvertor Standalone to do a HOTCLONE of the Our Master FreePBX server above. The hot-copy went perfectly. As a test, I brought down the master, and brought up the hotapre copy. Everything seemed to go well, and I could even make phone calls…however, after a few minutes, I got a message saying the system could not connect to MySQL. I rebooted, and sure enough I got a message saying the system could not connect to MySQL, so the system seemed to be running…however NO FreePBX gui and no MySQL connection. (I seemed to be able to make calls still)
Now, this is more than likely because I did the hot clone of MySQL/MariaDB and maybe it was writing records…but does anyone have any suggestions? Has anyone seen this before, should I be doing something when I bring up the hot clone, so MySQL/MariaDB is happy?
Have you checked if MariaDB/MySQL is running?
Also, check the boot logs to see if any anomalies reported during the initial boot. Could be that the cloning of a live system with an active file system might have had some file system issues that were automatically repaired but perhaps the DB was damaged enough to keep the DB from running.
No, I had not. I hope this doesn’t sound too ignorant, but being that, I’m quite experienced with Linux, but since sangoma, has chosen a flavor of Linux in the back ground, can I just use the standard MySQL/Maria start command, or is there a specific process for FreePBX to start MySQL/maria DB.
I have another question based on that. I figure if I dumped the whole DB with a script every night IE all of this:
If I had any problems, with the DB I could just RESTORE all of this, and in my eyes, if there was DB Damage, I should be running again. What do you think. This seems the safest, since I’d always have a clean DUMP of the whole Database. What do you guys think? I’ve always heard so many horror stories of people loading the DB from backup and restore and having issues.
If you can pin down the issue to mysq/mariadb not being correctly copied by the cloning due to being in use (or maybe just because the DB files in memory haven’t been written to the filesystem when cloned) - do a mysqldump before the cloning and then using that to restore with on the hot clone is probably the way to go.
Success. I wanted to test our centralized backup system with this anyways, for disaster recovery. And I felt it was best to test a BARE METAL scenario, with our centralized backup system. We use SEP SESAM, and it’s always been awesome. However, it’s worked so well, I have not had to do many full Linux bare metal recoveries. So here is what I did. Just some background about the backup system:
SEP can do full file-level recovery, and Bare Metal recovery of the whole system
I use the relax and recover project to make a skeleton bootable iso of the network and file system components, so that I can just boot into this 300MB skeleton, and then any BLANK system can boot with its original IP and relax and recover will cut the same partition structure.
I created a BLANK VM on a new hypervisor with the same blank virtual HDD size and similar specs
I cold booted the EFI VM, with the skeleton relax and recover skeleton boot iso
The system came up on it’s original IP on the network as it should, and then relax and recover cut the original boot and file partitions, mounting /mnt/local in preparation for the restore
Once this happened, I engaged a COMPLETE RESTORE/Recovery from my latest backup
The restore only took 15 minutes (12.6 GB of data)
I rebooted the VM and it happily booted with EVERYTHING WORKING!
SEP SESAM knows how to deal with the backup of databases, during a hot backup, so that solved that issue, and the system was fully functional.
Thank you everyone for all the hints. I just don’t think VMware convert is robust enough for hot migration with Databases. So I’ll just stick with our centralized backup system. Thanks again everyone! I can now sleep at night knowing I can recover in 15 minutes if I lose our centralized FreePBX.
This might be a silly question, but why not run FreePBX as a VM? That’s how I have our production system deployed. Backups to a local repository and then streamed out to a cloud repo. While I can perform file-level restores, if there were a major hardware issue on the VMware host appliance then restoring the guest VM onto an another host is relatively quick.
Are there major gotchas or performance penalties involved with a FreePBX VM deployment? So far after a couple of years with 5 sites hanging off of it things have run smoothly. Hopefully I’m not jinxing myself by openly stating that…lol.
My System is a VM hence the term BLANK & EFI VM on VMWARE, and it works great. I don’t have VEEM, or any Migration tools for ESXi, and since Relax and recover and SEP SESAM allow me to do file-level restores and Bare Metal restore to a BLANK VM, I figure that’s just as good. Relax and recover is ONLY needed for the Bare Metal restore part, I can do file level recovery anytime. Relax and recover is only an extra 20-second step for Bare Metal. Both the MASTER, and the TEST HOT SPARE were VM’s, and both work GREAT!
PS - The database dump is just paranoia. Paranoia has saved me several times
I’ve been using fpbx VM’s on vsphere for years now. We are using VEEAM to backup the VM but I’ve found that for system crashes, the database is sometimes corrupted - probably due to losing whatever was in memory at the time for both the DB and filesystem. So having a separate backup of the DB can be helpful. I have a cron job that dumps the DB to disk, and that just gets backed up in the VEEAM backup.
hence the term BLANK & EFI VM on VMWARE, and it works great. I don’t have VEEM, or any Migration tools for ESXi, and since Relax and recover and SEP SESAM allow me to do file-level restores and Bare Metal restore to a BLANK VM, I figure that’s just as good. Relax and recover is ONLY needed for the Bare Metal restore part, I can do file level recovery anytime. Relax and recover is only an extra 20-second step for Bare Metal. Both the MASTER, and the TEST HOT SPARE were VM’s, and both work GREAT!