Best way to create a backup hard disk

[herbw]

My general principle for every server is to have at least two full backups – one onsite, and one offsite. I believe freepbx merits the same backup strategy.

It appears as if the recommended way to create a backup is to install freepbx on a spare workstation, upgrade all modules to the latest version, and then restore an existing backup to that spare workstation (Best way to backup entire FreePBX 6.12.65/CentOS 6.5 server?). (I discovered the hard way that if you do not first upgrade all modules prior to doing the restore, the resulting configuration may be invalid, requiring you reinstall freepbx and start over.)

This is somewhat time consuming if you wish to create two backup disks. But even this approach does not give you an exact clone of the freepbx hard disk. Comparing the output of yum list installed shows that the installed packages on the two systems do not match. And there may be other locally installed items such as scripts that are not part of the backup.

I looked at rsync as a possible approach to clone the hard disk, but since freepbx uses the LVM file system, and since I have not discovered any way to simultaneously mount two hard disks containing the same LVM Volume Groups (as would be the case with a clone of the hard disk), I can not find any way to make rsync work. (Perhaps it would be possible to create a clone of the freepbx hard disk using rsync to a different workstation – if so, this would be an acceptable solution.)

So is there a good way to simply and reliably create multiple clones of the freepbx hard disk? Note that I need to be able to do this without shutting down freepbx during the process. I understand, and am willing to accept the possibility that cloning the hard disk in a running system may result in inconsistencies – for example, discrepancies between the contents of voice mail and the contents of the CDR records.

Thanks for any recommendations.

[dicko]

Well, rsync doesnt care about partitioning, it works at the filesystem level, you can trivially rsync a zfs file system on an lvm partitioned disk, no matter what it is named, to a btrfs mount on a nas device on the other side of the world .

As to realtime online cloning, i have always used mondorescue. it will hapily clone a hp hardware array to a raspberry pi, using a disk image,an nfs mount or a usb thumb drive booting to mindi, some complain it doesn’t work on ‘the distro’ but i have no reasonable way of testing that and there is no logical reason ever given as to why it fails to install.

Another higher level solution is to run rsnapshot (kindofa super rsync wrapper using hardlinks to minimize disk use) from your servers to “some where in the cloud”, /var/{spool,lib}/{asterisk,mysql}/* and /etc/asterisk/* will get most everything, All modules and web behavior is in /var/www/html/admin/* and a little in /etc/{httpd,apache2}/* but be wary, these have been the source of problems when upgrading or having been penetrated (yes we have been told that that is not possible anymore but . . . .) .mysqldumping the asterisk and asteriskcdrdb tables gets the rest if needed .

Another feasable solution is to use snapshot-able filesystems for your / mount (zfs,xfs,btrfs. . .) and, well, snapshot it when you feel the urge

[herbw]

Thanks, dicko, for your suggestions.

I now have a backup solution that almost works. I’ll describe the method I use, since it may be helpful to others. I’ll then describe why it fails, and hopefully someone can recommend an approach to solve the last remaining problem.

I installed freepbx on spare hard disk, but did not perform any updates or configuration once the install completed.

I installed Ubuntu on a spare PC. I explicitly do not use LVM for the Ubuntu file system, to ensure that the Ubuntu file system and the freepbx file system do not interact with each other. I boot the Ubuntu system, with the freepbx disk attached.

I then ran the following commands:

sudo mount /dev/mapper/SangomaVG-root /mnt
sudo rsync -aAXve ssh --delete --exclude=/dev/* --exclude=/proc/* --exclude=/sys/* --exclude=/tmp/* --exclude=/run/* --exclude=/mnt/* --exclude=/media/* --exclude=swapfile --exclude="lost+found" --exclude=".cache" root@myasteriskhostname:/ /mnt
cp -p etc-sysconfig-network-scripts-ifcfg-eth0 /mnt/etc/sysconfig/network-scripts/ifcfg-eth0
umount /mnt

The problem is that this results in a corrupted MySQL database, forcing freepbx to boot into emergency mode. In hindsight, this is really not at all surprising, since rsync is not a valid way to copy a MySQL data base.

So, here are my questions:

1. What additional files do I need to exclude from the rsync to ensure that no MySQL databases are copied. (Is there a specific filename pattern or suffix that identifies a MySQL database?)

2. What is the best way to copy those databases after the rsync completes? (I do use freepbx Backup and Restore to create backup copies of my databases.) I suspect that it would be preferable to restore my freepbx databases while freepbx is not running, to ensure that there are no inconsistencies between the versions of various freepbx components, and the versions of configuration information recorded in the database.

[dicko]

rsync’ing /var/lib/mysql while the mysql/mariadb service on either end is running is risky but you could

ssh [email protected] "mysqldump -pyourpassword -uroot  --databases asterisk asteriskcdrdb" > restore.sql
mysql -u root -pyourpassword  < restore.sql

on the backup machine instead

[herbw]

Thanks, dicko. I will try this.

[dicko]

To script it you might want to

ssh-copy-id [email protected] -p theportsshisrunningon

the necessary key to make ssh password less

[herbw]

Thanks dicko for your assistance. It turns out that if I ssh [email protected] the mysqldump does not require a password.

Since the mysql < restore.sql must be done on the freepbx workstation rather than the Ubuntu workstation, is there an easy way to tell freepbx to boot but not start freepbx, so that freepbx does not go into emergency mode? Perhaps there’s a file or switch that freepbx checks when starting up.

Also, is there any way to identify the MySQL files (by filename or extension) so that I don’t even copy them using rsync?

[dicko]

This is getting a little convoluted, did you ever try mondo?

“is there an easy way to tell freepbx to boot but not start freepbx”

makes no sense at all to me.
all the Asterisk mysql state is in /var/lib/mysql/asterisk and less critically /var/lib/mysql/asteriskcdrdb

There is (certainly was) a Sangoma module for “High Availablity” that might suit you better.

[herbw]

Thanks, dicko. You are correct. It is getting a bit convoluted.

I’m not sure that the Sangoma module for “High Availablity" meets one of my critical requirements, which is having an offsite backup.

[dicko]

again, is installing mondo not within your abilities? it will do everything you want .

[herbw]

Thanks, dicko. I will take a look at mondo.

[herbw]

I now have a good (not great, but good) solution for creating a backup (clone) of my freepbx hard disk.

There might potentially be three different environments in which the clone could be created:

• Create the clone while the freepbx system is running.
• Create the clone while the freepbx system is running, but with asterisk and freepbx stopped (fwconsole stop).
• Create the clone while the freepbx system is not running, but the hard disk is mounted on a different system (e.g. Ubuntu).

I was unable to find a simple and reliable solution in the first environment, because of difficulties backing up and restoring the MySQL databases. I did not attempt to find a solution in the second environment, because it did not offer sufficient advantages over the third environment, and I did not have confidence that there would not be any latent issues. I have Ubuntu installed on a small SSD in my freepbx workstation, so it is easy for me to shutdown freepbx and boot Ubuntu for the purposes of creating the backup (clone).

The only reason that I rate my solution as good rather than great is that I need to shutdown freepbx to perform the operation. I did that last night, since Saturday after midnight seemed like a reasonable time to take the system down for maintenance.

dicko suggested that I try mondorescue. This solution looked promising, but I ultimately abandoned it because I could not find any evidence that this software could safely backup MySQL databases on a running system, rendering this approach no better than rsync, which I already had mostly working.

My solution involves two steps. The first step creates a freepbx backup on my file server, and must be performed while freepbx is not running:

#!/bin/sh
sudo mount /dev/mapper/SangomaVG-root /mnt
sudo rsync -aAXve ssh --delete --exclude="lost+found" --exclude=".cache" /mnt [email protected]:/path-to-backup
sudo umount /mnt

The second step can be performed on a spare workstation, and therefore does not require shutting down freepbx:

#!/bin/sh
sudo mount /dev/mapper/SangomaVG-root /mnt
sudo rsync -aAXve ssh --delete --exclude="lost+found" --exclude=".cache" --exclude="fstab" --exclude="ifcfg-eth0" [email protected]:/path-to-backup/ /mnt
sudo umount /mnt

Some notes about the restore step:

• Excluding fstab ensures that the UUID (Universally Unique Identifier) for the hard drive is not altered. (I am not sure what the consequences would be of altering this.)

• Excluding ifcfg-eth0 ensures that the network configuration (IP address, etc.) of the target workstation is not altered. This allows the target workstation to be booted and tested without creating a duplicate IP address on the network. If you need a clone with the exact same IP address as the existing workstation, for example to replace a failed hard disk, do not exclude ifcfg-eth0 .

• The restore step includes a slash after path-to-backup/ – the backup step does not include this slash. The slash on the restore step ensures that the restore copies the contents of the backup directory rather than the directory itself.

[dicko]

Another solution might be

http://relax-and-recover.org/

It takes a little RTFM’ing and I haven’t used in a while but . . . .

[herbw]

Thanks, dicko. It is always helpful to have an alternative if one option does not solve your problem.