High Availability Backup and Restore

In our never ending quest to make FreePBX the best phone system that doesn’t require money to buy (and even better than most that do cost money…), allow us to introduce you to the latest features in the Backup & Restore module. Last week we alluded to a critical server error, but left you guessing as to how we address that. This is a big step in that direction.

Along with the kind assistance provided by some customers of FreePBX Professional Support (in the form of time donated for a feature they needed for their business), the boys over at Schmoozecom (disclaimer: including yours truly) have once again surpassed all expectations. As of FreePBX 2.8, (currently in beta - more on that later) the Backup module includes built in support for High Availability clustering!

The following is an interview I had with Me (aka myself), elaborating on the new features:

I: So, what is this new HA (High Availability) feature, and why are we so excited with it?
Me: To ensure that their critical infrastructure maintains a high level of reliability, many business seek out HA solutions of one type of another. To that end, we have introduced an automated capability for a designated server (herein: backup server) to periodically backup a production server (herein: primary server) and to restore said backup on the backup server. This ensures that the backup server always has a fresh copy of the primary server’s settings, and is ready to take over should the primary server fail.

I: Billions of bilious blistering barnacles - that’s not true HA! True HA means the backups in real time and instant hand off between the two servers?!
Me: Actually, according to ( wait for it - you’ll never guess!) Wikipedia, HA is “is a system design … to ensures a certain degree of operational continuity…“. While there may be some benefit to having both the backups and hand-offs in real-time, such solutions can to be overly complex and way above the requirements of many organizations. Additionally, such solution tend to require a unique and complex setup with additional components not usually installed on FreePBX based distributions. Being OS agnostic, FreePBX is best not left to deal with the configuration and maintenance of such solutions. Instead, FreePBX takes a simplistic approach, removing much obscurity and confusion from the picture. Keep in mind, however, that FreePBX’s HA implementation can be used as part of a greater HA solution, perhaps including real-time fail-over support.

I: So, how does FreePBX backup a server?
Me: The backup server can perform backups at any requested interval - ranging from every minute (not recommended for most scenarios) to multiple times per hour/day/week/month, etc. It then restores the backup to itself, wiping away any previous configurations, and replacing them with the latest setting from the production server.

Optionally, the backup server can be configured NOT to restore the setting locally, and act as a storage area for backing up MANY primary servers. In this configuration, the backup server keeps copies of backups handy, ready to be restored locally in the event of a failure to any of the primary servers. Please note that in this scenario, the backup server will continue to function as a backup server, even in the capacity of a surrogate to the failed production server.

I: How does production move to the backup server?
Me: As mentioned previously, on its own - it doesn’t. The sysadmin is required to manually change the ip address of the server so that all peers can find it. This also allows the sysadmin more control of the switching over process, more notification to error’s and an incentive to ensure that the primary server doesn’t fail in the first place!

Additionally, there are services that can do the fail over automatically, but those are beyond the scope of this discussion. Apparently, these can be enabled in certain routers as well.

I: Great! This is the perfect way to backup my CDR’s!
Me: Actually, if you require real-time backups of cdr’s, your probably better off with a master-slave setup for your database (for the cdrs that is).

I: After a backup is restored, what happens to my trunks - will they automatically try to register to my provider and ‘steal’ the incoming calls from the primary server?
Me: No. FreePBX requires that you click the orange bar at the top of the screen for any changed settings to go in to affect. This can be done manually or, if you’re using other elements as part of you HA solution, programmaticly by calling /var/lib/asterisk/bin/module_admin reload. If you have traditional PSTN trunks (not VoIP), additional measures will have to be taken to connect those lines to your PBX and assure they are up and running.

I: Bougainvillea! How do I backup the backup server?!
Me: Exposed now in the GUI are many different options that can be run after a backup is complete. For example, copy the backup to an ftp server, ssh the backup to another host, or even have it emailed to you.

I: Doesn’t the ability to pull a backup off a primary server pose a security risk?
Me: We have gone to great lengths to ensure that your phone system remains safe and secure throughout this process. The backups are all executed over ssh and encrypted with the public key.

I: Public keys rock! (Uh, What are public keys?)
Me: Public/private keys are a method to encrypt data sent over insecure connections - easily considered one of the securest method in the world. See here for more info.

I: Right, I knew that! But just to make sure, can you ‘remind’ me how to set up ssh keys?
Me: That’s probably something better left for the wiki/forums, but here is a very, very, very quick primer (designed for Cent OS systems, assuming asterisk is running as Linux user asterisk):

sudo -u asterisk ssh-keygen
hit [enter]
hit [enter]
hit [enter]

Then copy the public key to the primary server:

sudo -u asterisk ssh-copy-id -i /var/lib/asterisk/.ssh/id_rsa.pub  [email protected]<primary server ip or hostname>

Your output should look something like this:

[[email protected] /]# sudo -u asterisk ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/var/lib/asterisk/.ssh/id_rsa):
Created directory '/var/lib/asterisk/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /var/lib/asterisk/.ssh/id_rsa.
Your public key has been saved in /var/lib/asterisk/.ssh/id_rsa.pub.
The key fingerprint is:
ae:2d:fd:b7:19:d3:e8:34:8a:a9:7d:76:1c:71:c4:a8 [email protected]

[[email protected] /]# sudo -u asterisk ssh-copy-id -i /var/lib/asterisk/.ssh/id_rsa.pub  [email protected]

The authenticity of host 'myserver.example.com (21.158.66.3)' can't be established.
RSA key fingerprint is 8e:ae:6a:49:bb:1b:1b:91:3f:02:4f:65:ab:e7:5e:6b.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'myserver.example.com,21.158.66.3' (RSA) to the list of known hosts.
[email protected]'s password:
Now try logging into the machine, with "ssh '[email protected]'", and check in:

  .ssh/authorized_keys

to make sure we haven't added extra keys that you weren't expecting.
[[email protected] /]#

I: Duh! Now, what do I need to do in the backup module?
Me: Under Remote Backup Options, put in the hostname or IP address of the primary server, the user name (probably root), and the private ssh key (if you followed the steps above, that should be: /var/lib/asterisk/.ssh/id_rsa). To restore the backup immediately, check “Restore to this server”.

I: Great, I’m all set. One last thing - why don’t I see these options in my backup module?
Me: These options were included starting with FreePBX 2.8, currently in beta. Feel free to start beta testing 2.8! Have a look here for more info.

I: Thank you so much for sharing your thoughts with me. Is there anything else you would like to add?
Me: Yes. As of last week there were some issues threatening the continuity of Custom Contexts as of FreePBX 2.8. Congratulation and a tip of the hat to all those that step up to contribute and help resolve the issue!

Moshe Brevda

works great!

Your welcome!

I have a standby FreePBX server . The only difference between the primary and secondary is the i/p address .

Is it possible to have some kind of exclude i/p address at ‘Remote Backup Options’ ?

That way there would be less to change in the event of a switch over.

The primary would never get a backup from the remote,

And when the roles switch [which we do every month to ensure things are working ok] the backup/restore would continue to work.

Do you know what I mean?

Hello
the improvements to the backup module are great! However on a fresh install a remote backup/restore failled due to missing /var/lib/asterisk/backups

so I added this to our Debian FreePBX install script:

this is needed in case you want to o a remote backup/restore

if it is missing the remote backup/restore will fail as ot 2010-07-12

mkdir -p /var/lib/asterisk/backups
chown asterisk:asterisk /var/lib/asterisk/backups

PS: is there a better place to put suggestions like this?

in a ticket with a few more details.

I assume what you are saying is that the backup module is installed on the remote system, but was never run.

Assuming that is the case, it sounds like the backup module does not create the backup directory until it is run locally for the first time, and you are saying that resulted in it failing.

Assuming my above understanding is true, you should file a bug indicating that the backup module needs to create the needed directory in its install script.

But just one more question to check for clarification. Was that the only directory you needed to create on the remote system, or did you need to have a subdirectory with the backup set name as well?

Please provide any of those relevant details in the bug report in the ticket system. Thanks.

is there any chance at all to automate the fail over process? Thanks for this documentation! ITS GREEAAAT, saved my life

when i try to < sudo -u asterisk ssh-copy-id -i /var/lib/asterisk/.ssh/id_rsa.pub [email protected] > I get Permission denied (publickey,gssapi-with-mic).
But it works on other box’s .151 & .152 so there must be wrong permissions on box .150 ? not sure were to look or how to fix.

Gary

Seems like you’re box is configured not to use passwords. You will need to change that momentarily or copy the with some other method. There are a ton of guide out there, this really isn’t the place

Thanks for pointing me in the right place.

Gary.

type=friend
username=yournumber
fromuser=yournumber
secret=yourpassword
host=2talk.co.nz
context=default ; or your own selected context if desired
dtmfmode=rfc2833
disallow=all
allow=ilbc
allow=gsm
allow=alaw
allow=ulaw
;allow=g729 ; only if you have licenses to use it
nat=yes
canreinvite=no
insecure=very ; use insecure=invite,port in later versions of Asterisk such as v1.4

This is way easier than me having to setup a MySQL cluster…

Just to note: I believe this feature was added in 2.8

Has the option of changing the port for the ssh connection ever been addressed as was requested above, (not sure if he put in a feature request) this would be nice to have as I’m sure other people also try to hide there ssh port to the outside world.

Gary.

I havnt seen a feature request for that

best not to post the same thing multiple places,

there is an open issue on this, #5021

With FreePBX 2.8 the Remote Backup Option of the Backup & Restore module worked.
I could make a change on the master server, and then run the backup job on the slave server, with the “Restore to this server” box checked. The change would appear on the slave.

I did the update to 2.9.0rc1.1 along with all the other updates for 2.9 including Backup & Restore 2.9.0.2 now the process fails.

The error in /var/log/httpd/error_log
is scp: /backups/AsteriskP2-backup/10.0.19.27.20110406.15.44.52.tar.gz: No such file or directory. Expect for the updates nothing else was changed.

i am trying to add the tftpboot folder using the additional files feature,
i tried what jonstar mentioned previously using the quotes, but it still doesnt
work.
I want the tftpboot directory and subdirectories.
so i used
"/tftpboot/“
and i also tried
”/tftpboot"

yet i got errors (from httpd/error_log)

/bin/tar: /tftpboot/ : Cannot stat: No such file or directory
/bin/tar: Error exit delayed from previous errors

and the tared files do not include these additional files.
any tips? how can I include this?

thanks,
-gk

just in case someone else comes looking for error logging info, it can be found in the apache error.log on your system as it is generated by the FreePBX code running in php… this is true for backups that you run yourself in realtime via the web interface… scheduled jobs are run as a cron not sure where they are logged to.

The reason i was trying to look for errors was that i was attempting to backup multiple files under Additional Files -> Additional files and folders… I tried multiple methods of separating multiple files (one per line, comma separated, space separated w/ no quotes) and found that each of those methods only backs up the 1st file listed.

So, it might be good to notate in the context help that the proper way to add “additional files” to be backed up; is on a single line, separated by spaces, and enclosed by quotes. Example below:

“/var/log/asterisk/foo /var/log/asterisk/bar /var/log/asterisk/TEST”

where are failures logged to ? I am attempting to get this running with no luck and no feedback from application.

Thanks,
Jon

Ok Thank you for your reply. I will keep playing with it and will let you know if i ever figure it out.