RFC: Backup & Restore renovations

Yes, it is finally here: the long anticipated deep renovations/rewrite of the Backup module is finally on the drawing board, and here is your chance to make a difference! This thread is a RFC (request for comment) to hear what the community finds important and what you guys feel should/needs to be included in the next iteration of the module. Here are the rules for participation:

[list]
[] Please be concise, short, and extremely clear
[
] While we generally dont apply heavy censorship to these forums (if any), THIS THREAD WILL BE MODERATED, as necessary, to keep it strictly on topic. Feel free to take other discussions up in another thread, and even link to it if you feel its absolutely necessary
[] This projects primary sponsor, Schmooze Com, Inc. will have the final say as to wich features will be included. If you would like to guaranty the inclusion of a feature by means of a private sponsorship, please use Profession Support to make your request.
[
] End date for this RFC: June 17, 2011 at 0:00 GMT (unless otherwise noted) UPDATE: Extended until July 1, 2011
[/list]

Could you include in the resulting file-name some flags to indicate what is included in each backup?

I currently run separate jobs for System Recordings, Voice Mail and System Config.

Just as an example:
The filename could include “SR” “VM” “SC” “OP” "AWD"
e.g SR-VM_backup-$date.tar.gz

While not in the filename per se, we have plans for a revamped index that will list the contents of the backup --mbrevda

i would add privileges for e.g. the main sysadmin would be able to set all options and a sub admin would be able to do less and wouldn’t have access to backup sets made by the admin
it could be expanded

Interesting idea. Unfortunately, FreePBX isn’t currently built to handel ‘class of service’ or multi-level permissions/acl’s, making this idea pretty hard to implement at the moment. Thanks for the suggestion! --mbrevda

There are two things I have always wished for, and maybe they are mutually exclusive and maybe not, but I will throw them out anyway.

One would be a page you would go to that would show you a single line corresponding to every single configuration page on FreePBX, with no exceptions or omissions (there should even be some mechanism for third-party modules to tie into this, though obviously you couldn’t force any third-party developer to use it). In the case where there can be multiple items under a heading (extensions, inbound routes, outbound routes, etc.) you could click on a small icon to expand the section and it would list each individual page. Next to each line would be a checkbox. You could check the box next to any page, or for any section if you wanted all the pages in that section. At the bottom there would be a “Select All” button and an “Export to File” button. If you exported to a file, it would create a .csv file of ALL your settings. Basically this would probably be mostly a glorified MySQL export but it would grab any other data it needs as well from wherever that is stored (for example, the Asterisk database).

For the above there should be a corresponding import page that lets you import a generated .csv file, with or without overwriting existing conflicting settings or pages (options should be Overwrite always, Ask, Don’t Overwrite).

My second thought, particularly for the FreePBX distribution, is that there should be some equivalent to the “Time Machine” function on the Mac, where all changed files (or all files on the initial run) are backed up once per hour, to a complete archive that can be used to restore any file inadvertently deleted or corrupted. In case of a problem it should be possible to roll back the system to a previous known good state. But, IDEALLY, you should also at any point be able to generate a full system ISO file that could be burned to a CD/DVD and later popped into a machine (not necessarily the same one) to restore the system. I know this would not be the easiest thing because when installing to a new machine certain configurations might be changed, but that’s my point - when you generate the ISO it should know which files can be copied verbatim and which need to be regenerated during installation of the operating system.

Or maybe you could generate a disk intended to be used only after a base installation of CentOS has been completed, so moving a system would consist of burning a “transfer” CD, installing a basic CentOS/utility package (available from the FreePBX site) on the new machine, and at the end of that installation it would ask for the transfer disk and after loading that you’d have an effective copy of the previous system. In an case, the idea is to make transfer of a functional FreePBX system to new hardware, or restoration of a completely crashed system, as painless as possible. Personally I’d prefer the full system backup that would also grab any “extras” the system admin may have installed and bring them along (those will be wanted too, and it would make things a lot easier if they didn’t have to be reinstalled separately). Again, think along the lines of “Time Machine” under OS X, for those familiar with that program (there’s a similar Linux program called “Back in Time” but the interface isn’t as intuitive as on Time Machine, and also, we would probably need something that can be run from either the web interface or from a SSH session command line prompt, since most users don’t have a GUI installed).

Those are two types of backups I have wished for at various times, so maybe one or both of those will give you some ideas. By the way, thanks for asking for input on this.

I 2nd the motion for a complete system image, currently I use clonezilla which requires a system shutdown and there for is only run once a year and freebpx backups every week. It would be nice to grab an image once a month this way updated freepbx modules will be captured more often making it easier for the reinstallation of a broken machine.

Gary

@ou812 - thanks for your suggestion.

Unfortunately, system imaging is way beyond the scope of what FreePBX can provide. FreePBX currently runs as a limited user (meaning limited permissions) and doesn’t have access to the data needed to do system imaging. Additionally, php - the language that FreePBX is written in - is ill suited (if at all) for such a task.

Currently, it would seem that the scripts are hard-coded to write into /tmp as a “staging post”. There are security arguments that /tmp should be considered ephemeral, and probably be nosuid,noexec if on a physical partition (which some argue is preferable to being in /), some argue that it should be in ram. The trouble is if one where to do that ram thing then there is a fair chance that /tmp would get full before completing a backup., and if on a partition you would have to “pre-think” how big to make that partition if using said ampbackup script

Just my two cents worth.

Regards

While I’m not sure there is an immediate security risk, there is no reason why we can’t move the staging area to something more asterisk-y, say /var/spool/asterisk/tmp. Great idea, thanks! --mbrevda

I request the ability to restore a backup file that is either on your local workstation OR in a directory on the server other than the default (/var/lib/asterisk/backups).

Great idea! Noted. --mbrevda

I would like to see a way to limit the number of backups. When that number is exceeded the oldest file would be replaced.

Also, would there be merit to implementing a incremental backup feature?

Already on the list. No there wouldn’t, explained in next post. Thanks! --mbrevda

Regarding incremental backups:

There are two parts to “core” FreePBX backups: the static parts (a.k.a the FreePBX web files, the generated config files, etc) and the dynamic parts (i.e. the databases that store you settings, voice mails, etc). The required frequency for backing up the static parts is really low - say as low as once a month (remember - you can always download FreePBX again if your system needs to be reinstalled el al). And while the required frequency for backing up the dynamic parts is much higher (once per day/week - depending on how often you back up), the file size when backing up ONLY those items is so small that it incremental backups would be simple inefficient, and cause more harm the help.

While I generally agree with your post Moshe, and voicemail’s are indeed dynamic by nature. I would argue that the /var/spool/asterisk/monitor directory would indeed benefit by incremental backups. I know that FreePBX does not include that structure currently by default, not surprisingly, it can indeed be huge, yet so depended on by many.

So many of us (myself included) have had to add some sort of mechanism outside of Backup/Restore to manage that “statefullness”, so maybe a reconsideration of the engine is appropriate?

Perhaps rsync instead of tar would be more economic in the long run all around.

Unless you are “storage bound”, it would be both quicker, with lower CPU load and generally network ready beyond 127.0.0.1, no?

For example I can rsync /var/spool/asterisk/voicemail/* and /var/spool/asterisk/monitor/* every five minutes on or off site with a negligible load on the busiest server, on a catastrophic, failure there would be almost no friction from the clients about “today’s voice-mails”, the catastrophic failure will probably cause a few more whines though:) .

Just my 2 cents worth.

regards

dicko

Please use a different thread for discussion so that we can keep this one for feature requests (and not “the validity of”) --mbrevda

@dicko - sounds like we are saying the same thing then. Indeed, for large installs (say call centers) where every call and cdr is important, I recommend (and have implemented) a second server to act purely as a “data” server. The cdr’s are pushed in real time with a mysql master-slave configuration, and the call recordings are synced/copied in near real time, by running a post-record script that copies them and a cron based rsync to catch any stragglers.

However, what does this all have to do with backup :slight_smile:

Backup is designed to handel the extraction and insertal of critical bits of data to keep your pbx running. Using the same mechanism, we can generally “overload” backups and have them help out with other, less critical assets (tftpboot, network settings, voice prompts, etc). Heck, we can utilise the same mechanism for ANYTHING that is similar in nature to the core bits - by and large, that means files or databases - but only in the same way: simple extraction and exclusion. FreePBX’s Backup is lacking both the mechanism - and the need - to be able to implement the above incrementally.

Also, please keep in mind that aside for the technical/permission limitations to do real incremental (as I mentioned above), incremental solutions usually work at low level’s, reading certain data directly from the disk (say - file system tables). The order of magnitude (difficulty wise) to implement something similar is so huge, that even beyond FreePBX I doubt you’de ever see anything (written in php) doing incremental backups.

As a final note on this topic, there really aren’t many usages beyond those discussed here (full disc, which is close to imposible, and recordings, which are better implemented with a post recording script or a rsync server) that would benefit from incremental backups. We have discussed this idea internally many times, and always come back to the same conclusion:
[list]
[]Static content doesn’t need to be backed often, providing no real value in incremental backups (does it really matter if you can restore your entire backup with Call forwarding 8.2 module instead of the later Call forwarding 8.3?!). Also, the backups are small enough that its cheaper to buy storage space (and store many complete copies of your backups) then to spend the (development) time implementing it
[
]Dynamic content is so small, that a. as above - the price per mb/gb makes the price of development seem outrageous, and b. there really is no “fun” (read: efficient and worthwhile) way to incrementally backup a database (yes there are way, but if your going to go down that route your better off just implementing a mysql cluster)
[/list]

I second this, some function to ‘browse’ or the ability to point to a different FTP server would be a nice addition.

not sure if you going to implement a browser, but I hope that, at very least, you will be able to paste a full path. Thanks! --mbrevda

Can we also backup /tftpboot

We already can. --mbrevda

So not to beat a dead horse, but . . .

We already have the intrinsic infrastructure to ssh/scp/rsync to/from remote machines.

I totally agree with mbrevda as to his arguments as to the timely fashion of “the state” of the machine, but the current backup is largely hardly “stateful” in time, perhaps a month late perhaps last night, no ?, who needs voicemails restored from a month ago?.

My suggestion is to rethink that whole backup concept, a la mbrevda’s arguments, and if external backup is available then monitor files and voicemail files DO NOT belong in the weekly/monthly/daily backup, they indeed belong in a more synchronous and robust backup location and should be excluded from the current backup process for reasons of time and space, and added to a new scheduler that bears in mind reality.

That backup location can be synced to in a far more timely fashion, Voicemail’s maybe every five minutes, monitor files maybe every evening. ( And of course a real-time push/pull from an external mysql server for cdr records by pbx)

(JM2CWAE, I’ll shut up for now :slight_smile: )