Warm Standby PABX failing

Hello,

I am trying to set up a Warm Standby PABX as per: Warm+Spare+Setup on the freepbx wiki (I am not able to link as this is my first post)

This has worked perfectly for 3 sites but for the latest site I am getting the following message about 30 seconds into the backup:

Intializing Backup 6
Connecting to remote server...
/bin/tar: Removing leading `/' from member names
/bin/tar: Removing leading `/' from member names
/bin/tar: Removing leading `/' from member names
/bin/tar: Removing leading `/' from member names
/bin/tar: Removing leading `/' from member names
PHP Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 524288 bytes) in /var/www/html/admin/libraries/utility.functions.php on line 877
Verifying received file...
File verification failed. 
Here are the first few lines of the file as sent by the remote server:```

I have tried:
 1. Increasing the memory available to php from 128M to 1024M (confirmed on the phpinfo.php page)
 2. Running the backup with different backup options omitted
 3. Adding in a line to *utility.functions.php* to log the directory variable just before line 877 (which the script complains about as per the message above) and tailing the /var/log/httpd/error_log file but nothing is being output here.

    error_log("$dir",0);
    
    //get directory contents
       foreach (scandir($dir) as $d) {

Any assistance would be much appreciated

That warning has been gone for a while. Please ensure all your machines are up to date. Iā€™m pretty sure that was fixed about 6 months ago in the 12 release cycle.

The cause of the problem is that youā€™re trying to back up an insanely huge number of files. You need to reduce the number of files youā€™re backing up. You could put something like this:
print "$d\n";
after that foreach and youā€™ll see the directories itā€™s trying to back up, which will help you remove the erroneous entry.

Hi Rob,

Thanks for the reply. I will look at upgrading and I have added the echo and will make another attempt to backup tonight and identify the problem directory.

Regarding the actual backup and size:

  1. Does it do a full copy or a copy of the diffs? In my mind the script would use either and scp or an rsync so I am thinking that if I can get an initial snapshot of the primary pbx backed up the the future backups may be less huge.
  2. How can I control the size of the backup or can I perhaps copy in smaller segments? In the wiki I followed I dragged a preconfigured ā€œFull Backupā€ set of options and while I can control what I backup, I donā€™t seem to be able to control how much of the files I backup.
  3. Given that I can control what I backup, would it perhaps be an option to run multiple backups with only different components of the full backup selected?

Thanks in advance

It does a full copy. Iā€™m guessing that you have a multiple recursive copy of ā€¦ something, thatā€™s trying to be backed up.

Youā€™ll probably notice it immediately and thatā€™ll let you delete it and continue on normally.

Edit: Note the print is of $d, not of $dir, I edited my original comment.

Hi Rob,

I have followed your edit and added some other lines (see inline comments) to try and identify a directory giving an error. It does not echo to the web interface and i do not see anything in the /var/log/apache/error_log when I tail. Changes are as follows:

869 function scandirr($dir, $absolute = false) {
 870         $list = array();
 871         if ($absolute) {
 872                 global $list;
 873         }
 874
 875         error_log("$dir",0)//added by me
 876         //get directory contents
 877         foreach (scandir($dir) as $d) {
 878                 print "$d\n";//added by me
 879                 echo "$d\n";//added by me
 880                 error_log("$d",0);//added by me
 881                 //ignore any of the files in the array
 882                 if (in_array($d, array('.', '..'))) {
 883                         continue;
 884                 }

Here is the latest output:

Saving Backup 6...Intializing Backup 6
Connecting to remote server...
/bin/tar: Removing leading `/' from member names
/bin/tar: Removing leading `/' from member names
/bin/tar: Removing leading `/' from member names
/bin/tar: Removing leading `/' from member names
/bin/tar: Removing leading `/' from member names
PHP Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 524288 bytes) in /var/www/html/admin/libraries/utility.functions.php on line 877
Verifying received file...
File verification failed. 
Here are the first few lines of the file as sent by the remote server:

rm: cannot remove `/var/spool/asterisk/tmp/backup-6/astdb': Permission denied
rm: cannot remove `/var/spool/asterisk/tmp/backup-6/mysql-3.sql': Permission denied
rm: cannot remove `/var/spool/asterisk/tmp/backup-6/etc/dahdi': Permission denied
rm: cannot remove `/var/spool/asterisk/tmp/backup-6/etc/asterisk': Permission denied
rm: cannot remove `/var/spool/asterisk/tmp/backup-6/tftpboot': Permission denied
rm: cannot remove `/var/spool/asterisk/tmp/backup-6/mysql-4.sql': Permission denied
rm: cannot remove `/var/spool/asterisk/tmp/backup-6/var/lib/asterisk/bin': 

I am guessing that the remove follows an unsuccessful backup as a cleanup if the backup fails?

Note I havenā€™t yet upgraded the version of Freepbx as I need to organize internet breakout with the network admin. Could this perhaps be causing or contributing to the issue?

Again, Thanks in advance

Yes Please ensure that your FreePBX machine always has internet connectivity.

Iā€™d also remove that ā€˜tmpā€™ directory on the other machine, too, that may be your problem.

Thanks for the speedy response. Will look into getting this asap.

Just to confirm, is this the tmp folder on the Primary, active machine or the secondary, standby machine?

I did try clearing the contents of /tmp on the primary of any backup folders but these get generated when the backup attempt is made.

1 Like