Restored Backup to HA master, now amportal will not start

We recently set up an HA cluster and successfully restored a full backup. When it came time to test failover, we shut down node freepbx-a. Upon restart, all services are stopped and the GUI is gone.

Trying the cluster repair script from the wiki, all errors were repaired and restraints removed however when we tried to do anything with amportal we got:

amportal restart

Fetching settings from amportal.conf file..
/etc/amportal.conf: line 732: reg: command not found
FATAL: can not find freepbx_engine to start Asterisk

I don’t see similar errors to this post, so it doesn’t appear on face to be a permissions issue. I also fixed the &reg escape bug, but amportal still cannot find freepbx_engine. What should I do to repair?

If I run pcs status to check the health of the cluster itself, it appears everything is in order but since amportal can’t start all services are stopped:

[[email protected] httpd]# pcs status
Cluster name: 
Last updated: Wed Nov 26 12:47:21 2014
Last change: Wed Nov 26 12:16:08 2014 via crmd on freepbx-a
Stack: cman
Current DC: freepbx-a - partition with quorum
Version: 1.1.10-14.el6-368c726
2 Nodes configured
20 Resources configured


Online: [ freepbx-a freepbx-b ]

Full list of resources:

 spare_ip       (ocf::heartbeat:IPaddr2):       Stopped (unmanaged) 
 floating_ip    (ocf::heartbeat:IPaddr2):       Stopped (unmanaged) 
 Master/Slave Set: ms-asterisk [drbd_asterisk] (unmanaged)
     Stopped: [ freepbx-a freepbx-b ]
 Master/Slave Set: ms-mysql [drbd_mysql] (unmanaged)
     Stopped: [ freepbx-a freepbx-b ]
 Master/Slave Set: ms-httpd [drbd_httpd] (unmanaged)
     Stopped: [ freepbx-a freepbx-b ]
 Master/Slave Set: ms-spare [drbd_spare] (unmanaged)
     Stopped: [ freepbx-a freepbx-b ]
 spare_fs       (ocf::heartbeat:Filesystem):    Stopped (unmanaged) 
 Resource Group: mysql
     mysql_fs   (ocf::heartbeat:Filesystem):    Stopped (unmanaged) 
     mysql_ip   (ocf::heartbeat:IPaddr2):       Stopped (unmanaged) 
     mysql_service      (ocf::heartbeat:mysql): Stopped (unmanaged) 
 Resource Group: asterisk
     asterisk_fs        (ocf::heartbeat:Filesystem):    Stopped (unmanaged) 
     asterisk_ip        (ocf::heartbeat:IPaddr2):       Stopped (unmanaged) 
     asterisk_service   (ocf::heartbeat:freepbx):       Stopped (unmanaged) 
 Resource Group: httpd
     httpd_fs   (ocf::heartbeat:Filesystem):    Stopped (unmanaged) 
     httpd_ip   (ocf::heartbeat:IPaddr2):       Stopped (unmanaged) 
     httpd_service      (ocf::heartbeat:apache):        Stopped (unmanaged) 


Having the cluster marked as ‘unmanaged’ is somewhat worrying. It means you’ve put the cluster into maintenance mode.

http://wiki.freepbx.org/display/FCM/FreePBX+HA-Setting+the+cluster+to+maintenance+mode

You’ll need to undo that.

Also, if you’ve restored a backup of a NON-HA system to a HA system, you’ve possibly confused things, too.

It had, in fact been restored from a non-HA backup, but this did not seem to cause any other issues.

In any case it does not matter; This morning we reinstalled both node distributions and tried the restore before configuring HA, and the restore breaks the GUI, rendering all buttons and links useless. Trying a reload of amportal a ma reload returns the following:

Fetching FreePBX settings with gen_amp_conf.php..

Error(s) have occured, the following is the retrieve_conf output:
exit: 255
PHP Fatal error:  Call to undefined function core_users_list() in /var/www/html/admin/modules/campon/functions.inc.php on line 295

I noticed the HA module was upgraded since last week from 12.0.1.1 to 12.0.2. Perhaps this is significant?

When you did the restore, it has disabled a pile of modules. You’ll need to re-enable them. Going into Module Admin will show you the problem.

If you have a bunch of problems about being unable to connect to asterisk, it’s because your manager settings don’t match. Compare /etc/asterisk/manager.conf to what’s in advanced settings.

Okay, so it appears everything is almost back to a fully working state on the HA cluster. We restored only the mysql backup (PBX Settings), and got the same error as above but now at least amportal a ma updateall does in fact work and the GUI is back to normal.

The only warning detected during backup restore this time around was:

Intialized!!
Intializing Restore...
Running pre-restore hooks, if any...
WARNING!
Web Root restore not detected, not restoring module table
You should run "amportal a ma updateall" to ensure system integrity
Restoring Database...
Restored Database
Restoring Settings complete
Running post-restore hooks, if any...
Cleaning up...
Restore complete!
Reloading...
Done!

Our extensions etc. appear to have been restored as well. However the only issue remaining is that, as you rightly mentioned, several of the modules are now disabled.

Some cannot be updated because the Core module needs to be updated (it’s reporting version 2.11) however when running the update it fails because it’s looking for version 12.

What can we do to update the Core module to the current version get the other modules properly up to date?

We get this when trying to update core:

Errors with selection:

Core cannot be upgraded:
FreePBX version 12.0.1rc16 or higher is required, you have 2.11.0.42
Please try again after the dependencies have been installed.

I should add that according to The FreePBX website upgrading via the source should be possible, but it looks like the mirror site is down (and the instructions are for old versions of FreePBX).