Backup/restore uninstalls sysadmin, fails

billsimon · July 27, 2021, 2:12pm

Hope someone can help me understand what’s going on here running a warm spare backup/restore on v15.

On the spare where a restore is being performed:

[2021-07-27 02:37:03] [6e2e794e-2804-4f75-8972-ae8df6a5c611.INFO]: Processing sysadmin [] []
[2021-07-27 02:37:03] [6e2e794e-2804-4f75-8972-ae8df6a5c611.INFO]: Resetting sysadmin module data [] []
[2021-07-27 02:37:04] [6e2e794e-2804-4f75-8972-ae8df6a5c611.ERROR]: Error uninstalling sysadmin reason(s): Failed to run un-installation scripts on line 41 of file /var/www/html/admin/modules/backup/Handlers/FreePBXModule.php [] []
[2021-07-27 02:37:04] [6e2e794e-2804-4f75-8972-ae8df6a5c611.INFO]: #0 /var/www/html/admin/modules/backup/Handlers/FreePBXModule.php(20): FreePBX\modules\Backup\Handlers\FreePBXModule->uninstall('sysadmin')

as a result of this, other modules fail to restore:

[2021-07-27 02:44:31] [6e2e794e-2804-4f75-8972-ae8df6a5c611.INFO]: Resetting sangomaconnect module data [] []
[2021-07-27 02:44:55] [6e2e794e-2804-4f75-8972-ae8df6a5c611.ERROR]: SQLSTATE[42S02]: Base table or view not found: 1146 Table 'asterisk.sysadmin_options' doesn't exist:: on line 123 of file /var/www/html/admin/libraries/utility.functions.php [] []

…

[2021-07-27 02:45:16] [6e2e794e-2804-4f75-8972-ae8df6a5c611.INFO]: Processing vega [] []
[2021-07-27 02:45:16] [6e2e794e-2804-4f75-8972-ae8df6a5c611.INFO]: Resetting vega module data [] []
[2021-07-27 02:45:20] [6e2e794e-2804-4f75-8972-ae8df6a5c611.ERROR]: SQLSTATE[42S02]: Base table or view not found: 1146 Table 'asterisk.sysadmin_options' doesn't exist on line 968 of file /var/www/html/admin/modules/sysadmin/Sysadmin.class.php [] []

Why is sysadmin being uninstalled?

jfinstrom · July 27, 2021, 4:08pm

Could be part of the upgrading modules. That said this seems to be part of a bigger trend of the developers NOT using internals and not doing sanity checks.

They should check is sysadmin is installed
They should then call an internal method if it is installed and fail if not.
Each module should be individually responsible for its own data. This means you shouldn’t be backing up data from tables not owned by the module

The backup module automatically backs up dependencies.

I hate to be negative bit it is getting harder and harder with such fundamental mistakes. What’s worse is they all seem to get past code review and QA.
Even index errors which are basic programming and not FreePBX specific.

There are internal docs, public docs and code examples for all of the FreePBX stuff. Hell before I left I created a new developer onboarding doc that breaks down every thing you need to know about FreePBX and where to find code documentation for specific tasks.

Recent discussions show that was a complete waste of my time. It is almost like I didn’t spend almost a decade writing this stuff down

billsimon · July 27, 2021, 4:29pm

I did not realize this was part of the process.

It’s probably a downgrade attempt actually. The issue here is that the primary is on a version installed about 6 weeks ago and the client wants updates on a fixed schedule. So that one can’t be upgraded right now.

The warm spare is an appliance that came with an older version of pbxact so I ran the updater to get it current when it arrived last week.

I am going to try to run the WS process again with the module versions matched up. The only problem is I don’t see a way to tell it to grab the versions of all the modules as of a certain date in June. One possibility is to tar up the primary’s whole /var/www/html/admin/modules directory and untar it on the warm spare and run fwconsole ma installall… Not sure how that will go. Alternatively, install each module on the spare using the --tag parameter to force the same version as the primary. Thoughts?

jfinstrom · July 27, 2021, 5:34pm

Backup buillds a manifest of versions but that is simply passed in the data structure to the module. It can then look at what it needs to do to process the data for added fields etc. Again the idea was every module handles the data it owns.

Looking at the code which has been a few years you are getting hung up here:

github.com

FreePBX/backup/blob/840c8dff142f62d6e42706fee4efcb0028733a12/Handlers/FreePBXModule.php#L15


namespace FreePBX\modules\Backup\Handlers;
use Symfony\Component\Process\Process;
use Symfony\Component\Process\Exception\ProcessFailedException;


class FreePBXModule{
	public $moduleXML = false;
	public function __construct($freepbx) {
		$this->freepbx = $freepbx;
		$this->mf = \module_functions::create();
	}
	public function reset($module,$version){
		$developer = $this->freepbx->Config->get('DEVEL');
		$module = \strtolower($module);
		$info = $this->mf->getInfo($module, false, true);
		if(!empty($info[$module]) && ($info[$module]['status'] === MODULE_STATUS_ENABLED)) {
			$uninstall = $this->uninstall($module);
		}
		$install = $this->install($module);
		return $this;
	}

To reset the data (since no other way is defined in the master scheme) it does an uninstall and reinstall.

The only way to skip this is to add it in to the modules backup process and add “skip_reset”.

They have hardcoded some modules because well… Anyway they need to add “skip_reset” to the sysadmin module backup settings OR fix the direct calls. In any case this is out of your hands unless you modify the restore handler.

jfinstrom · July 27, 2021, 5:37pm

This is where things are hard coded ಠ_ಠ

github.com

FreePBX/backup/blob/840c8dff142f62d6e42706fee4efcb0028733a12/Handlers/Restore/Common.php#L116


				return;
			}
			$this->log(_("Using fallback restore strategy"),'WARNING');
			$className = 'FreePBX\modules\Backup\RestoreBase';
		}
		$class = new $className($this->freepbx, $this->backupModVer, $this->getLogger(), $this->transactionId, $modData, $this->tmp, $this->defaultFallback);
		//Change the Text Domain
		$this->log(sprintf(_('Resetting %s module data'),$module));
		modgettext::push_textdomain($module);
		//FREEPBX-20646 Warm spare backup,  resets will clear all data so the extenalip ,nat,bindaddress we can not preserve
		if ($module === 'sipsettings' && $modData['backupInfo']['backupInfo']['warmspareenabled'] == 'yes' || $module === 'zulu' || ($modData['backupInfo']['backupInfo']['warmspareenabled'] == 'yes' && $modData['backupInfo']['backupInfo']['warmspare_excludetrunks'] == 'yes' && $module === 'core')) {
			$this->log(sprintf(_('NOT Resetting %s module data'),$module));
		} else {
			if(!$skipModuleReset){
				$class->reset();
			}else {
				$this->log(sprintf(_('Based on Module settings We are NOT Resetting %s module data'),$module));
			}
		}
		$this->log(sprintf(_("Restoring from %s [%s]"), $module, get_class($class)));
		$this->runRestore($class);

billsimon · July 27, 2021, 5:54pm

Is the uninstall() method different than what happens when I do fwconsole ma uninstall ...? (Yes I am being a little lazy as I could look up this code myself) Because the uninstall of sysadmin simply won’t work due to dependencies:

# fwconsole ma uninstall sysadmin
The following error(s) occured:
 - Cannot disable: The following modules depend on this one: areminder,broadcast,callerid,calllimit,conferencespro,cos,extensionroutes,faxpro,firewall,oembranding,pagingpro,parkpro,pinsetspro,qxact_reports,recording_report,restapps,vmnotify,voicemail_report,vqplus,webcallback,zulu

At the point it restores, the only other modules that have been processed so far are userman, pm2, certman, and queues. The other modules are already installed and would interfere with the uninstall of sysadmin.

I am doing warmspare backup/restore on a different pair of servers and it works fine. Apparently the reset() of sysadmin works there.

I am going to try synchronizing the versions on the two servers and see whether that makes a difference.

jfinstrom · July 27, 2021, 6:41pm

That is in the hardcoded section to skip reset.

The uninstall is equivalent to

fwconsole ma uninstall --force sysadmin

it does a force because it knows it is followed by an install so those dependent modules will have it when needed.

billsimon · July 28, 2021, 3:01am

I filed a bug on the issue of different module versions causing failure.

To work around it for now I did some awk-ward scripting!

On the primary:

fwconsole ma list | awk '{print $2, $4}' > modlist

edit it to get rid of headers…

on the warmspare:

awk '{ system("fwconsole ma downloadinstall " $1 " --tag " $2 )}' modlist

All versions matched up and the backup/restore went through.

franckdanard · July 28, 2021, 8:42am

Hi Bill
A ticket is on going to fix this kind of issue. A dev is working on it.
Be patient.

billsimon · July 28, 2021, 1:24pm

Sorry Franck, I did not see a prior ticket, which is why I opened one yesterday.

In the meantime it was necessary to get the process going.

franckdanard · July 28, 2021, 3:20pm

yes, no problem Bill.
Don’t worry.
The main is to fix this issue

kgupta · July 30, 2021, 2:45am

Hi @billsimon Thanks for reporting https://issues.freepbx.org/browse/FREEPBX-22694.
Issue has been resolved and fix is available in sysadmin edge version v15.0.21.70.
Please give try and update the jira if the issue still persist on v15.0.21.70.

Thanks
Kapil

system · August 29, 2021, 2:45am

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.