No GUI, system not working following upgrade to 16

I just completed the update to version 16. The screen came up and said it was successful. The freepbx16-upgrade.log file says it upgraded successfully. However… the GUI won’t load and the phones are all dead.
The GUI has an error showing

"Exception thrown with message “SQLSTATE[HY000] [2002] No such file or directory::SQLSTATE[HY000] [2002] No such file or directory”

Stacktrace:
#7 Exception in /var/www/html/admin/libraries/utility.functions.php:120
#6 die_freepbx in /var/www/html/admin/libraries/BMO/Database.class.php:150
#5 PDOException in /var/www/html/admin/libraries/BMO/Database.class.php:144
#4 PDO:__construct in /var/www/html/admin/libraries/BMO/Database.class.php:144
#3 FreePBX\Database:__construct in /var/www/html/admin/libraries/BMO/FreePBX.class.php:77
#2 FreePBX:__construct in /var/www/html/admin/bootstrap.php:144
#1 require_once in /etc/freepbx.conf:9
#0 include_once in /var/www/html/admin/config.php:127

I rebooted the system and the GUI now just says:

“0 Vega 15.0.12 Copyright 2021 by Sangoma Technologies Inc., All rights reserved By installing, copying, downloading, distributing, inspecting or using the materials provided herewith, you agree to all of the terms of use as outlined in our End User Agreement which can be found and reviewed at https://www.freepbx.org/legal/

But there’s still no phone.

I got something very similar on my attempted upgrade last night to v16. Hopefully someone can shed some light on it. Nothing I did seemed to work either afterward.

Looks like the upgrade didn’t remove/upgrade this commercial module from 15. So it is looking for ZendGuard which is now gone. You can remove it and reinstall it (if you need it to manage Vega gateways). fwconsole ma delete vega

fwconsole ma list should now only show modules with major version 16. If you see any other older versions you should delete them; if you are unsure, post the fwconsole ma list output here.

I’m in serious trouble over here and I’m starting to panic big time. I SO regret attempting this upgrade. I don’t know how to go back, or forward or even get the help I need.

When I enter ‘fwconsole ma list’ it just returns 0

In fact, it seem to just return 0 pretty much not matter what command I enter.

Did you take a backup before performing the upgrade? FreePBX 16 is pretty good at restoring from any version of backup to a full version. Worst case you could do a clean install of FreePBX 16 and then just restore from backup.

I did a backup but I’ve never tried to restore it to make 100% sure it worked. I only have the one system, the live one, so I didn’t have a way to test it. My knowledge of linix is pretty limited so without the GUI, I don’t even know where to start. This is my worst nightmare with upgrading.

So is my system beyond hope? Do I download a distro ISO and boot onto it and start over that way? I know once I’ve done that there’s zero chance of fixing the system.

I really hope I haven’t lost all the routes, IVRs, etc.

If it said that the backup was successful and you have the full file I’d feel pretty good about that. The backup really does backup all of the configuration so I wouldn’t be worried about having lost that.

You could try and setup a small virtual machine somewhere like Vultr and stand up a test machine to restore to and see what it looks like. It really doesn’t take that much time and it’s not very expensive to get started.

Once you’ve verified that it’s working there you can then just download an ISO and run through the setup (not much Linux knowledge needed) and then just restore the now verified backup to it once it’s up and running again.

While you are testing with a virtual machine I’ll let other chime in with possible troubleshooting steps for the failed upgrade.

Testing with a VM. duh… why didn’t I think of that. I’ll download the ISO and start spinning up a VM

Or just install Virtual Box on your PC. Install a new Distro and restore your backup to test… Vs Cloud VM like Vultr, DO, etc…

This makes more sense for sure.

I’m back up. Bill Simon was a real life saver and was able to fix everything that broke during the update script. I won’t pretend I could tell you everything he did. It was several fwconsole ma upgradeall commands, restarts, an npm install, reactivating my system and more. He did an excellent job and I’m very grateful for his help.

4 Likes

I did a backup, which was successful. Just now I tried to restore it on a VM of FreePBX. When I select the file (in Windows) it shows as 0 k. And then FreePBX just spins and spins saying “processing” with 0.00% so clearly the backup and restore isn’t working.

I’m super lucky we were able to get the system back up and running thanks to Bill Simon because the backup and restore simply doesn’t work. Perhaps it’s operator error but I’m not sure how that can be when I’m using the GUI. The backup had 89 modules selected and reported success every time.

What your specific issue is, Im not sure. But Ive done backups and restores to both VM and physical on-premise hardware numerous times over the past 12+ months. Both v15 to v15 and v15 to v16… Not an issue…

Thanks for the kind words @cdsJerryw … [Not a paid actor]

0-byte file sounds like the download failed. But I bet the backup file is there on the server intact. Check the file size from SSH or an SFTP connection. If the files really are 0 bytes then that is worth investigating carefully and the backup logs are in /var/log/asterisk/backup.log.

I don’t know all the issues either. One thing I found is that I was backing up to a NAS which FreePBX sees just fine but since it’s a Linux share, it appears that Windows can’t access it so when I try to do a restore using the GUI on a Windows machine, I can’t get to the backup file at all.

I changed the backup location and did a new backup, which was successful. I get a report saying it backed up all these files but again, Windows sees it as 0 K. But Windows at least sees the file so I can select it for the GUI. The GUI then says “processing…” and the little wheel spins but nothing more ever happens.

The end file I see with Windows is named “test” which matches the file name I put in the backup.

Here’s a snipit from the log file:

Generated Backup process result email to jerryremoved. Status: Success

[2023-03-01 19:00:02] [ed668595-617a-4832-982e-282ef2284d2b.DEBUG]: Running Backup ID: 7removedbyauthor
[2023-03-01 19:00:02] [ed668595-617a-4832-982e-282ef2284d2b.DEBUG]: Transaction: ed668595-617a-4832-982e-282ef2284d2b
[2023-03-01 19:00:02] [ed668595-617a-4832-982e-282ef2284d2b.DEBUG]: Starting backup test
[2023-03-01 19:00:02] [ed668595-617a-4832-982e-282ef2284d2b.DEBUG]: This backup will be stored locally and is subject to maintenance settings
[2023-03-01 19:00:02] [ed668595-617a-4832-982e-282ef2284d2b.INFO]: Backup File Name: 20230301-190002-1677715202-16.0.33-794217254.tar.gz
[2023-03-01 19:00:02] [ed668595-617a-4832-982e-282ef2284d2b.INFO]: Working with amd module
[2023-03-01 19:00:02] [ed668595-617a-4832-982e-282ef2284d2b.INFO]: Exporting KVStore from Amd
[2023-03-01 19:00:02] [ed668595-617a-4832-982e-282ef2284d2b.DEBUG]: Adding module manifest for amd
[2023-03-01 19:00:02] [ed668595-617a-4832-982e-282ef2284d2b.INFO]: Working with announcement module
[2023-03-01 19:00:02] [ed668595-617a-4832-982e-282ef2284d2b.DEBUG]: Adding module manifest for announcement
[2023-03-01 19:00:02] [ed668595-617a-4832-982e-282ef2284d2b.INFO]: Working with areminder module
[2023-03-01 19:00:02] [ed668595-617a-4832-982e-282ef2284d2b.DEBUG]: Adding module manifest for areminder
[2023-03-01 19:00:02] [ed668595-617a-4832-982e-282ef2284d2b.INFO]: Working with arimanager module
[2023-03-01 19:00:02] [ed668595-617a-4832-982e-282ef2284d2b.DEBUG]: Adding module manifest for arimanager

Here are the last few lines from the log file.
It seems like it has to be a Windows-Linux thing because I’m not seeing any .gz files in the backup folder at all. I just re-ran the backup again but… no file (using Windows).
I’ll blame that on operator error simply because I don’t know Linux very well BUT it does mean that as a Windows guy, I can’t do a restore using the GUI.

[2023-03-01 19:00:34] [ed668595-617a-4832-982e-282ef2284d2b.INFO]: Finished created backup file: 20230301-190002-1677715202-16.0.33-794217254.tar.gz
[2023-03-01 19:00:34] [ed668595-617a-4832-982e-282ef2284d2b.INFO]: Saving to selected Filestore locations
[2023-03-01 19:00:34] [ed668595-617a-4832-982e-282ef2284d2b.DEBUG]: Saving to: Local:‘NAS 3 local xxx’ instance ,File location: 192.168.xxx.xxx/nfs/PBX/20230301-190002-1677715202-16.0.33-794217254.tar.gz
[2023-03-01 19:00:34] [ed668595-617a-4832-982e-282ef2284d2b.INFO]: Finished Saving to selected Filestore locations
[2023-03-01 19:00:34] [ed668595-617a-4832-982e-282ef2284d2b.DEBUG]: Generated Backup process result email to removed. Status: Success

I’ve just discovered the Dropbox addition. I think that’s going to solve my problem as I’ve been able to backup there and can see it in Dropbox. I’m doing a restore to a VM now as a test and it seems to be working (it’s still restoring).

This seems like a better solution than my network NAS drives.

1 Like

@billsimon any clues on the wizardry you used to get this all back up? im in the same state, every fwconsole command I use returns back with

"0

Vega 15.0.12

Copyright 2021 by Sangoma Technologies Inc., All rights reserved

By installing, copying, downloading, distributing, inspecting or using

the materials provided herewith, you agree to all of the terms of use as

outlined in our End User Agreement which can be found and reviewed at

and the GUI says the same

Delete the vega module (rm -rf /var/www/html/admin/modules/vega)

You can reinstall it later if you use it.

Then you should be able to issue fwconsole ma list and get an inventory of module versions on disk. Framework should be version 16 at this point. Try:

fwconsole ma install framework (I don’t know whether this is actually necessary)
fwconsole ma upgradeall

If all the modules showing are already version 16, you can use fwconsole ma installall

One problem we encountered was that the pm2 module (node.js process manager) wasn’t running correctly and as a result some of the modules weren’t installing. If you see this, you can reinstall it with fwconsole ma downloadinstall pm2 and then run fwconsole ma installall again to finish the job.

2 Likes

I opened a ticket on this since several people have experienced it. [FREEPBX-24078] Vega module breaks FreePBX 15->16 upgrade - Sangoma Issue Tracker

2 Likes