Unable to boot after 4.211.64-100 upgrade

After 4.211.64-100 upgrade to a 5.211.65-1 version and track the system no longer boots.

I get to the boot menu
SHMZ (2.632-431.el6.i686)
Centos (2.6.32-358.14.1.el6.i686)
the system tries to boot with the SHMZ default and dies.

If I manually select Centos the system boots ok but the next time I reboot it tries again to boot to SHMZ and dies again.

never mind

Not to go off topic but the distro seems to have a problem running in virtual machines. We got it working in 4.211.64-100 by enabling ACPI but after upgrade 5.211.65-3 it’s not booting again. Is anyone running the current distro in a VM? Any special settings?

Sorry on the Centos option it boots but not ok as Asterisk does not load.

Any help? This is not a production system but it wasn’t an Alpha system either. We were running 4.211.64-9 prior to the upgrade so this should have worked without any issues like any other upgrade.

And the FreePBX Distro 4.211.64-10 step was missed before applying the -100?

Not on purpose that’s for sure. I meticulously check each step before going to the next one but nothing is impossible I guess. Is there any way out of this mess?

@Tony. Wouldn’t it be possible for upgrade scripts to validate the distro version before running? If this type of mistake happened to me it can happen to anyone.

I checked and the script upgrade-4.211.64-10.sh is on the system so pretty sure I ran it. I believe this is the error I got when running the script.

PHP Fatal error: Call to undefined method DB_Error::query() in /usr/sbin/sysadmin_update_system on line 237
PHP Fatal error: Call to undefined method DB_Error::query() in /usr/sbin/sysadmin_update_system on line 237
PHP Fatal error: Call to undefined method DB_Error::query() in /usr/sbin/sysadmin_update_system on line 237
PHP Fatal error: Call to undefined method DB_Error::query() in /usr/sbin/sysadmin_update_system on line 219
REBOOT YOUR BOX NOW FOR NEW KERNEL, DAHDI AND ASTERISK

Broadcast message from [email protected]
(/dev/pts/0) at 13:52 …

The system is going down for reboot NOW!

if someone can edit the post pls remove the domain name. Thanks.

Revmoved for privacy reason, as per request.

Regarding Shell Update Script: each script will check if the system is a FreePBX Distro (to obtain a clearance to go on) and which version is running (but not to obtain a clearance to going on with the update)…so a very basic preliminary check is made as a first step but, AFAIK, there isn’t a mechanism to avoid that a -(n) FreePBX Distro shell update script can be successfully applied to a -(n-m), with m>1, FreePBX Distro minor version revision (repeat: AFAIK).

So basically nothing will avoid that manually a FreePBX Distro Shell Update Scritp is executed blindly.

Said that if your system executed one (or more) shell update script then automatically a relevant Log file was created (Look for it under /var/log/pbx/upgrade/ folder) and so you can gather some info about what happened (time correlating with the yum history log too) if you think the issue rose after a specific update step.

Thanks for the privacy edit.
I checked the logs as you suggested and I was right. I did not skip any update. 4.211.64-10 was completed before the next update (log below). Just wondering if maybe the recent changes with CentOS may be the issue. Maybe the update points to the wrong repo?

Wed Feb 12 16:24:31 EST 2014 This appears to be a FreePBX Distro system as it has a Distro Version of 4.211.64-9
VARIABLES SET FOR UPGRADE
asterisk=Asterisk 11.7.0 built by root @ jenkins-el6-32.schmoozecom.net on a i686 running Linux on 2013-12-18 22:22:09 UTC
Privilege escalation protection disabled!
See https://wiki.asterisk.org/wiki/x/1gKfAQ for more details.
kernel=Linux xx.xxxxxxx.com 2.6.32-358.14.1.el6.i686 #1 SMP Tue Jul 16 21:12:30 UTC 2013 i686 i686 i386 GNU/Linux
version=4.211.64-9
host=xx.xxxxxxx.com
upgradeversion=4.211.64-10
virtual=

Wed Feb 12 16:24:31 EST 2014 Your FreePBX Distro System is being upgraded to 4.211.64-10. Please standby…

Wed Feb 12 16:24:31 EST 2014 STAGE 1 STARTING - GUI Modules
Wed Feb 12 16:24:31 EST 2014 Upgrade All FreePBX GUI Modules
Wed Feb 12 16:24:37 EST 2014 STAGE 1 COMPLETED - GUI Modules - Moving to Stage 2

Wed Feb 12 16:24:37 EST 2014 STAGE 2 STARTING - RPM’s
Wed Feb 12 16:24:37 EST 2014 Replace repos with only FreePBX Distro since some people have added other repos which can break updates
Wed Feb 12 16:24:37 EST 2014 Update Asterisk RPMs
Wed Feb 12 16:24:51 EST 2014 Update Prosody XMPP RPM
Wed Feb 12 16:24:52 EST 2014 Update iSymphony RPM
Wed Feb 12 16:24:53 EST 2014 Make sure all DAHDI firmware is installed
Wed Feb 12 16:24:54 EST 2014 Updating all remaining RPMS now to Centos 6.4
Wed Feb 12 16:24:55 EST 2014 STAGE 2 COMPLETED - Misc Items - Moving to Stage 4

Wed Feb 12 16:24:55 EST 2014 STAGE 3 STARTING - Misc Items
Wed Feb 12 16:24:55 EST 2014 STAGE 3 COMPLETED - Misc Items - Moving to Stage 4

Wed Feb 12 16:24:55 EST 2014 STAGE 4 STARTING - Clean Up
Wed Feb 12 16:24:55 EST 2014 updatedb for locate command
Wed Feb 12 16:24:56 EST 2014 Restart incron to be safe
Wed Feb 12 16:24:56 EST 2014 STAGE 4 COMPLETED - Clean Up - Moving to Stage 5

Wed Feb 12 16:24:56 EST 2014 STAGE 5 STARTING - Final Verifications
Wed Feb 12 16:24:56 EST 2014 STAGE 5 COMPLETED - Final Verifications - Moving to Stage 6

Wed Feb 12 16:24:56 EST 2014 UPGRADE 100% COMPLETED


Wed Feb 12 16:52:45 EST 2014 Check to make sure this is a FreePBX Distro system before executing
Wed Feb 12 16:52:45 EST 2014 This appears to be a FreePBX Distro system as it has a Distro Version of 4.211.64-10
Wed Feb 12 16:52:45 EST 2014 Make sure this is version 4.211.64-9 or greater before executing
Wed Feb 12 16:52:45 EST 2014 This appears to be FreePBX Distro version 4.211.64.10 so we can now upgrade you to the 5.211.65 track as version 5.211.65-1
VARIABLES SET FOR UPGRADE
asterisk=Asterisk 11.7.0 built by root @ jenkins-el6-32.schmoozecom.net on a i686 running Linux on 2013-12-18 22:22:09 UTC
Privilege escalation protection disabled!
See https://wiki.asterisk.org/wiki/x/1gKfAQ for more details.
kernel=Linux xx.xxxxxxx.com 2.6.32-358.14.1.el6.i686 #1 SMP Tue Jul 16 21:12:30 UTC 2013 i686 i686 i386 GNU/Linux
version=4.211.64.10
brand=FreePBXDistro
host=xx.xxxxxxx.com
upgradeversion=5.211.65-1
virtual=

Wed Feb 12 16:52:45 EST 2014 Your FreePBX Distro System is being upgraded to 5.211.65-1. Please standby…

Wed Feb 12 16:52:45 EST 2014 STAGE 1 STARTING - GUI Modules
Wed Feb 12 16:52:45 EST 2014 Upgrade All FreePBX GUI Modules
Wed Feb 12 16:52:53 EST 2014 STAGE 1 COMPLETED - GUI Modules - Moving to Stage 2

Wed Feb 12 16:52:53 EST 2014 STAGE 2 STARTING - RPM’s
Wed Feb 12 16:52:53 EST 2014 Replace repos with only FreePBX Distro since some people have added other repos which can break updates
Wed Feb 12 16:52:53 EST 2014 Cleanup old Kernels to make space in boot
Wed Feb 12 16:53:28 EST 2014 Updating all remaining RPMS now to Centos 6.5
Wed Feb 12 12:01:29 EST 2014 STAGE 2 COMPLETED - Misc Items - Moving to Stage 4

Wed Feb 12 12:01:29 EST 2014 STAGE 3 STARTING - Misc Items
Wed Feb 12 12:01:29 EST 2014 STAGE 3 COMPLETED - Misc Items - Moving to Stage 4

Wed Feb 12 12:01:29 EST 2014 STAGE 4 STARTING - Clean Up
Wed Feb 12 12:01:29 EST 2014 updatedb for locate command
Wed Feb 12 12:01:30 EST 2014 Restart incron to be safe
Wed Feb 12 12:01:30 EST 2014 STAGE 4 COMPLETED - Clean Up - Moving to Stage 5

Wed Feb 12 12:01:30 EST 2014 STAGE 5 STARTING - Final Verifications
Wed Feb 12 12:01:30 EST 2014 STAGE 5 COMPLETED - Final Verifications - Moving to Stage 6

Wed Feb 12 12:01:30 EST 2014 UPGRADE 100% COMPLETED
Wed Feb 12 12:01:30 EST 2014 REBOOT YOUR BOX NOW FOR NEW KERNEL, DAHDI AND ASTERISK

Wrong repo during updates? I doubt. Your (update) Logs appear exactly like mine. Have you tried to force shell script updates up to 5.211.65-5 or this is not possible at all at this point?

A “yum clean all”, “updatedb” then a “yum upgrade” which list of (if any) RPM updates will show you?

Thanks for your help. I didn’t try to continue the upgrades since the system did not boot with the SHMZ kernel but I can definitely try. Is there a special syntax to force the shell scripts? I use chmod +x

[root@XX ~]# yum clean all
Loaded plugins: fastestmirror, kmod
Cleaning repos: base extras pbx schmooze-commercial updates
Cleaning up Everything
Cleaning up list of fastest mirrors
[root@XX ~]# updatedb
[root@XX ~]# yum upgrade
Loaded plugins: fastestmirror, kmod
Determining fastest mirrors
base | 2.0 kB 00:00
base/primary | 2.0 MB 00:00
base 4807/4807
extras | 1.3 kB 00:00
extras/primary | 5.9 kB 00:00
extras 12/12
pbx | 1.3 kB 00:00
pbx/primary | 100 kB 00:00
pbx 407/407
schmooze-commercial | 1.3 kB 00:00
schmooze-commercial/primary | 16 kB 00:00
schmooze-commercial 105/105
updates | 1.3 kB 00:00
updates/primary | 100 kB 00:00
updates 235/235
Setting up Upgrade Process
No Packages marked for Update
[root@XX ~]#

There isn’t; just follow the Wiki page about FreePBX Distro update. FreePBX Distro Shell Update Scripts updates, in a controlled way, both RPM (as per official repositories and only if the system has RPM that needs to be updated) and FreePBX Modules. I think you can safely apply a script once again, if necessary (eventually the system doesn’t change if the previous attempt ended as expected or it may change if something has been changed on the Repositories side in meantime).

The system seems to be updated (from the point of view of installed RPM). Check which Repositories are involved with a “cat /etc/yum.repos.d/freepbx.repo”) so you will be able remove your doubts about them (no wrong repo).

I just tried to run the next update and here is the error I get.

Moving to Next Step

Update Asterisk RPMs
Loaded plugins: fastestmirror, kmod
Determining fastest mirrors
base | 2.0 kB 00:00
base/primary | 2.0 MB 00:01
base 4807/4807
extras | 1.3 kB 00:00
extras/primary | 5.9 kB 00:00
extras 12/12
pbx | 1.3 kB 00:00
pbx/primary | 100 kB 00:00
pbx 407/407
schmooze-commercial | 1.3 kB 00:00
schmooze-commercial/primary | 16 kB 00:00
schmooze-commercial 105/105
updates | 1.3 kB 00:00
updates/primary | 100 kB 00:00
updates 235/235
Setting up Update Process
No Packages marked for Update
Privilege escalation protection disabled!
See https://wiki.asterisk.org/wiki/x/1gKfAQ for more details.
Unable to connect to remote asterisk (does /var/run/asterisk/asterisk.ctl exist?)

Moving to Next Step
STAGE 2 STARTING - RPM’s
PHP Fatal error: Call to undefined method DB_Error::query() in /usr/sbin/sysadmin_update_system on line

STAGE 3 STARTING - Misc Items
PHP Fatal error: Call to undefined method DB_Error::query() in /usr/sbin/sysadmin_update_system on line 237
Moving to Next Step

Moving to Next Step
STAGE 3 COMPLETED - Misc Items - Moving to Stage 4
PHP Fatal error: Call to undefined method DB_Error::query() in /usr/sbin/sysadmin_update_system on line 237

STAGE 4 STARTING - Clean Up
PHP Fatal error: Call to undefined method DB_Error::query() in /usr/sbin/sysadmin_update_system on line 237
updatedb for locate command

Restart incron to be safe

STAGE 4 COMPLETED - Clean Up - Moving to Stage 5
PHP Fatal error: Call to undefined method DB_Error::query() in /usr/sbin/sysadmin_update_system on line 237
PHP Fatal error: Call to undefined method DB_Error::query() in /usr/sbin/sysadmin_update_system on line 237
PHP Fatal error: Call to undefined method DB_Error::query() in /usr/sbin/sysadmin_update_system on line 237
PHP Fatal error: Call to undefined method DB_Error::query() in /usr/sbin/sysadmin_update_system on line 237
PHP Fatal error: Call to undefined method DB_Error::query() in /usr/sbin/sysadmin_update_system on line 219

Seems like you have a DB issue.

Is the DB running at all?

I simply upgraded from 4.211.64-10 to 4.211.64-100. Never touched the DB or anything else but the DBD issue could be because I am booting to the CentOS kernel instead of the SHMZ kernel as that one dies on boot every time.