System in standby after failed upgrade to 10.13.66-17. Please help!

During an upgrade from 10.13.66-12 to 10.13.66-18, the system hung during the 10.13.66-17. rebooted hoping it would respond but both of my nodes are in standby and will not go active.

Here is what I got when I upgraded.

Please give any idea to get the system back up

Thanks.

FreePBX High Availability

You have connected to a standby server. Asterisk is not running
on this machine. Very few commands are available.

Usually, the only thing you do on a standby node is yum update.

[root@freepbx-a ~]# Starting UCP Server
-bash: Starting: command not found
[root@freepbx-a ~]# ucpnode
-bash: ucpnode: command not found
[root@freepbx-a ~]# cd /
[root@freepbx-a /]# ucpnode
-bash: ucpnode: command not found
[root@freepbx-a /]#
[root@freepbx-a /]# pcs cluster unstandby freepbx-b
[root@freepbx-a /]#
[root@freepbx-a /]# cat /proc/drbd
version: 8.4.6 (api:1/proto:86-101)
GIT-hash: 833d830e0152d1e457fa7856e71e11248ccf3f70 build by [email protected] chmoozecom.net, 2015-07-20 01:39:28

1: cs:Connected ro:Secondary/Secondary ds:UpToDate/UpToDate C r-----
ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
2: cs:Connected ro:Secondary/Secondary ds:UpToDate/UpToDate C r-----
ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
3: cs:Connected ro:Secondary/Secondary ds:UpToDate/UpToDate C r-----
ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
4: cs:Connected ro:Secondary/Secondary ds:UpToDate/UpToDate C r-----
ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
[root@freepbx-a /]#
[root@freepbx-a /]# pcs resource show floating_ip
Resource: floating_ip (class=ocf provider=heartbeat type=IPaddr2)
Attributes: ip=192.168.10.5 iflabel=ext-ip
Operations: start interval=0s timeout=20s (floating_ip-start-timeout-20s)
stop interval=0s timeout=20s (floating_ip-stop-timeout-20s)
monitor interval=30s (floating_ip-monitor-interval-30s)
[root@freepbx-a /]# /root/upgradescripts/upgrade-10.13.66-17.sh
Unable to connect to remote asterisk (does /var/run/asterisk/asterisk.ctl exist? )
Check to make sure this is a FreePBX Distro system before executing
This appears to be a FreePBX Distro system as it has a Distro Version of 10.13.6 6-17

Your FreePBX Distro System is being upgraded to 10.13.66-17. Please standby…

STAGE 1 STARTING - GUI Modules

PHP Fatal error: Call to undefined method DB_Error::query() in /usr/sbin/sysadm in_update_system on line 237

Upgrade All FreePBX GUI Modules

/root/upgradescripts/upgrade-10.13.66-17.sh: line 89: /var/lib/asterisk/bin/free pbx_setting: No such file or directory
/root/upgradescripts/upgrade-10.13.66-17.sh: line 92: /usr/local/sbin/amportal: No such file or directory
/root/upgradescripts/upgrade-10.13.66-17.sh: line 102: /usr/sbin/fwconsole: No s uch file or directory
/root/upgradescripts/upgrade-10.13.66-17.sh: line 103: /usr/sbin/fwconsole: No s uch file or directory
/root/upgradescripts/upgrade-10.13.66-17.sh: line 104: /usr/sbin/fwconsole: No s uch file or directory
/root/upgradescripts/upgrade-10.13.66-17.sh: line 105: /usr/sbin/fwconsole: No s uch file or directory
/root/upgradescripts/upgrade-10.13.66-17.sh: line 106: /usr/sbin/fwconsole: No s uch file or directory
/root/upgradescripts/upgrade-10.13.66-17.sh: line 107: /usr/sbin/fwconsole: No s uch file or directory
/root/upgradescripts/upgrade-10.13.66-17.sh: line 108: /usr/sbin/fwconsole: No s uch file or directory
/root/upgradescripts/upgrade-10.13.66-17.sh: line 109: /usr/sbin/fwconsole: No s uch file or directory

STAGE 1 COMPLETED - GUI Modules - Moving to Stage 2

PHP Fatal error: Call to undefined method DB_Error::query() in /usr/sbin/sysadmin_update_system on line 237

STAGE 2 STARTING - RPMs

PHP Fatal error: Call to undefined method DB_Error::query() in /usr/sbin/sysadmin_update_system on line 237

Replace repos with only FreePBX Distro since some people have added other repos which can break updates

Moving to Next Step

Update Asterisk RPMs
Loaded plugins: fastestmirror, kmod
Setting up Update Process
Loading mirror speeds from cached hostfile
base | 2.0 kB 00:00
extras | 1.3 kB 00:00
pbx | 2.9 kB 00:00
schmooze-commercial | 2.9 kB 00:00
updates | 1.3 kB 00:00
No Packages marked for Update

Moving to Next Step

Update all misc 6.6 RPM’s
Loaded plugins: fastestmirror, kmod
Setting up Update Process
Loading mirror speeds from cached hostfile
No Packages marked for Update

Moving to Next Step

STAGE 2 COMPLETED - RPM’s - Moving to Stage 3
PHP Fatal error: Call to undefined method DB_Error::query() in /usr/sbin/sysadmin_update_system on line 237

STAGE 3 STARTING - Misc Items

Moving to Next Step

Moving to Next Step

STAGE 3 COMPLETED - Misc Items - Moving to Stage 4

PHP Fatal error: Call to undefined method DB_Error::query() in /usr/sbin/sysadmin_update_system on line 237

STAGE 4 STARTING - Clean Up

PHP Fatal error: Call to undefined method DB_Error::query() in /usr/sbin/sysadmin_update_system on line 237
updatedb for locate command

Restart incron to be safe

STAGE 4 COMPLETED - Clean Up - Moving to Stage 5

PHP Fatal error: Call to undefined method DB_Error::query() in /usr/sbin/sysadmin_update_system on line 237

STAGE 5 STARTING - Final Verifications

PHP Fatal error: Call to undefined method DB_Error::query() in /usr/sbin/sysadmin_update_system on line 237

STAGE 5 COMPLETED - Final Verifications - Moving to Stage 6

PHP Fatal error: Call to undefined method DB_Error::query() in /usr/sbin/sysadmin_update_system on line 237

PHP Fatal error: Call to undefined method DB_Error::query() in /usr/sbin/sysadmin_update_system on line 237
Sat Feb 18 00:26:58 EST 2017 UPGRADE 100% COMPLETED
PHP Fatal error: Call to undefined method DB_Error::query() in /usr/sbin/sysadmin_update_system on line 237
PHP Fatal error: Call to undefined method DB_Error::query() in /usr/sbin/sysadmin_update_system on line 237
PHP Fatal error: Call to undefined method DB_Error::getOne() in /usr/sbin/sysadmin_update_system on line 255
REBOOT YOUR BOX NOW FOR DAHDI CHANGES TO TAKE EFFECT
If you would like to change your Asterisk release version to
11 or 13 you can do so by typing asterisk-version-switch
from the linux CLI at anytime

[root@freepbx-a /]#

Here is the services

[root@freepbx-a asterisk]# service --status-all
acpid (pid 1833) is running…
apcupsd dead but subsys locked
asterisk is stopped
auditd (pid 1650) is running…
avahi-daemon (pid 1801) is running…
cluster is running.
corosync (pid 2326) is running…
corosync-notifyd is stopped
crond (pid 2136) is running…
dnsmasq (pid 1902) is running…
drbd driver loaded OK; device status:
version: 8.4.6 (api:1/proto:86-101)
GIT-hash: 833d830e0152d1e457fa7856e71e11248ccf3f70 build by [email protected], 2015-07-20 01:39:28
m:res cs ro ds p mounted fstype
1:asterisk Connected Primary/Secondary UpToDate/UpToDate C /drbd/asterisk ext4
2:mysql Connected Primary/Secondary UpToDate/UpToDate C /drbd/mysql ext4
3:httpd Connected Secondary/Primary UpToDate/UpToDate C
4:spare Connected Primary/Secondary UpToDate/UpToDate C /drbd/spare ext4
fail2ban-server is stopped
hald (pid 1843) is running…
htcacheclean is stopped
httpd is stopped
incrond (pid 3590) is running…
ip6tables: Firewall is not running.
ipmi_si module not loaded.
ipmi_devintf module not loaded.
/dev/ipmi0 does not exist.
ipmievd is stopped
iptables: Firewall is not running.
iSymphonyServer
iSymphonyServer is stopped
iSymphonyServerV3 is stopped
started
No sensors found!
Make sure you loaded all the kernel drivers you need.
Try sensors-detect to find out which these are.
lvmetad is stopped
mdmonitor is stopped
messagebus (pid 1789) is running…
modclusterd is stopped
mysqld is stopped
netconsole module not loaded
Configured devices:
lo eth0 eth1
Currently active devices:
lo eth0 eth1
rpc.svcgssd is stopped
rpc.mountd is stopped
nfsd is stopped
rpc.rquotad is stopped
rpc.statd (pid 1760) is running…
ntpd (pid 1935) is running…
oddjobd is stopped
openvpn: service not started
pacemakerd (pid 2551) is running…
pcsd is stopped
master (pid 2124) is running…


A problem occured while reading the config file /etc/prosody/prosody.cfg.lua
Error: Error loading included /etc/asterisk/prosody_additional.conf: /etc/asterisk/prosody_additional.conf: No such file or directory

More help on configuring Prosody can be found at http://prosody.im/doc/configure
Good luck!


queue-callback.php is stopped
quota_nld is stopped
radiusd is stopped
rdisc is stopped
restapps.php is stopped
ricci is stopped
rpcbind (pid 1740) is running…
rpc.gssd is stopped
rpc.idmapd is stopped
rpc.svcgssd is stopped
rsyslogd (pid 1670) is running…
sandbox is stopped
saslauthd is stopped
openssh-daemon (pid 1917) is running…
svnserve is stopped
vsftpd is stopped

Router is stopped !

winbindd is stopped
xinetd (pid 1926) is running…
presence.php is stopped
[root@freepbx-a asterisk]#

Issue has been resolved by support.

1 Like