Both PBX HA Nodes stuck in Standby mode after reboot

Commercial Modules
1

FreePBX High Availability

You have connected to a standby server. Asterisk is not running
on this machine. Very few commands are available.

Usually, the only thing you do on a standby node is yum update.

how can i get one of the nodes running again?

2

Hi @rbrumitt you need to manually bring it out of standby via the command line.

For example to bring out of standby of node-a run the below command:-
" pcs cluster unstandby freepbx-a"
https://wiki.freepbx.org/display/FPG/FreePBX+HA-Critical+CLI+Commands+to+know

3

@pramarajan how do i know what the real node name is?

4

@pramarajan or better than that, what do i do after that?

5

HI, @rbrumitt I think you can get the node name from the “pcs status” command output.
https://wiki.freepbx.org/display/FPG/FreePBX+HA-PCS+Status

6

can you share the “pcs status” command out put here?

7

@pramarajan
[root@freepbx-a ~]# pcs status
Cluster name: freepbx-ha
Last updated: Thu Mar 19 02:24:07 2020
Last change: Thu Mar 19 02:11:52 2020
Stack: cman
Current DC: freepbx-b - partition with quorum
Version: 1.1.11-97629de
2 Nodes configured
26 Resources configured

Node freepbx-a: UNCLEAN (online)
Online: [ freepbx-b ]

Full list of resources:

spare_ip (ocf::heartbeat:IPaddr2): Started freepbx-a
floating_ip (ocf::heartbeat:IPaddr2): Started freepbx-a
Master/Slave Set: ms-asterisk [drbd_asterisk]
Masters: [ freepbx-a ]
Slaves: [ freepbx-b ]
Master/Slave Set: ms-mysql [drbd_mysql]
Masters: [ freepbx-a ]
Slaves: [ freepbx-b ]
Master/Slave Set: ms-httpd [drbd_httpd]
Masters: [ freepbx-a ]
Slaves: [ freepbx-b ]
Master/Slave Set: ms-spare [drbd_spare]
Masters: [ freepbx-b ]
Slaves: [ freepbx-a ]
spare_fs (ocf::heartbeat:Filesystem): Started freepbx-b
Resource Group: mysql
mysql_fs (ocf::heartbeat:Filesystem): Started freepbx-a
mysql_ip (ocf::heartbeat:IPaddr2): Started freepbx-a
mysql_service (ocf::heartbeat:mysql): FAILED freepbx-a
Resource Group: asterisk
asterisk_fs (ocf::heartbeat:Filesystem): Stopped
asterisk_ip (ocf::heartbeat:IPaddr2): Stopped
asterisk_service (ocf::heartbeat:freepbx): Stopped
Resource Group: httpd
httpd_fs (ocf::heartbeat:Filesystem): Stopped
httpd_ip (ocf::heartbeat:IPaddr2): Stopped
httpd_service (ocf::heartbeat:apache): Stopped
Clone Set: ClusterMon-SNMP-clone [ClusterMon-SNMP]
Started: [ freepbx-a freepbx-b ]
Clone Set: ClusterMon-SMTP-clone [ClusterMon-SMTP]
Started: [ freepbx-a freepbx-b ]
fence_a (stonith:fence_ipmilan): Started freepbx-a
fence_b (stonith:fence_ipmilan): Started freepbx-a

Failed actions:
fence_a_start_0 on freepbx-b ‘unknown error’ (1): call=104, status=Timed Out, last-rc-change=‘Thu Mar 19 02:16:55 2020’, queued=0ms, exec=30023ms
mysql_service_stop_0 on freepbx-a ‘unknown error’ (1): call=117, status=complete, last-rc-change=‘Thu Mar 19 02:17:56 2020’, queued=1ms, exec=33ms
mysql_service_stop_0 on freepbx-a ‘unknown error’ (1): call=117, status=complete, last-rc-change=‘Thu Mar 19 02:17:56 2020’, queued=1ms, exec=33ms

8

@pramarajan i am stuck as i can not get asterisk to start, I have little time to get this up and online before work starts in a couple hours.

9

@pramarajan
[root@freepbx-a ~]# amportal start

Please wait…
PHP Warning: require_once(/var/www/html/admin/bootstrap.php): failed to open stream: No such file or directory in /etc/freepbx.conf on line 9
PHP Fatal error: require_once(): Failed opening required ‘/var/www/html/admin/bootstrap.php’ (include_path=’.:/usr/share/pear:/usr/share/php’) in /etc/freepbx.conf on line 9

PHP Warning: require_once(/var/www/html/admin/bootstrap.php): failed to open stream: No such file or directory in /etc/freepbx.conf on line 9
PHP Fatal error: require_once(): Failed opening required ‘/var/www/html/admin/bootstrap.php’ (include_path=’.:/usr/share/pear:/usr/share/php’) in /etc/freepbx.conf on line 9
**** WARNING: ERROR IN CONFIGURATION ****
astrundir in ‘/etc/asterisk’ is set to but the directory
does not exist. Attempting to create it with:
'mkdir -p ’

mkdir: missing operand
Try `mkdir --help’ for more information.
**** ERROR: COULD NOT CREATE ****
Attempt to execute 'mkdir -p ’ failed with an exit code of 1
You must create this directory and the try again.

10

It looks something a problem with your node-a and due to that MySQL service is failed.
Can you try with below points one by one and check the status.

  1. Check the disk space on node-a

  2. Try to clear the MySQL error by running the below command.
    “crm_resource --resource mysql_service -C --node freepbx-a”

  3. Power off Node-a and then reboot node-b.

If the above points didn’t help then please contact our support by raising a ticket at https://support.sangoma.com/

11

@pramarajan - node b rebooted and came back up in standby mode

12

@pramarajan I have like one hour, do you have any ideas what else i could try?

13

@pramarajan do you know of any other things I could try?

14

We need to login on your system to dig further so I would recommend you to raise the ticket @ support.sangoma.com

15

I opened a ticket, support is online in 2.5 hours

16

There is a cluster fix script you can try:
https://wiki.freepbx.org/pages/viewpage.action?pageId=28771168

17

same thing:

18

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.