Apply config caused error. Had to start httpd manually. Now HA is broke

[Kafluke]

I had an error on Apply config. Afterwards I couldn’t get to the gui. Ran “service httpd restart” and the gui came back up but ever since applying config I’m getting about 2 of these emails every minute:

Cluster notification

====

Operation st_notify_fence for resource - on freepbx-a failed: Operation st_notify_fence requested by freepbx-b for peer freepbx-a: Timer expired (ref=fde36091-90b0-4d07-a4d3-bbe0629e39ca)

Details:

operation status: (0) complete

script returned: (-62) Exceptionally unknown error

expected return value: (0) OK

And everything in High Availability is blank:

[BlazeStudios (Tom Ray)]

You’re going to need to open a support ticket with Sangoma. This is a commercial product and comes with support.

[avayax (Johann Zurner)]

What is the output of pcs status?

[Kafluke]

[root@freepbx-a asterisk]# pcs status
Cluster name: freepbx-ha
Last updated: Tue Mar 17 17:52:27 2020
Last change: Mon Mar 16 22:53:25 2020
Stack: cman
Current DC: freepbx-b - partition with quorum
Version: 1.1.11-97629de
2 Nodes configured
26 Resources configured


Node freepbx-a: UNCLEAN (online)
Online: [ freepbx-b ]

Full list of resources:

 spare_ip       (ocf::heartbeat:IPaddr2):       Started freepbx-a
 floating_ip    (ocf::heartbeat:IPaddr2):       Started freepbx-a
 Master/Slave Set: ms-asterisk [drbd_asterisk]
     Masters: [ freepbx-a ]
     Slaves: [ freepbx-b ]
 Master/Slave Set: ms-mysql [drbd_mysql]
     Masters: [ freepbx-a ]
     Slaves: [ freepbx-b ]
 Master/Slave Set: ms-httpd [drbd_httpd]
     Masters: [ freepbx-a ]
     Slaves: [ freepbx-b ]
 Master/Slave Set: ms-spare [drbd_spare]
     Masters: [ freepbx-a ]
     Slaves: [ freepbx-b ]
 spare_fs       (ocf::heartbeat:Filesystem):    Started freepbx-a
 Resource Group: mysql
     mysql_fs   (ocf::heartbeat:Filesystem):    Started freepbx-a
     mysql_ip   (ocf::heartbeat:IPaddr2):       Started freepbx-a
     mysql_service      (ocf::heartbeat:mysql): Started freepbx-a
 Resource Group: asterisk
     asterisk_fs        (ocf::heartbeat:Filesystem):    Started freepbx-a
     asterisk_ip        (ocf::heartbeat:IPaddr2):       Started freepbx-a
     asterisk_service   (ocf::heartbeat:freepbx):       Started freepbx-a
 Resource Group: httpd
     httpd_fs   (ocf::heartbeat:Filesystem):    Started freepbx-a
     httpd_ip   (ocf::heartbeat:IPaddr2):       Started freepbx-a
     httpd_service      (ocf::heartbeat:apache):        FAILED freepbx-a
 Clone Set: ClusterMon-SNMP-clone [ClusterMon-SNMP]
     Started: [ freepbx-a freepbx-b ]
 Clone Set: ClusterMon-SMTP-clone [ClusterMon-SMTP]
     Started: [ freepbx-a freepbx-b ]
 fence_a        (stonith:fence_ipmilan):        Started freepbx-a
 fence_b        (stonith:fence_ipmilan):        Started freepbx-a

Failed actions:
    fence_a_start_0 on freepbx-b 'unknown error' (1): call=320, status=Timed Out, last-rc-change='Mon Mar 16 22:53:25 2020', queued=0ms, exec=30023ms
    httpd_service_stop_0 on freepbx-a 'unknown error' (1): call=153, status=complete, last-rc-change='Tue Mar 17 14:35:03 2020', queued=0ms, exec=27026ms
    httpd_service_stop_0 on freepbx-a 'unknown error' (1): call=153, status=complete, last-rc-change='Tue Mar 17 14:35:03 2020', queued=0ms, exec=27026ms