Announcement: New Advanced Recovery module for FreePBX

Hey All,

We just released a new module for FreePBX called the Advanced Recovery module. This module can help FreePBX users and administrators to ensure better uptime and availability (in case of system failure) through automated data migration and regular syncing to a hot spare. It can also automatically provisions phones to use both systems in case of failure in the primary. You can read more about it at:

Best wishes,
Matthew Fredrickson

8 Likes

I have a few questions regarding the technical implementation of this module, especially the differences between that and FreePBX High Availability, which we still have running on our installations.
Please correct me if one of my assumptions below are incorrect.

-With this module installed, unlike with the old HA where I had a floating IP address, I have two separate machines each with its own IP address.
-They have identical configurations, except that on the standby server the trunks are disabled to avoid registrations coming from two machines simultaneously.
-My non Sangoma phones need to be configured to register to the backup server address if registration to primary is unsuccessful.
-Services like Asterisk, FreePBX, etc are all running on both machines no matter whether in standby or active.

-The primary machine is monitored for the status of Asterisk, FreePBX, database and its network interface.
-If any of those is down a failover will take place which basically only means the trunks are enabled on the standby server (plus execution of a custom hook that I might configure, as well as notifications).

Does option “Bring down Primary server after switchover” mean that primary server will be powered off in case of a failover event?

1 Like

Hi @avayax Thanks a lot for bringing up your queries. As your queries are quite valid so I have captured them in wiki thinking it might helps others as well.

Please refer to https://wiki.freepbx.org/display/FPG/Advanced+Recovery#AdvancedRecovery-FrequentlyAskedQuestions and let me know if you still have any further queries.

Thanks
Kapil

2 Likes

Do non Sangoma phones need to be configured to register to the backup server address if registration to primary is unsuccessful.
A: Yes we need to manually set the Fall back destination or sip server address to SIP phones so during the primary system failure time, phone can get register to secondary system.

I mean outside of the poor grammar and structure of this sentence, this means that only Sangoma/Digium phones have a way to be updated via the EPM. So that would mean anyone using other phones would have to touch all their phones manually to deploy this solution.

There are plans to make this for work other devices in the EPM right?

EPM supports hundreds of devices. If there is no native EPM support for backup registrar config in EPM for a specific device, then we’ll need feature requests from interested users:
https://wiki.freepbx.org/display/SUP/How+to+open+a+feature+request

Our initial release target was to support our phones first (for the automatic provisioning of the backup SIP server) - to a large degree to keep the development schedule for this module in, but sales of our phones very directly help fund the development of the FreePBX project as well. Support for other phones is under consideration, but we’re still trying to decide how to handle it.

EDIT: Just to clarify, the module doesn’t disallow support for other vendor’s phones with the Advanced Recovery replication and failover functionality, just the easy configuration support for setting up phones for both servers (tight integration with EPM) has initially only been released for Sangoma’s S and D phones. You can manually setup primary and backup servers appropriately for your vendor’s phone (if it supports that style of registration).

I apologize, I think my statement above might not have made that clear.

Best wishes,
Matthew Fredrickson

Thanks.
More questions:

-What does it mean exactly when a trunk is in a disabled state on the standby server?

I am mostly using IP authentication and not registration, so after a switch over I would have to change a rule on my firewall to forward inbound traffic to the correct server.

-What is the “Exclude bind address” option for, when would I want to include bind address in the syncing?

It’s not up. If you’re using IP auth all you need to do is have both IPs listed with your carrier and setup to be using for inbound in a failover/sequenced manner. Disabling the trunks stop any trunks that register from registering when they shouldn’t.

The bind address is the IP address Asterisk is bound to. So you have two servers each with their own IP. When the configs come over from the Primary server you don’t want the backup to bind Asterisk (or other services) to the IP of the primary server. It will break things.

You can test this yourself by disabling a trunk on your system. In older versions, this setting ONLY disabled the registration for the trunk. On current 14-15 systems, disabling a chan_sip trunk removes the registration and the user/peer details. Disabling a pjsip trunk removes everything, registration, aor, endpoint, auth, etc.

It’s a holdover from 13HA. Generally you are going to leave the bind address at the default 0.0.0.0, and if you do then there is no issue. If you have a technical reason to change the bind address, it’s possible (likely even) that setting will break something if you migrate it to another system. In those cases, you have the option of excluding it from the backup.

1 Like

Hi to all,

Actually this module is based on warm spare concept and is using API and backup and restore module. So, it’s different with HA module or other HA solution with Pacemaker/DRBDand etc.

I know for 2 telephony servers, there is split brain issue, but as same as the name, this module is not HA replacement for v14 or 15+.

Please correct me if it’s wrong.

BR

Yes it is.
It’s the new HA alternative and the drbd/pacemaker solution is gone, which was quite complex and never worked well anyway.

Sure it can happen that when the two servers lose connection to each other you will end up with some phones registering to one server and others to the other, but imo that will be easier to recover from than some of the issues that happened on my old HA installs, which were unstable, with the whole cluster sometimes crashing and both servers being out of service. Issues that can be quite difficult to troubleshoot.

Upgrades will also be easier and less prone to causing problems as you’ll basically just have a regular FreePBX install with only one module being different.

What we still have in place from the old HA is a direct fiber connection between the two servers for inter-cluster communication and I am wondering if that can be reused somehow with the new module so that the two servers monitoring each other can be done over the direct link.
That would be to avoid triggering the switch over due to things happening in the network where the servers otherwise could not be talking to each other but where a failover would actually not be desired.

As long as both machines are routable to each other in some way or another, I don’t think the module cares if it’s on a dedicated link or not. @kgupta1, do you agree?

I agree with @mattf. As long as both servers can talk to each other, this module is able to do monitoring and having dedicated fiber link between server is even much better as that will provide faster network connectivity or bandwidth to ensure syncing from primary to secondary is happening quite fast as well.

1 Like

i have the HA license. is this a separate purchase ?

My servers will each have two NICs. One will be providing regular access to the network to which SIP will bind to, and another NIC to directly connect to the other FreePBX server.

How do I configure that the monitoring and syncing between servers will happen over the direct link?
What needs to be done there?

1 Like

Hi @avayax Nice question.

I have added the same into FAQ section of wiki for future/others help as well. Please refer to last FAQ in below section,

https://wiki.freepbx.org/display/FPG/Advanced+Recovery#AdvancedRecovery-FrequentlyAskedQuestions

Please let me know if you have any further query.

Best Regards
Kapil

Is there a possibility to live sync Queue Members as well as voicemail data, or maybe at least every few minutes?

Other data, such as CDR or extension changes are not so crucial to be synced over immediately, but voicemail and queue member data can make a mess when switching between nodes.

Also, I noticed when using the EPM failover section for the Sangoma phones, it asks you to specify a port, so if you use ChanSIP and PJSIP, extensions with both technology will try to connect on the same port.

1 Like

Hi @PitzKey

Other data, such as CDR or extension changes are not so crucial to be synced over immediately, but voicemail and queue member data can make a mess when switching between nodes.

Good point, could you please raise aFeature request.

Also, I noticed when using the EPM failover section for the Sangoma phones, it asks you to specify a port, so if you use ChanSIP and PJSIP, extensions with both technology will try to connect on the same port.

If you check, EPM integration section only asking(or confirming) Backup server IP to make sure EPM is updating proper fallback destination IP in phones, the port number will be internally filled by EPM based on the extension (pjsip or chan_sip).

Best Regards
Kapil