Please let me first apologize for the impact that yesterday’s outage of Sangoma infrastructure had on your business. We understand the trust you have placed in us to perform reliably, and we did not live up to those expectations yesterday. For that we are truly sorry, and commit to using this event as a learning experience to improve our service to you.
In summary, yesterday we suffered a major outage of our internal database cluster that powers many of the public facing services we provide. This outage had many impacts on our customers including the Sangoma Portal, Support Ticketing systems, FreePBX and PBXact updates, and many other business transactions with Sangoma. While many of Sangoma’s Cloud Services customers were unaffected, the event did create a service disruption for some SIPStation customers, as well as an outage on our FAXStation, VPN and SMS services.
As to what specifically happened, at approximately 2:08PM CST yesterday, one of our production database clusters crashed, and indicated data corruption as a result of the crash. Subsequently we were unable to simply restore it to a working state. Our internal infrastructure teams worked through the night to restore the data, redeploy the database (in a manner that would not be susceptible to another similar problem), and return the services to normal operation. All services were fully restored by 6AM CST today.
In addition to the outage, there was also a lack of effective communication and support for our customers and partners during the event. While there was not a lot of definitive information we could provide during the early portion of the outage, we should have done a better job of keeping you informed about the scope of the outage, and our progress. Some of you have expressed your displeasure with our lack of communication, and we certainly heard you, and commit to doing better in the future. As we continue our investigation into the root cause of this event, we will provide more details over the next few days, along with a plan for corrective actions.
We know that you depend on Sangoma products and services to run your business, and our goal is to exceed your expectations. Yesterday, we fell short of that goal, and for that we apologize. We take the availability and reliability of our services and infrastructure very seriously, and will take steps to prevent this in the future. Thank you for your patience, understanding and continued support.
