Online Module Management
The Challenges of Serving Millions of Diverse Installations
FreePBX has grown in many ways over the years and that growth has been both vertically and horizontally. Vertically, there are millions of installations, many of which check back with our servers daily to get up-to-date modules and security notices. Horizontally there are multiple distributions, multiple PHP versions, and multiple Asterisk versions to account for. Some of these factors affect which modules and which module versions will be presented to a system as potentials for upgrades.
The online Module Admin system has been around since 2006, and for adventurous developers, all the tools have been and still are available to generate and create your own repository. In fact, several other projects over the years have done exactly this. In the past, the process was straight forward. The packaging tools we created would mark a module with a version number, check source code into SVN, create the tarball, calculate an MD5 hash of the tarball, and then check that back into SVN. To create an online repository, you could simply pull all the latest module.xml files, concatenate them together into a single XML file, put that up on your server and put the corresponding tarballs in the appropriate location and you were done.
Growing Demands
As the project grew, the demands from our customer base also grew and put strains on this process to the breaking point. We needed to find ways to:
Dynamically generate an XML manifest that could vary depending on the version of FreePBX, Asterisk, PHP and even the Distro you are running. This is needed to allow us to choose the proper version of a module, or not include a module that might break a given system.
We were asked to provide the ability to go backwards, offering up previous versions of modules so people could revert to older versions.
More security, a way to not only authenticate the tarball and its contents at the time of delivery, but to retain that manifest and regularly check the integrity of the installation to detect a hacked system that has had portions of modules modified or entirely new modules installed on the system that were not intended and possibly malicious(1).
An ability to transmit security notifications about known vulnerabilities and the required modules versions to patch them.
Migrating to Git
The online system was already strained at the time we migrated from SVN to Git two years ago. Furthermore, the migration implied that all of the tools would have to be re-written since much of the process was specific to SVN. As such, we started to attack this problem at the same time we made our move to Git.
The first step was to change how we would represent module branches in a Git repository. In the SVN era, each major release of FreePBX required the entire set of modules to be branched, even if the exact same module was used on multiple versions of FreePBX. This was largely done to support the online system and tools which required this structure and made it difficult to maintain multiple versions of the same module. With Git, we defined a new branching standard that would allow each module to contain only as many branches as were necessary for different versions of the module. The <supported> tag in the module.xml of a branch indicates the major versions of FreePBX that branch works with. Using this, the tools that interact with the Git repository can determine which branch to use on the target FreePBX version. The new structure also fits well with Git and has made it much easier for the community to contribute and collaborate on module development.
All of this is managed by the development tools, also available in Git, for developers to create and tag modules. By using the supplied management script, package.php, "minor branch tags" are inserted into the module's repo delineating the minor branch points of a module. The script also statically checks for and tries to block errors such as PHP syntax violations or other integrity issues. Once properly packaged back into the Git repo, the tags can be used to find and pull out any specific version of a module for subsequent preparation into a consumable tarball for FreePBX. There is also a module.sig "packaging slip" that can be created by the Git devtools using sign.php. This is discussed in more detail below. Furthermore, we created the ability for FreePBX to accept any name combination file during upload. This means not only does the old format of "module-1.1.1.tgz" work but even Github's format of "master.zip" works since FreePBX now reads the module.xml to figure out which module it is, and if a URL is provided, will fetch it from there first.
Dynamic Manifest Generation
Along with moving to Git, we needed to find a new way to address the dynamic nature of module delivery and the differing requirements that were listed above. A single XML manifest per FreePBX version would not suffice. As such, we ended up rebuilding the entire backend of our delivery system and moving it from statically generated XML files to be completely database driven. When a system checks for module updates now, what they get back looks very similar to what they always got back, an XML manifest with information about the latest versions of the different modules that will work on their system. The creation of that manifest is now generated on the fly, at the time the request is made, upon receiving various information such as their versions of FreePBX, PHP and Asterisk as well as the Distro type being used. This mechanism allows us, for instance, to not offer up a module that would require PHP 5.4, to a system running PHP 5.3, which would otherwise result in their system crashing with a "white screen" because of the incompatibility. This same mechanism also allows us to determine previously available module versions to offer up, compile up a list of security vulnerabilities that we may want to inform the system about, and more.
Originally this system consisted of a single XML file, modules-<version>.xml containing a concatenated list of all the module.xml files. It was later enhanced to deliver a second file, security-<version>.xml which provided security CVE announcements that would allow FreePBX to inventory itself and determine if there were any known vulnerable modules that urgently needed to be updated. In version 12, with the new rollback and beta features we added two more XML files, namely old-<version>.xml and beta-<version>.xml. The proliferation of XML manifests began to make slower systems run very sluggishly simply to check for updates. We thus combined all 4 manifests into a single one, called all-<version>.xml resulting in significant performance gains on both the FreePBX client side as well as the server side distribution(2).
Module Security
Our dynamic online system has been in place for over two years now and solved some aspects of the original problem. However, it didn’t solve all the evolving security concerns. There are many aspects of security that have been, and continue to be addressed within FreePBX as we take a 10 year old application and continue to "rebuild" it from the inside out with security, stability, usability and features in mind.
However, the very modular nature of FreePBX in conjunction with it being written in PHP, which is a scripted language, provides many opportunities for malicious attackers to compromise a system. This meant we had to come up with a solution that would not only allow you to authenticate a module being downloaded to your system, but also a way to continuously monitor that same system against malicious changes once loaded or a malicious module being installed undetected.
In the past, FreePBX did nothing more then check that the MD5 hash of the tarball matched what the online XML manifest said it should. If you loaded a module manually there was no way to authenticate it. We thus took the path that many other open source projects have followed and implemented GPG signing and verification into the process.
Module security starts when a developer wants to release a module. As part of the development tools (licensed under the GPL, and available via Git) we run sign.php which creates a simple manifest, module.sig, that is nothing more then a catalog of every file that is included in that module, along with a SHA1 hash for each of those files. This is effectively a "packaging list" that, once prepared, is signed by the developer's trusted private key so its authenticity can be verified. All of the files from Git, for this tagged version, are then included and packaged into a compressed tarball that is generated for distribution so that FreePBX can ultimately authenticate the integrity of each included file.
Next, we take that tarball, which is effectively a "binary file" and add another GPG binary signature to it so that it can be verified before ever opening it up once delivered to a system for installation. At this stage there are two tarballs a TGZ and a GPG that are available for download.
To make an analogy, each tarball of source code (and other files) contains a packaging list in it. That packaging list, which is not code, is “sealed” so that if tampered with, a consumer can detect such. The entire package is then wrapped up for shipping, and on the outside of the package, we put a type of "tracking label," just like FedEx might do. The recipient can then examine that label to determine if the package is authentic before even opening it. Once you open it, you can check the seal on the package list, just imagine that wax seal with the king’s stamp on it…, and then you can use the package list to confirm that the contents of the package are intact.
What does all of this do? After checking for updates and receiving the dynamically generated XML manifest of available modules, the user may now choose to update or install some or all modules presented to it. The manifest tells FreePBX where to get those updates, which are just the above mentioned signed TGZ or GPG tarballs. FreePBX then proceeds to download the GPG signed tarballs one at a time. The first part of the GPG validation involves obtaining the most recent public keys available from the online GPG Web of Trust. For systems that don't have internet access and are loading modules locally, they will attempt to fall back to a manually distributed list of keys. Once the tarball is on the system, the key is used to authenticate the package. Only after the package is authenticated is the tarball extracted. This assures that we don't extract the content of a tarball if it is signed by a key that can’t be authenticated.
Once authenticated and the package has been opened, we are ready to run the module’s install script to upgrade (or install) the module. The files are extracted and FreePBX is able to use the signed "packaging list" to verify each file against its SHA1 hash to make sure it's exactly what was checked out from the publically available Git repository used to originally prepare it(3). This same "packaging list" is used to verify the integrity of every module on the system on a regular basis, to detect if files have been maliciously tampered with and alert the system if it detects such. If there is no packaging list or the signature can't be verified, the system will also alert you which is how Trojan horses(1) can be detected.
Summary
The code required to generate the key components of this process is part of our "devtools" repository, and the documentation to use it is explained in the corresponding Wiki. We want, and encourage, people to extend and develop our code, which is why we license it under the GPL (and the AGPL). We make sure that all of it is mirrored to Github (with a 'Fork This!' button on the top of every page), and we try our hardest to make sure that everything is thoroughly documented.
When you look at the underlying workings of the FreePBX Module Admin code, or the XML manifest that gets returned, most of it has minimal change since 2006 when the first Online System was introduced. The addition of some security monitoring components, GPG signing, rollbacks and module beta release tracking are the more evident changes. That is quite a testament to the original designers and their very simple but elegant solution that has been able to stay largely intact.
The simplicity and elegance of the original XML format has made it easy for us to ensure that there aren't any incompatibilities between distributions using the old way of generating an online module list and the new way, which also links in security alerts and the ability to roll back to previous versions along with beta releases.
However, the huge changes required to scale vertically with the volume of users and horizontally with the diverse range of versions and Distros currently putting demands on the system has been an immense undertaking and a tribute to the FreePBX developers who have continued to engineer a world class system while maintaining the legacy of compatibility and virtually no bumps to the installed base. The project has become more scalable, stable and secure as a result of these changes and innovations and the success of the project and diversity of the community who uses it is a result of this hard work.
Â
  (1) When we first introduced module signing we were immediately alerted by several customers to a rapidly spreading virus on hacked systems. The particular exploit was not a FreePBX exploit in this case. These hacked systems had a new, unauthorized module installed on them that was named "Admin Dashboard." Since FreePBX is a modular system, it allows new modules, user modules, etc., to be installed. This module was actually a "Trojan horse" which would allow unauthorized remote access into a FreePBX system. Prior to the signature checking, and without it, this module would not have been detected. With the new security measures put in place the customers were quickly notified of the unauthentic and suspect module. A few of them notified the project and we were able to take other measures in addition to the signature signing to inform the community as well as active measures to "search and destroy" in order to squash this exploit as quickly as reasonably possible.
  (2) Prior to combining the 4 XML manifests into a single all-<version>.xml, we were seeing some slower systems, such as Raspberry PIs, take 40 seconds to download and process the set of manifests. This was simply to get to the point of presenting what modules are available for update. The inefficiencies were accounted for at all stages of the process. Four separate requests to the server which means setting up and tearing down TCP/IP connections four times along with the round trip latency. Four instances of the server side parsing the requests, querying the database and generating the various manifests dynamically on the fly. But most notably, four sets of XML manifests that need to be parsed by the local machine for internal representation of which the XML parser is a fairly significant computational task. With these changes, this has been reduced down to 2 seconds or less in some instances.
  (3) The module.sig "packaging list" can and is used not only on open source modules but any FreePBX module, whether obfuscated, Zended, or any other format that FreePBX is able to consume.
Â
