The FreePBX development team has identified with some zero day security vulnerabilities related to httpd and php. These vulnerabilities may allow a remote user to gain full root control over a system, and are present in lots of popular asterisk-related distro’s.
The FreePBX development team strongly urges all user of the FreePBX Distro to immediately upgrade their systems and patch these vulnerabilities. Additionally, users are reminded never to keep their web port accessible to the internet.
To secure your system, please download the latest scripts found here. Please remember that the upgrade scripts must be executed sequentially.
A big round of applause to my colleagues at Schmooze Com., Inc. for their tireless dedication to the community, for the sleepless nights they spent working on this (and many other!) issue, and for their swift response in releasing a patch to protect the users of the distro.
UPDATE: it seems this post has left a host of questions in its wake - please read the following replies to see if your questions have been answered yet, or reply with them if they havent been!
Guys at this time we do not have the specific CVE numbers and these could be older exploits. What was happening is we were getting reports in FreePBX Support of systems being hacked and were experiencing hackers going after our customers PBX’s and some of our core servers. The common thing we kept seeing was they were trying to get in through apache. We than had a customer report to us that they had a PCI Compliance scan done on there PBX and were told the following information.
PHP- Remote attackers may be able to gain unauthorized access to the web server, cause a denial of service or information disclosure, or execute arbitrary code. Resolution PHP should be upgraded to 5.2.17 or higher for 5.2.x, to a version higher than 5.3.6 for 5.3.x
APACHE- A remote attacker could crash the web server or execute arbitrary commands. Upgrade Apache 1.x to version 1.3.41-dev or higher, 2.0.x to version 2.0.64-dev or higher when available, or a version higher than 2.2.18
Once we upgraded apache and php to the version they recommended the servers that were being attacked were no longer being attacked. Before upgrading apache and PHP if we killed the running exploits on a server they would spawn back up a hour later. Once we upgraded both packages they were no longer being spawned.
This is all we know but for us it has stopped anyone that was being exploited. You will see that Apache in Centos is older than what is recommended here hence why we recommend that everyone upgrade.
The updates are done using the root account on the machine or using terminal. The update is downloaded onto the FreePBX machine, made executable, and then run. I always use the tmp folder, but any directory will do. The line-by-line commands used for this latest (8/11/11) update would be:
If you don’t know the actual bugs that need to be fixed, how did you fix them? How do you know the system is now secure? And what about those of us who use FreePBX on non-distro OS? Sorry but this “security advisory” is crap as it provides no useful advice whatsoever. This is a technical group; you need to provide some actual technical details.
[root@mysystem tmp]# chmod +x upgrade-1.8.2.0-2.sh
[root@mysystem tmp]# ./upgrade-1.8.2.0-2.sh
-bash: ./upgrade-1.8.2.0-2.sh: /bin/sh^M: bad interpreter: No such file or directory
Okay, I did this and it went fine until it got to Stage 3, then it got a bunch of PHP warning messages - see http://pastebin.com/Z4mY1hHw
EDIT: I think I fixed the complaints by doing
yum update php-mcrypt
BUT I don’t know if the script completed properly or not due to the complaints - do I need to re-run it? Stages 1 and 2 seemed to go fine.
EDIT #2: And where did php-mcrypt come from in the first place, I hear you ask, since it’s not part of the FreePBX distro? Well just a couple of days ago I installed phpMyAdmin, because I think it’s a real handy tool to have, and I wanted to get a bunch of “garbage” records out of the CDR that had been generated by an endless loop I had inadvertently created (my own dumb mistake). And when I installed phpMyAdmin, it brought along libmcrypt and php-mcrypt as dependencies. I tried updating both, but only php-mcrypt actually had an update available today.
When I do cat /etc/asterisk/freepbxdistro-version it seems to show that the update “took” so I guess I’m okay?
If I had more info I would provide it. We have no been able to figure out what the exact exploit was and have spent the past 5 days straight on it and even brought in some outside consultants. All we know is that once we upgrade to the latest apache the exploits die off as they can not get back into the box but if we downgrade back than we see them back in and doing port scans from the box to other boxes mainly trying to force there way into other peoples apache boxes or use there boxes for email relay.
Believe me we are all very technical guys and are just as frustrated over this as the next guy.
What do you propose next time we just keep out mouth shut and let others get exploited as we have seen in the forums has already happened. Some information is better than none don’t you agree.
billsimon,
Unfortunately, in the world of security vulnerabilities you can really never know if the NEXT vulnerability will affect you. As Tony mentioned:
"Once we upgraded apache and php to the version they recommended the servers that were being attacked were no longer being attacked"
and hence we feel that these upgrade are necessary for anyone that wants to protect agains the issue that we saw.
Additionally, as Tony mentioned, anyone on other distors need only ensure that there running patched version of php and apache:
and
Finally, we have always stressed that FreePBX is built to be a front end to asterisk - useful for a sysadmin or a group of sysadmin. It was not intended to be a public facing web app/site. Hence, we always recommend to user that the web portion is NOT ACCESSIBLE from the internet. While we take FreePBX’s security seriously, this vulnerability is way beyond the scope of FreePBX (or anything that we could patch/fix at the FreePBX level) and should help to convince users to keep their boxes off the net at all costs.
I’m sure everyone appreciates the warning but the problem is there is not enough information to work out what the compromise is. I know you say you don’t have enough information but some more clues would be good.
For instance, you say you saw lot of boxes that had root compromised. I’m assuming that you did not just upgrade Apache/PHP on these same boxes and the problem was cured?
So I’m assuming you updated Apache and PHP on others boxes, and have not seen these ones compromised. But how do you know they just haven’t been compromised yet? If you don’t know what the attack vector is how are you detecting it?
If this was a general issue affecting the current released version of Apache and/or PHP with CentOS then this would be a widespread issue, widely reported surely? Not just with FreePBX.
So isn’t it more likely that it’s part of FreePBX being compromised?
Also, do you know for sure the root account was compromised? You don’t need root access to scan for other servers to attack.
Either way more information about what you were seeing in the Apache logs, what form the attacks took, and how to detect if your servers are being scanned would definitely be welcome.
We would ps awx and see a ton of process called exim running which was executing some perl script and most of the time you could see a SCREEN session running.
If you did a screen -x you could watch the port scan running. In each example we would see they were running scans of IP’s for /phpmyadmin or /myadmin. At other times they were doing scans to find email SMTP relay servers to exploit from other peoples boxes.
Most of these scripts were being fed from other hacked boxes running IRC servers to command and control the IP addresses the scripts should be executing.
Most of the time the scripts were being save in /usr/game/
That was my point above - you can never know 1000% for sure. In this case it definitely helped with at least some issues. Are you a million precent safe after you apply the upgrades? Only time will tell. Unfortunately, we do not have the resources to employ the cream of the crop in security people to try to detect every posible theoretical issue.
indeed.
No, this issue was also seen on systems that weren’t running FreePBX at all
[quote]
Also, do you know for sure the root account was compromised? You don’t need root access to scan for other servers to attack.[/quote]
No. However the hacks were extremely sophisticated (as described by the security consultants we brought it in) and we have reason to suspect “they” may have had root-level access
Yes, I will agree that some information is better than none. Thank you. Have you gotten any response from Apache on this issue? Attempting to research it is frustrating. Google blocks the search for “apache exploit” in quotes, claiming no results. (Really?) Apache’s change log for httpd from 2.2.18 to .19 shows only one patch:
[quote] -- coding: utf-8 --
Changes with Apache 2.2.19
*) Revert ABI breakage in 2.2.18 caused by the function signature change
of ap_unescape_url_keep2f(). This release restores the signature from
2.2.17 and prior, and introduces ap_unescape_url_keep2f_ex().
[Eric Covener]
[/quote]
If 2.2.18 has the vulnerability and 2.2.19 does not, then more was done than was written here in the log.
Yes I would love to know more also but after investing 5 grand of money into finding the root exploit and not getting anywhere we gave up as we just do not want to keep spending money on the issue since the upgrade resolved it for us.
All I can say is when we killed the running processes on a box they would spawn back up within 30 minutes. After upgrading apache and php as outlined above and killing the processes they never started back up on any boxes that we did this on. That was enough for us to close out our 5 days of hell and move back to productive work.
