There was nothing to report to apache since the upgrade resolved the issue for the boxes we were working on.
Okay, this is upsetting…
I have pretty strong feelings about doing upgrades that take over your system. Basically, it’s my feeling that it’s MY system, not YOURS, even if I am running your software. One of the reasons I’ve declined to use a competing distro is because I feel they take just a little too much control away from a system administrator.
I’m not sure of everything this upgrade did, but it definitely did one thing I don’t like, and may have done another (I can’t be sure). The thing I’m not sure about is that now ALL FreePBX modules are enabled, and I seem to recall that after the original distro install two or three were disabled by default. This isn’t a big deal as long as the newly-enabled modules (if any) don’t screw up anything on my system.
But the thing that more upsets me is that I happened to go into /etc/yum.repos.d/ and noticed that all repos except the FreePBX repo were GONE - I mean they were totally deleted! I personally only had two additional repos — the dag repo and the webmin repo — but some may have taken the trouble to install others.
Now, i understand that during an upgrade you may not want to be pulling software from a repo you can’t trust. Fair enough, but it’s overstepping your bounds to simply delete any other repos that someone may have installed. What the script SHOULD do is move them to a safe place, and then at the end of the script, move them back. If you want, you can even print a warning that there are repos that you think shouldn’t be there, and ask the system administrator to confirm that they are really wanted. But ultimately, you shouldn’t be deleting anything that the administrator may have taken the trouble to install on a system, because down the road he or some automated process might assume it’s still there. I can see why you might not like the dag repo being enabled during an upgrade, but I can’t see how the webmin repo would cause you any issues, and Webmin probably depends on it to do software and module upgrades.
Please don’t turn into that “other” distro that treats us like we’re all children!
By the way, is there anything else you delete or change during the update process that we might want to know about (besides actual software updates, of course)?
You can simple look at the upgrade script and see what it does. We do not compile or hide what is in our update scripts. We always recommend that you look at upgrade scripts regardless what platform you are using them from.
tonyclewis, I’m not a programmer. And I’m just saying, there are certain boundaries you shouldn’t cross, like not permanently deleting software (or repositories) that a user may have installed.
Would it really be that hard to be considerate and copy anything you don’t want present to a safe place, then restore it after the fact? Even if I can figure out what you are doing in your scripts (which might not always be that easy), there are doubtless users that are far less technically astute, that could look at your script all day and not really understand what you’re doing. I’ll grant that such users are probably less likely to be installing additional software or repos in the first place, but just because it’s less likely doesn’t mean it’s impossible. They might have followed “cookbook” style instructions from a web site or blog to install a particular program (such as Webmin, which is very popular with users such as myself that don’t care much for the Linux command line).
And to answer your question all we do is a update all for modules not install any modules that have not been installed already and yes we clear out all other repos to make sure there are not conflicts. I can look at just moving them and moving them back after the upgrade for the future as that did not occur to me but once again it is all in plain english to see and we spit out comment as the upgrade runs like “Replace repos with only FreePBX Distro since some people have added other repos which can break updates”
tonyclewis, what I was really asking was, do you by any chance set any modules to “enabled” that were previously present on the system but disabled? I know you didn’t install anything that wasn’t already installed.
Anyway, thank you for giving this some consideration. When a script is running sometimes things scroll by pretty quickly. Looking at the script, I now see where that’s in there, but didn’t really consciously notice it as it scrolled by (on the other hand, I DID look in that directory, so maybe I noticed it subconsciously or something). At the end I was a bit more concerned about those PHP warning messages and trying to figure out what might have caused them, so that was what pretty much occupied my attention for several minutes.
Thanks for the explanation, I now feel more reassured! And THANK YOU for helping keep our systems secure, THAT is very much appreciated. The thing about the disappearing repos was really one of only a VERY few things I’ve found in the FreePBX Distro that I would consider a negative — overall, installing and using the distro has been a pretty smooth experience, given that it’s relatively new and a few of the kinks are still being worked out. On the other hand, I wasn’t even aware until today that you guys had upgrade scripts, so that is good to know!
What about PHP version 5.1.6? Does it need upgrading?
And the FreePBX servers we have are exposed over internet via HTTPS (custom certificates) - but not via 443 port, but a random redirected port, is this HTTPS with SSL also a problem?
Ok so we just had a client in paid support inform us they thought they were hacked. We logged in and saw in the ps aux the following
root 12379 3.4 0.2 49960 10984 pts/0 S+ 13:25 0:00 /usr/bin/perl ./mycnf 174.142.177.241 10000 0
root 12380 3.2 0.2 49960 10984 pts/0 S+ 13:25 0:00 /usr/bin/perl ./mycnf 174.142.177.242 10000 0
root 12381 3.3 0.2 49960 10980 pts/0 S+ 13:25 0:00 /usr/bin/perl ./mycnf 174.142.177.243 10000 0
root 12382 3.4 0.2 49960 10984 pts/0 S+ 13:25 0:00 /usr/bin/perl ./mycnf 174.142.177.244 10000 0
There were about 2000 lines of that and the IP’s kept changing as the port scan was running. Load average was up to 150 in TOP.
The file this time was located in /usr/books/.n I tar’d up the files and if anyone wants them please PM me with your email and I will forward them to you.
The only ports open on the firewall for this customer was 80, 5060 and 10000-20000 for SIP. Everything else was closed down. I see nothing in the access files for apache on someone failing to log in and fail2ban did not detect any failed logins. This was not a FreePBX Distro but a different FreePBX based system. They were running an older apache and php and advised them to upgrade those packages but the customer decided to do a re install instead and than upgrade the packages.
If anyone else runs into this please be very careful killing the processes as each time we have done this they than DoS attack the IP that you killed the process from.
And here is another one from the same customer different box.
root 10209 2.0 0.0 1740 384 pts/0 S+ 15:20 0:00 ./exim new vuln
root 10210 2.0 0.0 1740 384 pts/0 R+ 15:20 0:00 ./exim new vuln
root 10211 1.8 0.0 1740 384 pts/0 R+ 15:20 0:00 ./exim new vuln
root 10212 2.0 0.0 1740 384 pts/0 R+ 15:20 0:00 ./exim new vuln
root 10213 2.0 0.0 1740 384 pts/0 S+ 15:20 0:00 ./exim new vuln
root 10214 1.8 0.0 1740 384 pts/0 R+ 15:20 0:00 ./exim new vuln
root 10215 0.2 0.1 7948 2060 ? Ss 15:20 0:00 sshd: root@pts/
root 10217 2.2 0.0 1740 384 pts/0 S+ 15:20 0:00 ./exim new vuln
root 10218 2.2 0.0 1740 384 pts/0 S+ 15:20 0:00 ./exim new vuln
root 10219 1.8 0.0 1740 384 pts/0 S+ 15:20 0:00 ./exim new vuln
root 10220 1.8 0.0 1740 384 pts/0 S+ 15:20 0:00 ./exim new vuln
root 10221 2.0 0.0 1740 384 pts/0 S+ 15:20 0:00 ./exim new vuln
root 10222 2.0 0.0 1740 384 pts/0 S+ 15:20 0:00 ./exim new vuln
root 10223 1.7 0.0 1740 384 pts/0 S+ 15:20 0:00 ./exim new vuln
root 10224 1.7 0.0 1740 384 pts/0 S+ 15:20 0:00 ./exim new vuln
root 10225 1.7 0.0 1740 384 pts/0 R+ 15:20 0:00 ./exim new vuln
root 10226 1.7 0.0 1740 384 pts/0 S+ 15:20 0:00 ./exim new vuln
root 10227 1.7 0.0 1740 384 pts/0 R+ 15:20 0:00 ./exim new vuln
root 10228 1.6 0.0 1740 384 pts/0 S+ 15:20 0:00 ./exim new vuln
root 10229 1.6 0.0 1740 384 pts/0 S+ 15:20 0:00 ./exim new vuln
root 10230 0.5 0.1 4752 1492 pts/2 Ss 15:20 0:00 -bash
root 10254 1.5 0.0 1740 384 pts/0 R+ 15:20 0:00 ./exim new vuln
root 10255 1.5 0.0 1740 384 pts/0 S+ 15:20 0:00 ./exim new vuln
root 10256 1.5 0.0 1740 384 pts/0 S+ 15:20 0:00 ./exim new vuln
root 10257 1.0 0.0 1740 384 pts/0 S+ 15:20 0:00 ./exim new vuln
root 10258 1.5 0.0 1740 384 pts/0 R+ 15:20 0:00 ./exim new vuln
root 10259 1.0 0.0 1740 384 pts/0 S+ 15:20 0:00 ./exim new vuln
root 10260 1.5 0.0 1740 384 pts/0 S+ 15:20 0:00 ./exim new vuln
root 10261 1.5 0.0 1740 384 pts/0 S+ 15:20 0:00 ./exim new vuln
root 10262 1.0 0.0 1740 384 pts/0 S+ 15:20 0:00 ./exim new vuln
root 10263 1.5 0.0 1740 384 pts/0 R+ 15:20 0:00 ./exim new vuln
root 10264 1.5 0.0 1740 384 pts/0 S+ 15:20 0:00 ./exim new vuln
root 10265 1.5 0.0 1740 384 pts/0 R+ 15:20 0:00 ./exim new vuln
root 10266 1.0 0.0 1740 384 pts/0 S+ 15:20 0:00 ./exim new vuln
root 10267 1.0 0.0 1740 384 pts/0 R+ 15:20 0:00 ./exim new vuln
root 10268 1.0 0.0 1740 384 pts/0 S+ 15:20 0:00 ./exim new vuln
root 10269 1.0 0.0 1740 384 pts/0 R+ 15:20 0:00 ./exim new vuln
root 10270 1.0 0.0 1740 384 pts/0 S+ 15:20 0:00 ./exim new vuln
root 10271 1.0 0.0 1740 384 pts/0 S+ 15:20 0:00 ./exim new vuln
root 10272 1.0 0.0 1740 384 pts/0 S+ 15:20 0:00 ./exim new vuln
root 10273 1.0 0.0 1740 384 pts/0 S+ 15:20 0:00 ./exim new vuln
root 10274 1.0 0.0 1740 384 pts/0 S+ 15:20 0:00 ./exim new vuln
root 11773 0.0 0.0 1748 412 pts/0 S+ 14:20 0:00 ./exim new vuln
root 11814 0.0 0.0 4564 1064 pts/0 S+ 14:20 0:00 sh -c perl x.pl
root 11815 0.0 2.6 38448 34564 pts/0 S+ 14:20 0:00 perl x.pl 202.1
root 12314 0.0 0.0 1748 412 pts/0 S+ 14:21 0:00 ./exim new vuln
root 12377 0.0 0.0 4564 1068 pts/0 S+ 14:21 0:00 sh -c perl x.pl
root 12378 0.0 1.8 28208 24328 pts/0 S+ 14:21 0:00 perl x.pl 202.1
nagios 18092 0.0 0.0 5080 944 ? Ss 12:05 0:00 /usr/local/nagi
root 18899 0.0 0.0 1748 412 pts/0 S+ 14:29 0:00 ./exim new vuln
root 18953 0.0 0.0 4564 1068 pts/0 S+ 14:29 0:00 sh -c perl x.pl
root 18954 0.0 8.1 110128 106244 pts/0 S+ 14:29 0:00 perl x.pl 202.1
root 23242 0.0 0.0 5032 1156 ? Ss 13:28 0:02 SCREEN
root 23243 0.0 0.1 4752 1528 pts/0 Ss 13:28 0:00 /bin/bash
root 23261 0.0 0.0 4756 860 pts/0 S+ 13:28 0:00 /bin/bash
Notice the screen session. If you go into screen you will see the following.
[+] 202.161.45.177
[+] 202.162.217.59
[+] 202.162.217.57
[+] 202.162.217.56
[+] 202.162.217.58
[+] 202.162.217.60
[+] 202.162.217.61
[+] 202.162.77.52
sh: fork: Cannot allocate memory
[+] 202.162.78.9
[+] 202.162.77.36
sh: fork: Cannot allocate memory
[+] 202.162.78.8
sh: fork: Cannot allocate memory
[+] 202.162.77.25
sh: fork: Cannot allocate memory
[+] 202.162.78.59
sh: fork: Cannot allocate memory
[+] 202.164.172.195
[+] 202.164.172.197
sh: fork: Cannot allocate memory
[+] 202.164.189.226
[+] 202.164.189.225
[+] 202.164.189.228
sh: error while loading shared libraries: libtermcap.so.2: failed to map segment from shared object: Cannot allocate memory
sh: fork: Cannot allocate memory
[+] 202.165.247.91
[+] 202.167.231.47
[+] 202.167.231.46
[+] 202.168.64.111
[+] 202.168.64.110
[+] 202.170.120.224
[+] 202.170.120.225
[+] 202.170.122.83
[+] 202.170.126.52
[+] 202.170.127.103
[+] 202.170.127.76
[+] 202.170.127.66
[+] 202.170.127.73
[+] 202.170.127.9
[+] 202.170.127.84
[+] 202.170.38.231
[+] 202.171.133.31
[+] 202.171.136.198
[+] 202.171.136.195
[+] 202.171.136.205
[+] 202.171.136.214
[+] 202.171.136.217
[+] 202.172.169.16
[+] 202.172.169.42
[+] 202.172.169.18
[+] 202.172.32.144
[+] 202.172.32.223
[+] 202.173.130.5
[+] 202.173.138.29
[+] 202.173.141.190
[+] 202.173.142.160
[+] 202.173.140.192
[+] 202.173.151.16
[+] 202.173.180.34
I tar’d up the files this time from /usr/books if anyone wants them.
That machine is totally compromised. It’s a goner.
They’re exploiting exim - the mail program. Check out this slashdot article: http://it.slashdot.org/story/10/12/1...it-In-the-Wild
Bite the bullet and take that machine off line immediately!
You can just shut down the outward facing eth0 or eth1 if you want to do a forensic examination the system while sitting in front of the machine.
The hacker is using a file called “x.pl” to do nasty things. Find it and delete it. That will clean up the stuff you can see.
If it were me, I’d format the disk and reinstall the most recent version of the OS.
Well that is obvious. I was just trying to pass on info since everyone was screaming they wanted more info. They were doing a complete rebuild.
I’ve tried to collect all of what we currently know about this vulnerability in one place.
Moderator note: No need to go off to other site, we will keep the information available here. Thanks.
Here’s a summary of what we currently know about this vulnerability:
Vulnerability: Any stock CentOS 5.x or 6.x system with Apache and PHP exposed to Internet access with no IP address restrictions (WhiteList).
How Do They Get In: Some (perhaps unknown) vulnerability in stock versions of Apache and PHP on CentOS systems allows the attacker to gain system access. We really don’t know any more than that at this juncture. But this does not appear to be a PHPmyAdmin exploit as that utility is locked down by secure htaccess password on some systems that have been compromised. Fail2Ban is not detecting hack attempts so it appears the attacker is walking right in with this exploit.
Privileges: Still unclear whether attacker is gaining root access or merely same access as enjoyed by Apache on the attacked system. To do what they’re doing would NOT require root privileges on your system. The attacker brings a customized version of WebMin with their own password.
What Happens Once They’re In: In a nutshell, your system is turned into a zombie. Using perl and WebMin (their own version), they can interconnect your server into a worldwide network of machines used to launch denial-of-service and other malicious attacks against other systems on the Internet.
How Do I Know If My Machine Has Been Compromised? Examine some of the previous comments in this thread. Run ps awx on your server and look for long lists of processes running perl scripts. Look in the /usr directory for a directory called game, games, books, etc. Inside those directories, run ls -all which will show hidden files beginning with a period. There will be a directory called .n or .s or something similar. Look in /etc/cron.daily. There will be a new script as outlined in this thread. NOTE: The zombie software is old and signatures already exist in anti-virus programs. The exploit to gain access may be entirely new.
What Should Be in /usr? On a stock FreePBX Distro system, you should see the following directories:
bin etc games include java kerberos lib libexec local sbin share src tmp X11R6
The games directory will be empty when you ls -all
What Should Be in /etc/cron.daily? On a stock FreePBX Distro system, you should see the following files:
0anacron 0logwatch cups logrotate makewhatis.cron mlocate.cron prelink rpm tmpwatch
How to Fix It: If your system has been compromised, reformat the disk and reinstall. If they haven’t gotten in or if you’ve started over, (1) immediately turn off Internet access to web services on your servers. You can implement a whitelist of safe IP addresses for web access using IPtables or your hardware-based firewall. (2) Upgrade Apache to latest version and PHP to latest 5.2 or 5.3 version.
NOTE: Just because your system has not yet been compromised does NOT mean you are safe. Your system still needs to be secured. Turn OFF Web Access Now!
[B]For Standard PIAF servers and Proxmox PIAF installs with IPtables:[/B]
[B]How to Temporarily Disable Web Access[/B]: Log in as root. Issue command: [B]iptables -D INPUT -p tcp -m tcp --dport 80 -j ACCEPT[/B]
[B]How to Temporarily Enable Web Access[/B]: Log in as root. Issue command: [B]iptables -A INPUT -p tcp -m tcp --dport 80 -j ACCEPT[/B]
[B]How to Temporarily Enable Web Access for Specific IP Address [/B]: [B]iptables -A INPUT -p tcp -m tcp -s 123.45.67.8 --dport 80 -j ACCEPT[/B]
To start Im very new to linux and asterix and FreePBX, I downloaded the disro and installed on a old PC, bought a linksys PAP2 ATA and a Linksys SPA3000. Followed how-tos and easly got 2 EXTNs on the PAP2, 1 on my laptop using a softphone, setup the SPA3000 to make and recive PSTN call, and a trunk form Sipgate, All working great.
After following the instruction above I can no longer login to “Free BPX Administrator” the system is running and calls are routing.
Can anyone help me get back into the Web admin page, it dose popup the login dialoge but then sits on a blank page.!!? grr
Hi scottmoss. Perhaps the wrong place to post this, but I’ll respond so some of the developers will see this. Before 2.9 you could edit /etc/amportal.conf The parameter authtype could be changed from =database to =none and after an amportal restart you could get into the GUI with no password and reset you password. Then set it back to database and another amportal restart you you were back in business. Now amportal.conf says don’t edit it. So I don’t know where to change this, or if it is still possible to gain access if you have ssh or console access.
Any word from the higher ups? Thanks
Can you install a wireshark in front of a reinstalled server to see the attack attempt? [might be your 5 grand answer]
Is it wise to schedule a daily yum -y update? [we need automatic updates]
How about install a FreePBX onto a public IP with no firewall. First, package FreePBX into an OpenVZ container ( wink, wink ). Then host it on a $10 VPS. Login every day to take a system snapshot (recursive file listing, hidden files, file sizes, ps auxw) and cron an email to yourselves which shows you diff from the day before. [maintain a honeypot for us]