FreePBX Hacker, help please!

Hi everyone

I have a very big problem because recently my freePBX has been hacked and somebody has been changing my FreePBX username and password and when I try to change it back from the asterisk databse (AMPUSERS) from the commandline it doesn’t work as this hacker has done some evil things on my system to restrict me changing the usernam and password that he has created on my system.

More strangely, he has been doing this again and again as I have tried to reinstall the whole system for a few times so far, unfortunately he manages to change it back to the same username (mgknight) in am a matter of a few hours and locks me out from accessing the freepbx admin portal.

Please someone help me as this is a security matter and I have already tried the security checker in this link http://wiki.freepbx.org/display/L1/FreePBX+Security+Scan and also I have updated all the modulels but it is still the same for this hacker as he/she can easily change the admin username and password and also makes the calls on my system not going through.

I am left with a locked system and I seriously need your help.

Thanks.

Probably you have yet to update the OS level of Freepbx. I did encounter the same issue like you. After went through the steps and I thought that it is. Hell no! I got hit again and I went to irc and someone suggested to upgrade the freepbx version to latest one. yum update will not do anything and module updates are not enough either…google or look at freepbx side to look for upgrading freepbx distro

Update your machine COMPLETELY behind a firewall including restoring the Backup and updating any broken modules BEFORE you put it back out “In The Wild” otherwise you are just putting the vulnerable box back where the Hacker expects it to be with the vulnerabilities still open for exploitation.

Updating completely means both a “yum update -y” from the cli (OS Updates) and completely updating all the FreePBX modules through Admin->Module Admin - Select Basic and Extended -> Check Online -> Click Download All and Upgrade All and then Process and Apply.

Greg

Thank you both of you very much for your quick reply.

I have done almost done everything you people have suggested above, except for the distro updating which I find a bit difficult to do because I am not quite familiar with that, however I have now changed all the passwords on my box and hopefully they can’t access my box again.

Please note that I always appreciate your useful tips on resolving this issue because this has driven to the point that I shall never use freepbx again if this happens again because it is so easy for the hackers to control over it nowadays.

Thank you.

I am sorry to hear that. Of note we did announce the security release 19 days ago, this was 2-3 days before the “mgknight” admin attack was developed and exposed into the wild. This combined with the recent bash exploits (which are not from FreePBX and were announced 24 days ago) can cause issues for anyone.

Furthermore we learned that once the hacker has gotten into the system he is uploading custom index.php files all over the web directory. This is something we have tried to catch but unfortunately you can’t expect us to know every routine the hacker runs. This means once you get hacked the best course of action is a complete reinstall because the hacker could have laid down more “backdoors” than we even know about.

That said. moving forward we have removed the “ARI” module completely to prevent this issue from happening again.

The other key element you need to ask yourself is why is your machine exposed to the internet? Is there a reason you can’t put it behind a VPN or similarly firewall the machine so only you and your provider can get in?

1 Like

Wow Andrew, that is the question I keep asking myself over and over again - most people in the forums get REALLY grumpy when you ask them why they are exposing their boxes saying the security on them should be good enough that posting them in the wild shouldn’t be a problem - but the potential for fraud is HUGE!

How hard is it to first connect to a VPN before you do maintenance?

We have almost 100 Asterisk boxes in service now, and not a single one of them is in the wild. It’s not that I think the security of FreePBX is lacking, it’s just that I see no benefit for exposing a box - maybe I am weird, we have been doing Asterisk installs for over 10 years now and the only box we have ever had hacked was when a CUSTOMER moved it to a public IP.

It’s just not worth it.

Greg

2 Likes

Here is why I keep all of my machines behind a VPN/Firewall as well. FreePBX itself has been around since about 2004. None of the developers that first released the project in 2004 are involved any longer. There are over 170,000 lines of code ( https://www.openhub.net/p/freepbx/analyses/latest/languages_summary ) the project itself has had over 70 contributors, ranging from people like “David Herselman” who changed 41 lines of code back in May to someone like myself with over 335,000 lines of code changed in FreePBX. At some point you have to sit here and think about how much of FreePBX is what we call internally “old code”. It’s code that works right, but was designed when PHP was version 4 back in 2004. Over time we do go through much of the code, rip it out and rewrite it in new and better ways but we are always fighting the battle of how much do we change vs how much time do we want to waste. It’s a daunting task.

The ARI vulnerability was an example of an area of code that no one has touched since the originally author submitted ARI to FreePBX back in 2006. That means the vulnerability has been around since 2006 and no one (not even hackers) discovered it until 2014. Now you may be thinking “that is outlandish, I can’t believe you’d let that sit in there for so long”. But that is my point, there are over 170,000 lines of code. We’ve had people email us telling us that they have looked through all the lines of code in FreePBX looking for security weaknesses only for them to come up empty handed or for them to find trivial things. These same people never found the ARI vulnerability and the hacking community itself never found the vulnerability until 2014. Not even they have the time and energy to go through hundreds of thousands of lines of code.

Anyways why am I saying this? I am a developer. Do I not trust in my own project? I trust in FreePBX 100%. However there are a couple of things working against FreePBX.

  1. It’s open source. The code is there for anyone to look at. It’s far easier for someone to find vulnerabilities in an open source project than it is in a closed source one. This is and will always be true. It’s easy to find glitches when you can see how things work.
  2. It’s PHP and while PHP has gotten better throughout the years (by disabling register_globals for example). It’s still PHP, a widely known language. Heck the exploits for the CVE are all written in PHP.
  3. FreePBX was never designed to be exposed to the open internet. From the start of the project it was designed to work internally. This is a different practice then say Drupal or Wordpress (and might I mention Drupal just patched an SQL injection vulnerability two weeks ago).

Back to the topic, I trust FreePBX but I trust it more knowing it’s behind a firewall and the only way I can access it is through openvpn.

In closing, did you know that certain USB Pen drives are now hackable as well?

http://www.forbes.com/sites/gordonkelly/2014/08/01/usb-security/

And a favorite “hacking” TED talk. Where he doesn’t only hack desktop computers he hacks everything WITH a computer.

Nothing is safe. I don’t even trust my own credit cards.

You need to update your freepbx…Changing password will not solve. I went through exactly your path…Take a look at how to upgrade freepbx distro. Just follow the steps

Hey Greg, lttn chat

Anyway, I was one of the largest advocates of VPN in day. But with today’s SSL boxes you don’t even need to do that. Sure they can act as a full SSL VPN put most have http/https/ssh proxies. You just login to your VPN site from your browser and boom you are in to any web app you need.

This isn’t just a FreePBX problem, cable companies get hammered on their equipment by their own customers. I know TWC has move to an out of band management network. Still doesn’t solve the millions of CPE devices.

Anything with an interface open to the Internet is suspect today.

So yes, VPN’s are a must if you have any kind of assets on the web. If you are running one PBX on some cheeze ball hosting platform install a software firewall, only open SIP/RTP to the people that need it and change the port of the SSH server to something uncommon. Set the SSHD options to not allow root login and very short timeout. Fail2ban will kill you after 3 tries. Don’t forget to create an admin account on the machine then sudo up to root.

Now for those who complain they can’t get to web interface do a Google search on “Putty SSH SOCKS proxy” and you will be shocked at how simple it is. One more tip. Firefox is the only modern browser that has it’s own IP interface. You can add your Putty proxy without screwing up access for whole WIN box.

Hey Scott - Yeah, I haven’t been around here much lately - too busy. Will I see you this week at Astricon?

Greg

Hi All

I am glad that quite a few people have tried to help to overwhelm this issue, however I still think that it is not to do with my server box because there is also a2billing installed on the same server and the hacker has not been able to touch my a2billing at all and this makes me think that this little hacker has possibly got access to the freepbx update files and things and he can get access to the freepbx web interface or the asterisk database and he can do what he wants to do but what has been upsetting to me the most is that he always uses the same username and I am not able to change it even from the command line by going into the MySQL database.\

I hope you experts can tackle this hacker because it is clear that it is to do with FreePBX and not anything else.

Please excuse me if I am wrong because I am only speaking of my limited knowledge.

Thanks.

You have compromised files on your system still. Our security checker would tell you that. However it is not full proof. We can’t beging to guess what the hacker added

You can blame freepbx all day long but the fact remains if you kept your system up to date you would not be running into these issues.

Comparing a2billing to freepb is like comparing apples and oranges. Different products.

You will most likely have to reinstall at this point. Strangely you keep acting as though the freepbx team has not solved this issue. But we have. Two weeks ago.

Please go and download the checker and rerun it and post the output here for all of us to see

1 Like

Once the extension_custom file had been modified, your a2b also will not work. and your callback will not work after a while if you left with it.

This was what I did, ran security checker, insert back for a2b in extension_config,manager_custom, change the ampusers in asterisk db, update update update freepbx, I spent 3hrs to upgrade to the latest version from where I was.
Then I spent another 1 day to figure out why my fail2ban is not banning.

I believe fail2ban was modified by the exploited as well. Notice my fail2ban was not banning sip wrong password attempts.

If I had a compromised box I would go for full reinstall. Otherwise you can never be sure you have discovered all backdoors left by the hacker. This is why I do once a month a full backup of my asterisk using fsarchiver. This gives me an easy option of restoring backup I know is not compromised and then do a full update.
All what was said here about VPN I fully support, no reason to have http accessible outside the network, if it takes 15 seconds to login to a VPN.

1 Like

Hi tm1000

I am not sure why someone from FreePBX get so upset by my last post as I have only been trying to tell the truth? and I don’t know why some very technical people can’t read posts properly since I have clearly mentioned it in my first post that I have tried annd reinstalled the system a few times and by this I mean reinstalling from a system image to 7 months ago when my system was all fine, however I am going to explain how I have done it below:

1/ reinstalled the whole server to a backup image that was taken 7 months ago.
2/ yum update -y.
3/ update all the freepbx modules.
4/ reboot.
5/ Mgknight is in again in a matter of three hours.

I bet if you can tell me how I to remove this Mgknight user from my ampusers, because I only believe if you are able to resolve this issues after you have been able to explain above as your security checker also not helps as I have also downloaded it and done all the steps above again.

What I mean by a2billing no been touched is that if this hacker has comprised my system why is he not stealing my a2billing accounts so that he can use my customers’ credits to make free phone calls? why is he a so free hacker?

The ANSWER is CLEAR that this virus only affects the freepbx apps and it comes from the freepbx side and it is nothing to do with hackers compromising, but I am not blaming you personally but I can’t still accept what you are saying.

Thanks.

You seem to be looking for a silver bullet: “just do this and it’s fixed” response. Ain’t going to happen. The hacker is in your system, where and what has he compromised: “who knows?”. The only solution is to re-install. You’re also thinkig that it is a FreePBX exploit that he is doing. Maybe it’s CentOS hack or whatever other program or service is running. You just don’t know what he has done once he’s in the system.

Not sure why a hacker would care about a2billing accounts for credits when they have access to the entire phone system. Your statement doesn’t disprove the hack is coming in through a2billing. Of course, it also doesn’t prove it is an a2billing problem.

There are lots of vectors for attacks on the system. It very well may be freepbx, but it also may be something else.

There is a language barrier here. I am not upset. If you are restoring from a backup that restores system files then you are restoring the compromises. There are plenty of people who have gotten hacked and are now fine (meaning they have not gotten hacked again) because the successfully followed the steps people have provided in this topic. Why don’t you just restore your system from a backup and update then zip up the html directory and send/post it online for all of us to inspect to make sure you are doing what you say you are doing.

Furthermore if you really think this is another security vulnerability then next time you get hacked email [email protected] instead with details of how to get into your system so that we can inspect it ourselves.

One of my Trunk providers alerted me about the compromise, I then came here and found the very document that referenced the upgrade and the steps, I updated my server within about 4 hours of the notification.

I try to do updates as soon as I see them, I do get emails from my server to notify me of module upgrades, I also bought sys admin pro which can handle server upgrades also. It did not do the major release and I did that by hand, but it was very simple to do.

My thanks and respect to the devs for the continued work…