I got a notification today from my SIP provider that a known fraudulent number had been used on my account and they shut down my SIP. I’ve gotten this before and typically I go in and change all my passwords and life is good, not this time.
When I fired up my FreePBX site I got a notice that my module signing was invalid and Ajax.php could not be found. I went into the HTML folder and saw that someone renamed the Ajax.php to a random name and in it’s place was something that said I was hacked. Fine, I renamed the file back and followed the instructions on http://wiki.freepbx.org/display/F2/Module+Signing to make sure everything was correct. FreePBX is back up and running now.
The issue I’m having is when I SSH’d into my box to change my passwords I noticed a new account called ‘dude’, I tried to remove the user but couldn’t because it says it was used by process 1 (/sbin/init). I changed the password and I’m unsure what to do to my system so that ‘dude’ is not running /sbin/init any longer.
I ran a find / -user dude and saw tons of files “owned” by him but when I check the files they say they are owned by root, making me wonder if ‘dude’ is some kind of alias for ‘root’.
the best course of action from here is to rebuild as soon as possible
rebuild and restore with a known good backup prior to the incident
insure you are fully up to date; reexamine security and make any changes deemed necessary-- ie if the admin interface doesnt need to be exposed to the public … dont
im assuming you had not patched for the latest CVE released last week ?
I have not patched yet, it was on my list of things to do this week. When you say rebuild what do you mean? Rebuild the operating system? It’s hosted and I haven’t a clue how I would go about doing that.
you say It’s hosted, but have you got a chance for physical (i.e. keyboard & screen) access to the server? Even via ‘server built in’ remote management like ILO ? So that you won’t be running any infected binary voluntarily.
Are you using the official Distro or is it your own spin, with FreePBX on top ?
(and if it’s your own spin, what OS was your base ?)
It’s hosted on FreePBXHosting, nothing special about the distro it’s all stock (as far as I know). I don’t have any access to the physical machine, only SSH
Hmm VPS or Dedicated?
But either way, cyberlynk may be better suited to deal with it, (I think) including (if you get them approximate timestamp of when the breach happened) a re-spin of your system and restoring config from a nearest clean backup of config… vps or not, you probably WANT cybelynk on board with this anyway - cause the infection may find ways to spread beyond what you are seeing.
Also since it’s not really possible to run a system with substituted init without rebooting/kexecing into it, it’s hard to believe they wouldn’ have noticed this (a restart/reboot) happening. Get them on board.
I just finished updating all my FreePBX to the latest version 15 and it’s all working fine. I have another PBX on the same hosting service and it got hacked identically to this one (except not the SIP hack in addition). If I do a full back up of my FreePBX using the provided utilities and have Cyberlynk respin up my PBX do you think that restore is safe?
Ok, thank you for the advice, I’ll do full backups of all my PBX stuff and ask them to respin the VPS. Any advice on additional steps with my backup to make sure that I go from a fresh OS reload (with FreePBX installed by Cyberlynk) to being back to regular operations? I’ve looked over my last backup and it seems complete but since this is a one-way street I want to make sure I get absolutely everything that I will need to get back to where I am now.
Provided they respin the image so that all binaries get replaced by proper, verified copies (all means all - bootloader, kernel, libraries, program binaries) and there is no residues of the old binary files, and if you verify that the config files [the freepbx config dump, and the *_custom.conf files] are clean, there is only one way to find out: install FreePBX distro on a machine in your lab and restore the config to it (keeping the network access shut from outside world - or the rest of your company internal network as well - you wouldn’t care about the trunks not going up there anyway - e.g. behind another router with firewall - which will tell help you to find out if it’s not trying to do something weird after backup restoration… just look at what your respun box is trying to access) I never had to do it that way ( so far, beginners^Wluck ) so I don’t have a first-hand experience.
I would highly suggest backing up your FreePBX and just spinning up a new instance. I am sure Cyberlink would do that for you and keep both up for the time needed.
Do a Zend reset and move your deployment ID to the new system
I would lake to take this chance to remind everyone there is another “boutique” corporate class certified FreePBX hosting provider, our company. Micro Advantage and we provide fully managed systems and have fully certified FreePBX/Sangoma technicians on staff that can not only provide a fantastic hosting environment but also take care of FreePBX for you.