FreePBX server compromised


(Ted) #1

Hello,

It looks like one of the test FreePBX servers that we are using has somehow been compromised.
The server is hosted by a cloud server provider and we have received a message from them that, they have received a message from Tepucom Abuse Dept. that our FreePBX server is sending our spam invites. This is a piece of what they wrote:

Tepucom Abuse Dept:

IP-OF-OUR-SERVER/32 (root IP: IP-OF-OUR-SERVER) (PTR: IP-OF-OUR-SERVER.our-vps-provider.com.) was added to the blackholes.tepucom.nl RBLDNS for the following reason:
“Caught scanning for web/mail exploits / compromised hosts”

A T T E N T I O N ! T H I S I S A C O M P R O M I S E D H O S T !

Then they attached a piece of what they have been receiving from our server:

IP-OF-OUR-SERVER tpc-035.mach3builders.nl 20200906/22:46:30 22:44:07.952660 rule 0/0(match): block in on vmx0: IP-OF-OUR-SERVER.5121 > 91.190.98.8.5060: SIP: INVITE sip:100@91.190.98.8 SIP/2.0
IP-OF-OUR-SERVER tpc-035.mach3builders.nl 20200906/22:46:30 22:44:07.953053 rule 0/0(match): block in on vmx0: IP-OF-OUR-SERVER.5121 > 91.190.98.76.5060: SIP: INVITE sip:100@91.190.98.76 SIP/2.0
IP-OF-OUR-SERVER tpc-035.mach3builders.nl 20200906/22:46:33 22:44:07.953314 rule 0/0(match): block in on vmx0: IP-OF-OUR-SERVER.5121 > 91.190.98.79.5060: SIP: INVITE sip:100@91.190.98.79 SIP/2.0

It looks like the invites are somehow being sent from our server to scan different IPs for open SIP ports, is that right? From what I understand usually this process is in reverse, directed to our FreePBX server. Why is it being originated from our server?

It also looks like there is extension 100 involved. We do not have that extension, nor do we have guest/unanimous calls enabled.

Can someone help us figure out what is going on here?


#2

I’m no expert, but I would assume they are using your server to scan other hosts for SIP vulnerabilities. It may be they are using a tool hosted on your server to scan for other vulnerabilities as well.

Since is a test box for you, I would take it off the network so you can stop the scanning and see if you can figure out how they got access to the system (i.e. via ssh, freepbx web gui, etc.) if you have remote console access. If this test box shares credentials with any other box of yours or it has passwordless access to any other servers, I would check if those have been hacked as well.


(Nate) #3

I’ve seen a few reports of hosts compromised recently and most of them seem to be on older FreePBX versions with the firewall disabled/bypassed and un-patched module vulnerabilities. Found a cron job in root’s crontab that was re-downloading and executing a malicious script that creates a Linux user ‘supports’. Then there’s the usual bunch of malicious PHP scripts scattered throughout the web root, mostly ‘config.php’ and ‘ajax.php’. More than likely there’s something on the system that’s being executed by this ‘supports’ user which is created as a root equivalent (ID=0) that is originating these outgoing attacks/scans. Logging doesn’t seem to be affected, so you should see the user being created in /var/log/secure as well as potentially any shell logins to the system, but I don’t know that I would trust it.


(Ted) #4

The server’s CPU is currently being loaded at up to 100% and we’re trying to figure out what process is causing this. Below are the currently running processes of our FreePBX instance. Does any of these look suspicious or irregular? Thanks for all your inputs!

[root@ourpbxserver ~]# ps -e
PID TTY TIME CMD
1 ? 01:40:05 systemd
2 ? 00:00:03 kthreadd
3 ? 00:43:10 ksoftirqd/0
5 ? 00:00:00 kworker/0:0H
7 ? 00:00:00 migration/0
8 ? 00:00:00 rcu_bh
9 ? 01:33:29 rcu_sched
10 ? 00:01:24 watchdog/0
12 ? 00:00:00 kdevtmpfs
13 ? 00:00:00 netns
14 ? 00:00:55 khungtaskd
15 ? 00:00:00 writeback
16 ? 00:00:00 kintegrityd
17 ? 00:00:00 bioset
18 ? 00:00:00 kblockd
19 ? 00:00:00 md
25 ? 00:08:15 kswapd0
26 ? 00:00:00 ksmd
27 ? 00:02:33 khugepaged
28 ? 00:00:00 crypto
36 ? 00:00:00 kthrotld
38 ? 00:00:00 kmpath_rdacd
39 ? 00:00:00 kpsmoused
40 ? 00:00:00 ipv6_addrconf
59 ? 00:00:00 deferwq
91 ? 00:03:16 kauditd
273 ? 00:00:01 ata_sff
280 ? 00:00:00 scsi_eh_0
281 ? 00:00:00 scsi_tmf_0
282 ? 00:00:00 scsi_eh_1
283 ? 00:00:00 scsi_tmf_1
286 ? 00:00:00 ttm_swap
316 ? 00:10:04 kworker/0:1H
357 ? 00:00:00 kdmflush
358 ? 00:00:00 bioset
369 ? 00:00:00 kdmflush
370 ? 00:00:00 bioset
383 ? 00:00:00 bioset
384 ? 00:00:00 xfsalloc
385 ? 00:00:00 xfs_mru_cache
386 ? 00:00:00 xfs-buf/dm-0
387 ? 00:00:00 xfs-data/dm-0
388 ? 00:00:00 xfs-conv/dm-0
389 ? 00:00:00 xfs-cil/dm-0
390 ? 00:00:00 xfs-reclaim/dm-
391 ? 00:00:00 xfs-log/dm-0
392 ? 00:00:00 xfs-eofblocks/d
393 ? 01:11:32 xfsaild/dm-0
467 ? 12:37:44 systemd-journal
485 ? 00:00:00 lvmetad
491 ? 00:00:00 systemd-udevd
510 ? 00:01:21 hwrng
571 ? 00:00:00 edac-poller
580 ? 00:00:00 jbd2/vda1-8
582 ? 00:00:00 ext4-rsv-conver
600 ? 00:18:32 auditd
626 ? 11:16:39 rsyslogd
628 ? 00:39:08 systemd-logind
629 ? 00:27:13 polkitd
636 ? 01:06:22 dbus-daemon
646 ? 00:00:17 incrond
649 ? 00:00:36 chronyd
661 ? 00:07:18 avahi-daemon
672 ? 00:00:00 avahi-daemon
687 ? 00:00:00 cfg80211
867 ? 00:00:01 dhclient
927 ? 00:00:01 dnsmasq
928 ? 00:18:38 sshd
930 ? 00:36:41 tuned
935 ? 00:00:04 vsftpd
937 ? 00:23:58 httpd
947 ? 00:00:00 atd
949 ? 00:08:58 crond
962 tty1 00:00:00 agetty
1009 ? 00:00:00 mysqld_safe
1014 ? 00:03:51 letschat
1020 ? 14:55:04 mongod
1056 ? 00:00:00 dio/dm-0
1372 ? 00:01:35 master
1390 ? 00:00:21 qmgr
1960 ? 00:00:01 httpd
2091 ? 00:00:00 safe_asterisk
2094 ? 2-08:27:01 asterisk
2706 ? 03:08:03 PM2 v2.10.6: Go
3904 ? 00:00:00 pnp_server
7148 ? 00:00:02 httpd
7491 ? 00:00:01 httpd
8226 ? 00:00:01 kworker/0:3
9790 ? 00:00:00 sshd
9834 pts/0 00:00:00 bash
10352 ? 00:00:01 httpd
12994 ? 00:00:08 httpd
14090 ? 00:00:00 kworker/0:0
14894 ? 00:00:02 httpd
14913 ? 00:00:02 httpd
14919 ? 00:00:02 httpd
16566 ? 00:00:00 pickup
17569 ? 00:00:02 python
17620 ? 00:00:02 python
17634 ? 00:00:02 python
17668 ? 00:00:01 python
17692 ? 00:00:00 crond
17700 ? 00:00:00 sh
17758 ? 00:00:00 kworker/0:1
17795 ? 00:00:00 php
17858 ? 00:00:00 sshd
17864 ? 00:00:00 sshd
17906 ? 00:00:00 sh
17907 ? 00:00:00 sleep
17908 ? 00:00:00 php
17910 ? 00:00:00 sed
17911 pts/0 00:00:00 ps
18867 ? 00:00:00 kworker/u2:0
20576 ? 00:00:01 node /var/www/h
20929 ? 00:00:02 httpd
21373 ? 00:28:50 sh
21374 ? 00:00:00 sh
21378 ? 00:13:57 start
22079 ? 00:01:10 php
22614 ? 00:03:31 fail2ban-server
22630 ? 00:00:37 gam_server
23268 ? 00:00:04 php
24492 ? 00:00:57 rpcbind
24849 ? 09:44:16 mysqld
24898 ? 00:11:31 php
27126 ? 00:00:00 xinetd
28415 ? 00:00:00 httpd
30618 ? 00:00:00 kworker/u2:1


(Ted) #5

We just ran a TOP commant to see what’s taking up most of the CPU resources and this is what we are getting. There are a ton of these python processes that are taking up a lot of CPU. We checked other FreePBX servers, there is none of these. What could these be?

511 root 20 0 212364 10484 4016 R 1.5 0.6 0:00.06 python
479 root 20 0 212364 10484 4012 R 1.2 0.6 0:00.05 python
482 root 20 0 212364 10484 4016 S 1.2 0.6 0:00.05 python
486 root 20 0 212364 10480 4012 R 1.2 0.6 0:00.06 python
487 root 20 0 212364 10480 4012 R 1.2 0.6 0:00.05 python
488 root 20 0 212364 10480 4012 S 1.2 0.6 0:00.05 python
492 root 20 0 212364 10484 4016 R 1.2 0.6 0:00.06 python
493 root 20 0 212364 10480 4012 S 1.2 0.6 0:00.05 python
494 root 20 0 212364 10484 4012 R 1.2 0.6 0:00.06 python
498 root 20 0 212364 10480 4012 S 1.2 0.6 0:00.05 python
499 root 20 0 212364 10480 4012 S 1.2 0.6 0:00.05 python
501 root 20 0 212364 10484 4016 R 1.2 0.6 0:00.05 python
502 root 20 0 212364 10484 4016 R 1.2 0.6 0:00.05 python
504 root 20 0 212364 10480 4012 R 1.2 0.6 0:00.05 python
506 root 20 0 212364 10480 4012 R 1.2 0.6 0:00.05 python
507 root 20 0 212364 10480 4012 S 1.2 0.6 0:00.05 python
509 root 20 0 212364 10480 4012 R 1.2 0.6 0:00.05 python
512 root 20 0 212364 10480 4012 R 1.2 0.6 0:00.05 python
513 root 20 0 212364 10484 4012 R 1.2 0.6 0:00.05 python
514 root 20 0 212364 10480 4012 S 1.2 0.6 0:00.05 python
515 root 20 0 212364 10480 4012 R 1.2 0.6 0:00.05 python
516 root 20 0 212364 10484 4016 R 1.2 0.6 0:00.05 python
529 root 20 0 212364 10484 4012 R 1.2 0.6 0:00.04 python
531 root 20 0 212364 10480 4012 R 1.2 0.6 0:00.04 python
536 root 20 0 199456 9776 3820 R 1.2 0.5 0:00.04 python
541 root 20 0 193260 9504 3620 R 1.2 0.5 0:00.04 python
543 root 20 0 208168 10280 3884 R 1.2 0.5 0:00.04 python
557 root 20 0 193264 9440 3620 R 1.2 0.5 0:00.04 python
480 root 20 0 212364 10480 4012 R 0.9 0.6 0:00.05 python
481 root 20 0 212364 10480 4012 S 0.9 0.6 0:00.05 python
484 root 20 0 212364 10480 4012 S 0.9 0.6 0:00.05 python
489 root 20 0 212364 10484 4016 R 0.9 0.6 0:00.05 python
490 root 20 0 212364 10484 4016 S 0.9 0.6 0:00.05 python
496 root 20 0 212364 10480 4012 S 0.9 0.6 0:00.05 python
497 root 20 0 212364 10480 4012 R 0.9 0.6 0:00.04 python
500 root 20 0 212364 10484 4012 R 0.9 0.6 0:00.04 python
503 root 20 0 212364 10480 4012 S 0.9 0.6 0:00.04 python
505 root 20 0 212364 10484 4016 S 0.9 0.6 0:00.05 python
508 root 20 0 212364 10484 4012 S 0.9 0.6 0:00.04 python
517 root 20 0 212364 10484 4016 R 0.9 0.6 0:00.04 python
519 asterisk 20 0 336952 10004 6728 R 0.9 0.5 0:00.03 php
521 root 20 0 199632 10004 3844 R 0.9 0.5 0:00.03 python
522 root 20 0 199456 9772 3820 R 0.9 0.5 0:00.03 python


(Nate) #6

Those python processes are probably a script that is scanning/attacking other people’s servers.


(Ted) #7

Oh wow, this is what they are looks like. How would they get originated in our server? Does that mean our server indeed has been compromised and we should reinstall it completely? And how can I delete these scripts?

[root@ourpbx ~]# ps x | grep python
930 ? Ssl 36:41 /usr/bin/python -Es /usr/sbin/tuned -l -P
7837 ? R 0:00 python svmap.py 153.17.0.0/17 -A -t0 -m INVITE -v -p 5060
7843 ? S 0:00 python svmap.py 171.246.252.0/23 -A -t0 -m INVITE -v -p 5060
7845 ? R 0:00 python svmap.py 57.142.128.0/18 -A -t0 -m INVITE -v -p 5060
7846 ? R 0:00 python svmap.py 44.140.0.0/17 -A -t0 -m INVITE -v -p 5060
7853 ? R 0:00 python svmap.py 223.241.128.0/18 -A -t0 -m INVITE -v -p 5060
7867 ? R 0:00 python svmap.py 101.166.128.0/18 -A -t0 -m INVITE -v -p 5060
7894 ? R 0:00 python svmap.py 54.241.0.0/17 -A -t0 -m INVITE -v -p 5060
7898 ? R 0:00 python svmap.py 216.95.128.0/18 -A -t0 -m INVITE -v -p 5060
7902 ? R 0:00 python svmap.py 126.101.128.0/18 -A -t0 -m INVITE -v -p 5060
7906 ? R 0:00 python svmap.py 219.234.128.0/18 -A -t0 -m INVITE -v -p 5060
7910 ? R 0:00 python svmap.py 128.112.0.0/17 -A -t0 -m INVITE -v -p 5060
7919 ? R 0:00 python svmap.py 9.101.128.0/18 -A -t0 -m INVITE -v -p 5060
7924 ? R 0:00 python svmap.py 40.138.128.0/18 -A -t0 -m INVITE -v -p 5060
7926 ? R 0:00 python svmap.py 149.93.128.0/18 -A -t0 -m INVITE -v -p 5060
7934 ? R 0:00 python svmap.py 212.236.128.0/18 -A -t0 -m INVITE -v -p 5060
7936 ? R 0:00 python svmap.py 28.34.0.0/17 -A -t0 -m INVITE -v -p 5060
7938 ? R 0:00 python svmap.py 111.254.0.0/17 -A -t0 -m INVITE -v -p 5060
8121 pts/0 R+ 0:00 grep --color=auto python
22614 ? Sl 3:33 /usr/bin/python /usr/bin/fail2ban-server -b -s /var/run/fail2ban/fail2ban.sock -p /var/run/fail2ban/fail2ban.pid -x
[root@ourpbx ~]#


(Ted) #8

This is what has been running on our server.

svmap

this is a sip scanner. When launched against
ranges of ip address space, it will identify any SIP servers 
which it finds on the way. Also has the option to scan hosts 
on ranges of ports.

#9

Sorry I can’t be of help, but this is fascinating in another way – could you PM your public IP address to @lgaetz to see whether your scanner made it into the APIBAN blacklist? I’m wondering if you hit one of their honeypots.


#10

As this is a test box, I would just take it off the network and make another. Ideally once off the network, you could try to determine how the server was hacked - if you have the experience as it not necessarily easy.
But if one was hacked, I would check the others to see if they were hacked as well. Look for the same weird process, same files on them, odd ssh logins, etc.


(Ted) #11

Where shoud I go to at least stop the current script from running?


#12

That is hard to tell as they could do a number of things to keep it running, obviously you could kill all the running processes.
You could try a reboot to see if they didn’t make it able to auto-restart.
What about changing your firewall to block outbound SIP on that box?


#13

This is a waste of time, because the attacker has root access to the system. You could kill all the python jobs, but he’s probably got a cron job or similar to restart them. If not, he will log in and restart them manually. When something like this happens, the only solution is to reinstall from scratch, or from a backup or snapshot that you are certain was taken before the breach, and secure the system properly.


(Ted) #14

Thanks! How do I block SIP on outbound?


(Ted) #15

Do you think they are mostly logging in through the web or ssh?
Also where can I check login log for the web and ssh as well?


#16

This is the primary software they have deployed:


and I suppose you could remove it, after setting your firewall to allow only authorized IP addresses. However, it’s likely that they also installed a back door, which may be very difficult to find.

As this is not a production system, there is no reason to keep running it. If you want to analyze it to see how it was breached, take a backup and use an uncompromised system to examine it.

If the intruders have not attempted to cover their tracks, the
last
command may show their SSH logins and the logs in /var/log/httpd may show their web access.


(Ted) #17

Wow, I just went to Admin > Administrators and found to malicious users created and granted all admin rights! That is crazy!
What are the ways for them to create users like that. Would they have to be logged into the main admin account first, or would they need to login through ssh to create those users?


#18

After you work it out, for next time

  • Don’t use port 22, its much quieter . . .
  • exchange ssl keys and then . . .
  • disable password logins.
  • Make sure your fail2ban sshd jail is properly configured and working

(Ted) #19

That is definitely what we’re going to do first thing!
Other than ssh or web, there is not really any other way of creating those users right?


(Ted Mittelstaedt) #20

Actually what you NEED to be doing when you install these is setup a firewall that blocks access from all other IP addresses than the ones that you are expecting to have incoming SIP registrations from and incoming management access from. It sounds to me like you are wanting to deploy FreePBX in a business or some such and are just using the cloud provider to host it. In that case there’s zero reason for your FreePBX server to be accessible in any fashion whatsoever to the general internet. Just open it up to the IP address you have on your external interface that is connected to the Internet. Many of the cloud providers have firewalls themselves, ask about this. Even if you are trying to setup as a SIP extension provider (virtual PBX) you can do this you just require your business customers to have a static IP on their Internet connections.

If, however, your intent is to SELL sip access to anyone on the general Internet, then IMHO you don’t have to security knowledge or skills to do this safely. Hire someone who does or find another line of work.