After the break I came back to my CPU pegged at 100% I was unable to SSH in or connect via the web GUI. I power cycled the unit. I can now access through SSH but cannot access through the GUI, the phone apps are not working as well. I have also noticed under systemctl that fail2ban & httpd services are failing.
Any Ideas?
Currently my restapps are at v15.0.20 will this still affect me?
If your server was exposed to the Internet during the time that the vulnerable restapps was installed then you might have been hacked before the 15.0.20 update was applied.
Search your server for some of these clues.
/etc/passwd file - does it have a “supports” or “supermaint” user with ID 0?
/tmp/k - does this file exist?
/var/www/html/r0r.php - ?
Do a ps ax and see whether there are any weird scripts running, or wgets.
Under /etc/passwd file I found two instances of
supermaint:x:0:0::/home/supermaint:/bin/bash
sugarmaint:x:0:0::/home/sugarmaint:/bin/bash
As for the /tmp/k it does exist.
/var/www/html/r0r.php Does not
Unfortunately for ps ax I do not know what I am looking at really.
PID TTY STAT TIME COMMAND
1 ? Ss 0:13 /usr/lib/systemd/systemd --switched-root --system --d
2 ? S 0:00 [kthreadd]
4 ? S< 0:00 [kworker/0:0H]
5 ? S 0:02 [kworker/u8:0]
6 ? S 0:00 [ksoftirqd/0]
7 ? S 0:00 [migration/0]
8 ? S 0:00 [rcu_bh]
9 ? R 0:57 [rcu_sched]
10 ? S< 0:00 [lru-add-drain]
11 ? S 0:00 [watchdog/0]
12 ? S 0:00 [watchdog/1]
13 ? S 0:00 [migration/1]
14 ? S 0:00 [ksoftirqd/1]
16 ? S< 0:00 [kworker/1:0H]
17 ? S 0:00 [watchdog/2]
18 ? S 0:00 [migration/2]
19 ? S 0:00 [ksoftirqd/2]
21 ? S< 0:00 [kworker/2:0H]
22 ? S 0:00 [watchdog/3]
23 ? S 0:00 [migration/3]
24 ? S 0:01 [ksoftirqd/3]
26 ? S< 0:00 [kworker/3:0H]
28 ? S 0:00 [kdevtmpfs]
29 ? S< 0:00 [netns]
30 ? S 0:00 [khungtaskd]
31 ? S< 0:00 [writeback]
32 ? S< 0:00 [kintegrityd]
33 ? S< 0:00 [bioset]
34 ? S< 0:00 [bioset]
35 ? S< 0:00 [bioset]
36 ? S< 0:00 [kblockd]
37 ? S< 0:00 [md]
38 ? S< 0:00 [edac-poller]
39 ? S< 0:00 [watchdogd]
45 ? S 0:10 [kswapd0]
46 ? SN 0:00 [ksmd]
47 ? SN 0:00 [khugepaged]
48 ? S< 0:00 [crypto]
56 ? S< 0:00 [kthrotld]
59 ? S< 0:00 [kmpath_rdacd]
60 ? S< 0:00 [kaluad]
62 ? S< 0:00 [kpsmoused]
64 ? S< 0:00 [ipv6_addrconf]
77 ? S< 0:00 [deferwq]
114 ? S 0:01 [kauditd]
281 ? S 0:01 [kworker/u8:2]
296 ? S< 0:00 [ata_sff]
301 ? S 0:00 [scsi_eh_0]
302 ? S< 0:00 [scsi_tmf_0]
304 ? S 0:00 [scsi_eh_1]
305 ? S< 0:00 [scsi_tmf_1]
346 ? S< 0:00 [kworker/0:1H]
347 ? S< 0:00 [kworker/u9:0]
348 ? S 0:00 [i915/signal:0]
349 ? S 0:00 [i915/signal:1]
350 ? S 0:00 [i915/signal:2]
391 ? S< 0:00 [kdmflush]
392 ? S< 0:00 [bioset]
403 ? S< 0:00 [kdmflush]
404 ? S< 0:00 [bioset]
418 ? S< 0:00 [bioset]
419 ? S< 0:00 [xfsalloc]
420 ? S< 0:00 [xfs_mru_cache]
421 ? S< 0:00 [xfs-buf/dm-0]
422 ? S< 0:00 [xfs-data/dm-0]
423 ? S< 0:00 [xfs-conv/dm-0]
424 ? S< 0:00 [xfs-cil/dm-0]
425 ? S< 0:00 [xfs-reclaim/dm-]
426 ? S< 0:00 [xfs-log/dm-0]
427 ? S< 0:00 [xfs-eofblocks/d]
428 ? S 0:17 [xfsaild/dm-0]
514 ? Ss 0:30 /usr/lib/systemd/systemd-journald
532 ? Ss 0:00 /usr/sbin/lvmetad -f
539 ? Ss 0:00 /usr/lib/systemd/systemd-udevd
581 ? S< 0:00 [kworker/1:1H]
609 ? S< 0:00 [kvm-irqfd-clean]
610 ? S< 0:00 [kworker/3:1H]
626 ? S 0:00 [jbd2/sda2-8]
627 ? S< 0:00 [ext4-rsv-conver]
644 ? S<sl 0:04 /sbin/auditd
669 ? Ss 0:01 /usr/sbin/irqbalance --foreground
671 ? Ss 0:04 /usr/lib/systemd/systemd-logind
672 ? Ssl 0:03 /usr/lib/polkit-1/polkitd --no-debug
673 ? Ss 0:11 /usr/bin/dbus-daemon --system --address=systemd: --no
675 ? Ss 0:00 /sbin/rpcbind -w
680 ? Ssl 11:39 /etc/.etcservice/linuxservice
682 ? Ss 0:02 avahi-daemon: running [uc-47634720.local]
688 ? Ss 0:01 /usr/sbin/incrond
697 ? S 0:00 avahi-daemon: chroot helper
703 ? S 0:00 /usr/sbin/chronyd -f /etc/sangoma_chrony.conf
717 ? S< 0:00 [cfg80211]
742 ? S< 0:03 [kworker/2:1H]
1001 ? Ssl 0:28 /usr/bin/redis-server 127.0.0.1:6379
1003 ? Ssl 0:12 /usr/sbin/rsyslogd -n
1006 ? Ss 0:00 /usr/sbin/xinetd -stayalive -pidfile /var/run/xinetd.
1008 ? Ssl 0:03 /usr/bin/python2 -Es /usr/sbin/tuned -l -P
1011 ? Ss 0:00 /usr/sbin/sshd -D
1014 ? Ss 0:01 /usr/sbin/dnsmasq -k
1044 ? Ss 0:00 /usr/sbin/atd -f
1057 ? Ss 0:00 /usr/sbin/crond -n
1107 ? S 0:04 /usr/bin/python3.6 -m aiohttp.web aiovega.web:app_fac
1111 tty1 Ss+ 0:00 /sbin/agetty --noclear tty1 linux
1148 ? Sl 2:09 /usr/bin/mongod --quiet -f /etc/mongod.conf run
1169 ? S 0:00 /bin/bash /opt/xactview3/server//startup.sh -- XactVi
1196 ? Ss 0:00 /bin/sh /usr/bin/mysqld_safe --basedir=/usr
1206 ? S 0:00 /bin/bash XactViewServer.sh start
1577 ? Sl 2:35 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib
1708 ? Ss 0:01 /usr/libexec/postfix/master -w
1728 ? S 0:03 qmgr -l -t unix -u
1744 ? Sl 3:51 /usr/lib/jvm/jre-openjdk//bin/java -Xms256m -XX:MinHe
1827 ? Ss 0:00 /usr/bin/python /usr/local/bin/pnp_server
2361 ? S 0:04 [kworker/0:9]
2607 ? S 0:25 php /var/www/html/admin/modules/firewall/hooks/voipfi
2749 ? S 0:00 /bin/sh /usr/sbin/safe_asterisk -U asterisk -G asteri
2753 ? Sl 8:46 /usr/sbin/asterisk -f -U asterisk -G asterisk -vvvg -
3606 ? Ssl 0:51 PM2 v4.5.0: God Daemon (/home/asterisk/.pm2)
3712 ? Ss 0:21 php /var/www/html/admin/modules/restapps/restapps.php
3934 ? Ssl 0:02 /usr/local/proxy-client/bin/proxy-client --remoteHost
4038 ? Ssl 1:12 node /var/www/html/admin/modules/sangomartapi/node/ki
4167 ? Ssl 1:28 node /var/www/html/admin/modules/ucp/node/index.js
4363 ? Ssl 1:27 letschat
4508 ? Ssl 3:14 node /var/www/html/admin/modules/zulu/node/index.js
4617 ? S 0:00 /var/www/html/admin/modules/zulu/node/node_modules/me
4618 ? S 0:00 /var/www/html/admin/modules/zulu/node/node_modules/me
4619 ? S 0:00 /var/www/html/admin/modules/zulu/node/node_modules/me
4620 ? S 0:00 /var/www/html/admin/modules/zulu/node/node_modules/me
6909 ? S 0:25 voipfirewalld (Monitor thread)
11181 ? S 0:00 cleanup -z -t unix -u
11706 ? S 0:00 local -t unix
13852 ? S 0:00 pickup -l -t unix -u
14728 ? R 0:00 [kworker/1:1]
15074 ? S 0:00 cleanup -z -t unix -u
15749 ? S 0:00 [kworker/3:0]
16184 ? S 0:00 bounce -z -t unix -u
18621 ? S 0:00 [kworker/1:2]
18879 ? S 0:00 [kworker/3:1]
19199 ? S 0:00 local -t unix
19345 ? S 0:00 [kworker/2:0]
21245 ? S 0:00 [kworker/0:0]
21532 ? S 0:00 trivial-rewrite -n rewrite -t unix -u
22503 ? S 0:00 [kworker/2:1]
24180 ? Ds 0:00 sshd: root@pts/0
24232 pts/0 Ss 0:00 -bash
24366 pts/0 Sl 0:07 irqbalance --foreground
24376 ? S 0:00 [kworker/0:2]
24970 ? S 0:00 [kworker/3:2]
25320 ? S 0:00 [kworker/1:0]
25711 ? R 0:00 [kworker/2:2]
26221 ? S 0:00 /usr/sbin/CROND -n
26224 ? Ss 0:00 /bin/sh -c [ -e /usr/sbin/fwconsole ] && sleep $((RAN
26228 ? S 0:00 sleep 29
26599 ? Ss 0:00 sshd: [accepted]
26611 ? S 0:00 sshd: [net]
26615 pts/0 R+ 0:00 ps ax
27454 ? S 0:01 [kworker/0:1]
32270 ? Ssl 0:14 node /var/www/html/admin/modules/sangomaconnect/node/
Thanks,
Those are signs of the exploit @dicko linked to. To stop the bleeding I recommend you stop httpd and crond, remove the bogus accounts and get rid of the /tmp/k. Asterisk will still work and you can have phone service while you work on recovery.
Thank you very much. I have a temporary back up pc I can use. Luckily we are a small office I might just wipe it and start clean.
You rock @billsimon
Gotta say those ‘guys’ in the Netherlands/Iceland/CountryCode7 did a very quick , very clever and very nasty opportunistic compromise in a very short time span.
They are well organized, well distributed, very skillful and likely well funded.
The script touches large parts of a compromised system.
I suggest anyone with a tendency to ‘paranoid’ or ‘wise virgin’ to check the existence , timestamp and content of all of
/etc/passwd
/tmp/test.sh
/usr/local/asterisk/ha_trigger
/var/spool/asterisk/tmp/k
/var/spool/asterisk/tmp/test.sh
/var/www/html/admin/assets/ajax.php
/var/www/html/admin/assets/config.php
/var/www/html/admin/assets/js/config.php
/var/www/html/admin/modules/core/ajax.php
/var/www/html/admin/modules/freepbx_ha/license
/var/www/html/admin/modules/freepbx_ha/license.php
/var/www/html/admin/views/ajax.php
/var/www/html/admin/views/footer.php
/var/www/html/digium_phones/ajax.php
/var/www/html/rest_phones/ajax.php
Properly installing and configuring a ‘root kit’ detector can help detect future compromises. I use http://rkhunter.sourceforge.net/
Also DROP 37.49.230.0/24 in your firewall
whois -h whois.cymru.com ’ -v -f 37.49.230.74’
and related networks belonging to AS213371
This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.