Gui failing to load

After the break I came back to my CPU pegged at 100% I was unable to SSH in or connect via the web GUI. I power cycled the unit. I can now access through SSH but cannot access through the GUI, the phone apps are not working as well. I have also noticed under systemctl that fail2ban & httpd services are failing.

Any Ideas?

Currently my restapps are at v15.0.20 will this still affect me?

If your server was exposed to the Internet during the time that the vulnerable restapps was installed then you might have been hacked before the 15.0.20 update was applied.

Search your server for some of these clues.

/etc/passwd file - does it have a “supports” or “supermaint” user with ID 0?

/tmp/k - does this file exist?

/var/www/html/r0r.php - ?

Do a ps ax and see whether there are any weird scripts running, or wgets.

Under /etc/passwd file I found two instances of

supermaint:x:0:0::/home/supermaint:/bin/bash
sugarmaint:x:0:0::/home/sugarmaint:/bin/bash

As for the /tmp/k it does exist.

/var/www/html/r0r.php Does not

Unfortunately for ps ax I do not know what I am looking at really.

  PID TTY      STAT   TIME COMMAND
    1 ?        Ss     0:13 /usr/lib/systemd/systemd --switched-root --system --d
    2 ?        S      0:00 [kthreadd]
    4 ?        S<     0:00 [kworker/0:0H]
    5 ?        S      0:02 [kworker/u8:0]
    6 ?        S      0:00 [ksoftirqd/0]
    7 ?        S      0:00 [migration/0]
    8 ?        S      0:00 [rcu_bh]
    9 ?        R      0:57 [rcu_sched]
   10 ?        S<     0:00 [lru-add-drain]
   11 ?        S      0:00 [watchdog/0]
   12 ?        S      0:00 [watchdog/1]
   13 ?        S      0:00 [migration/1]
   14 ?        S      0:00 [ksoftirqd/1]
   16 ?        S<     0:00 [kworker/1:0H]
   17 ?        S      0:00 [watchdog/2]
   18 ?        S      0:00 [migration/2]
   19 ?        S      0:00 [ksoftirqd/2]
   21 ?        S<     0:00 [kworker/2:0H]
   22 ?        S      0:00 [watchdog/3]
   23 ?        S      0:00 [migration/3]
   24 ?        S      0:01 [ksoftirqd/3]
   26 ?        S<     0:00 [kworker/3:0H]
   28 ?        S      0:00 [kdevtmpfs]
   29 ?        S<     0:00 [netns]
   30 ?        S      0:00 [khungtaskd]
   31 ?        S<     0:00 [writeback]
   32 ?        S<     0:00 [kintegrityd]
   33 ?        S<     0:00 [bioset]
   34 ?        S<     0:00 [bioset]
   35 ?        S<     0:00 [bioset]
   36 ?        S<     0:00 [kblockd]
   37 ?        S<     0:00 [md]
   38 ?        S<     0:00 [edac-poller]
   39 ?        S<     0:00 [watchdogd]
   45 ?        S      0:10 [kswapd0]
   46 ?        SN     0:00 [ksmd]
   47 ?        SN     0:00 [khugepaged]
   48 ?        S<     0:00 [crypto]
   56 ?        S<     0:00 [kthrotld]
   59 ?        S<     0:00 [kmpath_rdacd]
   60 ?        S<     0:00 [kaluad]
   62 ?        S<     0:00 [kpsmoused]
   64 ?        S<     0:00 [ipv6_addrconf]
   77 ?        S<     0:00 [deferwq]
  114 ?        S      0:01 [kauditd]
  281 ?        S      0:01 [kworker/u8:2]
  296 ?        S<     0:00 [ata_sff]
  301 ?        S      0:00 [scsi_eh_0]
  302 ?        S<     0:00 [scsi_tmf_0]
  304 ?        S      0:00 [scsi_eh_1]
  305 ?        S<     0:00 [scsi_tmf_1]
  346 ?        S<     0:00 [kworker/0:1H]
  347 ?        S<     0:00 [kworker/u9:0]
  348 ?        S      0:00 [i915/signal:0]
  349 ?        S      0:00 [i915/signal:1]
  350 ?        S      0:00 [i915/signal:2]
  391 ?        S<     0:00 [kdmflush]
  392 ?        S<     0:00 [bioset]
  403 ?        S<     0:00 [kdmflush]
  404 ?        S<     0:00 [bioset]
  418 ?        S<     0:00 [bioset]
  419 ?        S<     0:00 [xfsalloc]
  420 ?        S<     0:00 [xfs_mru_cache]
  421 ?        S<     0:00 [xfs-buf/dm-0]
  422 ?        S<     0:00 [xfs-data/dm-0]
  423 ?        S<     0:00 [xfs-conv/dm-0]
  424 ?        S<     0:00 [xfs-cil/dm-0]
  425 ?        S<     0:00 [xfs-reclaim/dm-]
  426 ?        S<     0:00 [xfs-log/dm-0]
  427 ?        S<     0:00 [xfs-eofblocks/d]
  428 ?        S      0:17 [xfsaild/dm-0]
  514 ?        Ss     0:30 /usr/lib/systemd/systemd-journald
  532 ?        Ss     0:00 /usr/sbin/lvmetad -f
  539 ?        Ss     0:00 /usr/lib/systemd/systemd-udevd
  581 ?        S<     0:00 [kworker/1:1H]
  609 ?        S<     0:00 [kvm-irqfd-clean]
  610 ?        S<     0:00 [kworker/3:1H]
  626 ?        S      0:00 [jbd2/sda2-8]
  627 ?        S<     0:00 [ext4-rsv-conver]
  644 ?        S<sl   0:04 /sbin/auditd
  669 ?        Ss     0:01 /usr/sbin/irqbalance --foreground
  671 ?        Ss     0:04 /usr/lib/systemd/systemd-logind
  672 ?        Ssl    0:03 /usr/lib/polkit-1/polkitd --no-debug
  673 ?        Ss     0:11 /usr/bin/dbus-daemon --system --address=systemd: --no
  675 ?        Ss     0:00 /sbin/rpcbind -w
  680 ?        Ssl   11:39 /etc/.etcservice/linuxservice
  682 ?        Ss     0:02 avahi-daemon: running [uc-47634720.local]
  688 ?        Ss     0:01 /usr/sbin/incrond
  697 ?        S      0:00 avahi-daemon: chroot helper
  703 ?        S      0:00 /usr/sbin/chronyd -f /etc/sangoma_chrony.conf
  717 ?        S<     0:00 [cfg80211]
  742 ?        S<     0:03 [kworker/2:1H]
 1001 ?        Ssl    0:28 /usr/bin/redis-server 127.0.0.1:6379
 1003 ?        Ssl    0:12 /usr/sbin/rsyslogd -n
 1006 ?        Ss     0:00 /usr/sbin/xinetd -stayalive -pidfile /var/run/xinetd.
 1008 ?        Ssl    0:03 /usr/bin/python2 -Es /usr/sbin/tuned -l -P
 1011 ?        Ss     0:00 /usr/sbin/sshd -D
 1014 ?        Ss     0:01 /usr/sbin/dnsmasq -k
 1044 ?        Ss     0:00 /usr/sbin/atd -f
 1057 ?        Ss     0:00 /usr/sbin/crond -n
 1107 ?        S      0:04 /usr/bin/python3.6 -m aiohttp.web aiovega.web:app_fac
 1111 tty1     Ss+    0:00 /sbin/agetty --noclear tty1 linux
 1148 ?        Sl     2:09 /usr/bin/mongod --quiet -f /etc/mongod.conf run
 1169 ?        S      0:00 /bin/bash /opt/xactview3/server//startup.sh -- XactVi
 1196 ?        Ss     0:00 /bin/sh /usr/bin/mysqld_safe --basedir=/usr
 1206 ?        S      0:00 /bin/bash XactViewServer.sh start
 1577 ?        Sl     2:35 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib
 1708 ?        Ss     0:01 /usr/libexec/postfix/master -w
 1728 ?        S      0:03 qmgr -l -t unix -u
 1744 ?        Sl     3:51 /usr/lib/jvm/jre-openjdk//bin/java -Xms256m -XX:MinHe
 1827 ?        Ss     0:00 /usr/bin/python /usr/local/bin/pnp_server
 2361 ?        S      0:04 [kworker/0:9]
 2607 ?        S      0:25 php /var/www/html/admin/modules/firewall/hooks/voipfi
 2749 ?        S      0:00 /bin/sh /usr/sbin/safe_asterisk -U asterisk -G asteri
 2753 ?        Sl     8:46 /usr/sbin/asterisk -f -U asterisk -G asterisk -vvvg -
 3606 ?        Ssl    0:51 PM2 v4.5.0: God Daemon (/home/asterisk/.pm2)
 3712 ?        Ss     0:21 php /var/www/html/admin/modules/restapps/restapps.php
 3934 ?        Ssl    0:02 /usr/local/proxy-client/bin/proxy-client --remoteHost
 4038 ?        Ssl    1:12 node /var/www/html/admin/modules/sangomartapi/node/ki
 4167 ?        Ssl    1:28 node /var/www/html/admin/modules/ucp/node/index.js
 4363 ?        Ssl    1:27 letschat
 4508 ?        Ssl    3:14 node /var/www/html/admin/modules/zulu/node/index.js
 4617 ?        S      0:00 /var/www/html/admin/modules/zulu/node/node_modules/me
 4618 ?        S      0:00 /var/www/html/admin/modules/zulu/node/node_modules/me
 4619 ?        S      0:00 /var/www/html/admin/modules/zulu/node/node_modules/me
 4620 ?        S      0:00 /var/www/html/admin/modules/zulu/node/node_modules/me
 6909 ?        S      0:25 voipfirewalld (Monitor thread)
11181 ?        S      0:00 cleanup -z -t unix -u
11706 ?        S      0:00 local -t unix
13852 ?        S      0:00 pickup -l -t unix -u
14728 ?        R      0:00 [kworker/1:1]
15074 ?        S      0:00 cleanup -z -t unix -u
15749 ?        S      0:00 [kworker/3:0]
16184 ?        S      0:00 bounce -z -t unix -u
18621 ?        S      0:00 [kworker/1:2]
18879 ?        S      0:00 [kworker/3:1]
19199 ?        S      0:00 local -t unix
19345 ?        S      0:00 [kworker/2:0]
21245 ?        S      0:00 [kworker/0:0]
21532 ?        S      0:00 trivial-rewrite -n rewrite -t unix -u
22503 ?        S      0:00 [kworker/2:1]
24180 ?        Ds     0:00 sshd: [email protected]/0
24232 pts/0    Ss     0:00 -bash
24366 pts/0    Sl     0:07 irqbalance --foreground
24376 ?        S      0:00 [kworker/0:2]
24970 ?        S      0:00 [kworker/3:2]
25320 ?        S      0:00 [kworker/1:0]
25711 ?        R      0:00 [kworker/2:2]
26221 ?        S      0:00 /usr/sbin/CROND -n
26224 ?        Ss     0:00 /bin/sh -c [ -e /usr/sbin/fwconsole ] && sleep $((RAN
26228 ?        S      0:00 sleep 29
26599 ?        Ss     0:00 sshd: [accepted]
26611 ?        S      0:00 sshd: [net]
26615 pts/0    R+     0:00 ps ax
27454 ?        S      0:01 [kworker/0:1]
32270 ?        Ssl    0:14 node /var/www/html/admin/modules/sangomaconnect/node/

Thanks,

Those are signs of the exploit @dicko linked to. To stop the bleeding I recommend you stop httpd and crond, remove the bogus accounts and get rid of the /tmp/k. Asterisk will still work and you can have phone service while you work on recovery.

Thank you very much. I have a temporary back up pc I can use. Luckily we are a small office I might just wipe it and start clean.

You rock @billsimon

Gotta say those ‘guys’ in the Netherlands/Iceland/CountryCode7 did a very quick , very clever and very nasty opportunistic compromise in a very short time span.

They are well organized, well distributed, very skillful and likely well funded.

The script touches large parts of a compromised system.

I suggest anyone with a tendency to ‘paranoid’ or ‘wise virgin’ to check the existence , timestamp and content of all of

/etc/passwd
/tmp/test.sh
/usr/local/asterisk/ha_trigger
/var/spool/asterisk/tmp/k
/var/spool/asterisk/tmp/test.sh
/var/www/html/admin/assets/ajax.php
/var/www/html/admin/assets/config.php
/var/www/html/admin/assets/js/config.php
/var/www/html/admin/modules/core/ajax.php
/var/www/html/admin/modules/freepbx_ha/license
/var/www/html/admin/modules/freepbx_ha/license.php
/var/www/html/admin/views/ajax.php
/var/www/html/admin/views/footer.php
/var/www/html/digium_phones/ajax.php
/var/www/html/rest_phones/ajax.php

Properly installing and configuring a ‘root kit’ detector can help detect future compromises. I use http://rkhunter.sourceforge.net/

Also DROP 37.49.230.0/24 in your firewall
whois -h whois.cymru.com ’ -v -f 37.49.230.74’

and related networks belonging to AS213371

1 Like

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.