FreePBX 13 Hacked! Update to FreePBX 14 left with YUM repository issues

andy_woolford · June 15, 2018, 5:26pm

I am sorry to say that I was hacked, with responsive firewall and iptables active.

The offending IPs were mainly based in Palestine.

The hacker created a new extension on my system with the caller ID “EL BACHIR CHANFAR” <222> on the 10 June 2018 and then proceeded to make around 200 call attempts around the world, of which around 100 seem to have been successfully connected. The cost to me was around £50. My sip service provider identifed some but by no means all of these attempts and blocked them.

Reviewing my security arrangements, I have:

Blocked the entire offending IP superblocks x.x.x.x/8
Changed the password for the GUI and moved the ports.
Port 80 is now disabled save for access to the Letsencrypt ./well-known directories.
I also changed the asterisk manager password (it was already changed from the default).

I would like to turn off responsive firewall, but unfortunately I cannot as I have some dynamic clients, (mobile phones). (See my questions at the end)

According to my SIP service provider, he says the hacker had probably used direct SQL injection to create the extension on FreePBX and probably wouldn’t need to access the GUI. This would suggest I also need to change the SQL table password but I don’t see a setting for that.

To try and improve security, I have also now completed the upgrade to FreePBX 14, following these instructions:

https://wiki.freepbx.org/display/PPS/Upgrading+from+FreePBX+10.13.66+to+SNG7

I don’t know if that implements any more security features, but I felt it was best to ensure all upgrades were in place.

Unfortunately the upgrade did not go without issues and I had to run the post_upgrade repair:

wget http://package1.sangoma.net/post_upgrade
chmod 755 ./post_upgrade
./post_upgrade

This did actually fix the upgrade but left some additional installed packages in an uncertain state, which was fixed with yum update.

The new FreePBX 14 appears to be running perfectly along with my other packages (postfix / dovecot / openvpn). However YUM is still reporting:

There are 351 System updates available. Run yum update to update them.

However, if I run yum update again, it prepares to Install 7 Packages + 49 Dependent packages and Upgrade 336 Packages.

When I answer “Y” to proceed I get the following error:

Downloading packages:
Running transaction check
ERROR with transaction check vs depsolve:
epel-release = 6 is needed by (installed) ius-release-1.0-14.ius.el6.noarch
** Found 112 pre-existing rpmdb problem(s), 'yum check' output follows:

I then get a very long list of packages, some of which claim to be a “duplicate” and some of which say has “Missing requires”, and then yum aborts.

I have tried cleaning yum with no effect with yum clean all and rm -rf /var/cache/yum.

The output from repoclosure lists all the unmet dependencies from the repo depositories as:

~# repoclosure
Reading in repository metadata - please wait....
Checking Dependencies
Repos looked at: 5
   sng-base
   sng-epel
   sng-extras
   sng-pkgs
   sng-updates
Num Packages in Repos: 24274
package: 2:nodejs-devel-8.11.1-1.5.x86_64 from sng-pkgs
  unresolved deps:
     nodejs(x86-64) = 0:8.11.1-1.5
package: 2:nodejs-devel-8.11.2-1.6.x86_64 from sng-pkgs
  unresolved deps:
     nodejs(x86-64) = 0:8.11.2-1.6
package: 2:nodejs-devel-8.9.4-1.4.x86_64 from sng-pkgs
  unresolved deps:
     nodejs(x86-64) = 0:8.9.4-1.4
package: airinv-1.00.1-2.el7.x86_64 from sng-epel
  unresolved deps:
     libzmq.so.4()(64bit)
package: banshee-2.6.2-11.el7.x86_64 from sng-epel
  unresolved deps:
     libgpod-sharp >= 0:0.8.2
package: beets-1.4.3-2.el7.noarch from sng-epel
  unresolved deps:
     python-jellyfish
     python-musicbrainzngs >= 0:0.4
     python-mutagen >= 0:1.23
package: beets-plugins-1.4.3-2.el7.noarch from sng-epel
  unresolved deps:
     pylast
     python-acoustid
     python-mpd
     python-musicbrainzngs >= 0:0.4
package: bionetgen-2.2.5-2.el7.x86_64 from sng-epel
  unresolved deps:
     libsundials_cvode.so.1()(64bit)
     libsundials_nvecserial.so.0()(64bit)
package: collectd-amqp-5.8.0-3.el7.x86_64 from sng-epel
  unresolved deps:
     librabbitmq.so.1()(64bit)
package: dragonegg-3.4-5.el7.x86_64 from sng-epel
  unresolved deps:
     gcc = 0:4.8.5-4.el7
package: golang-bazil-fuse-devel-0-0.2.20160811git371fbbd.el7.noarch from sng-epel
  unresolved deps:
     golang(golang.org/x/net/context)
package: golang-github-aws-aws-sdk-go-devel-1.4.22-0.1.git6c577e9.el7.noarch from sng-epel
  unresolved deps:
     golang(github.com/go-ini/ini)
     golang(github.com/gucumber/gucumber)
     golang(github.com/jmespath/go-jmespath)
     golang(golang.org/x/net/html)
     golang(golang.org/x/tools/go/loader)
package: golang-github-google-go-genproto-devel-0-0.3.git411e09b.el7.noarch from sng-epel
  unresolved deps:
     golang(golang.org/x/net/context)
package: golang-github-goraft-raft-devel-0-0.5.git73f9c44.el7.noarch from sng-epel
  unresolved deps:
     golang(code.google.com/p/goprotobuf)
package: golang-github-grpc-grpc-go-devel-1.0.0-0.2.git231b4cf.el7.noarch from sng-epel
  unresolved deps:
     golang(golang.org/x/net/context)
     golang(golang.org/x/net/http2)
     golang(golang.org/x/net/http2/hpack)
     golang(golang.org/x/net/trace)
package: golang-github-pkg-sftp-devel-0-0.1.git8197a2e.el7.noarch from sng-epel
  unresolved deps:
     golang(golang.org/x/crypto/ssh)
package: golang-github-rackspace-gophercloud-devel-1.0.0-14.el7.noarch from sng-epel
  unresolved deps:
     golang(github.com/mitchellh/mapstructure)
package: golang-github-rackspace-gophercloud-unit-test-1.0.0-14.el7.x86_64 from sng-epel
  unresolved deps:
     golang(golang.org/x/crypto/ssh)
package: golang-github-smartystreets-assertions-devel-1.6.0-0.7.git287b434.el7.noarch from sng-epel
  unresolved deps:
     golang(golang.org/x/net/context)
package: golang-github-spacemonkeygo-spacelog-devel-0-0.6.gitae95ccc.el7.noarch from sng-epel
  unresolved deps:
     golang(github.com/spacemonkeygo/flagfile/utils)
package: golang-golangorg-oauth2-devel-0-0.18.git1364adb.el7.noarch from sng-epel
  unresolved deps:
     golang(golang.org/x/net/context)
     golang(google.golang.org/appengine)
     golang(google.golang.org/appengine/urlfetch)
package: golang-google-golangorg-cloud-devel-0-0.10.git872c736.el7.noarch from sng-epel
  unresolved deps:
     golang(golang.org/x/net/context)
     golang(google.golang.org/api/bigquery/v2)
     golang(google.golang.org/api/container/v1)
     golang(google.golang.org/api/googleapi)
     golang(google.golang.org/api/logging/v1beta3)
     golang(google.golang.org/api/pubsub/v1)
     golang(google.golang.org/api/storage/v1)
     golang(google.golang.org/appengine)
     golang(google.golang.org/appengine/file)
     golang(google.golang.org/appengine/log)
package: gthumb-3.3.4-1.el7.x86_64 from sng-epel
  unresolved deps:
     libexiv2.so.12()(64bit)
package: jabber-roster-0.1.1-7.el7.noarch from sng-epel
  unresolved deps:
     python-xmpp
package: kf5-frameworkintegration-5.36.0-2.el7.x86_64 from sng-epel
  unresolved deps:
     qt5-qtbase(x86-64) = 0:5.6.2
package: kf5-kdeclarative-5.36.0-2.el7.x86_64 from sng-epel
  unresolved deps:
     qt5-qtbase(x86-64) = 0:5.6.2
package: kmod-dahdi-linux-2.11.1-12.sng7.x86_64 from sng-pkgs
  unresolved deps:
     kernel(sysfs_remove_link) = 0:0x5d6346c9
     kernel(sysfs_create_link) = 0:0x533a4987
package: kmod-dahdi-linux-2.11.1-50.sng7.x86_64 from sng-pkgs
  unresolved deps:
     kernel(sysfs_remove_link) = 0:0x5d6346c9
     kernel(sysfs_create_link) = 0:0x533a4987
package: kmod-forcedeth-0.64-3.sng7.x86_64 from sng-pkgs
  unresolved deps:
     kernel(napi_complete_done) = 0:0x905307be
package: kmod-forcedeth-0.64-4.sng7.x86_64 from sng-pkgs
  unresolved deps:
     kernel(napi_complete_done) = 0:0x905307be
package: kmod-via-rhine-1.5.1-3.sng7.x86_64 from sng-pkgs
  unresolved deps:
     kernel(napi_complete_done) = 0:0x905307be
package: kmod-via-velocity-1.15-2.sng7.x86_64 from sng-pkgs
  unresolved deps:
     kernel(napi_complete_done) = 0:0x905307be
package: kmod-wanpipe-7.0.21-1.sng7.x86_64 from sng-pkgs
  unresolved deps:
     kernel(inet_dgram_ops) = 0:0x7f8e2719
package: kmod-wanpipe-7.0.22-2.sng7.x86_64 from sng-pkgs
  unresolved deps:
     kernel(inet_dgram_ops) = 0:0x7f8e2719
package: libxfcegui4-4.10.0-5.el7.x86_64 from sng-epel
  unresolved deps:
     libxfce4util.so.6()(64bit)
package: llvm-ocaml-3.4.2-8.el7.x86_64 from sng-epel
  unresolved deps:
     ocaml(Int32) = 0:ad06f04cfca6d404d1de76c3dc67324a
     ocaml(Int64) = 0:3945db6e8df0d5a79bcbc949ee550d52
     ocaml(Pervasives) = 0:36b5bc8227dc9914c6d9fd9bdcfadb45
     ocaml(Unix) = 0:93736a394d3d85d6d127fe238ddc6092
     ocaml(runtime) = 0:4.01.1
package: mediawiki123-HTTP302Found-2.0.1-3.el7.noarch from sng-epel
  unresolved deps:
     mediawiki123
package: mediawiki123-RSS-2.25.0-1.el7.noarch from sng-epel
  unresolved deps:
     mediawiki123
package: mediawiki123-intersection-1.7.0-1.el7.noarch from sng-epel
  unresolved deps:
     mediawiki123
package: nodejs-bson-0.2.9-1.el7.x86_64 from sng-epel
  unresolved deps:
     nodejs(abi) = 0:0.10
     nodejs(v8-abi) = 0:3.14
package: nodejs-follow-0.11.4-2.el7.noarch from sng-epel
  unresolved deps:
     nodejs(engine) < 0:0.11
package: nodejs-fs-ext-0.4.2-2.el7.x86_64 from sng-epel
  unresolved deps:
     nodejs(abi) = 0:0.10
     nodejs(v8-abi) = 0:3.14
package: nodejs-i2c-0.1.4-9.el7.x86_64 from sng-epel
  unresolved deps:
     nodejs(abi) = 0:0.10
     nodejs(v8-abi) = 0:3.14
package: nodejs-is-builtin-module-1.0.0-1.el7.noarch from sng-epel
  unresolved deps:
     npm(builtin-modules) >= 0:1.0.0
     npm(builtin-modules) < 0:2
package: nodejs-libxmljs-0.9.0-1.el7.x86_64 from sng-epel
  unresolved deps:
     nodejs(abi) = 0:0.10
     nodejs(v8-abi) = 0:3.14
package: nodejs-node-expat-2.1.4-5.el7.x86_64 from sng-epel
  unresolved deps:
     nodejs(abi) = 0:0.10
     nodejs(v8-abi) = 0:3.14
package: nodejs-node-stringprep-0.2.3-5.el7.x86_64 from sng-epel
  unresolved deps:
     nodejs(abi) = 0:0.10
     nodejs(v8-abi) = 0:3.14
package: nodejs-pg-0.12.3-2.el7.x86_64 from sng-epel
  unresolved deps:
     nodejs(abi) = 0:0.10
     nodejs(v8-abi) = 0:3.14
package: notify-sharp3-3.0.3-2.el7.x86_64 from sng-epel
  unresolved deps:
     mono(mscorlib) = 0:2.0.0.0
package: opensips-event_rabbitmq-1.10.5-3.el7.x86_64 from sng-epel
  unresolved deps:
     librabbitmq.so.1()(64bit)
package: phototonic-1.7.20-2.el7.x86_64 from sng-epel
  unresolved deps:
     libexiv2.so.12()(64bit)
package: php-drush-drush-6.2.0-6.el7.noarch from sng-epel
  unresolved deps:
     php-channel(pear.drush.org)
package: php-pecl-amqp-1.4.0-1.el7.x86_64 from sng-epel
  unresolved deps:
     librabbitmq.so.1()(64bit)
package: php-pecl-krb5-1.1.2-1.el7.x86_64 from sng-epel
  unresolved deps:
     libkadm5clnt_mit.so.8()(64bit)
     libkadm5clnt_mit.so.8(kadm5clnt_mit_8_MIT)(64bit)
package: pix-1.6.1-3.el7.x86_64 from sng-epel
  unresolved deps:
     libexiv2.so.12()(64bit)
package: pyexiv2-0.3.2-22.el7.x86_64 from sng-epel
  unresolved deps:
     libexiv2.so.12()(64bit)
package: python-atomic-reactor-1.6.23.2-1.el7.noarch from sng-epel
  unresolved deps:
     python-docker-squash >= 0:1.0.0-0.3
package: python-django-1.6.11.6-1.el7.noarch from sng-epel
  unresolved deps:
     python-django-bash-completion = 0:1.6.11.6-1.el7
package: python-dnf-langpacks-0.15.1-1.el7.noarch from sng-epel
  unresolved deps:
     dnf
     dnf-plugins-core
package: python-proliantutils-2.1.0-1.el7.noarch from sng-epel
  unresolved deps:
     python-oslo-concurrency
     python-oslo-utils
package: python-qpid-qmf-1.35.0-1.el7.x86_64 from sng-epel
  unresolved deps:
     qpid-qmf(x86-64) = 0:1.35.0-1.el7
package: python2-pyfakefs-3.1-1.el7.noarch from sng-epel
  unresolved deps:
     python-pytest >= 0:2.8.6
package: python2-wikitcms-2.3.0-1.el7.noarch from sng-epel
  unresolved deps:
     python2-openidc-client >= 0:0.4.0
package: python3-yamlordereddictloader-0.3.0-1.el7.noarch from sng-epel
  unresolved deps:
     python3-PyYAML
package: qt-creator-4.1.0-3.el7.x86_64 from sng-epel
  unresolved deps:
     qt5-qtbase(x86-64) = 0:5.6.2
package: qt5-qtquick1-5.6.2-1.64faeb0git.el7.x86_64 from sng-epel
  unresolved deps:
     qt5-qtbase(x86-64) = 0:5.6.2
     qt5-qtscript(x86-64) = 0:5.6.2
package: qt5-qtquickcontrols2-5.6.2-1.el7.x86_64 from sng-epel
  unresolved deps:
     qt5-qtbase(x86-64) = 0:5.6.2
     qt5-qtdeclarative(x86-64) = 0:5.6.2
     qt5-qtgraphicaleffects(x86-64) = 0:5.6.2
package: qt5-qtstyleplugins-5.0.0-15.el7.x86_64 from sng-epel
  unresolved deps:
     qt5-qtbase(x86-64) = 0:5.6.2
package: ruby-qpid-qmf-1.35.0-1.el7.x86_64 from sng-epel
  unresolved deps:
     qpid-qmf(x86-64) = 0:1.35.0-1.el7
package: rubygem-apipie-bindings-0.0.10-2.el7.noarch from sng-epel
  unresolved deps:
     rubygem(awesome_print)
     rubygem(oauth)
package: simcrs-1.01.1-2.el7.x86_64 from sng-epel
  unresolved deps:
     libzmq.so.4()(64bit)
package: slim-1.3.6-6.el7.x86_64 from sng-epel
  unresolved deps:
     desktop-backgrounds-basic
package: uwsgi-plugin-python36u-2.0.15-1.ius.centos7.x86_64 from sng-pkgs
  unresolved deps:
     uwsgi-plugin-common = 0:2.0.15

So my questions are:

How do I secure the mysql database from further hacking attempts
Which logs would tell me how he actually gained access and why would fail2ban not have prevented this?
Can anyone tell me if I were to turn off responsive firewall and whitelist all my known networks, if the firewall will still allow connections on the ports listed in the “Services Tab” from unknown IP addresses?
What can I do about the YUM errors mentioned?

Many thanks

Andy

tm1000 · June 16, 2018, 3:08am

Your sip provider is just throwing out random baseless facts. Where’s the proof of any of this? There hasn’t been a direct sql vulnerability against freepbx in several years and there hasn’t been a vulnerability that would allow extension creation in over two years. The last one being the RCE in our wiki Sangoma Documentation

Also. Even if there was an sql vulnerability changing the “table password” wouldn’t do you any good. Firstly there’s no such thing as a table password. Secondly MySQL is bound to the local address. The only way someone externally can access mysql is through freepbx and if you prevent freepbx then you can’t use freepbx.

I wouldn’t put a lot of faith into this provider that is throwing around baseless opinions with no data to back it up. They haven’t even looked at your logs they just assumed.

You can forward the logs to me in a private message for review.

As for yum. I’m on mobile right now. Someone should come along to help you soon

andy_woolford · June 16, 2018, 10:21am

Hi Andrew,

Thanks for the reply. The SIP service provider mentioned “SQL injection”. The phrase “table password” was mine. Sorry I had assumed Freepbx needed a password to access the MySQL database in which the table(s) reside.

I would love to know how this hacker did what he did - and I couldn’t decode the logs. The service provider wasn’t interested in doing that as it wasn’t his system that was hacked. I am very grateful for your kind offer to take a look.

Could you tell me exactly which logs you need? There is of course the asterisk full log, but I guess you would like to see the apache and system logs too?

Regards

Andy

tm1000 · June 16, 2018, 3:09pm

Yes it does need a password. But FreePBX can only talk to mysql using the freepbx database user over the local interface 127.0.0.1. Therefore if a user had obtained the password for MySQL they wouldn’t be able to do anything with it.

github.com

FreePBX/framework/blob/release/13.0/installlib/installcommand.class.php#L479


      
          			} elseif((empty($amp_conf['AMPDBUSER'])) && empty($amp_conf['AMPDBPASS'])) {
          				$amp_conf['AMPDBUSER'] = $answers['dbuser'];
          				$amp_conf['AMPDBPASS'] = $answers['dbpass'];
          			}
          
          			if($dbroot) {
          				if($force) {
          					$pdodb->query("DROP DATABASE IF EXISTS ".$amp_conf['AMPDBNAME']);
          				}
          				$pdodb->query("CREATE DATABASE IF NOT EXISTS ".$amp_conf['AMPDBNAME']." DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_unicode_ci");
          				$sql = "GRANT ALL PRIVILEGES ON ".$amp_conf['AMPDBNAME'].".* TO '" . $amp_conf['AMPDBUSER'] . "'@'localhost' IDENTIFIED BY '" . $amp_conf['AMPDBPASS'] . "'";
          				$pdodb->query($sql);
          			} else {
          				//check collate
          			}
          
          			$bmo = new \FreePBX($amp_conf);
          
          			$dsn = $amp_conf['AMPDBENGINE'] . ":host=" . $amp_conf['AMPDBHOST'];
          			$db = new \DB(new \FreePBX\Database($dsn, $answers['dbuser'], $answers['dbpass']));

There is no unauthenticated SQL injection and there’ hasn’t been in freepbx in a long long time.

Don’t really need the asterisk logs. Need the apache logs and freepbx logs.

andy_woolford · June 16, 2018, 7:43pm

I see what you mean.

I will send you the logs by private message.

Thanks again Andrew.

Andy

tm1000 · June 16, 2018, 9:22pm

also include @xrobau in the message

andy_woolford · June 16, 2018, 10:15pm

Wilco. Done

cynjut · June 20, 2018, 3:18pm

As with HTTP; make sure your external access to SSH is likewise blocked. Once you have access to the localhost, lots of bad things can happen.

andy_woolford · June 22, 2018, 5:55am

Yes indeed. SSH has always been on a completely different port and the root password is a long alphanumeric. I was thinking of blocking it entirely and just use VPN from now on.

system · June 22, 2019, 5:55am

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.