FreePBX Unauthorized Calls

So I inherited a PBX system which for the most part has not given me much issue… until this weekend…

I had a notice of significant charges from our service provider… I did some digging… looks at CDRs and had a TON of calls “from-trunk” and calls from random numbers… they appear to mostly if not all last for approximately 13 seconds…

I did learn through reading that there are two levels of anonymous access…

Allow Anonymous Inbound SIP Calls - was set to NO still set to NO
Allow SIP Guests - was set to YES has since been changed to NO

I notice now that my CDRs have dropped to a fraction of what they were… like over a similar sample size 10,000+ records to 30 records… more importantly the 30 records appear to be legitimate records…

I am waiting on the call detail from the service provider… but I am expecting a million 12 second calls…

can someone explain to me what these are? how this happens? I need to be able to correctly explain this to senior management…

YES the PBX is open to the internet…
YES we are looking at changing that…

but for now I have to at least be able to understand exactly what happened… and I dont think I fully get it…

any help would be really appreciated…

Allow SIP guests, by itself, should not result in charges from your provider. Such calls appear in the CDR with Destination [from-sip-external] and play a ‘not in service’ message. Those are the 12 second calls you see.

The fraudulent charges are likely caused by a different configuration error. If you export the CDRs to Excel and filter out the from-sip-external calls, you may be able to see the other unwanted calls mixed in with the legitimate traffic.

1 Like

OK I do see some calls going out from an old extension that is no longer used (but was still provisioned)…

what can I look at to figure out how the extension was used to make these unauthorized calls. and more importantly stop it from happening again…

I really appreciate the input…

in looking at the logs further I see thousands of wrong password attempts for this extension before finally making successful calls…

any pointers on why these are not getting banned?

apologies in advance as I am certainly not an expert on this…

You might want to check that Intrusion Detection is enabled and running. This is one tool that’s a part of the firewall that should be helping to mitigate those brute force attacks. You’ll find this under Admin -> System Admin -> Intrusion Detection (Right Side). You may want to fine tune those options to your preferences.

1 Like

JUST fixed that… it was not running… it said it was running but was clearly not… I stopped and restarted the service and it already caught two IPs… so its looking promising…

I need to figure out how to update the white list…

Awesome! There is a section on that same page to add IP’s to the whitelist. By default, you should see 127.0.0.1 listed there.

I see that but there is no way to like save them…

and for entire subnet… is it correct to say I could write

10.10.8.0/24
to denote the entire scope of 10.10.8.x

Yes you should be able to add anything in the CIDR format, just the way you listed should work. When you make changes here, there is a blue Submit button down in the bottom right of the screen. Clicking that will save it, but make sure you Restart the service after that in order to take the new changes.

1 Like

OK perfect… I am really thankful for all the help… I have two last questions (I think)

I had to restart fail2ban via CLI… GUI was giving me some trouble but CLI it started right up…

how do I remove IP from jail… LOL

thanks again!!

Restarting the fail2ban service clears the current blacklist. So restarting it should have no entries.

you are completely right…thats exactly what happened… so it appears at least…

fail2ban stopped working at some point…
this allowed someone to attack with 10s of thousands of password attempts (there are literally like 50,000) bad password log items…
they finally gained access and were able to make calls…
our provider flagged the traffic and notified us…

to remedy we have removed devices that are no longer active minimizing the opportunities for a breach
restarted the fail2ban service which is now working
and will adopt the policy to check on fail2ban as part of our daily work
we have also disabled SIP guest to minimize unnecessary traffic towards the servers

sound good? LOL

1 Like

all such problems fixed if you install fail2ban >= 0.9

bans remain over a restart/reboot

they moved to sqlite3 , which maintains bans.

Unless you used a ridiculously weak password such as an English word, or the password was also used at a compromised website, I suspect that there was some other weakness. For example, a password of only 8 random lower case letters has ~208 billion combinations; the probability of finding it in 50,000 probes is less than 1 in 4 million. And, I bet that if you look at the password failures, most of them are for other extensions.

The most common cause of this breach is provisioning data open to the world, not password protected and unencrypted. All the attacker has to do is guess the MAC address of one of your devices. There should be logs showing if this is what happened, but you should lock it down in any case.

If the device for the compromised extension was still online at the time of the breach, its web pages may have been accessible to the attacker (with no password or a default password). If it had been taken offline, the attacker may have gained physical access to it.

There are many other possibilities including network sniffing of unencrypted data, phishing attacks, etc.

1 Like

I thought the same thing… and yes there were other extensions tried… but all I can attest to is what I see which is the following…

an IP address tried to authenticate thousands of times for that extension alone…
then eventually this IP address was able to initiate calls…

I can also tell you that the physical device was absolutely not only… the device that was associated with that extension is unplugged in a storage closet…

with that said… I am certainly NOT stating what you are saying is invalid… I agree with it all… but I only know what I know… and that is a lengthy series of failed connections from that IP to that extension… and ultimately a successful connection…

if there is somewhere else to look… I am all for looking… and understanding…

I am definitely advocating for locking the system down… in the past we had a lot of mobile users to deal with… now we are connecting a much smaller group of people sitting on static IPs which should be pretty simple to lock down… in fact I can’t think of a reason not to…

thats fine… and makes sense… but I am not sure in this case it would have mattered… at some point Fail2Ban stopped banning… and needed to be manually restarted…

How long was the old password? What characters did it use? Was it partly an English word, or something like abcde or qwerty? Had the password been used on another site?

Is the person who previously used it a disgruntled employee or former employee? Did they have access to the phone’s GUI when it was online? Was the same password used e.g. on a mobile app that may have been used on public Wi-Fi?

the password was reasonable… over 8 char… alpha numeric…

as for the rest I dont know…

it was well before me being in this position… but now that I think about it… the IP address for the system was not even the same when that phone was provisioned and that user existed… so even if they had the physical phone (which as I stated they dont)… it would have been configured to connect to an old no longer used IP… and they would have no knowledge of the current IP…

the curious part to me is if they did have some of that info from some other means… why go through the trouble to make 50k bad log in attempts…

now there is something not considered… to be totally honest… I am not sure for how many days… weeks… it tried to connect… this system is not used to the extent that it used to be used… so I definitely need to do my part and pay more attention to it… it is in theory at least possible that that person has been trying to gain access for weeks… meaning that its not 50k tries… its exponentially more times than that…

I don’t know what happened, but I do know that online brute force cracking of a decent random password is essentially impossible. With 9 characters, each randomly chosen from uppercase, lowercase and digits, there are 13+ quadrillion possibilities. Trying only 50,000 daily, on average it would take 370+ million years to succeed.

Some guesses as to what may have happened:

  1. In parallel with guessing (hoping for something stupid like abcd1234), the attacker was accessing your provisioning data and when he hit the correct MAC address, he got the password and stopped the other search. There would likely be logs showing this.

  2. The attacker had a list of e.g. 100,000 passwords somehow stolen from many systems, but did not know which was yours, so he tried them all until he succeeded.

  3. The attacker had obtained the password by some other means, but pretended to do brute force guessing as a decoy to distract you from the real vulnerability.

But, it’s silly to guess what might have happened. Just lock down all aspects that you know about. If you see fail2ban banning even one IP, take a look at why they even got that far.

1 Like

If you are are using fail2ban 0.8 it will do exactly that, when it restarts, it has no ‘memory’ of what got banned, so you will have to add all your bans again manually, if however you have fail2ban >=0.9 it will retain those bans over a restart.

from a shell

fail2ban-client -V

1 Like