It’s unfortunate that you had to find this out the hard way, but glad to hear you’re sorted. Please take a moment and familiarize yourself with the firewall config and concepts, so that you fully understand where you went wrong.
I know how frustrated you feel. I know that feeling very well. Lucky you that is ‘just’ a pbx: ransomware is boiling everywhere.
My few cents to mitigation.
About sip provider. I have a low amount per refill (actually just $30 or any amount that meets your daily use) with a limited number of refills per day (just one): if this number is exceeded I will receive an mail notification, I will know that
a) One of my users was hacked and his extension is being used for calls or
b) The PBX was breached.
c) Business is doing great!! increase the refill amount.
SSH: I have SSH with no password, only SSL key. Not difficult to setup and very effective. To stop the infinite login attempts, limit port 22 to a few familiar know IPs or networks.
HTTP: same as above. Limit who can do it.
SIP/PJSIP ports: Why more than a couple of password attempts? I block the IP after the second failure. No exceptions. Ahh you are blocking a legal user? get your IP from any get your ip web and manually remove the lock. 99.9% of the user leave the password on the app or phone, so only a hacker will ‘try’ different options.
MYSQL: only access from 127.0.0.1 Note: this is a VERY easy way to access your PBX.
Use a virtual machine for your PBX, even an small one. take frequent checkpoints: if you are in trouble, just go back to the last good one. Checkpoint after every change. Before you go back, inspect (or save) the logs for the autopsy.
Use Fail2Ban is very effective and default rules accomplish a lot.
I manage ten FreePBX systems. I keep the ports on my router closed to external traffic, and configure IPTables to limit everyone’s local access to needed ports (phones get 5060 and 10-20,000), and haven’t been hacked ever in the nine years I’ve been doing so.
From what I can see this is a system config issue and not (completely) a FreePBX issue. Some basic and initial ideas would be to look at HOW you access the system as an admin. If you can access from anywhere in the world then so can those with malicious intent. If you allow for unlimited login attempts and/or have no policies to protect logins then an attacker can simply brute force a password. The other thing to look at is the logs. For example is the PBX being breached or it is a client? What I mean by that is in these posts you seem to be narrowly focused on the fact that the server is the only issue. The fact of the matter is that if a client was breached (desktop, laptop etc.) they can cause just as much damage. And last but not least the firewall and protocols. The firewall for the organization needs to be aggressive as do your security settings. Limit where and how you can access the admin settings from (ie- require a VPN). Turn off unneeded admin features (FTP, SSH etc.) or at least force them to only be accessible from a fixed list of networks/IPs.
I really appreciate all the replies!
UPDATE: Vitelity informed me that they still detected calls to a high cost domestic number.
I’m so frustrated. I’m also confused. Vitelity sends me the fraudulent CDR. My Freepbx does not show these calls.
Vitelity was nice enough to give me a good will credit of $1000 for the fraud calls (after I paid with a credit card). However, the tech support / fraud support seems to be clueless.
I have my server with Cyberlynk. I have the DID through Vitelity. Is it possible that the Vitelity account can be hacked?
Assuming that the attacker had control of your PBX (rather than just a stolen extension password), he could view (and steal) the credentials for your Vitelity (and other) trunks. He could then use his own system to make calls on your nickel, even with your PBX shut down. I recommend that after securing the PBX, you set up Vitelity to use IP authentication, so they will only accept calls from your PBX’s IP address. Remove any registration-based sub-accounts (or at least change the passwords to new strong values that you don’t put in the PBX).
Vitelity presumably has records showing whether your existing sub accounts were used from other than your PBX’s IP. If not, your system is still compromised.
BTW, it’s not surprising that someone named “Dr. re hak” would be hacked again.
This ^^^ was nearly verbatim what I was going to write.
- Reset your SIP providers WEB admin and any other Admin credentials
- Call your SIP provider and reset the password for the SIP AUTH
- Have them terminate any and all AUTH’d connections (force all connections to Re-AUTH)
- Have them lock down the ‘allowed IPs’ which they will accept AUTH’s from to JUST your PBX and/or others you need to add
- Re-connect your PBX to the SIP provider
Now you should be able to confirm pretty darn quickly which system has the issue. If it is your PBX OR the server it sits on - the calls will keep flowing. If it is the SIP provider then those calls should stop. You can further check this by requesting the AUTH failed logs from your SIP provider. This could (but unlikely) allow them to go after the malicious parties involved.
perhaps THIS is the problem, put it on your own network where you 100% control the security.
Or, move to a complete hosted solution and let them manage it all for you.
You need to learn the definition of toxic, nothing thats been said here is such.
Too many here seem to live in cotton wool balls, yeah go on James you have a history of flagging my posts so dont let me down this time.
That’s the forum self-moderation at work. There are posts in this thread that were flagged by multiple members for different reasons. Enough flags and the post gets hidden.
Fail2ban does not always work. Years ago, I had fail2ban running, and the FreePBX web interface open to the world. Using some script, they got through without needing a password to get through with some exploit in PHP.
Using the Responsive Firewall seems to be a good way to go now (or hide it behind VPN) than depending solely on fail2ban.
Internet-facing servers - whether PBX, web, email, or whatever - need to be professionally configured and managed or it’s likely they will be hacked. If you hide behind a tightly-configured firewall you can probably get away with a reasonable amount of self-management (I realize money is very tight for a lot of businesses right now), but any server or device that takes inbound connections from the Internet is going to get hammered with attacks on an almost non-stop basis. For most of our managed systems we track the numbers on an “attempts per second” basis. That’s just life on the Internet. Nearly all of these attempts are fairy dumb, automated scripts looking for low-hanging fruit and aren’t worth worrying about if you’re locked down and well-maintained. But a dumb, automated script will break through something with a weak configuration or an unpatched vulnerability.
In fairness, Sangoma and the FreePBX team do a flat-out awesome job of producing a security-aware system with some pretty decent self-defense built-in - better and smarter than many expensive commercial products. But it still needs to be set up and managed correctly. I don’t think they could have made it much easier for laypersons, but it still requires a certain amount of skill, and a large amount of attention to detail.
Well, Fail2Ban does a very good job of what you tell it to do, but is not a one button panacea for everything. It does come with a lot of pre-prepared jails but they need to enabled and checked for utility, I am unaware of which ones that are enabled in ‘The Distro’ but your PHP intrusion would NOT be caught by the asterisk Jail, there is an ‘apache-no-script’ jail which might well have, but the intrusion would need to be first captured and then maybe a regex written for Fail2Ban to verify that it can catch and ban the transgressor. I am unaware of any protection offered against TCP/5038 if it is open for example, but every open port needs to be well considered as to whether is should be open, if so how widely and if either of the above , what policing is needed.
As the bad guys get more and more sophisticated, the methods get sly’er and sly’er . Fully protecting your SIP server is a good start but being equally aware of the threats against your web services.
Being I was one of the people who implemented this system I know you have no way to know who flags you… that said I have no idea who you are so I would think you to be mistaken.
On a related, and probably not identical note, we had a system hacked early this month. They brute-forced an account via port 5060, despite Fail2Ban and the Responsive Firewall running, and system-generated complex passwords.Given the right access and enough time, they will get through Fail2Ban and the Responsive Firewall with the default settings.
I would be curious in how that’s possible with the responsive firewall banning the IP after a low number of failed tries and FreePBX generated SIP passwords being very long.
If there is a 'brute force ’ attack there will be a brutal number of ‘catches’ of the failed attempts in Fail2Ban, if they have found another way to compromise an account, perhaps through Asterisk Manager or Web stuff, it might well not be noticed.
It was brute force, thanks. The only open ports were 5060 and 10001-20000. The address to which the notifications were sent wasn’t being monitored. When I checked it had many fail2ban notifications.
I’m not here to quibble. I’m here to reinforce that, in some scenarios, the default Fail2Ban and Responsive Firewall rules aren’t enough to keep unwanted entities out.
. . When I checked it had many fail2ban notifications. . .
Perhaps file a bug with your findings then ?