How can this happen? Unauthorized access again!

What the heck is going on here? You all have helped me secure this system and yet suddenly we have new unauthorized calls. How is this possible, why do the logs not show the call details?

We have:
IP Auth enabled and these calls have the tech ID prepended.
The extension has a 256b password.
fail2ban and firewall are enabled

The log entries for these calls only show following with no other details.

  1. full-20201005:[2020-10-03 02:57:57] VERBOSE[23860][C-00001ddb] pbx.c: Executing [s@macro-user-callerid:1] Set(ā€œPJSIP/205-00000763ā€, ā€œTOUCH_MONITOR=1601715477.1891ā€) in new stack

  2. full-20201005:[2020-10-03 02:58:58] VERBOSE[23860][C-00001ddb] pbx.c: Executing [s@crm-hangup:4] NoOp(ā€œPJSIP/205-00000763ā€, ā€œMASTER CHANNEL: 1601715477.1891 = 1601715477.1891ā€) in new stack

https://pastebin.freepbx.org/view/4e6e69b0

Is that the entire contents of full-20201005 from 02:57:00 through 02:59:00 ? If not, please post the unfiltered log for that time period.

I went to verify and, yes, that is all the log shows for full-20201005 from 02:57:00 through 02:59:00.

If you make a call from ext. 205, does it get logged normally?

What do the CDR records show for that period?

What is the last entry in full-20201005 prior to 02:57:00 ?

Is extensions_custom.conf empty? Or at least contains only things you put in it?

Wrong pastebin and wrong notes. Try this again. Here is a log of one of the unauthorized calls.

https://pastebin.freepbx.org/view/7e943134

How is this an unauthorized call? I mean this went through the dialplan like a call that authed just fine. Are you seeing this extension registered with an IP it shouldnā€™t be? Are you seeing a lot of these calls? Do you have the responsive firewall enabled?

Search the log for
Added contact 'sip:205
to see whether the attacker registered. If so, see where his IP address is and why it got through your firewall.

Correct, it looks like a legit call from an approved IP but that is not one of my remotes. As you say, it appears that some IP has been allowed to make calls. How it can spoof my extension I do not understand. An I reading the log correctly that it was a call from Extension 205?

I have stopped and restarted the firewall and have not seen any odd calls since.

Yes, a lot of them. I agree that it points at the firewall. I have stopped and restarted it and thus far nothing weird.

Still have pirate calls after stopping and restarting firewall. Can anyone give advise here, I cannot see how an outside ip address can access my system.

Here is a log from a call last night, 5 minutes after the office closed.
https://pastebin.freepbx.org/view/73d78d8a

Call traces for this issue are worthless, it just confirms what you know to be true. Based on the bare info provided, I would guess you have malicious users registering to SIP extensions. I assume your SIP extension secrets are secure, and I assume youā€™re using the Distro with a default Intrusion Detection config in place. That means your provisioning service is likely exposed to unstrusted traffic (and if you use tftp, that means hours/days at most between exploits) and your SIP service(s) are exposed to untrusted traffic.

You need a firewall, either external or the PBX firewall module (or both) and you need to configure it so there is NO untrusted traffic to your system.

I agree. Just found that there is no password on the Trunk > PJSIP Settings. The systems uses IP authentication.

Trunk without a sip secret should have authentication set to ā€˜noneā€™, sloppy config but unrelated to your exploit.

Why is the Responsive Firewall not blocking these calls?

I used these settings suggested by Flowroute:

These are different subjects. The first is about phones registering to your PBX from random addresses throughout the world, the second is about trunk interaction from your ITSP.

The Responsive Firewall is a specialized firewall component that manages the interaction of your SIP ports with the outside world and is all about people connecting to your server from phones outside your network. If you donā€™t have anyone connecting from the WWW, thereā€™s no reason to have the Responsive Firewall working.

This firewall works in conjunction with Integrated Firewall, which should be set up to block everything trying to connect to your server, pretty much ā€˜periodā€™. If you are using the Responsive Firewall, your inbound SIP ports (phones and Flowroute) are managed that way. If you donā€™t, you have to manage who can connect to your SIP ports manually (more or less) by defining the networks that are trusted.

The second part of your issue: if you are using IP Authentication, there shouldnā€™t be any registration at all - no username, no password, no registration type, and your authentication is set to ā€˜Noneā€™. The SIP server is your outbound destination.

IIRC, there are several servers that can initiate POTS calls to your PBX from Flowroute. These are documented in the ā€œMatchā€ field so that you donā€™t have to allow anonymous calling.

So, having said all of that, you need to know where the phones that are supposed to connect to your system are connecting from. If they have fixed addresses, you set them up in the regular firewall as trusted addresses. If they come from random addresses all over the world, the Responsive Firewall will catch them. Note, however, that the Responsive Firewall will only catch bad actors that fail to authenticate. If someone manages to acquire credentials on your machine, the Responsive Firewall will gladly let them in.

Let me thank you for your clear and concise summary of this aspect of PBX security.

We have IP Authentication enabled and Flowroute confirms that the pirate calls show our Tech ID prepended to the packets. Responsive Firewall has been enabled all along. We have all the remote phones whitelisted. Could it be that the pirate calls are being allowed because Responsive Firewall is enabled?

The Responsive Firewall is for remote SIP/IAX connections. Allowing you to open the firewall up for that traffic. Someone authenticates properly they end up in a goodip list and that has a byproduct of opening UCP access for them.

It also has rate limiting so if they are hammering the box with REGISTER/INVITES it will blacklist them, however, as long as they auth correctly they get through.

Yes but you are overlooking that a call was made from 205 and 202 (given your logs) and those calls can only happen if they are AUTHING correctly. Which means they have creds of those accounts. You need to really check to see if there is still something on the PBX from when they compromised it with the XSS vulns.

From the system CLI run crontab -u root -l this should be empty if it is not, thereā€™s a problem. Also do crontab -u asterisk -l pastebin the output of that so we can double check no one has thrown a cronjob in there somehow.

We still have a problem, more pirate calls.

The CDR log:

Wed, 7 Oct 2020 13:26 1602098723.842 16073942448 Return ANSWERED 14:28
Wed, 7 Oct 2020 13:26 1602098718.838 16073942448 Return ANSWERED 14:09

This is Full
https://pastebin.freepbx.org/view/b67e9cdd

grep 1602098723.842 full* Returns nothing.