And what version of Asterisk are you on?
FreePBX 15.0.29
Asterisk : 16.28.0
all modules are activated after installation, but most of them are not used
Asterisk 16.13.0
FreePBX 16.0.33
FreePBX 16.0.33
Asterisk 18.16.0
I have saved off the logs and am going through them. I don’t see any obvious changes but I have 4 other people making changes on a daily basis. It sure would be nice to have logging of changes…
FreePBX 15.0.9
Asterisk 18.9-cert4 But since someone pointed out that I shouldn’t be running this unless I have a contract for support I will be moving to whatever version of asterisk is 18 without the cert.
Distro: 12.7.8-2203-2.sng7
PBX: PBXact 15.0.29
Asterisk: 16.30.0
OK so far I’m seeing only two recent versions of Asterisk, 16.30.0 (EOL but last release) and 18.16.0 (Jan/2023). All the others are out of date versions.
I’m going disappoint everyone. This was installed, I’m assuming, on the 4th and my logs only go back to the 5th. I did check my CDR and only found one attempt on the 4th and they called one of my analog fax machines through the DAHDI card. Then it looks like fail2ban kicked in and dumped them so there is that.
Lorne, you did ask for passwords from non-compromised systems the impacted users had too?
I have checked at least 4 systems updated from v13 to either v15 or v16. I have also looked at boxes that are v16 or v15 or v15 to v16. None of them have the same ARI password.
So if every infected box has the same password but non-infected boxes don’t (and I dont mean cloned systems) then perhaps the password was changed by the bad actor. That possibility is on the table right?
Does anyone know if fail2ban intercepts failed ARI auths?
The res_security_log Asterisk module is configured in FreePBX to write to /var/log/asterisk/fail2ban which is a log file fail2ban reads. I’d check that for ARI requests because it logs authenticated sessions too, including registration auths.
I have very little data to work with. I received the ARI password from two compromised systems from the same user and a number of passwords from uncompromised systems across a diverse set of users and systems. The most notable thing is that the identical ARI passwords from the two compromised systems ALSO matches the password from two unrelated uncompromised systems.
There may yet be more to this, but the fact with my small data set, I still found 4 systems across 3 unrelated users using the same password was enough to convince me to create the pinned post.
Cool, thanks for the update. I know the lack of information is a real speed bump in all this.
I too have had systems hit with this nastiness. 4 client PBXs have been hit with this (out of 28).
On 3 of these servers, I also found a rogue web PHP file in /var/www/html/hex_cmd.php which basically allows direct shell access to whatever command is passed to it. With this, they have deleted the cdr table in asteriskcdrdb and from the web logs, appear they used this to grab /etc/asterisk/ari_additional.conf, which provided the password for ARI. Then used that against port 8089 to make phone calls.
I do note that even the Sangoma Ports used on your PBX references allowing 8089 being open: “Safe to open this up to untrusted networks as the traffic is encrypted with SSL and requires username and password authentication.” We have shut down this port on all our systems now as well, but that does apparently disable to WebRTC phone capability.
What I haven’t been able to determine is how the entry was made to place the hex_cmd.php file on the webserver, or if something else is being used as the initial entry point. One at least one system, only 8089 was open, not 8088 (with an SBC in front of the PBX)
Awesome detective work. I’m curious about the fourth PBX that does not have hex_cmd.php. Does its ARI password match one of the other machines? Does its web log show errors that could indicate attempts to install rogue files (that failed because the vulnerability wasn’t present)?