Concerned about a possible vulnerability with SSL or SEC-2020-004 or New LetsEncrypt FW rules

adolfoc · July 30, 2020, 5:54pm

I suddenly got several hundred alerts from Fail2Ban for services (SSH, SIP, and recidive!!) that were not exposed to Internet/Public zones… upon further inspection somehow the FreePBX firewall services were inadvertently turned off but not by me…

diving deeper into the freepbx_security.log I found the following:
[2020-Jul-29 22:14:08] [freepbx_security.NOTICE]: [Restapps] WARNING!!! Unexpected activity has been detected from: 176.126.175.10 [] []
[2020-Jul-30 00:22:09] [freepbx_security.NOTICE]: Authentication failure for admin from 45.143.220.116 [] []
[2020-Jul-30 00:22:09] [freepbx_security.NOTICE]: Authentication failure for admin from 45.143.220.116 [] []
[2020-Jul-30 02:19:59] [freepbx_security.NOTICE]: [Restapps] WARNING!!! Unexpected activity has been detected from: 222.186.61.19 [] []
[2020-Jul-30 09:41:34] [freepbx_security.NOTICE]: Authentication failure for admin from 45.143.220.116 [] []
[2020-Jul-30 09:41:34] [freepbx_security.NOTICE]: Authentication failure for admin from 45.143.220.116 [] []

So then i looked into the full logs and found these that stood out:
[2020-07-30 03:56:08] ERROR[11557] iostream.c: Problem setting up ssl connection: error:00000005:lib(0):func(0):DH lib, System call EOF
[2020-07-30 03:56:08] ERROR[11557] tcptls.c: Unable to set up ssl connection with peer ‘185.56.80.49:22591’
[2020-07-30 03:56:08] ERROR[11557] iostream.c: SSL_shutdown() failed: error:00000001:lib(0):func(0):reason(1), Internal SSL error
[2020-07-30 04:18:30] WARNING[3172] chan_sip.c: Timeout on 1170279209-1432990090-985608681 on non-critical invite transaction.
[2020-07-30 04:21:22] NOTICE[3172][C-0000038d] chan_sip.c: Failed to authenticate device <sip:love@REDACTED>;tag=598158323 for INVITE, code = -1
[2020-07-30 04:21:23] NOTICE[3172][C-0000038e] chan_sip.c: Failed to authenticate device <sip:love@REDACTED>;tag=760492329 for INVITE, code = -1
[2020-07-30 04:21:24] NOTICE[3172][C-0000038f] chan_sip.c: Failed to authenticate device <sip:love@REDACTED>;tag=1972265539 for INVITE, code = -1
[2020-07-30 04:21:24] NOTICE[3172][C-00000390] chan_sip.c: Failed to authenticate device <sip:love@REDACTED>;tag=423412600 for INVITE, code = -1
[2020-07-30 04:21:25] NOTICE[3172][C-00000391] chan_sip.c: Failed to authenticate device <sip:love@REDACTED>;tag=563426531 for INVITE, code = -1
[2020-07-30 04:21:26] NOTICE[3172][C-00000392] chan_sip.c: Failed to authenticate device <sip:love@REDACTED>;tag=1654117528 for INVITE, code = -1
[2020-07-30 04:21:54] WARNING[3172] chan_sip.c: Retransmission timeout reached on transmission 2053166945-1726588841-646329173 for seqno 2 (Critical Response) – See https://wiki.asterisk.org/wiki/display/AST/SIP+Retransmissions
Packet timed out after 32000ms with no response
[2020-07-30 04:21:55] WARNING[3172] chan_sip.c: Retransmission timeout reached on transmission 971922622-1176669823-258880245 for seqno 2 (Critical Response) – See https://wiki.asterisk.org/wiki/display/AST/SIP+Retransmissions
Packet timed out after 32000ms with no response
[2020-07-30 04:21:56] WARNING[3172] chan_sip.c: Retransmission timeout reached on transmission 1581264549-1332467517-79792049 for seqno 2 (Critical Response) – See https://wiki.asterisk.org/wiki/display/AST/SIP+Retransmissions
Packet timed out after 32001ms with no response
[2020-07-30 04:21:56] WARNING[3172] chan_sip.c: Retransmission timeout reached on transmission 530972814-865645333-393049948 for seqno 2 (Critical Response) – See https://wiki.asterisk.org/wiki/display/AST/SIP+Retransmissions
Packet timed out after 32000ms with no response
[2020-07-30 04:21:57] WARNING[3172] chan_sip.c: Retransmission timeout reached on transmission 1483041143-890867727-1979619675 for seqno 2 (Critical Response) – See https://wiki.asterisk.org/wiki/display/AST/SIP+Retransmissions
Packet timed out after 31999ms with no response
[2020-07-30 04:21:58] WARNING[3172] chan_sip.c: Retransmission timeout reached on transmission 128706528-922952693-656273059 for seqno 2 (Critical Response) – See https://wiki.asterisk.org/wiki/display/AST/SIP+Retransmissions
Packet timed out after 32000ms with no response
[2020-07-30 04:30:05] NOTICE[3172][C-00000393] chan_sip.c: Failed to authenticate device <sip:1001@REDACTED>;tag=459698780 for INVITE, code = -1
[2020-07-30 04:30:06] NOTICE[3172][C-00000394] chan_sip.c: Failed to authenticate device <sip:1000@REDACTED>;tag=1471970940 for INVITE, code = -1
[2020-07-30 04:30:06] NOTICE[3172][C-00000395] chan_sip.c: Failed to authenticate device <sip:1001@REDACTED>;tag=2001042774 for INVITE, code = -1
[2020-07-30 04:30:07] NOTICE[3172][C-00000396] chan_sip.c: Failed to authenticate device <sip:1000@REDACTED>;tag=233748044 for INVITE, code = -1
[2020-07-30 04:30:07] NOTICE[3172][C-00000397] chan_sip.c: Failed to authenticate device <sip:1001@REDACTED>;tag=1834721715 for INVITE, code = -1
[2020-07-30 04:30:08] NOTICE[3172][C-00000398] chan_sip.c: Failed to authenticate device <sip:1000@REDACTED>;tag=1133866189 for INVITE, code = -1
[2020-07-30 04:30:08] NOTICE[3172][C-00000399] chan_sip.c: Failed to authenticate device <sip:1001@REDACTED>;tag=112346606 for INVITE, code = -1
[2020-07-30 04:30:09] NOTICE[3172][C-0000039a] chan_sip.c: Failed to authenticate device <sip:1000@REDACTED>;tag=608406118 for INVITE, code = -1
[2020-07-30 04:30:09] NOTICE[3172][C-0000039b] chan_sip.c: Failed to authenticate device <sip:1001@REDACTED>;tag=471499846 for INVITE, code = -1
[2020-07-30 04:30:10] NOTICE[3172][C-0000039c] chan_sip.c: Failed to authenticate device <sip:1000@REDACTED>;tag=1013204358 for INVITE, code = -1
[2020-07-30 04:30:13] NOTICE[3172][C-0000039d] chan_sip.c: Failed to authenticate device <sip:201@REDACTED>;tag=1853710243 for INVITE, code = -1
[2020-07-30 04:30:13] NOTICE[3172][C-0000039e] chan_sip.c: Failed to authenticate device <sip:1001@REDACTED>;tag=1973201575 for INVITE, code = -1
[2020-07-30 04:30:14] NOTICE[3172][C-0000039f] chan_sip.c: Failed to authenticate device <sip:201@REDACTED>;tag=747196418 for INVITE, code = -1
[2020-07-30 04:30:14] NOTICE[3172][C-000003a0] chan_sip.c: Failed to authenticate device <sip:1001@REDACTED>;tag=1300656798 for INVITE, code = -1
[2020-07-30 04:30:15] NOTICE[3172][C-000003a1] chan_sip.c: Failed to authenticate device <sip:201@REDACTED>;tag=2051311799 for INVITE, code = -1
[2020-07-30 04:30:15] NOTICE[3172][C-000003a2] chan_sip.c: Failed to authenticate device <sip:1001@REDACTED>;tag=337912026 for INVITE, code = -1
[2020-07-30 04:30:16] NOTICE[3172][C-000003a3] chan_sip.c: Failed to authenticate device <sip:201@REDACTED>;tag=1501366844 for INVITE, code = -1
[2020-07-30 04:30:16] NOTICE[3172][C-000003a4] chan_sip.c: Failed to authenticate device <sip:1001@REDACTED>;tag=1062036484 for INVITE, code = -1
[2020-07-30 04:30:17] NOTICE[3172][C-000003a5] chan_sip.c: Failed to authenticate device <sip:201@REDACTED>;tag=1470032640 for INVITE, code = -1
[2020-07-30 04:30:17] NOTICE[3172][C-000003a6] chan_sip.c: Failed to authenticate device <sip:1001@REDACTED>;tag=1619141091 for INVITE, code = -1
[2020-07-30 04:30:17] NOTICE[3172][C-000003a7] chan_sip.c: Failed to authenticate device <sip:201@REDACTED>;tag=1398681484 for INVITE, code = -1
[2020-07-30 04:30:18] NOTICE[3172][C-000003a8] chan_sip.c: Failed to authenticate device <sip:1001@REDACTED>;tag=1371450778 for INVITE, code = -1
[2020-07-30 04:30:37] WARNING[3172] chan_sip.c: Retransmission timeout reached on transmission 344409493-654093979-1893151406 for seqno 2 (Critical Response) – See https://wiki.asterisk.org/wiki/display/AST/SIP+Retransmissions
Packet timed out after 32000ms with no response
[2020-07-30 04:30:38] WARNING[3172] chan_sip.c: Retransmission timeout reached on transmission 1144715390-1950013116-820163247 for seqno 2 (Critical Response) – See https://wiki.asterisk.org/wiki/display/AST/SIP+Retransmissions
Packet timed out after 32001ms with no response
[2020-07-30 04:30:38] WARNING[3172] chan_sip.c: Retransmission timeout reached on transmission 81410782-1036223805-210602602 for seqno 2 (Critical Response) – See https://wiki.asterisk.org/wiki/display/AST/SIP+Retransmissions
Packet timed out after 32000ms with no response
[2020-07-30 04:30:39] WARNING[3172] chan_sip.c: Retransmission timeout reached on transmission 879592745-565242646-1681703610 for seqno 2 (Critical Response) – See https://wiki.asterisk.org/wiki/display/AST/SIP+Retransmissions
Packet timed out after 31999ms with no response
[2020-07-30 04:30:39] WARNING[3172] chan_sip.c: Retransmission timeout reached on transmission 42347264-644194217-246084590 for seqno 2 (Critical Response) – See https://wiki.asterisk.org/wiki/display/AST/SIP+Retransmissions
Packet timed out after 32000ms with no response
[2020-07-30 04:30:40] WARNING[3172] chan_sip.c: Retransmission timeout reached on transmission 719349474-1978769998-666519812 for seqno 2 (Critical Response) – See https://wiki.asterisk.org/wiki/display/AST/SIP+Retransmissions
Packet timed out after 32000ms with no response
[2020-07-30 04:30:40] WARNING[3172] chan_sip.c: Retransmission timeout reached on transmission 1397925985-2049680111-1145075041 for seqno 2 (Critical Response) – See https://wiki.asterisk.org/wiki/display/AST/SIP+Retransmissions
Packet timed out after 32000ms with no response
[2020-07-30 04:30:41] WARNING[3172] chan_sip.c: Retransmission timeout reached on transmission 190691075-1821182216-965419007 for seqno 2 (Critical Response) – See https://wiki.asterisk.org/wiki/display/AST/SIP+Retransmissions
Packet timed out after 32000ms with no response
[2020-07-30 04:30:41] WARNING[3172] chan_sip.c: Retransmission timeout reached on transmission 465694773-1582094534-57845845 for seqno 2 (Critical Response) – See https://wiki.asterisk.org/wiki/display/AST/SIP+Retransmissions
Packet timed out after 32000ms with no response
[2020-07-30 04:30:42] WARNING[3172] chan_sip.c: Retransmission timeout reached on transmission 620020169-1991082022-143485199 for seqno 2 (Critical Response) – See https://wiki.asterisk.org/wiki/display/AST/SIP+Retransmissions
Packet timed out after 32000ms with no response
[2020-07-30 04:30:45] WARNING[3172] chan_sip.c: Retransmission timeout reached on transmission 563296442-815921870-1205174152 for seqno 2 (Critical Response) – See https://wiki.asterisk.org/wiki/display/AST/SIP+Retransmissions
Packet timed out after 32000ms with no response
[2020-07-30 04:30:45] WARNING[3172] chan_sip.c: Retransmission timeout reached on transmission 1171765980-515304150-2015434982 for seqno 2 (Critical Response) – See https://wiki.asterisk.org/wiki/display/AST/SIP+Retransmissions
Packet timed out after 32000ms with no response
[2020-07-30 04:30:46] WARNING[3172] chan_sip.c: Retransmission timeout reached on transmission 221272472-1937019104-846465968 for seqno 2 (Critical Response) – See https://wiki.asterisk.org/wiki/display/AST/SIP+Retransmissions
Packet timed out after 32000ms with no response
[2020-07-30 04:30:46] WARNING[3172] chan_sip.c: Retransmission timeout reached on transmission 195530063-387829032-1196144748 for seqno 2 (Critical Response) – See https://wiki.asterisk.org/wiki/display/AST/SIP+Retransmissions
Packet timed out after 32001ms with no response
[2020-07-30 04:30:47] WARNING[3172] chan_sip.c: Retransmission timeout reached on transmission 404696427-1049245587-1017069336 for seqno 2 (Critical Response) – See https://wiki.asterisk.org/wiki/display/AST/SIP+Retransmissions
Packet timed out after 31999ms with no response
[2020-07-30 04:30:47] WARNING[3172] chan_sip.c: Retransmission timeout reached on transmission 1258627545-983998471-1507022237 for seqno 2 (Critical Response) – See https://wiki.asterisk.org/wiki/display/AST/SIP+Retransmissions
Packet timed out after 32000ms with no response
[2020-07-30 04:30:48] WARNING[3172] chan_sip.c: Retransmission timeout reached on transmission 119292003-1654983023-1133277836 for seqno 2 (Critical Response) – See https://wiki.asterisk.org/wiki/display/AST/SIP+Retransmissions
Packet timed out after 32001ms with no response
[2020-07-30 04:30:48] WARNING[3172] chan_sip.c: Retransmission timeout reached on transmission 788207180-1713809638-1719142516 for seqno 2 (Critical Response) – See https://wiki.asterisk.org/wiki/display/AST/SIP+Retransmissions
Packet timed out after 32001ms with no response
[2020-07-30 04:30:49] WARNING[3172] chan_sip.c: Retransmission timeout reached on transmission 689081081-52814014-204869644 for seqno 2 (Critical Response) – See https://wiki.asterisk.org/wiki/display/AST/SIP+Retransmissions
Packet timed out after 32001ms with no response
[2020-07-30 04:30:49] WARNING[3172] chan_sip.c: Retransmission timeout reached on transmission 1901163883-246237524-580613550 for seqno 2 (Critical Response) – See https://wiki.asterisk.org/wiki/display/AST/SIP+Retransmissions
Packet timed out after 32000ms with no response
[2020-07-30 04:30:49] WARNING[3172] chan_sip.c: Retransmission timeout reached on transmission 1707910763-1257200638-704735722 for seqno 2 (Critical Response) – See https://wiki.asterisk.org/wiki/display/AST/SIP+Retransmissions
Packet timed out after 32000ms with no response
[2020-07-30 04:30:50] WARNING[3172] chan_sip.c: Retransmission timeout reached on transmission 1596770768-345970907-651838136 for seqno 2 (Critical Response) – See https://wiki.asterisk.org/wiki/display/AST/SIP+Retransmissions
Packet timed out after 32000ms with no response
[2020-07-30 04:30:50] WARNING[3172] chan_sip.c: Timeout on 603219844-2079745078-1293337300 on non-critical invite transaction.

than another:
[2020-07-30 04:51:24] ERROR[9625] iostream.c: Problem setting up ssl connection: error:00000005:lib(0):func(0):DH lib, System call EOF
[2020-07-30 04:51:24] ERROR[9625] tcptls.c: Unable to set up ssl connection with peer ‘185.56.80.49:52890’
[2020-07-30 04:51:24] ERROR[9625] iostream.c: SSL_shutdown() failed: error:00000001:lib(0):func(0):reason(1), Internal SSL error

followed by a ton more requests for hours…

Looked at fail2ban logs for warnings:
2020-07-30 04:21:26,962 fail2ban.actions[25654]: WARNING [asterisk-iptables] Ban 164.132.201.36
was the first one… and then just continued to unban and ban

Looked at secure log and found these:
Jul 30 04:32:50 testpbx sshd[32648]: Did not receive identification string from 80.82.70.118 port 60000
Jul 30 04:33:01 testpbx sshd[639]: Did not receive identification string from 5.8.10.202 port 15014
Jul 30 04:33:01 testpbx sshd[641]: Connection closed by 5.8.10.202 port 46906 [preauth]
Jul 30 04:55:20 testpbx sshd[18634]: Connection closed by 139.162.122.110 port 52862 [preauth]
Jul 30 04:55:21 testpbx sshd[18636]: Invalid user from 139.162.122.110 port 53050
Jul 30 04:55:21 testpbx sshd[18636]: input_userauth_request: invalid user [preauth]
Jul 30 04:55:21 testpbx sshd[18636]: Failed none for invalid user from 139.162.122.110 port 53050 ssh2
Jul 30 04:55:21 testpbx sshd[18636]: Connection closed by 139.162.122.110 port 53050 [preauth]
Jul 30 05:02:11 testpbx sshd[1884]: Did not receive identification string from 125.160.17.32 port 59593
Jul 30 05:50:20 testpbx sshd[11829]: Invalid user admin from 217.182.192.217 port 51756
Jul 30 05:50:20 testpbx sshd[11829]: input_userauth_request: invalid user admin [preauth]
Jul 30 05:50:21 testpbx sshd[11829]: pam_unix(sshd:auth): check pass; user unknown
Jul 30 05:50:21 testpbx sshd[11829]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=217.182.192.217
Jul 30 05:50:23 testpbx sshd[11829]: Failed password for invalid user admin from 217.182.192.217 port 51756 ssh2
Jul 30 05:50:23 testpbx sshd[11829]: Connection closed by 217.182.192.217 port 51756 [preauth]
Jul 30 05:50:24 testpbx sshd[11934]: Invalid user admin from 217.182.192.217 port 54078
Jul 30 05:50:24 testpbx sshd[11934]: input_userauth_request: invalid user admin [preauth]
Jul 30 05:50:24 testpbx sshd[11934]: pam_unix(sshd:auth): check pass; user unknown
Jul 30 05:50:24 testpbx sshd[11934]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=217.182.192.217
Jul 30 05:50:27 testpbx sshd[11934]: Failed password for invalid user admin from 217.182.192.217 port 54078 ssh2
Jul 30 05:50:27 testpbx sshd[11934]: Connection closed by 217.182.192.217 port 54078 [preauth]
Jul 30 06:57:59 testpbx sshd[2303]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=106.52.93.109 user=root
Jul 30 06:58:01 testpbx sshd[2303]: Failed password for root from 106.52.93.109 port 56182 ssh2
Jul 30 06:58:01 testpbx sshd[2303]: Connection closed by 106.52.93.109 port 56182 [preauth]
Jul 30 06:58:04 testpbx sshd[2699]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=106.52.93.109 user=root
Jul 30 06:58:06 testpbx sshd[2699]: Failed password for root from 106.52.93.109 port 56262 ssh2
Jul 30 06:58:07 testpbx sshd[2699]: Connection closed by 106.52.93.109 port 56262 [preauth]
Jul 30 06:58:08 testpbx sshd[2948]: Invalid user admin from 106.52.93.109 port 56346
Jul 30 06:58:08 testpbx sshd[2948]: input_userauth_request: invalid user admin [preauth]
Jul 30 06:58:08 testpbx sshd[2948]: pam_unix(sshd:auth): check pass; user unknown
Jul 30 06:58:08 testpbx sshd[2948]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=106.52.93.109
Jul 30 06:58:11 testpbx sshd[2948]: Failed password for invalid user admin from 106.52.93.109 port 56346 ssh2
Jul 30 08:10:18 testpbx sshd[3913]: Did not receive identification string from 192.241.237.229 port 35544
Jul 30 09:58:07 testpbx sshd[20798]: Did not receive identification string from 107.173.181.20 port 53010

This deployment (is a test non-production system of mine) is running
PBX Version:15.0.16.64
PBX Distro:12.7.6-2002-2.sng7
Asterisk Version:16.9.0
PhoneApps 15.0.19.7.
How could there have been outside access to RestFul/RestApps ports when they were closed to the public/internet zone (only Local Zone).
(Also Responsive Firewall is not enabled.)

The only Internet accessible ports are LetsEncrypt, UCP, OpenVPN, and XMPP.
For LetsEncrypt: Allow full Internet zone access to the Let’s Encrypt acme-challenge folder on port 80.
No Custom Rules are enabled.

The only IPs permitted as Trusted in the FPBX firewall are my WAN IP for myself, office, SBC, a SIP trunk provider, and a server that I have ping it as a ping/latency monitor.

Very concerning…
Doesn’t seem like there was any other kind of intrusion … no calls were recorded in CDRs, no routes added, no trunks modified, no users added…

what else can I look for/at?

PitzKey · July 30, 2020, 6:24pm

If the firewall was stopped, then there was no police to stop them.

If any API users were created.
If any new lines in extensions_custom.conf or any of the custom conf files.

It would be nice if FreePBX can send firewall and fail2ban “Has been stopped” emails, instead of all these module security emails that we can’t stop.

adolfoc · July 30, 2020, 6:44pm

Yeah… what could cause the firewall to stop without an admin’s intervention?

Thanks for the other things to look at:
No API users or tokens, no REST users, no Asterisk Manager users
No custom context, custom destinations, custom extensions, or any conf files modified.

However… in looking at changed files i noticed under /etc/schmooze
the schmooze.zl, pbxid, and license-XXXXXXXXX.zl files changed… is that normal, like a chron script to check for licensing?

tonyclewis · July 30, 2020, 6:57pm

Do you use let’s encrypt. If so when did you let cert renew? Others are reporting of the firewall crashing on the renewal of the cert.

adolfoc · July 30, 2020, 6:59pm

I just noticed there was a new Firewall Logs option added in one of the last Firewall updates and looked through the logs there…
The last logged event was:
OUT >>> [2020-07-28 05:26:18] - /sbin/iptables -w5 -W10000 -D fpbxhosts -s REDEACTED -j zone-trusted
Which was a DynDNS resolved host I trusted for an office.

Then it’s just
OUT >>> Redirecting to /bin/systemctl restart ip6tables.service
OUT >>> Redirecting to /bin/systemctl restart iptables.service
on repeat until I manually started the firewall:
OUT >>> [2020-07-30 16:23:59] - Wall: 'Firewall service now starting.

adolfoc · July 30, 2020, 7:03pm

Yes I do use the LetsEncrypt Rules in the firewall… looks like my cert is valid until 2020-08-28… 29 days left… so does the LetsEncrypt renew every 30 days?

I do see an error " There was an error updating certificate “testpbx.REDACTED.com”: REMOTE_ADDR didn’t parse -" 15 Hours, 58 Minutes, 32 Seconds, Ago

eeeeh… so does this mean I’m have the same issue as other people?

jerrm · July 30, 2020, 8:36pm

LetsEncrypt renewals are broken, crash the firewall and fail2ban and will leave your system wide open.

Open ticket: https://issues.freepbx.org/browse/FREEPBX-21683

Only mitigating factor is it will sometimes self repair in 24 hours during the next renewal attempt, but for at least 24 hours your system is naked to the world.

adolfoc · July 30, 2020, 8:41pm

oh geeez!

To be clear this only effects systems that have Firewall LetsEncrypt Rules (allow full Internet zone access…) enabled right? Disabling this should mitigate the issue right?

jerrm · July 30, 2020, 8:46pm

It impacts any system using the firewall module with renewing LE certs. It won’t happen if the LE certs are not due to renew yet.

If your LE certs managed to renew, you’re probably OK for the next 60 days or so. Just be sure to check that the firewall is now enabled.

system · August 6, 2020, 8:46pm

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.