LetsEncrypt Error: Connection refused when requesting certificates

There was an error updating the certificate: Error ‘Requested ‘http://xxxxxxx.co.uk//.freepbx-known/b7ee2c4268b04a1ab097d4055f193896’ - Failed connect to xxxxxxxx.co.uk:80; Connection refused’ when requesting http://xxxxxxxxx.co.uk//.freepbx-known/b7ee2c4268b04a1ab097d4055f193896

This is a new install of FreePBX14 and I am trying to obtain certificates for the first time.

There is no external firewall. I have a FQDN and a public fixed IP which resolves perfectly.
Port 80 is pointed to the LetsEncrypt /.freepbx-known directory using the System Admin > Ports.
Port 443 is set on the HTTPS admin interface
Firewall is configured correctly (shows green) for all the letsencrypt and freepbx mirrors. I have explicitly added these to the “Trusted” zone on the firewall.

I have tested that I can actually access the /.freepbx-known directory by manually placing a “hello world” htm file inside that directory and pointing an external browser to it.

I have also checked that mod_rewrite is enabled in the 00-base.conf file located in /etc/httpd/conf.modules.d/ directory, and that the following entry exists and is not commented.

LoadModule rewrite_module modules/mod_rewrite.so

I have also restarted httpd and rebooted the entire machine, however the error persists.

I would welcome any thoughts on this.

Sounds like something upstream is filtering connections to port 80. You might want to speak to whoever is providing your internet.

… Oh. LetsEncrypt have changed the way they validate certificates. They can come from any number of IP addresses now (including TOR endpoints). That’s why the LetsEncrypt port now exists, in Sysadmin, because you have to expose 80 to the entire internet.

Hi Rob,

Thanks for the suggestion, but there were no upstream firewalls as:

I have tested that I can actually access the /.freepbx-known directory by manually placing a “hello world” htm file inside that directory and pointing an external browser to it.

However, I resolved the issue after I spent a number of hours going around in circles. Eventually, I gave up and copied the entire /etc/asterisk/keys folder from my old machine to the new machine. This brought with it, the three existing sub-directories for my subdomain certificates, (complete with the original certificates).

Initially, I tried to import the original certificates directly into the HTTPS settings, however they did not show up presumably because they weren’t written into the FreePBX database tables. However, returning to the “Certificate Management” module they did show up in there and, furthermore, the generation of NEW LetsEncrypt certificates worked flawlessly.

I can only surmise that there was a problem creating a new subdirectory under the empty Keys folder, which caused the rather confusing error:

Failed connect to… Connection refused’

Once the subdomain directories were created, the rest of the process went as well as could be expected.

Many thanks again for your assistance.

Andy

I am resurrecting this thread, 2 months after the original posting due to the auto-renew failing.

I received this error:

There was an error updating the certificate: Error ‘Requested ‘http://xxx.co.uk//.freepbx-known/9d39e1fe3b0234b93f615c4fa28bdc90’ - Failed connect to xxx.co.uk:80; Connection refused’ when requesting http://xxx.co.uk//.freepbx-known/9d39e1fe3b0234b93f615c4fa28bdc90

Observations:

  • I have Lets Encrypt set to Port 80 in System Admin and the firewall is letting connections in
  • The error message occurs whether I try to renew or create a new certificate
  • The previous “solution” above (recreating /etc/asterisk/keys folder) does not fix this error

External Port Scan Result:

Starting Nmap 7.40 ( https://nmap.org ) at 2018-08-21 05:52 UTC
Nmap scan report for xxx.co.uk (xx.xxx.xx.xxx)
Host is up (0.074s latency).
PORT STATE SERVICE
21/tcp filtered ftp
22/tcp filtered ssh
23/tcp filtered telnet
80/tcp open http
110/tcp filtered pop3
143/tcp filtered imap
443/tcp open https
3389/tcp filtered ms-wbt-server

Nmap done: 1 IP address (1 host up) scanned in 2.10 seconds

This only left one possibility, that the Lets Encrypt IP address was blacklisted. Sure enough, the error was cured by turning off the firewall from the console:

% fwconsole firewall stop

Then running the certificate renewals worked without a problem. After the certificates were renewed, then:

% fwconsole firewall start

So, I draw the following conclusions:

  • The originating IP address used by Lets Encrypt was on my blacklist, but it was not possible to identify exactly which one it was.
  • Blacklisted IPs ended up there because they were either the source of spam email or had been repeatedly logged in mail.log as failed logins. Some blacklisted IPs and ranges were also created after being identified as the source of hacking attempts.
  • Lets Encrypt is using IP addresses that may also be used by hackers, such as TOR exits, unsecured relays or public proxies.

Opening Port 80 specifically, does not override a blacklisted IP. This makes it very difficult to whitelist any IPs from Lets Encrypt, even if they were known.

Can anyone suggest another solution? For example permitting any otherwise blacklisted IPs to access port 80 only, when port 80 is directed ONLY to the Lets Encrypt directory?

Might it also be possible to create an index.htm file in that directory that serves a “Go Away… you’re not welcome here” notice, with the user’s IP that would be served by default to any generic accesses to the root of that directory?

I am open to suggestions.

LetsEncrypt can come from ANY IP ADDRESS now. You have to expose all of port 80 to the internet. That’s why there is a LetsEncrypt service, that ONLY PROVIDES letsencrypt.

Hi Rob,

Yes, that’s what I understand and that’s what I’m TRYING to do.

Can you advise HOW I can do that, if I have certain IP’s part of my Blacklist? It seems that the Blacklist overrides the Port 80 service “Whitelist” that I created.

Ideally I would like all of the Blacklisted IPs to still access Port 80 which is only exposed to the /.freepbx-known directory.

At the moment, the easiest way is to create a custom service that is port 80, and allow access to it from external/internal/local.

I don’t see an “external” option on the custom ports. I assume you mean “internal/local/other”?

Does that over-ride the IP Blacklist though? Because I tried that and it didn’t work. I had to turn off the firewall before it let Lets Encrypt through.

For example, I have a mail custom port already which allows Multiple Ports: 25,587,993,995 to internet/local/other. If someone spams my email, I add it to the Blacklist and that blocks it.

Firewall REJECT Rules:

~# iptables -nL | grep REJECT
REJECT all – 166.172.186.229 0.0.0.0/0 reject-with icmp-port-unreachable
REJECT all – 103.0.0.0/8 0.0.0.0/0 reject-with icmp-port-unreachable
REJECT all – 104.0.0.0/8 0.0.0.0/0 reject-with icmp-port-unreachable
REJECT all – 137.59.52.0/22 0.0.0.0/0 reject-with icmp-port-unreachable
REJECT all – 142.44.128.0/17 0.0.0.0/0 reject-with icmp-port-unreachable
REJECT all – 144.217.174.88/29 0.0.0.0/0 reject-with icmp-port-unreachable
REJECT all – 145.239.0.0/16 0.0.0.0/0 reject-with icmp-port-unreachable
REJECT all – 158.69.0.0/16 0.0.0.0/0 reject-with icmp-port-unreachable
REJECT all – 163.172.0.0/16 0.0.0.0/0 reject-with icmp-port-unreachable
REJECT all – 172.93.96.0/20 0.0.0.0/0 reject-with icmp-port-unreachable
REJECT all – 172.98.192.0/20 0.0.0.0/0 reject-with icmp-port-unreachable
REJECT all – 173.208.128.0/17 0.0.0.0/0 reject-with icmp-port-unreachable
REJECT all – 173.240.13.0/24 0.0.0.0/0 reject-with icmp-port-unreachable
REJECT all – 176.58.0.0/16 0.0.0.0/0 reject-with icmp-port-unreachable
REJECT all – 185.0.0.0/8 0.0.0.0/0 reject-with icmp-port-unreachable
REJECT all – 186.0.0.0/8 0.0.0.0/0 reject-with icmp-port-unreachable
REJECT all – 187.0.0.0/8 0.0.0.0/0 reject-with icmp-port-unreachable
REJECT all – 188.0.0.0/8 0.0.0.0/0 reject-with icmp-port-unreachable
REJECT all – 192.162.101.0/24 0.0.0.0/0 reject-with icmp-port-unreachable
REJECT all – 192.99.0.0/16 0.0.0.0/0 reject-with icmp-port-unreachable
REJECT all – 195.154.0.0/16 0.0.0.0/0 reject-with icmp-port-unreachable
REJECT all – 198.50.128.0/17 0.0.0.0/0 reject-with icmp-port-unreachable
REJECT all – 199.0.0.0/8 0.0.0.0/0 reject-with icmp-port-unreachable
REJECT all – 208.100.0.0/18 0.0.0.0/0 reject-with icmp-port-unreachable
REJECT all – 208.234.15.0/24 0.0.0.0/0 reject-with icmp-port-unreachable
REJECT all – 212.129.0.0/18 0.0.0.0/0 reject-with icmp-port-unreachable
REJECT all – 213.0.0.0/8 0.0.0.0/0 reject-with icmp-port-unreachable
REJECT all – 217.182.0.0/16 0.0.0.0/0 reject-with icmp-port-unreachable
REJECT all – 217.79.176.0/20 0.0.0.0/0 reject-with icmp-port-unreachable
REJECT all – 218.104.128.0/20 0.0.0.0/0 reject-with icmp-port-unreachable
REJECT all – 23.239.64.0/19 0.0.0.0/0 reject-with icmp-port-unreachable
REJECT all – 37.0.0.0/8 0.0.0.0/0 reject-with icmp-port-unreachable
REJECT all – 46.0.0.0/8 0.0.0.0/0 reject-with icmp-port-unreachable
REJECT all – 5.133.29.0/24 0.0.0.0/0 reject-with icmp-port-unreachable
REJECT all – 51.15.0.0/16 0.0.0.0/0 reject-with icmp-port-unreachable
REJECT all – 54.240.0.0/18 0.0.0.0/0 reject-with icmp-port-unreachable
REJECT all – 54.39.35.160/28 0.0.0.0/0 reject-with icmp-port-unreachable
REJECT all – 62.0.0.0/8 0.0.0.0/0 reject-with icmp-port-unreachable
REJECT all – 83.0.0.0/8 0.0.0.0/0 reject-with icmp-port-unreachable
REJECT all – 84.247.18.0/23 0.0.0.0/0 reject-with icmp-port-unreachable
REJECT all – 85.17.0.0/16 0.0.0.0/0 reject-with icmp-port-unreachable
REJECT all – 85.25.217.0/24 0.0.0.0/0 reject-with icmp-port-unreachable
REJECT all – 89.163.0.0/16 0.0.0.0/0 reject-with icmp-port-unreachable
REJECT all – 91.121.0.0/16 0.0.0.0/0 reject-with icmp-port-unreachable
REJECT all – 91.192.40.0/22 0.0.0.0/0 reject-with icmp-port-unreachable
REJECT all – 93.115.24.0/21 0.0.0.0/0 reject-with icmp-port-unreachable
REJECT all – 94.23.0.0/16 0.0.0.0/0 reject-with icmp-port-unreachable
REJECT all – 0.0.0.0/0 0.0.0.0/0 reject-with icmp-port-unreachable
REJECT udp – 0.0.0.0/0 0.0.0.0/0 udp dpt:2049 reject-with icmp-port-unreachable
REJECT tcp – 0.0.0.0/0 0.0.0.0/0 tcp dpt:2049 reject-with icmp-port-unreachable
REJECT udp – 0.0.0.0/0 0.0.0.0/0 udp dpt:892 reject-with icmp-port-unreachable
REJECT udp – 0.0.0.0/0 0.0.0.0/0 udp dpt:662 reject-with icmp-port-unreachable
REJECT udp – 0.0.0.0/0 0.0.0.0/0 udp dpt:32769 reject-with icmp-port-unreachable
REJECT tcp – 0.0.0.0/0 0.0.0.0/0 tcp dpt:892 reject-with icmp-port-unreachable
REJECT tcp – 0.0.0.0/0 0.0.0.0/0 tcp dpt:662 reject-with icmp-port-unreachable
REJECT tcp – 0.0.0.0/0 0.0.0.0/0 tcp dpt:32803 reject-with icmp-port-unreachable
REJECT udp – 0.0.0.0/0 0.0.0.0/0 udp dpt:137 reject-with icmp-port-unreachable
REJECT udp – 0.0.0.0/0 0.0.0.0/0 udp dpt:138 reject-with icmp-port-unreachable
REJECT tcp – 0.0.0.0/0 0.0.0.0/0 tcp dpt:139 reject-with icmp-port-unreachable
REJECT tcp – 0.0.0.0/0 0.0.0.0/0 tcp dpt:445 reject-with icmp-port-unreachable
REJECT tcp – 0.0.0.0/0 0.0.0.0/0 tcp dpt:5222 reject-with icmp-port-unreachable

No idea where that is in your chain, which is why it’s much better to do this:

iptables-save | pastebin

However, the simple answer is to just try it!

Thats a very cool command :sunglasses:

https://pastebin.freepbx.org/view/05a01309

I swear I did try adding port 80 as a custom port and it didn’t work though.

image

Yes, the firewall blacklist takes priority. You have MASSIVE amounts of the internet blacklisted, including almost all of AWS which is why it’s not working. I would strongly suggest removing those blacklist entries 8)

Also, RTP is already whitelisted, it’s the second rule in the chain as it gets hit the most.

Do you mean the entries with /8 in the CIDR? (xxx.xxx.xxx.xxx/8) ? I know I have a chunk of the internet blacklisted but basically my machine got taken down in June by some hacker who gained root access. There were a lot of attacks (and email spam) from various nets and many had the first octet in common so that’s why I blocked them.

What is “AWS”? I’m a bit nervous about unblocking them again - even though the machine has had a complete rebuild, passwords changed etc. I figured if they could get in once, they could get in again.

I would have no issues selectively letting them into port 80, knowing it only goes to that one .freepbx-known directory. How could the blacklist chain be modified to do that?

Suggestion:

For example in the blacklist there is this rule:

-A fpbxblacklist -s 103.0.0.0/8 -j REJECT --reject-with icmp-port-unreachable

How about if it were written instead as:

-A fpbxblacklist -s 103.0.0.0/8 ! --dport 80 -j REJECT --reject-with icmp-port-unreachable

So the command “! --dport 80” is inserted if the Lets Encrypt port is activated?

Amazon web services

Thanks Andrew. Is blocking AWS a bad thing then? (I don’t have any Amazon accounts running through this server).

How about my proposed modification to the blacklist? Would that work?

Thanks again Andrew.

AWS is all new to me. Quite an eye-opener!

As it happens, I do run a VPN through this server and Netflix is one of the services I access via the tunnel, Amazon Prime being another. Despite the blocked IP ranges, there seems to be no impact on the useability of these services - perhaps because related, established traffic is always allowed through? I’d be happy to block spam emails from many of these companies though, so perhaps maintaining the block isn’t so bad.

My question really was whether this was bad in relation to Lets Encrypt? As I cannot tell which of my blacklisted IP addresses belong to AWS and which do not, I’m still happy for any of these blacklisted IP addresses to access port 80 when the Lets Encrypt dedicated port is opened, to allow specific access to the /var/www/html/.freepbx-known/ directory only. Black hats can poke around in there all they like and find nothing of value.

My question to you is whether or not this is a feature you might consider including in the next firewall update, or if there is a workaround you might suggest so I can implement it myself?

Many thanks again for your time.

Firewall blocks everything by default. So I’m not really sure what you are doing here. If you don’t want people to access SSH from the outside world, do not allow port 22 from ‘Internet’.

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.