LetsEncrypt update error

My LetsEncrypt cert tried to update last night and sent me this error

SECURITY NOTICE:

Some Certificates are expiring or have expired:
 There was an error updating certificate "pbx.xxxxxx.com": Error
'Requested
'http://pbx.xxxxxx.com//.freepbx-known/7251bf30b7ecfe7885a1c9259c760a74'
- Failed connect to pbx.xxxxxx.com:80; Connection timed out' when
requesting
http://pbx.xxxxxx.com//.freepbx-known/7251bf30b7ecfe7885a1c9259c760a74</br>

I am using System Admin version 14.0.12.2, which includes the LetsEncrypt port 80 setting. I also have port 80 forwarded in my router.

I attempted to disable the LetsEncrypt port in System Admin and change my UCP and my admin port to 80 (at separate times) but when the page reloaded no changes were made.

I have my firewall correctly set up as well, according to the certificate management page.

Why dont you first check if you can connect to port 80 yourself. Doesn’t seem like you even tried.

Yep, didn’t even think about that. I can not connect to port 80 when my FreePBX firewall is enabled. But if I disable it, I can. I used http://canyouseeme.org/ to check.

I disabled my firewall and successfully updated my cert. But I’d like to find the root cause. According to my system, I have my firewall rules set up correctly.

Firewall Validated
LetsEncrypt requires the following hosts to be permitted for inbound http access:
outbound1.letsencrypt.org, outbound2.letsencrypt.org, mirror1.freepbx.org, mirror2.freepbx.org

These entries are correctly set up in the Firewall module. However, it's possible that other external firewalls may block access. If you are having problems validating your certificate, this could be the issue.

So it’s firewall related then I would say. Hard to tell now that you updated.

Sounds like your “other” firewall needs to be open to that port. I’d set it up with a redirect from port 80 on those four names to port 80 on your PBX and see what happens.

1 Like

I’m sure I’ll have the problem again in two months, I can revisit it then.

Cynjut, I disabled the FreePBX firewall to get the cert to update, so presumably my router firewall isn’t causing any issues, right? Would it help to put my PBX in the DMZ on my router?

The way I understand it - the Integrated Firewall knows about the cert sites and should allow them in. You can “belt and suspender” that by adding those four sites to the Sysadmin whiltelkist and to the firewall.

Your PBX is non-routable, so your external firewall should (correctly) block SYN Access to port 80, or at least answer on port 80 that you aren’t authorized to access the network. That would be a common and “best practice” method. Remember, the cert request isn’t coming from port 80 (IIRC), so NAT will not have opened to that port from the cert host.

Moving the PBX to your DMZ is dangerous, because it exposes the system to more scrutiny. Leaving it in the secure enclave would be my recommendation. Open the ports from certain hosts to certain port (In this case, the four cert hosts, port 80) and redirect those bi-directionally to port 80 on the PBX. Then, in the integrated firewall, you can just list the hosts in the Trusted Zone, knowing that the only SYN traffic coming to your PBX from there will be port 80 stuff.

I’m not sure why turning the firewall off on the PBX would have made any difference. From what we understand, it shouldn’t have.

Thanks, Cynjut.

I’ve added the LetsEncrypt addresses to my systemadmin whitelist to be safe. Below are the addresses I have trusted in my firewall. I assume these are accurate, correct? Being that disabling the firewall allowed the cert to renew tells me that the issue is in my firewall settings, correct?

1 Like

Maybe try setting it to “Trusted - Excluded from Firewall”

I’ll give that a shot for the next renewal in two months. I used the “wizard” to set my firewall settings when first installing the LetsEncrypt cert and it set it to “Local”.

Hopefully setting it to “Excluded” will do the trick.

Thanks!

Hello,

I too use LetsEncrypt (LE) to provide trusted connections, however my test environment located at my home office has port 80 blocked by the ISP, although for extra $$ I could get that removed.

LE has has another option of renewing the certificate, which is to edit the DNS record with some text to prove you are making a legitimate request. Thus, if you have access to your domain records, and can make a quick edit, you could use this second method to renew your certs.

Personally, I would drop the firewall on renewal day, renew the cert, and then bring them back up. I also wish that LE would relax the tight window, and let the certs live for 6 months or a year at a crack.

I myself have done the same thing. When I get the alert advising it could not renew, I disable the FreePBX firewall, run the cert renewal via the FreePBX portal and then enable the firewall afterwards.

Seems like there is a bug in the Certificate Management in setting up the firewall to permit the LetsEncrypt renewals.

Certificate manager doesn’t talk to firewall. So the bug would be in firewall itself

Can you advise what the wizard in the Certificate Manager does? I did not have the rules in the firewall before running the wizard for LetsEncrypt in Certificate Manager.

Perhaps as part of the Certificate Manager process for auto-renewing the LetsEncypt certificate it can open up port 80 for that process?

That’s what sysadmin already does.

Is this something new? There are over a dozen systems I manage that every 3 months I am disabling firewall inside freepbx and then running the certificate manager update on the letsencrypt cert so it would update properly.

Each time I take the opportunity to also update the modules before doing this to see if a module update would correct this to no avail.

It’s new as of a month ago. Assigns let’s encrypt only to port 80 so you can put everything on other ports

As for firewall just whitelist the domains let’s encrypt comes from. The problem is sounds like you are facing is either you need to whitelist them or they aren’t hitting you from outbound.theirdomain

Chiming in here - I just got a new Sangoma PBX with the latest firmware. I have the Lets Encrypt Port set to port 80, have my firewall port forwarding correctly (I am able to connect to it externally from whitelisted IPs found on both my firewall and the PBX firewall), had the firewall entries for Let’s Encrypt and could not get the certificate to work without disabling my firewall on the PBX first.

The issue most likely with the firewall rules on the PBX for Lets Encrypt, the ones provided might not be sufficient here? How are DNS entries validated against requests incoming from various IPs? Are reverse DNS calls made to see if an IP resolves to a whitelisted entry? Let’s Encrypt documentation says their validation requests can come from a variety of IPs and there’s no set list to “whitelist”. Do we know if Let’s Encrypt even guarantees there will be a hostname tied to the requests/IPs they send? From what I’m reading, Let’s Encrypt does not recommend trying to do any sort of whitelist when using http challenges and to switch to dns challenges instead if this isn’t feasible.

Unfortunately, LetsEncrypt are no longer only originating connections from their well known IP addresses, and can (and do) now establish connections from anywhere on the internet.

To address that, we added the ‘LetsEncrypt’ service to Sysadmin, which is perfectly safe to expose to the internet on port 80, as it only provides letsencrypt and nothing else.