The issue with let's encrypt certificate updating

Having been through three Let’s Encrypt setups in the last week, here’s what we found was required:

  1. A valid DNS entry for the host name (resolves to the public IP used by the FreePBX install).
  2. Port 80 forwarded (PAT/NAT) through the router to the FreePBX system.
  3. Port 80 open to FreePBX on the site firewall.
  4. The FreePBX firewall OFF. The integrated mechanism to whitelist the four hosts in the FreePBX firewall isn’t adequate as the requests don’t come just from those four hosts.
  5. In at least one case, fwconsole chown to fix perms issues
1 Like

If I am using Admin port 8080 instead of the default 80 does this mean I open 8080 for LE or do I forward external 80 to internal 8080?

image

There is no need to disable the Firewall, and to suggest such is irresponsible.

1 Like

I’m reporting my experience on the last three setups, Lorne: The initial cert request failed until I disabled it. I re-enabled as soon as the cert was provided.

In each case we didn’t change anything else. Disable firewall in Connectivity, request LetsEncrypt cert, receive cert, enable firewall.

I’m not trying to be irresponsible. I’m trying to save others the hours of hair-ripping we’ve had on this in the last week or so. If others can make it work with the firewall enabled, that’s great.

I haven’t actually tried this, just trying to get some input here on whether or not this would be viable. Couldn’t we just firewall out port 80 to only allow anonymous requests to /.well-known/acme-challenge? Something like this…

/sbin/iptables -I INPUT -p tcp --dport 80 -m string --string “GET /.well-known/acme-challenge” --algo kmp -j ALLOW

This is completely untested, and is result of some googling to see if pattern matching against the http header is something iptables is capable of. I just wanted to get some feedback, on whether or not this would work, and whether or not allowing public access to ‘/.well-known/acme-challenge’ to the world is a bad idea.

I’m thinking that v15 has that done by default, based on the snip below. Maybe Lorne can confirm.

Sure looks that way, huh. I recently provisioned 3 FreePBX’s. The first 2 had no problems with the LE cert acquisition. The last one I did was stuck until I disabled the firewall. All systems were installed via the most recent ISO, and all have current/updated modules.

The only thing I can find that’s related to LE in iptables-save is an empty chain called ‘fpbxsvc-letsencrypt’, and the IP exceptions for the 2 LE hosts. I’m wondering if that ‘fpbxsvc-letsencrypt’ chain is updated on the fly when a cert request is made. I used ‘watch’ to dump the output of iptables-save to a text file every second, and ran a new cert request, but couldn’t locate any changes.

I’m starting to wonder if there isn’t an issue with the responsive firewall at play here. I removed my remote IP from the whitelist in it and now I’m locked out of the admin panel, and the phone I have sitting here next to me won’t register.

The admin panel block I can see, but the phone should register (it was working before I removed the whitelist entry).

I’ll be able to get back in later this eve.

After removing the entry for our public remote IP from the responsive firewall trusted list I can’t access the web gui or telnet from inside the office, so either the firewall is haywire or the web service is dead.

I’ll have to go in Monday to sort it out.

To anyone facing the following error message.

VerificationTimedOut

.

Below is a tested and surefire way to always allow Let’s Encrypt certificate verification without exposing the PBX administration web page to the Internet.

a) Change the Admin Insecure (http) Port to something other than 80 (such as 85). Then set the Let’s Encrypt Insecure (http) Port to 80.

.

b) If you have the recommended Let’s Encrypt Networks defined in your Firewall, delete them. They are now useless because of Let’s Encrypt’s change to Multi-Perspective Validation.

.

c) Now that the Let’s Encrypt Port is set in Port Management, you will be able to set the Let’s Encrypt Service to Internet, Local, and Other.

.

d) Validation will now succeed.

UpdatedCertificate

So if we’re not using the internal firewall are we now forced to manually open port 80 every time we want to renew our certificates?

If you open port 80 only to Let’s Encrypt, there’s no reason not to leave port 80 open permanently.

There was a discussion about this 4-6 months ago. IIRC, the Integrated Firewall knows to open port 80 for LE Updates.

No. The PBX Firewall can be configured to allow inbound access to port 80 from world, but it will not do it automatically***. There was some recent speculation about how this would be a nice feature, have the cert renewal hook the firewall rules prior to renewal, but I don’t believe a feature request was ever made.

*** 2020-06-24 - this was true at the time of writing, but now firewall rules are opened automatically during LE cert creation/renewal.

1 Like

With respect, given any knowledge of letsencrypt’s published methods of validation also

Would not a needed ‘feature request’ have been obvious and self generated by the devs a year or so ago ?

I am attempting to change the Admin to Port 85 but when I click update now the gear in the top right just spins and the update doesnt happen. I am having this exact problem with the error for updating the certificate.

Thanks
Phil

Possibly you have pending system updates: Recent update to sysadmin rpm

Ok so updates must have been an issue. I thought I had done them all last night however there were more today. So I am now able to get the Admin port changed to 85 and the LetsEncrypt port to 80 however I still get the “There was an error updated the certificate: Verification timed out” message. I made sure the lets encrypt networks were not in the PBX Firewall however I do have them noted in my pFSense firewall to pass port 80 to the pbx server.

Things I’ve now done three times to work around around this very issue in the last week:

  1. fwconsole chown
  2. TEMPORARILY disable the FreePBX firewall. Standard warnings and noting that Lorne has said this should not be necessary.
  3. Reboot the system.

In two cases I could load the challenge file remotely via the URL but the request still failed until I rebooted the system.

I wish I had more insight into what’s actually going on, but I don’t; we’re doing these installs on fixed-cost basis so we have to be as efficient as possible.

Confirm that your pfSense passes port 80 with no restrictions on source IP.