The issue with let's encrypt certificate updating


#24

I’m reporting my experience on the last three setups, Lorne: The initial cert request failed until I disabled it. I re-enabled as soon as the cert was provided.

In each case we didn’t change anything else. Disable firewall in Connectivity, request LetsEncrypt cert, receive cert, enable firewall.

I’m not trying to be irresponsible. I’m trying to save others the hours of hair-ripping we’ve had on this in the last week or so. If others can make it work with the firewall enabled, that’s great.


#25

I haven’t actually tried this, just trying to get some input here on whether or not this would be viable. Couldn’t we just firewall out port 80 to only allow anonymous requests to /.well-known/acme-challenge? Something like this…

/sbin/iptables -I INPUT -p tcp --dport 80 -m string --string “GET /.well-known/acme-challenge” --algo kmp -j ALLOW

This is completely untested, and is result of some googling to see if pattern matching against the http header is something iptables is capable of. I just wanted to get some feedback, on whether or not this would work, and whether or not allowing public access to ‘/.well-known/acme-challenge’ to the world is a bad idea.


#26

I’m thinking that v15 has that done by default, based on the snip below. Maybe Lorne can confirm.


#27

Sure looks that way, huh. I recently provisioned 3 FreePBX’s. The first 2 had no problems with the LE cert acquisition. The last one I did was stuck until I disabled the firewall. All systems were installed via the most recent ISO, and all have current/updated modules.

The only thing I can find that’s related to LE in iptables-save is an empty chain called ‘fpbxsvc-letsencrypt’, and the IP exceptions for the 2 LE hosts. I’m wondering if that ‘fpbxsvc-letsencrypt’ chain is updated on the fly when a cert request is made. I used ‘watch’ to dump the output of iptables-save to a text file every second, and ran a new cert request, but couldn’t locate any changes.


#28

I’m starting to wonder if there isn’t an issue with the responsive firewall at play here. I removed my remote IP from the whitelist in it and now I’m locked out of the admin panel, and the phone I have sitting here next to me won’t register.

The admin panel block I can see, but the phone should register (it was working before I removed the whitelist entry).

I’ll be able to get back in later this eve.


#29

After removing the entry for our public remote IP from the responsive firewall trusted list I can’t access the web gui or telnet from inside the office, so either the firewall is haywire or the web service is dead.

I’ll have to go in Monday to sort it out.


(Jean Sebastien Carle) #30

To anyone facing the following error message.

VerificationTimedOut

.

Below is a tested and surefire way to always allow Let’s Encrypt certificate verification without exposing the PBX administration web page to the Internet.

a) Change the Admin Insecure (http) Port to something other than 80 (such as 85). Then set the Let’s Encrypt Insecure (http) Port to 80.

.

b) If you have the recommended Let’s Encrypt Networks defined in your Firewall, delete them. They are now useless because of Let’s Encrypt’s change to Multi-Perspective Validation.

.

c) Now that the Let’s Encrypt Port is set in Port Management, you will be able to set the Let’s Encrypt Service to Internet, Local, and Other.

.

d) Validation will now succeed.

UpdatedCertificate


(Mvogel4949) #31

So if we’re not using the internal firewall are we now forced to manually open port 80 every time we want to renew our certificates?


(Jean Sebastien Carle) #32

If you open port 80 only to Let’s Encrypt, there’s no reason not to leave port 80 open permanently.


(Dave Burgess) #33

There was a discussion about this 4-6 months ago. IIRC, the Integrated Firewall knows to open port 80 for LE Updates.


(Lorne Gaetz) #34

No. The PBX Firewall can be configured to allow inbound access to port 80 from world, but it will not do it automatically***. There was some recent speculation about how this would be a nice feature, have the cert renewal hook the firewall rules prior to renewal, but I don’t believe a feature request was ever made.

*** 2020-06-24 - this was true at the time of writing, but now firewall rules are opened automatically during LE cert creation/renewal.


#35

With respect, given any knowledge of letsencrypt’s published methods of validation also

Would not a needed ‘feature request’ have been obvious and self generated by the devs a year or so ago ?


(sBITS) #36

I am attempting to change the Admin to Port 85 but when I click update now the gear in the top right just spins and the update doesnt happen. I am having this exact problem with the error for updating the certificate.

Thanks
Phil


(Lorne Gaetz) #37

Possibly you have pending system updates: Recent update to sysadmin rpm


(sBITS) #38

Ok so updates must have been an issue. I thought I had done them all last night however there were more today. So I am now able to get the Admin port changed to 85 and the LetsEncrypt port to 80 however I still get the “There was an error updated the certificate: Verification timed out” message. I made sure the lets encrypt networks were not in the PBX Firewall however I do have them noted in my pFSense firewall to pass port 80 to the pbx server.


#39

Things I’ve now done three times to work around around this very issue in the last week:

  1. fwconsole chown
  2. TEMPORARILY disable the FreePBX firewall. Standard warnings and noting that Lorne has said this should not be necessary.
  3. Reboot the system.

In two cases I could load the challenge file remotely via the URL but the request still failed until I rebooted the system.

I wish I had more insight into what’s actually going on, but I don’t; we’re doing these installs on fixed-cost basis so we have to be as efficient as possible.


#40

Confirm that your pfSense passes port 80 with no restrictions on source IP.


(sBITS) #41

Well using a port checker from an outside address port 80 does not show as open. Are you familiar with pfsense? Below is my NAT Rule and the destination is set to the correct IP address.


#42

LE changed their policy and now send verification from random IP addresses that are not publicized in advance. I don’t know pfSense but you need to change the Source field of your redirect to allow any IPv4 address.


#43

Technically, LetsEncrypt has never changed their policy at least as of 2017, they just chose to enforce it as of February 19th 2020 :slight_smile:

Further, Acme v1 is deprecated