Let's Encrypt broken on two FreePBX servers

FreePBX Security
1

I have two public-facing (no NAT) servers each with seemingly unique problems
renewing or creating new certificates for Let’s Encrypt. Port 80 and 443 are open
to the world and using Responsive Firewall.

I tried deleting the Let’s Encrypt certificate via Certificate Manager, although it
doesn’t show in the list anymore, the FreePBX server is still using the soon to
expire certificate that I “deleted” (even after making the self-signed the default
and being the only cert in the list).

If I try to create a new certificate I get the following Error:
There was an error updating the certificate: 400 { "type": "urn:ietf:params:acme:error:malformed", "detail": "Unable to update challenge :: authorization must be pending", "status": 400 }

This is on FreePBX 14.0.13.28 and Certificate Manager 14.0.8.
I updated all modules, ran yum update and rebooted. Problem persists.

From first server access_log:

11.22.33.217 - - [03/May/2020:15:55:57 +0000] "GET /admin/ajax.php?module=search&command=global HTTP/1.1" 200 14113 "https://my.pbx1.com/admin/config.php?display=certman&action=add&type=le" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.113 Safari/537.36"
199.102.239.170 - - [03/May/2020:15:56:05 +0000] "GET /.freepbx-known/76dc5a067650f17666ca81b3116d93c1 HTTP/1.1" 200 32 "-" "-"
::1 - - [03/May/2020:15:56:37 +0000] "GET /.well-known/acme-challenge/_x8RfrLKyC4Rmk1T3iTiHmkXGgWHJGURgSfHvfHDWjA HTTP/1.0" 200 87 "-" "Wget/1.10.2 (Red Hat modified)"
64.78.149.164 - - [03/May/2020:15:56:40 +0000] "GET /.well-known/acme-challenge/_x8RfrLKyC4Rmk1T3iTiHmkXGgWHJGURgSfHvfHDWjA HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
64.78.149.164 - - [03/May/2020:15:56:44 +0000] "GET /.well-known/acme-challenge/_x8RfrLKyC4Rmk1T3iTiHmkXGgWHJGURgSfHvfHDWjA HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
66.133.109.36 - - [03/May/2020:15:56:49 +0000] "GET /.well-known/acme-challenge/_x8RfrLKyC4Rmk1T3iTiHmkXGgWHJGURgSfHvfHDWjA HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
11.22.33.217 - - [03/May/2020:15:56:05 +0000] "POST /admin/config.php?display=certman HTTP/1.1" 200 40259 "https://my.pbx1.com/admin/config.php?display=certman&action=add&type=le" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.113 Safari/537.36"
11.22.33.217 - - [03/May/2020:15:56:53 +0000] "GET /admin/assets/less/cache/lessphp_b3fab7250c25f1dff3dd25b88d72e7fe98a04a9f.css HTTP/1.1" 200 87893 "https://my.pbx1.com/admin/config.php?display=certman" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.113 Safari/537.36"
11.22.33.217 - - [03/May/2020:15:56:53 +0000] "GET /admin/assets/certman/less/cache/lessphp_c2126c9c8880c85219c33383fec832da3dcd1707.css HTTP/1.1" 200 260 "https://my.pbx1.com/admin/config.php?display=certman" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.113 Safari/537.36"
11.22.33.217 - - [03/May/2020:15:56:53 +0000] "GET /admin/ajax.php?module=search&command=global HTTP/1.1" 200 14113 "https://my.pbx1.com/admin/config.php?display=certman" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.113 Safari/537.36"

The second server is also Distro FreePBX 14.0.13.26 (not yet updated) running Certificate Manager 14.0.6.

I still have the soon to expire cert and when trying to renew I get the following:
There was an error updating the certificate: Verification timed out

From apache access_log on second server:

    11.22.33.217 - - [03/May/2020:11:41:06 -0400] "GET /admin/ajax.php?module=search&command=global HTTP/1.1" 200 14103 "http://my.pbx.com/admin/config.php?display=certman&action=view&id=1" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.113 Safari/537.36"
    ::1 - - [03/May/2020:11:41:14 -0400] "OPTIONS * HTTP/1.0" 200 - "-" "Apache/2.4.6 (Sangoma) OpenSSL/1.0.2k-fips PHP/5.6.40 (internal dummy connection)"
    199.102.239.170 - - [03/May/2020:11:41:25 -0400] "GET /.freepbx-known/b99328a7bb2299536f878611215bd0db HTTP/1.1" 200 32 "-" "-"
    ::1 - - [03/May/2020:11:41:26 -0400] "GET /.well-known/acme-challenge/OLrAjsFN6favT3rcIqo3c3DVNGxJAJ6Rp_Yo65ngeLQ HTTP/1.0" 200 87 "-" "Wget/1.10.2 (Red Hat modified)"
    66.133.109.36 - - [03/May/2020:11:41:28 -0400] "GET /.well-known/acme-challenge/OLrAjsFN6favT3rcIqo3c3DVNGxJAJ6Rp_Yo65ngeLQ HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
    66.133.109.36 - - [03/May/2020:11:41:29 -0400] "GET /.well-known/acme-challenge/OLrAjsFN6favT3rcIqo3c3DVNGxJAJ6Rp_Yo65ngeLQ HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
    66.133.109.36 - - [03/May/2020:11:41:30 -0400] "GET /.well-known/acme-challenge/OLrAjsFN6favT3rcIqo3c3DVNGxJAJ6Rp_Yo65ngeLQ HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
    64.78.149.164 - - [03/May/2020:11:41:31 -0400] "GET /.well-known/acme-challenge/OLrAjsFN6favT3rcIqo3c3DVNGxJAJ6Rp_Yo65ngeLQ HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
    64.78.149.164 - - [03/May/2020:11:41:32 -0400] "GET /.well-known/acme-challenge/OLrAjsFN6favT3rcIqo3c3DVNGxJAJ6Rp_Yo65ngeLQ HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
    11.22.33.217 - - [03/May/2020:11:41:25 -0400] "POST /admin/config.php?display=certman HTTP/1.1" 200 41010 "http://my.pbx.com/admin/config.php?display=certman&action=view&id=1" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.113 Safari/537.36"
    11.22.33.217 - - [03/May/2020:11:41:33 -0400] "GET /admin/assets/certman/less/cache/lessphp_c2126c9c8880c85219c33383fec832da3dcd1707.css HTTP/1.1" 200 260 "http://my.pbx.com/admin/config.php?display=certman" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.113 Safari/537.36"
    11.22.33.217 - - [03/May/2020:11:41:33 -0400] "GET /admin/assets/less/cache/lessphp_24bcf3c8d621d820ac16f650e93b3cb2f1544dcf.css HTTP/1.1" 200 87893 "http://my.pbx.com/admin/config.php?display=certman" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.113 Safari/537.36"
    11.22.33.217 - - [03/May/2020:11:41:34 -0400] "GET /admin/ajax.php?module=search&command=global HTTP/1.1" 200 14103 "http://my.pbx.com/admin/config.php?display=certman" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.113 Safari/537.36"
2

You don’t use Responsive to open ports 80/443. This doesn’t make sense unless you intend the above statement to convey 2 separate and unrelated facts.

Here’s a recent video showing LE creation: Open Source Pro Tips #4 - Setting Up a TLS Cert with Let's Encrypt

3

@lgaetz thanks for the fast response.

Just after posting I looked into the Firewall > Services and noticed
http and https were set to local zones only. I temporarily added “Internet zone”
and was able to get certs to renew.

So it seems all that has changed is Let’s Encrypt is using different public IP addresses
for their services, which causes the Let’s Encrypt exceptions to be no longer effective
in FreePBX.

I couldn’t understand what changed. I had always just assumed port 80 and 443 were
handled like pjsip and sip in the Responsive firewall. Lesson learned!

4

Another vid for Firewall

5

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.