Let's Encrypt certificate error Pest_NotFound - 404: Nothing to do

FreePBX 15.0.17.24 / all modules up to date. I’m running pfSense 2.5.0

I’m getting a critical dialog in the FreePBX Dashboard that says:

Security Issue: Some certificates are expiring or have expired. This is a critical issue and should be resolved urgently.

There was an error updating certificate “www.fqdn.com”: Self test error: Pest_NotFound - 404 Not Found
404 Not Found
nginx
There was an error updating certificate “www.anotherfqdn.com”: Self test error: Pest_NotFound - 404 Not Found
404 Not Found
nginx
Resolve

Port 80 (TCP) is forwarded to FreePBX temporarily on pfSense. In the Certificate Management module in the FreePBX GUI, there are three certificates two of which are Let’s Encrypt, and one of those is the Default certificate. I can ping www.fqdn.com:

$ ping www.fqdn.com
PING www.fqdn.com (11.22.15.72): 56 data bytes
64 bytes from 11.22.15.72: icmp_seq=0 ttl=64 time=0.077 ms

The details of the Default LE cert are:

Certificate Host name: www.fqdn.com
Owner’s Email: [email protected]
Country:XYX
State:ABC
Challenge Over:HTTP (Port 80)
Certificate Common Name:www.fqdn.com
Certificate Alternative Names:DNS:www.fqdn.com
Certificate Valid Until:2021-04-13 (19 days)
Certificate Policies:
Policy: 2.23.140.1.2.1
Policy: 1.3.6.1.4.1.44947.1.1.1
CPS: http://cps.letsencrypt.org

The details of the second LE certificate are similar.

When I update the two Let’s Encrypt certificates, the response in the GUI for both is “Nothing to do”.

How can I resolve this issue? Thanks.

I think you will find that the module assumes apache2 (httpd in RH speak) as your web server

Most likely a DNS issue.

From the pbx, “www.fqdn.com” must resolve to the internal IP of the pbx. .

During the self test, certificate manager tries to download a locally hosted file using the fqdn. The separate self test was added because the actual Lescript.php library does the same and would fail even more cryptically.

The fact it’s hitting an nginx server makes me think the fqdn resolves to another internal address or possibly the pfsense external IP and the temp rule is clashing with another rule that has nat loopback enabled.

@dicko I’m using a stock FreePBX 15 and haven’t modified the web server. So I presume nginx is the Certificate Manager module in the FreePBX GUI, not apache2.

Then it is a DNS issue. Fix internal resolution for the fqdn, or add it to the pbx hosts file.

I have to admit to being clueless with certificates in FreePBX and pfSense. In the Certificate Manager module in the FreePBX GUI, it states:

Let’s Encrypt certificate creation and validation requires unrestricted inbound http access on port 80 to the Let’s Encrypt token directories

If security is managed by the PBX Firewall module, this process should be automatic. Alternate security methods and external firewalls will require manual configuration.

In pfSense I have installed ACME (i.e., automated cert management environment for automated use of LE certs). Is ACME conflicting with the certificate management in FreePBX? If so, is there a way to temporarily disable ACME in pfSense without un-installing it so the certificates in FreePBX can be updated?

If your acme client is working, just copy the *.crt and *.key (rename if needed) into /etc/asterisk/keys hopefully with a client post hook then import/updateall/setdefault either in the gui or using fwconsole. Make sure your webserver is referencing these certs

That way you are not unnecessarily opening up your firewall or worrying about the FreePBX acme client.

No clue how greedy the pfsense acme client is.

Easiest fix to get past the self test error is adding the fqdn(s) to the pbx hosts file pointing to localhost. Then see how far it gets.

To renew the certs from the cli, use:

fwconsole cert --udpdateall --force

Here’s the reply I get from that command:

sudo fwconsole certificates --updateall --force
Forced update enabled !!!
Processing: www.fqdn.com, Local IP: 11.22.15.72, Public IP: 11.22.15.72
Self test: trying http://www.fqdn.com/.freepbx-known/dfb800eaf037f0d1c4bb4d882a469c2b
Self test error: Pest_NotFound - 404 Not Found

404 Not Found
nginx
Processing: www.anotherfqdn.com, Local IP: 11.22.15.72, Public IP: 11.22.15.72
Self test: trying http://www.anotherfqdn.com/.freepbx-known/0e31820c50e4dbfc265270e56a752979
Self test error: Pest_NotFound - 404 Not Found

404 Not Found
nginx

** Does DNS name resolution for www.fqdn.com resolve correctly?
Local DNS result: 11.22.15.72, External DNS result: 11.22.15.72

** Does DNS name resolution for www.anotherfqdn.com resolve correctly?
Local DNS result: 11.22.15.72, External DNS result: 11.22.15.72

Certificate named “default” is going to expire in less than a month. Please update this certificate in Certificate Manager
There was an error updating certificate “www.fqdn.com”: Self test error: Pest_NotFound - 404 Not Found

404 Not Found
nginx
There was an error updating certificate “www.anotherfqdn.com”: Self test error: Pest_NotFound - 404 Not Found

404 Not Found
nginx
$

I’m assuming that is actually your external IP, and the internal is something like 10.x.x.x or 192.168.x.x.

You have few options:

  • Properly configure internal DNS to point to the PBX Internal IP.
  • Add the fqdn(s) to the PBX hosts file pointing to either localhost, or the PBX Internal IP.
  • Configure NAT Loopback on the pfsense box (I think it’s “NAT Reflection” in pfsense terminology).
  • Wait and hope I finish my refactor of certman backed by acme.sh before the certs expire (I wouldn’t take that bet).
  • Use the pfsense box to renew the certs and import them into FreePBX as @dicko suggested.
  • Use another LetsEncrypt client (acme.sh, certbot, lego, etc) on the PBX and import the certs into FreePBX.

Let’s see if I understand what you recommend. In pfSense, as user root and starting from “/”, I look for either a .crt file or a .key file:

# find . -type f ( -name “.key" -o -name ".crt” ) -exec ls -l {} ;
-rw-r–r-- 1 root wheel 786648 Feb 5 12:11 ./usr/local/share/certs/ca-root-nss.crt
-rw-r–r-- 1 unbound unbound 758 Mar 25 00:31 ./var/unbound/root.key
-rw-r----- 1 unbound unbound 2459 Aug 24 2020 ./var/unbound/unbound_server.key
-rw-r----- 1 unbound unbound 2459 Aug 24 2020 ./var/unbound/unbound_control.key
-rw-r–r-- 1 root wheel 3294 Mar 25 07:05 ./tmp/acme/pfblockng/ca/acme-staging-v02.api.letsencrypt.org/account.key
-rw-r–r-- 1 root wheel 1675 Mar 25 07:05 ./tmp/acme/pfblockng/www.fqdn.com/www.fqdn.com.key
#

There is only one candidate for .crt, that is ca-root-nss.crt. I presume the other candidate is www.fqdn.com.key.

You suggest I should copy those two files into /etc/asterisk/keys on the FreePBX server. Here’s what that directory looks like:

$ cd /etc/asterisk/keys
$ ls -l
total 80
drwxr-xr-x 2 asterisk asterisk 43 Dec 25 22:04 _account
-rw-------. 1 asterisk asterisk 1675 Nov 25 06:42 api_oauth.key
-rw-------. 1 asterisk asterisk 451 Nov 25 06:42 api_oauth_public.key
-rw-rw-r-- 1 asterisk asterisk 187 Dec 25 22:04 ca.cfg
-rw-rw-r-- 1 asterisk asterisk 1712 Dec 25 22:04 ca.crt
-rw-rw-r-- 1 asterisk asterisk 3243 Dec 25 22:04 ca.key
-rw------- 1 asterisk asterisk 1176 Jan 15 19:53 default.crt
-rw------- 1 asterisk asterisk 558 Dec 25 22:04 default.csr
-rw------- 1 asterisk asterisk 887 Jan 15 19:53 default.key
-rw------- 1 asterisk asterisk 2064 Jan 15 19:53 default.pem
drwxrwxr-x. 2 asterisk asterisk 71 Jan 13 16:28 integration
drwxr-xr-x 2 asterisk asterisk 113 Jan 15 22:23 anotherfqdn.com
-rw------- 1 asterisk asterisk 2789 Jan 15 22:23 anotherfqdn.com-ca-bundle.crt
-rw------- 1 asterisk asterisk 2198 Jan 15 22:23 anotherfqdn.com.crt
-rw------- 1 asterisk asterisk 3272 Jan 15 22:23 anotherfqdn.com.key
-rw------- 1 asterisk asterisk 8261 Jan 15 22:23 anotherfqdn.com.pem
drwxr-xr-x 2 asterisk asterisk 113 Jan 13 16:28 www.fqdn.com
-rw------- 1 asterisk asterisk 2789 Jan 13 16:28 www.fqdn.com-ca-bundle.crt
-rw------- 1 asterisk asterisk 2198 Jan 15 19:53 www.fqdn.com.crt
-rw------- 1 asterisk asterisk 3272 Jan 15 19:53 www.fqdn.com.key
-rw------- 1 asterisk asterisk 5471 Jan 15 19:53 www.fqdn.com.pem
$

If I understand correctly, I should copy ca-root-nss.crt from pfSense to www.fqdn.com.crt and www.fqdn.com.key to www.fqdn.com.key.

not sure how pfSense acme client works but the crt might well be called .ert or .pem it is likely in the same directory as ./tmp/acme/pfblockng/www.fqdn.com/www.fqdn.com.key and not the root cert

There must be other people running FreePBX with pfSense with ACME installed. Do they have to go through this just to update a certificate? Sorry to whine, but am I missing something or is this process overly convoluted? Is there no other way?

ACME is the protocol, correctly you should be using acme client most acme clients set up a cron job or a systemd timer to automatically renew certs FreePBX will automatically check for expred certs in /etc/asterisk/keys if your cron job updates your FPBX cert files in a timely fashion you are all done

https://docs.netgate.com/pfsense/en/latest/packages/acme/index.html

Cron Entry

A checkbox which enables the ACME renewal cron job. When set, the ACME package will check all certificates each night and if any are up for renewal, it will attempt to renew them.

Write Certificates

When set, the ACME package will write the certificate files out in  `/conf/acme` . From there, other scripts or processes which do not support GUI integration can pick up the certificate.

Here is what that directory looks like:

# cd /tmp/acme/pfblockng/www.fqdn.com/
# ls -l
total 16
-rw-r–r-- 1 root wheel 293 Mar 25 07:05 www.fqdn.com.conf
-rw-r–r-- 1 root wheel 985 Mar 25 07:05 www.fqdn.com.csr
-rw-r–r-- 1 root wheel 213 Mar 25 07:05 www.fqdn.com.csr.conf
-rw-r–r-- 1 root wheel 1675 Mar 25 07:05 www.fqdn.com.key
#

Maybe RTFM and set the write thingy ? :slight_smile:

The ‘hook’ I mentioned is called the ‘action list’

@dicko Thanks.
In Services==>Acme==>Settings in the pfSense GUI, I have the Cron Entry enabled, but not the Write Certificates. I will enable Write Certificates.

At the moment, with Write Certificates enabled, the /conf/acme directory is empty. The blurb next to Cron Entry inServices==>Acme==>Settings==>General Settings says:

Enable Acme client renewal job. This will configure cron to renew certificates once a day at 3:16. Keeping track of the last successful renewal and the number of days set after to renew again. When renewal happens a service can be restarted or a shell script run to load the new certificate for services that need it, if needed this needs to be configured as a action under the certificate settings.

I’ll wait until tomorrow (i.e., after 3:16am) and check if anything in /conf/acme has changed. [It’s pretty obvious that I don’t know WTF I’m doing here…]

If that doesn’t happen, there is a 5/week rate limit on ssuances/renew per domain, so it would be safe to delete what you have and start over. Then set up the ‘action list’ to push the key and cert to /etc/asterisk/keys on your FreePBX.

If DNS and firewall forwarding/loopback are properly configured, it should “just work.”

If your sharing a single external IP and need multiple services listening on a single port, it’s on you to have proper a proxy/forward setup. It’s not a beginner config, but shouldn’t be overly difficult either.

If the pfsense acme client listening on port 80 is the root of your problem, remove the conflict. Setup the pfsense acme client to use tls-alpn auth(port 443) or dns-01auth (no port needed) and keep port 80 forwarded exclusively to the PBX.

For now, you have to make the clients play nice together when sharing a single IP.

As long as both clients use http-01 auth, managing port 80 will be an issue. Changing the pfsense auth method should be an easy workaround.

The “self test” issue is a problem with the current acme library used by FreePBX. If/when the FreePBX backend is ported to acme.sh, that specific problem should go away, but “sharing” port 80 will remain an issue.

If the FreePBX backend is migrated to acme.sh, dns-01 auth (no port needed) will be an option for FreePBX. I doubt tls-alpn auth (port 443) will be a FreePBX option. No reason tls-alpn couldn’t be made available, but it really needs SysAdmin(closed source) changes for the distro.

On that line, pfSense has grown up a lot since I last used it, I see they have an HAProxy package if you need/want multiple sites behind your Router and all of them to ‘play nice’

If you do that, you can do all the certs for all the sites on the pfSense box rewriting 80 to 443 (or anything else) on the front-ends and just forward to your various relevant backends without TLS being needed. ( Add strict SNI and you are pretty rock solid)