Let's encrypt root CA cert expiring on October 1st, 2021

While helpful, there is a 2 month lead time on LE certs. So everything will still puke on my cert until I renew, after applying this update.

I stand by my first post in this thread.

@sorvani I don’t think that’s how it works. All LE certs are dual signed for the OLD and NEW CA. The problem is CentOS 7 made the decision that the both CAs must validate a certificate to consider it valid. The patch, simply removes the OLD cert from the validation so that it will let the NEW CA do its job. So no LE renewal is necessary. This is just a client issue (where FreePBX is the client) connecting externally to other LE secured servers. People connecting to a FreePBX instance that is secured with LE certificate should have no issue, even if that server is not patched. At least, no issue until the LE certificate that is securing the instance fails to automatically renew, which like you said, could be anytime within a 2 month period.

1 Like

Do you think it will work if we upgrade the Certs package as described here? or do we need a different patch for FreePBX itself? https://www.queuemetrics.com/blog/2021/09/29/lets-encrypt-tutorial/

If you’re running CentOS 7 that will possibly work. I’m glad to see CentOS finally released an update, looks like we were a little bit ahead of them. For SNG7, we’ll have to release a new rpm based on this.

1 Like

So when you do it will be enough upgrading the ca-certs package?

@lynx87 it looks like we got the yum package upgraded last night, so you should ALSO be able to run ‘yum upgrade ca-certificates’ to resolve this issue.

1 Like

Deliberately left my home prod server on the old certman module version without this fix to see what would happen today. Haven’t tested TLS SIP transport yet but:

:white_check_mark: browsing to GUI with https - no issues (as expected)
:white_check_mark: LE cert renewal using module - no issues

2 Likes

If something went wrong for you, don’t feel bad. Some big players messed up too…

Browser support was NEVER a concern

The root CA expiring on October 1 affected TLS with Polycom VVX 411s. Other PBXes recommended modifying the certificate chain, for example, removing step 2 below:

Certificate chain
 0 s:/CN=tel.whitestone.link
   i:/C=US/O=Let's Encrypt/CN=R3
 1 s:/C=US/O=Let's Encrypt/CN=R3
   i:/C=US/O=Internet Security Research Group/CN=ISRG Root X1
 2 s:/C=US/O=Internet Security Research Group/CN=ISRG Root X1
   i:/O=Digital Signature Trust Co./CN=DST Root CA X3

My guess is that the Polycoms are experiencing the same problem @mbrooks mentioned that both CAs must be valid. We had no issues with the system module updates and /etc/pki/ca-trust/source/blacklist/DST-Root-CA-X3.pem exists.

After beating our heads against the wall for a while, we changed the Polycoms to use a self-signed certificate for TLS.

Is it possible for certbot (we’re at v15.0.47) to generate certs that no longer include X3 in the chain? This appears to be the root problem from a best-practices perspective, irrespective of client settings.

-rw-------  1 asterisk asterisk 2203 Oct  2 12:05 cert.pem
-rw-------  1 asterisk asterisk 3752 Oct  2 12:05 chain.pem
-rw-------  1 asterisk asterisk 5956 Oct  2 12:05 fullchain.pem
-rw-------  1 asterisk asterisk 1765 Oct  2 12:05 last.csr
-rw-------  1 asterisk asterisk 3272 Oct  2 11:28 private.pem
-rw-------  1 asterisk asterisk  800 Oct  2 11:28 public.pem

chain.pem and fullchain.pem still reference X3:
keytool -printcert -file fullchain.pem
Certificate[3]:
Owner: CN=ISRG Root X1, O=Internet Security Research Group, C=US
Issuer: CN=DST Root CA X3, O=Digital Signature Trust Co.
Serial number: 4001772137d4e942b8ee76aa3c640ab7
Valid from: Wed Jan 20 10:14:03 AKST 2021 until: Mon Sep 30 10:14:03 AKDT 2024
Certificate fingerprints:

Signature algorithm name: SHA256withRSA
Subject Public Key Algorithm: 4096-bit RSA key
Version: 3

It seems odd that Freepbx is still generating certificates via certbot with an expired certificate in the chain, if I understand things correctly. (I regenerated the letsencrypt cert 2x to be sure.)

I don’t think that the FreePBX acme client is ‘certbot’.

Thanks for sharing that info. Can you tell us what Polycom firmware version those phones are running?

Correct, it uses lescript, last updated 2 years ago.

You will note the project hasn’t been touched since that update

Off topic but interesting , Neil Pang’s acme.sh now uses certs from “ZeroSSL” by default for newly issued domains, (just like the one used on this forum) as an alternative to LetsEncrypt , it doesn’t have X3 in its chain and so wouldn’t be prone to old devices like android => 4 or IE >= 11 ‘breaking’

Fully updated yet unable to update the Let’s Encrypt cert.
FreePBX 15.0.17.55
Certificate Manager 15.0.47
Certificate Valid Until 2021-10-04 Certificate Expired! (-1 days)

Update Certificate result: Nothing to do, no changes made

@fastdrw Please run the following to see if it resolves your issue:

yum update ca-certificates

I have already run - yum update ca-certificates prior to posting.

To further add to this and @wsfadmin comment above.
I checked my certificates and he is indeed correct. The 3rd certificate added/signed with is the expired DST Root CA X3. It does seem that my Yealink’s authenticating via TLS still verify against this just fine. They must use the logic of ANY valid CA = pass. Whereas the Polycom phones use the ALL VALID = pass logic. I manually deleted the 3rd certificate block off of my [FQDN].com-fullchain.crt file and boom, Polycom registers via TLS now. So whatever script / application manages and creates those certificates needs to remove that expired CA off the chain. I’m not sure how even new certificates are being generated with it expired now. Hopefully this little hiccup can be resolved before the next batch of certificates renews.

I’m really not sure what is going on here, but I don’t think this is related to the LE CA cert expiring. I do believe this is an issue with the Let’s Encrypt renewal mechanism, which is something different and shouldn’t be affected by this. The Let’s Encrypt renewal mechanism could fail for a number of reasons.

Are you by chance running SNG7 (The FreePBX Distro)? Or are you running something else like CentOS or Debian?

@wsfadmin and @MrXirtam, so Let’s Encrypt is saying that they are still distributing the expired DST-Root-CA-X3 CA in the bundle to maintain compatibility with older clients that do not expire root ca certs. So, we’ve added a feature where you can remove the DST-Root-CA-X3 CA from the bundle that Let’ Encrypt returns. My assumption here is that removing this may fix some older clients but break others, so just be aware of the trade-offs.

For now, we’re going to just return what Let’s Encrypt gives us by default, so you’ll have to check that ‘Remove DST Root CA X3’ box if you want to remove it from the CA Bundle.

This is now available in edge:
Certman v14.0.20
Certman v15.0.48
Certman v16.0.19

You can install the module in edge by running the following CLI command or wait on GA:

fwconsole ma --edge upgrade certman

1 Like