Let's encrypt root CA cert expiring on October 1st, 2021

Distro Discussion & Help
1

Let’s encrypt is a Certificate Authority that provides SSL/TLS certificates to 260 million websites. On Oct 1st, an older root CA used by Let’s encrypt is expiring and will no longer validate websites signed by Let’s encrypt.

SNG7, the most current FreePBX distro, ships with the impacted older Let’s encrypt root CA cert, so on Oct 1st, web requests that are accessed internally from FreePBX that use Let’s encrypt to sign their SSL/TLS certificate will quit working due to certificate errors.

So what does this mean? Well for most people, it doesn’t mean anything. FreePBX should still be up and continue to function properly, however, if you use Let’s encrypt to sign the SSL/TLS certificate on your FreePBX instance, the mechanism that keeps the certificate automatically up-to-date will stop functioning and eventually the certificate that is used to secure your FreePBX instance will expire. It should ALSO be a concern if FreePBX is acting as a client that is connecting to some other server that uses a Let’s encrypt cert. Possible servers could include but not limited to HTTPS, FTP, mail, or potentially an ITSP using a Let’s encrypt certificate.

That being said, to minimize any possible disruption, we are recommending that everyone patch their system to prepare for this event. To fix this issue, we have released patches to the certman module to fix that problem by removing the old Let’s encrypt root CA. We currently recommend that you update the certman module to one of the following:

Certman v14.0.19
Certman v15.0.47
Certman v16.0.17

The issue can ALSO be fixed via SNG7 by updating to the latest version of ca-certificates via yum:

yum update ca-certificates

Alternatively, the issue can be fixed on SNG7 or CentOS7 by manually running the following commands from the CLI:

sudo trust dump --filter "pkcs11:id=%c4%a7%b1%a4%7b%2c%71%fa%db%e1%4b%90%75%ff%c4%15%60%85%89%10" | openssl x509 | sudo tee /etc/pki/ca-trust/source/blacklist/DST-Root-CA-X3.pem
sudo update-ca-trust extract

How to test if your system is affected:

Run the test against one of Let’s encrypt’s services:

wget https://acme-staging-v02.api.letsencrypt.org/directory -O-

A Failed Response will look like this:

–2021-10-01 00:00:00-- https://acme-staging-v02.api.letsencrypt.org/directory
Resolving acme-staging-v02.api.letsencrypt.org (acme-staging-v02.api.letsencrypt.org)… 172.65.46.172, 2606:4700:60:0:f41b:d4fe:4325:6026
Connecting to acme-staging-v02.api.letsencrypt.org (acme-staging-v02.api.letsencrypt.org)|172.65.46.172|:443… connected.
ERROR: cannot verify acme-staging-v02.api.letsencrypt.org’s certificate, issued by ‘/C=US/O=Let’s Encrypt/CN=R3’:
Issued certificate has expired.
To connect to acme-staging-v02.api.letsencrypt.org insecurely, use `–no-check-certificate’.

A Successful Response will look like this:

--2021-10-01 00:00:00--  https://acme-staging-v02.api.letsencrypt.org/directory
Resolving acme-staging-v02.api.letsencrypt.org (acme-staging-v02.api.letsencrypt.org)... 172.65.46.172, 2606:4700:60:0:f41b:d4fe:4325:6026
Connecting to acme-staging-v02.api.letsencrypt.org (acme-staging-v02.api.letsencrypt.org)|172.65.46.172|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 724 [application/json]
Saving to: 'STDOUT'
 0% [                                                                                         ] 0           --.-K/s              {
  "keyChange": "https://acme-staging-v02.api.letsencrypt.org/acme/key-change",
  "meta": {
    "caaIdentities": [
      "letsencrypt.org"
    ],
    "termsOfService": "https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf",
    "website": "https://letsencrypt.org/docs/staging-environment/"
  },
  "newAccount": "https://acme-staging-v02.api.letsencrypt.org/acme/new-acct",
  "newNonce": "https://acme-staging-v02.api.letsencrypt.org/acme/new-nonce",
  "newOrder": "https://acme-staging-v02.api.letsencrypt.org/acme/new-order",
  "revokeCert": "https://acme-staging-v02.api.letsencrypt.org/acme/revoke-cert",
  "xK2Vc0EsDik": "https://community.letsencrypt.org/t/adding-random-entries-to-the-directory/33417"
100%[========================================================================================>] 724         --.-K/s   in 0s
2021-10-01 00:00:00 (37.9 MB/s) - written to stdout [724/724]

Notes on FreePBX 13 and below

We currently have no plans on patching FreePBX 13 and below, so on Oct 1st, 2021 the Let’s encrypt hooks inside of FreePBX will stop functioning properly. We recommend that anyone running FreePBX 13 upgrade to FreePBX 15 and above.

4 Likes
2

As of today, the patch is currently in edge. We hope to make it generally available soon. For now you can install the module in edge by running the following CLI command:

fwconsole ma --edge upgrade certman

You can install a specific tag by running the following:

fwconsole ma downloadinstall certman --tag 15.0.47

4

is the updated certman module expected to be Generally Available prior to Oct 1?

5

Yes, hopefully it should be available on Monday after QA qualifies it.

1 Like
6

ouch. Just outside of the default “Update every saturday morning” cycle.

No chance of QA qualifying it be EOD so that affected systems auto-update prior to the weekly updates?

7

I would say it is highly unlikely that QA will be able to fully qualify it before Monday.

8

Thanks for the update!
worst case, SOME systems are affected for ~35 hours between 00:00 Oct 1 and the autoupdates on Saturday the 2nd, or admins schedule a non-standard upgrade.
I know from my side, we really appreciate you guys!

9

Perhaps add a ‘cron job’ (for the user asterisk or root , either should work)

30 23 30 9 * “/usr/sbin/fwconsole moduleadmin upgradeall && /usr/sbin/fwconsole reload”

and check your email a few minutes later.

10

That this is even something possible to be an issue is complete incompetence on Sangoma’s part.

More than one of us have been quite vocal about the horrible implementation of the ACME protocol that was chosen to be used. But if you are going to choose to be your own thing, then you have to choose to do even the most basic level of maintenance for it.

It has never happened for the Digium cert until a few months out, it has never happened for fail2ban, and now it is not happening for certman…

Horrible.

1 Like
11

I am seriously trying hard not to downplay this issue in anyway, but yes, we’re being safe than sorry here. I do think there is a chance, granted a small one, that something bad may happen if someone doesn’t upgrade, but, if you choose to do nothing and not upgrade, there is a high possibility that let’s encrypt renewal mechanism will continue to work anyways. This is actually because of the library we chose. In fact, the other mechanisms such as certbot, would be more likely to stop working. You can feel free to read more about all of our findings here as to why, but the TLDR is that PHP manages certificates differently than the system: https://issues.freepbx.org/browse/FREEPBX-22849?filter=-3

1 Like
12

Just for the sake of this thread, it looks like Certman is indeed GA for version 15.0.47

changelog:
15.0.47: #22849 Fix for Let’’'s encrypt root CA

2 Likes
13

While helpful, there is a 2 month lead time on LE certs. So everything will still puke on my cert until I renew, after applying this update.

I stand by my first post in this thread.

14

@sorvani I don’t think that’s how it works. All LE certs are dual signed for the OLD and NEW CA. The problem is CentOS 7 made the decision that the both CAs must validate a certificate to consider it valid. The patch, simply removes the OLD cert from the validation so that it will let the NEW CA do its job. So no LE renewal is necessary. This is just a client issue (where FreePBX is the client) connecting externally to other LE secured servers. People connecting to a FreePBX instance that is secured with LE certificate should have no issue, even if that server is not patched. At least, no issue until the LE certificate that is securing the instance fails to automatically renew, which like you said, could be anytime within a 2 month period.

1 Like
15

Do you think it will work if we upgrade the Certs package as described here? or do we need a different patch for FreePBX itself? https://www.queuemetrics.com/blog/2021/09/29/lets-encrypt-tutorial/

16

If you’re running CentOS 7 that will possibly work. I’m glad to see CentOS finally released an update, looks like we were a little bit ahead of them. For SNG7, we’ll have to release a new rpm based on this.

1 Like
17

So when you do it will be enough upgrading the ca-certs package?

18

@lynx87 it looks like we got the yum package upgraded last night, so you should ALSO be able to run ‘yum upgrade ca-certificates’ to resolve this issue.

1 Like
19

Deliberately left my home prod server on the old certman module version without this fix to see what would happen today. Haven’t tested TLS SIP transport yet but:

:white_check_mark: browsing to GUI with https - no issues (as expected)
:white_check_mark: LE cert renewal using module - no issues

2 Likes
20

If something went wrong for you, don’t feel bad. Some big players messed up too…