Browser support was NEVER a concern
The root CA expiring on October 1 affected TLS with Polycom VVX 411s. Other PBXes recommended modifying the certificate chain, for example, removing step 2 below:
Certificate chain
0 s:/CN=tel.whitestone.link
i:/C=US/O=Let's Encrypt/CN=R3
1 s:/C=US/O=Let's Encrypt/CN=R3
i:/C=US/O=Internet Security Research Group/CN=ISRG Root X1
2 s:/C=US/O=Internet Security Research Group/CN=ISRG Root X1
i:/O=Digital Signature Trust Co./CN=DST Root CA X3
My guess is that the Polycoms are experiencing the same problem @mbrooks mentioned that both CAs must be valid. We had no issues with the system module updates and /etc/pki/ca-trust/source/blacklist/DST-Root-CA-X3.pem exists.
After beating our heads against the wall for a while, we changed the Polycoms to use a self-signed certificate for TLS.
Is it possible for certbot (we’re at v15.0.47) to generate certs that no longer include X3 in the chain? This appears to be the root problem from a best-practices perspective, irrespective of client settings.
-rw------- 1 asterisk asterisk 2203 Oct 2 12:05 cert.pem
-rw------- 1 asterisk asterisk 3752 Oct 2 12:05 chain.pem
-rw------- 1 asterisk asterisk 5956 Oct 2 12:05 fullchain.pem
-rw------- 1 asterisk asterisk 1765 Oct 2 12:05 last.csr
-rw------- 1 asterisk asterisk 3272 Oct 2 11:28 private.pem
-rw------- 1 asterisk asterisk 800 Oct 2 11:28 public.pem
chain.pem and fullchain.pem still reference X3:
keytool -printcert -file fullchain.pem
Certificate[3]:
Owner: CN=ISRG Root X1, O=Internet Security Research Group, C=US
Issuer: CN=DST Root CA X3, O=Digital Signature Trust Co.
Serial number: 4001772137d4e942b8ee76aa3c640ab7
Valid from: Wed Jan 20 10:14:03 AKST 2021 until: Mon Sep 30 10:14:03 AKDT 2024
Certificate fingerprints:
Signature algorithm name: SHA256withRSA
Subject Public Key Algorithm: 4096-bit RSA key
Version: 3
It seems odd that Freepbx is still generating certificates via certbot with an expired certificate in the chain, if I understand things correctly. (I regenerated the letsencrypt cert 2x to be sure.)
I don’t think that the FreePBX acme client is ‘certbot’.
Thanks for sharing that info. Can you tell us what Polycom firmware version those phones are running?
Correct, it uses lescript, last updated 2 years ago.
You will note the project hasn’t been touched since that update
Off topic but interesting , Neil Pang’s acme.sh now uses certs from “ZeroSSL” by default for newly issued domains, (just like the one used on this forum) as an alternative to LetsEncrypt , it doesn’t have X3 in its chain and so wouldn’t be prone to old devices like android => 4 or IE >= 11 ‘breaking’
Fully updated yet unable to update the Let’s Encrypt cert.
FreePBX 15.0.17.55
Certificate Manager 15.0.47
Certificate Valid Until 2021-10-04 Certificate Expired! (-1 days)
Update Certificate result: Nothing to do, no changes made
I have already run - yum update ca-certificates prior to posting.
To further add to this and @wsfadmin comment above.
I checked my certificates and he is indeed correct. The 3rd certificate added/signed with is the expired DST Root CA X3. It does seem that my Yealink’s authenticating via TLS still verify against this just fine. They must use the logic of ANY valid CA = pass. Whereas the Polycom phones use the ALL VALID = pass logic. I manually deleted the 3rd certificate block off of my [FQDN].com-fullchain.crt file and boom, Polycom registers via TLS now. So whatever script / application manages and creates those certificates needs to remove that expired CA off the chain. I’m not sure how even new certificates are being generated with it expired now. Hopefully this little hiccup can be resolved before the next batch of certificates renews.
I’m really not sure what is going on here, but I don’t think this is related to the LE CA cert expiring. I do believe this is an issue with the Let’s Encrypt renewal mechanism, which is something different and shouldn’t be affected by this. The Let’s Encrypt renewal mechanism could fail for a number of reasons.
Are you by chance running SNG7 (The FreePBX Distro)? Or are you running something else like CentOS or Debian?
@wsfadmin and @MrXirtam, so Let’s Encrypt is saying that they are still distributing the expired DST-Root-CA-X3 CA in the bundle to maintain compatibility with older clients that do not expire root ca certs. So, we’ve added a feature where you can remove the DST-Root-CA-X3 CA from the bundle that Let’ Encrypt returns. My assumption here is that removing this may fix some older clients but break others, so just be aware of the trade-offs.
For now, we’re going to just return what Let’s Encrypt gives us by default, so you’ll have to check that ‘Remove DST Root CA X3’ box if you want to remove it from the CA Bundle.
This is now available in edge:
Certman v14.0.20
Certman v15.0.48
Certman v16.0.19
You can install the module in edge by running the following CLI command or wait on GA:
fwconsole ma --edge upgrade certman
1 Like
Thank you @mbrooks
We have tested this workaround with Polycom VVX (6.3.1) and Panasonic TGP 11.110 and report that both HTTPS provisioning and TLS appear to be fully functioning. We do not have older clients to support as you mentioned above.
Thanks again!
The problem is not the expired root CA DST cert although I know a lot of people think it is. The problem is that the intermediate R3 cert that is supplied by LetsEncrypt with even new certificates was deliberately cross-signed with the expired DST cert so that old antique Android clients that don’t validate expirations on their DST CA certs and don’t have the ISRG CA root in their cert store would continue to work. I already turned the flamethrower on the idiots at LE about this. Antique android clients all allow the user to override an invalid cert on their email clients, and antique android clients have web browers that are too old to render anything on today’s web. LE should have stared down the old Android users and told them to go buy new phones or to just manually override the invalid cert when the warning message pops up. Unfortunately they blinked, probably because none of them bothered testing what would have actually happened with an antique Android client.
Until such time that LE releases a new R3 intermediate cert that is ONLY signed with the ISRG CA root, buggy SSL implementations will continue to give problems with LE certs. Removing the DST root cert from the bundle does nothing because the buggy client is going to have that DST root CA cert in it’s own cert store. Replacing fullchain.pem with a new fullchain.pem also does nothing because LE didn’t change anything in fullchain.pem. Replacing the LE cert with a new one also does nothing because all LE certs are signed with the intermediate R3 cert and that hasn’t been changed either. Sorry about all this folks but it just won’t work. The problem is in the client. The ONLY time doing anything on the server would help is if the server has a really super old R3 cert that wasn’t cross-signed with the ISRG CA. (maybe SNG7 or Centos7 is like this I haven’t checked) and is handing that out and the client lacks the R3 intermediate cert in it’s intermediate cert store.
When the client is validating a cert chain if it hits a cross-signed cert it needs to use recursion to travel up to one of the root CA’s and if that CA is expired it needs to recurse back down to the cross-sign and then travel up to the other root CA to see if it’s valid. If it did that properly and hit the DST root CA it would then check the ISRG root CA and see that was valid, and then approve the cert.
To reiterate, deleting the expired DST CA cert is unnecessary if the client does it right. In fact it is not even necessary to present the intermediate or root CA certs to a modern client that does it right at all, because they are going to have the R3 cert in their intermediate cert store and the ISRG cert in their root CA store. All they will need is the LE cert itself. But admittedly they would have to be really modern and an older client that also does it right but lacks the newer R3 intermediate cert would need the server to send along the intermediate cert along with the LE cert.
On the bright side it gives a good excuse to tell users to finally pitch that old Windows 7 system but there’s a lot more stuff than web browsers that certs are used in. Such as phones.
I will point out that you can get a 1 year cert off of a number of resellers signed by a chain ending with Sectigo AKA Comodo for about $10. Your time is not worth screwing around with LE certs if you have older phones, etc. that have buggy SSL implementations.
2 Likes
@tmittelstaedt, excellent writeup. Thanks! Three questions:
- Can you post a link or a google search that points us to the $10 certs?
- Is there a forum thread for the “flamethrower” to LE? I’d like to add my voice to it.
- Does anyone have a writeup on how to convert FreePBX from LE to standalone cert?[1]
We have over 110 active endpoints that will no longer autoprovision because of this bug. 
[1]: I’m pretty confident I can sort this put, but it’s an issue a whole lot of FreePBX users should be having. A script or a writeup would make their life easier.
You can use
- It uses the same authority as this site does ( ZeroSSL , not LetsEncrypt )
- It’s cheaper
- It won’t have an X3 cert
- It works with Polycoms and old Androids.
- it is easy to script
- You can use DNS-01 instead of HTTP-01 if you want an easier firewall setup
just drop the keys and certs into /etc/asterisk/keys and update (easy to script, it’s in the FM) and will from then on cron the whole process
Namecheap.com I use them for customers who I don’t run certbot on and use LE certs for. Note that the ACME protocol is the future for cert issuing and I think that even the “traditional” SSL sellers are going to shift to this. LE is notable because they hand out free certs but they are the vanguard of where the SSL industry is going. I completely agree with the ACME approach that SHOULD have been part of the original SSL standard IMHO.
There’s no dogpile thread on the letsencrypt.org site. Most people don’t completely understand the whole SSL thing, it’s something they do once a year and a lot of the corporate types get 5 year SSL certs and may only do this once in their careers at that particular company. And what LE is handing out is free and since it is free they can do what they want and people like me that complain about free ice cream are probably looked on as jerks anyway.
Note that unless your phones have the Sectigo root cert CA in their CA stores then you are just going out of the frying pan into the fire. Of course, a single domain namecheap cert is cheap enough to experiment with.
It’s my considered opinion that this is the Achilles heel of SSL. If you have a client that has NO root CA store and is entirely depending on the server handing it root CAs then you have ZERO authentication. You might have encryption but without authentication what you have is worthless. A MiM attacker can generate their own CA cert and fake cert chain and your client won’t know the difference.
The SSL system is only as secure as the root CA certs that are in the devices because when they connect to a SSL server and the server hands them a root CA they can compare it to the one they have on file and see if it’s real or not. They actually SHOULD be ignoring any root CAs in any chains the server hands them anyway, the entire point of public private key encryption is the server hands you a cert that was signed by a root CA that you already know and trust. Bypassing that by just accepting any old CA cert in the chain handed to you ruins the entire point of SSL. So if a manufacturer of a device - like a phone for example - decides to “stop supporting it” and quits handing out firmware updates and root CA updates then its basically setting a time bomb for that device to become useless.
I understand WHY LE did what they did even though I think ultimately they are going to cause more trouble for people. But who knows. Nobody really has any idea out there of how many devices are using certs that are headed for expiration like decaying radium. LE is probably getting unfairly thrashed on just because they were the first in line, but ALL of the root CA certs out there in the world have expiration dates in them so as “unsupported” devices age other cert issuers are going to get it, also.
Namecheap and ZeroSSL use the same Sectigo CA root. It’s also possible to use certbot to get a cert from ZeroSSL, here’s a script for it:
GitHub - zerossl/zerossl-bot: The repository for the ZeroSSL certbot wrapper
win-acme also supports ZeroSSL now:
Linking win-acme to ZeroSSL · Issue #1648 · win-acme/win-acme · GitHub
Lots of ‘partner’ clients that use ACME, and zerossl-bot is a near drop in replacement for ‘certbot’
https://zerossl.com/features/acme/
Unfortunately, not partnered with the client the FPBX distro uses 
I prefer acme.sh because of
https://twitter.com/neilpangxa/status/1222792801835872256
and apilayer acquired zerossl last year (chain of authority ? )
This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.