I’ve been having to ssh in and manually renew my let’s encrypt cert for the last couple of times after it stopped renewing automatically within freepbx. It had been renewing automatically fine for about a year.
I’ve found several posts on here about people having trouble with it randomly deciding to stop renewing the letsencrypt certificates, but have not found anyone who has said how to get it to renew automatically again…
Right now I have to ssh into my freepbx box and do a “fwconsole cert --updateall --force” to get it to renew. I would like to find out what has caused the disconnect so that freepbx will go back to renewing it automatically if anyone has found the answer…
So I went skimming the posts and there is one every few months or so. I would think these to be edge cases or something with the individual systems rather than a broader issue. In other words if this was a general bug people would be at the gates with pitch forks. It is heavily used functionality so if it was broken hell would rain down.
My thought would be to run it without --force and see what the error is. Then work forward from there…
Note almost every post about this has minimal interaction then the OP typically dispersal so I assume they found out something was configured wrong and fixed it. Then didn’t have the courtesy to resolve and update their post.
I am the one that’s participated and reported this problem in multiple posts in the past. Out of the 60 deployments that we manage about 10% to 20% of them need to be renewed manually. I suspect that recreating the certificate would get it to work properly on those deployments but we haven’t done that in the past as I feel like it would eventually break again.
I live in the shell, so doing a quick “ssh pbx fwconsole cert --updateall” (I am about to create a shell function for that command actually), no --force required, when we get notified that a certificate is about to expire takes all of 2 seconds.
The reason I haven’t spent any time banging down doors is simple math. I feel like the 4 to 5 times a month that we have to do it times 2 seconds is far less of a time investment for me then trying to track this down, especially because I haven’t been able to locate any logs when I’ve tried to look into it in the past.
I also don’t expect much out of this functionality on FreePBX especially when it’s missing something as essential as a DNS auth option. My assumption is that we are just using a shitty version of certbot that nobody is bothering to update until hopefully the next version of FreePBX, whatever that may be and whenever it may come out.
This assumption may be completely wrong actually. I went to check after putting this up there and it doesn’t seem like FreePBX is using cerbot for certificate renewals as I can’t really find a binary for it in the usual spots.
I’m pretty sure I’ve already tested that theory, but just in case I didn’t, I’ll give that theory a try and update this thread whether it works or not.
I just deleted and recreated both the existing let’s encrypt cert and created a brand new lets encrypt cert for freepbx. The self signed cert has already been removed and the new let’s encrypt cert that I just created is set to the default. So the new let’s encrypt cert, the new full chain cert, and the sangoma connect cert are the only three certs on the freepbx system.
If your theory pans out, I should be able to renew the let’s encrypt cert by clicking on the Update Certificate button in the freepbx gui in a few days and tell weather or not feepbx can renew it successfully. I’ll post the result back here.
Do you have any thoughts on where I could find that output to see what the error is in freepbx? There isn’t any log that I can find of the automatic renewal, and clicking the “Update Certificate” button does not create any error output when using the freepbx gui.
Manually using SSH and the fwconsole command to renew the lets encrypt cert from the command line successfully renews it manually. If you can think of any place there is a diagnostic logging for the automatic renewal process, I’m game for trying to find it again.
I don’t know that this will be a good short term test. Manual renewals always work. You’ll have to wait for it to stop renewing again to be able to tell if this made any difference. You may end up waiting a long time as some certificate stop auto renewing quite a while after they were initially setup.
I don’t know specifically what’s going on here, but if the scheduled task doesn’t work and it always works from bash as root, then it could be a permissions issue. Next time you go to renew from command line, try running the task as the asterisk user and see if you get a different result:
Since we are just checking permissions I actually decided to try and renew a cert on one of the machines that’s stopped doing it automatically before we get the notification that it’s about to expire and I did get an interesting result in the output:
Done !!§§!
Unable to access the running directory (Permission denied). Changing to '/' for compatibility.
Unable to access the running directory (Permission denied). Changing to '/' for compatibility.
Successfully updated certificate named "REDACTED"
Normally don’t get that error message when I do it without sudo -u asterisk.
Chiming in to agree with the 10% to 20% number of installs doing this.
Also…at the risk of directing ire at myself…I’ve almost entirely stopped reporting problems/bugs/whatever. I tried with several issues and just got nowhere with getting things fixed. I dumped a lot of hours into the issue with EPM Pro creating invalid config files and never got anywhere. I strongly suspect I’m not the only one who’d gone quiet, and that it contributes to the lack of a flood of reports of things like this.
Yea, I guess if I am being honest, on top of it not taking a lot of time for us to rectify manually we don’t really bother putting a bunch of effort into reporting minor issues like this as I don’t really see Sangoma put a lot of effort or resources, for whatever business reasons, into rectifying them.
I’ll report and try to fix PBX breaking problems and I’ll continue to give back by helping out here as much as I can and we’ll continue to buy things from Sangoma that make sense for us to purchase from them instead of another vendor in hopes that FreePBX continues to be a thing but in all honesty I really didn’t see continuing to complain about this particular issue (or others like it of the same magnitude) as ever receiving any sort of serious attention from Sangoma.
So I’ve always gone with “Maybe it’ll be incidentally fixed in the next version of FreePBX.”
Just got an email from Let’s Encrypt Expiry Bot that one of the certificates is about to expire in 19 days for one of the PBX that have a Let’s Encrypt certificate setup on them that we typically need to manually renew.
@lgaetz want me to look at anything in particular to see why it’s not doing it automatically?
Other than being aware that there are reports of this, I don’t know what’s going on. I have LE certs on most of my boxes and all of them renew automatically without issue. My comment above was a response to a more generic situation where “It doesn’t work from GUI but does work from CLI” might indicate a permissions issue. Running the cert update command as the ‘asterisk’ user would expose that.
I’ll stop after this but just wanted to illustrate the frequency. Just got another notification for a deployment about a let’s encrypt cert that needs to be renewed.
This happens to me as well repeatedly. I did the delete and reapply trick on this particular cert a few cycles ago and it was renewing until this time and has expired again =(
Another example that caused me a heart attack for about an hour that was caused by a non renewing certificate.
In an environment with two PBXact servers in a redundancy configuration using the Advanced Recovery module, starting at midnight tonight the primary server’s asterisk wouldn’t stay running for more than two minutes. Nothing we did would ensure that asterisk stayed up. We would watch it start up after running asterisk -vvvc and then a minute into the startup procedure we would see it getting shut down. It wasn’t crashing or freezing up, asterisk was simply going through the shutdown procedure and then when we checked right after there was no running asterisk on the primary system to process calls.
Restarting the system would make no difference either.
After about an hour of sheer panic we figured out that the certificate on the secondary unit was expired. It took us a while to think of checking on the secondary server because the problem was manifesting itself only on the primary.
The expired certificate caused heartbeat connectivity issues between the servers and basically was causing the secondary server to keep shutting down asterisk on the primary without really anything showing up in the asterisk full logs on the primary side (side note, it would be great if Advanced Recovery would somehow log that it was shutting down asterisk when it was shutting it down inside the asterisk full logs).
Again, it took us a while to consider checking anything related to the secondary server and Advanced Recovery because nothing obvious was showing up in the asterisk full logs of the primary.
Renewing the certificate on the secondary unit immediately resolved the issue.
No idea how we got unbelievably lucky that the expiration fell on a Sunday, the only day this environment is shut down and receives no calls so we had some breathing room to troubleshoot.
Confirmed that the cert renewal cron line 44 1 * * * /usr/sbin/fwconsole certificates --updateall -q 2>&1 >/dev/null is missing inside the /var/spool/cron/asterisk file on the affected systems.
Going to try adding just that line in manually and see if it disappears again.
