Check the /var/spool/cron/asterisk file and look for a line similar to 44 1 * * * /usr/sbin/fwconsole certificates --updateall -q 2>&1 >/dev/null toward the top. If it’s missing add it back in. That should restore the functionality… at least until the next time whatever is causing that to get removed happens again.
1 Like
dobrosavljevic,
Is this cron job created with the creation of a new LE certificate? I have a handful of installations that suffer from this issue too. I just checked a system that reported today that the cert would expire in 19 days. When I checked the system, sure enough, the cron job was missing so I added this to my crontab for the user “asterisk”. The certificate immediately when the cron job ran.
Like many others, I have just been fixing these by hand in a one-off fashion because it is an easier fix than to report it. Thanks for pointing this out and sharing the knowledge!!
Not sure if that line in the crontab is present weather you have an LE cert or not. I just noticed it missing vs other systems that are working just fine so figured it’s as simple as adding it back to fix it on non working systems… at least hopefully temporarily until it disappears again for whatever reason.
This happened over a year ago on all our deployments. The cronjob to renew certificates had disappeared. We never figured out why and Sangoma wasn’t able to figure out why either. I suspect a bad update… Then it worked and renewed certificates fine 2-3 times during that year and recently all deployments are having this issue. Once again it’s the same issue, the cronjob has simply disappeared.
The cronjob is configured when certman is installed so here’s the quickest way to fix it
fwconsole ma downloadinstall certman --force
fwconsole certificates --updateall
fwconsole reload
We’ll see in 90 days if something breaks it again…
2 Likes
We just had this break at another deployment.
Update the /var/spool/cron/asterisk file and see how long before it breaks again.
Simple fix, make it ‘immutable’.
chattr +i /var/spool/cron/asterisk
1 Like
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.
Just some quick notes on the code:
- The cron job is added at install. If anything removes it then it is gone unless install is run again.
- There doesn’t seem to be any code that removes it EXCEPT uninstall in the certman module.
- Assuming the cron job IS present it will run as asterisk. Everything apache should be asterisk so this shouldn’t be a problem. That said when people run it from the CLI they are running it as root. A test would be on a CLI run to use
sudo -u asterisk fwconsole...to see if it fails. If it fails this may need to be a hook.
I don’t use this module personally on my dev machines so some possible fixes I can’t validate:
- Test if the user matters. If so see if that can be resolved. If not move to a hook which breaks the modules independence on commercial modules so this should be a last resort.
- Add a hook to reload to check if the cron job is present, if not re-add it.
NOTE: There is an open PR to move from cron to the job class already with a signed CLA
I will note that I have done all those tests previously and running manually as asterisk worked in every case. Really the root cause is that SOMETHING removes that cron entry line. On every system I’ve checked but I was unable to determine what. Adding that cron line back in manually to the cron file seems to be a fix for now. All the systems that have had that line missing haven’t lost it again as of yet and I’ve been doing this fix for a couple of months now.
Seems like the PR you linked to might actually be the answer to the long term fix for this.
1 Like
Also not to keep beating a dead horse here but there is another system that had a similar symptom that I’ve noticed for a very long while that has something in common with this problem, a cron line missing in that exact same cron file. It’s the scheduled backups.
They would randomly stop working I’d say at the same rate that the certificate renewals would stop working. When we would check the backup line would be missing from that cron file. Saving the same backup job would add the line back in but then after a while it would disappear again and backups would stop.
Somebody recently mentioned that same symptom here: Scheduled Backup not working
It’s been working a lot better for us on newer versions of FreePBX (so 16 and up) but for sure it was a pain for a very long time for us on older FreePBX systems as well.
This is all to say, there is something for sure removing these cron jobs from the cron file seemingly randomly, I was just never able to determine what.
Running two FreePBX installations here each with very little customisation. On one the cert updates flawlessly and on the other it doesn’t.
On that affected box when I try to renew in the GUI it says “Nothing to do”, despite the fact that the cert was due to expire the following day.
Adding a new cert works fine from the GUI but they never renew.
After reading this thread, I checked the cron, and on the affected box the relevant line was missing. I’ve reinstated it and will monitor at the next renewal.
Whilst I can see that this is a relatively low priority bug - in that it is fixable either by adding a new cert in the GUI or by renewing on the command line - it is certainly affecting quite a number of users.
It seems, however, that this thread has identified the root cause - the missing cron line. Logically then, either there was something in a particular version of the OS or module code that failed to add the appropriate line at installation, or (perhaps more worryingly) there is something which is causing it to be removed.
Whichever the issue is, it should be relatively easy to fix by issuing an update to CertMan which checks for the line and re-inserts it during upgrade. Perhaps this check should always be part of the upgrade process for the CertMan module?
Im down to 1 deployment that is still auto updating the LetsEncypt cert… 20+ deployments all no longer auto update the certs and they ALL used to auto update. Trying the update feature in the gui never works…
Running as root fwconsole certificates --updateall from CLI always
This is supper frustrating. As we now have to log into all these systems and manually update. We just had one literally not auto update early this week and P series phone apps stopped working because of it… thats how we discovered it. Logged in, updated manually via cli and everything back working again… This was another deployment to add to the manual update list that always auto updating before a week ago…
These are all distro installs, fully licensed, commercial modules… Nothing custom. All are updated with latest updates monthly so the only thing I can come up with is some update at some point stopped this from working…
Now that you mention it, we’ve seen this as well.
So the root of it seems to be something messing with cron.
Yea, that specific cron file at /var/spool/cron/asterisk.
Adding my 2 pence worth here as well, mostly given the “no one is complaining so it can’t be that bad” comment.
And ohhhh am I glad I found this thread.
We have 10 x freepbx 16 installs in the wild, all done using a standardised process so they should be identical most (turns out, now I check, all but one) were not renewing certs properly so I decided that today was the day to get that resolved.
One of the servers was installed on Tuesday this week and has the following crontab entries for the asterisk user (crontab -u asterisk -l)
38 2 * * * /usr/sbin/fwconsole certificates --updateall -q 2>&1 >/dev/null
* * * * * [ -e /usr/sbin/fwconsole ] && sleep $((RANDOM\%30)) && /usr/sbin/fwconsole queuestats --syncall >> /var/log/asterisk/queuestats_syncall.log 2>&1
0 0 * * * [ -e /usr/sbin/fwconsole ] && sleep $((RANDOM\%30)) && /usr/sbin/fwconsole queuestats --purnedata >> /var/log/asterisk/queuestats_purnedata.log 2>&1
*/15 * * * * [ -e /usr/sbin/fwconsole ] && sleep $((RANDOM\%30)) && /usr/sbin/fwconsole userman --syncall -q
@daily [ -x /var/lib/asterisk/bin/freepbx_voipinnovations_check ] && /var/lib/asterisk/bin/freepbx_voipinnovations_check 2>&1 > /dev/null
0 18 * * * /usr/sbin/fwconsole backup --backup=7a4868c5-c8fd-4123-a9a7-4778ab0905c4 > /dev/null 2>&1
10 18 * * 5 /usr/sbin/fwconsole backup --backup=ccf61e20-29d9-45be-91f9-bf29019f7024 > /dev/null 2>&1
1 0 1 * * /usr/sbin/fwconsole backup --backup=0eba4ef9-ddbe-4904-8ef3-535b7bdabf32 > /dev/null 2>&1
* 2 * * * /usr/sbin/fwconsole util cleanplaybackcache -q
10 11 * * 6 [ -e /usr/sbin/fwconsole ] && /usr/sbin/fwconsole ma listonline --sendemail -q > /dev/null 2>&1
10 13 * * 6 [ -e /usr/sbin/fwconsole ] && /usr/sbin/fwconsole ma upgradeall --sendemail -q > /dev/null 2>&1
* * * * * [ -e /usr/sbin/fwconsole ] && /usr/sbin/fwconsole job --run --quiet 2>&1 > /dev/null
Of the remaining 9 servers;
- 6 only have the final line from above
- 3 have the final 4 lines
- None of them have all the lines
- None of them seem to be running BACKUPS
- None of them are renewing certs
This does highlight a failing on our part… I relied on the fact that I would get an email telling me a backup failed to know something was wrong… but if the backup doesn’t run and you don’t check for the success email… lesson learned, task added to the weekly jobs list.
I can also confirm that running either of the following does renew the cert but with the same error noted by @dobrosavljevic when using sudo (permission denied error);
sudo -u asterisk fwconsole cert --updateall
fwconsole cert --updateall
Additionally, as others have commented. “update certificate” via admin->certificate management just gives a “nothing to do” message. Deleting and re-creating looks like it works but does not add the crontab line back in.
Happy to share my “process” for setting up a machine if anyone wanted to set something up as a test bed. Also thinking that for now, I’ll monitor the new server and see at what point the line(s) start to vanish…
This is not good… I’m off to try and fix all my backups
1 Like
For the record;
Editing and saving each of the backups does seem to have put them back into the crontab for asterisk. Not sure how long for but they are at least back for now.
And… just to document this…
I added the 3 backups back (daily, weekly, monthly) and ran the daily one. Just before closing my ssh session I decided to run crontab -u asterisk -l again just to check and I’m greeted with
18 0 * * * /usr/sbin/fwconsole backup --backup=5c0f4623-2b8b-4ada-84ad-99318d60f036 > /dev/null 2>&1
18 10 * * 5 /usr/sbin/fwconsole backup --backup=a587d845-d71a-4c24-b011-0ab4e0cd2fe0 > /dev/null 2>&1
0 1 1 * * /usr/sbin/fwconsole backup --backup=e6da9d42-d81f-47d3-9e87-b1a6d76d74f8 > /dev/null 2>&1
* 2 * * * /usr/sbin/fwconsole util cleanplaybackcache -q
54 1 * * * [ -e /usr/sbin/fwconsole ] && /usr/sbin/fwconsole ma listonline --sendemail -q > /dev/null 2>&1
54 2 * * * [ -e /usr/sbin/fwconsole ] && /usr/sbin/fwconsole sys upgradeall --sendemail -q > /dev/null 2>&1
54 3 * * * [ -e /usr/sbin/fwconsole ] && /usr/sbin/fwconsole ma upgradeall --sendemail -q > /dev/null 2>&1
* * * * * [ -e /usr/sbin/fwconsole ] && /usr/sbin/fwconsole job --run --quiet 2>&1 > /dev/null
So, whilst my LE certs still aren’t getting renewed, something seems to have put some of the lines back again…
But only one one of the servers, not on the other 5 that are also missing those lines.
Hi all, I also have the same bug (crontab entry missing). I already pushed a PR to fix the issue by switching to Job class but nobody seems to care…
It’s here Switch from Cron to Job class by Massi-X · Pull Request #4 · FreePBX/certman · GitHub
2 Likes