If you tried to edit it without making it mutable again of course it wouldn’t work, that’s the whole point of chattr i . . . from the manual
i A file with the 'i' attribute cannot be modified: it cannot be deleted or renamed, no link can be cre‐
ated to this file, most of the file's metadata can not be modified, and the file can not be opened in
write mode. Only the superuser or a process possessing the CAP_LINUX_IMMUTABLE capability can set or
clear this attribute.
If something is writing to that file more than ’ very occasionally’ then whoever wrote that did a big oopsy
Just checking back in on this as this issue has just come back to the top of my hit list. Of the 10 servers I have running only the one that I most recently setup still has the fwconsole certificates --updateall crontab line intact. The good news is that for me at least all of the servers do still have the backups working as expected.
None of my servers have the 30 second random wait on them so I do think that’s something specific to the server you looked at @jfinstrom.
I always like to look at why… Why would the devs think that we need to check that nobody accidentally wiped out the cron jobs. and… why would we rely on a process called from the crontab to check that the crontab is ok… because… well… if the crontab is gone then the task to check it’s ok isn’t going to get called is it… Maybe someone had a bad day.
I also think you are making an assumption here @miken32 by saying “that isn’t it”. It’s reading and re-writing the crontab every min (and in at least one of my servers it seems multiple times a min) so some sort of edge timing issue where the read doesn’t finish before the write starts or where two instances cross over… that could easily be the issue here. Can you find anything else that is making changes to the crontab. When I searched I only found
AND… if the only line that is being removed is the fwconsole job --run one then that task would only be run once and would then delete it’s self surely?
My servers are numbered sequentially, I’m going to manually add the renewal line to the even numbered ones and see what happens. Maybe it was some random one off event that caused the issue on any servers that were running at the time?
For anyone following along at home
crontab -u asterisk -e
probably press “i” to enter insert mode then add to the bottom of the file
I believe the direction was that cron would only be used as a trigger to load FreePBX every minute and fire off any jobs contained in its jobs table.
But – and there may be a technical reason behind this, I don’t know – it does that (fwconsole job) and also runs more cron jobs that… are also FreePBX. (fwconsole backup and fwconsole certificates for example)
Makes no sense to me. And because there’s apparently the risk of a user or some other process corrupting the crontab, it needs to be validated all the time.
We don’t want to run php a bunch of times, we therefore put lots of the scheduled tasks into a job within freePBX and just run that once. Makes sense (kinda). (I’m using “the Royal we” there, I had nothing to do with it)
I also guess that for some jobs, backups being one, it makes sense to spend the resources on additional php instances to make sure they run independently of other jobs.
I guess it also makes sense that authors of modules could choose to have their module either use system cron or the fwconsole job system to run regular tasks.
Good to understand the history a bit.
But…
Why does the fwconsole task need to messing around with cron jobs and…
Something is still causing lines to go missing from crontabs…
Interesting thread. I have a friend who I built a FreePBX system for and he calls me like clockwork every 90 days “hey I got this email about my certificate expiring, what do I do?”.
I’ve been ssh’ing in and manually renewing this every time but it’s getting tedious. I just checked and found that the cron entry is/was missing from /var/spool/cron/asterisk so I guess that’s why. No idea what is causing that, along with apparently everyone else on this thread.
Thinking of paving his system and installing 16 to see if it’s better, but with 17 around the corner I am going to wait a bit.
Does anyone know if this problem is unique to v15.x installs?
I just upgraded our office production deployment from FreePBX 16 to FreePBX 17. Certificate update is working again in the FreePBX 17 gui to update the cert. Hopefully this is an issue of the past now on FreePBX 17… We shall see
I updated some modules (system admin and framework) and some lines returned (even though they are not certman related, and this confirms the theory that this is a system-wide problem)
Ok… maybe I spoke too hastily. I basically have some 4 or 5 installations where this keeps happening. So it is reproducible in the sense of “I can find you a system that has manifestations of this problem”.
But if this is technically difficult to figure out (I should have read more of the earlier posts, sorry), then I admit I don’t have a reproducible case in the sense of “something that we can go ahead and debug”.
Can we creatively come up with something?
For example, if I set a watch on the crontab file to see when it is changed, to record when the backup schedule line disappears. I also keep my FreePBX system logging at the highest levels (if that is a thing in FreePBX, log levels such as “detailed” or “debug”?)
Then we try to match what was happening in FreePBX when the file got destroyed. Do you think this could work to provide some clues? Is it worth a try?
Ok, so I have a few results back… I (or more honestly, an army of xAI computers at my service) wrote a script to monitor file changes in `/var/spool/cron/asterisk`
I conclude that the deletion of the backup service happened during a server reboot at 5:15 AM.
I recognize that this was part of a weekly process in my HyperV server that hosts the FreePBX server as a VM. It is a full backup of the entire VM that starts by shutting down the VM, then copies the files, then starts it up again.
When FreePBX boots, apparently it cleans up the cron file. entries are removed, and only this is left:
When I log in to the UI, I notice the familiar red button on the top-left saying Apply config. I had noticed this before, that I was being asked to apply configs when I knew for sure I hadn’t changed anything.
If I do apply the configs, the backup job is re-added to the cron file.
I have a production system (16.0.45) that is outside of hours right now and just tried this. Unfortunately, it did not happen here.
crontab -l -u asterisk – backup job exists
reboot
log back in by SSH before doing anything else; crontab -l -u asterisk and the backup job is still there
However, this reboot did not cause an “Apply Config” to appear. I believe you got that because your reboot included a FreePBX update. Do you have automatic updates enabled? Perhaps whatever module(s) was/were updated caused the crontab refresh.
Thanks @billsimon your answer made look at this more carefully and although it is true that when the cron gets deleted, it happens during reboots, it is not obvious that every reboot leads to the cron getting deleted. It is an erratic behaviour, which is really frustrating.
I am giving up on this, and my current approach is this:
I set up the Backup in the UI, because it is convenient
I go to the cron file and copy the backup command, something like this: 29 3 * * 1,3,6 /usr/sbin/fwconsole backup --backup=b133c506-a1bc-4e27-80d2-a337cea01f2f > /dev/null 2>&1
I then disable the backup in the UI, to stop FreePBX from putting that line in the cron that is (mis-)managed by FreePBX
I create a new file in sudo nano /etc/cron.d/custom-asterisk with these contents:
# manual addition by Pgr, copied from what FreePBX UI writes in /var/spool/cron/asterisk, adding asterisk username before cmd
29 3 * * 1,3,6 asterisk /usr/sbin/fwconsole backup --backup=b133c506-a1bc-4e27-80d2-a337cea01f2f > /dev/null 2>&1
As the comment says, note that you add asterisk as the user name because here we are in a generic system crontab, not one specific to the user asterisk.
This new file stays stable, I just need to remember this if I change the scheduled times.
maybe it is periodically purging the cron entries?