Cron vs Job (also: Certificate Management failed renewals fix)

FreePBX Development
1

Hi guys, I’m starting this thread to hopefully shed a light on the obnoxious problem of cron management in FreePBX.

The main consequence that most of the users notice is the tls certificate expiring because the relative cron entry gets removed, but this is just the peak of the iceberg! I’m monitoring the issue for more than a year now and I’ve pretty clear in my mind what is going on (I had it clear years ago already, oh well…)

First a little summary of some discussions regarding this issue:

And now a brief description with identified cause and proposed solution:

FreePBX switched to a unified Job interface in v.14 to manage recurring task with the objective of avoiding issues with modules directly editing the system crontab and messing with it. And that’s great!

The Job class works really well but does not implement (yet, my fault I didn’t have the time to continue the PR) a parallel system to handle multiple “jobs” coming in, so it is not suitable for intensive tasks that will block subsequent calls until completed.

Now it’s all fine and dandy until we reach modules that still uses cron to handle their recurring tasks! I encountered two for now: one is fwconsole itself, and one is certmanager. Following is the content of the crontab when not purged (on my system)

*/15 * * * * [ -e /usr/sbin/fwconsole ] && sleep $((RANDOM\%30)) && /usr/sbin/fwconsole userman --syncall -q
8 3 * * * /usr/sbin/fwconsole certificates --updateall -q 2>&1 >/dev/null
* 5 * * * /usr/sbin/fwconsole util cleanplaybackcache -q
36 0 * * 1 [ -e /usr/sbin/fwconsole ] && /usr/sbin/fwconsole ma listonline --sendemail -q > /dev/null 2>&1
36 1 * * 1 [ -e /usr/sbin/fwconsole ] && /usr/sbin/fwconsole sys listonline --sendemail -q > /dev/null 2>&1
* * * * * [ -e /usr/sbin/fwconsole ] && sleep $((RANDOM\%30)) && /usr/sbin/fwconsole job --run --quiet 2>&1 > /dev/null

This is a pretty consistent crontab, but on FPBX versions > 14 should only contain one line (the last one)! This because all the recurring tasks should have been migrated to the Job class, but evidently the switch have never been completed.

Now as pointed out multiple times this would be a non-issue if the crontab was left untouched, but instead it gets periodically purged (found the suspect here! FBI there’s no need to thank me :slight_smile:).
From my experience the update checks continue to work even if it gets purged (why? no clue. Maybe it is periodically brought back and then purged again) but certman does not restore its crontab, at least not that often and this causes the expiration of certificates, the first thing users notice. Now, I saw people saying that they never encountered the issue, but this seems impossible to me! The code to purge the crontab is here and certman cannot update in any other way (see certman code for cron entry).

Now, would I suggest removing the code to purge the crontab from Job class? Absolutely not! I think the concept is great, but we should pursue it till the end, finishing what we started.

So, what are the next steps?

  • I’m gonna release a module that uses the Job class to update the certificates. This is only a workaround, meant as a temporary fix
  • Create a new PR for Job to allow for background/parallel execution of jobs
  • Migrate all existing modules that still use the old cron entries to use the “new” Job class
  • Completely deprecate the old cron calls in BMO, to avoid the rise of new issues like this one

I hope it is all clear and hope that somebody will jump in to help us improving FreePBX for every one, in the meantime
Thank you!

2

First step completed: released module to fix certificate renewal.

More info:

To install from Module Admin: Click Upload Modules > Download (From Web) > paste "https://github.com/Massi-X/freepbx-certmanjob/releases/download/1.0.0/certmanjob_1.0.0_signed.tar.gz" > Download then go back to module admin and install it normally. You could also download it from GitHub directly and upload it locally if your PBX does not have direct internet access.

Module is signed and eventual updates will be directly notified by your system.

For the moderators: It is possible to let me post/edit even after the discussion gets closed? I don’t know when I’m gonna complete the other steps in the checklist (from post #1) but I still want to keep this thread as a reference for everyone

3

Did you consider patches to existing functionality instead of a new module entirely ?

Also you can request the thread be open in the future via DMs to mods.

4

Yes it is clearly explained in the first post. This module is a temporary fix because the certificate issue has been bugging me out for an entire year (or more) so I quickly built it (no AI involved, as in all of my projects).
The rest will be a long process (read the code and modify it to implement background execution, edit the modules that still uses cron accordingly, test, submit the patches and wait for reviews and successive modifications, backport to 16 at least) and I don’t know how much time I have for this right now. You would agree with me that this workload requires time and effort and can’t be done as quickly as this module specific to this issue only? Also: I could reutilize the background code to eventually implement the changes to the framework, so this is not a totale waste of time.

5

Firing off cron every minute (+ random number of seconds) is what needs to change – that should probably be a service daemon running continuously, handling new jobs as they come in.

6

I don’t know what the impact might be but I suppose not that high?
You mean a init.d or systemd service? I’m not so expert of that…

7

I’ll be honest here, specially since the move to Debian. Why cron jobs are still being used for a lot of this stuff is beyond me. This is classic “We built this solution 15 years ago and have ignored any modern options that are better suited since” logic this project has. There’s plenty of this that would be better suited in systemd timers for various reasons with one being that an actual log of what happened, especially when there is a failure, exists.

“My backup job didn’t run” - check journald to find out what happened and why it didn’t run.
“My LE cert didn’t renew” - check journald to find out what happened and why it didn’t run.

“My server was down for 6 hours and all the overnight cron jobs didn’t run” – yup, stuff will just be missed. That nightly backup never happened, the CDR cleanup never happened, any custom stuff never happened. While with systemd timers setup correctly, when the server comes back up all those jobs that should have run in that 6 hour downtime…run. Backup happens, CDR cleanup happens.

Why are cron jobs still handling way important things that need to run when there are better options for this?

8

That’s a good point and surely the fact that we are still talking about this is a consequence of code updates missing for a long time. Now while using systemd is surely a better option I want to spend two words defending the current job management: in an ideal word, the job class would handle all updates in background being the only Cron entry in the entire system. Assuming it is called every minute (and assuming this doesn’t cause a big overhead, which I don’t think it does), the overnight problem is not a problem at all, the Job class already takes into account the last and next execution of jobs. The logging is made through standard freepbx (BMO) logging, and it can be found among all the other logs inside the UI.
If all the fixes gets implemented, I think Job could became a pretty robust method for handling background tasks in a unified fashion. Also, implementing this would be very easy (for me that I have a clear idea at least) while changing the entire way things work right now would surely require time and a lot of work.

9

The Job class makes a database call every second, that database call gets all the cron jobs in the table that is enabled and then loops through them. As it loops through each of them it compares their actual cron schedule. For example, the firewall job has a schedule of */15 * * * * it’s going to compare that schedule as if it was a normal cronjob. It’s going to execute the script like it’s a normal cron job including no logging output. I have yet to see /var/log/asterisk/fwjobs.log actually be created due to there being output.

Nothing tracks when it ran last. It will track if something is in use/has a lock but nothing tracks when it was executed last or when it should be executed next. So again, server goes down for three hours when it comes back up and “fwconsole --job” is run again…it runs like everything is just fine. It won’t go back and run the jobs that should have run during the downtime. It just continues on and ignores the downtime.

I’m also not sure making database calls and looping through the entries is lightweight considering it has to load a crap tonne of stuff just to run the Jobs function(s).

Basically the Jobs BMO stuff just replaces putting 25 cron jobs in the crontab by just running a single cron job that checks every enabled entry in the table to see if it runs. So those once a day or once a week cron jobs actually get checked to see if they should run every single second when the Job class cron is executed. It’s not some magic black ju-ju doing extra stuff. It’s running cron jobs that act like they ran from the crontab and since those jobs don’t produce output there are no logs produced for said job.

10

Ok the database call Is a good point. I think this could be circumvented by generating a static file every time a reload is done and then checking against this.
The log could be easily implemented if not already there, I did it in this module I just released (although a general method should be implemented then).
The great thing about using Job is that your module is inside BMO and it can reliably access methods and settings without any particular consideration.
Also, allowing the modules to edit the crontab is a big no-no as bad things could happen (but I imagine this was not your point anyway).

11

Here’s the thing, cron cannot do a lot of things systemd can do. However, if you want to write a lot of bash syntax you can make cron do some of the things systemd can do. But then you’re really calling on external things.

Logging the output to a file. No date/timestamps or consistent formatting. What ever the stdout/stderr output format is…that’s what you get.

* * * * * /path/to/script >> /var/log/myjob.log 2>&1

Now my logging has a timestamp and exit code.
* * * * * { echo "[$(date)] START"; /path/to/script; rc=$?; echo "[$(date)] END rc=$rc"; } >> /var/log/myjob.log 2>&1

It just seems cron + shell + external (BMO) seems pretty convoluted over something that does all of it natively along with so much more control, reliability and flexibility.

12

Sorry but I don’t see the point of logging to an external file unrelated to everything. The module specific logging is obviously a responsibility of the module and we should not consider it here; logging of exceptions can be catch and sent to the standard freepbx.log file (or the specific one), the log we are all used to see in report > asterisk logfiles.
Oh and I forgot for the missing schedule if the server goes down a “last run” check would be very easy to implement if not already here (I’m going by memory now).

Is not that I don’t want to go for a better solution if possible, it’s only that I think that modules (internal but mainly external to freepbx) expect a standard interface and methods even inside a Job, so if we move to systemd it is necessary to implement all of those things anyway. Or your idea is to let every module handle it’s own service by itself?