Mult emals - Cron <asterisk@24408486> [ -e /usr/sbin/fwconsole ] && sleep $((RANDOM

updating did not solve the problem

And the asterisk users shell is ?

Interesting history on this.

In 2019 @tm1000 took this out of cron and made it into a job.

Two weeks ago it was made back into a cron. :face_with_raised_eyebrow:

Anyway, the “%” fix only got merged into version 16 and not 15… so if you are using 15, it’s still not fixed. (Merge pull request #128 in FREEPBX/userman from ~VSC55/userman:bugfix… · FreePBX/userman@336f459 · GitHub)

ticket: [FREEPBX-23154] userman --syncall -q Crontab needs to escape % - Sangoma Issue Tracker

Yes everything is suppose to move in to “jobs” The library added in 2019 allowed everything to run under 1 cron job so your crontab didn’t look like a steven king novel. I have noticed new devs not reading the docs and existing devs not enforcing standards.

2 Likes

And indeed @afonsolaw2020 's crontab does look like a Christine car-wreck, what happens at 23:30 ?

1 Like

This was also a fix that I think @dicko proposed which was run all cron jobs under one cron task so that CPU load of launching PHP 20x times would not take down a system, instead PHP would launch once and then all the processes that needed to cron would be done at the same time utilizing the same “shared” memory and CPU and reducing overall load on the system.

This is in-fact how other PHP Frameworks do scheduling like Laravel

The job manger (/usr/sbin/fwconsole job --run) itself has to run in the cron every minute. $((RANDOM%30)) is there because of PBXact in the cloud where-as they all use the same storage so every top of the minute there was a large spike of I/O. Randomizing this with a sleep solves the issue of disk I/O but unfortunately it doesn’t translate to other distros well.

The Job functionality in FreePBX is also poorly documented because when I started working on this was right around the time Tony Lewis announced he was leaving and it was pretty chaotic.

Sidebar: Im sorry to shoot everyone who does this in the foot but PBXact and FreePBX don’t scale well in cloud environments because they are single tenant and FreePBX has expectations that it is running on its own dedicated hardware. This is one (of the many) reasons why.

5 Likes

What sucks about this change is that there was zero community discussion as to why everything should or shouldn’t be under a single php process for cron jobs. The change to userman was just done because a FreePBX Internal ticket said to do it (see FREEI-1194 Possible for job cron task to become locked indefinitely · FreePBX/userman@d03b875 · GitHub)

FREEI-1194 Possible for job cron task to become locked indefinitely
Moving userman sync from JOBS to Cron.
the reason for moving sync from job to cron is , sync takes lots of time
in certain case , which cause other process not being executed and
fwconsole JOB itself get struck for long

Since this was obviously an issue wouldn’t it be wise for Sangoma to fix the bug in the Job class instead of reverting back to using a cron line for each individual process? (I mean copy Laravel’s implementation of task scheduling, its non-blocking and its easy enough to do). Now you have yet another PHP process taking CPU and memory away from your system which basically brings back the issues @dicko and others had in 2019. It’s just reverting without looking at the actual history of why things were done in the first place. Sangoma’s own FreePBX/PBXact Cloud implementations will suffer the most from this (because of I/O bound issues)

It is what it is

1 Like

I question whether there was even any internal discussion since someone would know the history of moving crons to jobs and would say “hey we want this to be a job, let’s not go back to cron.”

FYI, The history was literally me arguing with @dicko and someone else (who also ran cloud instances of FreePBX) in a thread here on community for a couple of days until I actually did the thing they/he was proposing

Additionally in FreePBX 2.5 or so there was also a cron manager of sorts. Which was abandoned at some point which is why the crontab looked horrendous after a couple of years. The Jobs class was there to solve that issue.

This is all too familiar with the recent sks gpg key servers issue as well. FYI if you didn’t know Jason Parker was the last one working on GPG issues (Publically) in FreePBX for Sangoma and he was “terminated” from Sangoma in October 2021 (see: Jason Parker - Jeltz | LinkedIn). Mergers and Acquisitions can be a b*tch.

1 Like

It is 2021, systemd ‘timers’ would likely better supervise maintenance tasks in an orderly fashion.

over to you @kgupta :wink:

4 Likes

I think we have agreed on this a few times for a few things already…

Changing $((RANDOM%30)) to $((RANDOM %30)) dependently fixed my issue.

\# sh -c "echo $((RANDOM %30))"
27
\# sh -c "echo $((RANDOM\%30))"
-bash: RANDOM\%30: syntax error: invalid arithmetic operator (error token is "\%30")

But some process keep rewriting /var/spool/cron/asterisk. Does any one know which process?

Checking my emails now I see this message started appearing for me today straight after the 06:35 UT automatic module Update email. It appears every 15 minutes.

So whatever is causing this came via the update. I just did a manual module update but there is no change.

I have not read through this long thread - is there a fix for this?

I have the same issue from this morning.

Same issue here.

Current PBX Version:
15.0.17.64
Current System Version:
12.7.8-2107-3.sng7

Same problem as documented in this thread. Basically FreePBX pushed an untested and broken module to your PBX. You will need to either wait for a fix, presumably next Friday, or manually revert the module. It is not a real killer but an incredibly incompetent release

Same here

FreePBX 15.0.17.64

Started after this mornings automated module updates, now recieving an email every 15 minutes from our FreePBX server.

Please advise.

Did you read this thread ?

Could someone please advise how to go back to the previous version.

1 Like

It might be a little more complicated than that, because the previous module didn’t inject a crontab and the broken one might not remove it on being ‘replaced’.

Any takers here?

1 Like