@lgaetz @kgupta who is handling the review and merge of community contributions on Github?
1 Like
I’m not on the dev team so clearly my opinion should be taken with a pinch of salt (is that a British thing? does it translate well?) but…
Rather than changing the way the jobs are called would it not make more sense to work out what the actual heck is going on and why lines are being deleted… seems like the kind of thing you don’t want to risk leaving unresolved…
I’ve added
-w /var/spool/cron/asterisk -p w
to
/etc/audit/rules.d/audit.rules
and it looks like (from running ausearch -f /var/spool/cron/asterisk) cron is being called twice a min by the asterisk user and is updating (deleting and then creating) the cron file. Is this “normal” behaviour? Does anyone know if it’s possible to get auditd to ignore writes by a specific binary / user?
I may be totally misunderstanding what I am seeing here but I added -a always,exit -S execve -F path=/usr/bin/crontab to the audit log and it actually looks like crontab is being called something like 10 times a min, sometimes with the -l command line option and sometimes with just a -
time->Fri Feb 9 19:51:01 2024
type=PROCTITLE msg=audit(1707508261.418:6001782): proctitle=2F7573722F62696E2F63726F6E746162002D6C
type=PATH msg=audit(1707508261.418:6001782): item=1 name="/lib64/ld-linux-x86-64.so.2" inode=33591534 dev=fd:00 mode=0100755 ouid=0 ogid=0 rdev=00:00 objtype=NORMAL cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0
type=PATH msg=audit(1707508261.418:6001782): item=0 name="/usr/bin/crontab" inode=17319856 dev=fd:00 mode=0104755 ouid=0 ogid=0 rdev=00:00 objtype=NORMAL cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0
type=CWD msg=audit(1707508261.418:6001782): cwd="/home/asterisk"
type=EXECVE msg=audit(1707508261.418:6001782): argc=2 a0="/usr/bin/crontab" a1="-l"
type=SYSCALL msg=audit(1707508261.418:6001782): arch=c000003e syscall=59 success=yes exit=0 a0=21b8e70 a1=21b93f0 a2=21b8f40 a3=7ffe04de4260 items=2 ppid=8121 pid=8122 auid=999 uid=999 gid=1000 euid=0 suid=0 fsuid=0 egid=1000 sgid=1000 fsgid=1000 tty=(none) ses=205427 comm="crontab" exe="/usr/bin/crontab" key=(null)
----
time->Fri Feb 9 19:51:01 2024
type=PROCTITLE msg=audit(1707508261.429:6001785): proctitle=2F7573722F62696E2F63726F6E746162002D6C
type=PATH msg=audit(1707508261.429:6001785): item=1 name="/lib64/ld-linux-x86-64.so.2" inode=33591534 dev=fd:00 mode=0100755 ouid=0 ogid=0 rdev=00:00 objtype=NORMAL cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0
type=PATH msg=audit(1707508261.429:6001785): item=0 name="/usr/bin/crontab" inode=17319856 dev=fd:00 mode=0104755 ouid=0 ogid=0 rdev=00:00 objtype=NORMAL cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0
type=CWD msg=audit(1707508261.429:6001785): cwd="/home/asterisk"
type=EXECVE msg=audit(1707508261.429:6001785): argc=2 a0="/usr/bin/crontab" a1="-l"
type=SYSCALL msg=audit(1707508261.429:6001785): arch=c000003e syscall=59 success=yes exit=0 a0=1041e70 a1=10423f0 a2=1041f40 a3=7ffe0c5b61a0 items=2 ppid=8123 pid=8124 auid=999 uid=999 gid=1000 euid=0 suid=0 fsuid=0 egid=1000 sgid=1000 fsgid=1000 tty=(none) ses=205427 comm="crontab" exe="/usr/bin/crontab" key=(null)
----
time->Fri Feb 9 19:51:01 2024
type=PROCTITLE msg=audit(1707508261.451:6001788): proctitle=2F7573722F62696E2F63726F6E746162002D6C
type=PATH msg=audit(1707508261.451:6001788): item=1 name="/lib64/ld-linux-x86-64.so.2" inode=33591534 dev=fd:00 mode=0100755 ouid=0 ogid=0 rdev=00:00 objtype=NORMAL cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0
type=PATH msg=audit(1707508261.451:6001788): item=0 name="/usr/bin/crontab" inode=17319856 dev=fd:00 mode=0104755 ouid=0 ogid=0 rdev=00:00 objtype=NORMAL cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0
type=CWD msg=audit(1707508261.451:6001788): cwd="/home/asterisk"
type=EXECVE msg=audit(1707508261.451:6001788): argc=2 a0="/usr/bin/crontab" a1="-l"
type=SYSCALL msg=audit(1707508261.451:6001788): arch=c000003e syscall=59 success=yes exit=0 a0=1b62e70 a1=1b633f0 a2=1b62f40 a3=7ffcd6931ca0 items=2 ppid=8125 pid=8126 auid=999 uid=999 gid=1000 euid=0 suid=0 fsuid=0 egid=1000 sgid=1000 fsgid=1000 tty=(none) ses=205427 comm="crontab" exe="/usr/bin/crontab" key=(null)
----
time->Fri Feb 9 19:51:01 2024
type=PROCTITLE msg=audit(1707508261.461:6001791): proctitle=2F7573722F62696E2F63726F6E746162002D6C
type=PATH msg=audit(1707508261.461:6001791): item=1 name="/lib64/ld-linux-x86-64.so.2" inode=33591534 dev=fd:00 mode=0100755 ouid=0 ogid=0 rdev=00:00 objtype=NORMAL cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0
type=PATH msg=audit(1707508261.461:6001791): item=0 name="/usr/bin/crontab" inode=17319856 dev=fd:00 mode=0104755 ouid=0 ogid=0 rdev=00:00 objtype=NORMAL cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0
type=CWD msg=audit(1707508261.461:6001791): cwd="/home/asterisk"
type=EXECVE msg=audit(1707508261.461:6001791): argc=2 a0="/usr/bin/crontab" a1="-l"
type=SYSCALL msg=audit(1707508261.461:6001791): arch=c000003e syscall=59 success=yes exit=0 a0=1700e70 a1=17013f0 a2=1700f40 a3=7ffc410526e0 items=2 ppid=8127 pid=8128 auid=999 uid=999 gid=1000 euid=0 suid=0 fsuid=0 egid=1000 sgid=1000 fsgid=1000 tty=(none) ses=205427 comm="crontab" exe="/usr/bin/crontab" key=(null)
----
time->Fri Feb 9 19:51:01 2024
type=PROCTITLE msg=audit(1707508261.472:6001794): proctitle=2F7573722F62696E2F63726F6E746162002D6C
type=PATH msg=audit(1707508261.472:6001794): item=1 name="/lib64/ld-linux-x86-64.so.2" inode=33591534 dev=fd:00 mode=0100755 ouid=0 ogid=0 rdev=00:00 objtype=NORMAL cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0
type=PATH msg=audit(1707508261.472:6001794): item=0 name="/usr/bin/crontab" inode=17319856 dev=fd:00 mode=0104755 ouid=0 ogid=0 rdev=00:00 objtype=NORMAL cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0
type=CWD msg=audit(1707508261.472:6001794): cwd="/home/asterisk"
type=EXECVE msg=audit(1707508261.472:6001794): argc=2 a0="/usr/bin/crontab" a1="-l"
type=SYSCALL msg=audit(1707508261.472:6001794): arch=c000003e syscall=59 success=yes exit=0 a0=1f21e70 a1=1f223f0 a2=1f21f40 a3=7ffcacc435e0 items=2 ppid=8129 pid=8130 auid=999 uid=999 gid=1000 euid=0 suid=0 fsuid=0 egid=1000 sgid=1000 fsgid=1000 tty=(none) ses=205427 comm="crontab" exe="/usr/bin/crontab" key=(null)
----
time->Fri Feb 9 19:51:01 2024
type=PROCTITLE msg=audit(1707508261.481:6001797): proctitle=2F7573722F62696E2F63726F6E746162002D
type=PATH msg=audit(1707508261.481:6001797): item=1 name="/lib64/ld-linux-x86-64.so.2" inode=33591534 dev=fd:00 mode=0100755 ouid=0 ogid=0 rdev=00:00 objtype=NORMAL cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0
type=PATH msg=audit(1707508261.481:6001797): item=0 name="/usr/bin/crontab" inode=17319856 dev=fd:00 mode=0104755 ouid=0 ogid=0 rdev=00:00 objtype=NORMAL cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0
type=CWD msg=audit(1707508261.481:6001797): cwd="/home/asterisk"
type=EXECVE msg=audit(1707508261.481:6001797): argc=2 a0="/usr/bin/crontab" a1="-"
type=SYSCALL msg=audit(1707508261.481:6001797): arch=c000003e syscall=59 success=yes exit=0 a0=22eac50 a1=22eb360 a2=22eae90 a3=7ffd28e8daa0 items=2 ppid=8108 pid=8131 auid=999 uid=999 gid=1000 euid=0 suid=0 fsuid=0 egid=1000 sgid=1000 fsgid=1000 tty=(none) ses=205427 comm="crontab" exe="/usr/bin/crontab" key=(null)
----
time->Fri Feb 9 19:51:01 2024
type=PROCTITLE msg=audit(1707508261.493:6001801): proctitle=2F7573722F62696E2F63726F6E746162002D6C
type=PATH msg=audit(1707508261.493:6001801): item=1 name="/lib64/ld-linux-x86-64.so.2" inode=33591534 dev=fd:00 mode=0100755 ouid=0 ogid=0 rdev=00:00 objtype=NORMAL cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0
type=PATH msg=audit(1707508261.493:6001801): item=0 name="/usr/bin/crontab" inode=17319856 dev=fd:00 mode=0104755 ouid=0 ogid=0 rdev=00:00 objtype=NORMAL cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0
type=CWD msg=audit(1707508261.493:6001801): cwd="/home/asterisk"
type=EXECVE msg=audit(1707508261.493:6001801): argc=2 a0="/usr/bin/crontab" a1="-l"
type=SYSCALL msg=audit(1707508261.493:6001801): arch=c000003e syscall=59 success=yes exit=0 a0=b45e70 a1=b463f0 a2=b45f40 a3=7ffc6fc9b3a0 items=2 ppid=8132 pid=8133 auid=999 uid=999 gid=1000 euid=0 suid=0 fsuid=0 egid=1000 sgid=1000 fsgid=1000 tty=(none) ses=205427 comm="crontab" exe="/usr/bin/crontab" key=(null)
----
time->Fri Feb 9 19:51:01 2024
type=PROCTITLE msg=audit(1707508261.503:6001804): proctitle=2F7573722F62696E2F63726F6E746162002D6C
type=PATH msg=audit(1707508261.503:6001804): item=1 name="/lib64/ld-linux-x86-64.so.2" inode=33591534 dev=fd:00 mode=0100755 ouid=0 ogid=0 rdev=00:00 objtype=NORMAL cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0
type=PATH msg=audit(1707508261.503:6001804): item=0 name="/usr/bin/crontab" inode=17319856 dev=fd:00 mode=0104755 ouid=0 ogid=0 rdev=00:00 objtype=NORMAL cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0
type=CWD msg=audit(1707508261.503:6001804): cwd="/home/asterisk"
type=EXECVE msg=audit(1707508261.503:6001804): argc=2 a0="/usr/bin/crontab" a1="-l"
type=SYSCALL msg=audit(1707508261.503:6001804): arch=c000003e syscall=59 success=yes exit=0 a0=2037e70 a1=20383f0 a2=2037f40 a3=7fff376805e0 items=2 ppid=8134 pid=8135 auid=999 uid=999 gid=1000 euid=0 suid=0 fsuid=0 egid=1000 sgid=1000 fsgid=1000 tty=(none) ses=205427 comm="crontab" exe="/usr/bin/crontab" key=(null)
----
time->Fri Feb 9 19:51:01 2024
type=PROCTITLE msg=audit(1707508261.513:6001807): proctitle=2F7573722F62696E2F63726F6E746162002D
type=PATH msg=audit(1707508261.513:6001807): item=1 name="/lib64/ld-linux-x86-64.so.2" inode=33591534 dev=fd:00 mode=0100755 ouid=0 ogid=0 rdev=00:00 objtype=NORMAL cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0
type=PATH msg=audit(1707508261.513:6001807): item=0 name="/usr/bin/crontab" inode=17319856 dev=fd:00 mode=0104755 ouid=0 ogid=0 rdev=00:00 objtype=NORMAL cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0
type=CWD msg=audit(1707508261.513:6001807): cwd="/home/asterisk"
type=EXECVE msg=audit(1707508261.513:6001807): argc=2 a0="/usr/bin/crontab" a1="-"
type=SYSCALL msg=audit(1707508261.513:6001807): arch=c000003e syscall=59 success=yes exit=0 a0=20d0c50 a1=20d1360 a2=20d0e90 a3=7ffdf2750fe0 items=2 ppid=8108 pid=8136 auid=999 uid=999 gid=1000 euid=0 suid=0 fsuid=0 egid=1000 sgid=1000 fsgid=1000 tty=(none) ses=205427 comm="crontab" exe="/usr/bin/crontab" key=(null)
----
time->Fri Feb 9 19:51:01 2024
type=PROCTITLE msg=audit(1707508261.523:6001811): proctitle=2F7573722F62696E2F63726F6E746162002D6C
type=PATH msg=audit(1707508261.523:6001811): item=1 name="/lib64/ld-linux-x86-64.so.2" inode=33591534 dev=fd:00 mode=0100755 ouid=0 ogid=0 rdev=00:00 objtype=NORMAL cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0
type=PATH msg=audit(1707508261.523:6001811): item=0 name="/usr/bin/crontab" inode=17319856 dev=fd:00 mode=0104755 ouid=0 ogid=0 rdev=00:00 objtype=NORMAL cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0
type=CWD msg=audit(1707508261.523:6001811): cwd="/home/asterisk"
type=EXECVE msg=audit(1707508261.523:6001811): argc=2 a0="/usr/bin/crontab" a1="-l"
type=SYSCALL msg=audit(1707508261.523:6001811): arch=c000003e syscall=59 success=yes exit=0 a0=17eae70 a1=17eb3f0 a2=17eaf40 a3=7ffe43b932e0 items=2 ppid=8137 pid=8138 auid=999 uid=999 gid=1000 euid=0 suid=0 fsuid=0 egid=1000 sgid=1000 fsgid=1000 tty=(none) ses=205427 comm="crontab" exe="/usr/bin/crontab" key=(null)
Nice debug! It’s surely some of the freepbx code (asterisk is the default user, doesn’t mean it is asterisk itself). I have no idea what is causing this but I highly suspect it is the fwconsole process.
Anyway switching to Job is the right thing, using Cron is deprecated for a long time now (see https://help.sangoma.com/community/s/article/FreePBX-OpenSource-Project-Job)
I may be sharing things that people already know here… sorry if that’s the case but I’ve started down a bit of a rabbit hole and am learning new things… who knew auditd was so useful!
so it looks like we have;
argc=3 a0=sh a1=-c a2=/usr/bin/crontab -l 2>/dev/null
that is called by
argc=5 a0=php a1=/usr/sbin/fwconsole a2=job a3=--run a4=--quiet
at that point I’m getting lost but it looks like that might be called by crond but that doesn’t make much sense in my head … crond calling crontab calling the fwconsole job makes sense but the ppid’s go the wrong way for that…
my brain hurts.
so (still learning here…) fwconsole job --run seems to run all pending jobs. So I guess that is being constantly run by crond and one of the jobs listed in there (fwconsole job --list) is the one doing all the crontab stuff…
grep -R -n 'crontab -l' /var/www/html
gives
grep: /var/www/html/admin/images/bullet.png: No such file or directory
grep: /var/www/html/admin/images/bullet_checked.png: No such file or directory
/var/www/html/admin/libraries/utility.functions.php:494: $exec = '/usr/bin/crontab -l ' . $cron_user;
/var/www/html/admin/modules/sysadmin/bin/letsencrypt:552: if ! crontab -l | grep 'letsencrypt renewAll' ; then
/var/www/html/admin/modules/sysadmin/bin/letsencrypt:553: crontab -l | { cat; echo "0 0 * * * /bin/letsencrypt renewAll"; } | crontab -
/var/www/html/admin/modules/sysadmin/bin/letsencrypt:569: if ! crontab -l | grep 'letsencrypt renewAll' ; then
/var/www/html/admin/modules/sysadmin/bin/letsencrypt:570: crontab -l | sed "letsencrypt renewAll/d" | crontab -
Looking into utility.functions.php from around lne 466 (for me) there is a function to manage crontab entries. It has a $remove paramater, worth investigating?
grep -R -n 'edit_crontab' /var/www/html
leads me to
/var/www/html/admin/modules/queues/functions.inc/cron.php
That does seem to have a line that “remove all stale backup’s” (which I think should be backups) that might be the cause of the backup problems maybe but I’m getting out of my depth a little here…
Sorry, to be clear I think that it is the right thing long term but I also think that if something id nixing lines from config files then it should probably be bottomed out rather than just left flapping around knackering things up…
Noticed in /var/log/cron today,
Feb 12 07:44:14 uc-x crontab[29594]: (asterisk) LIST (asterisk)
Feb 12 07:44:14 uc-x crontab[29596]: (asterisk) LIST (asterisk)
Feb 12 07:44:15 uc-x crontab[29622]: (asterisk) LIST (asterisk)
Feb 12 07:44:15 uc-x crontab[29624]: (asterisk) LIST (asterisk)
Feb 12 07:44:15 uc-x crontab[29626]: (asterisk) LIST (asterisk)
Feb 12 07:44:15 uc-x crontab[29627]: (asterisk) REPLACE (asterisk)
Feb 12 07:44:15 uc-x crontab[29629]: (asterisk) LIST (asterisk)
Feb 12 07:44:15 uc-x crontab[29631]: (asterisk) LIST (asterisk)
Feb 12 07:44:15 uc-x crontab[29632]: (asterisk) REPLACE (asterisk)
Feb 12 07:44:15 uc-x crontab[29634]: (asterisk) LIST (asterisk)
...
every minute. I don’t know why this is happening but it looks like either a bug or a really sloppy implementation.
This is still a problem for a number of my freepbx boxes.
When I try to renew it under Certificate management, it says Nothing to do, No Changes made
I just followed @PYoung commands and it’s now updated.
Just wanted to mention here that I was editing the file /var/spool/cron/asterisk to add the cron job to renew certificates as posted by @dobrosavljevic and when I attempted to save it, it said that the file was changed while I had it open (in nano). I had only been inside the file for no more than 2ish minutes.
It certainly appears that the asterisk user crontab entry is being read and re-written, possibly every 60 seconds by something.(Although @billsimon seems to be getting it multiple times a second!)
I ran out of brain when I was looking a week or so ago, it would be really useful if we could get some input from the actual dev / customer service team but it looks like they aren’t here very often?
I’ve been working on some ansible stuff to monitor and “fix” things when they go wrong. Will post when it’s somewhere near ready.
You should have a cron job running /usr/sbin/fwconsole job every minute. One of the things it does is rewrite the crontab.
Hey @miken32 do you have any insight as to why the crontab is being read and re-written every min? Or why that might start happening multiple times a min? (as seems to be the case from the audit logs I recorded)
Something is managing to remove lines from the crontab for multiple users on multiple machines so it would be really good to work out what and why.
Can you think of any serious adverse affects that would occur from disabling the cron re-write?
So the cron job that calls the job task is:
* * * * * [ -e /usr/sbin/fwconsole ] && sleep $((RANDOM\%30)) && /usr/sbin/fwconsole job --run --quiet 2>&1 > /dev/null
* * * * * Run every minute
[ -e /usr/sbin/fwconsole ] check if fwconsole exist
sleep $((RANDOM\%30)) Wait a random delay from 0 to 29 seconds
/usr/sbin/fwconsole job --run --quiet Run the actual command
2>&1 > /dev/null Dump all output to the abyss because cron hates output.
So fwconsole job --run --quiet Loops through all the Job classes registered and does whatever is inside. One of the modules touches the crontab if not Job itself (I didn’t look at the code).
tl;dr the “job” job runs every minute +/- 29 seconds and one of the tasks is saying do a thing and it does.
It’s just making sure the scheduled jobs are run and that nobody accidentally wipes out the cron job. See FreePBX\Job::init() method for where it happens. The only line it’s removing is one that matches “fwconsole job --run” so if something is removing random lines from your crontab, that isn’t it.
FYI, the JOBSRANDOMSLEEP variable is set to 0 by default, with no random sleep before running the job. You must have set it to 30 at some point.
pulled from a random server I have in my dev rack god only knows what it has had done to it lol ¯\_(ツ)_/¯
Not understanding what thinking would rewrite the crontable, if there is an other condition to check, it should be done in the job, been quite a while to have tihs obvious ‘bug’ squished though ;-),
Although I don’t have that problem , a likely work around would be to make the table ‘immutable’, anybody tried that yet?
I’ve been following this thread, because it seems to be very similar to my problem here. Scheduled Backup not working - #5 by dobrosavljevic
I have not tried your suggestion of making the file immutable, because I’m not sure if it would cause any other issues.
It’s definitely related to the same bug.
I haven’t done it because it’s a pretty crude workaround and would break some things. For example if you tried to change the backup schedule or some other change in the system that’s managed by that file it probably wouldn’t work.