Cron job for backup disappears

[jsv (J V)]

I’ve run into an issue with the Backup & Restore module not running a scheduled backup. The backup can be successfully run manually. When it doesn’t run, the backup failure notification email doesn’t get sent either. I only clued in on the failure since I run a script on the backup storage server that checks for fresh backups and notifies if they aren’t there.

I found that the job disappears from /var/spool/cron/asterisk, and I’m not sure what is removing it. If I change the timing on the backup by a minute and save it, it gets put back into the cron config, works for a couple days, then disappears again. This started happening on one system, but yesterday started happening on a 2nd system.

Both systems are FreePBX 15.0.17.64 and use the Backup & Restore module version 15.0.10.68.

Any ideas?

[davidg]

Is the value being replaced by something else or just disappearing?

[jsv (J V)]

It was disappearing altogether. I’ve been checking the /var/spool/cron/asterisk file daily to see if it disappears, and today I found something interesting.

Yesterdays:

* 1 * * * /usr/sbin/fwconsole util cleanplaybackcache -q
14 1 * * 0 [ -e /usr/sbin/fwconsole ] && /usr/sbin/fwconsole ma listonline --sendemail -q > /dev/null 2>&1
14 3 * * 0 [ -e /usr/sbin/fwconsole ] && /usr/sbin/fwconsole ma upgradeall --sendemail -q > /dev/null 2>&1
* * * * * [ -e /usr/sbin/fwconsole ] && sleep $((RANDOM\%30)) && /usr/sbin/fwconsole job --run --quiet 2>&1 > /dev/null
58 1 * * * /usr/sbin/fwconsole backup --backup=b38d473c-6e92-46ba-a43c-a140c96bb6b2  > /dev/null 2>&1

Todays:

* 1 * * * /usr/sbin/fwconsole util cleanplaybackcache -q
14 1 * * 0 [ -e /usr/sbin/fwconsole ] && /usr/sbin/fwconsole ma listonline --sendemail -q > /dev/null 2>&1
14 3 * * 0 [ -e /usr/sbin/fwconsole ] && /usr/sbin/fwconsole ma upgradeall --sendemail -q > /dev/null 2>&1
58 1 * * * /usr/sbin/fwconsole backup --backup=b38d473c-6e92-46ba-a43c-a140c96bb6b2  > /dev/null 2>&1
* * * * * [ -e /usr/sbin/fwconsole ] && sleep $((RANDOM\%30)) && /usr/sbin/fwconsole job --run --quiet 2>&1 > /dev/null

The backup job (after being updated and saved in the modue, which adds it the cron config again) was the last item in the cron config yesterday. No longer the case today. So whatever is automatically updating this file strips out the backup job sometimes.

[jfinstrom (James Finstrom)]

The backup is in both of those blocks, just reordered

[jsv (J V)]

Apparently this system was compromised with this:

K.php - a RestApps malicious script

If you were impacted by the restapps security regression a week or two ago, it is possible you were hit with a php script that is currently labeled “k.php” If you want to see if you were compromised by this script, I’ve included some content below that you can check. I do not claim to have identified everything, but I’m hoping this might help someone. Anyways, here are the places you can look. crontab -e -u root → if this line exists, you’re compromised. DO NOT run this for testing. */1 * …

Was notified by the backup server that the backup file for last night was missing. Checked the cron file and the only thing in it was a wget to the malicious script.

[dicko]

Unless you are very confident that you have purged EVERY part of that compromise (and that is not trivial) , I suggest you consider your system failed, restore a backup taken prior to the compromise or rebuild from scratch, and know that all your ‘family jewels’ are now compromised and need to be replaced piecemeal but completely.

I would take the opportunity to install a rootkit detector at this point in time ‘just in case’/‘before’ it happens again