How to get automatic updates to run on my schedule? (i.e. monthly)

FreePBX Configuration
1

Hello! So, I started down the rabbit hole of automatic updates via the Admin>Updates>Scheduler & Alerts tab, which is going really well, except that it only has daily and weekly options as shown (is there a reason monthly is no longer available like in older versions?)

Screenshot%20(53)
Screenshot%20(53)1237×490 28.7 KB

I really was wanting monthly automatic updates, unless there is a specific reason that ensures updates are more reliably installed if run every week - well, I went to edit the cron jobs that the automatic updates settings in the GUI writes in with a crontab -e -u asterisk, which appears to be these three lines:

[ -e /usr/sbin/fwconsole ] && /usr/sbin/fwconsole ma listonline --sendemail -q > /dev/null 2>&1
[ -e /usr/sbin/fwconsole ] && /usr/sbin/fwconsole sys upgradeall --sendemail -q > /dev/null 2>&1
[ -e /usr/sbin/fwconsole ] && /usr/sbin/fwconsole ma upgradeall --sendemail -q > /dev/null 2>&1

Of course, the above is just an example without the cron scheduling. I removed the actual times and schedule that goes on the left in cron for the purposes of this post.

Basically, the GUI sets randomly the dates and times, and if I change them to my preferred schedule (monthly), it will soon change them back to a randomly generated value between the day of week and time selected in the GUI (which is expected behavior).

Is there a problem with running monthly automatic updates, and also is there a way to get FreePBX to respect my own scheduling and/or play nicely with such?

2

Disable the gui schedule and use a cron.monthly script or a systemd.timer to

/usr/sbin/fwconsole ma upgradeall && /usr/sbin/fwconsole sys upgradeall && /usr/sbin/fwconsole reload

(Or perhaps have root do them)

3

Use this, it is 2021, and all of the main distributions use systemd now.

4

These are great ideas, thanks guys! Also having that anywhere except for the asterisk crontab would mean the GUI settings would leave them alone too, in case it is looking to disable them there.

I also like how you include the fwconsole reload at the end - I’ve noticed by my own experience and other posts that the config is not applied (and the red apply config button hangs around in the GUI) after automatic updates as for some reason this isn’t included as part of the automatic updates process when setup by the GUI.

I also have a less frequent fwconsole restart programmed in with mine too…just for those bigger things mentioned that need that as well. :slight_smile:

Thanks!!

5

I know this is more of a deep dive linux question, but does anyone know what this is for in the default FreePBX GUI added cron entries?

6

I think I may have found it: shell - What does "if [ -e $name ]" mean? Where $name is a path to a directory - Unix & Linux Stack Exchange

7

That is just a check that fwconsole actually ‘exists (-e)’ at /usr/sbin/fwconsole (it will almost certainly be there) , if it doesn’t then && (which is a logical AND) will fail so nothing further gets done

1 Like
8

Okay, I have a (very quiet) line of code that I think will work well when placed somewhere. Complete with an fwconsole reload followed by an fwconsole restart.

[ -e /usr/sbin/fwconsole ] && /usr/sbin/fwconsole ma listonline --sendemail -q > /dev/null 2>&1 && /usr/sbin/fwconsole sys upgradeall --sendemail -q > /dev/null 2>&1 && /usr/sbin/fwconsole ma upgradeall --sendemail -q > /dev/null 2>&1 && /usr/sbin/fwconsole reload -q -n > /dev/null 2>&1 && /usr/sbin/fwconsole restart -q -n > /dev/null 2>&1

Should be good for a while so long as the ideal fwconsole commands to update don’t change for a while. In which case, I wouldn’t know not using GUI defined automatic updates, but I’m trying to get subscribed to blog updates or something so maybe I would catch such a change.

That or I could turn on GUI updates occasionally just long enough to cat /var/spool/cron/asterisk or do a crontab -l -u asterisk to see if the the way they are doing updates is the same as mine. Should be good for a long time though (hopefully).

9

I would suggest that you as a ‘concerned user’ don’t want a “very quiet” script (it is not quiet, it sends it’s output to whatver you have configured FreePBX to send informational mail to) unless you ultimately trust what fwconsole ma upgradeall will do, if you do but don’t read your emails then why are you concerned about how the GUI does it’s job?

10

Basically I’d rather not have it quiet, I was writing it that way to keep things quiet for cron and figured the --sendemail tag would take care of things by emailing me.

But, you’re right I guess it would still have to complete in some fashion for that to happen. I do have an SMTP relay setup and receive and look for the emails - are you suggesting that I remove the -q and > /dev/null 2>&1 from each line (and then will cron email me the output)? Or maybe there’s a way to just get emails when things don’t complete properly…What’s best practice to get all error related information to my inbox besides just using those --sendemail tags? (of which, fwconsole reload and fwconsole restart doesn’t support anyways)

11

cron will email the output of any job (if there is one) to the MAILTO= address you define in your crontab, by default that will be the crontab ‘user’ 's account. the --sendmail -q presumably ‘quietens’ the output so cron won’t have any output. .

Unless you have a very good reason to disagree with a weekly rather than a monthly check, I would suggest you leave well alone what you don’t completely understand yet :wink:

12

Makes sense. No, I don’t have a MAILTO= defined but could easily have one and go that route. I’ve experimented enough that’s exactly what it is doing, --sendmail sends the notification emails (oddly enough fwconsole sys upgradeall --sendemail sends them everytime, whereas fwconsole ma listonline --sendemail and fwconsole ma upgradeall --sendemail only send email when there are actually updates available and/or being installed)…and like you said the -q just quietens the console output so cron doesn’t have any to deal with and the > /dev/null 2>&1 of course is just bulletproofing so cron again doesn’t have any output to deal with.

Agreed if I don’t understand…but unless I am missing something I just need to decide if I want to add a MAILTO= line and remove the -q and > /dev/null 2>&1 and let cron email me in case something goes wrong so I can get it straight from the console output.

That or write in / call some sort of custom mail or sendmail script sending me an email letting me know the entire thing from start to finish completed…so if I don’t get that at the end of the other update notification emails I’ll know something is awry…

13

You can of course be as convoluted as you want :slight_smile: , this is linux and there are many ways to do any one thing, the weekly GUI based process works for most everyone but you though.

14

Of course; if there are problems I’d rather only have to deal with it once a month unless FreePBX will operate (and update) more smoothly with weekly updates. I was actually kind of concerned that might be the case (like an implicit recommendation from the devs) when there used to be a monthly option and now it has been removed…

Also - I wanted to schedule in the fwconsole reload and the fwconsole restart and I like that I can chain everything together with && and have it run all at once, those at the end, instead of having to have three separate commands in cron, randomly scheduled an hour apart, and trying to dodge those with my reload and restart commands, and have that extra downtime for restart every week instead of monthly. Also…makes my backup scheduling easier to schedule around a monthly update and I can keep less backups because it will be “known good, working, tried and true” or whatever for a month instead of a week for me to catch it and make sure I have a backup. Just less to deal with I guess…

Thank you all so much for your help. Any thoughts on why the GUI now has weekly updates but in previous versions there was a monthly option?

15

When you better understand, you will realize that a reload before a restart is totally spurious but a restart is probably a bad thing to do in a cron job :slight_smile:

16

You know, I was thinking that was the case, but when I experimented with it the red “Apply Config” button remained in the GUI after a restart. I thought that was enough reason to throw that in there beforehand.

17

Please, just stop randomly ‘doing things’ that really don’t make much sense, you might eventually break what works just fine for everyone else.

18

I really appreciate your advice and your knowledge, however you don’t have to fight with everyone. I’m not trying to prove you wrong about anything or call you out. I know that’s how you sometimes operate with others (and believe me I know and can be guilty as anyone at getting frustrated with others like that), but I’m really just here trying to learn.

That was a controlled experiment to learn - yes risks are involved for sure, and I hate that. I’ve searched the documentation but not everything is covered and I have nowhere else to go but to testing. If the “Apply Config” was supposed to clear after a restart then I guess that’s a bug. It may not be though. If you know why I’d love to learn more and to understand…I’m just trying to learn, not offend anyone with my questions.

Also still interested in the experts like yourself - you are and you and sorvani and so many others carry so much knowledge about this project that is often hard to find (without reverse engineering or digging through code) - but interested to hear from you guys why it’s only weekly updates instead of monthly when the automatic updates were rewritten.

Also since the GUI does do quiet output for everything in cron, is there something other than the --sendmail tag that it may use to notify of any problems that happen?

I know that’s a lot of questions, and you don’t have to answer, but short of reverse engineering or digging through code a lot of it is hard to find. And, I love to hear the opinions of the experts like yourself. Thank you so much for your help and your time!!!

19

Okay! So after a lot of trouble I decided to go manual for a while longer, but I couldn’t stop myself from trying to figure this out and having this semi-ready in case I want to script automatic monthly updates in the future. So here is some kind of a resolution in case people want that (monthly or “your own schedule” updates), maybe they can learn from some of my pain (at their own risk, of course - preferably you’re running on a VM, in the cloud, etc. and can shut things down and make a clean snapshot backup before messing with any of this).

So, I couldn’t find a good way to script certain fwconsole commands (mainly fwconsole reload and fwconsole restart) except for running them as root with a runuser command, and putting them inside of their own bash script, and then calling that script from the root user cron.

I’ve noticed that reloads (and sometimes restarts, or sometimes even full server reboots) are needed after certain module and system updates, respectively. Restarting things like that can also help in the event of an automatic Let’s Encrypt security certificate update, to have things settle and make sure the correct certificate is loaded and being used (it sends notifications about every quarter when it updates saying to restart things), but if you’re running monthly updates and rebooting the server along with it, then the problem should theoretically be solved there as well…provided nothing else breaks along the way here.

Without getting a ton into it, there were permissions or path issues every other way when trying to run all of the fwconsole commands needed from a single place - when run directly from the asterisk crontab fwconsole reload or fwconsole restart will have problems (which is fine since FreePBX manages that - I’d rather not be messing with the asterisk crontab anyway) or when run directly from the root crontab there are some other issues, maybe because the crontab (I think) uses /usr/bin/sh instead of /usr/bin/bash. Anyway, if you put this in your own separate script, say, /opt/mystuff/myscript.sh and make it executable with a chmod +x /opt/mystuff/myscript.sh then you can call this script from the root user crontab (which is empty by default, and a reasonable place to put your own cron jobs since the asterisk one is heavily managed by the system) - or like it was earlier stated cron.monthly, or systemd.timer may be a better route.

I’m still not entirely sure about wrapping these in runuser commands since I think fwconsole uses runuser some itself, but things appeared to work and exit with a successful exit code. Read on though because I did have some major problems at one point during testing, and was glad I had a snapshot backup on hand.

So, your script could look something like this:

[ -e /usr/sbin/fwconsole ] \
&& /usr/sbin/runuser -l root -c ‘/usr/sbin/fwconsole ma listonline --sendemail -q’ > /dev/null 2>&1 && sleep 90 \
&& /usr/sbin/runuser -l root -c ‘/usr/sbin/fwconsole ma upgradeall --sendemail -q’ > /dev/null 2>&1 && sleep 90 \
&& /usr/sbin/runuser -l root -c ‘/usr/sbin/fwconsole reload -q -n’ > /dev/null 2>&1 && sleep 90 \
&& /usr/bin/yum update -y > /dev/null 2>&1 && sleep 90 \
&& /usr/sbin/shutdown -r now > /dev/null 2>&1

of course instead of the yum update line you could use (if your system allows it, I think it either has to be activated or have sysadmin pro module or both - I can’t remember)

&& /usr/sbin/runuser -l root -c ‘/usr/sbin/fwconsole sys upgradeall --sendemail -q’ > /dev/null 2>&1 && sleep 90 \

but I just hated to wrap the system updater inside of another runuser command (but still the only way it would work - I tested by turning off all of the -q tags and > /dev/null 2>&1 and receiving the output via a crontab MAILTO= line), which you may or may not want to figure out a way to get output, such as scripting in an email to yourself with status on either success or failure.

I noticed that it is often recommended to do module updates before system updates, and even when enabled in the GUI FreePBX puts system ahead of module updates in the asterisk crontab, so the order is module then system here, and also I noticed that by not doing either an fwconsole restart or rebooting the server after updates sometimes there can be some minor firewall issues (at least there was for me) so I have a full restart in here at the end.

Be sure and back up with a snapshot before you try any of this. At some point in my testing things seemed to fry and no amount of restarts would hold a phone call. It seems like testing these commands should never have caused that, but maybe it had something to do with path and permissions issues not being correct when running these from different accounts and etc. as a test before I figured out I had to use my own bash script and runuser as root. I was glad to have a snapshot to quickly fall back on and I wouldn’t recommend for a second testing this without having some sort of snapshot backup system in place to grab a “known good” snapshot a few days before this runs, that way if something breaks you can roll back.

And then of course I almost forgot, to call such a script from the root user’s crontab (crontab -e -u root will let you edit it, just be ready to use vi)

0 0 1 * * [ -e /opt/mystuff/myscript.sh ] && /opt/mystuff/myscript.sh

(Of course that’s midnight on the first of every month - probably best to find another time that is more random than that)

Again, please note that I did come up with MAJOR problems during testing, which I think (but can’t verify) was due to many reloads and restarts with all of the permissions issues and things not being able to finish - this script appears to execute cleanly and properly in my testing so far. However, appearances and reality may be two different things, so if you are interested have a snapshot backup ready to go and use it or parts of it knowing the risks.

20

https://raw.githubusercontent.com/sorvani/freepbx-helper-scripts/master/InitialSetup/update.sh