FreePBX 17 Slow Shutdown (thread #3) [Solved]

[notorious.dds]

Hi I’ve been having the same slow shutdown issue as has been descirbed both here:

FreePBX 17 - Slow Shutdown

I was wondering if anyone else is seeing a slow shutdown on their new FreePBX 17 setups? We’ve migrated over nine of our PBXes now, and all are VMs running in Hyper-V. We’ve noticed that when we ask for a shutdown or restart from System Admin, it drops the ssh session and the Hyper-V console window changes to a blinking cursor. Then it seems to wait for about two or three minutes before the shutdown or reboot occurs. I have seen that with previous versions, but with FreePBX 17 it seems to be con…

and here:

FreePBX 17 - Slow Shutdown, second try

Hi, first time poster. English is not my first language. So please be kind, if I do wrong or stupid things. I found out, that my current problem seems to be the same as described here: [FreePBX 17 - Slow Shutdown] My freepbx needs nearly 4 Minutes to shut down (including the vm). Thats way to long for the currently configured USV which shuts down Power 180 seconds after sending the shutdown command … anyway. I googled a bit and found some interesting things not mentioned in the abouve Thread…

For clarity, I’m running FreePBX 17 on a VM (ESXi 8.3). My freepbx system was installed using the ISO provided by Sangoma.

As was pointed out by @utelit in the second link above, it appears that the problem seems to be more associated with how the freepbx service is started at boot vs how it actually shuts down. The rationale for this conclusion stems from the fact that so long as you restart freepbx at some time after the system boots, it will shut down normally. But, if you never restart it and simply try to shutdown (or reboot) the system, asterisk will hang until systemd hits its timeout and ultimately sends the sigkill signal some 2+ minutes later.

Further, it also appears that the hangup has something to do with mariadb. If, after booting up, I simply execute:
systemctl stop mariadb && shutdown
I can get the system to shut down in ~10 seconds.

To the best that I can tell, the problem seems to stem from the freepbx systemd service requiring that mariadb be running before it starts and also requiring that it be stopped before it stops. This goes against the default sequencing of systemd (e.g. That which starts earlier, stops later).

At any rate, I was able to create an override systemd drop-in file that makes things work as desired. See below.

The following commands need to be run as root, so prepend with ‘sudo’ or get to a root prompt.

1. Create override drop-in systemd file

systemctl edit freepbx

2. With the file open, add the following lines near the top. The file will tell you below and above which lines you need to place the new lines. BE SURE TO PUT THEM WHERE THE FILE INDICATES.

[Service]
ExecStartPre=/usr/bin/systemctl start mariadb

ExecStop=
ExecStop=/bin/sh -c '/usr/bin/kill -TERM $(/usr/bin/systemctl show --property MainPID --value mariadb)'
ExecStop=/usr/sbin/fwconsole stop

3. Save the file and then reload systemd
   systemctl daemon-reload

For those who are curious as to what this change is doing, essentially…

• ExecStop= clears the ExecStop command in the default service file. Without this, systemd will still try to stop freepbx before mariadb.
• The second ExecStop command stops mariadb and the third one stops freepbx.
• The ExecStartPre= command is now needed because without it, you cannot simply restart freepbx. In its absence, mariadb would be shutdown when freepbx shuts down, but it needs to be running when freepbx trys to start and there would be nothing telling it to do so.

At any rate, after doing this my system shut down in < 10 seconds vs 2.5+ minutes.

[michiganbroadband (N8LBV)]

Is this setting (in advanced settings) a factor?
This implies it would only be needed if there are calls in progress, and otherwise the delay here should not be applied.

But then again it is only implied.
I think the default is 200 seconds.

[notorious.dds]

Is this setting (in advanced settings) a factor?

I don’t think so. If you simply execute a
fwconsole restart
at some point after the system boots, asterisk won’t hang on shutdown (regardless of how much time passes bewteen restarting freepbx and restarting the system… as far as I can tell).

But, if you boot the system and attempt to restart it any point without having restarted freepbx at some point, it hangs.

What the difference is between freepbx starting at boot time vs starting/restaring at any other time is beyond me. And thus, I have no idea how to explain the default behavior that I’m seeing.

But, I can say with conviction that (at least with my system) freepbx won’t stop via system shutdown (if it’s the first shutdown of freepbx since boot) without fwconsole throwing an “Asterisk is still running and we can’t stop it!” error. That is to say… the freepbx service won’t stop until the systemd SIGTERM signal sent to freepbx times out after ~ 2+ minues at which point systemd sends the more agressive SIGKILL signal which does finally kill freepbx and allow the system to continue shutting down.

If the “Waiting Period to Stop Asterisk” setting you referenced above was the core of the issue, I’d expect the behavior to be present at all shutdowns, regardless of whether freepbx had ever been restarted or not.

[penguinpbx (C. Maj)]

Howdy! Welcome to the forums.

@notorious.dds

I was able to create an override systemd drop-in file that makes things work as desired

That’s a nice hack however, stopping MariaDB before FreePBX might result in lost CDRs, CELs, or other data.

[notorious.dds]

That’s a nice hack however, stopping MariaDB before FreePBX might result in lost CDRs, CELs, or other data.

Yeah… I realize that.

However, the alternative is simply to allow systemd to hit freepbx with a SIGKILL signal 2.5 minutes later essentially yielding all those same concerns.

So, as I see it… it’s six in one and a half dozen in the other. However, my approach takes < 10 seconds whereas the default process takes 2.5+ minutes.