FreePBX 15 MariaDB very long time to stop on VMware ESXI Hypervisor

Hi guys,

After installing the newest Freepbx15 distro on VMware ESXI Hypervisor, it was taking a very long time to reboot or shutdown the system.
I figured out that the reason is MariaDB server, which hangs during the stopping process and, actually, it stops only after 5 minutes, when it reaches the timeout.

Here is what I can see in /var/log/mariadb/mariadb.log when executing “systemctl stop mariadb” :

200129 11:21:21 [Note] /usr/libexec/mysqld: Normal shutdown
200129 11:21:21 [Note] Event Scheduler: Purging the queue. 0 events
200129 11:21:21 InnoDB: Starting shutdown…
200129 11:22:21 InnoDB: Waiting for worker threads to be suspended
200129 11:23:22 InnoDB: Waiting for worker threads to be suspended
200129 11:24:22 InnoDB: Waiting for worker threads to be suspended
200129 11:25:22 InnoDB: Waiting for worker threads to be suspended

Than, after executing “systemctl start mariadb” I get this output in /var/log/mariadb/mariadb.log :

200129 11:45:36 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
200129 11:45:36 [Note] /usr/libexec/mysqld (mysqld 5.5.60-MariaDB) starting as process 28309 …
200129 11:45:36 InnoDB: The InnoDB memory heap is disabled
200129 11:45:36 InnoDB: Mutexes and rw_locks use GCC atomic builtins
200129 11:45:36 InnoDB: Compressed tables use zlib 1.2.7
200129 11:45:36 InnoDB: Using Linux native AIO
200129 11:45:36 InnoDB: Initializing buffer pool, size = 128.0M
200129 11:45:36 InnoDB: Completed initialization of buffer pool
200129 11:45:36 InnoDB: highest supported file format is Barracuda.
200129 11:45:36 InnoDB: Starting crash recovery from checkpoint LSN=32237726
InnoDB: Restoring possible half-written data pages from the doublewrite buffer…
200129 11:45:36 InnoDB: Starting final batch to recover 40 pages from redo log
200129 11:45:36 InnoDB: Waiting for the background threads to start
200129 11:45:37 Percona XtraDB (http://www.percona.com) 5.5.59-MariaDB-38.11 started; log sequence number 32575033
200129 11:45:37 [Note] Plugin ‘FEEDBACK’ is disabled.
200129 11:45:37 [Note] Server socket created on IP: ‘0.0.0.0’.
200129 11:45:37 [Note] Event Scheduler: Loaded 0 events
200129 11:45:37 [Note] /usr/libexec/mysqld: ready for connections.
Version: ‘5.5.60-MariaDB’ socket: ‘/var/lib/mysql/mysql.sock’ port: 3306 MariaDB Server.

I found some info, that it can be related with the time settings, so have made the appropriate changes, but no luck. Time and date settings seems correct. The output of “timedatectl” is:

Local time: Wed 2020-01-29 12:01:08 +04
Universal time: Wed 2020-01-29 08:01:08 UTC
RTC time: Wed 2020-01-29 12:01:08
Time zone: Asia/Yerevan (+04, +0400)
NTP enabled: yes
NTP synchronized: yes
RTC in local TZ: yes
DST active: n/a

One interesting moment, when I shut down the virtual machine and disable the Ethernet from the Hypervisor settings and power on the VM again, MariaDB starts working great.

I even tried to clone the virtual machine, with malfunctioning MariaDB server to my Desktop PC and run it in VMware Workstation Pro, and it works awesome, even with enabled Ethernet.

Also, I can decrease the timeout time from 300 sec to, let’s say, 30 sec, but the main issue will not be resolved.

I couldn’t find any additional info about the possible reasons of the described issue.

Any help would be highly appreciated. Thanks!

Best.

If you shut down the PBX processes eg. Asterisk, before trying to shutdown Maria, then reboot, does that help speed it up ?

It could be a lingering database handle from Asterisk -> Maria.

Thanks for your comment.

No luck. I tried to stop Asterisk and than stop MariaDB. Got the same result :frowning:

This issue has been coming up on and off on various Virtual Servers for the past few months. I remember at least one thread did come up with a “sort of” solution, but I can’t remember what it was.

You might want to take a stroll down memory lane and see what’s in the archive.

1 Like

Insufficient system resources on the VM ? Need more RAM ? Or at least some temporary swap to/from disk ?

Hi! Not really… The VM has 12GB of RAM :frowning:

As a community, we’ve been talking about this for a couple of years. It seems to me there was some resolution on it, but I can’t for the life of me remember what that resolution is.

The reason I’m not paying attention (even though this seems to come up at least once a quarter) is that I always run on my own hardware and I’ve been doing this since VMs were invented (and I still don’t trust them - it’s an extra layer of complexity that people keep telling me is an improvement, right before they bitch about things like performance and inability to do simple stuff like shutting down a database).

Whew, apparently I needed to vent about that again. Sorry. :slight_smile:

On a serious note - we really have been around and around on this for the past couple of years. It’s not actually a resource thing, it’s something in the way ESXI handles the database.

This week’s Intel side-channel attack resolution recommendation is to please change the CPU, which I’m sure all the five-dollar-foot-long VM hosting “services” are getting to next week. :wink:

1 Like

I’ve installed FreePBX on baremetal and it also can take ~5 Mins to reboot becasue of the MariaDB shutting down. I noticed if I don’t perform any config changes over a long period of time my reboot time is greatly reduced. Kind of makes me think that some sort of housekeeping gets kicked off bringing the MariaDB into a “good” state to be shutdown/rebooted. I know this doesn’t help just wanted to pass along my observations.

Thanks,

Gordon

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.