Got a FreePBX system (Sangoma distro), and every few weeks (I believe always on Mondays) we come in and find the FreePBX GUI not working, and ultimately it’s because MySQL is down.
Checking the mariadb.log we can find the cause is being unable to allocate memory for the buffer pool. The system is not anywhere close to being out of disk space and the system should not otherwise be out of ram (It’s a 3 phone installation with light call volume, host has 1 core and 2gb ram on SSD). I am unsure what is triggering the condition, but the easiest fix is to reboot the host. I’m curious if it could be a memory leak somewhere which explains why after a couple weeks the condition is triggered. After a fresh reboot it’s showing 873MB free. I need to check that figure once a week to confirm it’s a memory leak I guess, but I don’t have this problem with any other freepbx system, including some larger/more busy ones on similar specs.
Reading the log it appears that some kind of maintenance is taking place at around 3AM and then the problem happens.
Can anyone interpret the log (I included everything for today, including the events leading up to the failure in case they are relevant) and offer any ideas?
InnoDB: Fatal error: cannot allocate memory for the buffer pool
That error means that MySQL couldn’t allocate the needed memory for the buffer pool. According to your log, the pool is configured to use 128m, so that would mean that there was no RAM available to allocate 128m to the InnoDB engine. Even though 2gb of RAM should be enough, you might be on the edge of RAM availability. Do you have the possibility to add more RAM? Also, have you checked if your installation has a swap filesystem configured?
While your point about ram might be valid, I have dozens of other VM’s all built from the same image with more extensions and far busier activity with zero issues. This particular box is apparently running out of RAM over time.
I suspect memory leak as after a reboot we’ve got 800M free memory. If the pool is 128mb, then over time the ram usage is growing to a point where there is less than that.
I’m going to start checking the available RAM every couple days and see if I can see a trend and can pinpoint who is getting hungry here. I’ll also tweak the swap to help out too if this is a temporal issue.
FWIW I pulled the old resource usage log entries, and can see everything is “normal” up until 2am when just about all RAM is consumed AND swap.
Doesn’t appear to be a memory leak, instead “something” runs and consumes everything. Will bump up swap and see if that helps.
Anyone have an idea as to how I can find out WHAT process ran at that time that would be that demanding? I thought it was auto-updates previously, so I’ve already disabled those as a temporary test.
All “Auto Updates” are set to “Email Only”, but interestingly the system is fully up to date and it’s been a couple weeks since I ran updates manually. Wondering if that’s related somehow.