FreePBX | Register | Issues | Wiki | Portal | Support

Mysql Server has Gone Away


#1

Received the following error which crashed my server and required a reboot to get things back up and running again – any ideas how to fix this?

PDOException: SQLSTATE[HY000]: General error: 2006 MySQL server has gone away in file /var/www/html/admin/libraries/modulefunctions.class.php on line 698 Stack trace:

  1. PDOException->() /var/www/html/admin/libraries/modulefunctions.class.php:698
  2. PDOStatement->execute() /var/www/html/admin/libraries/modulefunctions.class.php:698
  3. module_functions->getinfo() /var/www/html/admin/libraries/BMO/Modules.class.php:315
  4. FreePBX\Modules->getInfo() /var/www/html/admin/libraries/BMO/Self_Helper.class.php:193
  5. FreePBX\Self_Helper->loadObject() /var/www/html/admin/libraries/BMO/Self_Helper.class.php:106
  6. FreePBX\Self_Helper->autoLoad() /var/www/html/admin/libraries/BMO/Self_Helper.class.php:37
  7. FreePBX\Self_Helper->__get() /var/www/html/admin/libraries/BMO/FreePBX.class.php:105
  8. FreePBX->__callStatic() /var/www/html/admin/modules/timeconditions/bin/schedtc.php:8
  9. FreePBX->Timeconditions() /var/www/html/admin/modules/timeconditions/bin/schedtc.php:8

(Itzik) #2

Is your PBX up to date?
What’s your server specs?


#3

I am running the Free PBX Gui V14 on a Centos server

i just checked and a few modules needed to be updated – including core and dashboard

Yum updates seem to be clear

Just ran the updates now – do you think that was the cause of the issue?

thanks


(Itzik) #4

Physical, VM, ram, processor, disk etc?


#5

I am hosting the server on a 2GB Linode vm

Thanks


(Tom Ray) #6

Is mysql running?


#7

I rebooted the server so it’s running now

Assume it was not when the server crashed - when it crashed the cpu usage spiked to 90 plus percent and the whole system locked up

I was only able to reboot it from the Linode Mgmt console

Thanks


(Dave Burgess) #8

There is an error log file in the MySQL database directory. I’d start checking that first.


#9

Mar 30 14:44:07 schoolhousepbx systemd: systemd-logind.service: main process exited, code=exited, status=1/FAILURE Mar 30 14:45:37 schoolhousepbx systemd: systemd-logind.service stop-sigterm timed out. Killing. Mar 30 14:47:07 schoolhousepbx systemd: systemd-logind.service still around after SIGKILL. Ignoring. Mar 30 14:48:37 schoolhousepbx systemd: systemd-logind.service stop-final-sigterm timed out. Killing. Mar 30 14:50:07 schoolhousepbx systemd: systemd-logind.service still around after final SIGKILL. Entering failed mode. Mar 30 14:50:07 schoolhousepbx systemd: Failed to start Login Service. Mar 30 14:50:07 schoolhousepbx systemd: Unit systemd-logind.service entered failed state. Mar 30 14:50:07 schoolhousepbx systemd: systemd-logind.service failed. Mar 30 14:50:07 schoolhousepbx systemd: systemd-logind.service has no holdoff time, scheduling restart. Mar 30 14:50:07 schoolhousepbx systemd: Stopped Login Service. Mar 30 14:50:07 schoolhousepbx systemd: Starting Login Service... Mar 30 14:50:07 schoolhousepbx systemd-logind: Failed to register name: File exists Mar 30 14:50:07 schoolhousepbx systemd-logind: Failed to fully start up daemon: File exists Mar 30 14:50:07 schoolhousepbx systemd: systemd-logind.service: main process exited, code=exited, status=1/FAILURE Mar 30 14:51:37 schoolhousepbx systemd: systemd-logind.service stop-sigterm timed out. Killing. Mar 30 14:53:07 schoolhousepbx systemd: systemd-logind.service still around after SIGKILL. Ignoring. Mar 30 14:54:38 schoolhousepbx systemd: systemd-logind.service stop-final-sigterm timed out. Killing. Mar 30 14:56:08 schoolhousepbx systemd: systemd-logind.service still around after final SIGKILL. Entering failed mode. Mar 30 14:56:08 schoolhousepbx systemd: Failed to start Login Service. Mar 30 14:56:08 schoolhousepbx systemd: Unit systemd-logind.service entered failed state. Mar 30 14:56:08 schoolhousepbx systemd: systemd-logind.service failed. Mar 30 14:56:08 schoolhousepbx systemd: systemd-logind.service has no holdoff time, scheduling restart. Mar 30 14:56:08 schoolhousepbx systemd: Stopped Login Service. Mar 30 14:56:08 schoolhousepbx systemd: Starting Login Service... Mar 30 14:56:08 schoolhousepbx systemd-logind: Failed to register name: File exists Mar 30 14:56:08 schoolhousepbx systemd-logind: Failed to fully start up daemon: File exists Mar 30 14:56:08 schoolhousepbx systemd: systemd-logind.service: main process exited, code=exited, status=1/FAILURE Mar 30 14:57:38 schoolhousepbx systemd: systemd-logind.service stop-sigterm timed out. Killing. Mar 30 14:59:08 schoolhousepbx systemd: systemd-logind.service still around after SIGKILL. Ignoring. Mar 30 15:00:38 schoolhousepbx systemd: systemd-logind.service stop-final-sigterm timed out. Killing. Mar 30 15:02:08 schoolhousepbx systemd: systemd-logind.service still around after final SIGKILL. Entering failed mode. Mar 30 15:02:08 schoolhousepbx systemd: Failed to start Login Service. Mar 30 15:02:08 schoolhousepbx systemd: Unit systemd-logind.service entered failed state. Mar 30 15:02:08 schoolhousepbx systemd: systemd-logind.service failed. Mar 30 15:02:08 schoolhousepbx systemd: systemd-logind.service has no holdoff time, scheduling restart. Mar 30 15:02:08 schoolhousepbx systemd: Stopped Login Service. Mar 30 15:02:08 schoolhousepbx systemd: Starting Login Service... Mar 30 15:02:08 schoolhousepbx systemd-logind: Failed to register name: File exists

I am seeing the above error over and over in the logs – wondering if this is the root cause of the problem

I guess i can do the following to turn it off – but wanted to get your feedback before doing that

systemctl mask systemd-logind.service
pam-auth-update

thanks,


(Dave Burgess) #10

You are running out of resources. I don’t know if it’s PIDs, memory, disk, or what, but you are definitely running out of resources.


#11

Interesting – its a very small box with only a few extensions – the 2GB Linode should have enough resources?

What would the next steps be in terms of trouble shooting this?

thanks,


(Dave Burgess) #12

Check the MySQL error log in the MySQL data directory. It should give you some ideas.

Also, 2GB of RAM is barely enough to load and install the system. Running it in less than 4G is a technical challenge.


#13

Interesting – i have read multiple reports where people have said that a small system with less than 10 extensions should be fine on a similarly sized 2GB linode - have even seen some people running such a system with only 1GB

I looked in /var/log/mariadb/mariadb.log – but didnt really see anything

thanks,


(Dave Burgess) #14

I’m not as familiar with MariaDB, but since it’s a fork of MySQL, I’d expect the error log to be in the data directory with the FRM files. I expect something like “hostname_err.log”.

There was a thread on here about a year ago where someone was trying to go as small as possible, and (IIRC) the consensus was that 2G was the smallest install footprint for Asterisk and FreePBX. Smaller installations were possible with other Asterisk system (minus FreePBX) but I’m pretty sure those were the good old days.

The number of extensions isn’t really a critical element - sure, more extensions need more memory, but the overhead and the fixed/variable cost after the first phone in terms of memory is pretty forgiving. With everything that FreePBX has loaded (SNG-7, two firewalls, databases, asterisk, etc.) I just wouldn’t be comfortable trying to shoe-horn all of that into just 2G.


#15

ok – so to clarify – i am not using the distro – this is a centos 7 server with asterisk and Freepbx installed manually

second i just ran

top
echo [PID] [MEM] [PATH] && ps aux | awk ‘{print $2, $4, $11}’ | sort -k2rn | head -n 20
ps -eo pcpu,pid,user,args | sort -k 1 -r | head -20

and under the 3 commands MYSQL tops out at around 10% of memory

also i am using mysql as far as i can tell:

mysql /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --log-error=/var/log/mariadb/mariadb.log --pid-file=/var/run/mariadb/mariadb.pid --socket=/var/lib/mysql/mysql.sock

thanks


(Dave Burgess) #16

There should be (in my experience) a file in this directory called “something.err”. In there will be the actual failure for the database crash that killed your MySQL server back 14 messages ago.

I know I’m just a user, but sooner or later it’s going to come down to resources. Your failures are all common (across platforms and systems) and consistent with a system that is running out of memory or PIDs. I understand you want it to be something else, but I think you are just going to end up disappointed. The appearance that the problem moves around from process to process is my clue to this.

If I was a true “Kool-Aid drinking” FreePBX user, I’d tell you it’s because you’re running a roll-your-own system and leave you hanging. I’m not sure it isn’t that (the packaged software is chosen and tested as a unit, so …) but at this point, I’m leaning more toward resource starvation.

https://community.spiceworks.com/topic/575275-specs-for-a-50-line-asterisk-pbx is a good discussion of the how and why for sizing a system, and that’s before you start adding all of the other stuff that the newer FreePBX versions are adding.


#17

For what it’s worth, I have dozens of instances of opensource FreePBX running on 1G $5/month Vultr instances, they have been doing that for two or three years with up to 200 users each (they also have t38modem, hylafax servers and email2fax integrated) .

Unless you have “added” stuff or perhaps use some ‘expensive’ add-ons, then unless anyone can prove otherwise, (yum/apt install sysstat and then sar) I believe you have a ‘more than enough’ server.

Please analysts, show where resources are “running out” (again sar can be indicative in the 40000 foot view)

Be careful with your mysql daemon, on some systems that means mysql on others then mariadb masquerades itself, make sure you don’t have both installed.

JM2CWAE


#18

thanks dicko – I agree

I installed asterisk / freepbx based on the roll your own instructions on the FPBX website – nothing special or extra really

https://wiki.freepbx.org/display/FOP/Installing+FreePBX+14+on+CentOS+7#InstallingFreePBX14onCentOS7-EnableandStartMariaDB

i also believe based on running several analytics tools that i have more then enough horse power for the amount of users and trunks i have on this system

only mariadb is running and not mysqld

something definitely caused the cpu usage to spike and the server to crash – just need to figure out what

thanks,


#19

As I said, sysstat can help. It analyses every minute your resource usage, when things peg out, it shows the general area of overload, you will see the trend with sar , mostly the last thing to die on overload is sysstat itself.


#20

dicko,

On your Vultr instances – what is the size of your swap partition?

thanks,