Received the following error which crashed my server and required a reboot to get things back up and running again – any ideas how to fix this?
PDOException: SQLSTATE[HY000]: General error: 2006 MySQL server has gone away in file /var/www/html/admin/libraries/modulefunctions.class.php on line 698 Stack trace:
Mar 30 14:44:07 schoolhousepbx systemd: systemd-logind.service: main process exited, code=exited, status=1/FAILURE
Mar 30 14:45:37 schoolhousepbx systemd: systemd-logind.service stop-sigterm timed out. Killing.
Mar 30 14:47:07 schoolhousepbx systemd: systemd-logind.service still around after SIGKILL. Ignoring.
Mar 30 14:48:37 schoolhousepbx systemd: systemd-logind.service stop-final-sigterm timed out. Killing.
Mar 30 14:50:07 schoolhousepbx systemd: systemd-logind.service still around after final SIGKILL. Entering failed mode.
Mar 30 14:50:07 schoolhousepbx systemd: Failed to start Login Service.
Mar 30 14:50:07 schoolhousepbx systemd: Unit systemd-logind.service entered failed state.
Mar 30 14:50:07 schoolhousepbx systemd: systemd-logind.service failed.
Mar 30 14:50:07 schoolhousepbx systemd: systemd-logind.service has no holdoff time, scheduling restart.
Mar 30 14:50:07 schoolhousepbx systemd: Stopped Login Service.
Mar 30 14:50:07 schoolhousepbx systemd: Starting Login Service...
Mar 30 14:50:07 schoolhousepbx systemd-logind: Failed to register name: File exists
Mar 30 14:50:07 schoolhousepbx systemd-logind: Failed to fully start up daemon: File exists
Mar 30 14:50:07 schoolhousepbx systemd: systemd-logind.service: main process exited, code=exited, status=1/FAILURE
Mar 30 14:51:37 schoolhousepbx systemd: systemd-logind.service stop-sigterm timed out. Killing.
Mar 30 14:53:07 schoolhousepbx systemd: systemd-logind.service still around after SIGKILL. Ignoring.
Mar 30 14:54:38 schoolhousepbx systemd: systemd-logind.service stop-final-sigterm timed out. Killing.
Mar 30 14:56:08 schoolhousepbx systemd: systemd-logind.service still around after final SIGKILL. Entering failed mode.
Mar 30 14:56:08 schoolhousepbx systemd: Failed to start Login Service.
Mar 30 14:56:08 schoolhousepbx systemd: Unit systemd-logind.service entered failed state.
Mar 30 14:56:08 schoolhousepbx systemd: systemd-logind.service failed.
Mar 30 14:56:08 schoolhousepbx systemd: systemd-logind.service has no holdoff time, scheduling restart.
Mar 30 14:56:08 schoolhousepbx systemd: Stopped Login Service.
Mar 30 14:56:08 schoolhousepbx systemd: Starting Login Service...
Mar 30 14:56:08 schoolhousepbx systemd-logind: Failed to register name: File exists
Mar 30 14:56:08 schoolhousepbx systemd-logind: Failed to fully start up daemon: File exists
Mar 30 14:56:08 schoolhousepbx systemd: systemd-logind.service: main process exited, code=exited, status=1/FAILURE
Mar 30 14:57:38 schoolhousepbx systemd: systemd-logind.service stop-sigterm timed out. Killing.
Mar 30 14:59:08 schoolhousepbx systemd: systemd-logind.service still around after SIGKILL. Ignoring.
Mar 30 15:00:38 schoolhousepbx systemd: systemd-logind.service stop-final-sigterm timed out. Killing.
Mar 30 15:02:08 schoolhousepbx systemd: systemd-logind.service still around after final SIGKILL. Entering failed mode.
Mar 30 15:02:08 schoolhousepbx systemd: Failed to start Login Service.
Mar 30 15:02:08 schoolhousepbx systemd: Unit systemd-logind.service entered failed state.
Mar 30 15:02:08 schoolhousepbx systemd: systemd-logind.service failed.
Mar 30 15:02:08 schoolhousepbx systemd: systemd-logind.service has no holdoff time, scheduling restart.
Mar 30 15:02:08 schoolhousepbx systemd: Stopped Login Service.
Mar 30 15:02:08 schoolhousepbx systemd: Starting Login Service...
Mar 30 15:02:08 schoolhousepbx systemd-logind: Failed to register name: File exists
I am seeing the above error over and over in the logs – wondering if this is the root cause of the problem
I guess i can do the following to turn it off – but wanted to get your feedback before doing that
Interesting – i have read multiple reports where people have said that a small system with less than 10 extensions should be fine on a similarly sized 2GB linode - have even seen some people running such a system with only 1GB
I looked in /var/log/mariadb/mariadb.log – but didnt really see anything
I’m not as familiar with MariaDB, but since it’s a fork of MySQL, I’d expect the error log to be in the data directory with the FRM files. I expect something like “hostname_err.log”.
There was a thread on here about a year ago where someone was trying to go as small as possible, and (IIRC) the consensus was that 2G was the smallest install footprint for Asterisk and FreePBX. Smaller installations were possible with other Asterisk system (minus FreePBX) but I’m pretty sure those were the good old days.
The number of extensions isn’t really a critical element - sure, more extensions need more memory, but the overhead and the fixed/variable cost after the first phone in terms of memory is pretty forgiving. With everything that FreePBX has loaded (SNG-7, two firewalls, databases, asterisk, etc.) I just wouldn’t be comfortable trying to shoe-horn all of that into just 2G.
There should be (in my experience) a file in this directory called “something.err”. In there will be the actual failure for the database crash that killed your MySQL server back 14 messages ago.
I know I’m just a user, but sooner or later it’s going to come down to resources. Your failures are all common (across platforms and systems) and consistent with a system that is running out of memory or PIDs. I understand you want it to be something else, but I think you are just going to end up disappointed. The appearance that the problem moves around from process to process is my clue to this.
If I was a true “Kool-Aid drinking” FreePBX user, I’d tell you it’s because you’re running a roll-your-own system and leave you hanging. I’m not sure it isn’t that (the packaged software is chosen and tested as a unit, so …) but at this point, I’m leaning more toward resource starvation.
For what it’s worth, I have dozens of instances of opensource FreePBX running on 1G $5/month Vultr instances, they have been doing that for two or three years with up to 200 users each (they also have t38modem, hylafax servers and email2fax integrated) .
Unless you have “added” stuff or perhaps use some ‘expensive’ add-ons, then unless anyone can prove otherwise, (yum/apt install sysstat and then sar) I believe you have a ‘more than enough’ server.
Please analysts, show where resources are “running out” (again sar can be indicative in the 40000 foot view)
Be careful with your mysql daemon, on some systems that means mysql on others then mariadb masquerades itself, make sure you don’t have both installed.
i also believe based on running several analytics tools that i have more then enough horse power for the amount of users and trunks i have on this system
only mariadb is running and not mysqld
something definitely caused the cpu usage to spike and the server to crash – just need to figure out what
As I said, sysstat can help. It analyses every minute your resource usage, when things peg out, it shows the general area of overload, you will see the trend with sar , mostly the last thing to die on overload is sysstat itself.