Mysql won't start after hard shutdown, bug?

FreePBX Distro 5.211.65-1
MySQL Version 5.1.71-log

Had a weird issue today. We had a power failure and this particular FreePBX install didn’t have a battery backup (not my choice) So it encountered a hard shutdown. When it came back up, it had a slew of FreePBX errors. amportal wouldn’t start.

When going to the FreePBX admin site I’d get the following:

FATAL ERROR
DB Error: extension not found
Trace Back
/var/www/html/admin/libraries/db_connect.php:63 die_freepbx()
[0]: DB Error: extension not found

/var/www/html/admin/bootstrap.php:98 require_once()
[0]: /var/www/html/admin/libraries/db_connect.php

/etc/freepbx.conf:9 require_once()
[0]: /var/www/html/admin/bootstrap.php

/var/www/html/admin/config.php:97 include_once()
[0]: /etc/freepbx.conf

When doing an “amportal restart” I’d get the following:

/usr/local/sbin/amportal: line 46: [FATAL]: command not found

/var/lib/asterisk/bin/freepbx_engine: line 98: [FATAL]: command not found
**** WARNING: ERROR IN CONFIGURATION ****
astrundir in ‘/etc/asterisk’ is set to but the directory
does not exists. Attempting to create it with: 'mkdir -p ’

mkdir: missing operand
Try `mkdir --help’ for more information.
**** ERROR: COULD NOT CREATE ****
Attempt to execute 'mkdir -p ’ failed with an exit code of 1
You must create this directory and the try again.

I traced it down to MySQL not starting, even though it says it was. When I’d do a “service mysql restart”, it would always stop successfully, and when it would go to start I’d get “Another MySQL daemon already running with the same unix socket.” Top didn’t show anything running so I knew there was more to that error than it was letting on.

The solution:
I had to remove /var/lib/mysql/mysql.sock and allow the mysql server to recreate it on start.

In my opinion, this is a bug. On start, that file should always be cleared out. In the past, this was never an issue. But I pulled power again for giggles and was able to reproduce the issue. A little bit of code in /etc/init.d/mysqld to check, on start, if mysql.sock exists, and then delete it, should be all that is needed to fix this issue.

The init script makes this file to “claim” the service. Lock files are very common. On shutdown these files are cleaned up. On a hard shutdown there is no mechanism to do the clean up work because that is not suppose to happen. When you have a panic, lockup, power loss or have to flip the switch you should expect some orphan locks and possible corruption depending on the task that was happening during the event.

This has been covered at length. Its a change in the MySQL RPMs from oracle and RedHat. WE do not want to get into patching MySQL direct and having to fork it so we already include a fix that has a fixer run before MySQL starts on boot.

http://wiki.freepbx.org/display/FD/5.211.65+Release+Notes

Update 5.211.65-7 added this. Sounds like you are not keeping your system upgraded.

Thanks for the quick and detailed response. I did check the bug tracker for mysql related issues but didn’t look back to January. This server was physically replaced and fully updated on 2-26-14. I looked back at the disk I used-5.211.65-6. Literally days before the fix was added to the update scripts and days before scheduled maintenance. Unlucky timing on my part.