Unexpected Power Outage - Can't Access Web GUI or CLI (mysql corruption?)

General Help
1

FreePBX Distro
FreePBX: 14.0.11
Asterisk: 13.22.0
PBX Firmware: 12.7.6-1904-1.sng7
PBX Service Pack: 1.0.0.0

We had an unexpected loss of power on this server, and since then I get the following message when trying to access the web GUI:

Exception thrown with message "SQLSTATE[HY000] [2002] No such file or directory::SQLSTATE[HY000] [2002] No such file or directory"

Stacktrace:
#7 Exception in /var/www/html/admin/libraries/utility.functions.php:204
#6 die_freepbx in /var/www/html/admin/libraries/BMO/Database.class.php:142
#5 PDOException in /var/www/html/admin/libraries/BMO/Database.class.php:137
#4 PDO:__construct in /var/www/html/admin/libraries/BMO/Database.class.php:137
#3 FreePBX\Database:__construct in /var/www/html/admin/libraries/BMO/FreePBX.class.php:71
#2 FreePBX:__construct in /var/www/html/admin/bootstrap.php:153
#1 require_once in /etc/freepbx.conf:9
#0 include_once in /var/www/html/admin/config.php:100

I see this when logging in via SSH:

Last login: Fri May 17 12:13:41 2019 from 10.0.1.154
Exception: SQLSTATE[HY000] [2002] No such file or directory::SQLSTATE[HY000] [2002] No such file or directory in file /var/www/html/admin/libraries/utility.functions.php on line 204
Stack trace:
  1. Exception->() /var/www/html/admin/libraries/utility.functions.php:204
  2. die_freepbx() /var/www/html/admin/libraries/BMO/Database.class.php:142
  3. PDOException->() /var/www/html/admin/libraries/BMO/Database.class.php:137
  4. PDO->__construct() /var/www/html/admin/libraries/BMO/Database.class.php:137
  5. FreePBX\Database->__construct() /var/www/html/admin/libraries/BMO/FreePBX.class.php:71
  6. FreePBX->__construct() /var/www/html/admin/bootstrap.php:153
  7. require_once() /etc/freepbx.conf:9
  8. include_once() /var/lib/asterisk/bin/fwconsole:12
Exception: SQLSTATE[HY000] [2002] No such file or directory::SQLSTATE[HY000] [2002] No such file or directory in file /var/www/html/admin/libraries/utility.functions.php on line 204
Stack trace:
  1. Exception->() /var/www/html/admin/libraries/utility.functions.php:204
  2. die_freepbx() /var/www/html/admin/libraries/BMO/Database.class.php:142
  3. PDOException->() /var/www/html/admin/libraries/BMO/Database.class.php:137
  4. PDO->__construct() /var/www/html/admin/libraries/BMO/Database.class.php:137
  5. FreePBX\Database->__construct() /var/www/html/admin/libraries/BMO/FreePBX.class.php:71
  6. FreePBX->__construct() /var/www/html/admin/bootstrap.php:153
  7. require_once() /etc/freepbx.conf:9
  8. include_once() /var/lib/asterisk/bin/fwconsole:12
+------------------------------------------------------------+
| Your system is currently up to date!                       |
| No PBX Module updates information available                |
+------------------------------------------------------------+

I’m not a Linux expert, but from what I can tell, mysql (or mariadb?) isn’t running.

I tried:

mysqlcheck --repair --all-databases

and I got this:

[root@mobpbx2 ~]# mysqlcheck --repair --all-databases
mysqlcheck: Got error: 2002: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2) when trying to connect

I came across this FreePBX Community post: Mysqld service will not start after a improper shut down

That led me to remove /var/lib/mysql/mysql.sock then I ran service mariadb start, and mysql is running again.

Then when I ran mysqlcheck --repair --all-databases, it looks like all of the asterisk.* & asteriskcdrdb.* tables say “doesn’t exist” and “operation failed” next to them. The mysql.* tables all show “OK”.

Since the asterisk & asteriskcdrdb databases aren’t working, I get this when I login via CLI:

Last login: Mon May 20 07:59:01 2019 from 10.0.1.154
Exception: fatal error reading freepbx_settings:: in file /var/www/html/admin/libraries/utility.functions.php on line 207
Stack trace:
  1. Exception->() /var/www/html/admin/libraries/utility.functions.php:207
  2. die_freepbx() /var/www/html/admin/libraries/BMO/Freepbx_conf.class.php:222
  3. FreePBX\Freepbx_conf->__construct() /var/www/html/admin/libraries/BMO/Self_Helper.class.php:124
  4. FreePBX\Self_Helper->autoLoad() /var/www/html/admin/libraries/BMO/Self_Helper.class.php:53
  5. FreePBX\Self_Helper->__call() /var/www/html/admin/bootstrap.php:173
  6. FreePBX->Freepbx_conf() /var/www/html/admin/bootstrap.php:173
  7. require_once() /etc/freepbx.conf:9
  8. include_once() /var/lib/asterisk/bin/fwconsole:12
Exception: fatal error reading freepbx_settings:: in file /var/www/html/admin/libraries/utility.functions.php on line 207
Stack trace:
  1. Exception->() /var/www/html/admin/libraries/utility.functions.php:207
  2. die_freepbx() /var/www/html/admin/libraries/BMO/Freepbx_conf.class.php:222
  3. FreePBX\Freepbx_conf->__construct() /var/www/html/admin/libraries/BMO/Self_Helper.class.php:124
  4. FreePBX\Self_Helper->autoLoad() /var/www/html/admin/libraries/BMO/Self_Helper.class.php:53
  5. FreePBX\Self_Helper->__call() /var/www/html/admin/bootstrap.php:173
  6. FreePBX->Freepbx_conf() /var/www/html/admin/bootstrap.php:173
  7. require_once() /etc/freepbx.conf:9
  8. include_once() /var/lib/asterisk/bin/fwconsole:12
+------------------------------------------------------------+
| Your system is currently up to date!                       |
| No PBX Module updates information available                |
+------------------------------------------------------------+

This when I try to access the web GUI:

Exception thrown with message "fatal error reading freepbx_settings::"

Stacktrace:
#7 Exception in /var/www/html/admin/libraries/utility.functions.php:207
#6 die_freepbx in /var/www/html/admin/libraries/BMO/Freepbx_conf.class.php:222
#5 FreePBX\Freepbx_conf:__construct in /var/www/html/admin/libraries/BMO/Self_Helper.class.php:124
#4 FreePBX\Self_Helper:autoLoad in /var/www/html/admin/libraries/BMO/Self_Helper.class.php:53
#3 FreePBX\Self_Helper:__call in /var/www/html/admin/bootstrap.php:173
#2 FreePBX:Freepbx_conf in /var/www/html/admin/bootstrap.php:173
#1 require_once in /etc/freepbx.conf:9
#0 include_once in /var/www/html/admin/config.php:100

I have some backups, but this is the first time I’ve ever had to do anything with them. We’re supposed to have a full local back as well as a warm spare backup --> the primary & warm spare servers are both having the same issues.

In my backups I have mysql-2.sql & mysql-3.sql files, and I can see that these are the asterisk and asteriskcdrdb databases, respectively. I don’t have any of the other files from /var/lib/mysql though. On the server, I see these in /var/lib/mysql:

aria_log.00000001
aria_log_control
ib_logfile0
ib_logfile1
ibdata1
mysql.sock

I read in a forum somewhere that the ib* files are important for restoring. Again, I’m pretty far outside of my expertise here, so any direction would be appreciated. I don’t expect direct assistance with mysql, but if I’m missing something simple, I thought you all might see it.

Also, in spite of all of this, calls are still processing normally. I just can’t do anything in the interface.

Thanks!

2

I would stop your SQL server, restore /var/lib/mysql directory from a known good backup and hope all is good.

1 Like
3

Does FreePBX backup this directory as part of any of its backups? I had a full backup configured (and the warm spare setup). If FreePBX backed it up, where would I look for those locally stored backups?

4

No , generally the files are not actually backed up. But the tarball should include a mysqldump of both asterisk and asteriskcdrdb. The asterisk one should be restorable

1 Like
5

Thanks! That looks like it has worked.

I guess as long as we do a full nightly backup, we should always be restorable within the last 24 hours of a failure, right? We don’t have system changes that often, so I guess that may be good enough.

Any thoughts on a better backup strategy I could look into?

6

Personally I would feel very safe with a full clone of the system , I personally would use Mondo as it can be run in real-time without disruption.but in reality all my machines are now cloud based, so a dollar a month , no sweat.

1 Like
7

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.