Mysql/webadmin/asterisk down after ./upgrade-3.211.63-to-4.211.64-1.sh

Distro Discussion & Help
1

Recently took over as sysadmin for a small company. Customer has a freepbx server that was configured/installed by a former employee. Prior to my involvement today it had not been updated since install (roughly two years).

Followed the upgrade procedures all the way up to the script listed in title ./upgrade-3.211.63-to-4.211.64-1.sh and when it completed it forced a reboot. Up until this point everything worked and test calls were able to reach the phones beyond the freepbx. Following reboot, webadmin does not load and if I ssh to the server I am unable to manually start mysqld.

Error when visiting webadmin page

FATAL ERROR
DB Error: connect failed
Trace Back/var/www/html/admin/libraries/db_connect.php:71 die_freepbx()

0: DB Error: connect failed

/var/www/html/admin/bootstrap.php:85 require_once()

/etc/freepbx.conf:9 require_once()

/var/www/html/admin/config.php:99 include_once()

Attempt to manually start mysqld results in:
[root@pbx ~]# service mysqld start
Timeout error occurred trying to start MySQL Daemon.
Starting MySQL: [FAILED]

Relevant output from mysqld.log
141111 14:45:35 mysqld started
141111 14:45:36 InnoDB: Started; log sequence number 0 43655
141111 14:45:36 [Note] /usr/libexec/mysqld: ready for connections.
Version: ‘5.0.95’ socket: ‘/var/lib/mysql/mysql.sock’ port: 3306 Source distribution
141111 15:48:55 [Note] /usr/libexec/mysqld: Normal shutdown

141111 15:48:57 InnoDB: Starting shutdown…
141111 15:48:58 InnoDB: Shutdown completed; log sequence number 0 43655
141111 15:48:58 [Note] /usr/libexec/mysqld: Shutdown complete

141111 15:48:58 mysqld ended

141111 15:50:30 mysqld started
141111 15:50:30 [ERROR] /usr/libexec/mysqld: unknown variable 'general_log=1’
141111 15:50:30 mysqld ended

Help?

2

Relevant upgrade log (tried running it again when server did not behave as expected which was probably unwise)

Tue Nov 11 15:47:11 CST 2014 Check to make sure this is a FreePBX Distro system before executing
Tue Nov 11 15:47:11 CST 2014 This appears to be a FreePBX Distro system as it has a Distro Version of 3.211.63-10
Tue Nov 11 15:47:11 CST 2014 Make sure this is version 3.211.63-7 or greater before executing
Tue Nov 11 15:47:11 CST 2014 This appears to be FreePBX Distro version 3.211.63.10 so we can now upgrade you to the 4.211.64 track as version 4.211.64-1
VARIABLES SET FOR UPGRADE
asterisk=Asterisk 1.8.17.0 built by root @ jenkins4.schmoozecom.net on a x86_64 running Linux on 2012-10-16 21:31:18 UTC
kernel=Linux pbx 2.6.18-308.4.1.el5 #1 SMP Tue Apr 17 17:08:00 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux
version=3.211.63.10
brand=FreePBXDistro
host=pbx
upgradeversion=4.211.64-1
virtual=0

Tue Nov 11 15:47:11 CST 2014 Your FreePBX Distro System is being upgraded to 4.211.64-1. Please standby…

Tue Nov 11 15:47:11 CST 2014 STAGE 1 STARTING - GUI Modules
Tue Nov 11 15:47:11 CST 2014 Upgrade All FreePBX GUI Modules
Tue Nov 11 15:47:21 CST 2014 STAGE 1 COMPLETED - GUI Modules - Moving to Stage 2

Tue Nov 11 15:47:21 CST 2014 STAGE 2 STARTING - RPM’s
Tue Nov 11 15:47:21 CST 2014 Replace repos with only FreePBX Distro since some people have added other repos which can break updates
Tue Nov 11 15:47:21 CST 2014 Cleanup old Kernels to make space in boot
Tue Nov 11 15:47:38 CST 2014 Updating all remaining RPMS now to Centos 6.4
Tue Nov 11 15:47:46 CST 2014 STAGE 2 COMPLETED - Misc Items - Moving to Stage 4

Tue Nov 11 15:47:46 CST 2014 STAGE 3 STARTING - Misc Items
Tue Nov 11 15:47:46 CST 2014 STAGE 3 COMPLETED - Misc Items - Moving to Stage 4

Tue Nov 11 15:47:46 CST 2014 STAGE 4 STARTING - Clean Up
Tue Nov 11 15:47:46 CST 2014 updatedb for locate command
Tue Nov 11 15:47:46 CST 2014 Restart incron to be safe
Tue Nov 11 15:47:46 CST 2014 STAGE 4 COMPLETED - Clean Up - Moving to Stage 5

Tue Nov 11 15:47:47 CST 2014 STAGE 5 STARTING - Final Verifications
Tue Nov 11 15:47:47 CST 2014 STAGE 5 COMPLETED - Final Verifications - Moving to Stage 6

Tue Nov 11 15:47:47 CST 2014 UPGRADE 100% COMPLETED
Tue Nov 11 15:47:47 CST 2014 REBOOT YOUR BOX NOW FOR NEW KERNEL, DAHDI AND ASTERISK
Tue Nov 11 15:55:19 CST 2014 Check to make sure this is a FreePBX Distro system before executing
Tue Nov 11 15:55:20 CST 2014 This appears to be a FreePBX Distro system as it has a Distro Version of 4.211.64-1
Tue Nov 11 15:55:20 CST 2014 Make sure this is version 3.211.63-7 or greater before executing
Tue Nov 11 15:55:20 CST 2014 This appears to be FreePBX Distro version 4.211.64.1 so we can now upgrade you to the 4.211.64 track as version 4.211.64-1
VARIABLES SET FOR UPGRADE
asterisk=Unable to connect to remote asterisk (does /var/run/asterisk/asterisk.ctl exist?)
kernel=Linux pbx 2.6.18-308.4.1.el5 #1 SMP Tue Apr 17 17:08:00 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux
version=4.211.64.1
brand=FreePBXDistro
host=pbx
upgradeversion=4.211.64-1
virtual=0

Tue Nov 11 15:55:20 CST 2014 Your FreePBX Distro System is being upgraded to 4.211.64-1. Please standby…

Tue Nov 11 15:55:20 CST 2014 STAGE 1 STARTING - GUI Modules
Tue Nov 11 15:55:20 CST 2014 Upgrade All FreePBX GUI Modules
Tue Nov 11 15:55:21 CST 2014 STAGE 1 COMPLETED - GUI Modules - Moving to Stage 2

Tue Nov 11 15:55:21 CST 2014 STAGE 2 STARTING - RPM’s
Tue Nov 11 15:55:21 CST 2014 Replace repos with only FreePBX Distro since some people have added other repos which can break updates
Tue Nov 11 15:55:21 CST 2014 Cleanup old Kernels to make space in boot
Tue Nov 11 15:55:38 CST 2014 Updating all remaining RPMS now to Centos 6.4
Tue Nov 11 15:55:46 CST 2014 STAGE 2 COMPLETED - Misc Items - Moving to Stage 4

Tue Nov 11 15:55:46 CST 2014 STAGE 3 STARTING - Misc Items
Tue Nov 11 15:55:46 CST 2014 STAGE 3 COMPLETED - Misc Items - Moving to Stage 4

Tue Nov 11 15:55:46 CST 2014 STAGE 4 STARTING - Clean Up
Tue Nov 11 15:55:46 CST 2014 updatedb for locate command
Tue Nov 11 15:55:58 CST 2014 Restart incron to be safe
Tue Nov 11 15:55:58 CST 2014 STAGE 4 COMPLETED - Clean Up - Moving to Stage 5

Tue Nov 11 15:55:58 CST 2014 STAGE 5 STARTING - Final Verifications
Tue Nov 11 15:55:58 CST 2014 STAGE 5 COMPLETED - Final Verifications - Moving to Stage 6

Tue Nov 11 15:55:58 CST 2014 UPGRADE 100% COMPLETED
Tue Nov 11 15:55:58 CST 2014 REBOOT YOUR BOX NOW FOR NEW KERNEL, DAHDI AND ASTERISK

3

Possibly also a bad idea, I tried to run the next upgrade script hoping it might repair situation

Tue Nov 11 16:09:19 CST 2014 This appears to be a FreePBX Distro system as it has a FreePBX Distro Version of 4.211.64-1
VARIABLES SET FOR UPGRADE
asterisk=Unable to connect to remote asterisk (does /var/run/asterisk/asterisk.ctl exist?)
kernel=Linux pbx 2.6.18-308.4.1.el5 #1 SMP Tue Apr 17 17:08:00 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux
version=4.211.64-1
host=pbx
upgradeversion=4.211.64-2
virtual=0

Tue Nov 11 16:09:19 CST 2014 Your FreePBX Distro System is being upgraded to 4.211.64-2. Please standby…

Tue Nov 11 16:09:19 CST 2014 STAGE 1 STARTING - GUI Modules
Tue Nov 11 16:09:20 CST 2014 Upgrade All FreePBX GUI Modules
Tue Nov 11 16:09:20 CST 2014 STAGE 1 COMPLETED - GUI Modules - Moving to Stage 2

Tue Nov 11 16:09:20 CST 2014 STAGE 2 STARTING - RPM’s
Tue Nov 11 16:09:20 CST 2014 Replace repos with only FreePBX Distro since some people have added other repos which can break updates
Tue Nov 11 16:09:21 CST 2014 Install asterisk-version-switch RPM
Tue Nov 11 16:09:39 CST 2014 Install distro-motd RPM
Tue Nov 11 16:09:47 CST 2014 Install a few Misc Packages that use to be included in Centos 5 but are not in 6
Tue Nov 11 16:09:55 CST 2014 Updating all remaining RPMS now to Centos 6.2
Tue Nov 11 16:10:03 CST 2014 STAGE 2 COMPLETED - Misc Items - Moving to Stage 4

Tue Nov 11 16:10:03 CST 2014 STAGE 3 STARTING - Misc Items
Tue Nov 11 16:10:05 CST 2014 STAGE 3 COMPLETED - Misc Items - Moving to Stage 4

Tue Nov 11 16:10:05 CST 2014 STAGE 4 STARTING - Clean Up
Tue Nov 11 16:10:05 CST 2014 updatedb for locate command
Tue Nov 11 16:10:11 CST 2014 Restart incron to be safe
Tue Nov 11 16:10:11 CST 2014 STAGE 4 COMPLETED - Clean Up - Moving to Stage 5

Tue Nov 11 16:10:11 CST 2014 STAGE 5 STARTING - Final Verifications
Tue Nov 11 16:10:11 CST 2014 STAGE 5 COMPLETED - Final Verifications - Moving to Stage 6

Tue Nov 11 16:10:11 CST 2014 UPGRADE 100% COMPLETED

4

I would look at upgradeing your mysql server to a more recent version, as an interim you could comment out in

/etc/mysql/my.cnf

something like:-

#general_log_file        = /var/log/mysql/mysql.log
#general_log             = 1

logs are useful but only if you know how to use them :slight_smile:

5

That has asterisk and my webadmin interface reachable again and calls are completing! Now, is it safe to continue following the upgrade chain or should I upgrade mysql first?

6

Well, I don’t use those scripts so you would have to wait for an authoritative answer, I would comment though that the “general log file” can get huge especially if the mysql logrotate script does not cover your log files location, so generally you should not be logging “generally” unless you have a reason to, it is a waste of disk space and resources.