Distro Diag

[Russix]

In the last 30 days, I’ve twice experienced an unresponsive GUI and can’t log in via SSH until restarting the server. I’m using PBX Version: 15.0.16.76 PBX Distro: 12.7.8-2008-1.sng7 Asterisk Version: 16.13.0. Any suggestions on how to troubleshoot? Thank you.

[Stewart1 (Stewart)]

When this happens, does the system still process calls?

If so, it’s probably just a firewall issue, e.g. your IP got inadvertently banned. Log in from the console and look at iptables, etc.

If the system is frozen, likely a hardware issue, look for messages on the console or in dmesg.

[cynjut (Dave Burgess)]

I had a system do this and it was a bad spot in RAM. Doing a hardware diagnostic might be a good idea as well.

[Russix]

So it did it again, here’s what the console read:

Broadcast message from [email protected] (Thu Nov 5 14:13:40 2020):

Firewall was unable to connect to MySQL after 30 seconds.
Check Database!

Broadcast message from [email protected] (Thu Nov 5 14:14:14 2020):

Firewall was unable to connect to MySQL after 30 seconds.
Check Database!
packet_write_wait: Connection to XXX.XXX.XXX.XXX port 22: Broken pipe

[cynjut (Dave Burgess)]

I’d start with /var/log/messages and see if the hard drive is starting to through correctable errors. If so, back up the drive and replace it.

After that, I’d log into the console (or SSH) as ‘root’ and run the following to repair all of your databases:

GUI throws error when enabling firewall

Once you get MySQL running again, you need to make sure you repair all of the databases. Repair from the console as root with mysqlcheck --repair --all-databases IIRC, there’s also an ‘fwconsole repairall’ command, but I might have dreamed that.

[Russix]

I’m getting a ton of these messages:

[FAILED] Failed to start LSB: Bring up/down networking.

[cynjut (Dave Burgess)]

https://www.cyberithub.com/failed-to-start-lsb-bring-up-down-networking/ would be a good place to start.

[PitzKey (Itzik)]

I’ve seen that happening with machines that have low resources. If you are running the distro with ALLLLLLL modules, then you’ll need 2GB of RAM, if you remove UCP and other unnecessary modules you should be able to run it just fine with 1GB.

[Russix]

So I had a memory leak that would bring the system to a dead crawl after a week. I fixed it by upgrading MariaDB to version 10.4. Upgrade commands follow(its not a script):

#Upgrade FREEPBX DISTRO 7 to MariaDB 10.4
#stop mariadb
systemctl stop mariadb
#comment line “sangoma-pbx” in sangoma-pbx.conf
vi /etc/yum/protected.d/sangoma-pbx.conf
#find the old packages
rpm -qa | grep “mariadb”
#remove all packages listed in previous command(yours may differ)
rpm -e --nodeps “mariadb-libs-5.5.65-1.el7.x86_64”
rpm -e --nodeps “mariadb-5.5.65-1.el7.x86_64”
rpm -e --nodeps “mariadb-server-5.5.65-1.el7.x86_64”
#check packages removed
rpm -qa | grep “mariadb”
#configure repo, copy and paste next line to EOF
cat <<EOF | sudo tee /etc/yum.repos.d/MariaDB.repo
[mariadb]
name = MariaDB
baseurl = http://yum.mariadb.org/10.4/centos7-amd64
gpgkey=https://yum.mariadb.org/RPM-GPG-KEY-MariaDB
gpgcheck=1
EOF
sudo yum install mariadb mariadb-server
sudo systemctl start mariadb
sudo systemctl enable mariadb
mysql_upgrade
#uncomment line “sangoma-pbx” in sangoma-pbx.conf
vi /etc/yum/protected.d/sangoma-pbx.conf

[lgaetz (Lorne Gaetz)]

I don’t know of any other reports of maria issues on the Distro. As a general rule, upgrading packages from outside the Distro repo framework is not supported, so I caution anyone from following the method above. @Russix please provide an update in a few months time.