MariaDB hourly high IO usage

freepbx
configuration
Tags: #<Tag:0x00007f7028be6488> #<Tag:0x00007f7028be62f8>

(Omarchand) #1

Hi to all,

We’ve deployed about 20 FreePBX instances into Digital Ocean droplets. Each of them are using the same FreePBX ISO disk for initial installation : SNG7-PBX-64bit-1805 (FreePBX 14.0.13.26 , Current Asterisk Version: 13.32.0)

Server is using 2vCPU and 4GIGs of Ram.

Now, a single instance had a strange behaviour. At every hours, the IO usage is increasing up for one or two minutes, than returns to its normal “almost idle” state. It is the only instance experiencing that trouble.

All this started few days back, used to had no issue with that server.
Call volume is really low, in fact, almost no calls are being processed as our customer is still not ready to migrate its SIP phone lines to that server.

Using iotop command, I was able to find out that mariadb logs is the issue here:

09:00:55 18419 be/4 mysql 0.00 K/s 0.00 K/s 0.00 % 99.99 % mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/m
ysql/plugin --log-error=/var/log/mariadb/mariadb.log --pid-file=/var/run/mariadb/mariadb.pid --socket=/var/lib/mysql/mysql.sock
09:00:55 1597 be/4 mysql 0.00 K/s 7.89 K/s 0.00 % 0.00 % mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/m
ysql/plugin --log-error=/var/log/mariadb/mariadb.log --pid-file=/var/run/mariadb/mariadb.pid --socket=/var/lib/mysql/mysql.sock
09:00:56 1605 be/4 root 0.00 K/s 0.00 K/s 0.00 % 0.00 % rsyslogd -n [in:imjournal]
09:00:56 720 be/3 root 0.00 K/s 3.94 K/s 0.00 % 0.00 % auditd
09:00:56 13222 be/4 asterisk 0.00 K/s 3.94 K/s 0.00 % 0.00 % asterisk -f -U asterisk -G asterisk -vvvg -c
09:00:57 721 be/3 root 0.00 K/s 0.00 K/s 0.00 % 99.99 % auditd
09:00:57 1605 be/4 root 0.00 K/s 0.00 K/s 0.00 % 0.00 % rsyslogd -n [in:imjournal]
09:00:58 1597 be/4 mysql 0.00 K/s 142.02 K/s 0.00 % 99.99 % mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/m
ysql/plugin --log-error=/var/log/mariadb/mariadb.log --pid-file=/var/run/mariadb/mariadb.pid --socket=/var/lib/mysql/mysql.sock

As all our deployments are using same CPU size, same amount of RAM , and so on, did I missed something in the Advance Settings of FreePBX that might start up a debug log of some sort ?

I alredy went into all the advance settings and disable all FreePBX debug loggings and try all the usual reboots , module upgrades, yum upgrade, no luck.

Any advise what can be my next steps ?

Thanks.