Slow FreePBX GUI and DTMF reponses

1

After updating some modules recently (I can’t pinpoint exactly which ones because I did a few at a time) my FreePBX web GUI and asterisk system itself are slow. The GUI problem is self explanatory, but the phone system is slow to respond to call recording requests and remote extension pickups at a client level. Stopping and starting services also appears to happen much slower than it did before. At first I though it was an iSymphony/Java issue but now I’m finding that MySQL may be the culprit. I’m seeing a lot of processes in htop that look like this:

/usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --log-error=/var/log/mysqld.log --pid-file=/var/run/mysqld/m

and error log entries like this:

"Can’t connect to local MySQL server through socket… "

Any ideas?

2

I assume you tried a reboot?

3

Definitely have rebooted multiple times. I have gone down multiple paths of troubleshooting, including updating SysAdmin, reinstalling node.JS, disabling unused modules, repairing all MySQL databases, and applying new updates that have come out since. There appears to be high CPU activity on this 8 core server

4

We do all of our development on “generally unimpressive” virtual machines with terrible specs to make sure we work everywhere. I have FreePBX running on an Intel Edison. So a more powerful server should be super happy. Can you pastebin the output of top

5

Yes, exactly - it’s a very powerful machine with a lot of RAM. Here is the top output:

http://pastebin.com/gY5i9Z0R

6

Are you using iSymphony?

7

Yes, however I’ve tried stopping the service and the same slowness persists. The only thing I haven’t tried with iSymphony is stopping the service and then rebooting. iSymphony doesn’t believe it’s their issue (of course :wink:)s

8

I just saw the process using CPU is Java. FreePBX doesn’t use Java but ISymphony does. stopping the service then killing java would be a worth while test.

9

I’ve tried looking up how to kill java after stopping iSymphony, but I can’t find a command that kills Java. What command would I use?

10

pkill java should work.

11

I just tried this (pkill java and stopped iSymphony v3 server) and it’s still running slow. Any thoughts about the MySQL stuff?

12

The noticeable process is

35 root 20 0 0 0 0 R 51.6 0.0 220:55.27 events/0

you have a kernel thread stuck, I can only suggest

strace -cp 35

that might or might not track it down

1 Like
13

ptrace and strace don’t seem to be available commands for some reason:

[root@voice ~]# strace -cp 35
-bash: strace: command not found

14

That’s because it is not installed, use yum or apt-get to install it if it is in your repositories.

15

[root@voice ~]# strace -cp 35
attach: ptrace(PTRACE_ATTACH, …): Operation not permitted
[root@voice ~]# strace -p 35
attach: ptrace(PTRACE_ATTACH, …): Operation not permitted
[root@voice ~]# sudo strace -cp 35
attach: ptrace(PTRACE_ATTACH, …): Operation not permitted

This is the output after installing strace and trying to trace PID 35

16

sorry, I don’t have that problem.

17

researched my recent error and found that it’s a security feature:

"This is not a bug, it is a design issue. It is also possible to prevent your program being traced by using Linux specific prctl call and clear PR_SET_DUMPABLE flag, which OpenSSH agent in your system already does.

But, not many programmers aware of this behaviour. So, when looking at the Linux distributions perspective, it is good to activate additional protection mechanism on ptrace system call to make more secure system without need to change of any software code.

Newer Linux kernels has the Yama security module. When the Yama is built into the kernel, its behaviour can be changed through /proc/sys/kernel/yama/ptrace_scope file.

If you’re trying to use strace to attach another process which is also belongs to you but you’re not a parent of it and the value in ptrace_scope file greater than 0, it is normal to get “Operation not permitted” error.

Some recent Linux distributions sets ptrace_scope value to 1 in default. You can still use strace in this scenario with starting the programs with strace like below:

$ strace ./my_prog
but you can not attach a process which is already started and has different parent of your newly executed strace.

18

I wasn’t able to determine the root cause of events/0 hanging up, but I did research wanpipe issues on PRI cards and performance loss. I believe i resolved the performance issue by adding the following parameters to /etc/grub.conf: “apic=on” & “acpi=off” and installing/starting the IRQ Balance addon. The IRQ Balance addon appears to have increased buffered and cached disk reads on /dev/sda by 3-4 times. Thanks guys,

19

Well, in response to my own post I’m going to recommend that you NOT add “apic=on” and "acpi=off to your grub config file. The first reboot after adding this to the configuration resulted in kernel not loading. I’ve since removed it.