I’ve just installed a fresh copy of Freepbx (FreePBX-4.211.64-7-x86_64-Full-1379605000.iso) and without making any configuration changes (i just log into the web interface) the ‘App Memory’ is growing out of control. It starts off at 13% and after a few hours its at 80%+. I’m a linux noob so I’m not sure where to start.
Thanks
Damien
What “htop” (from Linux CLI) is telling?
Did you tried to see what happens (watching htop running) to memory allocation while, from another shell, you performs an “amportal restart” (or, more selectively, an “service httpd restart”)?
I really have nfi what I’m looking for but here’s a before/after restarting httpd.
CPU[|| 2.0%] Tasks: 55, 153 thr; 1 running
Mem[|||||||||||||||||||1665/1877MB] Load average: 0.00 0.00 0.00
Swp[| 15/4031MB] Uptime: 09:04:24
PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command
13299 root 20 0 110M 1788 1260 R 2.0 0.1 0:14.74 htop
1768 asterisk 20 0 1324M 29712 9192 S 0.0 1.5 2:04.51 /usr/sbin/asteris
1801 asterisk 20 0 1324M 29712 9192 S 0.0 1.5 0:50.07 /usr/sbin/asteris
1651 asterisk 20 0 1287M 57592 4644 S 0.0 3.0 0:30.66 /usr/java/latest/
1785 asterisk 20 0 1324M 29712 9192 S 0.0 1.5 1:09.79 /usr/sbin/asteris
1626 asterisk 20 0 1287M 57592 4644 S 0.0 3.0 0:43.35 /usr/java/latest/
1643 asterisk 20 0 1287M 57592 4644 S 0.0 3.0 0:01.55 /usr/java/latest/
1388 mysql 20 0 625M 19888 3644 S 0.0 1.0 0:15.29 /usr/libexec/mysq
1401 mysql 20 0 625M 19888 3644 S 0.0 1.0 0:09.67 /usr/libexec/mysq
1533 root 20 0 798M 12608 2256 S 0.0 0.7 0:19.50 /usr/bin/python /
1546 root 20 0 798M 12608 2256 S 0.0 0.7 0:01.05 /usr/bin/python /
1547 root 20 0 798M 12608 2256 S 0.0 0.7 0:02.81 /usr/bin/python /
1673 asterisk 20 0 1287M 57592 4644 S 0.0 3.0 0:04.71 /usr/java/latest/
13264 root 20 0 97924 3812 2888 S 0.0 0.2 0:00.18 sshd: root@pts/0
1536 root 20 0 798M 12608 2256 S 0.0 0.7 0:02.55 /usr/bin/python /
1402 mysql 20 0 625M 19888 3644 S 0.0 1.0 0:03.69 /usr/libexec/mysq
1858 nagios 20 0 38900 1300 700 S 0.0 0.1 0:05.48 /usr/local/nagios
CPU[|| 2.0%] Tasks: 53, 153 thr; 1 running
Mem[|||||||||||||||||||1422/1877MB] Load average: 0.83 0.22 0.07
Swp[| 12/4031MB] Uptime: 09:05:02
PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command
13299 root 20 0 110M 1792 1260 R 1.3 0.1 0:15.55 htop
1768 asterisk 20 0 1324M 29712 9192 S 0.7 1.5 2:04.84 /usr/sbin/asteris
1626 asterisk 20 0 1287M 57592 4644 S 0.7 3.0 0:43.41 /usr/java/latest/
1787 asterisk 20 0 1324M 29712 9192 S 0.7 1.5 0:00.91 /usr/sbin/asteris
1785 asterisk 20 0 1324M 29712 9192 S 0.0 1.5 1:09.96 /usr/sbin/asteris
1801 asterisk 20 0 1324M 29712 9192 S 0.0 1.5 0:50.21 /usr/sbin/asteris
13366 root 20 0 280M 11296 5268 S 0.0 0.6 0:00.72 /usr/sbin/httpd
13369 asterisk 20 0 280M 6696 640 S 0.0 0.3 0:00.02 /usr/sbin/httpd
1533 root 20 0 798M 12608 2256 S 0.0 0.7 0:19.55 /usr/bin/python /
1544 root 20 0 798M 12608 2256 S 0.0 0.7 0:01.07 /usr/bin/python /
13370 asterisk 20 0 280M 6696 640 S 0.0 0.3 0:00.01 /usr/sbin/httpd
1651 asterisk 20 0 1287M 57592 4644 S 0.0 3.0 0:30.69 /usr/java/latest/
1535 root 20 0 9228 1256 1024 S 0.0 0.1 0:00.28 /usr/libexec/gam_
1401 mysql 20 0 625M 19888 3644 S 0.0 1.0 0:09.67 /usr/libexec/mysq
1388 mysql 20 0 625M 19888 3644 S 0.0 1.0 0:15.30 /usr/libexec/mysq
1673 asterisk 20 0 1287M 57592 4644 S 0.0 3.0 0:04.73 /usr/java/latest/
1541 root 20 0 798M 12608 2256 S 0.0 0.7 0:01.21 /usr/bin/python /
F1Help F2
I had a better look today, sorted the items in htop by memory usage…
login as: root
[email protected]’s password:
Access denied
[email protected]’s password:
Last login: Fri Oct 18 09:37:31 2013 from 172.22.3.20
| | __ ___ | _ | __ ) / /
| | | '/ _ / _ \ |) | _ \ /
| || | | __/ __/ __/| |) /
|| || _|_|| |__//_\
Interface eth0 IP: 172.22.3.81
Please note most tasks should be handled through the FreePBX UI.
You can access the FreePBX GUI by typing one of the above IP’s in to your web br owser.
For support please visit http://www.freepbx.org/support-and-professional-service s
[root@localhost ~]# reboot
[root@localhost ~]#
Broadcast message from [email protected]
(/dev/pts/0) at 13:40 …
The system is going down for reboot NOW!
login as: root
[email protected]’s password:
Last login: Fri Oct 18 13:40:44 2013 from 172.22.3.20
| | __ ___ | _ | __ ) / /
| | | '/ _ / _ \ |) | _ \ /
| || | | __/ __/ __/| |) /
|| || _|_|| |__//_\
Interface eth0 IP: 172.22.3.81
Please note most tasks should be handled through the FreePBX UI.
You can access the FreePBX GUI by typing one of the above IP’s in to your web br owser.
For support please visit http://www.freepbx.org/support-and-professional-service s
[root@localhost ~]# htop
CPU[|| 2.0%] Tasks: 47, 62 thr; 1 running
Mem[|||||||||||||| 254/1877MB] Load average: 0.68 0.44 0.18
Swp[ 0/4031MB] Uptime: 00:01:58
PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command
1724 asterisk 20 0 1217M 33104 12804 S 0.7 1.7 0:00.22 /usr/sbin/asterisk -f -U asterisk -G asterisk -vvvg -c
1709 asterisk 20 0 1217M 33104 12804 S 0.7 1.7 0:01.70 /usr/sbin/asterisk -f -U asterisk -G asterisk -vvvg -c
1710 asterisk 20 0 1217M 33104 12804 S 0.0 1.7 0:00.00 /usr/sbin/asterisk -f -U asterisk -G asterisk -vvvg -c
1711 asterisk 20 0 1217M 33104 12804 S 0.0 1.7 0:00.00 /usr/sbin/asterisk -f -U asterisk -G asterisk -vvvg -c
1712 asterisk 20 0 1217M 33104 12804 S 0.0 1.7 0:00.06 /usr/sbin/asterisk -f -U asterisk -G asterisk -vvvg -c
1713 asterisk 20 0 1217M 33104 12804 S 0.0 1.7 0:00.00 /usr/sbin/asterisk -f -U asterisk -G asterisk -vvvg -c
1714 asterisk 20 0 1217M 33104 12804 S 0.0 1.7 0:00.00 /usr/sbin/asterisk -f -U asterisk -G asterisk -vvvg -c
1715 asterisk 20 0 1217M 33104 12804 S 0.0 1.7 0:00.00 /usr/sbin/asterisk -f -U asterisk -G asterisk -vvvg -c
1716 asterisk 20 0 1217M 33104 12804 S 0.0 1.7 0:00.00 /usr/sbin/asterisk -f -U asterisk -G asterisk -vvvg -c
1717 asterisk 20 0 1217M 33104 12804 S 0.0 1.7 0:00.00 /usr/sbin/asterisk -f -U asterisk -G asterisk -vvvg -c
1718 asterisk 20 0 1217M 33104 12804 S 0.0 1.7 0:00.00 /usr/sbin/asterisk -f -U asterisk -G asterisk -vvvg -c
1719 asterisk 20 0 1217M 33104 12804 S 0.0 1.7 0:00.00 /usr/sbin/asterisk -f -U asterisk -G asterisk -vvvg -c
1720 asterisk 20 0 1217M 33104 12804 S 0.0 1.7 0:00.00 /usr/sbin/asterisk -f -U asterisk -G asterisk -vvvg -c
1721 asterisk 20 0 1217M 33104 12804 S 0.0 1.7 0:00.00 /usr/sbin/asterisk -f -U asterisk -G asterisk -vvvg -c
1722 asterisk 20 0 1217M 33104 12804 S 0.0 1.7 0:00.00 /usr/sbin/asterisk -f -U asterisk -G asterisk -vvvg -c
1723 asterisk 20 0 1217M 33104 12804 S 0.0 1.7 0:00.00 /usr/sbin/asterisk -f -U asterisk -G asterisk -vvvg -c
1725 asterisk 20 0 1217M 33104 12804 S 0.0 1.7 0:00.00 /usr/sbin/asterisk -f -U asterisk -G asterisk -vvvg -c
1726 asterisk 20 0 1217M 33104 12804 S 0.0 1.7 0:00.00 /usr/sbin/asterisk -f -U asterisk -G asterisk -vvvg -c
1727 asterisk 20 0 1217M 33104 12804 S 0.0 1.7 0:00.00 /usr/sbin/asterisk -f -U asterisk -G asterisk -vvvg -c
1728 asterisk 20 0 1217M 33104 12804 S 0.0 1.7 0:00.00 /usr/sbin/asterisk -f -U asterisk -G asterisk -vvvg -c
1729 asterisk 20 0 1217M 33104 12804 S 0.0 1.7 0:00.00 /usr/sbin/asterisk -f -U asterisk -G asterisk -vvvg -c
1730 asterisk 20 0 1217M 33104 12804 S 0.0 1.7 0:00.00 /usr/sbin/asterisk -f -U asterisk -G asterisk -vvvg -c
1731 asterisk 20 0 1217M 33104 12804 S 0.0 1.7 0:00.00 /usr/sbin/asterisk -f -U asterisk -G asterisk -vvvg -c
1732 asterisk 20 0 1217M 33104 12804 S 0.0 1.7 0:00.00 /usr/sbin/asterisk -f -U asterisk -G asterisk -vvvg -c
1733 asterisk 20 0 1217M 33104 12804 S 0.0 1.7 0:00.00 /usr/sbin/asterisk -f -U asterisk -G asterisk -vvvg -c
1734 asterisk 20 0 1217M 33104 12804 S 0.0 1.7 0:00.00 /usr/sbin/asterisk -f -U asterisk -G asterisk -vvvg -c
1735 asterisk 20 0 1217M 33104 12804 S 0.0 1.7 0:00.00 /usr/sbin/asterisk -f -U asterisk -G asterisk -vvvg -c
1736 asterisk 20 0 1217M 33104 12804 S 0.0 1.7 0:00.00 /usr/sbin/asterisk -f -U asterisk -G asterisk -vvvg -c
1737 asterisk 20 0 1217M 33104 12804 S 0.0 1.7 0:00.00 /usr/sbin/asterisk -f -U asterisk -G asterisk -vvvg -c
1738 asterisk 20 0 1217M 33104 12804 S 0.0 1.7 0:00.00 /usr/sbin/asterisk -f -U asterisk -G asterisk -vvvg -c
1739 asterisk 20 0 1217M 33104 12804 S 0.0 1.7 0:00.00 /usr/sbin/asterisk -f -U asterisk -G asterisk -vvvg -c
1740 asterisk 20 0 1217M 33104 12804 S 0.7 1.7 0:00.16 /usr/sbin/asterisk -f -U asterisk -G asterisk -vvvg -c
1745 asterisk 20 0 1217M 33104 12804 S 0.0 1.7 0:00.00 /usr/sbin/asterisk -f -U asterisk -G asterisk -vvvg -c
F1Help F2Setup F3SearchF4FilterF5Tree F6SortByF7Nice -F8Nice +F9Kill F10Quit
Bah sorry cant edit post to clean that up :(.
I should also mention im running this vm on esx 5.1
I don’t know what could be the problem (if any, at this point).
I have a pretty little physical machine that has a system uptime of about 29 day 14 hours (Asterisk itself was restarted/reloaded many times due to updates) and its memory details are (just as an example to you):
top - 07:55:19 up 29 days, 14:04, 1 user, load average: 0.22, 0.14, 0.05
Tasks: 137 total, 1 running, 136 sleeping, 0 stopped, 0 zombie
Cpu0 : 4.0%us, 1.0%sy, 0.0%ni, 94.9%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu1 : 4.7%us, 1.0%sy, 0.0%ni, 94.3%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu2 : 1.7%us, 0.3%sy, 0.0%ni, 98.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu3 : 0.3%us, 0.7%sy, 0.0%ni, 99.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 1905232k total, 1655060k used, 250172k free, 301676k buffers
Swap: 785400k total, 47840k used, 737560k free, 734908k cached
As you can see the used RAM size has a value near the physical RAM size but I didn’t noticed nothing abnormal on the system (note that the “App” memory size reported by FreePBX GUI is 34% so there isn’t a clear relationship, or maybe I’m not able to see one, between what is reported by top/htop and what is reported on “App” memory size FreePBX GUI, at least to my eyes).
After a reboot the “App” memory size dropped to 14% (21% if I enter the GUI) and the used RAM size dropped to about 407 MBytes (about 570 MBytes once entered into the GUI) which is a value, compared to the one reported above before reboot, quite reasonable to me.
I only noticed that (htop, MEM% sorted) reports a lot of processes related to iSymphonyServer.jar on top.
Yeah i unintalled isymphony to see if that would help, but no luck. My production server has the same issue, when app memory gets into the red call quality starts to suffer.
I can give ssh access to someone if that would help troubleshoot the problem…
I’m having the same exact problem with a virtual instance of FreePBX - I rolled 64-bit Stable-4.211.64-8 and can report the same problem.
Restarting HTTPD did not seem to clear the memory utilization for me. I can post top output for anyone who can help.
Sounds like a virtual environment problem to me. I suggest installing on dedicated hardware and test.
I had thought of doing that. Of course, another FreePBX rolled from the same ISO in another VM environment doesn’t display this behavior.
To make matters more confusing, the virtual host reports that FreePBX is only using 768MB of host RAM, while the guest reports 80% utilization.
I can confirm the same issue in our VMWare ESXi 5.1 environment on a Dell T310 server with Perc Hardware raid. I have installed on standalone hardware and problem does not exist. Running version 2.11.0.11 64-bit of FreePBX distro.
I’m not really sure what to look for as the culprit, but it would be nice to get this running properly on virtual platforms. Unless you are running a analog to digital PCI card, I don’t know why you wouldn’t virtualize a system like this.
It appears that Java/iSymphony do not run properly on VMWare ESXi 5.1 X64. I think this is related to this thread: http://www.freepbx.org/forum/general-help/high-memory-usage-with-java-and-asterisk
On my machine Java/isymphony was using all of the memory, and also not reporting that it was consumed in vSphere for some reason.
I just disabled iSymphony and rebooted and I see a drastic improvement. I will continue to monitor:
service iSymphonyServer stop && chkconfig iSymphonyServer off
Would be nice to find the root of the problem though, and have this software working properly on virtual environments.
It also appears that DNSmasq was really slowing down our virtual FreePBX server. Reset our primary DNS server to 127.0.0.1 and did “service restart dnsmasq” and the web interface started loading much faster.
We had a problem in the past with DNSMasq interfering with our name resolution of our SIP providers, so I had switched to another DNS server. I guess you can’t do this with the way FreePBX distro is designed?
My hosted, virtualized FreePBX was running about 78% with 2G of memory. Today it’s running 39%. I did module updates, but I didn’t see when the usage changed. I did not disable iSymphony, wish I was of more help.
Yeah i did a module update a few weeks ago and the usage went down to <30% but if you keep refreshing the freepbx status page the app memory usage started to climb again. At the moment i’ve just stopped the httpd service after a reboot and it stops the memory usage growing out of control.
My memory is still reporting incorrectly. In vSphere it reports the server has only 875.00MB of Consumed Host Memory, but in the server using top it reports that approx 3GB is being used. I’ve got about 5 other CentOS, RedHat based distributions running on VMWare ESXi and I don’t see the same problem with those installations. Maybe something needs to be done to the FreePBX CentOS distro to make it virtual server compatible?
In the FreePBX System Status page it shows 79% App Memory used.
Although after changing the first DNS server to be 127.0.0.1 the FreePBX web interface is still loading way faster so that is good.
Well we dont build, test or support ESXI. We only test for and support KVM’s.
Surely a little ESX testing would be worth while, I can’t see many companies running wanting to run a physical server in an otherwise virtualised environment. I know of 1.8 servers that have been running on ESX for years flawlessly. I’m running plenty of other centos 6 VM’s on my ESX 5.1 server without issue.