App Memory / Memory Leak FreePBX 2.10 BUG

Whatever FOP is in the default install is in there.

The same server we upgraded PHP on yesterday is at this moment at 93% memory.
Do you want me to get you access, or should I reboot the process?
It’s a backup system not live so we can do a few things if necessary.

Warning: bad syntax, perhaps a bogus '-'? See /usr/share/doc/procps-3.2.7/FAQ
root      3578  0.0  0.1  26156  1012 ?        Ss   Feb16   0:02 /usr/sbin/httpd
asterisk  3580  0.8  9.2 128828 47548 ?        S    Feb16   9:08 /usr/sbin/httpd
asterisk  3581  0.8  9.0 129112 46800 ?        S    Feb16   9:08 /usr/sbin/httpd
asterisk  3582  0.8  9.3 130092 48080 ?        S    Feb16   9:10 /usr/sbin/httpd
asterisk  3583  0.8  9.1 129068 46832 ?        S    Feb16   9:04 /usr/sbin/httpd
asterisk  3584  0.8  9.1 129064 47084 ?        S    Feb16   9:06 /usr/sbin/httpd
asterisk  3585  0.8  9.2 129284 47392 ?        S    Feb16   9:14 /usr/sbin/httpd
asterisk  3586  0.8  9.3 129356 47872 ?        S    Feb16   9:07 /usr/sbin/httpd
asterisk  3587  0.8  9.2 128524 47340 ?        S    Feb16   9:11 /usr/sbin/httpd
asterisk  6695  0.6  5.3  51532 27300 ?        S    07:58   1:05 /usr/sbin/httpd
root     20671  0.0  0.1   4056   684 pts/0    S+   10:39   0:00 grep -i httpd

For those of you that are experiencing this same problem, can you please answer the following questions:

1 - Are running a “physical” server or a VPS?
2 - If it’s a real server can you please tell me what brand and model it is?
3 - How much memory do you have?
4 - Do you still have the problem?

Thank you!

VoIPTek, same issue with 1.89.210.57-2.
After one day of operation (without telephone load but with a browser remained open on the management window) memory is near to 100% and swap continuously increasing.

Did you find any solution?

Thanks

Carlo

top - 23:17:35 up 1 day, 4:51, 1 user, load average: 0.42, 0.40, 0.36
Tasks: 105 total, 2 running, 103 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.0%us, 0.3%sy, 0.0%ni, 99.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 384860k total, 373104k used, 11756k free, 5116k buffers
Swap: 779144k total, 349024k used, 430120k free, 38344k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1 root 15 0 2160 556 536 S 0.0 0.1 0:00.38 init
2 root RT -5 0 0 0 S 0.0 0.0 0:00.00 migration/0
3 root 34 19 0 0 0 S 0.0 0.0 0:00.01 ksoftirqd/0
4 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/0
5 root 10 -5 0 0 0 S 0.0 0.0 0:00.52 events/0
6 root 20 -5 0 0 0 S 0.0 0.0 0:00.00 khelper
7 root 11 -5 0 0 0 S 0.0 0.0 0:00.00 kthread
10 root 10 -5 0 0 0 S 0.0 0.0 0:00.97 kblockd/0
11 root 20 -5 0 0 0 S 0.0 0.0 0:00.00 kacpid
48 root 20 -5 0 0 0 S 0.0 0.0 0:00.00 cqueue/0
51 root 10 -5 0 0 0 S 0.0 0.0 0:00.05 khubd
53 root 14 -5 0 0 0 S 0.0 0.0 0:00.00 kseriod
117 root 15 0 0 0 0 S 0.0 0.0 0:00.00 khungtaskd
118 root 15 0 0 0 0 S 0.0 0.0 0:06.54 pdflush
119 root 15 0 0 0 0 S 0.0 0.0 0:09.28 pdflush
120 root 10 -5 0 0 0 S 0.0 0.0 0:00.96 kswapd0
121 root 17 -5 0 0 0 S 0.0 0.0 0:00.00 aio/0
270 root 11 -5 0 0 0 S 0.0 0.0 0:00.00 kpsmoused
290 root 17 -5 0 0 0 S 0.0 0.0 0:00.00 ata/0
291 root 17 -5 0 0 0 S 0.0 0.0 0:00.00 ata_aux
294 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 scsi_eh_0
301 root 14 -5 0 0 0 S 0.0 0.0 0:00.00 kstriped
310 root 10 -5 0 0 0 S 0.0 0.0 0:22.67 kjournald
335 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 kauditd
368 root 21 -4 2396 360 360 S 0.0 0.1 0:00.52 udevd
772 root 14 -5 0 0 0 S 0.0 0.0 0:00.00 iprt/0
1131 root 20 -5 0 0 0 S 0.0 0.0 0:00.00 kmpathd/0
1132 root 20 -5 0 0 0 S 0.0 0.0 0:00.00 kmpath_handlerd
1189 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 kjournald
1321 root 13 -5 0 0 0 S 0.0 0.0 0:00.00 iscsi_eh
1349 root 20 -5 0 0 0 S 0.0 0.0 0:00.00 cnic_wq
1369 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 ib_addr
1376 root 20 -5 0 0 0 S 0.0 0.0 0:00.00 ib_mcast
1377 root 20 -5 0 0 0 S 0.0 0.0 0:00.00 ib_inform
1378 root 20 -5 0 0 0 S 0.0 0.0 0:00.00 local_sa
1381 root 20 -5 0 0 0 S 0.0 0.0 0:00.00 iw_cm_wq
1386 root 20 -5 0 0 0 S 0.0 0.0 0:00.00 ib_cm/0
1388 root 20 -5 0 0 0 S 0.0 0.0 0:00.00 rdma_cm
1404 root 24 0 32704 280 280 S 0.0 0.1 0:00.00 brcm_iscsiuio
1410 root 15 0 3704 340 320 S 0.0 0.1 0:04.75 iscsid
1411 root 5 -10 4160 4156 3156 S 0.0 1.1 0:00.03 iscsid
1755 root 15 -4 12628 616 532 S 0.0 0.2 0:00.07 auditd
1757 root 7 -8 12164 628 576 S 0.0 0.2 0:00.02 audispd
1787 root 16 0 1816 520 476 S 0.0 0.1 0:01.12 syslogd
1790 root 15 0 1764 320 320 S 0.0 0.1 0:00.00 klogd
1875 rpc 21 0 1912 424 424 S 0.0 0.1 0:00.00 portmap
1910 root 12 -5 0 0 0 S 0.0 0.0 0:00.00 rpciod/0
1916 rpcuser 24 0 1964 592 592 S 0.0 0.2 0:00.00 rpc.statd
1952 root 25 0 5936 240 240 S 0.0 0.1 0:00.00 rpc.idmapd
1975 dbus 15 0 2848 864 724 S 0.0 0.2 0:00.09 dbus-daemon
2007 root 22 0 2256 552 552 S 0.0 0.1 0:00.00 hcid
2018 root 23 0 1832 368 368 S 0.0 0.1 0:00.00 sdpd
2031 root 9 -10 0 0 0 S 0.0 0.0 0:00.00 krfcommd
2075 root 25 0 12828 520 428 S 0.0 0.1 0:03.70 pcscd
2089 root 18 0 1760 448 448 S 0.0 0.1 0:00.00 acpid
2206 haldaemo 15 0 5984 1628 1308 S 0.0 0.4 0:30.82 hald
2207 root 25 0 3264 912 912 S 0.0 0.2 0:00.03 hald-runner
2216 haldaemo 25 0 2108 692 692 S 0.0 0.2 0:00.00 hald-addon-acpi
2222 haldaemo 25 0 2108 688 688 S 0.0 0.2 0:00.00 hald-addon-keyb
2235 root 16 0 2060 608 560 S 0.0 0.2 2:11.55 hald-addon-stor
2256 root 23 0 2008 336 336 S 0.0 0.1 0:00.00 hidd
2295 root 21 0 29424 940 832 S 0.0 0.2 0:01.50 automount
2382 root 25 0 8660 552 472 S 0.0 0.1 0:06.64 VBoxService
2405 root 15 0 7212 756 660 S 0.0 0.2 0:01.93 sshd
2421 root 18 0 2836 556 556 S 0.0 0.1 0:00.01 xinetd
2437 ntp 15 0 4512 4508 3496 S 0.0 1.2 0:00.26 ntpd
2474 root 25 0 4632 1028 1028 S 0.0 0.3 0:00.02 mysqld_safe
2524 mysql 15 0 137m 4688 2624 S 0.0 1.2 1:02.77 mysqld
2596 root 20 0 6968 1492 1420 S 0.0 0.4 0:00.55 master
2606 postfix 15 0 7092 1468 1404 S 0.0 0.4 0:00.30 qmgr
2611 root 18 0 2000 236 236 S 0.0 0.1 0:00.00 gpm
2625 root 15 0 25824 4340 3572 S 0.0 1.1 0:01.42 httpd
2654 root 25 0 4632 396 396 S 0.0 0.1 0:00.00 safe_asterisk
2666 asterisk 15 0 49644 5020 3580 S 0.0 1.3 1:31.91 asterisk
2672 root 18 0 5380 624 556 S 0.0 0.2 0:00.19 crond
2689 asterisk 18 0 94392 37m 3536 S 0.0 9.8 2:29.32 httpd
2690 asterisk 15 0 94908 35m 3832 S 0.0 9.6 2:29.79 httpd
2691 asterisk 20 0 94236 36m 3564 S 0.0 9.7 2:26.74 httpd
2692 asterisk 22 0 94908 36m 3548 S 0.0 9.7 2:29.30 httpd
2693 asterisk 18 0 94072 36m 3548 S 0.0 9.7 2:29.08 httpd
2694 asterisk 18 0 94452 37m 3836 S 0.0 9.9 2:27.76 httpd
2695 asterisk 21 0 95228 38m 3536 S 0.0 10.1 2:27.96 httpd
2696 asterisk 21 0 94332 37m 3544 S 0.0 9.9 2:28.77 httpd
2704 xfs 18 0 3400 576 544 S 0.0 0.1 0:00.00 xfs
2730 root 18 0 110m 2808 1580 S 0.0 0.7 0:25.48 fail2ban-server
2732 root 15 0 2768 1052 896 S 0.0 0.3 0:12.77 gam_server
2801 root 24 0 4628 1008 1008 S 0.0 0.3 0:00.01 startup.sh
2822 root 24 0 4632 1020 1020 S 0.0 0.3 0:00.01 bash
2838 root 18 0 2360 352 312 S 0.0 0.1 0:00.00 atd
2841 root 19 0 255m 5052 1268 S 0.0 1.3 0:10.71 java
2863 root 24 0 3068 508 508 S 0.0 0.1 0:00.00 incrond
2890 nobody 18 0 1916 316 292 S 0.0 0.1 0:00.02 dnsmasq
2995 root 34 19 25904 3576 2116 S 0.0 0.9 0:00.33 yum-updatesd
3039 root 15 0 3612 252 236 S 0.0 0.1 0:00.00 smartd
3042 root 17 0 1748 396 396 S 0.0 0.1 0:00.01 mingetty
3043 root 18 0 1748 376 376 S 0.0 0.1 0:00.00 mingetty
3044 root 18 0 1748 396 396 S 0.0 0.1 0:00.00 mingetty
3045 root 19 0 1748 376 376 S 0.0 0.1 0:00.00 mingetty
3046 root 20 0 1748 396 396 S 0.0 0.1 0:00.00 mingetty
3047 root 21 0 1748 376 376 S 0.0 0.1 0:00.00 mingetty
7043 postfix 18 0 7032 1816 1464 S 0.0 0.5 0:00.07 pickup
9716 asterisk 18 0 41876 19m 3672 S 0.0 5.3 0:04.88 httpd
12517 root 15 0 2408 996 784 R 0.0 0.3 0:00.01 top
16424 root 15 0 10200 2452 2228 R 0.0 0.6 0:00.48 sshd
16431 root 15 0 4636 1308 1148 S 0.0 0.3 0:00.09 bash

The bug is clearly related to the use of the System Administration GUI.
In a night, keeping any Management window carefully closed, the system didn’t increase memory usage.

One of my initial hunches was that keeping the status report open was possibly using up the memory, however on boxes that I was looking at frequently when I left the management page in the evening and opened it the next day memory DID increase buy a good jump.

With all the recent updates I have seen an improvement, memory is not disappearing as quickly. I will leave my servers status screen open today and see if we can confirm that one way or another.

crogialli, I see your current and I think at the moment that’s about all you can do. I would also say stay out of the status or for that matter the entire GUI to see if that improves the situation.

Are you running a real server or are you running a VPS?

It’s in a Virtual Machine, over a Mac OSX server.
Until it’s fixed, I’ll try to stay out the gui and I think I’ll schedule a machine restart every night at 3:00 am.

you don’t need to restart the whole system, simply restart apache and the memory clears up.

What “System Administration GUI” display page did you have opened that you observed this?

If you can create a 'reasonably reproducible" situation please feel free to file a bug so one of the developers can have a closer look.

Please make sure to provide details of FreePBX, apache, php, etc. versions that are being run in the ticket.

The browser was open on the FreePBX auto-reloading status page.
So - when left unadvertently open- it reloads at least one hundred times in a night.
This is the reason why IMHO it is a reasonable suspect, because the page auto-generates http sessions.

Thanks

Carlo

Thanks VoIPTek, sure I will follow your advice.

Carlo

When this was first opened memory was disappearing at an alarming rate to the point that you would have to reboot apache daily. After all the updates from when this was originally opened, I’m not sure the problem still exists.

The screen I believe that people run is the FreePBX System status which also happens to tell you the percentage of memory.

In my testing today keeping the GUI status open I do see I went from 20-22% but thats nothing, I would literally run out of memory within a day, so that has improved with all the updates. Now it may not be related to the GUI because I did have the problems as stated previously.

I don’t believe the problem is there at the level it was before because I could duplicate it on 6 different servers and right now I really cant say I can duplicate it.

crogialli, run yum check-update from the command line of the server, does it tell you that you need any updates? I know you ran the scripts already and should be current, this is just a way to double check. Also after your updates have you rebooted?

My system is updated except for the very minor release of ARI Framework and PBX Framework that I think have been introduced this night (my rev 2.10.0.0, current 2.10.0.1 for the two modules).

I understand the leak problem is far smaller than what you saw at the beginning of the thread, but on the other side the leak problem still exists and it is almost certainly related to status page reload.

The reason why I can monitor quite precisely the memory occupation is that I am running the system on a Virtual Machine on one of my servers, and the machine has been allocated with sufficient but not excessive resources. RAM is 384M, SWAP is 800 M.

If the status page is not continuously reloaded system stays between 57 and 59% RAM, ZERO swap. I left the system with 59% 24 hours ago, this evening is still 59%. So now it stays at a reassuring 59% since 48 hours, ZERO SWAP.

After One night (say 8 hrs) with a window partly open (the client computer went in stand-by at a certain time) the situation the following morning was 95% RAM and 70% SWAP.
I estimate my system, with the status page open, can survive more or less 24 hours before exhausting RAM and SWAP, not much more.
If i keep the status window open I can slowly see the RAM occupation increasing (1/2% in ten/twenty minutes). This probably is not so evident in systems with larger RAM amount, since the percentage increase in time is minor.
But I confirm, what I see is a modest memory leak in the apache related to status reloads.

I saw the same by reinstalling two times both 32 and 64bit version of the latest FreePBX distro.

  • duplicated -

I have started to take a look at this and have watched a system running distro 1.88.210.57-2 leaving the status page open, there is no doubt that memory usage does gradually increase as time passes whereas on a system with distro 1.87.29.55-2 memory usage remains absolutely rock solid and does not increase.

I found once I closed the status page and released the http connection memory use immediately stabilized on the 1.88.210.57-2 system.

I am happy to work with everyone to do whatever tests you might be required to track this issue down, please let me know if I can help. I do have a test system available that is up and running and is noot in production.

I’d like to get everyone here focused on the same page by starting with some minor “education” …

A web application can’t have a memory leak per-se over the course of staying up or being run multiple times.

It is not like Asteirsk which is started up in a daemon and run continuously in the background thus having the ability to continuously consume memory. Once a page is loaded or an ajax request is fulfilled, which is technically no different from the server’s perspective, the application ends and all resources that it has requested are relinquished. During the course of a page load it may use a lot of memory but then it id finished.

So with this in mind it’s important to try and focus a bit more on WHAT is actually leaking memory, WHY and what might be influencing it. It is clearly possible that an application may be doing something in particular that is triggering a memory leak elsewhere. In this case Apache which is thus most likely related to mod_php which is embedded in Apache and which is what is used to compile and interpret the page execution.

Now one thing that seems to be noted here is that the System Status page, when left up, exposes the apparent memory leak. Why is that? Well System Status page fires off ajax calls at regular intervals, which to the server is no different then if you sat there refreshing the page at those same intervals. So it is basically exercising page loads multiple times per minute thus potentially triggering what ever it may be in apache/mod_php that is creating this memory leak. As far as a bandaid, there is an advanced setting to change the frequency of the ajax refreshes to that page so if you do leave it up it won’t refresh as often which you may want to look at.

If someone wants to try and dig further, I would suggesting starting something really simple. First, create a page script that does almost nothing other than assures the php interpreter is engaged, and provides an ajax call that continuously reloads that page at a very rapid rate. That will tell us if the mere loading of a virtually empty PHP page is related to this. If that does not pan out, then augment that by using our existing libraries to open up a MySQL connection as FreePBX does when these ajax calls are called and go through the same thing to see if a memory leak is exposed. If not, do the same thing but with an Asterisk Manager connection as we do.

The bottom line here is … staying focused on WHERE the actually memory leak could be is going to to a lot further then thinking “The System Status Page” or FreePBX 2.10 has a memory leak. One or both of those could very likely be exercising something in the subsystems that they use that has the actual memory leak which is the point of all this, but staying focused on the fact that the web application can’t have a memory leak, per-se, will help to keep from going in circles.

I’ll close with one comment, the web application could be responsible for a memory leak in one place, which would be if it had a continuous build up of the php SESSION that is reloaded and does have a sense of “memory” across page loads, but that is something that should be pretty trivial to test for by simply adding a dbug() statement to dump the session every page load in the System Status and view if something is growing in there…

I am afraid my knowledge on Ajax is very limited so I am not qualified to write a script, however, I did put up a page that refreshed every second through a meta tag and ran a short piece of php code to print a message on the screen. I left this running for an hour and saw no increase in memory usage.

Not sure if this helps but I think it demonstrates that continually loading pages does not appear to affect memory usage.

stonet,

that helps to determine that just engaging PHP doesn’t have the affect. It’ll be a matter of working up from there to determine what we are engaging in php that may be affecting this.

Just an update, I ran the status page for 24+ hours with no change in the amount of memory being available on one server while on another server there were no pages or references open and it went from what was previously reported as 20% - 22% jumping to now 26%. This is not an alarming amount and everything is stable.
Both of these are physical servers.

On two virtual servers I did see memory in the high 30%-40% range, these servers were not looked at or monitored from the web GUI for several days. If I restarted apache on these two servers memory would go down to around 20% and literally climb backup up right in front of me within a few minutes back to around the 30%-40% however it would stabalize there. I’m not sure if that really tells me much, but it does point to some process which is being run upon starting apache and using that additional memory, however it is not “leaking” per say, once it reaches that point that everything is loaded it appears to be stable within a percentage or two.

To review, before all of the updates over the last 2 weeks the problem was significantly different putting the server in a critical condition and affecting voice quality because of the memory loss. The only way to correct this was to reboot apache or as some people did reboot the box. This has improved a lot and I believe that some other core application is responsible for the loss and the GUI simply made a call to that application further exhausting the memory problem and giving the appearance that the GUI was somehow responsible for that loss.

VoIPTek - I read this 3 times and I am not sure what you are trying to communicate. Do you no longer think it is the status page?

Sorry it didn’t make sense, I was just stating what I observed since the recent updates.

The original problem was not directly related to the status page because the memory would disappear through whatever unknown process and if you had checked the status page lets say at 9:00am and then checked the status page 3 hours later the memory was gone. The memory never returned when you stayed away from the status page either, so yes the GUI was not directly related to the problem in my opinion.

I believe the problem is gone, however others are experiencing it. I left one of my servers on the status page for around 3 days now and the used memory is stable.