App Memory / Memory Leak FreePBX 2.10 BUG

VoIPTek · February 9, 2012, 9:45pm

Hello All,

I’m running the latest of everything from the distro and then upgraded via the shell script upgrade-1.88.210.57-1.sh.
I verify that everything is updated both via yum and module admin, but over the course of 24 hours or so you can see via the web interface the App memory increasing into the red, eventually causing voice quality issues.

I spent a couple of hours looking for anything that made sense with no luck, until I finally decided to reboot the apache process being at that time the biggest user of memory and that was it, memory dropped from 93% to 14% ( 2GB Server ).

I can duplicate this on multiple servers and several times in a row so the question becomes is it something FreePBX is doing or is it apache.

I can see all the servers are running:
httpd-2.2.19-1.w5
httpd-tools-2.2.19-1.w5

I tried to force a reinstall in case maybe something got corrupted, but all I get is Installed package httpd-2.2.19-1.w5.i386 not available.

Any help appreciated.

wvroger · February 10, 2012, 12:50am

I am experiencing the same issue on my servers. To temp fix the problem I setup a cron job to restart httpd…it’s a ugly to do it, but its a temp fix to the problem.

tonyclewis · February 10, 2012, 2:54am

Are you running anything like FOP, FOP2 or other 3rd party applications. This should not be occurring and I have not seen this on any of our systems nor on any support customers as of yet.

VoIPTek · February 10, 2012, 3:09am

I’m not running anything extra or 3rd party…
I was debating setting up the cron job idea but was hoping to find something to fix the problem before trying that.

I have several servers and not all of them are experiencing this.
I have a case where I have 2 identical servers for different clients and one is having the problem to the point that it will within 24 hours crash and the other is running perfect. They are both upgraded from 2.9x.

My hunch is that there is some configuration entry which for lack of a better term is corrupted, has additional or wrong info which may be causing this.

One other note is on one of them both the primary and backup server have the same issue which is part of whats leaning me towards thinking a potential configuration type of issue.

VoIPTek · February 10, 2012, 3:17am

I just saw there are 5 updates, I will run those and see if by chance things get better unless anyone has some suggestions. Updates just released:

Backup & Restore 2.10.0.14 will be upgraded to online version 2.10.0.15
Core 2.10.0rc1.0 will be upgraded to online version 2.10.0rc1.1
FreePBX Framework 2.10.0rc1.0 will be upgraded to online version 2.10.0rc1.1
Paging and Intercom 2.10.0.2 will be upgraded to online version 2.10.0.3
Queues 2.10.0.1 will be upgraded to online version 2.10.0.3

losbravos · February 10, 2012, 3:20pm

We are experiencing the same issues. Our App Memory was at 11% (2GB Installed) in the morning and by evening had risen to 54%. By the next morning it was 72%. I shut down the system and installed another 2 GB of ram and it dropped to 4%. Now this morning it is up to 29% already. Not sure what to do now or what is causing it. We are running 2.10 FreePBX (all latest basic updates installed)

VoIPTek · February 10, 2012, 3:19pm

At this point simply restart apache and instead of restarting the box and loosing / dropping calls until a fix comes out.

Tony, if you want I can give you access to a box where we can duplicate this all day long.

tonyclewis · February 11, 2012, 6:18am

Please get a print out of top so we can see the processes and what each process is using for ram.

Also can you verify the php version and apache version are the same between the server with the issue and the server without the issue. Also are both of your systems built from the same Distro and kernel version?

VoIPTek · February 12, 2012, 3:42am

Hi Tony,

Both boxes are identical, apache version httpd-2.2.19-1.w5 and php php-5.3.3-1.3

The were all built from the same 2.9x and upgrade the same exact way via scripts.

There are 2 PDF links below from one of the boxes with the problem using ps -auxw

Results with 83% memory usage

Results after restart apache with 12% memory usage

SkykingOH · February 12, 2012, 8:16am

It’s Asterisk. Something is up. My first instinct would be a reinstall. Let’s see what other suggestions are offered.

VoIPTek · February 12, 2012, 5:22pm

Hi SkykingOH,

I don’t believe asterisk is the issue, it’s inconsistent across servers both physical and virtual but is duplicated so it seems difficult to believe there were several bad installs.

I tried a reinstall of just asterisk, no difference I also restarted the asterisk process while the memory was at 93% and no change, of course thats not a completely accurate way to test that.

VoIPTek · February 13, 2012, 12:21pm

93% Memory
{{{
Warning: bad syntax, perhaps a bogus '-'? See /usr/share/doc/procps-3.2.7/FAQ
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.0   2208   380 ?        Ss   Feb10   0:03 init [3]  
root         2  0.0  0.0      0     0 ?        S    Feb10   0:00 [kthreadd]
root         3  0.0  0.0      0     0 ?        S    Feb10   0:00 [ksoftirqd/0]
root         4  0.0  0.0      0     0 ?        S    Feb10   0:03 [migration/0]
root         5  0.0  0.0      0     0 ?        S    Feb10   0:03 [migration/1]
root         6  0.0  0.0      0     0 ?        S    Feb10   0:00 [ksoftirqd/1]
root         7  0.0  0.0      0     0 ?        S    Feb10   0:09 [events/0]
root         8  0.0  0.0      0     0 ?        S    Feb10   0:18 [events/1]
root         9  0.0  0.0      0     0 ?        S    Feb10   0:05 [khelper]
root        12  0.0  0.0      0     0 ?        S    Feb10   0:00 [netns]
root        13  0.0  0.0      0     0 ?        S    Feb10   0:00 [async/mgr]
root        16  0.0  0.0      0     0 ?        S    Feb10   0:00 [xenwatch]
root        17  0.0  0.0      0     0 ?        S    Feb10   0:00 [xenbus]
root        60  0.0  0.0      0     0 ?        S    Feb10   0:01 [sync_supers]
root        62  0.0  0.0      0     0 ?        S    Feb10   0:00 [bdi-default]
root        64  0.0  0.0      0     0 ?        S    Feb10   0:12 [kblockd/0]
root        65  0.0  0.0      0     0 ?        S    Feb10   0:10 [kblockd/1]
root        70  0.0  0.0      0     0 ?        S    Feb10   0:00 [kseriod]
root       178  5.5  0.0      0     0 ?        S    Feb10 258:55 [kswapd0]
root       179  0.0  0.0      0     0 ?        S    Feb10   0:00 [aio/0]
root       180  0.0  0.0      0     0 ?        S    Feb10   0:00 [aio/1]
root       181  0.0  0.0      0     0 ?        S    Feb10   0:00 [crypto/0]
root       182  0.0  0.0      0     0 ?        S    Feb10   0:00 [crypto/1]
root       193  0.0  0.0      0     0 ?        S    Feb10   0:00 [khvcd]
root       219  0.0  0.0      0     0 ?        S    Feb10   0:17 [kjournald]
root       243  0.0  0.0      0     0 ?        S    Feb10   0:00 [kauditd]
root       271  0.0  0.0      0     0 ?        S    Feb10   0:03 [flush-202:0]
root       277  0.0  0.0   2444   340 ?        S<s  Feb10   0:00 /sbin/udevd -d
root       672  0.0  0.0      0     0 ?        S    Feb10   0:00 [kjournald]
root      1072  0.0  0.0  10624   356 ?        S<sl Feb10   0:01 auditd
root      1074  0.0  0.0  11184   292 ?        S<sl Feb10   0:01 /sbin/audispd
root      1094  0.0  0.0   1860   408 ?        Ss   Feb10   0:12 syslogd -m 0
root      1099  0.0  0.0   1808   352 ?        Ss   Feb10   0:00 klogd -x
root      1126  0.0  0.0   2608   312 ?        Ss   Feb10   0:17 irqbalance
rpc       1162  0.0  0.0   1956   388 ?        Ss   Feb10   0:00 portmap
rpcuser   1189  0.0  0.1   2012   576 ?        Ss   Feb10   0:00 rpc.statd
dbus      1230  0.0  0.0   2896   364 ?        Ss   Feb10   0:00 dbus-daemon --system
root      1292  0.0  0.0  10824   316 ?        Ssl  Feb10   0:14 pcscd
68        1366  0.0  0.1   5220   536 ?        Ss   Feb10   0:00 hald
root      1367  0.0  0.0   3308   284 ?        S    Feb10   0:00 hald-runner
root      1418  0.0  0.1   7256   640 ?        Ss   Feb10   0:25 /usr/sbin/sshd
root      1439  0.0  0.0   2884   496 ?        Ss   Feb10   0:00 xinetd -stayalive -pidfile /var/run/xinetd.pid
ntp       1450  0.0  0.8   4552  4548 ?        SLs  Feb10   0:15 ntpd -u ntp:ntp -p /var/run/ntpd.pid -g
root      1484  0.0  0.1   4680   572 ?        S    Feb10   0:00 /bin/sh /usr/bin/mysqld_safe --datadir=/var/lib/mysql --socket=/var/lib/mysql/mysql.sock -
root      1601  0.0  0.1   7012   968 ?        Ss   Feb10   0:03 /usr/libexec/postfix/master
postfix   1610  0.0  0.1   7232   732 ?        S    Feb10   0:00 qmgr -l -t fifo -u
root      1611  0.0  0.0   2040   148 ?        Ss   Feb10   0:00 gpm -m /dev/input/mice -t exps2
root      1655  0.0  0.0   5384   500 ?        Ss   Feb10   0:01 crond
xfs       1674  0.0  0.0   3440   372 ?        Ss   Feb10   0:00 xfs -droppriv -daemon
root      1706  0.1  0.3  92916  1712 ?        Sl   Feb10   5:12 /usr/bin/python /usr/bin/fail2ban-server -b -s /var/run/fail2ban/fail2ban.sock
root      1708  0.0  0.1   2836   724 ?        S    Feb10   1:07 /usr/libexec/gam_server
root      1768  0.0  0.0   2408   208 ?        Ss   Feb10   0:00 /usr/sbin/atd
root      1787  0.0  0.0   3104   460 ?        Ss   Feb10   0:00 incrond
avahi     1796  0.0  0.0   2736   376 ?        Ss   Feb10   0:00 avahi-daemon: running [bpbx.local]
avahi     1797  0.0  0.0   2736   128 ?        Ss   Feb10   0:00 avahi-daemon: chroot helper
nobody    1814  0.0  0.0   1984   324 ?        S    Feb10   0:00 dnsmasq
asterisk  1920  0.0  0.0   4632   176 ?        S    Feb10   0:00 bash -c cd /var/www/html/admin/modules/fw_fop && /var/www/html/admin/modules/fw_fop/safe_o
asterisk  1921  0.0  0.1   4628   520 ?        S    Feb10   0:00 sh /var/www/html/admin/modules/fw_fop/safe_opserver
asterisk  1923  0.1  0.6  13052  3472 ?        S    Feb10   8:16 /usr/bin/perl /var/www/html/admin/modules/fw_fop/op_server.pl
root      1935  0.0  0.0   3652   152 ?        S    Feb10   0:00 /usr/sbin/smartd -q never
root      1938  0.0  0.0   1792   392 tty1     Ss+  Feb10   0:00 /sbin/mingetty tty1
root      2191  0.0  0.2  10084  1328 ?        Ss   Feb11   0:00 sshd: root@pts/0 
root      2195  0.0  0.1   4676   540 pts/0    Ss+  Feb11   0:00 -bash
asterisk 13253  1.6  4.7  47332 24420 ?        S    06:21   0:59 /usr/sbin/httpd
postfix  15568  0.0  0.2   7076  1124 ?        S    06:46   0:00 pickup -l -t fifo -u
asterisk 17568  2.2  2.4  35168 12512 ?        S    07:18   0:02 /usr/sbin/httpd
root     17716  0.2  0.5  10084  2860 ?        Ss   07:19   0:00 sshd: root@pts/2 
root     17722  2.3  0.2   4676  1340 pts/2    Ss   07:19   0:00 -bash
root     17761  0.0  0.1   4396   952 pts/2    R+   07:19   0:00 ps -auxw
mysql    30421  0.2  0.6 123188  3468 ?        Sl   Feb11   4:44 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --pid-file=/var/r
root     31284  0.0  0.0   4624   140 ?        S    Feb12   0:00 /bin/sh /usr/sbin/safe_asterisk -U asterisk -G asterisk -g
asterisk 31287  0.9  0.3  43284  1916 ?        Sl   Feb12  10:45 /usr/sbin/asterisk -f -U asterisk -G asterisk -g -vvvg -c
root     31427  0.0  0.2  26104  1088 ?        Ss   Feb12   0:03 /usr/sbin/httpd
asterisk 31431  1.5 11.3 149860 58316 ?        S    Feb12  17:31 /usr/sbin/httpd
asterisk 31432  1.5 11.3 149916 58408 ?        S    Feb12  17:35 /usr/sbin/httpd
asterisk 31433  1.5 11.2 148928 57824 ?        S    Feb12  17:33 /usr/sbin/httpd
asterisk 31434  1.5 11.6 150812 60032 ?        S    Feb12  17:31 /usr/sbin/httpd
asterisk 31435  1.5 11.8 153952 60920 ?        S    Feb12  17:35 /usr/sbin/httpd
asterisk 31436  1.5 11.6 150068 59868 ?        S    Feb12  17:29 /usr/sbin/httpd
asterisk 31485  1.5 11.9 153800 61632 ?        S    Feb12  17:28 /usr/sbin/httpd


after rebooting apache 9%
Warning: bad syntax, perhaps a bogus '-'? See /usr/share/doc/procps-3.2.7/FAQ
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.0   2208   384 ?        Ss   Feb10   0:03 init [3]  
root         2  0.0  0.0      0     0 ?        S    Feb10   0:00 [kthreadd]
root         3  0.0  0.0      0     0 ?        S    Feb10   0:00 [ksoftirqd/0]
root         4  0.0  0.0      0     0 ?        S    Feb10   0:03 [migration/0]
root         5  0.0  0.0      0     0 ?        S    Feb10   0:03 [migration/1]
root         6  0.0  0.0      0     0 ?        S    Feb10   0:00 [ksoftirqd/1]
root         7  0.0  0.0      0     0 ?        S    Feb10   0:09 [events/0]
root         8  0.0  0.0      0     0 ?        S    Feb10   0:18 [events/1]
root         9  0.0  0.0      0     0 ?        S    Feb10   0:05 [khelper]
root        12  0.0  0.0      0     0 ?        S    Feb10   0:00 [netns]
root        13  0.0  0.0      0     0 ?        S    Feb10   0:00 [async/mgr]
root        16  0.0  0.0      0     0 ?        S    Feb10   0:00 [xenwatch]
root        17  0.0  0.0      0     0 ?        S    Feb10   0:00 [xenbus]
root        60  0.0  0.0      0     0 ?        S    Feb10   0:01 [sync_supers]
root        62  0.0  0.0      0     0 ?        S    Feb10   0:00 [bdi-default]
root        64  0.0  0.0      0     0 ?        S    Feb10   0:12 [kblockd/0]
root        65  0.0  0.0      0     0 ?        S    Feb10   0:10 [kblockd/1]
root        70  0.0  0.0      0     0 ?        S    Feb10   0:00 [kseriod]
root       178  5.5  0.0      0     0 ?        S    Feb10 258:56 [kswapd0]
root       179  0.0  0.0      0     0 ?        S    Feb10   0:00 [aio/0]
root       180  0.0  0.0      0     0 ?        S    Feb10   0:00 [aio/1]
root       181  0.0  0.0      0     0 ?        S    Feb10   0:00 [crypto/0]
root       182  0.0  0.0      0     0 ?        S    Feb10   0:00 [crypto/1]
root       193  0.0  0.0      0     0 ?        S    Feb10   0:00 [khvcd]
root       219  0.0  0.0      0     0 ?        S    Feb10   0:17 [kjournald]
root       243  0.0  0.0      0     0 ?        S    Feb10   0:00 [kauditd]
root       271  0.0  0.0      0     0 ?        S    Feb10   0:03 [flush-202:0]
root       277  0.0  0.0   2444   344 ?        S<s  Feb10   0:00 /sbin/udevd -d
root       672  0.0  0.0      0     0 ?        S    Feb10   0:00 [kjournald]
root      1072  0.0  0.0  10624   308 ?        S<sl Feb10   0:01 auditd
root      1074  0.0  0.0  11184   276 ?        S<sl Feb10   0:01 /sbin/audispd
root      1094  0.0  0.0   1860   392 ?        Ss   Feb10   0:12 syslogd -m 0
root      1099  0.0  0.0   1808   352 ?        Ss   Feb10   0:00 klogd -x
root      1126  0.0  0.0   2608   320 ?        Ss   Feb10   0:17 irqbalance
rpc       1162  0.0  0.0   1956   388 ?        Ss   Feb10   0:00 portmap
rpcuser   1189  0.0  0.1   2012   580 ?        Ss   Feb10   0:00 rpc.statd
dbus      1230  0.0  0.0   2896   368 ?        Ss   Feb10   0:00 dbus-daemon --system
root      1292  0.0  0.0  10824   316 ?        Ssl  Feb10   0:14 pcscd
68        1366  0.0  0.1   5220   540 ?        Ss   Feb10   0:00 hald
root      1367  0.0  0.0   3308   284 ?        S    Feb10   0:00 hald-runner
root      1418  0.0  0.1   7256   640 ?        Ss   Feb10   0:25 /usr/sbin/sshd
root      1439  0.0  0.0   2884   500 ?        Ss   Feb10   0:00 xinetd -stayalive -pidfile /var/run/xinetd.pid
ntp       1450  0.0  0.8   4552  4548 ?        SLs  Feb10   0:15 ntpd -u ntp:ntp -p /var/run/ntpd.pid -g
root      1484  0.0  0.1   4680   576 ?        S    Feb10   0:00 /bin/sh /usr/bin/mysqld_safe --datadir=/var/lib/mysql --socket=/var/lib/mysql/mysql.sock -
root      1601  0.0  0.1   7012   916 ?        Ss   Feb10   0:03 /usr/libexec/postfix/master
postfix   1610  0.0  0.1   7232   732 ?        S    Feb10   0:00 qmgr -l -t fifo -u
root      1611  0.0  0.0   2040   148 ?        Ss   Feb10   0:00 gpm -m /dev/input/mice -t exps2
root      1655  0.0  0.0   5384   476 ?        Ss   Feb10   0:01 crond
xfs       1674  0.0  0.0   3440   372 ?        Ss   Feb10   0:00 xfs -droppriv -daemon
root      1706  0.1  0.3  92916  1968 ?        Sl   Feb10   5:12 /usr/bin/python /usr/bin/fail2ban-server -b -s /var/run/fail2ban/fail2ban.sock
root      1708  0.0  0.1   2836   724 ?        S    Feb10   1:07 /usr/libexec/gam_server
root      1768  0.0  0.0   2408   212 ?        Ss   Feb10   0:00 /usr/sbin/atd
root      1787  0.0  0.0   3104   460 ?        Ss   Feb10   0:00 incrond
avahi     1796  0.0  0.0   2736   376 ?        Ss   Feb10   0:00 avahi-daemon: running [bpbx.local]
avahi     1797  0.0  0.0   2736   132 ?        Ss   Feb10   0:00 avahi-daemon: chroot helper
nobody    1814  0.0  0.0   1984   328 ?        S    Feb10   0:00 dnsmasq
asterisk  1920  0.0  0.0   4632   180 ?        S    Feb10   0:00 bash -c cd /var/www/html/admin/modules/fw_fop && /var/www/html/admin/modules/fw_fop/safe_o
asterisk  1921  0.0  0.1   4628   524 ?        S    Feb10   0:00 sh /var/www/html/admin/modules/fw_fop/safe_opserver
asterisk  1923  0.1  0.7  13052  3740 ?        S    Feb10   8:17 /usr/bin/perl /var/www/html/admin/modules/fw_fop/op_server.pl
root      1935  0.0  0.0   3652   152 ?        S    Feb10   0:00 /usr/sbin/smartd -q never
root      1938  0.0  0.0   1792   396 tty1     Ss+  Feb10   0:00 /sbin/mingetty tty1
root      2191  0.0  0.2  10084  1328 ?        Ss   Feb11   0:00 sshd: root@pts/0 
root      2195  0.0  0.1   4676   540 pts/0    Ss+  Feb11   0:00 -bash
postfix  15568  0.0  0.2   7076  1080 ?        S    06:46   0:00 pickup -l -t fifo -u
root     17716  0.0  0.5  10084  2572 ?        Rs   07:19   0:00 sshd: root@pts/2 
root     17722  0.1  0.2   4676  1420 pts/2    Ss   07:19   0:00 -bash
root     17865  0.3  1.5  26104  7984 ?        Ss   07:20   0:00 /usr/sbin/httpd
asterisk 17867  3.9  3.2  37684 16524 ?        S    07:20   0:00 /usr/sbin/httpd
asterisk 17868  0.0  0.9  26104  4992 ?        S    07:20   0:00 /usr/sbin/httpd
asterisk 17869  1.1  1.9  31636 10216 ?        S    07:20   0:00 /usr/sbin/httpd
asterisk 17870  0.0  0.9  26104  4992 ?        S    07:20   0:00 /usr/sbin/httpd
asterisk 17871  0.0  0.9  26104  4992 ?        S    07:20   0:00 /usr/sbin/httpd
asterisk 17872  0.0  0.9  26104  4992 ?        S    07:20   0:00 /usr/sbin/httpd
asterisk 17873  0.0  0.9  26104  4992 ?        S    07:20   0:00 /usr/sbin/httpd
asterisk 17874  0.4  1.7  28916  9096 ?        R    07:20   0:00 /usr/sbin/httpd
root     17889  0.0  0.1   4396   948 pts/2    R+   07:20   0:00 ps -auxw
mysql    30421  0.2  0.6 123188  3472 ?        Sl   Feb11   4:45 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --pid-file=/var/r
root     31284  0.0  0.0   4624   144 ?        S    Feb12   0:00 /bin/sh /usr/sbin/safe_asterisk -U asterisk -G asterisk -g
asterisk 31287  0.9  0.3  43284  1912 ?        Sl   Feb12  10:46 /usr/sbin/asterisk -f -U asterisk -G asterisk -g -vvvg -c




}}}

SkykingOH · February 13, 2012, 1:30pm

Please, when posting this type of output in the forums use the code tags. I have edited your post.

You posted three pages of unreadable text that trashed the formatting on the page.

For lurkers, click “input format” at the bottom of the message input box, then “bbcode help”

VoIPTek · February 13, 2012, 1:41pm

Thanks skyking, was flying out the door but wanted the extra info before restarting apache on that server, I put {{{ and }}} which is from another board for entering code, pressed post and ran out the door.

FYI both of those entries are from one server and the entries I posted the other day are from a different server.

tonyclewis · February 13, 2012, 2:43pm

I wonder if it is a php 5.3 issue. I have not had time to dig into this yet as I am not able to see the same issues on any of our systems. Does anyone else have this issue and if so can we get your php version. Also anyone running FreePBX 2.10 not on php 5.3 are you having this issue?

VoIPTek · February 13, 2012, 2:55pm

I have one of my backup systems I can give you access to and if you trash it, thats fine since it’s a virtual box I can re provision it.
It’s up to you.
I can also downgrade php assuming it wouldn’t break anything, but I believe if your running 2.10 and you upgraded via scripts just about everyone will be on the same version of php.

tonyclewis · February 13, 2012, 5:55pm

Can you take a older distro and just update FreePBX from the GUI and not use the upgrade scripts. That way it stays on php 5.2 and report back if you still have the problems.

stonet · February 13, 2012, 6:15pm

I have seen this problem once on a 2.10 php 5.3 system, I rebooted and have not seen the problem again and try as I might I cant reproduce it. It seems to me that it is being caused by one particular process that is not always in use as the symptoms are not consistently present.

VoIPTek · February 13, 2012, 6:38pm

While I cant kill the production boxes, I can try it on a backup box.
If I use scripts up to the latest 2.9x will that keep php at 5.2x ?

Also stonet, if I reboot boxes ( which I have done several times ) no effect.

I can duplicate this problem on multiple boxes all day long.

tonyclewis · February 13, 2012, 7:36pm

Yes go up to anything that ends in .55 in the version. Than upgrade to FreePBX 2.10 in the GUI and let it run.