Memory consumption grows to 100%

General Help
1

Hello,
Two months ago I set up a new PBX system running as a Hyper-V guest. It has 4096 MB RAM allocated (dynamic memory is not enabled) and the traffic is usually quite low.
There is something wrong with the memory consumption, because one week after restart it shows nearly 100% usage and Asterisk becomes unstable.

Used memory: 96.45% Free memory: 3.55% Used buffers and cache: 95.48% Free buffers and cache: 4.52% <= available memory Used swap space: 18.66% Free swap space: 81.34%

Asterisk RAM consumption: 3.3 GiB

Asterisk load:
13 active calls
26 active channels

Versions:
SHMZ release 6.6 (Final)
FreePBXDistro10.13.66-17
Asterisk 13.12.2

I don’t want to restart the system every week to fix this…

2

Check which processes are consuming your memory. Try running this:
ps aux --sort -rss | less

What are the top memory consuming processes (they’ll be at the very bottom)?

3

Buffers and Cached don’t count towards used memory.

type in
free -m

[root@localhost ~]# free -m
             total       used       free     shared    buffers     cached
Mem:           995        764        230          0        198        247
-/+ buffers/cache:        319        676
Swap:          767         68        699

See how used appears large at 76%, but its not. It’s only 764 - 198 - 247 = 319 MB used or 32%.

[root@localhost ~]# echo 3 > /proc/sys/vm/drop_caches
[root@localhost ~]# free -m
             total       used       free     shared    buffers     cached
Mem:           995        318        676          0          0         27
-/+ buffers/cache:        290        705
Swap:          767         68        699
4

dmesg show me that asterisk was killed, when memory usage reached 100%

[ pid ]   uid  tgid total_vm      rss cpu oom_adj oom_score_adj name
[  398]     0   398     2715        0   0     -17         -1000 udevd
[  618]     0   618     2661        0   0     -17         -1000 udevd
[  906]     0   906    23289       34   1     -17         -1000 auditd
[  926]     0   926    62466       69   0       0             0 rsyslogd
[  990]    81   990     5358        1   0       0             0 dbus-daemon
[ 1002]    70  1002     6916        1   1       0             0 avahi-daemon
[ 1003]    70  1003     6916        0   0       0             0 avahi-daemon
[ 1034]     0  1034     1020        0   1       0             0 acpid
[ 1044]    68  1044     9476       94   0       0             0 hald
[ 1045]     0  1045     5099        1   0       0             0 hald-runner
[ 1077]     0  1077     5629        1   1       0             0 hald-addon-inpu
[ 1092]    68  1092     4501        1   0       0             0 hald-addon-acpi
[ 1104]    99  1104     3221       24   1       0             0 dnsmasq
[ 1119]     0  1119    16554        0   1     -17         -1000 sshd
[ 1128]     0  1128     5429        1   1       0             0 xinetd
[ 1137]    38  1137     7683        1   1       0             0 ntpd
[ 1160]     0  1160     2670        1   1       0             0 in.tftpd
[ 1188]     0  1188    27042        1   0       0             0 mysqld_safe
[ 1299]    27  1299   258938     1015   0       0             0 mysqld
[ 1393]     0  1393    20215        1   0       0             0 master
[ 1403]    89  1403    20278        1   1       0             0 qmgr
[ 1409]     0  1409    87904       20   0       0             0 httpd
[ 1419]     0  1419    29223       97   1       0             0 crond
[ 1440]   498  1440    22359      273   1       0             0 lua
[ 1454]     0  1454     3485       57   1       0             0 gam_server
[ 1480]     0  1480     3556        1   1       0             0 incrond
[ 1713]     0  1713    27041        0   1       0             0 safe_asterisk
[ 1715]     0  1715     2714        0   0     -17         -1000 udevd
[ 1716]   499  1716  1449370   832416   0       0             0 asterisk
[ 1821]     0  1821    27041        0   1       0             0 start-xmpp.sh
[ 1822]     0  1822    36250        1   1       0             0 su
[ 1823]   499  1823    85846     2705   0       0             0 presence.php
[ 1831]     0  1831     1016        1   1       0             0 mingetty
[ 1835]     0  1835     1016        1   1       0             0 mingetty
[ 1837]     0  1837     1016        1   1       0             0 mingetty
[ 1839]     0  1839     1016        1   1       0             0 mingetty
[ 1841]     0  1841     1016        1   0       0             0 mingetty
[ 1843]     0  1843     1016        1   0       0             0 mingetty
[ 2006]     0  2006   313073      670   0       0             0 fail2ban-server
[ 5551]   499  5551   116043        1   1       0             0 httpd
[ 5636]   499  5636   116043        1   0       0             0 httpd
[ 5640]   499  5640   116043        7   0       0             0 httpd
[ 6216]   499  6216   116043        1   1       0             0 httpd
[ 6624]   499  6624   116043        1   0       0             0 httpd
[54182]    89 54182    20235        1   0       0             0 pickup
[55885]   499 55885    34945      155   1       0             0 crond
[55888]   499 55888    26515       45   1       0             0 sh
[55892]   499 55892    83781     4398   0       0             0 php

Out of memory: Kill process 1716 (asterisk) score 840 or sacrifice child
Killed process 1716, UID 499, (asterisk) total-vm:5797480kB, anon-rss:3329432kB, file-rss:232kB

image.png526Ă—639 83.4 KB

5

can anyone help?

6

Have you tried upgrading everything?

7

Any updates could fix your issue.
If there’s no result, you could add a script in the crontab.
sync; echo 3 > /proc/sys/vm/drop_caches

Usually scheduled at 00h00 everyday.

However, the cause could come from a memory leak from any app.

If you’ve got 100% every 2 or 8 hours, the problem is serious.

A good link

But be careful. This is only a sticking.

1 Like
8

yes, this is the latest FreePBX version available, I have also installed Linux Integration Services Version 4.0 for Hyper-V but with no change to my problem

9

I decided to restart asterisk process every 3 a.m. with below cron command:

crontab -e -u asterisk
0 3 * * * /var/lib/asterisk/bin/fwconsole restart

I will also try your workaround, but how to find out a memory leak cause in the asterisk process? Is FreePBX or pure Asterisk not compatible with Hyper-V?

10

Did you tried to launch manually this cmd line :
sync; echo 3 > /proc/sys/vm/drop_caches

1 Like
11

This only drops the file cache, so doing this doesn’t actually help with applications that consume all the memory. The linux kernel will use filesystem cache then swap before going OOM.

Consider the following example:

[root@localhost log]# free -m
         total       used       free     shared    buffers     cached
Mem:          1876       1360        516          0        **154        435**
-/+ buffers/cache:        770       **1106**
Swap:          767         33        734


[root@localhost log]# free -m
             total       used       free     shared    buffers     cached
Mem:          1876        728       1148          0          **0         41**
-/+ buffers/cache:        685       **1191**
Swap:          767         33        734
12

I have very similar issues with Freepbx 13 & 14. I have tried them on physical boxes and virtual. No matter what install I do they only constant is that memory fills up. My Asterisk process doesn’t quit but the Swap files start getting very high and I then start getting audio issues.
I have tried with different amounts of ram ranging from 1gb-4gb.

Not a dev but from what I have seen of the past year + is that there is a mem leak somewhere. I use crontab and reboot my voice server once a week at 2am to deal with the issue.

13

Hi,

I was working on it for more than 6 months, this is not a memory cache issue and you CANNOT fix it by “sync; echo 3 > /proc/sys/vm/drop_caches”. It’s only the kernel issue that fixed in kernel 4.x lt. So you can use this instruction for upgrade your kernel: https://www.tecmint.com/install-upgrade-kernel-version-in-centos-7/

I hope it will be helpful everyone has this issue.