Fast AGI is enabled
Will prune the module list as suggested.
Thanks
XMPP can go as well.
Letās see what happens tomorrow.
Thankyou to all who are assisting.
1 Like
Some better news but weāre not there yet.
I disabled/uninstalled all the commercial modules that are not in use along with XMPP and anything else that wasnāt core function PBX related.
This dropped the overnight ābaselineā memory usage to a steady 25-30% as reported by sar -r
Down from 60-65% the previous night but still seems quite high. Thatās 8 GB for just standing around being a phone system !.
Hereās how the day looked.
Time period//memory used//calls answered in that hour
0000 to 0100 27% used 53 calls
0100 to 0200 27% used 15 calls
0200 to 0300 27-35% used 16 calls
0400 to 0800 it came up a little to 37% 230 calls
0800 to 0900 35-45% 386 calls
0900 to 1000 rising steadily to 99% 491 calls
1000 to 1020 steady on 99% 486 calls
1020 Our morning Asterisk oom crash and restart
1030 - 1100 <10% used
1100 -1200 <15% used 497 calls
1200 - 1300 <20% used 499 calls
1300 -1400 20% used ramping quite quickly to 55% 526 calls
1400 -1500 66-99% 561 calls
1500-1510 455 calls
1512 Our afternoon Asterisk oom crash and restart
1520 - 1600 <15% used
1600 - 1700 <20%used 515 calls
1700 - 1800 <20% used 270 calls
1800 - 1900 20-30% used 332 calls
1900 - 2000 30% ramping to 70% 312 calls
It looks like weāre going to have another Asterisk oom crash and restart in the next 30 minutes or so.
Maybe not, just checked sar again and we reached 75% mem in use now back down to 36%
Had a quick look in āmessagesā and the first crash was at 10:20
Hereās everything that relates. Not sure what is crash related and what is Asterisk restart related.
The afternoon crash at 15:12 appears similar. Can post more if required.
Mar 26 01:20:27 freepbx auditd[785]: Audit daemon rotating log files
Mar 26 10:20:43 freepbx kernel: asterisk invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Mar 26 10:20:43 freepbx kernel: asterisk cpuset=/ mems_allowed=0
Mar 26 10:20:43 freepbx kernel: CPU: 0 PID: 5451 Comm: asterisk Kdump: loaded Tainted: G IOE ------------ 3.10.0-1127.19.1.el7.x86_64 #1
Mar 26 10:20:43 freepbx kernel: Hardware name: HP ProLiant ML350 G6, BIOS D22 05/21/2018
Mar 26 10:20:43 freepbx kernel: Call Trace:
Mar 26 10:20:43 freepbx kernel: [] dump_stack+0x19/0x1b
Mar 26 10:20:43 freepbx kernel: [] dump_header+0x90/0x229
Mar 26 10:20:43 freepbx kernel: [] ? ktime_get_ts64+0x52/0xf0
Mar 26 10:20:43 freepbx kernel: [] oom_kill_process+0x25e/0x3f0
Mar 26 10:20:43 freepbx kernel: [] ? cpuset_mems_allowed_intersects+0x21/0x30
Mar 26 10:20:43 freepbx kernel: [] ? oom_unkillable_task+0xcd/0x120
Mar 26 10:20:43 freepbx kernel: [] ? find_lock_task_mm+0x56/0xc0
Mar 26 10:20:43 freepbx kernel: [] out_of_memory+0x4b6/0x4f0
Mar 26 10:20:43 freepbx kernel: [] __alloc_pages_slowpath+0x5db/0x729
Mar 26 10:20:43 freepbx kernel: [] __alloc_pages_nodemask+0x436/0x450
Mar 26 10:20:43 freepbx kernel: [] alloc_pages_current+0x98/0x110
Mar 26 10:20:43 freepbx kernel: [] __page_cache_alloc+0x97/0xb0
Mar 26 10:20:43 freepbx kernel: [] filemap_fault+0x270/0x420
Mar 26 10:20:43 freepbx kernel: [] __xfs_filemap_fault+0x7e/0x1d0 [xfs]
Mar 26 10:20:43 freepbx kernel: [] xfs_filemap_fault+0x2c/0x30 [xfs]
Mar 26 10:20:43 freepbx kernel: [] __do_fault.isra.61+0x8a/0x100
Mar 26 10:20:43 freepbx kernel: [] do_read_fault.isra.63+0x4c/0x1b0
Mar 26 10:20:43 freepbx kernel: [] handle_mm_fault+0xa20/0xfb0
Mar 26 10:20:43 freepbx kernel: [] ? rwsem_wake+0x8b/0xe0
Mar 26 10:20:43 freepbx kernel: [] __do_page_fault+0x213/0x500
Mar 26 10:20:43 freepbx kernel: [] do_page_fault+0x35/0x90
Mar 26 10:20:43 freepbx kernel: [] ? error_swapgs+0xaa/0xc0
Mar 26 10:20:43 freepbx kernel: [] page_fault+0x28/0x30
Mar 26 10:20:43 freepbx kernel: Mem-Info:
Mar 26 10:20:43 freepbx kernel: active_anon:5427881 inactive_anon:503411 isolated_anon:0#012 active_file:2675 inactive_file:2719 isolated_file:0#012 unevictable:0 dirty:0 writeback:0 unstable:0#012 slab_reclaimable:30296 slab_unreclaimable:21541#012 mapped:8423 shmem:8213 pagetables:38302 bounce:0#012 free:41192 free_pcp:327 free_cma:0
Mar 26 10:20:43 freepbx kernel: Node 0 DMA free:15820kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Mar 26 10:20:43 freepbx kernel: lowmem_reserve[]: 0 3441 23915 23915
Mar 26 10:20:43 freepbx kernel: Node 0 DMA32 free:91452kB min:9716kB low:12144kB high:14572kB active_anon:2799828kB inactive_anon:559808kB active_file:1784kB inactive_file:1204kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3774592kB managed:3527140kB mlocked:0kB dirty:0kB writeback:0kB mapped:5464kB shmem:5464kB slab_reclaimable:14848kB slab_unreclaimable:11816kB kernel_stack:2960kB pagetables:27956kB unstable:0kB bounce:0kB free_pcp:600kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:5149 all_unreclaimable? yes
Mar 26 10:20:43 freepbx kernel: lowmem_reserve[]: 0 0 20474 20474
Mar 26 10:20:43 freepbx kernel: Node 0 Normal free:57496kB min:57820kB low:72272kB high:86728kB active_anon:18911696kB inactive_anon:1453836kB active_file:8916kB inactive_file:9672kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:21364732kB managed:20965740kB mlocked:0kB dirty:0kB writeback:0kB mapped:28228kB shmem:27388kB slab_reclaimable:106336kB slab_unreclaimable:74332kB kernel_stack:11824kB pagetables:125252kB unstable:0kB bounce:0kB free_pcp:708kB local_pcp:120kB free_cma:0kB writeback_tmp:0kB pages_scanned:28767 all_unreclaimable? yes
Mar 26 10:20:43 freepbx kernel: lowmem_reserve[]: 0 0 0 0
Mar 26 10:20:43 freepbx kernel: Node 0 DMA: 14kB (U) 18kB (U) 016kB 032kB 164kB (U) 1128kB (U) 1256kB (U) 0512kB 11024kB (U) 12048kB (M) 34096kB (M) = 15820kB
Mar 26 10:20:43 freepbx kernel: Node 0 DMA32: 3274kB (UEM) 4398kB (UEM) 94316kB (UE) 44032kB (UEM) 38864kB (UEM) 126128kB (UEM) 28256kB (UEM) 13512kB (UEM) 31024kB (UM) 02048kB 04096kB = 91844kB
Mar 26 10:20:43 freepbx kernel: Node 0 Normal: 9714kB (UEM) 45428kB (UE) 31516kB (UEM) 7132kB (UEM) 11964kB (UEM) 18128kB (EM) 0256kB 0512kB 01024kB 02048kB 0*4096kB = 57452kB
Mar 26 10:20:43 freepbx kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Mar 26 10:20:43 freepbx kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Mar 26 10:20:43 freepbx kernel: 16758 total pagecache pages
Mar 26 10:20:43 freepbx kernel: 2689 pages in swap cache
Mar 26 10:20:43 freepbx kernel: Swap cache stats: add 14559759, delete 14557256, find 4298725/4711505
Mar 26 10:20:43 freepbx kernel: Free swap = 0kB
Mar 26 10:20:43 freepbx kernel: Total swap = 12386300kB
Mar 26 10:20:43 freepbx kernel: 6288829 pages RAM
Mar 26 10:20:43 freepbx kernel: 0 pages HighMem/MovableOnly
Mar 26 10:20:43 freepbx kernel: 161632 pages reserved
continues for quite some timeā¦well inexcess of the posting limitā¦
Just perusing some other posts looking for inspiration and there is an OP seeking help for a suspected memory leak.
He posted a screenshot from htop and Dicko commented that there were too many instances of Asterisk running.
Well, same here. Below is a screen shot from 2 minutes ago.
Top to bottom Asterisk processes. Different PIDās but all with the same memory allocation showing.
Some showing CPU usage, some not.
I just thought it was some display anomaly with htop as top just shows one Asterisk process.
What version of asterisk?
I would be inclined to try another version
Currently Freepbx 15/Asterisk 16.
Was Freepbx 13 /Asterisk 13 with similar issues
Are you using any asterisk add-ons at all? You nentioned AGI I believe.
Not that i am aware of.
AGI is just part of the normal feature set isnāt it ?
One thing i probably didnāt mention is that the site runs in Device & User mode.
1 Like
Iām seeing the same issue with Asterisk 13.32 which Iāve found to be quite stable.
Can you elaborate on which issue ?
This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.