Strange Kernel Errors (page fault and buffer overflow) on distro

Phades · July 30, 2012, 4:25am

I’ve NEVER seen this in my logs before…anyone have any ideas what it is? Is this a possible hack attempt/success by exploiting a page fault and/or buffer overflow bug?

PBX Firmware: 1.89.210.57-2

--------------------- Kernel Begin ------------------------


 WARNING:  Kernel Errors Present
    [2406386.516787]  [<c121297a>] ? error_code+0x66/0x6c ...:  1 Time(s)
    [2406386.573719]  [<c121297a>] ? error_code+0x66/0x6c ...:  1 Time(s)
    [2406386.576022]  [<c121297a>] ? error_code+0x66/0x6c ...:  1 Time(s)
    [2406445.658943]  [<c121297a>] ? error_code+0x66/0x6c ...:  1 Time(s)
    [2406445.721107]  [<c121297a>] ? error_code+0x66/0x6c ...:  1 Time(s)
    [2406445.724264]  [<c121297a>] ? error_code+0x66/0x6c ...:  1 Time(s)

 1 Time(s): [2406386.516584] local invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0
 1 Time(s): [2406386.516604] Pid: 16863, comm: local Not tainted 2.6.35.4-cloudPBX #1
 1 Time(s): [2406386.516612] Call Trace:
 1 Time(s): [2406386.516627]  [<c1060d84>] ? dump_header.clone.1+0x43/0x133
 1 Time(s): [2406386.516639]  [<c1212112>] ? _raw_spin_unlock_irqrestore+0xc/0xd
 1 Time(s): [2406386.516653]  [<c115e3bb>] ? ___ratelimit+0xbf/0xd0
 1 Time(s): [2406386.516663]  [<c1060e9e>] ? oom_kill_process.clone.0+0x2a/0xe1
 1 Time(s): [2406386.516674]  [<c1061055>] ? __out_of_memory+0x100/0x112
 1 Time(s): [2406386.516683]  [<c10610d7>] ? out_of_memory+0x70/0x94
 1 Time(s): [2406386.516693]  [<c1063513>] ? __alloc_pages_nodemask+0x3a5/0x46d
 1 Time(s): [2406386.516705]  [<c1064913>] ? __do_page_cache_readahead+0x96/0x15e
 1 Time(s): [2406386.516716]  [<c10649ef>] ? ra_submit+0x14/0x18
 1 Time(s): [2406386.516725]  [<c105fe14>] ? filemap_fault+0x170/0x2e4
 1 Time(s): [2406386.516736]  [<c106fc7c>] ? __do_fault+0x53/0x3cb
 1 Time(s): [2406386.516746]  [<c1070b45>] ? handle_mm_fault+0x48f/0x94c
 1 Time(s): [2406386.516757]  [<c102bb6d>] ? irq_exit+0x36/0x58
 1 Time(s): [2406386.516766]  [<c101a754>] ? do_page_fault+0x234/0x24c
 1 Time(s): [2406386.516776]  [<c101a520>] ? do_page_fault+0x0/0x24c
 1 Time(s): [2406386.516796]  [<c101a520>] ? do_page_fault+0x0/0x24c
 1 Time(s): [2406386.516803] Mem-Info:
 1 Time(s): [2406386.516808] DMA per-cpu:
 1 Time(s): [2406386.516814] CPU    0: hi:    0, btch:   1 usd:   0
 1 Time(s): [2406386.516822] CPU    1: hi:    0, btch:   1 usd:   0
 1 Time(s): [2406386.516828] Normal per-cpu:
 1 Time(s): [2406386.516834] CPU    0: hi:   90, btch:  15 usd:  83
 1 Time(s): [2406386.516842] CPU    1: hi:   90, btch:  15 usd:  72
 1 Time(s): [2406386.516852] active_anon:28922 inactive_anon:29017 isolated_anon:0
 1 Time(s): [2406386.516854]  active_file:2 inactive_file:35 isolated_file:0
 1 Time(s): [2406386.516855]  unevictable:1137 dirty:0 writeback:0 unstable:0
 1 Time(s): [2406386.516857]  free:739 slab_reclaimable:577 slab_unreclaimable:985
 1 Time(s): [2406386.516858]  mapped:908 shmem:34 pagetables:602 bounce:0
 1 Time(s): [2406386.516890] DMA free:1068kB min:124kB low:152kB high:184kB active_anon:6976kB inactive_anon:7148kB active_file:8kB inactive_file:100kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15868kB mlocked:0kB dirty:0kB writeback:0kB mapped:60kB shmem:0kB slab_reclaimable:24kB slab_unreclaimable:84kB kernel_stack:56kB pagetables:52kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:12 all_unreclaimable? yes
 1 Time(s): [2406386.516920] lowmem_reserve[]: 0 238 238 238
 1 Time(s): [2406386.516943] Normal free:1888kB min:1908kB low:2384kB high:2860kB active_anon:108712kB inactive_anon:108920kB active_file:0kB inactive_file:40kB unevictable:4548kB isolated(anon):0kB isolated(file):0kB present:243840kB mlocked:4548kB dirty:0kB writeback:0kB mapped:3572kB shmem:136kB slab_reclaimable:2284kB slab_unreclaimable:3856kB kernel_stack:1264kB pagetables:2356kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:270 all_unreclaimable? yes
 1 Time(s): [2406386.516976] lowmem_reserve[]: 0 0 0 0
 1 Time(s): [2406386.516994] DMA: 1*4kB 1*8kB 0*16kB 1*32kB 0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 1068kB
 1 Time(s): [2406386.517036] Normal: 286*4kB 1*8kB 0*16kB 1*32kB 1*64kB 1*128kB 0*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 1888kB
 1 Time(s): [2406386.517075] 958 total pagecache pages
 1 Time(s): [2406386.517081] 0 pages in swap cache
 1 Time(s): [2406386.517087] Swap cache stats: add 0, delete 0, find 0/0
 1 Time(s): [2406386.517095] Free swap  = 0kB
 1 Time(s): [2406386.517100] Total swap = 0kB
 1 Time(s): [2406386.517891] 65535 pages RAM
 1 Time(s): [2406386.517898] 0 pages HighMem
 1 Time(s): [2406386.517904] 2242 pages reserved
 1 Time(s): [2406386.517910] 15634 pages shared
 1 Time(s): [2406386.517916] 60630 pages non-shared
 1 Time(s): [2406386.517922] Out of memory: kill process 16958 (retrieve_conf) score 9036 or a child
 1 Time(s): [2406386.517932] Killed process 16958 (retrieve_conf) vsz:36144kB, anon-rss:17056kB, file-rss:1156kB
 1 Time(s): [2406386.573585] retrieve_conf: page allocation failure. order:0, mode:0x201da
 1 Time(s): [2406386.573602] Pid: 16958, comm: retrieve_conf Not tainted 2.6.35.4-cloudPBX #1
 1 Time(s): [2406386.573611] Call Trace:
 1 Time(s): [2406386.573626]  [<c1063595>] ? __alloc_pages_nodemask+0x427/0x46d
 1 Time(s): [2406386.573638]  [<c1064913>] ? __do_page_cache_readahead+0x96/0x15e
 1 Time(s): [2406386.573648]  [<c10649ef>] ? ra_submit+0x14/0x18
 1 Time(s): [2406386.573658]  [<c1064ce5>] ? page_cache_sync_readahead+0x34/0x3a
 1 Time(s): [2406386.573668]  [<c105fdc5>] ? filemap_fault+0x121/0x2e4
 1 Time(s): [2406386.573680]  [<c106fc7c>] ? __do_fault+0x53/0x3cb
 1 Time(s): [2406386.573689]  [<c1070b45>] ? handle_mm_fault+0x48f/0x94c
 1 Time(s): [2406386.573700]  [<c101a754>] ? do_page_fault+0x234/0x24c
 1 Time(s): [2406386.573708]  [<c101a520>] ? do_page_fault+0x0/0x24c
 1 Time(s): [2406386.573728]  [<c101a520>] ? do_page_fault+0x0/0x24c
 1 Time(s): [2406386.573735] Mem-Info:
 1 Time(s): [2406386.573740] DMA per-cpu:
 1 Time(s): [2406386.573746] CPU    0: hi:    0, btch:   1 usd:   0
 1 Time(s): [2406386.573753] CPU    1: hi:    0, btch:   1 usd:   0
 1 Time(s): [2406386.573759] Normal per-cpu:
 1 Time(s): [2406386.573770] CPU    0: hi:   90, btch:  15 usd:  83
 1 Time(s): [2406386.573777] CPU    1: hi:   90, btch:  15 usd:  72
 1 Time(s): [2406386.573788] active_anon:28922 inactive_anon:29017 isolated_anon:0
 1 Time(s): [2406386.573790]  active_file:2 inactive_file:35 isolated_file:0
 1 Time(s): [2406386.573791]  unevictable:1137 dirty:0 writeback:0 unstable:0
 1 Time(s): [2406386.573792]  free:739 slab_reclaimable:577 slab_unreclaimable:985
 1 Time(s): [2406386.573794]  mapped:908 shmem:34 pagetables:602 bounce:0
 1 Time(s): [2406386.573823] DMA free:1068kB min:124kB low:152kB high:184kB active_anon:6976kB inactive_anon:7148kB active_file:8kB inactive_file:100kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15868kB mlocked:0kB dirty:0kB writeback:0kB mapped:60kB shmem:0kB slab_reclaimable:24kB slab_unreclaimable:84kB kernel_stack:56kB pagetables:52kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:12 all_unreclaimable? yes
 1 Time(s): [2406386.573853] lowmem_reserve[]: 0 238 238 238
 1 Time(s): [2406386.573877] Normal free:1888kB min:1908kB low:2384kB high:2860kB active_anon:108712kB inactive_anon:108920kB active_file:0kB inactive_file:40kB unevictable:4548kB isolated(anon):0kB isolated(file):0kB present:243840kB mlocked:4548kB dirty:0kB writeback:0kB mapped:3572kB shmem:136kB slab_reclaimable:2284kB slab_unreclaimable:3856kB kernel_stack:1264kB pagetables:2356kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:270 all_unreclaimable? yes
 1 Time(s): [2406386.573908] lowmem_reserve[]: 0 0 0 0
 1 Time(s): [2406386.573926] DMA: 1*4kB 1*8kB 0*16kB 1*32kB 0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 1068kB
 1 Time(s): [2406386.573964] Normal: 286*4kB 1*8kB 0*16kB 1*32kB 1*64kB 1*128kB 0*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 1888kB
 1 Time(s): [2406386.574006] 958 total pagecache pages
 1 Time(s): [2406386.574015] 0 pages in swap cache
 1 Time(s): [2406386.574021] Swap cache stats: add 0, delete 0, find 0/0
 1 Time(s): [2406386.574028] Free swap  = 0kB
 1 Time(s): [2406386.574034] Total swap = 0kB
 1 Time(s): [2406386.574813] 65535 pages RAM
 1 Time(s): [2406386.574821] 0 pages HighMem
 1 Time(s): [2406386.574827] 2242 pages reserved
 1 Time(s): [2406386.574832] 15634 pages shared
 1 Time(s): [2406386.574838] 60630 pages non-shared
 1 Time(s): [2406386.575816] local invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0
 1 Time(s): [2406386.575829] Pid: 16863, comm: local Not tainted 2.6.35.4-cloudPBX #1
 1 Time(s): [2406386.575837] Call Trace:
 1 Time(s): [2406386.575854]  [<c1060d84>] ? dump_header.clone.1+0x43/0x133
 1 Time(s): [2406386.575867]  [<c1212112>] ? _raw_spin_unlock_irqrestore+0xc/0xd
 1 Time(s): [2406386.575887]  [<c115e3bb>] ? ___ratelimit+0xbf/0xd0
 1 Time(s): [2406386.575899]  [<c1060e9e>] ? oom_kill_process.clone.0+0x2a/0xe1
 1 Time(s): [2406386.575909]  [<c1061055>] ? __out_of_memory+0x100/0x112
 1 Time(s): [2406386.575920]  [<c10610d7>] ? out_of_memory+0x70/0x94
 1 Time(s): [2406386.575933]  [<c1063513>] ? __alloc_pages_nodemask+0x3a5/0x46d
 1 Time(s): [2406386.575946]  [<c1064913>] ? __do_page_cache_readahead+0x96/0x15e
 1 Time(s): [2406386.575958]  [<c10649ef>] ? ra_submit+0x14/0x18
 1 Time(s): [2406386.575967]  [<c105fe14>] ? filemap_fault+0x170/0x2e4
 1 Time(s): [2406386.575977]  [<c106fc7c>] ? __do_fault+0x53/0x3cb
 1 Time(s): [2406386.575986]  [<c1070b45>] ? handle_mm_fault+0x48f/0x94c
 1 Time(s): [2406386.575996]  [<c102bb6d>] ? irq_exit+0x36/0x58
 1 Time(s): [2406386.576005]  [<c101a754>] ? do_page_fault+0x234/0x24c
 1 Time(s): [2406386.576013]  [<c101a520>] ? do_page_fault+0x0/0x24c
 1 Time(s): [2406386.576031]  [<c101a520>] ? do_page_fault+0x0/0x24c
 1 Time(s): [2406386.576037] Mem-Info:
 1 Time(s): [2406386.576042] DMA per-cpu:
 1 Time(s): [2406386.576048] CPU    0: hi:    0, btch:   1 usd:   0
 1 Time(s): [2406386.576055] CPU    1: hi:    0, btch:   1 usd:   0
 1 Time(s): [2406386.576061] Normal per-cpu:
 1 Time(s): [2406386.576067] CPU    0: hi:   90, btch:  15 usd:  83
 1 Time(s): [2406386.576074] CPU    1: hi:   90, btch:  15 usd:  72
 1 Time(s): [2406386.576083] active_anon:28922 inactive_anon:29017 isolated_anon:0
 1 Time(s): [2406386.576085]  active_file:2 inactive_file:74 isolated_file:0
 1 Time(s): [2406386.576086]  unevictable:1137 dirty:0 writeback:0 unstable:0
 1 Time(s): [2406386.576088]  free:739 slab_reclaimable:577 slab_unreclaimable:985
 1 Time(s): [2406386.576089]  mapped:908 shmem:34 pagetables:602 bounce:0
 1 Time(s): [2406386.576117] DMA free:1068kB min:124kB low:152kB high:184kB active_anon:6976kB inactive_anon:7148kB active_file:8kB inactive_file:100kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15868kB mlocked:0kB dirty:0kB writeback:0kB mapped:60kB shmem:0kB slab_reclaimable:24kB slab_unreclaimable:84kB kernel_stack:56kB pagetables:52kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:12 all_unreclaimable? yes
 1 Time(s): [2406386.576146] lowmem_reserve[]: 0 238 238 238
 1 Time(s): [2406386.576166] Normal free:1948kB min:1908kB low:2384kB high:2860kB active_anon:108660kB inactive_anon:108920kB active_file:0kB inactive_file:196kB unevictable:4548kB isolated(anon):0kB isolated(file):0kB present:243840kB mlocked:4548kB dirty:0kB writeback:0kB mapped:3572kB shmem:136kB slab_reclaimable:2284kB slab_unreclaimable:3856kB kernel_stack:1264kB pagetables:2356kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
 1 Time(s): [2406386.576197] lowmem_reserve[]: 0 0 0 0
 1 Time(s): [2406386.576213] DMA: 2*4kB 1*8kB 0*16kB 1*32kB 0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 1072kB
 1 Time(s): [2406386.576246] Normal: 449*4kB 2*8kB 0*16kB 1*32kB 1*64kB 1*128kB 0*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 2548kB
 1 Time(s): [2406386.576279] 997 total pagecache pages
 1 Time(s): [2406386.576285] 0 pages in swap cache
 1 Time(s): [2406386.576291] Swap cache stats: add 0, delete 0, find 0/0
 1 Time(s): [2406386.576297] Free swap  = 0kB
 1 Time(s): [2406386.576302] Total swap = 0kB
 1 Time(s): [2406386.577076] 65535 pages RAM
 1 Time(s): [2406386.577084] 0 pages HighMem
 1 Time(s): [2406386.577089] 2242 pages reserved
 1 Time(s): [2406386.577095] 15633 pages shared
 1 Time(s): [2406386.577100] 59932 pages non-shared
 1 Time(s): [2406386.577107] Out of memory: kill process 16952 (module_admin) score 7002 or a child
 1 Time(s): [2406386.577117] Killed process 16957 (sh) vsz:2548kB, anon-rss:116kB, file-rss:416kB
 1 Time(s): [2406445.658752] fail2ban-server invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0
 1 Time(s): [2406445.658772] Pid: 7739, comm: fail2ban-server Not tainted 2.6.35.4-cloudPBX #1
 1 Time(s): [2406445.658782] Call Trace:
 1 Time(s): [2406445.658799]  [<c1060d84>] ? dump_header.clone.1+0x43/0x133
 1 Time(s): [2406445.658812]  [<c1212112>] ? _raw_spin_unlock_irqrestore+0xc/0xd
 1 Time(s): [2406445.658825]  [<c115e3bb>] ? ___ratelimit+0xbf/0xd0
 1 Time(s): [2406445.658834]  [<c1060e9e>] ? oom_kill_process.clone.0+0x2a/0xe1
 1 Time(s): [2406445.658843]  [<c1061055>] ? __out_of_memory+0x100/0x112
 1 Time(s): [2406445.658852]  [<c10610d7>] ? out_of_memory+0x70/0x94
 1 Time(s): [2406445.658862]  [<c1063513>] ? __alloc_pages_nodemask+0x3a5/0x46d
 1 Time(s): [2406445.658873]  [<c1064913>] ? __do_page_cache_readahead+0x96/0x15e
 1 Time(s): [2406445.658883]  [<c10649ef>] ? ra_submit+0x14/0x18
 1 Time(s): [2406445.658891]  [<c105fe14>] ? filemap_fault+0x170/0x2e4
 1 Time(s): [2406445.658903]  [<c106fc7c>] ? __do_fault+0x53/0x3cb
 1 Time(s): [2406445.658912]  [<c1070b45>] ? handle_mm_fault+0x48f/0x94c
 1 Time(s): [2406445.658923]  [<c101a754>] ? do_page_fault+0x234/0x24c
 1 Time(s): [2406445.658931]  [<c101a520>] ? do_page_fault+0x0/0x24c
 1 Time(s): [2406445.658951]  [<c101a520>] ? do_page_fault+0x0/0x24c
 1 Time(s): [2406445.658959] Mem-Info:
 1 Time(s): [2406445.658964] DMA per-cpu:
 1 Time(s): [2406445.658969] CPU    0: hi:    0, btch:   1 usd:   0
 1 Time(s): [2406445.658977] CPU    1: hi:    0, btch:   1 usd:   0
 1 Time(s): [2406445.658983] Normal per-cpu:
 1 Time(s): [2406445.658989] CPU    0: hi:   90, btch:  15 usd:  83
 1 Time(s): [2406445.658996] CPU    1: hi:   90, btch:  15 usd:  81
 1 Time(s): [2406445.659011] active_anon:28972 inactive_anon:29049 isolated_anon:0
 1 Time(s): [2406445.659012]  active_file:0 inactive_file:72 isolated_file:0
 1 Time(s): [2406445.659014]  unevictable:1137 dirty:0 writeback:0 unstable:0
 1 Time(s): [2406445.659015]  free:739 slab_reclaimable:557 slab_unreclaimable:974
 1 Time(s): [2406445.659016]  mapped:899 shmem:34 pagetables:583 bounce:0
 1 Time(s): [2406445.659045] DMA free:1060kB min:124kB low:152kB high:184kB active_anon:6888kB inactive_anon:7148kB active_file:4kB inactive_file:220kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15868kB mlocked:0kB dirty:0kB writeback:0kB mapped:12kB shmem:0kB slab_reclaimable:20kB slab_unreclaimable:68kB kernel_stack:72kB pagetables:36kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
 1 Time(s): [2406445.659074] lowmem_reserve[]: 0 238 238 238
 1 Time(s): [2406445.659095] Normal free:1896kB min:1908kB low:2384kB high:2860kB active_anon:109000kB inactive_anon:109048kB active_file:0kB inactive_file:68kB unevictable:4548kB isolated(anon):0kB isolated(file):0kB present:243840kB mlocked:4548kB dirty:0kB writeback:0kB mapped:3584kB shmem:136kB slab_reclaimable:2208kB slab_unreclaimable:3828kB kernel_stack:1224kB pagetables:2296kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:143 all_unreclaimable? yes
 1 Time(s): [2406445.659126] lowmem_reserve[]: 0 0 0 0
 1 Time(s): [2406445.659142] DMA: 1*4kB 0*8kB 0*16kB 1*32kB 0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 1060kB
 1 Time(s): [2406445.659175] Normal: 284*4kB 3*8kB 0*16kB 1*32kB 1*64kB 1*128kB 0*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 1896kB
 1 Time(s): [2406445.659208] 988 total pagecache pages
 1 Time(s): [2406445.659214] 0 pages in swap cache
 1 Time(s): [2406445.659220] Swap cache stats: add 0, delete 0, find 0/0
 1 Time(s): [2406445.659227] Free swap  = 0kB
 1 Time(s): [2406445.659232] Total swap = 0kB
 1 Time(s): [2406445.660080] 65535 pages RAM
 1 Time(s): [2406445.660088] 0 pages HighMem
 1 Time(s): [2406445.660094] 2242 pages reserved
 1 Time(s): [2406445.660099] 15008 pages shared
 1 Time(s): [2406445.660105] 60633 pages non-shared
 1 Time(s): [2406445.660112] Out of memory: kill process 16985 (retrieve_conf) score 9358 or a child
 1 Time(s): [2406445.660122] Killed process 16985 (retrieve_conf) vsz:37432kB, anon-rss:18576kB, file-rss:1060kB
 1 Time(s): [2406445.720910] retrieve_conf: page allocation failure. order:0, mode:0x201da
 1 Time(s): [2406445.720927] Pid: 16985, comm: retrieve_conf Not tainted 2.6.35.4-cloudPBX #1
 1 Time(s): [2406445.720937] Call Trace:
 1 Time(s): [2406445.720952]  [<c1063595>] ? __alloc_pages_nodemask+0x427/0x46d
 1 Time(s): [2406445.720963]  [<c1064913>] ? __do_page_cache_readahead+0x96/0x15e
 1 Time(s): [2406445.720974]  [<c10649ef>] ? ra_submit+0x14/0x18
 1 Time(s): [2406445.720983]  [<c1064ce5>] ? page_cache_sync_readahead+0x34/0x3a
 1 Time(s): [2406445.720994]  [<c105fdc5>] ? filemap_fault+0x121/0x2e4
 1 Time(s): [2406445.721060]  [<c106fc7c>] ? __do_fault+0x53/0x3cb
 1 Time(s): [2406445.721075]  [<c1070b45>] ? handle_mm_fault+0x48f/0x94c
 1 Time(s): [2406445.721086]  [<c101a754>] ? do_page_fault+0x234/0x24c
 1 Time(s): [2406445.721095]  [<c101a520>] ? do_page_fault+0x0/0x24c
 1 Time(s): [2406445.721116]  [<c101a520>] ? do_page_fault+0x0/0x24c
 1 Time(s): [2406445.721124] Mem-Info:
 1 Time(s): [2406445.721129] DMA per-cpu:
 1 Time(s): [2406445.721135] CPU    0: hi:    0, btch:   1 usd:   0
 1 Time(s): [2406445.721143] CPU    1: hi:    0, btch:   1 usd:   0
 1 Time(s): [2406445.721150] Normal per-cpu:
 1 Time(s): [2406445.721156] CPU    0: hi:   90, btch:  15 usd:  83
 1 Time(s): [2406445.721163] CPU    1: hi:   90, btch:  15 usd:  81
 1 Time(s): [2406445.721173] active_anon:28972 inactive_anon:29049 isolated_anon:0
 1 Time(s): [2406445.721175]  active_file:0 inactive_file:72 isolated_file:0
 1 Time(s): [2406445.721176]  unevictable:1137 dirty:0 writeback:0 unstable:0
 1 Time(s): [2406445.721178]  free:739 slab_reclaimable:557 slab_unreclaimable:974
 1 Time(s): [2406445.721179]  mapped:899 shmem:34 pagetables:583 bounce:0
 1 Time(s): [2406445.721208] DMA free:1060kB min:124kB low:152kB high:184kB active_anon:6888kB inactive_anon:7148kB active_file:4kB inactive_file:220kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15868kB mlocked:0kB dirty:0kB writeback:0kB mapped:12kB shmem:0kB slab_reclaimable:20kB slab_unreclaimable:68kB kernel_stack:72kB pagetables:36kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
 1 Time(s): [2406445.721271] lowmem_reserve[]: 0 238 238 238
 1 Time(s): [2406445.721295] Normal free:1896kB min:1908kB low:2384kB high:2860kB active_anon:109000kB inactive_anon:109048kB active_file:0kB inactive_file:68kB unevictable:4548kB isolated(anon):0kB isolated(file):0kB present:243840kB mlocked:4548kB dirty:0kB writeback:0kB mapped:3584kB shmem:136kB slab_reclaimable:2208kB slab_unreclaimable:3828kB kernel_stack:1224kB pagetables:2296kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:143 all_unreclaimable? yes
 1 Time(s): [2406445.721326] lowmem_reserve[]: 0 0 0 0
 1 Time(s): [2406445.721344] DMA: 1*4kB 0*8kB 0*16kB 1*32kB 0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 1060kB
 1 Time(s): [2406445.721382] Normal: 284*4kB 3*8kB 0*16kB 1*32kB 1*64kB 1*128kB 0*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 1896kB
 1 Time(s): [2406445.721424] 988 total pagecache pages
 1 Time(s): [2406445.721430] 0 pages in swap cache
 1 Time(s): [2406445.721436] Swap cache stats: add 0, delete 0, find 0/0
 1 Time(s): [2406445.721443] Free swap  = 0kB
 1 Time(s): [2406445.721449] Total swap = 0kB
 1 Time(s): [2406445.722354] 65535 pages RAM
 1 Time(s): [2406445.722366] 0 pages HighMem
 1 Time(s): [2406445.722372] 2242 pages reserved
 1 Time(s): [2406445.722560] 15008 pages shared
 1 Time(s): [2406445.722568] 60633 pages non-shared
 1 Time(s): [2406445.723979] fail2ban-server invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0
 1 Time(s): [2406445.724105] Pid: 7739, comm: fail2ban-server Not tainted 2.6.35.4-cloudPBX #1
 1 Time(s): [2406445.724114] Call Trace:
 1 Time(s): [2406445.724125]  [<c1060d84>] ? dump_header.clone.1+0x43/0x133
 1 Time(s): [2406445.724136]  [<c1212112>] ? _raw_spin_unlock_irqrestore+0xc/0xd
 1 Time(s): [2406445.724148]  [<c115e3bb>] ? ___ratelimit+0xbf/0xd0
 1 Time(s): [2406445.724157]  [<c1060e9e>] ? oom_kill_process.clone.0+0x2a/0xe1
 1 Time(s): [2406445.724167]  [<c1061055>] ? __out_of_memory+0x100/0x112
 1 Time(s): [2406445.724176]  [<c10610d7>] ? out_of_memory+0x70/0x94
 1 Time(s): [2406445.724186]  [<c1063513>] ? __alloc_pages_nodemask+0x3a5/0x46d
 1 Time(s): [2406445.724197]  [<c1064913>] ? __do_page_cache_readahead+0x96/0x15e
 1 Time(s): [2406445.724207]  [<c10649ef>] ? ra_submit+0x14/0x18
 1 Time(s): [2406445.724216]  [<c105fe14>] ? filemap_fault+0x170/0x2e4
 1 Time(s): [2406445.724226]  [<c106fc7c>] ? __do_fault+0x53/0x3cb
 1 Time(s): [2406445.724236]  [<c1070b45>] ? handle_mm_fault+0x48f/0x94c
 1 Time(s): [2406445.724246]  [<c101a754>] ? do_page_fault+0x234/0x24c
 1 Time(s): [2406445.724254]  [<c101a520>] ? do_page_fault+0x0/0x24c
 1 Time(s): [2406445.724273]  [<c101a520>] ? do_page_fault+0x0/0x24c
 1 Time(s): [2406445.724280] Mem-Info:
 1 Time(s): [2406445.724285] DMA per-cpu:
 1 Time(s): [2406445.724291] CPU    0: hi:    0, btch:   1 usd:   0
 1 Time(s): [2406445.724298] CPU    1: hi:    0, btch:   1 usd:   0
 1 Time(s): [2406445.724305] Normal per-cpu:
 1 Time(s): [2406445.724311] CPU    0: hi:   90, btch:  15 usd:  84
 1 Time(s): [2406445.724318] CPU    1: hi:   90, btch:  15 usd:  80
 1 Time(s): [2406445.724328] active_anon:28107 inactive_anon:29049 isolated_anon:0
 1 Time(s): [2406445.724347]  active_file:0 inactive_file:124 isolated_file:0
 1 Time(s): [2406445.724350]  unevictable:1137 dirty:0 writeback:0 unstable:0
 1 Time(s): [2406445.724352]  free:1599 slab_reclaimable:557 slab_unreclaimable:974
 1 Time(s): [2406445.724353]  mapped:899 shmem:34 pagetables:583 bounce:0
 1 Time(s): [2406445.724389] DMA free:1080kB min:124kB low:152kB high:184kB active_anon:6860kB inactive_anon:7148kB active_file:4kB inactive_file:220kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15868kB mlocked:0kB dirty:0kB writeback:0kB mapped:12kB shmem:0kB slab_reclaimable:20kB slab_unreclaimable:68kB kernel_stack:72kB pagetables:36kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
 1 Time(s): [2406445.724420] lowmem_reserve[]: 0 238 238 238
 1 Time(s): [2406445.724443] Normal free:5976kB min:1908kB low:2384kB high:2860kB active_anon:104944kB inactive_anon:109048kB active_file:0kB inactive_file:328kB unevictable:4548kB isolated(anon):0kB isolated(file):0kB present:243840kB mlocked:4548kB dirty:0kB writeback:0kB mapped:3584kB shmem:136kB slab_reclaimable:2208kB slab_unreclaimable:3828kB kernel_stack:1224kB pagetables:2296kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
 1 Time(s): [2406445.724534] lowmem_reserve[]: 0 0 0 0
 1 Time(s): [2406445.724551] DMA: 10*4kB 0*8kB 0*16kB 1*32kB 0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 1096kB
 1 Time(s): [2406445.724591] Normal: 1294*4kB 8*8kB 0*16kB 1*32kB 1*64kB 1*128kB 0*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 5976kB
 1 Time(s): [2406445.724630] 1053 total pagecache pages
 1 Time(s): [2406445.724637] 0 pages in swap cache
 1 Time(s): [2406445.724643] Swap cache stats: add 0, delete 0, find 0/0
 1 Time(s): [2406445.724650] Free swap  = 0kB
 1 Time(s): [2406445.724655] Total swap = 0kB
 1 Time(s): [2406445.725648] 65535 pages RAM
 1 Time(s): [2406445.725659] 0 pages HighMem
 1 Time(s): [2406445.725665] 2242 pages reserved
 1 Time(s): [2406445.725671] 15015 pages shared
 1 Time(s): [2406445.725677] 58899 pages non-shared
 1 Time(s): [2406445.725684] Out of memory: kill process 16979 (module_admin) score 7002 or a child
 1 Time(s): [2406445.725695] Killed process 16984 (sh) vsz:2548kB, anon-rss:116kB, file-rss:444kB

 ---------------------- Kernel End -------------------------

--------------------- XNTPD Begin ------------------------


 **Unmatched Entries**
   sendto(205.196.146.72) (fd=18): Operation not permitted: 84 time(s)
   sendto(208.87.221.228) (fd=18): Operation not permitted: 84 time(s)

 ---------------------- XNTPD End -------------------------

SkykingOH · July 30, 2012, 5:21am

This is a FreePBX distro downloaded from this web site?

The CloudPBX thing is throwing me.

Please clarify.

What kind of hardware is this running on? Any DAHDI cards?

Phades · July 30, 2012, 5:28am

I’ve never had that error before. I’ve been using the same virtual server for many months now with no changes besides the module updates:

http://www.vr.org/os/voip-hosting/freepbx-vps

SkykingOH · July 30, 2012, 5:59am

Ok, all I can tell you is that Asterisk was compiled/built custom by your provider. That is not the Asterisk that is distributed with the FreePBX Distro.

Hard to offer much advice without knowing why they customize it (my guess is for the virtual environment).

Phades · July 30, 2012, 6:02am

From my understanding the only change is to support the virtual enviro. Tho, I’ve never seen that before in the log and my server has been hit hard lately from unauthorized IP addresses. I want to make sure this isn’t a hack attempt.

SkykingOH · July 30, 2012, 6:27am

2.6.35.4-cloudPBX is your Kernel version.

kernel.i686 2.6.18-274.3.1.el5 installed

This is off a FreePBX distro. Your provider is using a newer Kernel.

It doesn’t look like hacking. You can eliminate most of the hacking by restricting your IP’s to only US addresses. If you don’t have a need to access the system outside the US.

If you have port 80 open you should also close it if only using for remote admin. If you have user accessing the ARI then this does not apply to you.

With regard to the error, what does top show for memory usage?