Unstable Freepbx - Continious Memory and other errors on cli

I have Freepbx 15 installed few days back. I see lots of errros on cli. After some times my extensions unregistered. Then i restart the server/Asterisk and then everything comes back.

What can be the cause? I am attaching picture of screen.

The postdrop errors are because the postdrop daemon is killed off.

The root of the situation seems pretty clear… you are out of memory.

Look through the process list and see what’s consuming it all.

How much memory does the machine have?

free -t

Machine has 2 GB RAM. Hereis some fresh errors

Please tell us that you failed to log in 3 times?
image

Otherwise I would be more worried about an insecure system than an out of memory error.

MySQL can’t start if the boot device is full.

Yes those failed attempt was by me. So what should i do? what minimum RAM i need? i am testing system with 1-2 calls only.

2 GB RAM is more than enough. Perhaps you are running out of disk or some other resource.
Please post the output of:
free -t
df -k

1 Like

You have a ‘no space left on device’, it might be necessary to delete some file from /var/log before you can do much else in your shell


ls -l /var/log/asterisk/*

would be a good start as it’s a “low resource” process

image

Two thinks wrong with that,

A) it’s a picture :slight_smile:
B) It doesn’t show the result of my suggestion to list a likely culprit directory, ( just confirms “Yes I’m out of space on my root device”)

here is log with your requested results:

[root@freepbx ~]# ls -l /var/log/asterisk/*
-rw-rw-r--. 1 asterisk asterisk          0 Oct  8 22:02 /var/log/asterisk/core-fastagi_err.log
-rw-rw-r--. 1 asterisk asterisk     430080 Oct 16 19:58 /var/log/asterisk/core-fastagi_out.log
--w------T  1 asterisk asterisk 1015074816 Oct 18 09:22 /var/log/asterisk/fail2ban
-rw-rw-r--  1 asterisk asterisk          0 Oct 13 03:10 /var/log/asterisk/fail2ban.0
-rw-rw-r--  1 asterisk asterisk 1221794774 Oct 12 19:28 /var/log/asterisk/fail2ban.1
-rw-rw-r--  1 asterisk asterisk 4195085291 Oct 11 18:20 /var/log/asterisk/fail2ban.2
-rw-rw-r--. 1 asterisk asterisk  659267792 Oct 10 03:41 /var/log/asterisk/fail2ban-20211010
-rw-rw-r--  1 asterisk asterisk 3679675349 Oct 11 03:39 /var/log/asterisk/fail2ban-20211011
-rw-rw-r--  1 asterisk asterisk  136189448 Oct 12 03:43 /var/log/asterisk/fail2ban-20211012
-rw-rw-r--  1 asterisk asterisk 1370082808 Oct 13 03:10 /var/log/asterisk/fail2ban-20211013
-rw-rw-r--  1 asterisk asterisk  676863664 Oct 14 03:46 /var/log/asterisk/fail2ban-20211014
-rw-rw-r--  1 asterisk asterisk  305151672 Oct 15 03:07 /var/log/asterisk/fail2ban-20211015
-rw-rw-r--  1 asterisk asterisk  104472767 Oct 16 03:31 /var/log/asterisk/fail2ban-20211016
-rw-rw-r--  1 asterisk asterisk          0 Oct 11 03:39 /var/log/asterisk/fail2ban.3
-rw-rw-r--  1 asterisk asterisk 3144026890 Oct 10 11:04 /var/log/asterisk/fail2ban.4
-rw-rw-r--  1 asterisk asterisk        110 Oct 12 19:52 /var/log/asterisk/firewall.err
-rw-rw-r--  1 asterisk asterisk      11934 Oct 17 12:06 /var/log/asterisk/firewall.log
-rw-rw-r--  1 asterisk asterisk  536690688 Oct 16 18:50 /var/log/asterisk/firewall.log.old
-rw-rw-r--. 1 asterisk asterisk   40869888 Oct 17 16:21 /var/log/asterisk/freepbx.log
--w--wx--T  1 asterisk asterisk      15167 Oct 17 12:05 /var/log/asterisk/freepbx_security.log
-rw-rw-r--  1 asterisk asterisk  283582464 Oct 18 09:23 /var/log/asterisk/full
-rw-rw-r--  1 asterisk asterisk 1101435115 Oct 11 18:20 /var/log/asterisk/full.0
-rw-rw-r--  1 asterisk asterisk          0 Oct 11 03:39 /var/log/asterisk/full.1
-rw-rw-r--. 1 asterisk asterisk  184068585 Oct 10 03:42 /var/log/asterisk/full-20211010
-rw-rw-r--  1 asterisk asterisk 1900405072 Oct 11 03:39 /var/log/asterisk/full-20211011
-rw-rw-r--  1 asterisk asterisk   37089968 Oct 12 03:43 /var/log/asterisk/full-20211012
-rw-rw-r--  1 asterisk asterisk  700984746 Oct 13 03:10 /var/log/asterisk/full-20211013
-rw-rw-r--  1 asterisk asterisk  301166437 Oct 14 03:47 /var/log/asterisk/full-20211014
-rw-rw-r--  1 asterisk asterisk   99373186 Oct 15 03:08 /var/log/asterisk/full-20211015
-rw-rw-r--  1 asterisk asterisk  308359934 Oct 16 03:31 /var/log/asterisk/full-20211016
-rw-rw-r--  1 asterisk asterisk          0 Oct 14 03:47 /var/log/asterisk/pms.log
-rw-rw-r--  1 asterisk asterisk        126 Oct 13 14:40 /var/log/asterisk/pms.log-20211014
-rw-rw-r--  1 asterisk asterisk       1940 Oct 17 21:29 /var/log/asterisk/queue_log
-rw-rw-r--. 1 asterisk asterisk        676 Oct  9 21:42 /var/log/asterisk/queue_log-20211010
-rw-rw-r--  1 asterisk asterisk        196 Oct 11 01:44 /var/log/asterisk/queue_log-20211011
-rw-rw-r--  1 asterisk asterisk        120 Oct 11 18:20 /var/log/asterisk/queue_log-20211012
-rw-rw-r--  1 asterisk asterisk        356 Oct 12 19:48 /var/log/asterisk/queue_log-20211013
-rw-rw-r--  1 asterisk asterisk        268 Oct 14 01:53 /var/log/asterisk/queue_log-20211014
-rw-rw-r--  1 asterisk asterisk         40 Oct 14 03:47 /var/log/asterisk/queue_log-20211015
-rw-rw-r--  1 asterisk asterisk        120 Oct 15 18:54 /var/log/asterisk/queue_log-20211016
-rw-rw-r--  1 asterisk asterisk     270336 Oct 16 21:20 /var/log/asterisk/ucp_err.log
-rw-rw-r--  1 asterisk asterisk      28672 Oct 16 21:20 /var/log/asterisk/ucp_out.log
-rw-rw-r--  1 asterisk asterisk      33328 Oct 12 19:47 /var/log/asterisk/xmpp_err.log
-rw-rw-r--  1 asterisk asterisk       5148 Oct 12 19:49 /var/log/asterisk/xmpp_out.log
-rw-rw-r--  1 asterisk asterisk          0 Oct 17 12:05 /var/log/asterisk/zulu_out.log

/var/log/asterisk/cdr-csv:

First off delete

rm /var/log/asterisk/firewall.log.old

that should give you a minimal working space without destroying evidence, Now work out why your fail2ban logs are so large (actually that’s because your full logs are so large, you need to investigate whatsupdoc) and then why your fail2ban and full log rotations are abnormal.

To get yourself up and running , I suggest

rm -rf /var/log/asterisk/full-2021101[0-5]
reboot

Would be the easiest. We can diagnose after that

Deleted. but still asterisk unable to run
[root@freepbx ~]# asterisk -rvvvv
Unable to connect to remote asterisk (does /var/run/asterisk/asterisk.ctl exist?)

As I just edited, you might need more disk space and lets just reboot (technically unnecessary but the mallet over tweezer solution)

No it doesn’t!

image

That’s less that 500 MB, although I guess it corresponds to 500 or 512 when you correct for some reserved for the BIOS or hardware.

Your machine has 512M physical memory plus another 1G swap memory, this is probably more than enough for 2 concurrent calls, but your firewall is woefully not working !!

It would be enough for a few concurrent calls on a trimmed-down FreePBX, but it’s not enough to run Distro with all the goodies (which is the default install).

I believe it’s up against the memory limit and then fail2ban is working on those huge logs and bringing everything down.

My suggestion, if more memory isn’t an option, would be to be done with fail2ban, clear out most of those logs, fix the firewall to block all the junk, and disable services/modules that are not in use.

I would suggest that a ‘default install’ would not exhibit that horrible level of apparent intrusion attempts . . . .

Absent of a working Firewall, the OP can only expect that. (while listening on UDP/5060 :slight_smile: )

2 Likes

Here is another rain of errors. What’s going on this server? how to do i get rid of and make this box stable? by the way i increased RAM to 1 GB.