Why is my Disk always close to full?

I have searched and searched but i cannot find anyone that has said anything that helps.
Where and why is my disk always full!

Try logging on to server via ssh as root, to system shell / console (not asterisk console!) and tell it to

cd /

to go to root directory, then

du -h | sort -h

(may/will run for a very long time and spew warnings in the process)
but should return the sizes and names of the biggest files shown last (just above the prompt).

1 Like

login as: root
[email protected]’s password:
Last login: Tue Jul 28 09:33:36 2015 from 204.101.240.36

======================================
Welcome to The FreePBX Distro

[[email protected] ~]#

Does this look like the right place to be? I am not aware of logging in anywhere else.

yes, that’s the place.
(the warning 'bout not using the asterisk console was because someone on this forum did confuse the two before).
you may also want to run

df -h

see which system devices are getting low (which partitions)

1 Like

Here are the top of that list. what can i delete?! Does anything here look too high for what it is?

100M ./var/lib/asterisk/sounds
104M ./usr/src
109M ./usr/java/jre1.7.0_05/lib
110M ./usr/java
110M ./usr/java/jre1.7.0_05
113M ./lib/modules
113M ./lib/modules/2.6.32-220.13.1.el6.i686
121M ./var/lib/mysql/asteriskcdrdb
130M ./root/installer-tmp
130M ./root/installer-tmp/tmp
131M ./root
134M ./usr/libexec/webmin
144M ./var/lib/mysql
152M ./lib
168M ./var/lib/asterisk
169M ./usr/libexec
259M ./usr/lib
290M ./var/log/asterisk
297M ./usr/share
361M ./var/lib
441M ./var/log
893M ./var
1.1G ./usr
2.4G .

Now add the output of:-

cat /proc/partitions
df -h
mount

then go to /var/log
and do the du -h |sort -h again…

1 Like

and, from the above output, are you using it on a 4GB card on Raspberry PI ? Or other PIAF / small flash disk solution ?

[[email protected] log]# du -h | sort -n
1.6M ./pbx
1.6M ./pbx/install
4.0K ./asterisk/cdr-csv
4.0K ./asterisk/cdr-custom
4.0K ./cluster
4.0K ./ntpstats
4.0K ./pbx/upgrade
4.5M ./httpd
8.0K ./ConsoleKit
30M ./audit
294M ./asterisk
459M .

hmm, in /var/log, do

ls -lShr

then again rinse, repeat:

cd /var/log/asterisk
du -h | sort -h
ls -lShr
1 Like
total 131M
-rw-------  1 root     root        0 Jul 19 03:20 spooler-20150726
-rw-------  1 root     root        0 Jul 12 03:47 spooler-20150719
-rw-------  1 root     root        0 Jul  5 03:10 spooler-20150712
-rw-------  1 root     root        0 Jun 28 03:36 spooler-20150705
-rw-------  1 root     root        0 Jul 26 03:24 spooler
-rw-------  1 root     root        0 Nov 25  2014 sangoma_mgd.log
-rw-------  1 root     root        0 Nov 25  2014 sangoma_bri.log
-rw-------  1 root     root       49 Dec 15  2014 yum.log-20150101
-rw-------  1 root     root       58 Apr  9 13:48 yum.log
----------  1 root     root      292 Jul 29 08:54 lastlog
-rw-------  1 root     root      339 Jul 17 14:44 secure-20150719
-rw-------  1 root     root      339 Jul  7 14:16 secure-20150712
-rw-------  1 root     root      834 Jul 20 08:35 secure-20150726
-rw-------  1 root     root     1006 Jul  4 10:26 secure-20150705
-rw-r--r--  1 root     root     3.6K Jun  8 09:39 boot.log
drwxr-xr-x. 4 asterisk asterisk 4.0K Apr 29  2013 pbx
drwxr-xr-x. 2 ntp      ntp      4.0K Nov 28  2011 ntpstats
drwx------. 2 root     root     4.0K Jul 26 03:24 httpd
drwxr-xr-x. 2 root     root     4.0K Apr 29  2013 ConsoleKit
drwx------. 2 root     root     4.0K Apr 30  2012 cluster
drwxr-x---. 2 root     root     4.0K May 23 15:00 audit
drwxrwxr-x. 4 asterisk asterisk 4.0K Jul 29 08:48 asterisk
-rw-------  1 root     root     6.1K Jul 29 08:54 secure
-rw-r-----  1 mysql    mysql     11K Jun  8 09:38 mysqld.log
-rw-------  1 root     root      17K Jul 29 08:36 fail2ban.log
-rw-r--r--  1 root     root      19K Jun  8 14:31 dmesg.old
-rw-r--r--  1 root     root      19K Jun  8 09:38 dmesg
-rw-------  1 root     root      23K Jul 29 08:01 cron
-rw-------  1 root     root      31K Jan 23  2015 fail2ban.log-20150123
-rw-------  1 root     root      32K Mar 19 03:30 fail2ban.log-20150319
-rw-------  1 root     root      35K Jun  9 03:16 fail2ban.log-20150609
-rw-------  1 root     root      41K Dec 16  2014 fail2ban.log-20141216
-rw-------  1 root     root      48K Jul  5 03:10 cron-20150705
-rw-------  1 root     root      48K Jul 19 03:20 cron-20150719
-rw-------  1 root     root      49K Jul 12 03:47 cron-20150712
-rw-------  1 root     root      49K Jul 26 03:24 cron-20150726
-rw-------  1 root     root      61K Jul 26 03:24 lfd.log-20150726.gz
-rw-rw-r--  1 root     utmp     112K Jul 29 08:54 wtmp
-rw-------  1 root     root     409K Jul 29 08:54 lfd.log
-rw-r-----  1 asterisk asterisk 2.4M Jul 29 08:54 maillog
-rw-r-----  1 asterisk asterisk 4.7M Jul 12 03:45 maillog-20150712
-rw-r-----  1 asterisk asterisk 5.1M Jul 19 03:05 maillog-20150719
-rw-r-----  1 asterisk asterisk 5.7M Jul 26 03:05 maillog-20150726
-rw-r-----  1 asterisk asterisk 6.4M Jul  5 03:05 maillog-20150705
-rw-------  1 root     root      35M Jul 29 08:55 messages
-rw-------  1 root     root      71M Jul 26 03:24 messages-20150726


[[email protected] asterisk]# du -h | sort -h
4.0K    ./cdr-csv
4.0K    ./cdr-custom
294M   


[[email protected] asterisk]# ls -lShr
total 294M
-rw-r-----  1 asterisk asterisk   40 Jul 26 03:24 queue_log-20150727
-rw-r-----  1 asterisk asterisk   40 Jul 25 03:28 queue_log-20150726
-rw-r-----  1 asterisk asterisk   40 Jul 19 03:20 queue_log-20150720
-rw-r-----  1 asterisk asterisk   40 Jul 18 03:37 queue_log-20150719
-rw-r-----  1 asterisk asterisk   40 Jul 12 03:47 queue_log-20150713
-rw-r-----  1 asterisk asterisk   40 Jul 11 03:25 queue_log-20150712
-rw-r-----  1 asterisk asterisk   40 Jul  5 03:10 queue_log-20150706
-rw-r-----  1 asterisk asterisk   40 Jul  4 03:12 queue_log-20150705
-rw-r-----  1 asterisk asterisk   40 Jul  1 03:26 queue_log-20150702
-rw-r-----  1 asterisk asterisk   93 Jul 29 08:36 fail2ban
-rw-r-----  1 asterisk asterisk  384 Jul 26 15:51 fail2ban-20150727
-rw-r-----  1 asterisk asterisk 1001 Jul 29 08:55 queue_log
-rw-r-----  1 asterisk asterisk 1.1K Jul 21 16:27 fail2ban-20150722
-rw-r-----  1 asterisk asterisk 1.4K Jul 22 21:19 fail2ban-20150723
-rw-r-----  1 asterisk asterisk 1.5K Jul 28 18:39 fail2ban-20150729
-rw-r-----  1 asterisk asterisk 1.6K Jul 24 21:51 fail2ban-20150725
-rw-r-----  1 asterisk asterisk 1.6K Jul 23 23:15 fail2ban-20150724
drwxrwxr-x. 2 asterisk asterisk 4.0K Nov 21  2012 cdr-custom
drwxrwxr-x. 2 asterisk asterisk 4.0K Apr 29  2013 cdr-csv
-rw-r-----  1 asterisk asterisk 5.7K Jul 27 22:27 fail2ban-20150728
-rw-r-----  1 asterisk asterisk  14K Jul 21 17:03 queue_log-20150722
-rw-r-----  1 asterisk asterisk  16K Jul 16 15:54 queue_log-20150717
-rw-r-----  1 asterisk asterisk  17K Jul 17 16:56 queue_log-20150718
-rw-r-----  1 asterisk asterisk  20K Jul 28 16:57 queue_log-20150729
-rw-r-----  1 asterisk asterisk  25K Jul 22 16:58 queue_log-20150723
-rw-r-----  1 asterisk asterisk  28K Jul 10 16:37 queue_log-20150711
-rw-r-----  1 asterisk asterisk  29K Jul 20 16:58 queue_log-20150721
-rw-r-----  1 asterisk asterisk  30K Jul 23 16:36 queue_log-20150724
-rw-r-----  1 asterisk asterisk  30K Jul  9 16:42 queue_log-20150710
-rw-r-----  1 asterisk asterisk  31K Jul 15 16:45 queue_log-20150716
-rw-r-----  1 asterisk asterisk  36K Jul  8 16:52 queue_log-20150709
-rw-r-----  1 asterisk asterisk  40K Jun 29 16:12 queue_log-20150630
-rw-r-----  1 asterisk asterisk  44K Jul  7 16:56 queue_log-20150708
-rw-r-----  1 asterisk asterisk  53K Jul 26 03:24 full-20150726
-rw-r-----  1 asterisk asterisk  54K Jul  3 16:56 queue_log-20150704
-rw-r-----  1 asterisk asterisk  58K Jun 30 16:50 queue_log-20150701
-rw-r-----  1 asterisk asterisk  64K Jul 24 16:51 queue_log-20150725
-rw-r-----  1 asterisk asterisk  65K Jul 27 16:37 queue_log-20150728
-rw-r-----  1 asterisk asterisk  69K Jul  6 16:56 queue_log-20150707
-rw-r-----  1 asterisk asterisk  79K Jul 14 16:22 queue_log-20150715
-rw-r-----  1 asterisk asterisk  93K Jul 13 17:01 queue_log-20150714
-rw-r-----  1 asterisk asterisk 130K Jul  2 17:13 queue_log-20150703
-rw-r-----  1 asterisk asterisk 199K Jul 27 03:44 full-20150727
-rw-r-----  1 asterisk asterisk 469K Jul 29 08:55 full
-rw-rw-r--  1 asterisk asterisk 636K Apr 29  2013 freepbx_debug
-rw-r-----  1 asterisk asterisk 6.9M Jul 29 03:06 full-20150729
-rw-r-----  1 asterisk asterisk 8.8M Jul 24 03:27 full-20150724
-rw-r-----  1 asterisk asterisk 8.9M Jul 23 03:07 full-20150723
-rw-rw-r--  1 asterisk asterisk 8.9M Jul 23 09:24 freepbx.log
-rw-r--r--  1 asterisk asterisk 9.1M Jul 29 08:56 freepbx_dbug
-rw-r-----  1 asterisk asterisk  19M Jul 25 03:28 full-20150725
-rw-r-----  1 asterisk asterisk  24M Jul 28 03:34 full-20150728
-rw-rw-r--  1 asterisk asterisk 207M Jul 29 08:55 cdr.db

OK so …
seems you have a lot in /var/log/messages, and that your logrotate may be configured to keep too much back history. Maybe delve into the messages (and fail2ban) logs, also see the /var/log/asterisk/full.log and its previous copies - see what fills them ? Also your maillog seems excessively large - basically, try looking into any of these large log files and try to understand what fills them (I AM NOT asking to post content(s) here, try understanding them first yourself…)
(useful commands : less [filename], cat filename | more, mc:slight_smile: )

So I think the general concensus is that his disk is so close to full because he doesn’t have much disk space.

Seems like a reasonable result to me :sunglasses:

Note, @Laveniuk, it would be REALLY HANDY if you could actually show us how much disk space you have. Run the command df -h and paste the output.

My guess is that the OP actually has a big disk but it is inappropriately partitioned/mounted (note the /cluster directory? ), hence my request for information, Pretty well everything fits nicely in 5G if a big partition is mounted on /var

cat /proc/partitions
df -h
mount

do that and it is fixable.

1 Like

@dicko - I have one (other, not FreePBX) server partitioned by Debian Installer (duh) and yeah that installer gives not a lot to /var out of the 75GB disk(s) used:

[server] ~$ df -h
Filesystem                                              Size  Used Avail Use% Mounted on
[...]
rootfs                                                  314M  193M  101M  66% /
/dev/md5                                                 61G   16G   43G  27% /home
/dev/md4                                                360M  2.1M  335M   1% /tmp
/dev/md1                                                8.2G  620M  7.1G   8% /usr
/dev/md2                                                2.7G  2.3G  309M  89% /var

(that’s btw guided partitioning of Debian Wheezy for you… )
But I suspect more that OP is running on a small 4GB flash somewhere.
(or, re-looking at his second post, on a VM in the cloud somewhere and he’s given it too little of disk).

A “guided partitioning” of Debian will indeed offer that schema. but in FreePBX, the only content of /home would be /home/asterisk with presumably a content only of .gpg, this is, also presumably, to have the asterisk user have access to GPG when and if your “HA” schema fails.

So in your case where you have the majority of the disk appropriated by home , and I guess you have 43G worth of NON asterisk stuff there
.
I would suggest you are yourself dangerously close to running out of space on /var my guess /var/log but you know how to figure that out with your bash-fu and fix it before your system breaks :wink:

@dicko this is

: - )

My proper freepbx server uses ‘default FreePBX distro partitioning’ /boot and / scheme :wink: and has about 29% usage on /

On the machine with 309M (89%) left you ARE close to critical. Journaled filesystems get very needy as you get past 80% and growing, check your logrotate scripts.

29% of what? it is all relative, /var will in any Asterisk/FreePBX/(Hylafax)/apache implementation amount to a vast majority of your disk needs, just look at it’s content, mysql, recordings, voicemail, faxes, etc etc etc. THAT / partition is going to b happy with 5-10 G the rest needs to be for var or just use a BIG_ENOUGH disk for / and don’t worry (until it is full :wink: then the whole thing dies not just asterisk) , this guy just didn’t RTFM . . .

Just look at his third post, do you have a /cluster directory, if so what does it contain ?

hint:-
du -h --max-depth=1 /

compare with

du -hx --max-depth=1 /

if you want to separate disk from partition usage

That machine with 21% space on /var left, is an (to be retired for hw reasons) auxilary backup file server :wink: running Debian Wheezy.

FreePBX machine is a Completely Different Machine and not even in the same room :wink: running Freepbx distro.

The FreePBX machine has

# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/md2         72G   20G   49G  29% /
tmpfs           499M     0  499M   0% /dev/shm
/dev/md0        291M   32M  245M  12% /boot

(this is default as FreePBX distro partitioned it while installing back in November… :wink: )

[[email protected] ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda2 59G 48G 7.9G 86% /
tmpfs 1.5G 0 1.5G 0% /dev/shm
/dev/sda1 97M 26M 67M 28% /boot

here is my disk space. 7.9 GB left out of a 60GB