Hello Everyone
i have 2 servers on 2 different location
it is running since couple years now
yesterday i got alert that server is getting full upon checking i found out that
storage is getting shrink each and every minutes until the server stopped working
i couldn’t find anything at all , at this moment both servers are down which creates lots of problem for us
when i login to cli i see the following message
Please help
I would first off
ls -lsrt /var/log/asterisk/
then delete files that are older than yesterday, that should give you space to tail the files that are growing so quick to see why.
i have deleted some of the log files but it is very strange all the sudden both server is running low on desk space , even they are not on the same network
That’s why I suggest you watch what is being written to the files growing so fast, intrusions often attack as a pack so you need to ‘nip them in the bud’
this might be helpful as well…
from the OS CLI:
find / -type f -size +20M -exec ls -lh {} \; | awk '{ print $NF ": " $5 }'
Find large (over 20M) files
du -m / | sort -n | tail -n 20
See what’s eating disk space
this is what the result shows
A lots of /tem/core.asterisk , i don’t know what this use for but it is a lot
/tftpboot/digium/0/D65.eff: 35M
/tftpboot/digium/0/D50.eff: 31M
/tftpboot/digium/0/D40.eff: 31M
/tftpboot/digium/0/D60.eff: 34M
/tftpboot/digium/0/D62.eff: 34M
/tftpboot/digium/0/D45.eff: 31M
/tftpboot/digium/0/D70.eff: 31M
/proc/kcore: 128T
find: ‘/proc/11213/task/11213/fdinfo/6’: No such file or directory
find: ‘/proc/11213/fdinfo/5’: No such file or directory
/sys/devices/pci0000:00/0000:00:08.0/resource0: 64M
/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.262.b10-0.el7_8.x86_64/jre/lib/rt.jar: 71M
/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.262.b10-0.el7_8.x86_64/jre/lib/amd64/server/classes.jsa: 22M
/usr/lib/modules/3.10.0-1127.el7.x86_64/extra/wanpipe/wanpipe.ko: 24M
/usr/lib/modules/3.10.0-1127.10.1.el7.x86_64/extra/wanpipe/wanpipe.ko: 24M
/usr/lib/modules/3.10.0-862.2.3.el7.x86_64/extra/wanpipe/wanpipe.ko: 23M
/usr/lib/modules/3.10.0-693.2.2.el7.x86_64/extra/wanpipe/wanpipe.ko: 23M
/usr/lib/modules/3.10.0-957.10.1.el7.x86_64/extra/wanpipe/wanpipe.ko: 24M
/usr/lib/locale/locale-archive: 102M
/usr/bin/node: 28M
/usr/lib64/libwireshark.so.3.1.14: 59M
/usr/share/freepbx/npm-bootstrap.tar.gz: 89M
/var/www/html/digium_phones/firmware_2_9_9_package/2_9_9_D62_firmware.eff: 34M
/var/www/html/digium_phones/firmware_2_9_9_package/2_9_9_D65_firmware.eff: 36M
/var/www/html/digium_phones/firmware_2_9_9_package/2_9_9_D45_firmware.eff: 32M
/var/www/html/digium_phones/firmware_2_9_9_package/2_9_9_D60_firmware.eff: 34M
/var/www/html/digium_phones/firmware_2_9_9_package/2_9_9_D50_firmware.eff: 32M
/var/www/html/digium_phones/firmware_2_9_9_package/2_9_9_D40_firmware.eff: 32M
/var/www/html/digium_phones/firmware_2_9_9_package/2_9_9_D70_firmware.eff: 32M
/var/www/html/digium_phones/firmware_1_4_2_0_package/1_4_2_0_63880_D40_firmware.eff: 33M
/var/www/html/digium_phones/firmware_1_4_2_0_package/1_4_2_0_63880_D70_firmware.eff: 33M
/var/www/html/digium_phones/firmware_1_4_2_0_package/1_4_2_0_63880_D45_firmware.eff: 33M
/var/www/html/digium_phones/firmware_1_4_2_0_package/1_4_2_0_63880_D50_firmware.eff: 33M
/var/www/html/digium_phones/firmware_2_9_14_package/2_9_14_D62_firmware.eff: 34M
/var/www/html/digium_phones/firmware_2_9_14_package/2_9_14_D50_firmware.eff: 32M
/var/www/html/digium_phones/firmware_2_9_14_package/2_9_14_D70_firmware.eff: 32M
/var/www/html/digium_phones/firmware_2_9_14_package/2_9_14_D45_firmware.eff: 32M
/var/www/html/digium_phones/firmware_2_9_14_package/2_9_14_D60_firmware.eff: 34M
/var/www/html/digium_phones/firmware_2_9_14_package/2_9_14_D40_firmware.eff: 32M
/var/www/html/digium_phones/firmware_2_9_14_package/2_9_14_D65_firmware.eff: 36M
/var/www/html/digium_phones/firmware_2_4_2_package/2_4_2_D62_firmware.eff: 34M
/var/www/html/digium_phones/firmware_2_4_2_package/2_4_2_D40_firmware.eff: 31M
/var/www/html/digium_phones/firmware_2_4_2_package/2_4_2_D65_firmware.eff: 35M
/var/www/html/digium_phones/firmware_2_4_2_package/2_4_2_D45_firmware.eff: 32M
/var/www/html/digium_phones/firmware_2_4_2_package/2_4_2_D60_firmware.eff: 34M
/var/www/html/digium_phones/firmware_2_4_2_package/2_4_2_D50_firmware.eff: 31M
/var/www/html/digium_phones/firmware_2_4_2_package/2_4_2_D70_firmware.eff: 31M
/var/www/html/digium_phones/firmware_2_9_10_package/2_9_10_D50_firmware.eff: 32M
/var/www/html/digium_phones/firmware_2_9_10_package/2_9_10_D62_firmware.eff: 34M
/var/www/html/digium_phones/firmware_2_9_10_package/2_9_10_D70_firmware.eff: 32M
/var/www/html/digium_phones/firmware_2_9_10_package/2_9_10_D65_firmware.eff: 36M
/var/www/html/digium_phones/firmware_2_9_10_package/2_9_10_D60_firmware.eff: 34M
/var/www/html/digium_phones/firmware_2_9_10_package/2_9_10_D40_firmware.eff: 32M
/var/www/html/digium_phones/firmware_2_9_10_package/2_9_10_D45_firmware.eff: 32M
/var/www/html/digium_phones/firmware_1_4_2_1_package/1_4_2_1_73372_D45_firmware.eff: 33M
/var/www/html/digium_phones/firmware_1_4_2_1_package/1_4_2_1_73372_D40_firmware.eff: 33M
/var/www/html/digium_phones/firmware_1_4_2_1_package/1_4_2_1_73372_D50_firmware.eff: 33M
/var/www/html/digium_phones/firmware_1_4_2_1_package/1_4_2_1_73372_D70_firmware.eff: 33M
/var/www/html/admin/modules/_cache/endpoint-14.0.68.48.tgz.gpg: 44M
/var/www/html/admin/modules/_cache/endpoint-14.0.68.62.tgz.gpg: 44M
/var/www/html/admin/modules/_cache/zulu-14.0.58.11.tgz.gpg: 22M
/var/www/html/admin/modules/_cache/endpoint-14.0.70.4.tgz.gpg: 44M
/var/www/html/admin/modules/_cache/endpoint-14.0.70.tgz.gpg: 44M
/var/www/html/admin/modules/_cache/endpoint-14.0.70.10.tgz.gpg: 44M
/var/www/html/admin/modules/_cache/zulu-14.0.58.3.tgz.gpg: 22M
/var/www/html/admin/modules/_cache/endpoint-14.0.68.74.tgz.gpg: 44M
/var/www/html/admin/modules/_cache/endpoint-14.0.68.39.tgz.gpg: 37M
/var/www/html/admin/modules/_cache/endpoint-14.0.68.73.tgz.gpg: 44M
/var/www/html/admin/modules/_cache/endpoint-14.0.68.59.tgz.gpg: 44M
/var/www/html/admin/modules/_cache/endpoint-15.0.41.11.tgz.gpg: 49M
/var/www/html/admin/modules/_cache/zulu-14.0.58.9.tgz.gpg: 22M
/var/cache/yum/x86_64/7/sng-epel/gen/primary_db.sqlite: 36M
/var/cache/yum/x86_64/7/sng-updates/gen/primary_db.sqlite: 23M
/var/cache/yum/x86_64/7/sng-base/gen/primary_db.sqlite: 31M
/var/cache/yum/x86_64/6/updates/primary.xml.gz.sqlite: 26M
/var/lib/rpm/Packages: 136M
/var/lib/mysql/ibdata1: 82M
/var/lib/mysql/asteriskcdrdb/cel.TMD: 1.2G
/var/lib/mysql/asteriskcdrdb/cel.MYI: 184M
/var/lib/mysql/asteriskcdrdb/cdr.MYI: 28M
/var/lib/mysql/asteriskcdrdb/cdr.MYD: 172M
/var/lib/mysql/asteriskcdrdb/cel.MYD: 1.2G
/var/lib/system-upgrade/asterisk-sounds-core-en-wideband-1.5-2.sng7.noarch.rpm: 48M
/var/lib/system-upgrade/sangoma-logos-70.0.6-6.sng7.noarch.rpm: 23M
/var/lib/system-upgrade/java-1.8.0-openjdk-headless-1.8.0.151-1.b12.el7_4.x86_64.rpm: 32M
/var/lib/system-upgrade/kernel-3.10.0-693.5.2.el7.x86_64.rpm: 43M
/var/lib/system-upgrade/asterisk-sounds-moh-opsound-sln-2.03-1.1.noarch.rpm: 42M
/var/lib/system-upgrade/asterisk-sounds-extra-en-wideband-1.5-2.sng7.noarch.rpm: 82M
/var/lib/system-upgrade/freepbx-14.1-1.sng7.noarch.rpm: 155M
/var/lib/system-upgrade/linux-firmware-20170606-56.gitc990aae.el7.noarch.rpm: 36M
/var/lib/mongodb/local.0: 64M
/var/lib/mongodb/journal/prealloc.2: 1.0G
/var/lib/mongodb/journal/prealloc.1: 1.0G
/var/lib/mongodb/journal/j._0: 128M
/var/lib/mongodb/letschat.1: 128M
/var/lib/mongodb/letschat.0: 64M
/var/spool/asterisk/monitor/1457982066.12836.wav: 35M
/var/spool/asterisk/monitor/1417715928.207.wav: 36M
/var/spool/asterisk/monitor/1416855674.76381.wav: 92M
/var/spool/asterisk/monitor/1421352164.10179.wav: 21M
/var/spool/asterisk/monitor/1465228503.12429.wav: 48M
/var/spool/asterisk/monitor/1444407879.79821.wav: 55M
/var/spool/asterisk/monitor/1460133617.9238.wav: 26M
/var/spool/asterisk/monitor/1464728599.8402.wav: 52M
/var/spool/asterisk/monitor/1446831712.16222.wav: 21M
/var/spool/asterisk/monitor/1439236829.31397.wav: 27M
/var/spool/asterisk/monitor/1435345144.33771.wav: 22M
/var/spool/asterisk/monitor/1464018408.3206.wav: 22M
/var/spool/asterisk/monitor/2018/01/26/ondemand–3200-20180126-092659-1516987620.7692.wav: 47M
/var/spool/asterisk/monitor/1442602629.65300.wav: 25M
/var/spool/asterisk/monitor/1443029170.68507.wav: 24M
/var/spool/asterisk/backup/Pre_Restore/20170712-182347-1499909027-13.0.192.9-1590956317.tgz: 599M
/var/spool/asterisk/backup/Backup_and_FTP/20211008-000002-1633676402-14.0.16.11-574871453.tgz: 2.1G
/var/spool/asterisk/backup/fresh-backup/20211011-134745-1633985265-15.0.17.55-1022350649.tar.gz: 348M
/var/log/asterisk/fail2ban-20211010: 24M
/var/log/asterisk/full-20211009: 25M
/var/log/asterisk/full-20211007: 27M
/var/log/asterisk/fail2ban-20211007: 25M
/var/log/asterisk/fail2ban-20211006: 24M
/var/log/asterisk/freepbx.log: 78M
/var/log/asterisk/fail2ban-20211005: 24M
/var/log/asterisk/full-20211008: 28M
/var/log/asterisk/full-20211011: 446M
/var/log/asterisk/full-20211005: 31M
/var/log/asterisk/full-20211006: 26M
/var/log/asterisk/freepbx.log-20211005: 105M
/var/log/asterisk/fail2ban-20211008: 25M
/var/log/asterisk/fail2ban-20211009: 25M
/boot/initramfs-redhat-upgrade-tool.img: 43M
/tmp/core.asterisk.kwtemecula.com-2021-10-10T17:14:54-0700: 152M
/tmp/core.asterisk.kwtemecula.com-2021-10-10T16:07:30-0700: 164M
/tmp/core.asterisk.kwtemecula.com-2021-10-10T21:15:36-0700: 183M
/tmp/core.asterisk.kwtemecula.com-2021-10-10T14:16:15-0700: 162M
/tmp/core.asterisk.kwtemecula.com-2021-10-10T17:44:38-0700: 181M
/tmp/core.asterisk.kwtemecula.com-2021-10-10T17:29:08-0700: 160M
/tmp/core.asterisk.kwtemecula.com-2021-10-10T14:04:09-0700: 163M
/tmp/core.asterisk.kwtemecula.com-2021-10-10T14:24:12-0700: 176M
/tmp/core.asterisk.kwtemecula.com-2021-10-10T15:30:32-0700: 154M
/tmp/core.asterisk.kwtemecula.com-2021-10-10T15:30:40-0700: 155M
/tmp/core.asterisk.kwtemecula.com-2021-10-10T21:27:08-0700: 161M
/tmp/core.asterisk.kwtemecula.com-2021-10-10T15:26:35-0700: 173M
/tmp/core.asterisk.kwtemecula.com-2021-10-10T16:01:52-0700: 183M
/tmp/core.asterisk.kwtemecula.com-2021-10-10T16:20:08-0700: 161M
/tmp/core.asterisk.kwtemecula.com-2021-10-10T16:14:55-0700: 166M
/tmp/core.asterisk.kwtemecula.com-2021-10-10T18:45:05-0700: 161M
/tmp/core.asterisk.kwtemecula.com-2021-10-10T15:31:24-0700: 158M
/tmp/core.asterisk.kwtemecula.com-2021-10-10T16:01:25-0700: 163M
/tmp/core.asterisk.kwtemecula.com-2021-10-10T16:19:02-0700: 162M
/tmp/core.asterisk.kwtemecula.com-2021-10-10T16:27:42-0700: 158M
/tmp/core.asterisk.kwtemecula.com-2021-10-10T13:52:06-0700: 172M
/tmp/core.asterisk.kwtemecula.com-2021-10-10T18:46:23-0700: 182M
/tmp/core.asterisk.kwtemecula.com-2021-10-10T17:28:38-0700: 153M
/tmp/core.asterisk.kwtemecula.com-2021-10-10T14:45:30-0700: 172M
/tmp/core.asterisk.kwtemecula.com-2021-10-10T17:44:51-0700: 183M
/tmp/core.asterisk.kwtemecula.com-2021-10-10T15:50:35-0700: 159M
/tmp/core.asterisk.kwtemecula.com-2021-10-10T15:15:04-0700: 166M
/tmp/core.asterisk.kwtemecula.com-2021-10-10T17:57:06-0700: 166M
/tmp/core.asterisk.kwtemecula.com-2021-10-10T15:50:07-0700: 167M
/tmp/core.asterisk.kwtemecula.com-2021-10-10T16:08:07-0700: 174M
/tmp/core.asterisk.kwtemecula.com-2021-10-10T14:08:00-0700: 164M
/tmp/core.asterisk.kwtemecula.com-2021-10-10T17:15:02-0700: 165M
/tmp/core.asterisk.kwtemecula.com-2021-10-10T17:50:14-0700: 161M
/tmp/core.asterisk.kwtemecula.com-2021-10-10T17:16:15-0700: 162M
/tmp/core.asterisk.kwtemecula.com-2021-10-10T17:11:18-0700: 154M
/tmp/core.asterisk.kwtemecula.com-2021-10-10T14:19:54-0700: 156M
/tmp/core.asterisk.kwtemecula.com-2021-10-10T14:40:18-0700: 166M
/tmp/core.asterisk.kwtemecula.com-2021-10-10T15:06:11-0700: 162M
/tmp/core.asterisk.kwtemecula.com-2021-10-10T17:06:45-0700: 162M
/tmp/core.asterisk.kwtemecula.com-2021-10-10T21:24:23-0700: 174M
/tmp/core.asterisk.kwtemecula.com-2021-10-10T16:06:17-0700: 179M
/tmp/core.asterisk.kwtemecula.com-2021-10-10T16:13:46-0700: 152M
/tmp/core.asterisk.kwtemecula.com-2021-10-10T15:23:17-0700: 177M
/tmp/core.asterisk.kwtemecula.com-2021-10-10T14:17:59-0700: 157M
/tmp/core.asterisk.kwtemecula.com-2021-10-10T18:09:18-0700: 157M
/tmp/core.asterisk.kwtemecula.com-2021-10-10T17:22:37-0700: 157M
/tmp/core.asterisk.kwtemecula.com-2021-10-10T16:11:03-0700: 175M
/root/installer-tmp/tmp/install.img: 146M
/home/asterisk/.package_cache/npm/6.4.1/a44d9cc169b29653a8aa0fe4e280abfb.tar.gz: 47M
/home/asterisk/.package_cache/npm/6.4.1/88f203e58167971e6fac356f1e40d054.tar.gz: 28M
You can automate a lot of this ‘log watching’ by installing logwatch, “yum install logwatch” should work, add a cron job
20 4 * * * logwatch --mailto [email protected]
It will mail you a daily report of things that you might want to look at like low disk space and untoward connections, by default it does well, you can customize much more if you want.
You need to find out why you are ‘core dumping’, at the same time stamp of a dump should be some lines in /var/log/messages (or perhaps /var/log/syslog) indicating how asterisk died.
This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.