Emergency, Hard drive full, can't connect to asterisk, logfiles huge

Hey guys,

I’m in a bit of a bind. A few days ago things started going crazy on our phones, and we realized it was because we had forgotten to take some of the call recordings off of the server. We removed them, but space just keeps getting filled up. I’ve figured out that it is because of our asterisk logfiles.

How can I delete these safely, and keep them from getting to these huge sizes. like 9G in a day?

We are not currently able to use our system because of this. Thankfully it is happening over the weekend and we were able to forward our calls from a sip provider, but the weekend is coming to a close and this needs to be solved today.

This is the output of /var/log/asterisk $ du -sh *

0 backup.log
4.0K backup.log-20171105
0 backup.log-20171112
0 backup.log-20171119
0 backup.log-20171126
4.0K backup.log-20171203
32M cdr-csv
4.0K cdr-custom
52M cdr.db
4.0K cel-custom
711M fail2ban
1.1G fail2ban.0
88M freepbx_dbug
0 freepbx_debug
0 freepbx_debug-20171112
0 freepbx_debug-20171119
0 freepbx_debug-20171126
0 freepbx_debug-20171203
0 freepbx.log
160K freepbx.log-20171029
116K freepbx.log-20171105
128K freepbx.log-20171112
116K freepbx.log-20171119
124K freepbx.log-20171126
256K freepbx.log-20171203
0 freepbx_security.log
96K freepbx_security.log-20171029
4.0K freepbx_security.log-20171105
12K freepbx_security.log-20171112
8.0K freepbx_security.log-20171119
0 freepbx_security.log-20171126
4.0K freepbx_security.log-20171203
15M full
4.0K full.0
23M full-20171128
14M full-20171129
19M full-20171130
17M full-20171201
9.0G full-20171202
2.1G full-20171203
8.0K messages
0 queue_log
4.0K queue_log.0
4.0K queue_log-20171029
4.0K queue_log-20171105
4.0K queue_log-20171112
4.0K queue_log-20171119
4.0K queue_log-20171126
4.0K queue_log-20171203

Notice:

9.0G full-20171202
2.1G full-20171203

Thanks for your help.

You can delete old logs safely, current logs can be truncated with cat /dev/null > name_of_file

What should be analyzed is why your log is reaching 9gb in only one day. Maybe it is normal according to the load and activity of your asterisk installation, but maybe it is an indication of possible issues, if your logs are filled with warning messages.

So you’re saying I can run cat /dev/null > full-20171202 and then run rm full-20171202. And that will delete the file safely?

No need to truncate the file, first delete all archived log files

rm -rf /var/log/asterisk/2017

Then to fix your problem down line, edit the /etc/logrotate.d/asterisk and replace rotate 5 with something less, then to cleanly rotate the logs with

logrotate -fv /etc/logrotate.conf

I suggest you install and setup logwatch so you get to know when it’s going to happen again

I meant that current log file can be truncated if you don’t want to stop the process writing on it. For example, file “full” is current asterisk log file. To delete it and actually free the space, asterisk must not be running, so you either stop asterisk and delete the file, or you truncate it without the need to stop asterisk.

I’m still not able to connect to asterisk, even after clearing space. What can I do to figure out what’s wrong with it?

I can restart it, but it never shows as being able to connect in the gui, and it soon quits anyways.

Make sure asterisk is NOT running., then start it in the forground

asterisk -vvvvvc

This is the output. Some I left out the last part, because it was setting some passwords as variables. But It looks like this is the section that contains the secret. I just don’t know how to read it.

asterisk -vvvvvc
Asterisk 11.5.1, Copyright © 1999 - 2012 Digium, Inc. and others.
Created by Mark Spencer [email protected]
Asterisk comes with ABSOLUTELY NO WARRANTY; type ‘core show warranty’ for detail s.
This is free software, with components licensed under the GNU General Public
License version 2 and other licenses; you are welcome to redistribute it under
certain conditions. Type ‘core show license’ for details.

== Parsing ‘/etc/asterisk/extconfig.conf’: Found
== Parsing ‘/etc/asterisk/logger.conf’: Found
== Parsing ‘/etc/asterisk/logger_general_additional.conf’: Found
== Parsing ‘/etc/asterisk/logger_general_custom.conf’: Found
== Parsing ‘/etc/asterisk/logger_logfiles_additional.conf’: Found
== Parsing ‘/etc/asterisk/logger_logfiles_custom.conf’: Found
== Parsing ‘/etc/asterisk/asterisk.conf’: Found
== Manager registered action DBGet
== Manager registered action DBPut
== Manager registered action DBDel
== Manager registered action DBDelTree
== Registered custom function ‘MESSAGE’
== Registered custom function ‘MESSAGE_DATA’
== Registered application ‘MessageSend’
== Manager registered action MessageSend
== Manager registered action DataGet
== Parsing ‘/etc/asterisk/codecs.conf’: Found
Asterisk Dynamic Loader Starting:
== Parsing ‘/etc/asterisk/modules.conf’: Found
[2017-12-03 16:20:03] NOTICE[9586]: loader.c:1190 load_modules: 7 modules will b e loaded.
[2017-12-03 16:20:03] WARNING[9586]: loader.c:423 load_dynamic_module: Error loa ding module ‘res_mwi_blf.so’: /usr/lib/asterisk/modules/res_mwi_blf.so: cannot o pen shared object file: No such file or directory
== Parsing ‘/etc/asterisk/res_odbc.conf’: Found
== Parsing ‘/etc/asterisk/res_odbc_custom.conf’: Found
[2017-12-03 16:20:03] WARNING[9586]: config.c:341 ast_include_new: ‘/etc/asteris k/res_odbc_custom.conf’, line 24: Same File included more than once! This data will be saved in res_odbc_custom.conf1 if saved back to disk.
== Parsing ‘/etc/asterisk/res_odbc_custom.conf’: Found
[2017-12-03 16:20:03] WARNING[9586]: config.c:341 ast_include_new: ‘/etc/asteris k/res_odbc_custom.conf’, line 24: Same File included more than once! This data will be saved in res_odbc_custom.conf
2 if saved back to disk.
== Parsing ‘/etc/asterisk/res_odbc_custom.conf’: Found
[2017-12-03 16:20:03] WARNING[9586]: config.c:341 ast_include_new: ‘/etc/asteris k/res_odbc_custom.conf’, line 24: Same File included more than once! This data will be saved in res_odbc_custom.conf3 if saved back to disk.
== Parsing ‘/etc/asterisk/res_odbc_custom.conf’: Found
[2017-12-03 16:20:03] WARNING[9586]: config.c:341 ast_include_new: ‘/etc/asteris k/res_odbc_custom.conf’, line 24: Same File included more than once! This data will be saved in res_odbc_custom.conf
4 if saved back to disk.
== Parsing ‘/etc/asterisk/res_odbc_custom.conf’: Found
[2017-12-03 16:20:03] WARNING[9586]: config.c:341 ast_include_new: ‘/etc/asteris k/res_odbc_custom.conf’, line 24: Same File included more than once! This data will be saved in res_odbc_custom.conf5 if saved back to disk.
== Parsing ‘/etc/asterisk/res_odbc_custom.conf’: Found
[2017-12-03 16:20:03] WARNING[9586]: config.c:341 ast_include_new: ‘/etc/asteris k/res_odbc_custom.conf’, line 24: Same File included more than once! This data will be saved in res_odbc_custom.conf
6 if saved back to disk.
== Parsing ‘/etc/asterisk/res_odbc_custom.conf’: Found
[2017-12-03 16:20:03] WARNING[9586]: config.c:341 ast_include_new: ‘/etc/asteris k/res_odbc_custom.conf’, line 24: Same File included more than once! This data will be saved in res_odbc_custom.conf7 if saved back to disk.
== Parsing ‘/etc/asterisk/res_odbc_custom.conf’: Found
[2017-12-03 16:20:03] WARNING[9586]: config.c:341 ast_include_new: ‘/etc/asteris k/res_odbc_custom.conf’, line 24: Same File included more than once! This data will be saved in res_odbc_custom.conf
8 if saved back to disk.
== Parsing ‘/etc/asterisk/res_odbc_custom.conf’: Found
[2017-12-03 16:20:03] WARNING[9586]: config.c:341 ast_include_new: ‘/etc/asteris k/res_odbc_custom.conf’, line 24: Same File included more than once! This data will be saved in res_odbc_custom.conf9 if saved back to disk.
== Parsing ‘/etc/asterisk/res_odbc_custom.conf’: Found
[2017-12-03 16:20:03] WARNING[9586]: config.c:341 ast_include_new: ‘/etc/asteris k/res_odbc_custom.conf’, line 24: Same File included more than once! This data will be saved in res_odbc_custom.conf
10 if saved back to disk.
[2017-12-03 16:20:03] WARNING[9586]: config.c:2481 ast_config_internal_load: Max imum Include level (10) exceeded
[2017-12-03 16:20:03] ERROR[9586]: config.c:1414 process_text_line: The file ‘re s_odbc_custom.conf’ was listed as a #include but it does not exist.
[2017-12-03 16:20:03] WARNING[9586]: config.c:2481 ast_config_internal_load: Max imum Include level (10) exceeded
[2017-12-03 16:20:03] ERROR[9586]: config.c:1414 process_text_line: The file ‘re s_odbc_additional.conf’ was listed as a #include but it does not exist.
[2017-12-03 16:20:03] WARNING[9586]: config.c:341 ast_include_new: ‘/etc/asteris k/res_odbc_custom.conf’, line 25: Same File included more than once! This data will be saved in res_odbc_additional.conf1 if saved back to disk.
[2017-12-03 16:20:03] WARNING[9586]: config.c:2481 ast_config_internal_load: Max imum Include level (10) exceeded
[2017-12-03 16:20:03] ERROR[9586]: config.c:1414 process_text_line: The file ‘re s_odbc_additional.conf’ was listed as a #include but it does not exist.
[2017-12-03 16:20:03] WARNING[9586]: config.c:341 ast_include_new: ‘/etc/asteris k/res_odbc_custom.conf’, line 25: Same File included more than once! This data will be saved in res_odbc_additional.conf
2 if saved back to disk.
[2017-12-03 16:20:03] WARNING[9586]: config.c:2481 ast_config_internal_load: Max imum Include level (10) exceeded
[2017-12-03 16:20:03] ERROR[9586]: config.c:1414 process_text_line: The file ‘re s_odbc_additional.conf’ was listed as a #include but it does not exist.
[2017-12-03 16:20:03] WARNING[9586]: config.c:341 ast_include_new: ‘/etc/asteris k/res_odbc_custom.conf’, line 25: Same File included more than once! This data will be saved in res_odbc_additional.conf3 if saved back to disk.
[2017-12-03 16:20:03] WARNING[9586]: config.c:2481 ast_config_internal_load: Max imum Include level (10) exceeded
[2017-12-03 16:20:03] ERROR[9586]: config.c:1414 process_text_line: The file ‘re s_odbc_additional.conf’ was listed as a #include but it does not exist.
[2017-12-03 16:20:03] WARNING[9586]: config.c:341 ast_include_new: ‘/etc/asteris k/res_odbc_custom.conf’, line 25: Same File included more than once! This data will be saved in res_odbc_additional.conf
4 if saved back to disk.
[2017-12-03 16:20:03] WARNING[9586]: config.c:2481 ast_config_internal_load: Max imum Include level (10) exceeded
[2017-12-03 16:20:03] ERROR[9586]: config.c:1414 process_text_line: The file ‘re s_odbc_additional.conf’ was listed as a #include but it does not exist.
[2017-12-03 16:20:03] WARNING[9586]: config.c:341 ast_include_new: ‘/etc/asteris k/res_odbc_custom.conf’, line 25: Same File included more than once! This data will be saved in res_odbc_additional.conf5 if saved back to disk.
[2017-12-03 16:20:03] WARNING[9586]: config.c:2481 ast_config_internal_load: Max imum Include level (10) exceeded
[2017-12-03 16:20:03] ERROR[9586]: config.c:1414 process_text_line: The file ‘re s_odbc_additional.conf’ was listed as a #include but it does not exist.
[2017-12-03 16:20:03] WARNING[9586]: config.c:341 ast_include_new: ‘/etc/asteris k/res_odbc_custom.conf’, line 25: Same File included more than once! This data will be saved in res_odbc_additional.conf
6 if saved back to disk.
[2017-12-03 16:20:03] WARNING[9586]: config.c:2481 ast_config_internal_load: Max imum Include level (10) exceeded
[2017-12-03 16:20:03] ERROR[9586]: config.c:1414 process_text_line: The file ‘re s_odbc_additional.conf’ was listed as a #include but it does not exist.
[2017-12-03 16:20:03] WARNING[9586]: config.c:341 ast_include_new: ‘/etc/asteris k/res_odbc_custom.conf’, line 25: Same File included more than once! This data will be saved in res_odbc_additional.conf7 if saved back to disk.
[2017-12-03 16:20:03] WARNING[9586]: config.c:2481 ast_config_internal_load: Max imum Include level (10) exceeded
[2017-12-03 16:20:03] ERROR[9586]: config.c:1414 process_text_line: The file ‘re s_odbc_additional.conf’ was listed as a #include but it does not exist.
[2017-12-03 16:20:03] WARNING[9586]: config.c:341 ast_include_new: ‘/etc/asteris k/res_odbc_custom.conf’, line 25: Same File included more than once! This data will be saved in res_odbc_additional.conf
8 if saved back to disk.
[2017-12-03 16:20:03] WARNING[9586]: config.c:2481 ast_config_internal_load: Max imum Include level (10) exceeded
[2017-12-03 16:20:03] ERROR[9586]: config.c:1414 process_text_line: The file ‘re s_odbc_additional.conf’ was listed as a #include but it does not exist.
[2017-12-03 16:20:03] WARNING[9586]: config.c:341 ast_include_new: ‘/etc/asteris k/res_odbc.conf’, line 25: Same File included more than once! This data will be saved in res_odbc_additional.conf~~9 if saved back to disk.
[2017-12-03 16:20:03] WARNING[9586]: config.c:2481 ast_config_internal_load: Max imum Include level (10) exceeded
[2017-12-03 16:20:03] ERROR[9586]: config.c:1414 process_text_line: The file ‘re s_odbc_additional.conf’ was listed as a #include but it does not exist.
[2017-12-03 16:20:03] WARNING[9586]: res_odbc.c:785 load_odbc_config: Unable to load config file res_odbc.conf
[2017-12-03 16:20:03] WARNING[9586]: loader.c:423 load_dynamic_module: Error loa ding module ‘res_mwi_blf.so’: /usr/lib/asterisk/modules/res_mwi_blf.so: cannot o pen shared object file: No such file or directory
[2017-12-03 16:20:03] WARNING[9586]: loader.c:902 load_resource: Module ‘res_mwi _blf.so’ could not be loaded.
[2017-12-03 16:20:03] NOTICE[9586]: config.c:2355 ast_config_engine_register: Re gistered Config Engine odbc

Also, my storage space is still rapidly filling up. I think asterisk must be throwing tons of errors or something.

Errors, warnings whatever

watch - d ls -l /var/log/asterisk/

Full is increasing like crazy. Probably about a mb a second. How can I fix this? I can’t figure out anything from looking at the logs. I really don’t have the expertise to find the solution on my own. I’m willing to pay for technical support. Who would you suggest?

Link to support at the top of the page.

1 Like

Until you get your help, comment out (with an initial ; ) the defines in

/etc/asterisk/logger_logfiles_additional.conf

thusly

;full => debug,error,notice,verbose,warning
;fail2ban => notice,warning,security

then from bash

rasterisk -x ‘logger reload’ , It will stem the continued growth of your log files