DISREGARD: Your system may NOT be logging properly and you don't know it (yet)!


(TheWebMachine Networks (Sangoma Software Development Partner)) #1

So, in the last week or so, we’ve been getting reports from customers saying that their /var/log/asterisk/full log is rather…um…sparse lately. In fact, some folks’ logs are damn near empty despite processing thousands of calls a day. Most have indicated that they have not recently performed updates on their own and our AMI has general automatic updates disabled. Automatic SECURITY updates ARE enabled by default, however, and I think that’s where this came from.

The only change I could find between a system with working logs and one without was in /etc/asterisk/logger_logfiles_additional.conf. On a system with broken full log:

In a system with proper working log (I have a few test systems with all automatic updates disabled exactly for these types of comparisons):

So, I think it’s pretty easy to spot the difference(3) here. :wink: Can someone “in the know” explain where this change came from and perhaps comment on if it should actually be here?

Is anyone else noticing a substantially slimmer full log in the last week or so? It has already cost me a diagnostic opportunity on a system that crashed today, because the log had less than 50 lines added to it all day and most of them (ALL of the VERBOSE) were:
image
No other verbose call activity is being recorded on the broken systems. Since the file is autogenerated, my hands appear tied on testing this out myself since I don’t know exactly where the (3) is being added.

This appears to be hitting a large number of systems, perhaps with admins unaware that this is even the case…until they attempt to check a log when something goes amiss, only to find the log effectively empty. That’s how we’ve stumbled upon it.

Versions on a broken system

±----------------------±-----------±---------±-----------+
| Module | Version | Status | License |
±----------------------±-----------±---------±-----------+
| accountcodepreserve | 13.0.2.2 | Enabled | GPLv2 |
| amd | 15.0.2 | Enabled | GPLv3+ |
| announcement | 15.0.3.10 | Enabled | GPLv3+ |
| api | 15.0.3.7 | Enabled | AGPLv3+ |
| areminder | 15.0.14.22 | Enabled | Commercial |
| arimanager | 15.0.3.8 | Enabled | GPLv3+ |
| asterisk-cli | 14.0.1 | Enabled | GPLv3+ |
| asteriskinfo | 15.0.10 | Enabled | GPLv3+ |
| asternic_cdr | 1.6.3 | Enabled | |
| awsfpbx_instance_info | 14.0.7 | Enabled | GPLv3 |
| backup | 15.0.10.28 | Enabled | GPLv3+ |
| blacklist | 15.0.2.10 | Enabled | GPLv3+ |
| broadcast | 15.0.12 | Enabled | Commercial |
| builtin | | Enabled | |
| bulkhandler | 13.0.19 | Enabled | GPLv3+ |
| calendar | 15.0.4.19 | Enabled | GPLv3+ |
| callback | 15.0.9 | Enabled | GPLv3+ |
| callerid | 15.0.15 | Enabled | Commercial |
| callforward | 15.0.10 | Enabled | AGPLv3+ |
| calllimit | 15.0.5.5 | Enabled | Commercial |
| callrecording | 15.0.7.11 | Enabled | AGPLv3+ |
| callwaiting | 15.0.4.2 | Enabled | GPLv3+ |
| campon | 13.0.4.1 | Enabled | GPLv3+ |
| cdr | 15.0.17.2 | Enabled | GPLv3+ |
| cel | 15.0.15.10 | Enabled | GPLv3+ |
| certman | 15.0.25.3 | Enabled | AGPLv3+ |
| cidlookup | 15.0.17 | Enabled | GPLv3+ |
| conferences | 15.0.7.4 | Enabled | GPLv3+ |
| conferencespro | 15.0.3.15 | Enabled | Commercial |
| configedit | 13.0.7.1 | Enabled | AGPLv3+ |
| contactmanager | 15.0.8.23 | Enabled | GPLv3+ |
| core | 15.0.12.17 | Enabled | GPLv3+ |
| cos | 15.0.9 | Enabled | Commercial |
| customappsreg | 15.0.13 | Enabled | GPLv3+ |
| cxpanel | 15.0.4 | Enabled | GPLv3 |
| dahdiconfig | 15.0.5.4 | Enabled | GPLv3+ |
| dashboard | 15.0.5 | Enabled | AGPLv3+ |
| daynight | 15.0.11 | Enabled | GPLv3+ |
| dictate | 15.0.6 | Enabled | GPLv3+ |
| digium_phones | 15.0.2 | Enabled | GPLv2 |
| directory | 15.0.16 | Enabled | GPLv3+ |
| disa | 15.0.4.7 | Enabled | AGPLv3+ |
| donotdisturb | 15.0.6 | Enabled | GPLv3+ |
| endpoint | 15.0.27.32 | Enabled | Commercial |
| extensionroutes | 15.0.5 | Enabled | Commercial |
| fax | 15.0.19 | Enabled | GPLv3+ |
| faxpro | 15.0.8.11 | Enabled | Commercial |
| featurecodeadmin | 13.0.6.4 | Enabled | GPLv3+ |
| filestore | 15.0.3.9 | Enabled | AGPLv3 |
| findmefollow | 15.0.22 | Enabled | GPLv3+ |
| firewall | 15.0.6.20 | Enabled | AGPLv3+ |
| framework | 15.0.16.72 | Enabled | GPLv2+ |
| fw_langpacks | 14.0.1 | Enabled | GPLv3+ |
| hotelwakeup | 15.0.5.4 | Enabled | GPLv2 |
| iaxsettings | 15.0.6 | Enabled | AGPLv3 |
| infoservices | 15.0.2 | Enabled | GPLv2+ |
| irc | 13.0.1 | Enabled | GPLv3+ |
| ivr | 15.0.26 | Enabled | GPLv3+ |
| languages | 15.0.10 | Enabled | GPLv3+ |
| logfiles | 15.0.4 | Enabled | GPLv3+ |
| manager | 15.0.9 | Enabled | GPLv2+ |
| miscapps | 15.0.8 | Enabled | GPLv3+ |
| miscdests | 15.0.2.10 | Enabled | GPLv3+ |
| music | 15.0.20 | Enabled | GPLv3+ |
| outroutemsg | 15.0.9 | Enabled | GPLv3+ |
| paging | 15.0.4.23 | Enabled | GPLv3+ |
| pagingpro | 15.0.1.20 | Enabled | Commercial |
| parking | 15.0.15.1 | Enabled | GPLv3+ |
| parkpro | 15.0.12.14 | Enabled | Commercial |
| pbdirectory | 2.11.0.6 | Enabled | GPLv3+ |
| phonebook | 15.0.11 | Enabled | GPLv3+ |
| phpinfo | 13.0.2 | Enabled | GPLv2+ |
| pinsets | 15.0.1.10 | Enabled | GPLv3+ |
| pinsetspro | 15.0.3 | Enabled | Commercial |
| pm2 | 15.0.3.8 | Enabled | AGPLv3+ |
| presencestate | 15.0.9 | Enabled | GPLv3+ |
| printextensions | 13.0.3.2 | Enabled | GPLv3+ |
| queueprio | 15.0.10 | Enabled | GPLv3+ |
| queues | 15.0.21 | Enabled | GPLv2+ |
| qxact_reports | 15.0.3.24 | Enabled | Commercial |
| recording_report | 15.0.4.31 | Enabled | Commercial |
| recordings | 15.0.3.13 | Enabled | GPLv3+ |
| restapi | 13.0.21.2 | Enabled | AGPLv3 |
| restapps | 15.0.19.8 | Enabled | Commercial |
| ringgroups | 15.0.11.8 | Enabled | GPLv3+ |
| sangomacrm | 15.0.15 | Enabled | Commercial |
| setcid | 15.0.8 | Enabled | GPLv3+ |
| sipsettings | 15.0.6.27 | Enabled | AGPLv3+ |
| sipstation | 15.0.5.8 | Enabled | Commercial |
| sms | 15.0.10 | Enabled | Commercial |
| sng_mcu | 13.0.5 | Disabled | Commercial |
| soundlang | 15.0.5.7 | Enabled | GPLv3+ |
| speeddial | 2.11.0.4 | Enabled | GPLv3+ |
| superfecta | 15.0.2.23 | Enabled | GPLv2+ |
| sysadmin | 15.0.16.4 | Enabled | Commercial |
| timeconditions | 15.0.15.3 | Enabled | GPLv3+ |
| tts | 15.0.9 | Enabled | GPLv3+ |
| ttsengines | 15.0.4.6 | Enabled | AGPLv3 |
| ucp | 15.0.6.17 | Enabled | AGPLv3+ |
| userman | 15.0.28 | Enabled | AGPLv3+ |
| vmblast | 15.0.11.4 | Enabled | GPLv3+ |
| vmnotify | 15.0.11 | Enabled | Commercial |
| voicemail | 15.0.18.15 | Enabled | GPLv3+ |
| voicemail_report | 15.0.5 | Enabled | Commercial |
| vqplus | 15.0.7.15 | Enabled | Commercial |
| weakpasswords | 13.0.2 | Enabled | GPLv3+ |
| webcallback | 15.0.6.2 | Enabled | Commercial |
| webrtc | 15.0.8 | Enabled | GPLv3+ |
| xmpp | 15.0.6.4 | Enabled | AGPLv3 |
| zulu | 15.0.58.9 | Enabled | Commercial |
±----------------------±-----------±---------±-----------+


(Avayax) #2

(TheWebMachine Networks (Sangoma Software Development Partner)) #3

Ugh…I should have held my tongue. I’ve been out for a week from surgery and was still catching up on recent posts. :man_facepalming: Maybe I need another week off! haha