Logrotate Bug?

[Albatros]

Hello everybody,
While I was carrying out maintenance operations on my FreePBX server, I noticed that some log files are no longer rotated since April.
Could it be due to a bug introduced by some update?
Below is a screenshot showing the full file as an example.

As you can see, on April 18, the rotation ends. Therefore, if I open the current full file, which has grown to 566MB in the meantime, the first line is dated April 18th as the last file rotated.

I have also checked the configuration files in /etc/logrotate.d/asterisk and the configuration seems correct to me. Here is how the configuration is set:

/ var / log / asterisk / queue_log {
daily
missingok
rotate 30
notifempty
sharedscripts
create 0640 asterisk asterisk
on asterisk asterisk
}

/ var / spool / mail / asterisk {
daily
missingok
rotate 7
notifempty
sharedscripts
create 0660 asterisk mail
on asterisk mail
}

/ var / log / asterisk / messages
/ var / log / asterisk / event_log
/ var / log / asterisk / full
/ var / log / asterisk / dtmf
/ var / log / asterisk / fail2ban {
daily
missingok
rotate 7
notifempty
sharedscripts
create 0640 asterisk asterisk
on asterisk asterisk
postrotate
/ usr / sbin / asterisk -rx ‘logger reload’> / dev / null 2> / dev / null
endscript
}
#This comment is to fix rpm file replacing
#Config file built on Fri Dec 11 09:16:15 UTC 2020

I ask you for help to understand what is happening. This anomaly, besides being annoying, makes system maintenance much more difficult.

Thank you so much.

[wmoon (Walter Moon)]

Can you give us the output of:
fwconsole ma list

and the output of:
/usr/sbin/logrotate -s /var/lib/logrotate/logrotate.status /etc/logrotate.conf -d
The info for /var/log/asterisk/full in that output might give some clues.

You might also want to try this if you see any permissions related complaints.
fwconsole chown

[dakudos]

@Albatros,

I suspect I know what your issue is… I’ve noticed this sporadically on systems, I’d look at the processes and would notice a hung logrotate due to pm2 reloadLogs. Like this :

asterisk 16054 16054     1  0 Jul06 ?        00:00:00 node /var/www/html/admin/modules/pm2/node/node_modules/pm2/bin/pm2 reloadLogs
asterisk 16054 16055     1  0 Jul06 ?        00:00:00   |_ node /var/www/html/admin/modules/pm2/node/node_modules/pm2/bin/pm2 reloadLogs
asterisk 16054 16056     1  0 Jul06 ?        00:00:00   |_ node /var/www/html/admin/modules/pm2/node/node_modules/pm2/bin/pm2 reloadLogs
asterisk 16054 16057     1  0 Jul06 ?        00:00:00   |_ node /var/www/html/admin/modules/pm2/node/node_modules/pm2/bin/pm2 reloadLogs
asterisk 16054 16058     1  0 Jul06 ?        00:00:00   |_ node /var/www/html/admin/modules/pm2/node/node_modules/pm2/bin/pm2 reloadLogs
asterisk 16054 16059     1  0 Jul06 ?        00:00:00   |_ node /var/www/html/admin/modules/pm2/node/node_modules/pm2/bin/pm2 reloadLogs
root     13938 13938     1  0 Jul06 ?        00:00:00 /usr/sbin/anacron -s
root     16014 16014 13938  0 Jul06 ?        00:00:00   |_/bin/bash /bin/run-parts /etc/cron.daily
root     16019 16019 16014  0 Jul06 ?        00:00:00       |_awk -v progname=/etc/cron.daily/logrotate progname { ????   print progname ":\n" ????   progname=""; ???       } ???       { print; }

In the example above, I’d kill 16054 and then awk -v progname=/etc/cron.daily/logrotate progname ... would continue like normal.

[sorvani (Jared Busch)]

Just got a call about poor performance form a client…

Look what I found.

killed pm2 and everything resolved itself.

[comtech]

What version of FreePBX is this box?

[sorvani (Jared Busch)]

Mine is 15. Last updated June 30. And then today. At least by my normal script. If I ran something manually I have not had time to check.

[sorvani (Jared Busch)]

@wmoon

/usr/sbin/logrotate -s /var/lib/logrotate/logrotate.status /etc/logrotate.conf -d

Actually seems there is something else also happening, so the pm2 kill might be unrelated to the log in my case.

I just ran logrotate manually via sudo /usr/sbin/logrotate -s /var/lib/logrotate/logrotate.status /etc/logrotate.conf -d

That is very obviously not rotated.

[comtech]

Having this same issue.

/dev/mapper/SangomaVG-root is maxed now.

Can I delete the full log without causing an issue?

[dicko]

better to truncate the log files rather than to delete them

echo : > /var/log/asterisk/full

you can call lograte with the -v (verbose) and -f (force) single instances logrotate -v -f /etc/logrotate.d/asterisk to better see what is going on here

[RealRuler2112 (Ruler2112)]

I remember having something like this on my PBXact 13 box once about 18 months ago! In my case, I started getting complaints of poor call quality. Found that the process was running away, consuming 100% of the CPU time and making audio sound like absolute crap. Killed the process and the problems disappeared. Hasn’t happened since.

edit - I didn’t notice if the logs hadn’t rotated or not, but DO remember the name of the process was pm2… the same as what @dakudos posted malfunctioned on his box.

[dicko]

Perhaps all three instances

/etc/logrotate.d/freepbx-core
/etc/logrotate.d/freepbx-ucp
/etc/logrotate.d/freepbx-xmpp

trying to /usr/sbin/fwconsole pm2 --reload-logs -q possibly close to simultaneously , might clog things up

time /usr/sbin/fwconsole pm2 --reload-logs

would show how long that process takes, it’s probably not a good thing to try to do that if another instance is running.

[sorvani (Jared Busch)]

I decided to let it run and see what happened this morning. Not surprised, it is all better.