High cpu usage issues

[bksales]

Not sure whats going on here but this server just keeps using more and more cpu until it maxes out. fwconsole restart resets it but it immediately starts climbing again. Looking at the logs the two that are continually writing stuff are the fail2ban.log-date and the freepbx.log

PBX Version: 15.0.29
PBX Distro: 12.7.8-2203-2.sng7
Asterisk Version: 18.9

in the fail2ban log it looks like it’s readding all the white listed IPs to the white list again. I see the Vitelity and Voip Innovations IPs in there every second, though that log isnt growing as fast as the other one.

In the freepbx.log I see these three messages repeating every

[2023-02-07 10:59:30] [freepbx.INFO]: Deprecated way to add Console commands for module api, adding console commands this way can have negative performance impacts. Please use module.xml. See: Sangoma Documentation

[2023-02-07 10:59:30] [freepbx.INFO]: Deprecated way to add Console commands for module backup, adding console commands this way can have negative performance impacts. Please use module.xml. See: Sangoma Documentation

[2023-02-07 10:59:30] [freepbx.INFO]: Deprecated way to add Console commands for module voicemail, adding console commands this way can have negative performance impacts. Please use module.xml. See: Sangoma Documentation

I’ve never had to go manually add the fwconsole commands, and they already work so I dont think that’s actually what needs to happen. Hoping someone has a suggestion.

[bksales]

Taking a few screen shots of the output of top

[dicko]

My guess is that fail2ban is crashing and the version you are using does not have ‘persistance’ over a restart.

try disabling the fail2ban service and then starting it in the foreground.

[bksales]

I forced it to stop and then started it again, will see things stay quiet.

[bksales]

Well that didnt last

[bksales]

I see some of these lines repeating in the cron log

sysadmin-hook[13310]: File ‘/var/spool/asterisk/incron/firewall.dynamic-jails.CONTENTS’ didn’t exist. That’s unpossible (Are you running incron commands manually?)

[dicko]

Unfortunately you are in closed source’ territory with this, only Sangoma can help.

[bksales]

Anyone from Sangoma have a suggestion on where to look? I found a second server doing this. Seeing a bunch of these in the cron log if it helps.

Feb 21 14:15:30 incrond[659]: (system::sysadmin) CMD (/usr/bin/sysadmin_manager sysadmin.fail2ban-getbanned)
Feb 21 14:15:31 sysadmin-hook[6197]: sysadmin hook started - [“/usr/bin/sysadmin_manager”,“firewall.dynamic-jails.CONTENTS”]
Feb 21 14:15:31 sysadmin-hook[6197]: File ‘/var/spool/asterisk/incron/firewall.dynamic-jails.CONTENTS’ didn’t exist. That’s unpossible (Are you running incron commands manually?)
Feb 21 14:15:31 sysadmin-hook[6198]: sysadmin hook started - [“/usr/bin/sysadmin_manager”,“firewall.dynamic-jails.CONTENTS”]
Feb 21 14:15:31 sysadmin-hook[6198]: File ‘/var/spool/asterisk/incron/firewall.dynamic-jails.CONTENTS’ didn’t exist. That’s unpossible (Are you running incron commands manually?)
Feb 21 14:15:31 sysadmin-hook[6197]: Security check passed. Running '/var/www/html/admin/modules/firewall/hooks/dynamic-jails ’
Feb 21 14:15:31 sysadmin-hook[6198]: Security check passed. Running '/var/www/html/admin/modules/firewall/hooks/dynamic-jails ’
Feb 21 14:15:31 sysadmin-hook[6216]: sysadmin hook started - [“/usr/bin/sysadmin_manager”,“sysadmin.fail2ban-getbanned”]
Feb 21 14:15:31 sysadmin-hook[6216]: Security check passed. Running '/var/www/html/admin/modules/sysadmin/hooks/fail2ban-getbanned ’
Feb 21 14:15:32 incrond[659]: (system::sysadmin) CMD (/usr/bin/sysadmin_manager firewall.get-dynamic-ignoreip)
Feb 21 14:15:33 sysadmin-hook[6332]: sysadmin hook started - [“/usr/bin/sysadmin_manager”,“firewall.get-dynamic-ignoreip”]
Feb 21 14:15:33 sysadmin-hook[6332]: Security check passed. Running '/var/www/html/admin/modules/firewall/hooks/get-dynamic-ignoreip ’
Feb 21 14:15:42 incrond[659]: (system::sysadmin) CMD (/usr/bin/sysadmin_manager firewall.jails-integrity)
Feb 21 14:15:42 sysadmin-hook[7018]: sysadmin hook started - [“/usr/bin/sysadmin_manager”,“firewall.jails-integrity”]
Feb 21 14:15:43 sysadmin-hook[7018]: Security check passed. Running '/var/www/html/admin/modules/firewall/hooks/jails-integrity ’

[dicko]

Anything sysadmin related is ‘closed source’ so only Sangoma can help you here. File a report there.