Fail2Ban not working anymore on HA system

Commercial Modules
1

I noticed that Fail2ban isn’t blocking any IP addresses anymore, but it should be.
E.g., as a test, I just tried to SSH into the server with wrong password, but Fail2ban isn’t jumping in. With failed SIP registration attempts it’s the same. Max retries set to 6.
FreePBX Distro 10.13.66

/var/secure:
Oct 19 10:13:33 freepbx-a sshd[63438]: Failed password for root from 192.168.1.254 port 58210 ssh2
Oct 19 10:13:37 freepbx-a sshd[63438]: Failed password for root from 192.168.1.254 port 58210 ssh2
Oct 19 10:13:40 freepbx-a sshd[63438]: Failed password for root from 192.168.1.254 port 58210 ssh2
Oct 19 10:13:44 freepbx-a sshd[63438]: Failed password for root from 192.168.1.254 port 58210 ssh2
Oct 19 10:13:48 freepbx-a sshd[63438]: Failed password for root from 192.168.1.254 port 58210 ssh2
Oct 19 10:13:50 freepbx-a sshd[63438]: Failed password for root from 192.168.1.254 port 58210 ssh2
Oct 19 10:13:50 freepbx-a sshd[63439]: Disconnecting: Too many authentication failures for root
Oct 19 10:13:50 freepbx-a sshd[63438]: PAM 5 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.1.254 user=root
Oct 19 10:13:50 freepbx-a sshd[63438]: PAM service(sshd) ignoring max retries; 6 > 3
Oct 19 10:14:10 freepbx-a sshd[64031]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.1.254 user=root

2

I’m guessing there are multiple instances of the server running on the master server of my HA setup, as well as clients trying to stop and restart fail2ban which are locked waiting to complete:

[root@freepbx-a asterisk]# ps aux | fgrep fail2ban
root 21522 0.0 0.0 108428 1800 ? S Sep06 0:00 /bin/sh /sbin/service fail2ban stop
root 21527 0.0 0.0 108432 1812 ? S Sep06 0:00 /bin/bash /etc/init.d/fail2ban stop
root 21530 0.0 0.0 168244 6184 ? S Sep06 0:00 /usr/bin/python /usr/bin/fail2ban-client stop
root 43635 0.0 0.0 100948 732 pts/0 S+ 17:37 0:00 fgrep fail2ban
root 88065 0.0 0.0 323520 14388 ? S 10:36 0:00 /usr/bin/php /usr/bin/sysadmin_manager sysadmin.fail2ban-start
root 88067 0.0 0.0 108164 1440 ? S 10:36 0:00 /bin/bash /var/www/html/admin/modules/sysadmin/hooks/fail2ban-start
root 88105 0.0 0.0 108428 1804 ? S 10:36 0:00 /bin/sh /sbin/service fail2ban restart
root 88110 0.0 0.0 108432 1808 ? S 10:36 0:00 /bin/bash /etc/init.d/fail2ban restart
root 88113 0.0 0.0 168248 7240 ? S 10:36 0:00 /usr/bin/python /usr/bin/fail2ban-client stop
root 94922 0.0 0.0 108164 1272 ? S Aug10 0:00 /bin/bash /var/www/html/admin/modules/sysadmin/hooks/fail2ban-start
root 94960 0.0 0.0 108428 1412 ? S Aug10 0:00 /bin/sh /sbin/service fail2ban restart
root 94965 0.0 0.0 108432 1400 ? S Aug10 0:00 /bin/bash /etc/init.d/fail2ban restart
root 94969 0.0 0.0 168864 2976 ? S Aug10 0:00 /usr/bin/python /usr/bin/fail2ban-client -x start
root 94970 0.0 0.0 0 0 ? Z Aug10 0:00 [fail2ban-server]
root 94972 0.0 0.0 188608 2556 ? S Aug10 0:00 /usr/bin/python /usr/bin/fail2ban-server -b -s /var/run/fail2ban/fail2ban.sock -p /var/run/fail2ban/fail2ban.pid -x
root 110935 0.0 0.0 108428 1800 ? S Sep08 0:00 /bin/sh /sbin/service fail2ban stop
root 110940 0.0 0.0 108432 1812 ? S Sep08 0:00 /bin/bash /etc/init.d/fail2ban stop
root 110943 0.0 0.0 168248 6184 ? S Sep08 0:00 /usr/bin/python /usr/bin/fail2ban-client stop
root 114855 0.0 0.0 188564 1720 ? S Jun01 0:23 /usr/bin/python /usr/bin/fail2ban-server -b -s /var/run/fail2ban/fail2ban.sock -p /var/run/fail2ban/fail2ban.pid -x
root 129292 0.0 0.0 108428 1620 ? S Aug31 0:00 /bin/sh /sbin/service fail2ban stop
root 129297 0.0 0.0 108432 1576 ? S Aug31 0:00 /bin/bash /etc/init.d/fail2ban stop
root 129300 0.0 0.0 168248 4212 ? S Aug31 0:00 /usr/bin/python /usr/bin/fail2ban-client stop
root 136972 0.0 0.0 108428 1800 ? S Oct15 0:00 /bin/sh /sbin/service fail2ban stop
root 136977 0.0 0.0 108432 1812 ? S Oct15 0:00 /bin/bash /etc/init.d/fail2ban stop
root 136980 0.0 0.0 168252 7228 ? S Oct15 0:00 /usr/bin/python /usr/bin/fail2ban-client stop

The oldest of these dates back to Aug 10 when the fail2ban logs stop. service stop wouldn’t work, and fail2ban-client commands also got stuck. I killed all the processes and restarted, and it now seems to be back up.

What was going on?