Freepbx gui stops responding, all calls drop, console remains operable

FreePBX 2.11.0.38
PBX Firmware: 5.211.65-15
PBX Service Pack: 1.0.0.0

Happened twice in 2 days, not sure where to start troubleshooting. 7 outside calls, 13 internal when this happened.

uptime? top? Something useful please?

1 day 7 hours uptime, 1 % CPU usage at the time, a page went out from the security desk and I see there have been 98 destinations added in the paging group - bet I need to decrease this significantly.

I would check iptables at the time, I have a feeling fail2ban banned your internal network.

Well, I kind of thought the same thing, but I turned off the intrusion detection in system admin pro because we had been troubleshooting some phones, and it remained off until the outage.

2014-08-19 13:42:37,460 fail2ban.server : INFO Changed logging target to /var/ log/fail2ban.log for Fail2ban v0.8.8
2014-08-19 13:42:37,460 fail2ban.jail : INFO Creating new jail 'recidive’
2014-08-19 13:42:37,489 fail2ban.jail : INFO Jail ‘recidive’ uses Gamin
2014-08-19 13:42:37,552 fail2ban.jail : INFO Initiated ‘gamin’ backend
2014-08-19 13:42:37,552 fail2ban.filter : INFO Added logfile = /var/log/fail2b an.log
2014-08-19 13:42:37,553 fail2ban.filter : INFO Set maxRetry = 20
2014-08-19 13:42:37,556 fail2ban.filter : INFO Set findtime = 86400
2014-08-19 13:42:37,556 fail2ban.actions: INFO Set banTime = 604800
2014-08-19 13:42:37,563 fail2ban.jail : INFO Creating new jail 'ssh-iptables '
2014-08-19 13:42:37,563 fail2ban.jail : INFO Jail ‘ssh-iptables’ uses Gamin
2014-08-19 13:42:37,563 fail2ban.jail : INFO Initiated ‘gamin’ backend
2014-08-19 13:42:37,585 fail2ban.filter : INFO Added logfile = /var/log/secure
2014-08-19 13:42:37,586 fail2ban.filter : INFO Set maxRetry = 8
2014-08-19 13:42:37,589 fail2ban.filter : INFO Set findtime = 600
2014-08-19 13:42:37,589 fail2ban.actions: INFO Set banTime = 1800
2014-08-19 13:42:37,650 fail2ban.jail : INFO Creating new jail 'apache-badbo ts’
2014-08-19 13:42:37,650 fail2ban.jail : INFO Jail ‘apache-badbots’ uses Gami n
2014-08-19 13:42:37,651 fail2ban.jail : INFO Initiated ‘gamin’ backend
2014-08-19 13:42:37,669 fail2ban.filter : INFO Added logfile = /var/log/httpd/ access_log
2014-08-19 13:42:37,670 fail2ban.filter : INFO Set maxRetry = 8
2014-08-19 13:42:37,673 fail2ban.filter : INFO Set findtime = 600
2014-08-19 13:42:37,673 fail2ban.actions: INFO Set banTime = 1800
2014-08-19 13:42:37,689 fail2ban.jail : INFO Creating new jail 'pbx-gui’
2014-08-19 13:42:37,690 fail2ban.jail : INFO Jail ‘pbx-gui’ uses Gamin
2014-08-19 13:42:37,690 fail2ban.jail : INFO Initiated ‘gamin’ backend
2014-08-19 13:42:37,690 fail2ban.filter : INFO Set maxRetry = 8
2014-08-19 13:42:37,694 fail2ban.filter : INFO Set findtime = 600
2014-08-19 13:42:37,694 fail2ban.actions: INFO Set banTime = 1800
2014-08-19 13:42:37,699 fail2ban.jail : INFO Creating new jail 'asterisk-ipt ables’
2014-08-19 13:42:37,699 fail2ban.jail : INFO Jail ‘asterisk-iptables’ uses G amin
2014-08-19 13:42:37,700 fail2ban.jail : INFO Initiated ‘gamin’ backend
2014-08-19 13:42:37,713 fail2ban.filter : INFO Added logfile = /var/log/asteri sk/fail2ban
2014-08-19 13:42:37,713 fail2ban.filter : INFO Set maxRetry = 8
2014-08-19 13:42:37,716 fail2ban.filter : INFO Set findtime = 600
2014-08-19 13:42:37,716 fail2ban.actions: INFO Set banTime = 1800
2014-08-19 13:42:37,745 fail2ban.jail : INFO Creating new jail 'apache-tcpwr apper’
2014-08-19 13:42:37,745 fail2ban.jail : INFO Jail ‘apache-tcpwrapper’ uses G amin
2014-08-19 13:42:37,745 fail2ban.jail : INFO Initiated ‘gamin’ backend
2014-08-19 13:42:37,777 fail2ban.filter : INFO Added logfile = /var/log/httpd/ error_log
2014-08-19 13:42:37,777 fail2ban.filter : INFO Set maxRetry = 8
2014-08-19 13:42:37,781 fail2ban.filter : INFO Set findtime = 600
2014-08-19 13:42:37,781 fail2ban.actions: INFO Set banTime = 1800
2014-08-19 13:42:37,788 fail2ban.jail : INFO Creating new jail 'vsftpd-iptab les’
2014-08-19 13:42:37,788 fail2ban.jail : INFO Jail ‘vsftpd-iptables’ uses Gam in
2014-08-19 13:42:37,788 fail2ban.jail : INFO Initiated ‘gamin’ backend
2014-08-19 13:42:37,789 fail2ban.filter : INFO Set maxRetry = 8
2014-08-19 13:42:37,792 fail2ban.filter : INFO Set findtime = 600
2014-08-19 13:42:37,792 fail2ban.actions: INFO Set banTime = 1800
2014-08-19 13:42:37,801 fail2ban.jail : INFO Jail ‘recidive’ started
2014-08-19 13:42:37,802 fail2ban.jail : INFO Jail ‘ssh-iptables’ started
2014-08-19 13:42:37,806 fail2ban.jail : INFO Jail ‘apache-badbots’ started
2014-08-19 13:42:37,812 fail2ban.jail : INFO Jail ‘pbx-gui’ started
2014-08-19 13:42:37,816 fail2ban.jail : INFO Jail ‘asterisk-iptables’ starte d
2014-08-19 13:42:37,819 fail2ban.jail : INFO Jail ‘apache-tcpwrapper’ starte d
2014-08-19 13:42:37,830 fail2ban.jail : INFO Jail ‘vsftpd-iptables’ started
2014-08-19 13:42:51,865 fail2ban.server : INFO Stopping all jails
2014-08-19 13:42:51,916 fail2ban.actions.action: ERROR iptables -D INPUT -p tcp -j fail2ban-PBX-GUI
iptables -F fail2ban-PBX-GUI
iptables -X fail2ban-PBX-GUI returned 100
2014-08-19 13:42:52,690 fail2ban.jail : INFO Jail ‘apache-tcpwrapper’ stoppe d
2014-08-19 13:42:53,667 fail2ban.jail : INFO Jail ‘recidive’ stopped
2014-08-19 13:42:53,918 fail2ban.jail : INFO Jail ‘pbx-gui’ stopped
2014-08-19 13:42:54,920 fail2ban.jail : INFO Jail ‘apache-badbots’ stopped
2014-08-19 13:42:55,918 fail2ban.jail : INFO Jail ‘ssh-iptables’ stopped
2014-08-19 13:42:56,920 fail2ban.jail : INFO Jail ‘asterisk-iptables’ stoppe d
2014-08-19 13:42:57,699 fail2ban.jail : INFO Jail ‘vsftpd-iptables’ stopped
2014-08-19 13:42:57,727 fail2ban.server : INFO Exiting Fail2ban
2014-08-19 13:42:58,888 fail2ban.server : INFO Changed logging target to /var/ log/fail2ban.log for Fail2ban v0.8.8
2014-08-19 13:42:58,889 fail2ban.jail : INFO Creating new jail 'recidive’
2014-08-19 13:42:58,889 fail2ban.jail : INFO Jail ‘recidive’ uses Gamin
2014-08-19 13:42:58,897 fail2ban.jail : INFO Initiated ‘gamin’ backend
2014-08-19 13:42:58,898 fail2ban.filter : INFO Added logfile = /var/log/fail2b an.log
2014-08-19 13:42:58,898 fail2ban.filter : INFO Set maxRetry = 20
2014-08-19 13:42:58,901 fail2ban.filter : INFO Set findtime = 86400
2014-08-19 13:42:58,902 fail2ban.actions: INFO Set banTime = 604800
2014-08-19 13:42:58,908 fail2ban.jail : INFO Creating new jail 'ssh-iptables '
2014-08-19 13:42:58,908 fail2ban.jail : INFO Jail ‘ssh-iptables’ uses Gamin
2014-08-19 13:42:58,908 fail2ban.jail : INFO Initiated ‘gamin’ backend
2014-08-19 13:42:58,908 fail2ban.filter : INFO Added logfile = /var/log/secure
2014-08-19 13:42:58,909 fail2ban.filter : INFO Set maxRetry = 8
2014-08-19 13:42:58,912 fail2ban.filter : INFO Set findtime = 600
2014-08-19 13:42:58,912 fail2ban.actions: INFO Set banTime = 1800
2014-08-19 13:42:58,971 fail2ban.jail : INFO Creating new jail 'apache-badbo ts’
2014-08-19 13:42:58,971 fail2ban.jail : INFO Jail ‘apache-badbots’ uses Gami n
2014-08-19 13:42:58,971 fail2ban.jail : INFO Initiated ‘gamin’ backend
2014-08-19 13:42:58,972 fail2ban.filter : INFO Added logfile = /var/log/httpd/ access_log
2014-08-19 13:42:58,972 fail2ban.filter : INFO Set maxRetry = 8
2014-08-19 13:42:58,975 fail2ban.filter : INFO Set findtime = 600
2014-08-19 13:42:58,976 fail2ban.actions: INFO Set banTime = 1800
2014-08-19 13:42:58,991 fail2ban.jail : INFO Creating new jail 'pbx-gui’
2014-08-19 13:42:58,991 fail2ban.jail : INFO Jail ‘pbx-gui’ uses Gamin
2014-08-19 13:42:58,991 fail2ban.jail : INFO Initiated ‘gamin’ backend
2014-08-19 13:42:58,992 fail2ban.filter : INFO Set maxRetry = 8
2014-08-19 13:42:58,995 fail2ban.filter : INFO Set findtime = 600
2014-08-19 13:42:58,995 fail2ban.actions: INFO Set banTime = 1800
2014-08-19 13:42:59,000 fail2ban.jail : INFO Creating new jail 'asterisk-ipt ables’
2014-08-19 13:42:59,000 fail2ban.jail : INFO Jail ‘asterisk-iptables’ uses G amin
2014-08-19 13:42:59,001 fail2ban.jail : INFO Initiated ‘gamin’ backend
2014-08-19 13:42:59,001 fail2ban.filter : INFO Added logfile = /var/log/asteri sk/fail2ban
2014-08-19 13:42:59,002 fail2ban.filter : INFO Set maxRetry = 8
2014-08-19 13:42:59,005 fail2ban.filter : INFO Set findtime = 600
2014-08-19 13:42:59,005 fail2ban.actions: INFO Set banTime = 1800
2014-08-19 13:42:59,032 fail2ban.jail : INFO Creating new jail 'apache-tcpwr apper’
2014-08-19 13:42:59,032 fail2ban.jail : INFO Jail ‘apache-tcpwrapper’ uses G amin
2014-08-19 13:42:59,033 fail2ban.jail : INFO Initiated ‘gamin’ backend
2014-08-19 13:42:59,033 fail2ban.filter : INFO Added logfile = /var/log/httpd/ error_log
2014-08-19 13:42:59,034 fail2ban.filter : INFO Set maxRetry = 8
2014-08-19 13:42:59,037 fail2ban.filter : INFO Set findtime = 600
2014-08-19 13:42:59,037 fail2ban.actions: INFO Set banTime = 1800
2014-08-19 13:42:59,043 fail2ban.jail : INFO Creating new jail 'vsftpd-iptab les’
2014-08-19 13:42:59,043 fail2ban.jail : INFO Jail ‘vsftpd-iptables’ uses Gam in
2014-08-19 13:42:59,044 fail2ban.jail : INFO Initiated ‘gamin’ backend
2014-08-19 13:42:59,044 fail2ban.filter : INFO Set maxRetry = 8
2014-08-19 13:42:59,047 fail2ban.filter : INFO Set findtime = 600
2014-08-19 13:42:59,047 fail2ban.actions: INFO Set banTime = 1800
2014-08-19 13:42:59,054 fail2ban.jail : INFO Jail ‘recidive’ started
2014-08-19 13:42:59,056 fail2ban.jail : INFO Jail ‘ssh-iptables’ started
2014-08-19 13:42:59,063 fail2ban.jail : INFO Jail ‘apache-badbots’ started
2014-08-19 13:42:59,072 fail2ban.jail : INFO Jail ‘pbx-gui’ started
2014-08-19 13:42:59,127 fail2ban.jail : INFO Jail ‘asterisk-iptables’ starte d
2014-08-19 13:42:59,134 fail2ban.jail : INFO Jail ‘apache-tcpwrapper’ starte d
2014-08-19 13:42:59,141 fail2ban.jail : INFO Jail ‘vsftpd-iptables’ started
2014-08-19 13:43:18,165 fail2ban.actions: WARNING [asterisk-iptables] Ban 200.20 0.7.88
2014-08-19 13:43:18,344 fail2ban.actions.action: ERROR printf %b “Subject: [Fai l2Ban] SIP: banned 200.200.7.88 on hostname -f
Date: date -u +"%a, %d %h %Y %T +0000"
From: Fail2Ban <>
To: [email protected]\n
Hi,\n
The IP 200.200.7.88 has just been banned by Fail2Ban after
20 attempts against SIP on hostname -f.\n
Regards,\n
Fail2Ban” | /usr/sbin/sendmail -f [email protected] returned 4b00
2014-08-19 13:43:18,344 fail2ban.actions: WARNING [asterisk-iptables] Ban 200.20 0.9.227
2014-08-19 13:43:18,358 fail2ban.actions.action: ERROR printf %b “Subject: [Fai l2Ban] SIP: banned 200.200.9.227 on hostname -f
Date: date -u +"%a, %d %h %Y %T +0000"
From: Fail2Ban <>
To: [email protected]\n
Hi,\n
The IP 200.200.9.227 has just been banned by Fail2Ban after
16 attempts against SIP on hostname -f.\n
Regards,\n
Fail2Ban” | /usr/sbin/sendmail -f [email protected] returned 4b00
2014-08-19 13:43:18,359 fail2ban.actions: WARNING [asterisk-iptables] Ban 200.20 0.8.192
2014-08-19 13:43:18,373 fail2ban.actions.action: ERROR printf %b “Subject: [Fai l2Ban] SIP: banned 200.200.8.192 on hostname -f
Date: date -u +"%a, %d %h %Y %T +0000"
From: Fail2Ban <>
To: [email protected]\n
Hi,\n
The IP 200.200.8.192 has just been banned by Fail2Ban after
17 attempts against SIP on hostname -f.\n
Regards,\n
Fail2Ban” | /usr/sbin/sendmail -f [email protected] returned 4b00
2014-08-19 14:13:18,310 fail2ban.actions: WARNING [asterisk-iptables] Unban 200. 200.7.88
2014-08-19 14:13:19,316 fail2ban.actions: WARNING [asterisk-iptables] Unban 200. 200.9.227
2014-08-19 14:13:19,321 fail2ban.actions: WARNING [asterisk-iptables] Unban 200. 200.8.192
2014-08-19 14:15:23,447 fail2ban.actions: WARNING [asterisk-iptables] Ban 200.20 0.7.88
2014-08-19 14:15:23,462 fail2ban.actions.action: ERROR printf %b “Subject: [Fai l2Ban] SIP: banned 200.200.7.88 on hostname -f
Date: date -u +"%a, %d %h %Y %T +0000"
From: Fail2Ban <>
To: [email protected]\n
Hi,\n
The IP 200.200.7.88 has just been banned by Fail2Ban after
8 attempts against SIP on hostname -f.\n
Regards,\n
Fail2Ban” | /usr/sbin/sendmail -f [email protected] returned 4b00
2014-08-19 14:15:59,497 fail2ban.actions: WARNING [asterisk-iptables] Ban 200.20 0.8.192
2014-08-19 14:15:59,512 fail2ban.actions.action: ERROR printf %b “Subject: [Fai l2Ban] SIP: banned 200.200.8.192 on hostname -f
Date: date -u +"%a, %d %h %Y %T +0000"
From: Fail2Ban <>
To: [email protected]\n
Hi,\n
The IP 200.200.8.192 has just been banned by Fail2Ban after
8 attempts against SIP on hostname -f.\n
Regards,\n
Fail2Ban” | /usr/sbin/sendmail -f [email protected] returned 4b00
2014-08-19 14:16:12,525 fail2ban.actions: WARNING [asterisk-iptables] Ban 200.20 0.9.227
2014-08-19 14:16:12,544 fail2ban.actions.action: ERROR printf %b “Subject: [Fai l2Ban] SIP: banned 200.200.9.227 on hostname -f
Date: date -u +"%a, %d %h %Y %T +0000"
From: Fail2Ban <>
To: [email protected]\n
Hi,\n
The IP 200.200.9.227 has just been banned by Fail2Ban after
8 attempts against SIP on hostname -f.\n
Regards,\n
Fail2Ban” | /usr/sbin/sendmail -f [email protected] returned 4b00
2014-08-19 14:38:14,803 fail2ban.server : INFO Stopping all jails
2014-08-19 14:38:15,598 fail2ban.actions.action: ERROR iptables -D INPUT -p tcp -j fail2ban-PBX-GUI
iptables -F fail2ban-PBX-GUI
iptables -X fail2ban-PBX-GUI returned 100
2014-08-19 14:38:15,600 fail2ban.jail : INFO Jail ‘apache-tcpwrapper’ stoppe d
2014-08-19 14:38:16,568 fail2ban.jail : INFO Jail ‘recidive’ stopped
2014-08-19 14:38:17,541 fail2ban.jail : INFO Jail ‘pbx-gui’ stopped
2014-08-19 14:38:18,457 fail2ban.jail : INFO Jail ‘apache-badbots’ stopped
2014-08-19 14:38:18,618 fail2ban.jail : INFO Jail ‘ssh-iptables’ stopped
2014-08-19 14:38:18,896 fail2ban.actions: WARNING [asterisk-iptables] Unban 200. 200.7.88
2014-08-19 14:38:18,901 fail2ban.actions: WARNING [asterisk-iptables] Unban 200. 200.8.192
2014-08-19 14:38:18,905 fail2ban.actions: WARNING [asterisk-iptables] Unban 200. 200.9.227
2014-08-19 14:38:18,915 fail2ban.jail : INFO Jail ‘asterisk-iptables’ stoppe d
2014-08-19 14:38:19,617 fail2ban.jail : INFO Jail ‘vsftpd-iptables’ stopped
2014-08-19 14:38:19,617 fail2ban.server : INFO Exiting Fail2ban