No Calls Till The System Gets A Reboot

x201_F12 · March 22, 2024, 4:05pm

For a week now, my calls outbound will get “all circuits are busy” and inbound says “this number is not in service”.
In the past it has been a Telnyx problem and I work with them to figure it out. However, starting last Friday the phones were down and I was waiting for Telnyx to get back to me (they never did) and I thought what the hell I will reboot the phone system (PBXact 40). I rebooted and the system was up. Now this is happening 2 to 3 times a day. Every time a reboot fixes the connection problem.

Does anyone have any advise on how to troubleshoot this? I am really at a loss on this one, I just don’t know how to see if it is the SIP provider (Telnyx) or the pbx (PBXact 40) problem. Or even why a reboot of the system is fixing this.

]

Stewart1 · March 22, 2024, 4:29pm

Have you tried
fwconsole reload
or
fwconsole restart
?

Router/firewall make/model? What VoIP related settings in it?

What errors, if any, appear in the Asterisk log on failing registration attempts?

x201_F12 · March 22, 2024, 4:52pm

No I have not tried

fwconsole reload
or
fwconsole restart

Firewall = Netgate 7100

The next time it goes down, I will look in the Asterisk logs before I reboot it.

And the next time , I should

fwconsole reload
or
fwconsole restart

instead of the web UI → system Admin → power options → Reboot

?

This has been running very well for about 3 years, no changes have been made.

Stewart1 · March 22, 2024, 6:00pm

What does the Asterisk log show prior to the reboot (that should still be present)? Anything obvious in /var/log/messages?

x201_F12 · March 22, 2024, 6:59pm

going into /var/log/messages I find this…

I found the system this morning at 6:28 not working so I rebooted.
This is what the logs say before that.

Mar 21 20:55:03 pbxsub php: /sbin/iptables -w5 -W10000 -D fpbxsmarthosts -s 64.16.250.10/32 -m mark --mark 0x1/0x1 -j ACCEPT
Mar 21 20:55:39 pbxsub php: /sbin/iptables -w5 -W10000 -A fpbxsmarthosts -s 64.16.250.10/32 -m mark --mark 0x1/0x1 -j ACCEPT
Mar 21 21:13:45 pbxsub php: /sbin/iptables -w5 -W10000 -D fpbxsmarthosts -s 64.16.250.10/32 -m mark --mark 0x1/0x1 -j ACCEPT
Mar 21 21:14:21 pbxsub php: /sbin/iptables -w5 -W10000 -A fpbxsmarthosts -s 64.16.250.10/32 -m mark --mark 0x1/0x1 -j ACCEPT
Mar 21 21:31:02 pbxsub php: /sbin/iptables -w5 -W10000 -D fpbxsmarthosts -s 64.16.250.10/32 -m mark --mark 0x1/0x1 -j ACCEPT
Mar 21 21:31:34 pbxsub php: /sbin/iptables -w5 -W10000 -A fpbxsmarthosts -s 64.16.250.10/32 -m mark --mark 0x1/0x1 -j ACCEPT
Mar 21 21:32:20 pbxsub php: /sbin/iptables -w5 -W10000 -D fpbxsmarthosts -s 64.16.250.10/32 -m mark --mark 0x1/0x1 -j ACCEPT
Mar 21 21:32:56 pbxsub php: /sbin/iptables -w5 -W10000 -A fpbxsmarthosts -s 64.16.250.10/32 -m mark --mark 0x1/0x1 -j ACCEPT
Mar 21 21:49:33 pbxsub php: /sbin/iptables -w5 -W10000 -D fpbxsmarthosts -s 64.16.250.10/32 -m mark --mark 0x1/0x1 -j ACCEPT
Mar 21 21:50:09 pbxsub php: /sbin/iptables -w5 -W10000 -A fpbxsmarthosts -s 64.16.250.10/32 -m mark --mark 0x1/0x1 -j ACCEPT
Mar 21 22:46:48 pbxsub php: /sbin/iptables -w5 -W10000 -D fpbxsmarthosts -s 64.16.250.10/32 -m mark --mark 0x1/0x1 -j ACCEPT
Mar 21 22:47:25 pbxsub php: /sbin/iptables -w5 -W10000 -A fpbxsmarthosts -s 64.16.250.10/32 -m mark --mark 0x1/0x1 -j ACCEPT
Mar 21 23:07:07 pbxsub php: /sbin/iptables -w5 -W10000 -D fpbxsmarthosts -s 64.16.250.10/32 -m mark --mark 0x1/0x1 -j ACCEPT
Mar 21 23:07:43 pbxsub php: /sbin/iptables -w5 -W10000 -A fpbxsmarthosts -s 64.16.250.10/32 -m mark --mark 0x1/0x1 -j ACCEPT
Mar 21 23:15:09 pbxsub rsyslogd: imjournal: journal reloaded… [v8.24.0-52.el7_8.2 try You searched for error 0 - rsyslog ]
Mar 21 23:26:04 pbxsub php: /sbin/iptables -w5 -W10000 -D fpbxsmarthosts -s 64.16.250.10/32 -m mark --mark 0x1/0x1 -j ACCEPT
Mar 21 23:26:40 pbxsub php: /sbin/iptables -w5 -W10000 -A fpbxsmarthosts -s 64.16.250.10/32 -m mark --mark 0x1/0x1 -j ACCEPT
Mar 21 23:51:38 pbxsub php: /sbin/iptables -w5 -W10000 -D fpbxsmarthosts -s 64.16.250.10/32 -m mark --mark 0x1/0x1 -j ACCEPT
Mar 21 23:52:14 pbxsub php: /sbin/iptables -w5 -W10000 -A fpbxsmarthosts -s 64.16.250.10/32 -m mark --mark 0x1/0x1 -j ACCEPT
Mar 22 00:16:36 pbxsub php: /sbin/iptables -w5 -W10000 -D fpbxsmarthosts -s 64.16.250.10/32 -m mark --mark 0x1/0x1 -j ACCEPT
Mar 22 00:17:58 pbxsub php: /sbin/iptables -w5 -W10000 -A fpbxsmarthosts -s 64.16.250.10/32 -m mark --mark 0x1/0x1 -j ACCEPT
Mar 22 00:22:22 pbxsub chronyd[687]: Selected source 69.10.223.132
Mar 22 00:26:55 pbxsub php: /sbin/iptables -w5 -W10000 -D fpbxsmarthosts -s 64.16.250.10/32 -m mark --mark 0x1/0x1 -j ACCEPT
Mar 22 00:27:31 pbxsub php: /sbin/iptables -w5 -W10000 -A fpbxsmarthosts -s 64.16.250.10/32 -m mark --mark 0x1/0x1 -j ACCEPT
Mar 22 01:00:38 pbxsub auditd[645]: Audit daemon rotating log files
Mar 22 01:07:07 pbxsub php: /sbin/iptables -w5 -W10000 -D fpbxsmarthosts -s 64.16.250.10/32 -m mark --mark 0x1/0x1 -j ACCEPT
Mar 22 01:07:43 pbxsub php: /sbin/iptables -w5 -W10000 -A fpbxsmarthosts -s 64.16.250.10/32 -m mark --mark 0x1/0x1 -j ACCEPT
Mar 22 01:16:25 pbxsub php: /sbin/iptables -w5 -W10000 -D fpbxsmarthosts -s 64.16.250.10/32 -m mark --mark 0x1/0x1 -j ACCEPT
Mar 22 01:17:01 pbxsub php: /sbin/iptables -w5 -W10000 -A fpbxsmarthosts -s 64.16.250.10/32 -m mark --mark 0x1/0x1 -j ACCEPT
Mar 22 01:22:36 pbxsub php: /sbin/iptables -w5 -W10000 -D fpbxsmarthosts -s 64.16.250.10/32 -m mark --mark 0x1/0x1 -j ACCEPT
Mar 22 01:23:13 pbxsub php: /sbin/iptables -w5 -W10000 -A fpbxsmarthosts -s 64.16.250.10/32 -m mark --mark 0x1/0x1 -j ACCEPT
Mar 22 01:23:59 pbxsub php: /sbin/iptables -w5 -W10000 -D fpbxsmarthosts -s 64.16.250.10/32 -m mark --mark 0x1/0x1 -j ACCEPT
Mar 22 01:24:35 pbxsub php: /sbin/iptables -w5 -W10000 -A fpbxsmarthosts -s 64.16.250.10/32 -m mark --mark 0x1/0x1 -j ACCEPT
Mar 22 01:29:08 pbxsub php: /sbin/iptables -w5 -W10000 -D fpbxsmarthosts -s 64.16.250.10/32 -m mark --mark 0x1/0x1 -j ACCEPT
Mar 22 01:29:44 pbxsub php: /sbin/iptables -w5 -W10000 -A fpbxsmarthosts -s 64.16.250.10/32 -m mark --mark 0x1/0x1 -j ACCEPT
Mar 22 01:30:30 pbxsub php: /sbin/iptables -w5 -W10000 -D fpbxsmarthosts -s 64.16.250.10/32 -m mark --mark 0x1/0x1 -j ACCEPT
Mar 22 01:31:07 pbxsub php: /sbin/iptables -w5 -W10000 -A fpbxsmarthosts -s 64.16.250.10/32 -m mark --mark 0x1/0x1 -j ACCEPT
Mar 22 01:33:41 pbxsub php: /sbin/iptables -w5 -W10000 -D fpbxsmarthosts -s 64.16.250.10/32 -m mark --mark 0x1/0x1 -j ACCEPT
Mar 22 01:34:17 pbxsub php: /sbin/iptables -w5 -W10000 -A fpbxsmarthosts -s 64.16.250.10/32 -m mark --mark 0x1/0x1 -j ACCEPT
Mar 22 01:39:12 pbxsub php: /sbin/iptables -w5 -W10000 -D fpbxsmarthosts -s 64.16.250.10/32 -m mark --mark 0x1/0x1 -j ACCEPT
M
Mar 22 02:16:58 pbxsub php: /sbin/iptables -w5 -W10000 -D fpbxsmarthosts -s 64.16.250.10/32 -m mark --mark 0x1/0x1 -j ACCEPT
Mar 22 02:17:34 pbxsub php: /sbin/iptables -w5 -W10000 -A fpbxsmarthosts -s 64.16.250.10/32 -m mark --mark 0x1/0x1 -j ACCEPT
Mar 22 02:18:56 pbxsub php: /sbin/iptables -w5 -W10000 -D fpbxsmarthosts -s 64.16.250.10/32 -m mark --mark 0x1/0x1 -j ACCEPT
Mar 22 02:19:32 pbxsub php: /sbin/iptables -w5 -W10000 -A fpbxsmarthosts -s 64.16.250.10/32 -m mark --mark 0x1/0x1 -j ACCEPT
Mar 22 02:36:40 pbxsub php: /sbin/iptables -w5 -W10000 -D fpbxsmarthosts -s 64.16.250.10/32 -m mark --mark 0x1/0x1 -j ACCEPT
Mar 22 02:37:16 pbxsub php: /sbin/iptables -w5 -W10000 -A fpbxsmarthosts -s 64.16.250.10/32 -m mark --mark 0x1/0x1 -j ACCEPT
Mar 22 02:39:51 pbxsub php: /sbin/iptables -w5 -W10000 -D fpbxsmarthosts -s 64.16.250.10/32 -m mark --mark 0x1/0x1 -j ACCEPT
Mar 22 02:40:27 pbxsub php: /sbin/iptables -w5 -W10000 -A fpbxsmarthosts -s 64.16.250.10/32 -m mark --mark 0x1/0x1 -j ACCEPT
Mar 22 02:48:22 pbxsub php: /sbin/iptables -w5 -W10000 -D fpbxsmarthosts -s 64.16.250.10/32 -m mark --mark 0x1/0x1 -j ACCEPT
Mar 22 02:48:59 pbxsub php: /sbin/iptables -w5 -W10000 -A fpbxsmarthosts -s 64.16.250.10/32 -m mark --mark 0x1/0x1 -j ACCEPT
Mar 22 02:51:17 pbxsub php: /sbin/iptables -w5 -W10000 -D fpbxsmarthosts -s 64.16.250.10/32 -m mark --mark 0x1/0x1 -j ACCEPT
Mar 22 02:51:53 pbxsub php: /sbin/iptables -w5 -W10000 -A fpbxsmarthosts -s 64.16.250.10/32 -m mark --mark 0x1/0x1 -j ACCEPT
Mar 22 03:54:51 pbxsub php: /sbin/iptables -w5 -W10000 -D fpbxsmarthosts -s 64.16.250.10/32 -m mark --mark 0x1/0x1 -j ACCEPT
Mar 22 03:56:09 pbxsub php: /sbin/iptables -w5 -W10000 -A fpbxsmarthosts -s 64.16.250.10/32 -m mark --mark 0x1/0x1 -j ACCEPT
Mar 22 03:58:17 pbxsub php: /sbin/iptables -w5 -W10000 -D fpbxsmarthosts -s 64.16.250.10/32 -m mark --mark 0x1/0x1 -j ACCEPT
Mar 22 03:58:53 pbxsub php: /sbin/iptables -w5 -W10000 -A fpbxsmarthosts -s 64.16.250.10/32 -m mark --mark 0x1/0x1 -j ACCEPT
Mar 22 04:25:54 pbxsub php: /sbin/iptables -w5 -W10000 -D fpbxsmarthosts -s 64.16.250.10/32 -m mark --mark 0x1/0x1 -j ACCEPT
Mar 22 04:26:31 pbxsub php: /sbin/iptables -w5 -W10000 -A fpbxsmarthosts -s 64.16.250.10/32 -m mark --mark 0x1/0x1 -j ACCEPT
Mar 22 04:45:27 pbxsub php: /sbin/iptables -w5 -W10000 -D fpbxsmarthosts -s 64.16.250.10/32 -m mark --mark 0x1/0x1 -j ACCEPT
Mar 22 04:46:03 pbxsub php: /sbin/iptables -w5 -W10000 -A fpbxsmarthosts -s 64.16.250.10/32 -m mark --mark 0x1/0x1 -j ACCEPT
Mar 22 04:54:18 pbxsub php: /sbin/iptables -w5 -W10000 -D fpbxsmarthosts -s 64.16.250.10/32 -m mark --mark 0x1/0x1 -j ACCEPT
Mar 22 04:54:54 pbxsub php: /sbin/iptables -w5 -W10000 -A fpbxsmarthosts -s 64.16.250.10/32 -m mark --mark 0x1/0x1 -j ACCEPT
Mar 22 05:29:30 pbxsub php: /sbin/iptables -w5 -W10000 -D fpbxsmarthosts -s 64.16.250.10/32 -m mark --mark 0x1/0x1 -j ACCEPT
Mar 22 05:30:06 pbxsub php: /sbin/iptables -w5 -W10000 -A fpbxsmarthosts -s 64.16.250.10/32 -m mark --mark 0x1/0x1 -j ACCEPT
Mar 22 05:35:16 pbxsub php: /sbin/iptables -w5 -W10000 -D fpbxsmarthosts -s 64.16.250.10/32 -m mark --mark 0x1/0x1 -j ACCEPT
Mar 22 05:35:52 pbxsub php: /sbin/iptables -w5 -W10000 -A fpbxsmarthosts -s 64.16.250.10/32 -m mark --mark 0x1/0x1 -j ACCEPT
Mar 22 05:50:24 pbxsub php: /sbin/iptables -w5 -W10000 -D fpbxsmarthosts -s 64.16.250.10/32 -m mark --mark 0x1/0x1 -j ACCEPT
Mar 22 05:51:01 pbxsub php: /sbin/iptables -w5 -W10000 -A fpbxsmarthosts -s 64.16.250.10/32 -m mark --mark 0x1/0x1 -j ACCEPT
Mar 22 05:53:35 pbxsub php: /sbin/iptables -w5 -W10000 -D fpbxsmarthosts -s 64.16.250.10/32 -m mark --mark 0x1/0x1 -j ACCEPT
Mar 22 05:54:11 pbxsub php: /sbin/iptables -w5 -W10000 -A fpbxsmarthosts -s 64.16.250.10/32 -m mark --mark 0x1/0x1 -j ACCEPT
Mar 22 06:20:19 pbxsub php: /sbin/iptables -w5 -W10000 -D fpbxsmarthosts -s 64.16.250.10/32 -m mark --mark 0x1/0x1 -j ACCEPT
Mar 22 06:20:51 pbxsub php: /sbin/iptables -w5 -W10000 -A fpbxsmarthosts -s 64.16.250.10/32 -m mark --mark 0x1/0x1 -j ACCEPT
Mar 22 06:28:41 pbxsub php: /sbin/iptables -w5 -W10000 -D fpbxsmarthosts -s 64.16.250.10/32 -m mark --mark 0x1/0x1 -j ACCEPT
Mar 22 06:28:59 pbxsub systemd: Stopping Fail2Ban Service…
Mar 22 06:28:59 pbxsub systemd: Removed slice system-selinux\x2dpolicy\x2dmigrate\x2dlocal\x2dchanges.slice.
Mar 22 06:28:59 pbxsub systemd: Stopping Availability of block devices…
Mar 22 06:28:59 pbxsub systemd: Stopped target Timers.

And then it went down again around 8am

Mar 22 06:46:52 pbxsub systemd: Started Cleanup of Temporary Directories.
Mar 22 06:47:32 pbxsub php: /sbin/iptables -w5 -W10000 -D fpbxsmarthosts -s 64.16.250.10/32 -m mark --mark 0x1/0x1 -j ACCEPT
Mar 22 06:48:08 pbxsub php: /sbin/iptables -w5 -W10000 -A fpbxsmarthosts -s 64.16.250.10/32 -m mark --mark 0x1/0x1 -j ACCEPT
Mar 22 07:09:58 pbxsub php: /sbin/iptables -w5 -W10000 -D fpbxsmarthosts -s 64.16.250.10/32 -m mark --mark 0x1/0x1 -j ACCEPT
Mar 22 07:10:34 pbxsub php: /sbin/iptables -w5 -W10000 -A fpbxsmarthosts -s 64.16.250.10/32 -m mark --mark 0x1/0x1 -j ACCEPT
Mar 22 07:29:14 pbxsub php: /sbin/iptables -w5 -W10000 -D fpbxsmarthosts -s 64.16.250.10/32 -m mark --mark 0x1/0x1 -j ACCEPT
Mar 22 07:29:50 pbxsub php: /sbin/iptables -w5 -W10000 -A fpbxsmarthosts -s 64.16.250.10/32 -m mark --mark 0x1/0x1 -j ACCEPT
Mar 22 07:32:56 pbxsub php: /sbin/iptables -w5 -W10000 -D fpbxsmarthosts -s 64.16.250.10/32 -m mark --mark 0x1/0x1 -j ACCEPT
Mar 22 07:33:32 pbxsub php: /sbin/iptables -w5 -W10000 -A fpbxsmarthosts -s 64.16.250.10/32 -m mark --mark 0x1/0x1 -j ACCEPT
Mar 22 07:51:32 pbxsub kernel: perf: interrupt took too long (2520 > 2500), lowering kernel.perf_event_max_sample_rate to 79000
Mar 22 08:16:31 pbxsub php: /sbin/iptables -w5 -W10000 -D fpbxsmarthosts -s 64.16.250.10/32 -m mark --mark 0x1/0x1 -j ACCEPT
Mar 22 08:17:07 pbxsub php: /sbin/iptables -w5 -W10000 -A fpbxsmarthosts -s 64.16.250.10/32 -m mark --mark 0x1/0x1 -j ACCEPT
Mar 22 08:26:59 pbxsub systemd: Stopping Fail2Ban Service…
Mar 22 08:26:59 pbxsub systemd: Closed LVM2 poll daemon socket.
Mar 22 08:26:59 pbxsub systemd: Removed slice system-selinux\x2dpolicy\x2dmigrate\x2dlocal\x2dchanges.slice.
Mar 22 08:26:59 pbxsub systemd: Stopping RPC bind service…
Mar 22 08:26:59 pbxsub systemd: Stopping LSB: DAHDI kernel modules…
Mar 22 08:26:59 pbxsub systemd: Stopping Authorization Manager…

I really don’t know what to make of this, any help would be appreciated.

x201_F12 · March 23, 2024, 9:24am

The system was down again so I made a call outbound and saw this in the asterisk logs when I made the call

491617 [2024-03-23 02:12:57] VERBOSE[22926][C-000000e7] pbx_builtins.c: Goto (macro-dialout-trunk,continue,1)
491618 [2024-03-23 02:12:57] VERBOSE[22926][C-000000e7] pbx.c: Executing [continue@macro-dialout-trunk:1] NoOp(“SIP/4405-000008ca”, “TRUNK Dial failed due to CHANUNAVAIL HANGUPCAUSE: 20 - failing through to other trunks”) in new stack
491619 [2024-03-23 02:12:57] VERBOSE[22926][C-000000e7] pbx.c: Executing [continue@macro-dialout-trunk:2] ExecIf(“SIP/4405-000008ca”, “1?Set(CALLERID(number)=4405)”) in new stack
491620 [2024-03-23 02:12:57] VERBOSE[22926][C-000000e7] pbx.c: Executing [2095815022@restrictedroute-453e406dcee4d18174d4ff623f52dcd8:13] Macro(“SIP/4405-000008ca”, “outisbusy,”) in new stack
491621 [2024-03-23 02:12:57] VERBOSE[22926][C-000000e7] pbx.c: Executing [s@macro-outisbusy:1] Progress(“SIP/4405-000008ca”, “”) in new stack
491622 [2024-03-23 02:12:57] VERBOSE[22926][C-000000e7] pbx.c: Executing [s@macro-outisbusy:2] GotoIf(“SIP/4405-000008ca”, “0?emergency,1”) in new stack
491623 [2024-03-23 02:12:57] VERBOSE[22926][C-000000e7] pbx.c: Executing [s@macro-outisbusy:3] GotoIf(“SIP/4405-000008ca”, “0?intracompany,1”) in new stack
491624 [2024-03-23 02:12:57] VERBOSE[22926][C-000000e7] pbx.c: Executing [s@macro-outisbusy:4] Playback(“SIP/4405-000008ca”, “all-circuits-busy-now&please-try-call-later, noanswer”) in new stack
491625 [2024-03-23 02:12:57] VERBOSE[22926][C-000000e7] file.c: <SIP/4405-000008ca> Playing ‘all-circuits-busy-now.ulaw’ (language ‘en’)

x201_F12 · March 23, 2024, 5:18pm

fwconsole reload

Did not fix the problem

fwconsole restart

Did fix the problem.

Again not sure what that means but now fwconsole restart and a reboots gets calls back.

tmittelstaedt · March 23, 2024, 6:35pm

If you have a service contract on this PBXact 40 system perhaps the regular support channel would help better?

My suggestion is as follows,

How to monitor the performance of “iptables” kernel module? - Unix & Linux Stack Exchange

What you describe sounds very much like a symptom of an attacker who has found a way to exploit a memory or other resource leak in the iptables module. They are sending tons of crap to your system and causing it to run out of some resource, that is a classic denial of service attack. I’m not saying that is what is happening, but the procedures and tools discussed in the thread I linked to will be just as useful if this happens to be a software bug.

x201_F12 · March 25, 2024, 11:25am

So this weekend I was able to play with the system a lot more. I have noticed that a reboot of my firewall (pfsense) also fixes the problem. Again, I don’t really know what that means but

Rebooting the PBX = Fixes the problem
fwconsole restart = Fixes the problem
Rebooting the pfsense = Fixes the problem

I’m starting to feel this is something due to inactivity and somehow the connection is closing.

On Saturday I checked on it sporadically and it was down a lot. The business was closed not many calls come in.

On Sunday I found it down at 2am rebooted the firewall and then, called it a lot (my phone says 40 times) and it was up every time. At 2pm I left the office went home called it around 6pm and it was down. I brought it back up and found it down 2:00am when I got to the office.

I really don’t know how to test for this.

system · April 25, 2024, 11:25am

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.