All circuits are busy intra-company pjsip trunks

FreePBX Endpoints
1

Hi,
I have three FreePBX 17 deployments. A (main), B (branch-1) and C (branch-2).
A and B are connected by a tunnel.
A and C are connected by a tunnel.

Each trunk’s pjsip Setting are
Authentication: NONE
Registration: NONE
I understand using NONE/NONE sets up IP based authentication.

A/B have a pjsip intra-company trunk ‘bridge_to_B’ setup between them.
A/C have a pjsip intra-company trunk ‘bridge_to_C’ setup between them.

Reports - Asterisk Info - Peers
A reports that ‘bridge_to_B’ is ‘Not in use’ and ‘Available’
B reports that ‘bridge_to_A’ is ‘Not in use’ and ‘Available’
A/B can direct dial between each other.

A reports that ‘bridge_to_C’ is ‘Unavail’ and ‘Unavailable’
C reports that ‘bridge_to_A’ is ‘Unavail’ and ‘Unavailable’
A/C receive message ‘All circuits are busy’ when trying to dial either direction.

The tunnel between A/C is up and connected.
I can ping C from A, and I can ping A from C.
I can reach the pbx C GUI from A, and A GUI from C.
Direct dialing between A/C has worked in the past. My first complaint of not working was 4 days ago.

I have narrowed down the problem to be the A/C bridge is not connecting. However I don’t know what steps to take to troubleshoot this problem. Also I don’t understand the process of how the intra-company trunks authenticate/register (I don’t know the correct word to put here) with each other.

I have tried rebooting A and C.
I have tried deleting/recreating the A/C trunks.

How does an intra-company that is set up as NONE/NONE IP authentication become available?
What steps do you suggest do troubleshoot this problem?

Your help is appreciated.
Thank you.

2

Asterisk tests trunk availability by sending OPTIONS requests to the remote end, by default every minute. Any reply (even an error such as 403 Forbidden) is considered ‘success’ for this purpose. So these requests (if directed to the correct IP address and port) are likely being blocked by your network, or by FreePBX firewall / fail2ban (or other software firewall you may be running on the PBX).

Turn on PJSIP logger on A and see whether OPTIONS are being sent to C. If so, check there if OPTIONS requests are being received. If not, run sngrep on C. If they show up there, problem is with the software firewall. If not, problem is with your tunnel.

3

I forgot to mention that if OPTIONS requests are getting through to C and replies are being sent, the path back to A may be where the blockage occurs.

4

Stewart,
Thank you for your clue about OPTIONS.
I’m not sure about how to turn on/off PJSIP logger. However, I have done packet captures on A, B and C.

A/B trunk that is working…
A will send an OPTIONS request to B.
B receives the request and sends a 200 OK (OPTIONS) back to A
A receives the 200 OK.
Later cycle repeats.

A/C trunk that is broken…
A will send an OPTIONS request to C.
C receives the request and sends a 200 OK (OPTIONS) back to A
A does not receive the 200 OK.
Later A will send another OPTIONS request to C and the cycle repeats.

My next step is to packet capture on my two palo alto gateways to see where the 200 OK packet is dropped/blocked.

Thank you for the guidance!

5

Stewart,
The 200 OK packet is being dropped by our firewall on A’s end. We are trying to figure out which rule/policy is responsible for dropping the packet.

That got me thinking about other options like changing pjsip authentication/registration from NONE/NONE to OUTBOUND/NONE to use user/password instead of IP address.
Would this change how the trunk becomes available?

Also thought about changing from pjsip to IAX2. I chose pjsip because it worked the first time I tried it so I haven’t considered anything else.
Is IAX2 a better choice?

Thank you,

6

Stewart,
We have Palo Alto firewalls on A, B and C end and can’t find a problem with our policies/rules. We opened a ticket with PA and attached packet captures. PA replied they can see that A is dropping OPTION replies from C but don’t know why. They have bumped it up to next level support. I think they want to do a remote session to look at the issue.

While waiting, I setup an IAX2 trunk/bridge between A-C and added the iax2-bridge as second route in the outbound route. The secondary iax2-bridge is allowing A-C to direct dial each other. The users are happy again!

The iax2-bridge is only temporary until PA can find the issue. Although after thinking about it, I may leave the iax2-bridge in place as a fail-over in case pjsip-bridge breaks again.

7

Stewart,
OK, we found the problem. The problem was that Palo Alto at location C was sending the reply down the wrong virtual tunnel because an update had changed all the Virtual Route metric(s) to the same level of 10. This caused the PA at A to drop the packet because the packet arrived on the wrong tunnel. After resetting PA-C metrics to the correct value then PA-A allowed the packet thru to FPBX-A.
Thank you,

8

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.