SIP Trunk Problems - outbound calls fail

This setup has been working fine for a months. Seemingly without reason in bound calls failed. I rebooted the FreePBX and in bound started working, but then outbound calls got that "all circuits are busy " Trunk Registration handshake looks fine in sngrep but sip show peers says its unreachable.

If I configure qualify=no sip show peers looks good, I don’t get " all circuits busy " message however it just rings, tcpdump shows it never attempting to call out.
Im behind Fortigate firewall using NAT and port translation the PBX is on port 5160 sip provider is on 5060. I noticed that outbound OPTIONS being sent to my SIP provider are sending from the PBX to a Fortinet blackhole ip address and the call never hits the pbx-outbound firewall rule. and of course never gets a response.
I very much appreciate any pointers. Thanks so much.

Hi been a while since ive been involved with fortine and fortigate. but here are a few things ive documented in the past

If this is the output of ‘sip show peers’ (as opposed to ‘sip show peers like xxx’), then all of your extensions and trunks except this one are using pjsip. Why are you using chan_sip?

In nearly all setups, OPTIONS (and INVITE) should be sent to the same IP address as REGISTER.
But in chan_sip, the register string has its own host parameter, independent of the PEER Details. So check those carefully, taking into account any applicable subtleties such as outbound proxy or SRV lookup.

In any case, the destination of your outbound INVITE should only be dependent on local settings and DNS lookup results; if it’s wrong but your PEER Details are correct I’d suspect a DNS problem. A packet capture should show whether this is happening.

I took another look at your blurry screenshot and found this:

So it appears that the OPTIONS and INVITE messages should indeed go to a different address than REGISTER.

Check that your outboundproxy domain name resolves correctly. If it does (PBX is attempting to send to the correct address), then fix the firewall to not blackhole it.

Weird, this is from tcpdump port 5160 on the freepbx:
17:09:03.745817 IP > SIP: REGISTER SIP/2.0
17:09:03.810811 IP > SIP: SIP/2.0 407 Proxy Authentication Required
17:09:03.811449 IP > SIP: REGISTER SIP/2.0
17:09:03.879267 IP > SIP: SIP/2.0 200 OK
17:09:04.374179 IP > SIP: OPTIONS SIP/2.0

the pbx’s os properly resolves the outbound proxy, yet it is sending packets to the blackhole. Ive checked the firewall and the ip addresses are white listed and no blocked or dropped packets in the logs. so for some reason the PBX is sending OPTION packets to the blackhole. SIP helper is disabled. is there some other way that the PBX is resolving names other than DNS? So weird.

In chan_sip, the outboundproxy parameter is not used for REGISTER. For Jive, this is exactly what is wanted. For providers that require registration through an outbound proxy, it can be added to the register string.

So for OPTIONS (and INVITE), it is using the outboundproxy parameter, which is what is wanted.
Just a guess, Jive moved their proxy server to a new address, which is why it suddenly stopped working.

Do the tcpdump with a -f option to see the numeric address attempted. If this is correct, fix the firewall to let it through. If incorrect, troubleshoot the DNS.

That’s even more true of chan_pjsip, at least at the Asterisk level.

tcpdump -f port 5160:

17:54:38.627679 IP > pbx01.xxxx.local.5160: SIP
17:54:44.588633 IP pbx01.xxxxx.local.5160 > SIP: OPTIONS SIP/2.0 is the fortinet black hole.
So it seems its not correct.

Indeed; is an address at Fortinet.

So whatever the PBX is doing with the outboundproxy parameter is not working right. What do you have set there? What are the contents of /etc/resolv.conf on the PBX?

/etc/resolv.conf just has my internal dns servers, which resolve the names perfectly.
/etc/nsswitch.conf: hosts: files dns myhostname
I put a manual entry in /etc/hosts. Ping and traceroute go through to the proper address ( the one in /etc/hosts) and is passed though properly by the firewall. Just seems the PBX is sending OPTION data for the trunk to directly the the blackhole. BTW… I super appreciate your help and expertise.

It looks like a DNS problem. Im getting different ip resolutions from different servers. I’ll let you know what i find out. Thanks again.

Thank you SO very much for your help and sage advice. I was able to track it down to a DNS issue, I guess the PBX uses DNS exclusively for name resolution and even if you put the proper ip address instead of FQDN. Regardless, some on my dns servers were resolving the proxy to the blackhole address, others were not. Ill get to the bottom of that straightway. Thanks again for the great help!