PJSIP Remote Extension Oneway Traffic Followup

[edricksmith]

PJSIP Extension only one way response

Rebooting the SPA device hasn’t had any change. I’ll have to get remote access to the SPA to see the expiration value, however as mentioned rebooting it doesn’t change it. Site 2 which is the one that was down, then randomly came back up a few hours later is still online as of now. Site 1 is still showing as unavailable.

This was the previous topic, I finally got my hands on the adapter to put it at a different site.

So I got the unit back and was able to do some further digging;

So I did notice I was able to find this in the log when attempting calls;

res_pjsip/pjsip_distributor.c: Request ‘INVITE’ from ‘“Anonymous” sip:anonymous@localhost’ failed for ‘REMOTEIP:5060’

I tried digging into the credentials and re-creating the extension and seeing if I could find what was causing it but no luck, changed the password to a simple one, updated the Display name and User ID in the SPA configuration to see if that was the issue.

So although the adapter worked when it first went out, something happened that made it stop working even when I brought it back in for testing.

The only “fix” I was able to get it working was to factory reset the unit and I set it up as a new extension and it worked.

I took the unit off site to my place and tested it and had no issues, brought it back and tested it on our separate testing network and it worked fine.

Then it went back out to another remote site (same area as before, just down the street to another location)

BAM same issue, they were able to initially make an outbound call, it showed up in the Asterisk Info section and I was able to see that it’s unavailable however, but it did show the IP address of where it had connected from. This location has a different ISP and doesn’t have any extra router attached to it as the old location had a Google Router.

The IP address is whitelisted in the Connectivity > Firewall section and is not listed in the Sys Admin > Intrusion Detection as being blocked, I have also whitelisted it there too.

---

What I see in the Asterisk Info Tab:

Endpoint: EXTENSION#/EXTENSION# Unavailable 0 of inf
InAuth: EXTENSION#-auth/EXTENSION#
Aor: EXTENSION# 1
Contact: EXTENSION#/sip:REMOTEIP@REMOTEIP:50 2a732c4636 Unavail 0.000

–

So it did connect and the contact point showed the remote IP

---

I’ve enabled PJSIP Logging for the host in question per the prior thread and I just see the request being sent out over and over, but no response this just gets sent out over and over

<— Transmitting SIP request (450 bytes) to UDP:REMOTEIP:5060 —>

OPTIONS sip:EXTENSION#@REMOTEIP:5060 SIP/2.0

Via: SIP/2.0/UDP PBXWANIP:5060;rport;branch=z9hG4bKPj32ffdfb7-43b9-4737-9bae-490295396c93

From: <sip:EXTENSION#@PBXLANIP>;tag=7fb51a69-b6f4-443e-84f1-bc70fe08e2d4

To: <sip:EXTENSION#@REMOTEIP>

Contact: <sip:EXTENSION#@PBXWANIP:5060>

Call-ID: 7f438b7a-390b-4acd-a9a7-ff4739d0b2b1

CSeq: 57694 OPTIONS

Max-Forwards: 70

User-Agent: PBXact-14.0.13.26(13.29.2)

Content-Length: 0

---

I have other SPAs to this same box that work fine, this one also worked fine again upon testing here and at my place but as soon as it went back out issue came back.

Different ISPs, different routers same issue

So I’m hoping something I’ve posted might give even the slightest bit further of a clue of what I can do to troubleshoot.

Also it would seem the PBX knows there’s some type of attempt because I would assume the box wouldn’t continuously Transmit a SIP request to that sites IP address randomly? It must know there’s a device trying to register and is transmitting to it but not receiving back if I’m thinking correctly?

---

It seems something is preventing the 200 OK request from making its way back perhaps? Is there a specific setting for PJSIP with the NAT settings like there is for the SIP Driver? It’s just rather confusing that it worked fine when I first tested it on the test network, worked fine at my place in another city, then as soon as it ended up at its final destination once again it started exhibiting the same symptoms

[edricksmith]

I’ve compared the message header between the SPA that does work that sends and receives the request and compared it to the request sent to the one that doesn’t work. The one that is working the request it sends matches basically the one that doesn’t work as far as IP information etc…

So something is blocking the response where it would be Receiving From REMOTESITEIP for the SIP 200ok

Just some more information incase anything helps;

uc-*CLI> pjsip show endpoint EXT#

Endpoint: <Endpoint/CID…> <State…> <Channels.>
I/OAuth: <AuthId/UserName…>
Aor: <Aor…>
Contact: <Aor/ContactUri…> <Hash…> <RTT(ms)…>
Transport: <TransportId…> <BindAddress…>
Identify: <Identify/Endpoint…>
Match: <criteria…>
Channel: <ChannelId…> <State…> <Time…>
Exten: <DialedExten…> CLCID: <ConnectedLineCID…>

Endpoint: EXT#/EXT# Unavailable 0 of inf
InAuth: EXT#-auth/EXT#
Aor: EXT# 1
Contact: EXT#/sip:EXT#@REMOTEWANIP:50 2a732c4636 Unavail 0.000

ParameterName : ParameterValue

100rel : yes
CHANNEL(parkinglot) : default
accept_multiple_sdp_answers : false
accountcode :
acl :
aggregate_mwi : true
allow : (ulaw|alaw|gsm|g726|g722)
allow_overlap : true
allow_subscribe : true
allow_transfer : true
aors : EXT#
asymmetric_rtp_codec : false
auth : EXT#-auth
bind_rtp_to_media_address : false
call_group :
callerid : “CID NAME” <EXT#>
callerid_privacy : allowed_not_screened
callerid_tag :
connected_line_method : invite
contact_acl :
context : from-internal
cos_audio : 5
cos_video : 4
device_state_busy_at : 0
direct_media : true
direct_media_glare_mitigation : none
direct_media_method : invite
disable_direct_media_on_nat : false
dtls_ca_file :
dtls_ca_path :
dtls_cert_file :
dtls_cipher :
dtls_fingerprint : SHA-256
dtls_private_key :
dtls_rekey : 0
dtls_setup : active
dtls_verify : No
dtmf_mode : rfc4733
fax_detect : false
fax_detect_timeout : 0
follow_early_media_fork : true
force_avp : false
force_rport : true
from_domain :
from_user :
g726_non_standard : false
ice_support : false
identify_by : username,ip
ignore_183_without_sdp : false
inband_progress : false
incoming_mwi_mailbox :
language : en
mailboxes :
media_address :
media_encryption : no
media_encryption_optimistic : false
media_use_received_transport : false
message_context :
moh_passthrough : false
moh_suggest : default
mwi_from_user :
mwi_subscribe_replaces_unsolicited : no
named_call_group :
named_pickup_group :
notify_early_inuse_ringing : false
one_touch_recording : true
outbound_auth :
outbound_proxy :
pickup_group :
record_off_feature : apprecord
record_on_feature : apprecord
refer_blind_progress : true
rewrite_contact : true
rpid_immediate : false
rtcp_mux : false
rtp_engine : asterisk
rtp_ipv6 : false
rtp_keepalive : 0
rtp_symmetric : true
rtp_timeout : 0
rtp_timeout_hold : 0
sdp_owner : -
sdp_session : Asterisk
send_connected_line : yes
send_diversion : true
send_pai : true
send_rpid : false
srtp_tag_32 : false
sub_min_expiry : 0
subscribe_context :
suppress_q850_reason_headers : false
t38_udptl : false
t38_udptl_ec : none
t38_udptl_ipv6 : false
t38_udptl_maxdatagram : 0
t38_udptl_nat : false
timers : yes
timers_min_se : 90
timers_sess_expires : 1800
tone_zone :
tos_audio : 184
tos_video : 136
transport :
trust_connected_line : yes
trust_id_inbound : true
trust_id_outbound : false
use_avpf : false
use_ptime : false
user_eq_phone : false
voicemail_extension :

---

PJSIP Driver Config:
[0.0.0.0-udp]
type=transport
protocol=udp
bind=0.0.0.0:5060
external_media_address=WANIP
external_signaling_address=WANIP
allow_reload=no
tos=cs3
cos=3
local_net=10.0.2.0/24 (LAN where server is)
local_net=10.8.0.0/24 (VPN LAN)

[edricksmith]

Kept digging into it over the weekend and thus far have found no solution. The other remote adapters continue to work fine

[wmoon (Walter Moon)]

Sorry if you already explained this, but can you talk a a bit more about the network setup between pbx and the failing environment? For the failing environment, is the device talking to the pbx over remote external addresses with port forwarding to the pbx, or is a vpn setup with differing subnets involved? If there is no Google router at the bad SPA location, what does it have instead?

As for those constant OPTIONS packets, they might be helpful to look at, but I think it would be easier to look at packets involved in a Registration attempt. If viewing the packets from the acli gets too messy, you can try using sngrep as others suggest, by entering ‘sngrep’ from the system command line. It’s a more interactive interface, where it’s easier to examine a particular transaction. You can also use it as a viewer if you’ve already recorded your packet capture to a file.

Another setting you can look at is the Rewrite Contact setting found in an Extension’s Advanced tab. Typically, you’d want to keep this at Yes, but given your situation, it could be worth checking to see if it makes a difference. Before playing with that, you’ll want to make sure your core module is up to date.

[edricksmith]

Hi, yes the core module is up to date, the appliance has the latest modules and system updates as of Friday.

The setup consists of the appliance at the main site on a static IP, with just the specific ports forwarded through the firewall. The appliance has an internal static address and then we have a static address for the WAN.

We have Sangoma phones working over VPN functionality with their built in client that are remote, we have SPA Phone adapters at a few sites that are just connecting over the internet.

This adapter originally months ago was setup and tested on site, but on a separate network segment / different public IP for testing, it then went out to the field and was connected to a cable connection with a Google Home router, it worked fine for a period of time, then started exhibiting issues where calling out would work occasionally, but calling in the system didn’t recognize it as being available and just sent the caller right to VM. Then calling out eventually stopped working and would just generate a busy signal. I went out to the site and bypassed the Google Home but the issue persisted, then finally they just brought it back in and wanted it sent to a different location.

So I plugged it back into the testing network here and it still continued to exhibit the inability to actually register and I noticed the res_pjsip/pjsip_distributor.c: Request ‘INVITE’ from ‘“Anonymous” sip:anonymous@localhost’ failed for ‘REMOTEIP:5060’

Popping up both in the logs from the previous site from the day before they brought it in and while it was here on the testing network. I tried modifying some settings (changing the password to a shorter string, trying a different extension, double checking the settings compared to another working SPA that is also on a remote network etc…) to no avail, I then factory reset the unit and was able to configure it to its new extension it was going to be repurposed for.

I then tested again on the testing network and it worked just fine, I brought it back home with me to test from another cable connection and it worked fine remotely, brought it back in once again and tested from the test network here no problems.

Then it went back out to the new site, which just has a Cox cable modem gateway, there is no additional router (which is actually the same setup as my house, just having a cox gateway and no additional router) and it once again stopped working. Exhibiting the same symptoms as before.

[edricksmith]

Using SNGREP the only transactions I see with that extension are the options, Invite and register never appear.

Edit: Oddly enough as soon as I posted that I went back and… it did register all of the sudden and I was able to make an inbound call…

However it since has dropped again;

I’ve attached 5 pictures 1) The Register Request 2) Unauthorized Response 3) The Second Register Request 4) The Reply 200 ok 5) The main filtered view showing the transactions for that IP

Right after that it however dropped back off and has a bunch of Option requests sent in a row again

[wmoon (Walter Moon)]

It’s a little hard to see what’s going on with the blanked out IPs, but I totally understand why you are hiding them. Something I would look into is the IP addresses, particularly in the Contact headers. I think I’m seeing the ATA’s LAN address in some cases, which might be the sign of an issue for the environment you describe.

Are any of the working devices on a similar network setup? That way you can compare the packets to make sure the external and internal addresses are listed in the packets the same way. On another note, have you considered updating the ATA firmware, just to rule that out? It looks like the version it’s reporting might not be the latest.

Also, have you tried looking into the registration expiration time on the ATA? It’s usually 1 hour/3600 seconds by default. If it’s still set around that duration, try something lower like 2 minutes. I don’t think this will fix the main issue, but it’s odd that there was a short period where things were working.

[edricksmith]

If it’s any easier I can PM you the unedited pictures with the IPs. Yes there are multiple SPAs working (two other on this system, same remote type of setup) and then I’ve got two other sangoma appliance systems that also run SPAs on the other side of the US.

I did compare the options request for one of the working ones and it did follow the same IP scheme. The registration is the default which I believe is 3600 as you mentioned, unfortunately this adapter is at a retirement community now and with the COVID they’re under lockdown and not allowing people in so I’m not sure how soon I’ll have access to the unit again. Which unfortunately makes troubleshooting all the more fun.

I’ve checked the firmware version as you say it is running an older version, however I do have another one of those adapters running the same firmware that is working.

That being said when I get the chance I will update the firmware on all units as I’ve downloaded the latest now.

I have also disabled / set rewrite contact to NO as it was on yes as default to see if that makes a difference

[wmoon (Walter Moon)]

@edricksmith

The registration is the default which I believe is 3600 as you mentioned, unfortunately this adapter is at a retirement community now and with the COVID they’re under lockdown and not allowing people in so I’m not sure how soon I’ll have access to the unit again. Which unfortunately makes troubleshooting all the more fun.

Understood. I don’t think that would have been the actual solution for this situation, but it’s odd that you get brief moments of what seemed like a successful registration. Decreasing it usually helps in a situation where the phone loses connection after some activity, but a reboot(forcing re-registration) will always get things working again for a bit. That’s not what we’re seeing here.

As for the comparing the OPTIONS packets, I wouldn’t focus too much on those for now. They are more like outbound pings from the pbx. We want to compare the REGISTER packets to see what info is coming from the phone during the initial part of the registration handshake.

For basic remote setups where the phone is registering to the PBX network’s external IP, it’s strange that the Contact header in the Register packet from the phone has an internal 192 address, instead of your ATA network’s external address. It might just add to the confusion, but I’m wondering if your working remote devices show the same in their Register packets. Also, do you know if your working ATAs have the same firmware as this problematic one?

[edricksmith]

The ATAs have a mix of firmware releases, I do know that other SPA boxes that have the same firmware as the non-working unit are working fine. Then the remainder of them that have a few different version releases all work fine. So it doesn’t seem to be firmware specific.

I do notice that the OPTIONS vs REGISTER have a difference

All the option requests are showing as for example;

SIP FROM: EXT#@LANIP OF SERVER
SIP To: EXT#@WAN IP OF ADAPTER SITE
Source: LAN IP OF SERVER
Destination: WAN IP OF ADAPTER SITE

Then the register event comes in and it shows;

SIP FROM: EXT#@ DDNS From Sangoma URL
SIP To: EXT#@ DDNS From Sangoma URL
Source: WAN IP OF ADAPTER SITE
Destination: LAN ADDRESS OF SERVER

Then all of the sudden after that when it drops back off all the option requests their DESTINATION address changes from using the WAN IP OF ADAPTER SITE to using LAN IP of ADAPTER

---

So all the initial option requests their destination is to the WAN IP of the adapter until the “successful” registration, then all of the sudden the Destination IP changed to the LAN IP of the adapter instead.

Now that is after I changed the contact rewrite to NO, however I did not pay attention to that field before hand so I’ll change it back to yes and see if it changes.

I’ll also compare the successful registration of one of the other adapters I suppose to the only successful registration for the one that’s not working right.

[edricksmith]

So around 5PM I started a log with SNGREP and just checked in on it now about 5 hours later.

There were about 300 events related to this extension during that time, of which those 300 were;

1x Registration at 5:18PM
1x Invite for In Call at 7:31PM
1x Invite for Canceled at 7:40 PM
298x Options

Between the Registration and the Invite for “In Call” were about 45% of the “options” that were sent
Then the Invite for In Call at 7:31PM came
About 10% of the “Options” were sent during this 9 minute period between the,
Invite for Canceled at 7:40 PM
Then the remaining 45% of the “Options” basically until I just looked at the log

I only mention that because I don’t know if this In Call was actually a call that was going on for 10 Minutes? Before the cancel or what it specifically means. But during that 10 minute window between In Call and Canceled there were still a number of Options being constantly sent with no response.

Then the “Canceled” Invite appears, followed by all the other Option requests constantly and the extension still shows as unregistered.

---

The register attempt at 5:18PM shows
Register —>
401 Un-Authorized <—
Register—>
200 OK <----
Register —>
401 Un-Authorized <—
Register —>
200 OK <—
(This Repeats another 3 or so times)

So basically it is sending out the register, getting un-authorized then sending out the register and getting a ok, then it repeats.

The Invite In-Call Has at 7:31PM

Invite SDP —>
401 Un-Authorized <—
ACK —>
Invite SDP —>
100 Trying <—
103 Session Progress (SPD).<—
ACK —>

---

The Cancel at 7:40PM Has
Invite SPD —>
401 Unauthorized <—
ACK —>
Invite SPD —>
100 Trying <—
183 Session Progress (SPD) <—
Cancel —>
200 OK <----
487 Request Terminated <----

I also saved the filtered output of that time period to a PCAP file that shows the transactions for that extension

Interestingly enough inside of the FreePBX interface under the Asterisk Info the PJSIP Extension where it shows its unavailable is reporting the LAN IP address of the adapter currently vs. the external WAN IP of the site it is at. I have now turned back on Rewrite Contact Header in the Advanced settings for the extension and watch over night again and compare.

[edricksmith]

Ok so after re-enabling Rewrite Contact header an hour or so later the extension did its registration again (and dropped right out just as it always does) and then at that point in SNGREP the Destination Column changed back and the Asterisk Info under the extension changed from reporting the LAN IP address to reporting the WAN address of the remote site. The extension has yet been able to register again.

Ok Latest Update

So I suppose the two biggest things I can see between the SPA that is remote and successfully able to register and the one that doesn’t correctly;

1. The From Line on the one that successfully registers show the from and To as an example;
   From: “Bob Smith” sip:EXT#@pbxurl.com
   To: “Bob Smith” sip:EXT#@pbxurl.com

Where as the one that doesn’t work is only showing it as
From: “EXT#” sip:EXT#@pbxurl.com
To: “EXT#” sip:EXT#@pbxurl.com

So not sure if it matters but for some reason the non-working one is showing the ext# instead of the name in the to and from

2) Instead of listing the Private IP of the Adapter the one that works correctly always has the WAN address of the remote site, where as the adapter that isn’t working always lists it as the LAN IP address and this is both with and without contact rewrite on.

So it seems item 2 here is more of the culprit but why would only this adapter be doing that?

[wmoon (Walter Moon)]

I’m not sure about the “Bob Smith” situation. At this time, I don’t think it matters but I’d look at the device’s config to see if that’s where it’s coming from. Maybe that old working one had a label or name field where that name was entered.

But yeah, I agree that point two is the likely culprit. Were you able to verify the differences in the Contact headers of the Register packets? To clarify, I mean when you have a Register packet highlighted, look for the IP that’s in the “Contact:” line. In your sngrep screenshots, we’re seeing 192.168.0.17, which is likely the problem. It sounds like you’re verifying the suspected issue based on the From/To values you’re looking at, but you never said what those “Contact” header values were. In the Contact line, is the working one showing the WAN ip and the failing one is showing the LAN ip? I’m guessing so based on what you’re seeing elsewhere in the packet, but it would help to have some confirmation.
Also, at this point, I’d just leave the Rewrite Contact option enabled since that doesn’t seem to be the issue, for now.

[edricksmith]

So on the register packet on the Contact Line for the non-working extension it always seems to be 192.168.0.17 wether Contact rewrite is on or off

Where as the other SPA which IS working that is at a remote site always has the contact as the WAN address of that site.

So it seems perhaps the question then is why is this the case, that the contact line on the non-working one is using the LAN IP, where as the Contact of the working one uses the correct WAN

[wmoon (Walter Moon)]

That makes sense. I agree that finding out why the LAN IP is in the contact of the failing ATA is the question to focus on. Usually for this, the first place to check is the router on the PBX side. It’s less obvious here since you have some units that work, but It’s still worth looking at. Sorry if you already mentioned this, but do you know what kind of router or other equipment is between the pbx and the outside world?

It would also help to see some info from the ATA’s side of things. It would help to know if there are any options on the ATA that could affect that header. It would seem odd that it’s the case since you reconfigured it from scratch, assuming you only set the basic registration settings. But at this point, it could still help to rule things out. If that device has any sort of packet tracing diagnostics, it would be helpful to see what that Register packet looks like as it leaves the device.

[edricksmith]

The first location it was tested at which is the same site the Appliance is at, but we have a separate network for testing was a Netgear Router consumer grade and it worked fine.

I then tested it at my place which is just a Arris TG1682G which is a ISP provided cable modem and it worked fine.

The site the SPA went to where it is not working now also should be that same Arris Modem or an equivalent as they also have the same internet provider I did.

The initial site it was out at where it also had issues was behind a Google Home Wifi setup.

What doesn’t make sense to me is why it worked fine here on the testing network, worked fine at my place on the cable modem then all of the sudden stopped working.

I’m going to setup another one of these SPAs I have and make sure its on the latest firmware and do some further testing with that oo.

[cynjut (Dave Burgess)]

@edricksmith

What doesn’t make sense to me is why it worked fine here on the testing network, worked fine at my place on the cable modem then all of the sudden stopped working.

Check the network ranges in each of the locations. Are they all the same (192.168.0.x, for example) or are they all different, or a mix?