Troubleshooting IAX2 trunks (stop arguing)

a5t1 · July 31, 2017, 1:16pm

I’d really like to bring up the other thread of why SIP or IAX is better for trunking, however, I don’t care.

At the end of the day I’d like to have some meaningful discussion on why my IAX2 trunks aren’t talking to each other.

To recap - I have two remote systems connected over a VPN. Both FreePBX systems are running the latest release.

Here is what I see:

 iax2 show peers
Name/Username    Host                                           Mask                                      Port           Status      Description
System1/System2  172.29.122.117                           (S)  255.255.255.255                           4569  (T)      UNREACHABLE

With debug:

 iax2 set debug on
IAX2 Debugging Enabled
Rx-Frame Retry[Yes] -- OSeqno: 000 ISeqno: 000 Type: IAX     Subclass: POKE
   Timestamp: 00014ms  SCall: 13277  DCall: 00000 172.29.122.117:4569

Tx-Frame Retry[ No] -- OSeqno: 000 ISeqno: 001 Type: IAX     Subclass: PONG
   Timestamp: 00014ms  SCall: 00001  DCall: 13277 172.29.122.117:4569
Tx-Frame Retry[000] -- OSeqno: 000 ISeqno: 000 Type: IAX     Subclass: POKE
   Timestamp: 00019ms  SCall: 14434  DCall: 00000 172.29.122.117:4569

Tx-Frame Retry[001] -- OSeqno: 000 ISeqno: 000 Type: IAX     Subclass: POKE
   Timestamp: 00019ms  SCall: 14434  DCall: 00000 172.29.122.117:4569

Tx-Frame Retry[002] -- OSeqno: 000 ISeqno: 000 Type: IAX     Subclass: POKE
   Timestamp: 00019ms  SCall: 14434  DCall: 00000 172.29.122.117:4569

Rx-Frame Retry[ No] -- OSeqno: 000 ISeqno: 000 Type: IAX     Subclass: POKE
   Timestamp: 00013ms  SCall: 00422  DCall: 00000 172.29.122.117:4569

Tx-Frame Retry[ No] -- OSeqno: 000 ISeqno: 001 Type: IAX     Subclass: PONG
   Timestamp: 00013ms  SCall: 00001  DCall: 00422 172.29.122.117:4569
Rx-Frame Retry[Yes] -- OSeqno: 000 ISeqno: 000 Type: IAX     Subclass: POKE
   Timestamp: 00013ms  SCall: 00422  DCall: 00000 172.29.122.117:4569

Tx-Frame Retry[ No] -- OSeqno: 000 ISeqno: 001 Type: IAX     Subclass: PONG
   Timestamp: 00013ms  SCall: 00001  DCall: 00422 172.29.122.117:4569

Here is the only other piece of information:
 tcpdump -X -s 0 -vv port 4569
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
09:14:16.864311 IP (tos 0xb8, ttl 64, id 5423, offset 0, flags [none], proto UDP (17), length 42)
    172.29.131.9.iax > 172.29.122.117.iax: [bad udp cksum f5c0!] UDP, length 14
        0x0000:  45b8 002a 152f 0000 4011 0f23 ac1d 8309  E..*./[email protected]#....
        0x0010:  ac1d 7a75 11d9 11d9 0016 55e1 b35b 8000  ..zu......U..[..
        0x0020:  0000 0005 0000 061e 3600                 ........6.

Notice the bad udp cksum, I wonder why?

Any thoughts?

cynjut · July 31, 2017, 1:40pm

This sure looks like a pure networking problem to me.

On your TCPDUMP, try a lot less options - at this point, you’re trying to analyze the protocol, you’re trying to figure out if the problem is a network route problem or a firewall problem.

Take a look at the output from the following command:

netstat -nr

and check to make sure your network routes and netmasks make sense.

Check your firewall settings all the way through the network - port 4569 is commonly blocked (it’s not a common port so it should be blocked by default almost everywhere).

Log into the console on both PBX servers and run your tcpdump:

tcpdump -I eth0 port 4569

and compare the output.

My suspicion is that your IAX2 packets are getting sent and not received because of a routing problem, but I’d be glad to be wrong.

a5t1 · July 31, 2017, 1:46pm

I was also thinking it could be a networking problem, but both devices (at this time) are whitelisted from the firewall.

Here are the tcpdump results:

from system 1:
09:44:13.480105 IP 172.29.131.9.iax > 172.29.122.117.iax: UDP, length 14
09:44:26.323708 IP 172.29.131.9.iax > 172.29.122.117.iax: UDP, length 14
09:44:26.816017 IP 172.29.131.9.iax > 172.29.122.117.iax: UDP, length 14
09:44:27.106470 IP 172.29.122.117.iax > 172.29.131.9.iax: UDP, length 14
09:44:27.121698 IP 172.29.131.9.iax > 172.29.122.117.iax: UDP, length 12
09:44:28.107713 IP 172.29.122.117.iax > 172.29.131.9.iax: UDP, length 14
09:44:28.119733 IP 172.29.131.9.iax > 172.29.122.117.iax: UDP, length 12
09:44:31.822534 IP 172.29.131.9.iax > 172.29.122.117.iax: UDP, length 14
09:44:44.646798 IP 172.29.131.9.iax > 172.29.122.117.iax: UDP, length 14
09:44:45.146852 IP 172.29.131.9.iax > 172.29.122.117.iax: UDP, length 14
09:44:47.105416 IP 172.29.122.117.iax > 172.29.131.9.iax: UDP, length 14
09:44:47.116870 IP 172.29.131.9.iax > 172.29.122.117.iax: UDP, length 12
09:44:48.106622 IP 172.29.122.117.iax > 172.29.131.9.iax: UDP, length 14
09:44:48.118174 IP 172.29.131.9.iax > 172.29.122.117.iax: UDP, length 12
09:44:50.146169 IP 172.29.131.9.iax > 172.29.122.117.iax: UDP, length 14
09:45:02.979638 IP 172.29.131.9.iax > 172.29.122.117.iax: UDP, length 14
09:45:03.479313 IP 172.29.131.9.iax > 172.29.122.117.iax: UDP, length 14
09:45:07.104660 IP 172.29.122.117.iax > 172.29.131.9.iax: UDP, length 14
09:45:07.115936 IP 172.29.131.9.iax > 172.29.122.117.iax: UDP, length 12
09:45:08.106239 IP 172.29.122.117.iax > 172.29.131.9.iax: UDP, length 14
09:45:08.118080 IP 172.29.131.9.iax > 172.29.122.117.iax: UDP, length 12
09:45:08.478972 IP 172.29.131.9.iax > 172.29.122.117.iax: UDP, length 14
09:45:21.312391 IP 172.29.131.9.iax > 172.29.122.117.iax: UDP, length 14
09:45:21.812456 IP 172.29.131.9.iax > 172.29.122.117.iax: UDP, length 14
09:45:26.815033 IP 172.29.131.9.iax > 172.29.122.117.iax: UDP, length 14
09:45:27.104069 IP 172.29.122.117.iax > 172.29.131.9.iax: UDP, length 14
09:45:27.124537 IP 172.29.131.9.iax > 172.29.122.117.iax: UDP, length 12
09:45:28.105275 IP 172.29.122.117.iax > 172.29.131.9.iax: UDP, length 14
09:45:28.132773 IP 172.29.131.9.iax > 172.29.122.117.iax: UDP, length 12

From System2:

09:44:13.473572 IP 172.29.131.9.iax > 172.29.122.117.iax: UDP, length 14
09:44:26.307349 IP 172.29.131.9.iax > 172.29.122.117.iax: UDP, length 14
09:44:26.806783 IP 172.29.131.9.iax > 172.29.122.117.iax: UDP, length 14
09:44:27.110892 IP 172.29.122.117.iax > 172.29.131.9.iax: UDP, length 14
09:44:27.111389 IP 172.29.131.9.iax > 172.29.122.117.iax: UDP, length 12
09:44:28.112573 IP 172.29.122.117.iax > 172.29.131.9.iax: UDP, length 14
09:44:28.113108 IP 172.29.131.9.iax > 172.29.122.117.iax: UDP, length 12
09:44:31.806288 IP 172.29.131.9.iax > 172.29.122.117.iax: UDP, length 14
09:44:44.640250 IP 172.29.131.9.iax > 172.29.122.117.iax: UDP, length 14
09:44:45.140182 IP 172.29.131.9.iax > 172.29.122.117.iax: UDP, length 14
09:44:47.110107 IP 172.29.122.117.iax > 172.29.131.9.iax: UDP, length 14
09:44:47.110412 IP 172.29.131.9.iax > 172.29.122.117.iax: UDP, length 12
09:44:48.111171 IP 172.29.122.117.iax > 172.29.131.9.iax: UDP, length 14
09:44:48.111749 IP 172.29.131.9.iax > 172.29.122.117.iax: UDP, length 12
09:44:50.139577 IP 172.29.131.9.iax > 172.29.122.117.iax: UDP, length 14
09:45:02.973082 IP 172.29.131.9.iax > 172.29.122.117.iax: UDP, length 14
09:45:03.472604 IP 172.29.131.9.iax > 172.29.122.117.iax: UDP, length 14
09:45:07.109180 IP 172.29.122.117.iax > 172.29.131.9.iax: UDP, length 14
09:45:07.109427 IP 172.29.131.9.iax > 172.29.122.117.iax: UDP, length 12
09:45:08.110591 IP 172.29.122.117.iax > 172.29.131.9.iax: UDP, length 14
09:45:08.111658 IP 172.29.131.9.iax > 172.29.122.117.iax: UDP, length 12
09:45:08.472378 IP 172.29.131.9.iax > 172.29.122.117.iax: UDP, length 14
09:45:21.305883 IP 172.29.131.9.iax > 172.29.122.117.iax: UDP, length 14
09:45:21.805636 IP 172.29.131.9.iax > 172.29.122.117.iax: UDP, length 14
09:45:26.804896 IP 172.29.131.9.iax > 172.29.122.117.iax: UDP, length 14
09:45:27.111229 IP 172.29.122.117.iax > 172.29.131.9.iax: UDP, length 14
09:45:27.111876 IP 172.29.131.9.iax > 172.29.122.117.iax: UDP, length 12
09:45:28.116663 IP 172.29.122.117.iax > 172.29.131.9.iax: UDP, length 14
09:45:28.117197 IP 172.29.131.9.iax > 172.29.122.117.iax: UDP, length 12

BlazeStudios · July 31, 2017, 1:52pm

@a5t1

My reply from the other thread still stands. This is a NAT/network issue, you need to look at your VPN tunnel and how the routing/NAT is setup for it.

a5t1 · July 31, 2017, 1:54pm

I appreciate the response, all ports are open and there is no NATing due to the VPN tunnel.

straight routing…

BlazeStudios · July 31, 2017, 2:00pm

I said NAT/Network, it can be either or a combo of both. You are not getting replies back to requests from one server to another. If there is no NAT then you need to look at your routes and how you are routing those packets.

So both networks on either end of the VPN are all part of the 172.29.131.x and/or 172.39.122.x ranges? Even without the VPN, those are the local IP ranges for both networks?

a5t1 · July 31, 2017, 2:11pm

Yes. The local networks on either end are 131.x and 122.x

The firewall(s) are the default gw for both subnets. Those firewalls are connected via site to site VPN.

172.29.131.0/24 -> 172.29.131.1 (which is the local firewall)

172.29.122.0/24->172.29.122.1 (other local firewall)

Routing tables on both firewalls show the above as well. Allow any any is set on the VPN site to site rules at the firewall level.

I appreciate the help, just not sure where to look from here.

a5t1 · July 31, 2017, 2:27pm

Another clue?

nmap -p 4569 172.29.131.9

Starting Nmap 5.51 ( http://nmap.org ) at 2017-07-31 10:27 EDT
Nmap scan report for 172.29.131.9
Host is up (0.00011s latency).
PORT STATE SERVICE
4569/tcp closed unknown

a5t1 · July 31, 2017, 2:30pm

I’m so confused…

netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name

udp 0 0 0.0.0.0:4569 0.0.0.0:* 2803/asterisk

cynjut · July 31, 2017, 2:44pm

OK - you looked at the network firewalls, but did you look at the integrated firewall? You might need to whitelist the opposite network in the PBXes and you might need to turn on the IAX2 ports in the integrated firewall.

To me, it looks like your systems are trying to connect but are failing - the fact that you are seeing traffic in both directions might be an artifact of both trying to register with the other end. If you are using the Adaptive Firewall, turn off the IAX2 part of that and manage it more precisely from the GUI.

a5t1 · July 31, 2017, 2:53pm

The firewall module is disabled in both appliances.

I’m going to enable it and then configure simply because I’m out of ideas.

a5t1 · July 31, 2017, 3:05pm

Still nothing.

Although the 172.29.122.117 system show this:

netstat -nau
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State

udp 230400 0 0.0.0.0:4569 0.0.0.0:*

a5t1 · July 31, 2017, 3:54pm

omg.

core restart now

on production pbx. problem fixed.

cynjut · July 31, 2017, 3:54pm

This really shouldn’t be this hard.

From the console of one of the machines, traceroute the other computer. Maybe (grasping at straws) there’s a machine in the middle that you’re not taking into account? I don’t know…

I tried something like this with a VPN firewall a few years ago and the VPN had a problem passing “unexpected” UDP traffic - basically, it would pass “the usual” server traffic, but stuff that was going to unusual ports had problems. We had to mess with the VPN configuration to make sure that all remote server port traffic was allowed.

a5t1 · July 31, 2017, 3:57pm

Thanks, I agree.

Simply forcing a restart of the asterisk core fixed the issue. Kept looking in all the wrong places, the network was/is fine. Thanks all for your suggestions. Learned an awful lot about IAX2 trunks.