Power outage now my SIP trunk won't connect


(Froglevelmc) #1

Asterisk 13.38.2 built by mockbuild @ jenkins7 on a x86_64 running Linux on 2021-04-07 06:49:36 UTC on a Sangoma40

Incoming and outgoing calls just with drop no rings.

  • I have a dedicated fiber line for SIP with a SIP trunk passing through and SBC.
  • We had a power outage and now I can’t get the asterisk to register with the SIP Trunk.
  • I can’t even ping the SBC now. (PBX =192.168.1.3) (SBC = 192.168.1.2)
  • All my subnets have been whitelisted in the firewall and IPS.
  • I have contacted the ISP and their engineers have so far not found anything wrong with the SBC
  • They say they can see my PBX but I can’t see the SBC…
  • I don’t see anything in the full logs that stands out, but I have only moderate asterisk experience…
    Any ideas, I’m trying to rule out everything on my end before I call them back and tell them their SBC is fried?

#2

Setups with an ISP-supplied SBC are generally statically configured and do not use registration or authentication, so lack of registration is probably normal.

Confirm that you are pinging from a shell prompt on the PBX.

Confirm that the SBC is supposed to respond to ping (possibly, its internal firewall is blocking ICMP).

If an attempted ping results in an error message (not merely no reply), post that.

Confirm that ifconfig shows an interface with IPv4 address 192.168.1.3 and an appropriate netmask.

Capture traffic with tcpdump and see whether your attempted pings are being sent out from the interface and whether there is any reply. (The tcpdump capture is after iptables for sent packets, and before iptables for received packets.)

If tcpdump shows ICMP echo (ping) being sent and nothing received, check for a switch or other networking hardware between PBX and SBC. If so, try running a cable directly. Note that depending on the interfaces, you may need to use a crossover cable.

If still no luck, get a laptop and configure its Ethernet interface for static IP 192.168.1.3 and netmask 255.255.255.0. Connect it directly to the SBC with a known good cable. If you still can’t ping the SBC, it is probably indeed fried. However, this conflicts with their claim that they can ‘see’ your PBX.


(Froglevelmc) #4

You are correct. It is statically configured, with the only info required being type=peer and host=SBC IP But it used to show the trunk info under “sip show registry” Now not so much…

[root@freepbx ~]# asterisk -rx “sip show registry”
Host dnsmgr Username Refresh State Reg.Time
0 SIP registrations.

The weird part is that the “PBX Statistics” on the Dashboard says there is 1 SIP trunk online.
https://seagateit-my.sharepoint.com/:i:/g/personal/chrishobgood_portcitydata_com/Eb7sGnI9irxDnlWtJlV2BTEBB5Hv1tG9_wUQAj5ZblXJ9Q?e=dDboL4

Yes I did from the from a putty session on the PBX and over VPN from the LAN

They suggested that it should not be be blocking ICMP. Although, he didn’t sound real sure.

[root@freepbx ~]# ping 192.168.1.2
From 192.168.1.3 icmp_seq=1 Destination Host Unreachable

Also tried it this way:
[root@freepbx ~]# ping -I eth0 192.168.1.2
PING 192.168.1.2 (192.168.1.2) from 192.168.1.3 eth0: 56(84) bytes of data.
^C
— 192.168.1.2 ping statistics —
110 packets transmitted, 0 received, 100% packet loss, time 109007ms

[root@freepbx ~]# ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.1.3 netmask 255.255.255.0 broadcast 192.168.1.255
inet6 fe80::2e2:69ff:fe18:7ed7 prefixlen 64 scopeid 0x20
ether 00:e2:69:18:7e:d7 txqueuelen 1000 (Ethernet)
RX packets 98206 bytes 20882670 (19.9 MiB)
RX errors 0 dropped 5807 overruns 0 frame 0
TX packets 65037 bytes 49882579 (47.5 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device interrupt 16 memory 0xd0900000-d0920000

I don’t think I have ever had to run a ping simultaneously with a tcpdump. How exactly is that done run a ping then pipe it into a tcpdump? Run multiple session? Anyway I have a SharkTap ethernet sniffer. I’ll run Wireshark on it tomorrow and will post back results.

Will be on site tomorrow to test this.

The link light on the SBC shows link. but that doesn’t necessarily mean its passing traffic. Also I am off site right now and I just set up a softphone so I could see what they were getting when dialing out. I get no rings only a message that says “all circuits are busy now please try your call again later” then hangs up.


#5

The Host Unreachable is an ICMP error sent from the SBC, presumably because it blocked the ping request. You could confirm this with tcpdump. But in any case, it shows that the connectivity is ok.

If you got the message within a second or two, the SBC rejected the call. There may be a clue in the Asterisk log showing a ‘hangup cause’. However, you can get all the details:
At the Asterisk command prompt (not a shell prompt), type
sip set debug on
You should see
SIP Debugging Enabled
Make a test call and the Asterisk log will now include a SIP trace. If you can’t figure out what is wrong, paste the complete log for the call at pastebin.freepbx.org and post the link here.

OTOH, if you hear ‘all circuits are busy’ after a long delay, then something (most likely FreePBX firewall) is blocking the request or the reply, and Asterisk gave up after many retries. You can run sngrep to see the SIP traffic on the network side of the firewall.


(David55) #6

Sending host unreachable in that context is a bit naughty, although I suppose it could be done to confuse a hacker. Normally it would be port unreachable, or administratively prohibited, for a firewall rejection on a destination machine.

An alternative mechanism for host unreachable, given the destination is on the same sub-net, would be that the SBC failed to respond to an address resolution protocol (ARP) query, which would indicate it had either forgotten its IP address, or the hardware was broken on that interface. The LInux CLI “arp -a” command can be used to see if it recently responded to an ARP request.

I’m not sure if the following will actually work, but you could try pinging 255.255.255.255, to see if anything responds from an unexpected address.


(Froglevelmc) #7

https://pastebin.freepbx.org/view/5d7b9707


(Froglevelmc) #8

Yeah it’s pretty much immediate.


(Froglevelmc) #9

? (192.168.1.2) at < incomplete > on eth0

Nothing unexpected…


(David55) #10

In that case, the SBC either doesn’t think its address is 192.168.1.2, or it isn’t physically connected to the network that it shares with Asterisk. E.g. this is what I get when pinging a non-existent address on my LAN, and then immediately running ARP. Again it says incomplete.

ping 192.168.1.66
PING 192.168.1.66 (192.168.1.66) 56(84) bytes of data.
^C
--- 192.168.1.66 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 41ms

root@dhcppc4:~# arp -a
? (192.168.1.66) at <incomplete> on enp3s0
_gateway (192.168.1.1) at XX:XX:XX:XX:XX:XX [ether] on enp3s0

#11

Sorry, I missed that one. @david55 is correct, and the log posted shows that the call was rejected because no trunks were available so an INVITE to the SBC was not even attempted. I assume that you have qualify set for the trunk and the OPTIONS requests had failed so the trunk was marked offline.

Something you can try quickly is capturing traffic with sngrep or tcpdump and seeing whether anything is sent by the SBC, either periodically or on an attempted incoming call.


(Froglevelmc) #12

It’s not I get no traffic at on an incoming call…


(Dave Burgess) #13

From your description, it sounds like your equipment is all local.

If that’s the case, I’d start with the obvious stuff, like are all of the link lights working. I had the power fail on one of my local PBX servers and it wiped out the ethernet card. In fact, over the years, the number 1 hardware problem I’ve had has been network cards failing.


(Froglevelmc) #14

Yeah that’s where I am on this too. The PBX is working fine with no errors that I have found.
Everything I do keeps leading me back to the SBC…


(Dave Burgess) #15

Log into the console on the PBX. Use the commands
“tcpdump -i etho” and let that run for a couple of minutes. You should see traffic from other hosts.

Next, try
“tcpdump -i eth0 host sbc.local.addr” where the address of the sbc is the sbc.local.addr.

If you get nothing from that machine, but get traffic from other hosts on your network, it’s likely to be the SBC or the switch the SBC is plugged into. If you are getting traffic from the SBC, it’s something software, like a bad routing entry or a bad firewall entry.

Remember, tcpdump sees traffic on the promiscuous port on the machine, so it can see traffic that the application is stopping.


(Froglevelmc) #16

[root@freepbx ~]# tcpdump -vv -i eth0 host 192.168.1.2
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes

That ran for about 5 min. Nothing…


(Froglevelmc) #17

I was getting some weird pcap entries because I remoted into the PBX using putty. I hooked a monitor and keyboard up to the PBX and ran the above command again. Now I could see packets coming in. Here is the .pcap file.
https://seagateit-my.sharepoint.com/:u:/g/personal/chrishobgood_portcitydata_com/EWX8LSyqwy9FlgRahL6avmMBdwKRkAhc--IK81Mv8scSHQ?e=nlIugZ

Summary of the pcap this was the only traffic coming in when dialing in from an outside line.
15 2021-07-01 12:48:59.700457 InnoMedi_0a:aa:c8 Broadcast ARP 64 Who has 192.168.1.3? Tell 192.168.1.2


(David55) #18

That suggests that 192.168.1.2 is able to transmit. Is an InnoMedia card, with MAC ending in 0a:aa:c8 plausible for the SBC. Maybe it’s ethernet receiver, or receive path, is broken, and it is transmitting blind.


#19

Assuming that the pcap was unfiltered, then the PBX should have replied to those ARP requests, but didn’t. I don’t know enough about the OS internals to know why that might be.

From a root shell prompt, issue
arp -i eth0 -s 192.168.1.2 00:10:99:0a:aa:c8
and then try to ping 192.168.1.2


(Froglevelmc) #20

Thanks everyone for all the input. I’d still be scratching my head without it. I got / we got it resolved. It was the switch after all; a Dell PowerEdge 3548P… It used to be set up with a Voice VLAN. This site only has 8 phones. So I deleted the VLAN to simplify things and put everything on the native LAN. Oddly it seem the power outage brought some of the setting back but not all of them. So when I was looking in the switch I didn’t see 2 ports were once again tagged as VLAN100. So I had to reset the switch to factory to get them to clear out. All is well now. I can even ping the SBC now. Thanks again for all the help.


#21

Sounds like someone forgot to save the switch config after the last set of changes.

But I’m glad that you got it working.