So a couple hours ago, a server of ours, with about 160 nodes stopped being able to receive calls. This server is virtualized, and has been for a few years, with no related problems.
When calling a DID on the server, the server would answer that the number was no longer available. When I looked at the CDR, the calls showed, but the DID column was blank, as if the Inbound Route didn’t exist or something.
One of the CDR lines says “Congestion” under the App column, with “Application: Congestion(5)” when you hold your mouse over it, and “s [from-sip-external]” in the Destination column. A call a minute or two later was different, but still didn’t go through. It said “Playback” under App and “Playback:ss(no-service)” when you hold your mouse over it.
In a panic, I reset the server. It came up fairly quickly, and the problem resided.
So I come to the community to ask if anyone has experienced anything similar, or if I can get some advice that will help us avoid this in the future.
I clicked on the Unique ID under the System column and got some more info. When you click it on the two failed CDR lines mentioned above, you actually see the DID number under the “exten” column, and what’s interesting is that the “context” column shows from-sip-external.
On calls after the reset, that worked fine, after clicking on the System column, they show the DID as well but the context is from-trunk-sip-“trunk name”.
So, noticed the following in the logs. Looks like it’s been going on at least all day. It stopped after the restart. So whatever it was was likely the cause, not our carrier. Any insight?
> [2017-06-01 14:07:46] WARNING[2345] chan_sip.c: sip_xmit of 0x7f91b84987e0 (len 453) to 192.168.80.101:54607 returned -2: No such file or directory
> [2017-06-01 14:07:46] ERROR[2345] chan_sip.c: Serious Network Trouble; __sip_xmit returns error for pkt data
> [2017-06-01 14:07:46] WARNING[2345] chan_sip.c: sip_xmit of 0x7f91b95714b0 (len 458) to 192.168.80.104:43153 returned -2: No such file or directory
> [2017-06-01 14:07:46] ERROR[2345] chan_sip.c: Serious Network Trouble; __sip_xmit returns error for pkt data
> [2017-06-01 14:07:46] WARNING[2345] chan_sip.c: sip_xmit of 0x7f91b83fd200 (len 450) to 192.168.6.75:44087 returned -2: No such file or directory
> [2017-06-01 14:07:46] ERROR[2345] chan_sip.c: Serious Network Trouble; __sip_xmit returns error for pkt data
> [2017-06-01 14:07:46] WARNING[2345] chan_sip.c: sip_xmit of 0x7f91b9258a70 (len 459) to 192.168.80.104:63845 returned -2: No such file or directory
> [2017-06-01 14:07:46] ERROR[2345] chan_sip.c: Serious Network Trouble; __sip_xmit returns error for pkt data
> [2017-06-01 14:07:46] WARNING[2345] chan_sip.c: sip_xmit of 0x7f91b84987e0 (len 459) to 192.168.70.159:55119 returned -2: No such file or directory
> [2017-06-01 14:07:46] ERROR[2345] chan_sip.c: Serious Network Trouble; __sip_xmit returns error for pkt data
> [2017-06-01 14:07:46] WARNING[2345] chan_sip.c: sip_xmit of 0x7f91b95714b0 (len 454) to 209.222.118.34:63575 returned -2: No such file or directory
> [2017-06-01 14:07:46] ERROR[2345] chan_sip.c: Serious Network Trouble; __sip_xmit returns error for pkt data
> [2017-06-01 14:07:46] WARNING[2345] chan_sip.c: sip_xmit of 0x7f91b83fd200 (len 454) to 10.20.23.45:55719 returned -2: No such file or directory
> [2017-06-01 14:07:46] ERROR[2345] chan_sip.c: Serious Network Trouble; __sip_xmit returns error for pkt data
> [2017-06-01 14:07:46] WARNING[2345] chan_sip.c: sip_xmit of 0x7f91b9258a70 (len 450) to 192.168.6.75:59942 returned -2: No such file or directory
> [2017-06-01 14:07:46] ERROR[2345] chan_sip.c: Serious Network Trouble; __sip_xmit returns error for pkt data
> [2017-06-01 14:07:46] WARNING[2345] chan_sip.c: sip_xmit of 0x7f91b90d3000 (len 461) to 96.81.178.217:25261 returned -2: No such file or directory
> [2017-06-01 14:07:46] ERROR[2345] chan_sip.c: Serious Network Trouble; __sip_xmit returns error for pkt data
> [2017-06-01 14:07:46] WARNING[2345] chan_sip.c: sip_xmit of 0x7f91b933ba30 (len 446) to 192.168.6.75:45185 returned -2: No such file or directory
> [2017-06-01 14:07:46] ERROR[2345] chan_sip.c: Serious Network Trouble; __sip_xmit returns error for pkt data
> [2017-06-01 14:07:46] WARNING[2345] chan_sip.c: sip_xmit of 0x7f91b84987e0 (len 456) to 96.81.178.217:58938 returned -2: No such file or directory
> [2017-06-01 14:07:46] ERROR[2345] chan_sip.c: Serious Network Trouble; __sip_xmit returns error for pkt data
> [2017-06-01 14:07:46] WARNING[2345] chan_sip.c: sip_xmit of 0x7f91b95714b0 (len 456) to 96.81.178.217:10878 returned -2: No such file or directory
> [2017-06-01 14:07:46] ERROR[2345] chan_sip.c: Serious Network Trouble; __sip_xmit returns error for pkt data
> [2017-06-01 14:07:46] WARNING[2345] chan_sip.c: sip_xmit of 0x7f91b83fd200 (len 455) to 192.168.80.104:34814 returned -2: No such file or directory
> [2017-06-01 14:07:46] ERROR[2345] chan_sip.c: Serious Network Trouble; __sip_xmit returns error for pkt data
> [2017-06-01 14:07:46] WARNING[2345] chan_sip.c: sip_xmit of 0x7f91b9258a70 (len 455) to 96.81.178.217:24985 returned -2: No such file or directory
> [2017-06-01 14:07:46] ERROR[2345] chan_sip.c: Serious Network Trouble; __sip_xmit returns error for pkt data
> [2017-06-01 14:07:46] WARNING[2345] chan_sip.c: sip_xmit of 0x7f91b84987e0 (len 452) to 96.81.178.217:5035 returned -2: No such file or directory
> [2017-06-01 14:07:46] ERROR[2345] chan_sip.c: Serious Network Trouble; __sip_xmit returns error for pkt data
> [2017-06-01 14:07:46] WARNING[2345] chan_sip.c: sip_xmit of 0x7f91b95714b0 (len 453) to 192.168.80.101:64005 returned -2: No such file or directory
> [2017-06-01 14:07:46] ERROR[2345] chan_sip.c: Serious Network Trouble; __sip_xmit returns error for pkt data
> [2017-06-01 14:07:46] WARNING[2345] chan_sip.c: sip_xmit of 0x7f91b83fd200 (len 459) to 192.168.80.104:36828 returned -2: No such file or directory
> [2017-06-01 14:07:46] ERROR[2345] chan_sip.c: Serious Network Trouble; __sip_xmit returns error for pkt data
> [2017-06-01 14:07:46] WARNING[2345] chan_sip.c: sip_xmit of 0x7f91b9258a70 (len 457) to 192.168.80.107:46665 returned -2: No such file or directory
> [2017-06-01 14:07:46] ERROR[2345] chan_sip.c: Serious Network Trouble; __sip_xmit returns error for pkt data
> [2017-06-01 14:07:46] WARNING[2345] chan_sip.c: sip_xmit of 0x7f91b84987e0 (len 456) to 96.81.178.217:17744 returned -2: No such file or directory
> [2017-06-01 14:07:46] ERROR[2345] chan_sip.c: Serious Network Trouble; __sip_xmit returns error for pkt data
> [2017-06-01 14:07:46] WARNING[2345] chan_sip.c: sip_xmit of 0x7f91b95714b0 (len 454) to 192.168.80.115:53288 returned -2: No such file or directory
> [2017-06-01 14:07:46] ERROR[2345] chan_sip.c: Serious Network Trouble; __sip_xmit returns error for pkt data
> [2017-06-01 14:07:46] WARNING[2345] chan_sip.c: sip_xmit of 0x7f91b83fd200 (len 450) to 192.168.6.75:49725 returned -2: No such file or directory
> [2017-06-01 14:07:46] ERROR[2345] chan_sip.c: Serious Network Trouble; __sip_xmit returns error for pkt data
> [2017-06-01 14:07:46] WARNING[2345] chan_sip.c: sip_xmit of 0x7f91b9258a70 (len 454) to 192.168.80.104:59682 returned -2: No such file or directory
> [2017-06-01 14:07:46] ERROR[2345] chan_sip.c: Serious Network Trouble; __sip_xmit returns error for pkt data
Do you think that perhaps we were getting attacked? I looked through the “full” logs that only go back a week or so. They all had similar errors. So if it was an attack, it had been going on for some time, and apparently the FreePBX Firewall didn’t stop the attack.
Thought it would be worth noting that the IP’s in the logs are either private or public IP’s that we’re familiar with.
Since the restart two days ago, we’re still getting the errors/warning, but far less. They’re now only coming form one IP, and it seems to only be this one:
[2017-06-03 16:48:40] WARNING[24006] chan_sip.c: sip_xmit of 0x7f7b48098e80 (len 450) to 96.81.XXX.XXX:16362 returned -2: No such file or directory
[2017-06-03 16:48:40] ERROR[24006] chan_sip.c: Serious Network Trouble; __sip_xmit returns error for pkt data
With a few “Interrupted system call” ones here and there.
These messages are appearing, again, for the same IP, every 6-7 seconds. No one is using the Web RTC phone at that location.
So as an update, I disabled TCP in Asterisk SIP Settings and the errors above stopped. We do use an obscure TCP port for most of our NAT’d endpoints, but not on this server, so I was able to disable it. We have other servers with this setting enabled though, with no problems.
I read somewhere that since the GUI added the “yes, no” option for Enable TCP, instead of having to add it manually with tcpenable=yes, there may have been a double entry of the TCP enable, thus causing the error. Still to be proven though.
I received the same errors after Public IP change.
I also had registration problems.
Tried disabling firewall and setting up the new public IP and verify it is configured in all the places it should (I am also using an HA topology with a floating IP address…)
What solved it for me, was I re-checked and fixed (and verified i am using the correct) External IP under the “Asterisk SIP settings” as well as the correct internal networks.
That fixed the “chan_sip.c: Serious Network Trouble; __sip_xmit returns error for pkt data” error for me . And afterwards my extensions were able to register successfully.
Not sure it is that relevant to your issue for 100% but hopefully it will help someone someday
Thanks for the input. We actually have the correct info there.
I think we’re going to switch our endpoints all to chan_pjsip, as it is supported by Digium, and all the errors we’re getting are chan_sip related. We will keep our SIP Trunks chan_sip though.
I chanced upon your response while looking for source of sip_xmit errors and having all extensions go unavailable.
We too have tcp and udp enabled, but have been able to spot some of the errors to be logged when some of the extensions (remote users on mobile data) tend to turn “unreachable”. We have been able to confirm the same by force unregistering such extensions and the errors instantly stop. However, there seems to be multiple sources for these errors. The more peculiar and worrying problem is when these errors get logged in hoards and reach a precipitation point when all extensions even though shown as connected with pingable IP, start returning unavailable and calls start going to voicemail or the queue calls remain on queue and don’t get passed onto the agents, as it gets an unavailable from them. I also read another post that indicated that there’s a chance that some ast db corruption can cause all extensions to seem to have DND turned on and that gets corrected post a server restart (which seems to be the case with us - a restart gets all exts back online) but I am unable to ascertain if disabling DND module could be a workaround to solve this problem nor am I able to find the reason for this ast db corruption which seems to be happening quite frequently (weekly once) at exactly one client site.
My question however is - were you able to solve it by moving all end points to pjsip? or any other finding that helped you debug this will be much appreciated since what you’d faced seems to come closest to what we have at our hands too. Thanks you.
There were some tricks with trunk based extensions we had setup, for FXO adapters for example, so we kept those on a separate server that still runs chan_sip.
We had disabled chan_pjsip so far. We are using only chan_sip so far - for both extensions and trunks, and both are on one micro-server box. Will try to convert all extensions to pjsip first and give it a try. What I am also thinking is - if it is a driver that is causing the difference between pjsip and chan_sip, then perhaps it is not related to network outside the box - but something in our setup or the chan_sip driver. Guess will need to get into the chan_sip code to decipher.
Btw, just a note - we noticed some errors being reported by Avahi Daemon too. Disabling it was harmless in our usecase. That fixed the Avahi originated errors but did not impact the sip_xmit errors.
Sounds good. Just so you know, we dug deep for a long time into logs and ran backtraces. It always led to something about chan_sip and TCP, but nothing conclusive. We had tickets in with Asterisk, but they closed them, saying they no longer supported chan_sip, and suggested we moved to pjsip. I suggest you don’t wast your time "decipher"ing chan_sip, but just do the move.
