Hi to all,
I have this annoying issue coming after 4 years of no problems and without changes to my environment. We hear on incoming external calls something like a CLOCK randomly (so not everytime) and periodically(so when it happens we hear the CLOCK every some seconds ). The issue is only on our side, so on the side of the caller the sound is perfect. After investigeting that there are no issue on the local phones sides, I made two tcp dumps, the first one on the freepbx console, and you can clearly view the interfacence in the picture below that is coming from the italian provider EHIWEB to the local freepbx
and I made another dump in the router console too(this is a pfsense box with a lot of resources) and I can see again the noise coming directly from their IP to our like this
at this point I was sure that the issue is coming from the providers side, and after sending them everything I have they told me that a second level IT replied in this way:
"From the analysis of the audio streams it is clear that it is a broacast msg on the customer LAN network, since it is present even before the conversation is established, on the carrier side this anomaly is not present in any other customer "
so I checked again and again the cap that I’m attaching( the router’ s side one router_cap.tgz (1,7 MB)
) but I cannot find any broadcast message (I searched on the wireshark wiki for the appropriate filter eth.dst == FF:FF:FF:FF:FF and other tools) without results, so I cannot understand where they found this message if we are looking at the same cap.
Any help or starting point will be very usefull for me… many thanks
Your provider is sending you corrupted data. I don’t see any way that this could be a networking problem on your end. The 16 bytes of hex 2a (maximum negative G.711a value) inserted into packet 1290 were almost certainly in the packet the provider sent.
The telco saying: "From the analysis of the audio streams it is clear that it is a broacast msg on the customer LAN network, since it is present even before the conversation is established, on the carrier side this anomaly is not present in any other customer "
makes no sense - I don’t see how other network traffic on your LAN would cause these sound “pops”. And looking at the packet capture doesn’t look like any packet loss, etc. happening. And the packet capture is from the internet side of your router, not the LAN side - correct? LAN broadcast messages would not be seen on the internet side of your router.
And I’m assuming you don’t have this happen on internal calls only to your FPBX server that never go to the telco provider correct?
yes internal calls even beetween extensions that are placed on remote departments over VPN are not affetcted by this issue.
So I’m pretty sure that this is an issue from the provider, but why he is reporting “since it is present even before the conversation is established” ??
I was looking exactly to this packet, but I cannot understand why he is telling to me that the issue is present even before the conversation is established.
my question is… what kind of network condition can give this disturb? If I’m for any reason having something that is taking my bandwith randomly, like a bot that is trying some sort of hack to my public IP, could this give to me a disturb of this type? I am trying to keep some logs on the wan interface of my router and deep analyse the throughput, but doesn’t seems to be anything strange, also because I have a symmetrical 1GB connectivity provided by optical fiber and this is very difficoult to saturate(the freepbx box is not even exposed), but due to the randomness of this issue I also have hard times to identify something like this.
I would think only packet loss or high latency would be noticable which don’t appear to be present here. That packet capture shows no packet loss.
I expect is something on the telco’s end, maybe they are having issues with their systems and don’t know it or not admitting it. LoL, I always get suspicious when a tech says something is not seen by any one else.
I agree, maybe the only fix is to wait until they receive more tickets with the same issue… the problem is that I’m in this situation from a mounth and it is really difficoult to explain this to my boss. anyway thank you
If you don’t have multiple telco providers, can you temporarily add a cheap one and test to show does not (hopefully) happen with that provider to help prove it is the telco that is having the issue, not you.
IMO, it’s very clear at this point that you are receiving corrupted RTP packets from the provider. There is no plausible problem at your end or caused by an attacker that would result in the symptoms you observe. Based on the pattern (10 bytes of 0xff followed by 12 or 16 bytes of 0x2a replacing the correct audio samples) I suspect a hardware issue with their media server.
This failing call has signaling at 83.211.227.21 (voip.eutelia.it) but media at 83.211.227.32. I assume that they have multiple media servers and if incoming calls use several different media addresses but the noisy ones all use 83.211.227.32 that should make it clear to them that this particular server is malfunctioning.
IMO, Hanlon’s razor applies in this case.
Speak with a supervisor or manager – this trouble may have been reported by others but handled by different agents. I can think of several reasons why other customers may not have experienced the problem. One is that they offer G.729 as the first preference codec. That seems really bizarre – quality is poor and very few would need the bandwidth savings. I suppose you could try that to see whether it makes a difference.
As @nielsen says, testing with another trunk is a good idea. Unfortunately, your problem is only on incoming and it’s not reasonable to ask customers to call a different number. However, I assume that the Vodafone mobile used for your test call is one of your own, so you can replicate the problem at will (though it may take several calls).
Two providers that I’m familiar with that I’d re commend for this test are AnveoDirect (US based) and Voxbeam (UK based). Both are Voxbone resellers and don’t proxy media; i.e. your audio will come directly from Voxbone. Both offer Ancona numbers. The experiment should cost you no more than $15 and other than a few hour’s delay while they verify your first payment manually, everything is automated so you could be testing tomorrow.
In this particular freepbx box, there are 4 trunks pjsip, each one is for a different geographically located branch offiice. The other offices reaches the freepbx box through a dedicated VPN. I never get the issue with the other numbers, only with the one that I’m reporting in the cap 07120723** that is in the same physical LAN with freepbx but… this one receives a massive amount of calls during the day so, due the randomness of the issue is easy to replicate this disturb. With the other numbers I tried to replicate the issue by myself around 50 times per number with no problems,
but I think that can be so many reasons for this to happens, that maybe even the fact that I tested this in the same way for 50 times cannot be safe to say that the other numbers are not affected.
In my opinion the issue is present only with this particular number, or better to say the media servers related to this number.
just to confirm this I tried some tcpdumps with the branch offices numbers and the media server has different IP like in the picture below(in this case is 83.211.227.16 )
