Both way audio loss after a few mins then disconnecting channel due to no RTP traffic (call drop)

No router/firewall in front of this PBX.

Remote extensions registered via TCP using port 42860 (no more unreachable/unregistering issues since switching to TCP), connecting via DDNS.

For most calls coming from remote extensions to the PBX (no matter if they’re calling another extension, or to voicemail, or listening to music while on park), there is an issue with loss of audio after usually 2-3 mins (rarely it comes back but when it does it fades again and then call drops) and then the call drops. Once again, audio loss then call drop also occurs on calls from remote extension>PBX utility (such as echo test, park, VM, etc.)
This is also occurring on some calls from local PBX network>SIP trunk. But it’s not a SIP trunk issue since it’s the same issue with remote extensions calling into PBX…
Phones are Cisco, some softphone apps, Grandstream etc. all the same issue. so I highly suspect it’s something with PBX.
Whenever the call drops the PBX log has this (of course with the extension of the call that got dropped in place of ‘123’)

Disconnecting channel 'PJSIP/123-00000056' for lack of audio RTP activity in 30 seconds

However the call doesn’t drop at regular times (it usually doesn’t drop less than 2 mins in); sometimes it is at 2:14 sometimes at 3:02, etc. So it’s not happening at exactly 30 seconds like some other very common issues related to local network address/RTP ports forward…
RTP ports on PBX set to 10001-20000.
RTP Checksums: Yes.
Strict RTP: Yes.
RTP Timeout: 30.
RTP hold timeout: 300.
RTP Keep Alve: 1.
STUN: makes no difference if using a STUN server or not, for this issue for me.
Codecs: using only ulaw and g729

I noticed that all the calls are dropping between 119-180 seconds, is also happening for calls from remote extensions even if there is no NAT on the remote extensions’ side (i.e. phone directly connected to modem or SIM card and no router/NAT device in between). It occasionally happens to calls from the PBX network to SIP trunk. which leads me to believe it is something with PBX config please suggest

I don’t think you have solved the original problem. I think it is probably the same problem that is causing your audio symptom.

However one thing I find very strange is the suggestion that the audio fades out. That can pretty much only happen in baseband audio processing, before the call enters a VoIP environment, or after it leaves it. With end to end SIP, that would only be in the the phone itself. Maybe you are using “fading out” loosely?

By fading in&out I mean this:

both ways audio fully present for ~1.5mins of the call as normal
then no audio at all for ~13 secs
audio back for ~14 secs
no audio and it does not return
call drops

Ever since phones connected thru TCP there have not been any registration problems. Qualify enabled and not ever getting “unreachable”/contact deleted/unregistering… anymore.
This audio problem present over TCP+UDP+TLS. And now there’s no firewall/router in front of the PBX.

The TCP error recovery is probably hiding the underlying problem, because it will probably retry for longer than SIP itself would, but the media is still on UDP, so is still vulnerable to your underlying problem.

I checked rtp debug and what happens is that first here is normal SENT/GOT rtp traffic, but then it only changes to SENT (PBX is not receiving RTP back from the phone/trunk) - what could be cause of this

Network congested. Dynamic firewall or NAT rule timed out in router.

I did “rtp show settings” and it came back with this:

General Settings:
----------------
Port start:      10002
Port end:        20000
Checksums:       Yes
DTMF Timeout:    1200
Strict RTP:      Yes
Probation:       4 frames
Replay Protect:  Yes
ICE support:     Yes
STUN address:    0.0.0.0:0

Why does it show Port Start as 10002 when my start port is configured as 10001 in Asterisk SIP settings?

I changed my RTP start port in Asterisk SIP settings to match 10002 to be consistent with rtp show settings (now it’s properly both configured and showing as 10002)
‘rtp show settings’ showed 10002 as the start port even when it was 10001 because RTP starts on an even number always. I am going to monitor the call quality to see if there are more drops.

Because ports are alternated between RTP and RTCP, with the RTP having even port numbers. It doesn’t make sense to start with an add number, as you can’t actually use it.

This sometimes shows up when no more RTP is coming from one side. What does this mean:

56004	[2022-09-09 19:34:40] DEBUG[11450][C-000000ad] res_rtp_asterisk.c: (0x7fd2e40e17f0) RTP NAT: Got audio from other end. Now sending to address 79.136.104.55:9427	
56005	[2022-09-09 19:34:40] DEBUG[11450][C-000000ad] res_rtp_asterisk.c: (0x7fd2e40e17f0) RTCP NAT: Got RTCP from other end. Now sending to address 79.136.104.55:9417

It says that it received audio, etc, from one of those addresses on the local media port for that SIP session, and because you have nat=comedia or rtp_symmetric=yes set, Asterisk believe is the true address of the peer for media, and it overrides that in the INVITE.

These options are intended to cope with the peer being behind NAT and sending a local media address, but normally they work anyway, because most SIP implementations use the same address to both send and receive RTP.

You might expect something like this, normally, at the start of a call, if you have one of these options set, but if it happens mid-call, something is not behaving nicely.

In the worst case, if you are getting this unexpectedly, it because the IP address of the peer has been changed by its provider (e.g. to discourage servers), or because a router is allocating a dynamic port number and has decided that it is dealing with a new UDP stream and has re-allocated the port number.

I have narrowed down the issue that the issue is that the UDP packets are being dropped after some time (and since RTP is over UDP then the audio stops and then the call drops), at this specific office location. Prior, remote phones connected via UDP were always experiencing unreachable/reachable and unregistering and as soon as they were switched to use TCP for SIP signaling, then all those issues disappeared; so all the problems at this location are related to UDP packet loss.

The cause is not FreePBX configuration since when I moved the PBX to another location and completed testing there, there are no issues at all. But all voip calls at this one office where the problem occurs are having RTP (UDP) packet loss NO MATTER WHAT the PBX/provider is; i.e. I’ve also tested with phones directly connected to RingCentral, VoipMS, Vonage, etc. and the issue persists across all services when the phone is connected to this location’s internet.
It’s the ISP that is probably the cause of this since the firewall is properly configured and running well, so I’ll be investigating their modem as there is probably some signal loss/interference/etc. in the cable line.
Thanks everyone for your help in narrowing this down.